An Empirical Study on Self-correcting Large Language Models for Data Science Code Generation

LLMs such as Codex, LaMDA, and GPT-4 have pushed code generation forward by translating natural-language specifications into working programs. Yet software engineering remains full of failures that are not simple syntax mistakes: API misuse, incorrect assumptions about outputs, hidden library behavior, and fragile data-processing pipelines. That is why Automated Program Repair (APR) has long sought automated fault localization and patch synthesis. Self-correcting LLM frameworks such as Self-Refine, Reflexion, and SelfEvolve moved the field toward post hoc repair, but they often lacked external knowledge, execution-grounded reasoning, or a strong focus on realistic data-science code. This paper positions itself exactly at that intersection.

The central problem is how to make an LLM generate data-science code that is not only syntactically valid but also executable and test-passing, especially under the failure modes common in NumPy/Pandas workflows. The challenge is hard because traceback messages are informative but incomplete: they indicate where execution broke, not necessarily the root cause or the exact fix. Purely internal prompting can repeat the same mistake, while naive regeneration wastes tokens and often fails on subtle API and assertion errors.

• �� External knowledge retrieval: The model queries StackOverflow and uses real developer discussions as context. This is needed because many debugging patterns already exist in human forums, and those patterns often encode the missing “how-to-fix” knowledge.\n• Dual Auto-CoT generators: One generator constructs an initial chain-of-thought prompt from the problem statement and retrieved documents; the second reshapes failure feedback into a repair prompt. This separates problem understanding from error correction.\n• Three-stage loop: generation, syntax/execution verification, and iterative refinement. The design is explicitly feedback-driven and focuses on API errors and assertion failures, which are common in data-science code.\n• Practical orientation: Unlike methods that stop at a single improved draft, this pipeline is designed for repeated execution in real engineering settings, making it relevant to continuous integration and notebook debugging.

• �� Input acquisition: Start from the problem description p_d.\n• Retrieval: Call external_knowledge_query(p_d) to obtain relevant StackOverflow documents doc.\n• Initial reasoning: auto_cot_1(p_d, doc) creates a CoT prompt that injects human debugging patterns into the model’s reasoning.\n• Code generation: The code_generator produces an initial candidate solution.\n• Syntax filtering: A syntax checker quickly removes malformed code before expensive execution.\n• Execution-based evaluation: The code_executor runs the candidate against unit tests in a Python environment and returns traceback/error feedback.\n• Feedback reformulation: If failures occur, auto_cot_2(p_d, f) converts the error signal f into a new repair-oriented prompt.\n• Iteration: The revised prompt is sent back to the code_generator, and the loop repeats until the code passes or the attempt budget n is exhausted.\n• Key design choice: feedback is not merely appended; it is structured into a reasoning chain so the model can infer root causes and repair steps instead of blindly patching surface symptoms.

The evaluation centers on DS-1000, a dataset for data-science code generation, with emphasis on Python libraries such as NumPy and Pandas. The external knowledge base is built from 558,402 StackOverflow posts and 972,513 related comments. The paper compares CoT-SelfEvolve against existing models across multiple LLM settings and studies three questions: overall performance, the effect of the Auto-CoT prompt generator, and how performance changes as the number of attempts increases. The protocol measures both the initial generation quality and the final outcome after iterative repair.

The headline result is that CoT-SelfEvolve performs substantially better than prior approaches on DS-1000, with the strongest gains on difficult, error-prone data-science tasks. The paper reports a clear positive trend: each additional repair iteration improves accuracy. A second result is that Auto-CoT materially improves initial generation quality by encoding StackOverflow-derived reasoning patterns. A third result is that syntax and executor feedback can be turned into actionable repair guidance, especially for API errors and assertion failures.

The framework is directly relevant to data-science assistants, notebook repair tools, and pre-merge validation in CI/CD pipelines. A team working with Pandas or NumPy could use it to catch common mistakes before they reach production, while educators could use it to explain debugging steps rather than just final answers. Its prerequisites are straightforward: an execution environment, unit tests or checks, and access to a retrieval source such as StackOverflow.

The approach is only as good as its retrieval and test signals. If StackOverflow lacks a close match, the prompt chain may be weak; if unit tests are incomplete, the model may overfit the tests rather than the underlying intent. It also incurs extra compute and token cost because it iterates through generation and execution multiple times. The current evidence is strongest for Python data-science code, so broader generalization remains to be shown.