AutoCodeRover: Autonomous Program Improvement
AutoCodeRover combines AST search, LLM agents, and SBFL to solve 19% of SWE-bench-lite at $0.43 per issue.
Key Findings
Methodology
AutoCodeRover treats a software repository as structured program artifacts rather than a bag of files. Given an issue, an LLM first extracts candidate keywords from the natural-language report, then iteratively invokes AST-backed retrieval APIs such as search_class, search_method_in_class, and search_code_in_file to collect class signatures, method bodies, and local context. If tests are available, spectrum-based fault localization (SBFL) provides suspiciousness signals to prioritize methods and classes. A separate LLM agent then generates a patch from the accumulated context and the inferred buggy location, followed by test-based validation and retry if needed.
Key Results
- On SWE-bench-lite, a benchmark of 300 real GitHub issues from 11 Python projects, AutoCodeRover solved 19% of the tasks, i.e., about 57 issues with pass@1. The paper states this is higher than the recently reported SWE-agent efficacy.
- The average completion time was about 4 minutes per issue, compared with 2.68 days on average for human developers, suggesting the system can operate within a practical maintenance window.
- The average cost was only $0.43 USD per task. The authors also report that roughly two-thirds of the produced patches were correct and acceptable, an engineering-quality metric not emphasized in Devin or SWE-agent reports.
Significance
The paper moves LLMs from code completion toward software maintenance and evolution. Its importance lies in addressing a real pain point: fixing issues in mature repositories requires root-cause analysis, cross-file reasoning, and test-aware validation, not just token-level code generation. For academia, AutoCodeRover provides a software-engineering-oriented agentic framework; for industry, it suggests a low-cost workflow in which structured retrieval narrows the search space before patch synthesis, reducing manual triage and debugging effort.
Technical Contribution
The key technical contribution is to make code search the centerpiece of LLM reasoning. Rather than treating the repository as text, AutoCodeRover searches the AST at the level of classes and methods, supports iterative retrieval, and combines issue text, local code context, and SBFL signals into a debugging-oriented loop. Unlike much APR work that assumes buggy locations are known, AutoCodeRover first localizes the fault and only then generates the fix, making it substantially closer to real maintenance workflows.
Novelty
Its novelty is system-level, not merely prompt-level. AutoCodeRover is not just a patch generator; it is an end-to-end autonomous program-improvement workflow. Compared with general-purpose agents that browse files, it explicitly leverages AST structure and test-guided localization to infer specification from code. The Django-13933 example illustrates this distinction well: iterative retrieval moves from ModelChoiceField to ModelMultipleChoiceField and then to to_python, showing how structure-aware search sharpens root-cause understanding.
Limitations
- The approach still depends on the LLM extracting useful keywords and converging through iterative search; vague issues, inconsistent naming, or highly distributed logic can derail retrieval and miss the true fault location.
- SBFL only helps when a test suite exists, so projects with weak or absent tests receive less benefit. The reported ~4-minute average is practical for benchmark issues, but runtime may increase in larger or more complex repositories.
- Evaluation is centered on 300 Python issues in SWE-bench-lite, so cross-language robustness and scalability to very large multi-module systems remain open questions.
Future Work
The authors position this work as a starting point for autonomous software engineering. Natural next steps include stronger structured retrieval, richer debugging support, incremental testing, and multi-round patch refinement. The community also needs broader evaluation across languages, repository sizes, and maintenance tasks, plus closed-loop pipelines where LLM-generated code is automatically improved before human review.
AI Executive Summary
AutoCodeRover asks a practical question that current coding assistants often dodge: can an LLM not only write code, but actually maintain software? The paper’s answer is to split repair into two stages—first find the right place to change, then generate the change. Instead of viewing a repository as a heap of files, AutoCodeRover treats it as program structure organized by classes, methods, and abstract syntax trees (ASTs). That choice matters because real GitHub issues usually require root-cause analysis across multiple files, not isolated text generation.
The system runs as a two-agent workflow. In the context-retrieval stage, an LLM extracts candidate keywords from the issue title and description, then repeatedly calls local AST-based search APIs such as search_class, search_method_in_class, and search_code_in_file to gather relevant signatures, method bodies, and surrounding code. If tests are available, spectrum-based fault localization (SBFL) adds suspiciousness scores so the agent can prioritize more likely fault sites. In the patch-generation stage, a second LLM agent uses the accumulated context plus the inferred buggy location to draft a patch, which is then checked against the test suite and regenerated if necessary.
The Django-13933 example makes the workflow concrete. The issue asks ModelChoiceField to show the invalid value in its validation error. AutoCodeRover first identifies ModelChoiceField and ModelMultipleChoiceField, then drills into validate and to_python, eventually locating django/forms/models.py and producing a patch that formats the invalid value into the error message. The point is not just that the model wrote a patch, but that it found the right semantic neighborhood through iterative, structure-aware search—much like a senior engineer reading the codebase with a debugger in hand.
On SWE-bench-lite, a benchmark of 300 real-world GitHub issues from 11 popular Python projects, AutoCodeRover achieved 19% efficacy, or about 57 solved issues at pass@1. The paper also reports an average time of about 4 minutes per issue, versus 2.68 days on average for human developers, and an average cost of only $0.43 USD. Perhaps most strikingly, about two-thirds of the produced patches were judged correct and acceptable. Together, these results suggest that structured retrieval plus LLM patching can already handle a meaningful slice of maintenance work at low cost.
The broader implication is that autonomous software engineering may be less about making LLMs “write more code” and more about giving them the right engineering workflow: understand structure, localize faults, validate with tests, and revise intelligently. AutoCodeRover shows that this is not just a theoretical idea; it can solve real GitHub issues in mature repositories. That makes it an important step toward systems that can repair, evolve, and eventually improve AI-generated or human-written software with minimal supervision.
At the same time, the paper is honest about what remains hard. Keyword extraction can mislead search; SBFL requires tests; and evaluation is limited to Python projects in SWE-bench-lite. Still, the direction is compelling: if future systems can better combine structured retrieval, debugging signals, and multi-turn repair, autonomous program improvement may become a practical layer in everyday development workflows.
Deep Analysis
Background
Software engineering has long pursued automation for tasks such as test generation, fault localization, and program repair. LLMs and tools like GitHub Copilot have recently improved code generation, but maintenance is a different challenge: real bug fixing and feature addition require reading issue reports, locating root causes, tracing dependencies, and validating repairs against tests. SWE-bench-lite standardizes this difficulty with 300 real GitHub issues from 11 popular Python repositories, including Django and SymPy, and has become an important benchmark for end-to-end software repair.
Core Problem
The central problem is to automatically produce a correct patch from only a natural-language GitHub issue and a large repository, without being told where the bug is. This is harder than HumanEval or MBPP because the system must reason about classes, methods, cross-file semantics, and test constraints. The paper argues that many APR and agent approaches either assume perfect fault localization or treat the codebase as a flat file collection, which is insufficient for authentic maintenance tasks in mature projects.
Innovation
AutoCodeRover contributes three main innovations. First, it uses AST-level program structure as the primary retrieval substrate, so code search targets classes, methods, and snippets rather than filenames. Second, it performs iterative context retrieval, letting each search result inform the next query and gradually narrowing the root cause. Third, it integrates SBFL when tests are available, turning test outcomes into suspiciousness signals that guide retrieval. Together, these ideas make the system resemble a human debugging workflow more than a generic code generator.
Methodology
- �� Inputs: a GitHub issue (title + description), the repository codebase, and optionally a test suite.
- �� Keyword extraction: an LLM reads the issue text and proposes names likely to map to program entities, such as class names, methods, exceptions, or code fragments.
- �� Structured retrieval: the agent calls AST-backed APIs like search_class, search_method_in_class, and search_code_in_file to obtain signatures, implementations, and nearby context.
- �� Iterative search: retrieval is multi-turn; the agent revises its next query based on what previous results revealed, e.g., from class to method to exact line region.
- �� Fault localization augmentation: if tests exist, SBFL assigns suspiciousness to methods so the agent can prioritize code that is both issue-relevant and test-suspicious.
- �� Patch synthesis: a second LLM agent generates a modification conditioned on the buggy location and collected context, often adjusting messages, conditions, or parameter passing.
- �� Validation and retry: the patch is run against available tests; if it fails, the system regenerates within a retry budget until success or exhaustion.
Experiments
The main evaluation is on SWE-bench-lite, a 300-instance benchmark spanning 11 real Python projects. Tasks include bug fixing and feature implementation, and the benchmark provides human pull requests and tests as ground truth. The paper compares AutoCodeRover with recent LLM-agent baselines, especially SWE-agent, and discusses Devin as the broader backdrop. Metrics include efficacy/pass@1, runtime, cost, and patch acceptability. A qualitative case study on Django-13933 illustrates end-to-end retrieval and patching.
Results
The headline result is 19% efficacy on SWE-bench-lite: about 57 of 300 issues were solved with pass@1. The paper explicitly states this is higher than the recently reported SWE-agent efficacy. A second important result is time: AutoCodeRover averages about 4 minutes per issue, versus 2.68 days on average for human developers, showing that the system can fit within a realistic maintenance budget. A third result is cost: only $0.43 USD on average per task. The authors also report that roughly two-thirds of the produced patches are correct and acceptable, which strengthens the case for practical use.
Applications
The most immediate use is open-source issue triage and first-pass patching: a maintainer can let the system localize the fault, draft a patch, and then review it. A second use is enterprise code maintenance, where large internal repositories often need rapid narrowing of the likely fix site. Because AutoCodeRover relies on a local code index and benefits from test suites, it is especially suitable for mature projects with structured source trees and recurring maintenance workloads.
Limitations & Outlook
The method still assumes that the issue description contains enough signal for keyword extraction and that the repository can be parsed into an AST. When naming is inconsistent, the problem is cross-cutting, or the code is highly fragmented, iterative retrieval may miss the true fault. SBFL also requires a test suite, so projects with sparse tests gain less benefit. The evaluation is limited to Python repositories in SWE-bench-lite, so cross-language generalization and scaling to very large systems remain open.
Plain Language Accessible to non-experts
Imagine a huge factory with thousands of machines. A worker reports: “Something is wrong with the red button on line 7.” You would not start by replacing random parts everywhere. First you would look at the map of the factory, find line 7, then check the machine panels near the red button, then inspect the tiny wires and switches inside. If there is a logbook that says which machines failed recent checks, you would start with the most suspicious ones.
AutoCodeRover works like that kind of careful mechanic. It reads the complaint, guesses which parts of the software might be involved, and then follows the structure of the program step by step. It does not treat the whole codebase like a pile of papers; it treats it like a machine with labels, parts, and connections. That is why it can avoid a lot of blind searching.
Once it finds the likely broken part, it makes a repair and then runs the factory’s safety checklist. If the repair fails the checks, it tries again. The big idea is simple: good repair starts with good finding. The paper shows that this strategy can fix many real bugs cheaply and quickly, instead of asking a human to hunt through the whole factory by hand.
ELI14 Explained like you're 14
Think of a giant game with tons of levels, menus, and hidden systems. A player writes in the bug forum: “When I click this thing, it crashes!” Now, if you were the fixer, would you open every file in the game and hope for the best? Nope—that would be chaos!
AutoCodeRover is like a super-organized game helper. First it reads the bug report and highlights clues like a class name, a function name, or an error message. Then it searches the game’s “map” of code in a smart order: first the big area, then the smaller room, then the exact shelf where the broken item is. If it has test results, it also uses them like hints from a walkthrough: “These spots are the most suspicious!”
After that, it writes a fix and checks whether the game still works. If the fix breaks something else, it tries again. That’s pretty cool, right? In the paper, this helper solved 57 out of 300 real issues, used about $0.43 per issue, and finished each one in about 4 minutes on average. Humans took 2.68 days on average!
So the big lesson is: smart repair is not just about writing code. It is about finding the right place to change, changing it carefully, and then checking everything. AutoCodeRover is basically teaching an AI to think like a careful game fixer instead of a random button-masher.
Glossary
AST (Abstract Syntax Tree)
A tree-shaped representation of code that exposes its grammatical structure, such as classes, methods, and calls. In technical terms, it is the substrate AutoCodeRover uses for search and localization.
Used by the local retrieval APIs to search classes and methods.
LLM agent
A large language model that can choose actions, call tools, and adapt based on feedback rather than generating one-shot text. Technically, AutoCodeRover uses agents for retrieval and for patch synthesis.
One agent retrieves context; another writes the patch.
SBFL (Spectrum-Based Fault Localization)
A debugging technique that uses test execution coverage to assign suspiciousness scores to program entities. In plain terms, it tells the system which methods look most likely to be broken.
Used when a test suite is available to sharpen retrieval.
SWE-bench-lite
A benchmark of 300 real GitHub issues from 11 Python projects. It measures whether a model can go from issue text to a working patch end to end.
The main experimental dataset in the paper.
pass@1
Success on the first try, without extra sampling or many retries. It is a strict metric for realistic automation.
AutoCodeRover solves about 57 issues at pass@1.
Open Questions Unanswered questions from this research
- 1 How well does this retrieval-plus-repair workflow transfer to non-Python ecosystems, very large monorepos, or codebases with weak naming conventions? The paper does not yet provide strong evidence beyond SWE-bench-lite.
- 2 Which repository properties most strongly predict success—test quality, code modularity, issue wording, or project size? A systematic cost-and-success model across these factors is still missing.
Applications
Immediate Applications
Open-source issue first responder
Maintainers can use the system to draft an initial patch from a GitHub issue, then inspect and refine it. This is most effective when the project has a local AST index and a usable test suite.
Internal debugging assistant
Engineering teams can deploy it on private repositories to shorten the time from bug report to likely fix location. It is especially useful for large codebases where manual search is slow.
Long-term Vision
Autonomous software maintenance pipeline
The longer-term vision is a closed loop where AI-generated code is automatically localized, repaired, tested, and improved before human review. The main obstacles are reliability, coverage, and broader language support.
Abstract
Researchers have made significant progress in automating the software development process in the past decades. Recent progress in Large Language Models (LLMs) has significantly impacted the development process, where developers can use LLM-based programming assistants to achieve automated coding. Nevertheless, software engineering involves the process of program improvement apart from coding, specifically to enable software maintenance (e.g. bug fixing) and software evolution (e.g. feature additions). In this paper, we propose an automated approach for solving GitHub issues to autonomously achieve program improvement. In our approach called AutoCodeRover, LLMs are combined with sophisticated code search capabilities, ultimately leading to a program modification or patch. In contrast to recent LLM agent approaches from AI researchers and practitioners, our outlook is more software engineering oriented. We work on a program representation (abstract syntax tree) as opposed to viewing a software project as a mere collection of files. Our code search exploits the program structure in the form of classes/methods to enhance LLM's understanding of the issue's root cause, and effectively retrieve a context via iterative search. The use of spectrum-based fault localization using tests, further sharpens the context, as long as a test-suite is available. Experiments on SWE-bench-lite (300 real-life GitHub issues) show increased efficacy in solving GitHub issues (19% on SWE-bench-lite), which is higher than the efficacy of the recently reported SWE-agent. In addition, AutoCodeRover achieved this efficacy with significantly lower cost (on average, $0.43 USD), compared to other baselines. We posit that our workflow enables autonomous software engineering, where, in future, auto-generated code from LLMs can be autonomously improved.