Probe-and-Refine Tuning of Repository Guidance for Coding Agents

The application of large language models (LLMs) in software engineering has led to significant progress in code synthesis, debugging, and documentation generation. Early efforts focused on fine-tuning models like CodeT5 and CodeBERT, or developing scaffolds such as tree search and local search algorithms, to improve task-specific performance. Recent advancements include integrating structural knowledge graphs (e.g., RepoGraph) and contextual information (e.g., AGENTS.md files) to enhance model understanding of repositories. Despite these developments, the effectiveness of repository-level guidance remains debated. Some studies report efficiency gains when using curated guidance, while others observe decreased bug resolution rates with automatically generated files. These conflicting results suggest that the quality and production process of guidance files are critical factors. Existing approaches often rely on static instructions or complex iterative training, which can be costly and inflexible. The need for a systematic, lightweight method to iteratively refine operational guidance based on failure feedback remains unmet. This research addresses this gap by proposing a content-driven, synthetic failure-based tuning framework that enhances guidance quality without extensive retraining.

The core challenge in deploying AI-based code agents is the variability and often suboptimal quality of repository guidance files, which encode operational knowledge such as debugging workflows, module navigation, and testing procedures. Static guidance files are typically generated once and remain unchanged, limiting their adaptability to new or unforeseen issues. When models encounter unfamiliar failures, they tend to jump to incorrect files or propose invalid fixes, partly because guidance lacks the diagnostic specificity needed for effective troubleshooting. The fundamental bottleneck is the absence of a systematic, scalable mechanism to diagnose and iteratively improve guidance content based on actual failure feedback. Existing methods either rely on costly fine-tuning, reinforcement learning, or manual curation, which are not scalable or adaptable enough for diverse repositories. The challenge is to develop a lightweight, automated process that can diagnose deficiencies, generate targeted guidance updates, and improve coverage and effectiveness in a scalable manner.

This paper introduces a novel probe-and-refine tuning process that leverages synthetic bug-fix probes to iteratively diagnose and improve repository guidance files. Its key innovations include:

• �� Generating diverse synthetic failure scenarios using the language model itself, simulating real debugging challenges.
• �� Conducting single-shot attempts to fix these synthetic bugs, followed by automatic evaluation of the fixes.
• �� Mechanically patching guidance files based on diagnostic feedback, expanding coverage and operational specificity.
• �� Eliminating the need for multi-step agent loops, reinforcement learning, or retraining, making the process lightweight and transparent.
• �� Validating the approach across multiple repositories and models, demonstrating consistent improvements in bug resolution rates.

This approach contrasts with traditional static guidance or complex fine-tuning, emphasizing content diagnosis and mechanical correction to achieve scalable, effective guidance refinement.

• �� Build repository structural knowledge using tree-sitter parsing, extracting key modules, dependencies, and entry points, summarized in natural language.
• �� Generate synthetic bug-fix probes by sampling from the model with temperature 0.9, producing diverse failure scenarios across subsystems.
• �� For each probe, attempt to generate a fix using the current guidance, simulating the code agent’s behavior.
• �� Evaluate the attempted fix with a single-shot model call, assessing success and identifying shortcomings.
• �� Based on evaluation, generate guidance edits—such as procedural rules, structural navigation, or quality gates—and mechanically patch the guidance file.
• �� Limit guidance length to 3000 characters, ensuring concise, operational instructions.
• �� Repeat the cycle for 3–5 iterations, stopping early if guidance stabilizes or no further improvements are detected.
• �� The entire process relies solely on single-shot LLM calls, avoiding multi-step reasoning or reinforcement learning, thus maintaining simplicity and efficiency.

The experimental setup involved four repositories—Django, SymPy, Matplotlib, and Scikit-learn—each tested on the SWE-bench verified subset. For each repository, the guidance was initially constructed from static structural and procedural knowledge, then refined through 3–5 iterations of the probe-and-refine process. The guidance content length increased from an average of 1687 to 2754 characters, mainly adding repo-specific navigation, debugging workflows, and quality rules. The models used were Qwen3.5-35B-A3B and NVIDIA-Nemotron-3-Nano-30B-A3B for cross-model validation. The primary metric was bug fix resolution rate, measured by the percentage of instances where the agent’s patch passed all tests. Baseline comparisons included unguided and static guidance conditions. The experiments also included step-budget analysis, varying the number of steps allowed, and cross-model tests to evaluate robustness under different model capacities. Results consistently showed that iterative guidance refinement improved resolution rates significantly, with p-values less than 0.001, confirming the method’s effectiveness.

The probe-and-refine procedure achieved an average bug resolution rate of 33.0%, outperforming static knowledge base guidance (28.3%) and unguided baselines (25.5%), with high statistical significance (p<0.001). Content analysis revealed that the refined guidance added detailed procedural workflows, structural navigation instructions, and quality gates, which collectively increased the coverage of evaluable patches by 14.5 percentage points. The per-patch precision remained stable at about 59%, indicating that the improvement was primarily due to better diagnosis and localization rather than more aggressive or less accurate fixes. Step-budget experiments demonstrated that guidance enabled the model to utilize larger step counts more productively, and cross-model validation showed that when the model’s diagnostic output diminished, the tuning loop’s effectiveness declined, though patch accuracy remained unaffected. These findings confirm that content-driven guidance refinement effectively enhances the operational performance of code agents.

This approach can be directly applied to automated bug fixing, continuous integration pipelines, and AI-assisted code maintenance workflows. By iteratively refining operational guidance based on synthetic failure feedback, developers can reduce manual debugging efforts and improve the robustness of autonomous code repair systems. The method is especially suitable for open-source projects and large-scale enterprise systems where frequent updates and bug fixes are needed. Long-term, integrating this framework into CI/CD pipelines could enable continuous, self-improving guidance that adapts to evolving codebases, significantly reducing downtime and manual intervention. It also opens avenues for developing more intelligent, context-aware guidance systems that learn from failures without extensive retraining.

The effectiveness of the method depends on the model’s ability to generate diagnostic outputs; when this capacity is limited, the tuning loop degrades, reducing improvements. The guidance content length increases by about 63% after refinement, which may pose challenges in contexts with strict token limits. The validation is currently limited to specific models and datasets, and its generalization to other models, languages, or larger codebases remains to be demonstrated. Additionally, the synthetic probes, while diverse, may not capture all real-world failure modes, potentially limiting coverage in complex scenarios. Future work should focus on adaptive probe generation, multi-modal feedback integration, and real-world deployment to address these limitations.