SWE-Synth: Synthesizing Verifiable Bug-Fix Data to Enable Large Language Models in Resolving Real-World Bugs
SWE-Synth uses LLMs to simulate debugging workflows, synthesizing verifiable bug-fix data, improving open-source model training by 2.3%.
Key Findings
Methodology
This approach employs a repository-level mutation framework, leveraging LLM agents to mimic debugging processes. It involves component selection based on test coverage, code re-implementation via LLMs, and iterative validation through test execution. The process generates bug-fix pairs, test cases, and structured repair trajectories, integrating reinforcement learning and rejection sampling to ensure realism and diversity. The pipeline minimizes manual effort, scales efficiently, and produces high-fidelity datasets. Experiments demonstrate a 2.3% improvement in repair success rate on SWE-Bench Lite when models are trained on SWE-Synth data, validating the method’s effectiveness.
Key Results
- Models trained on SWE-Synth achieved a 13.0% bug fix rate on SWE-Bench Lite, surpassing real-data trained models by 2.3%, indicating synthetic data’s high utility.
- Test coverage-driven component selection yielded more impactful bugs, enhancing model generalization, especially in complex debugging scenarios.
- Incorporating intermediate repair trajectories improved model interpretability and robustness, demonstrating the importance of process-aware data for complex bug fixing.
Significance
This work addresses the critical bottleneck of data scarcity in training open-source LLMs for automated program repair. By synthesizing realistic, verifiable bug-fix datasets, it enables scalable training without extensive manual annotation. The approach bridges the gap between real-world bug complexity and data availability, fostering advancements in autonomous software engineering. It also opens new avenues for research into process-aware learning, making models more interpretable and effective in real debugging tasks. The methodology’s scalability and verification mechanisms position it as a foundational step toward democratizing high-quality training data for open-source communities and industry applications.
Technical Contribution
The paper introduces a novel repository-level mutation framework that combines test coverage analysis, LLM-based code re-implementation, and structured trajectory sampling. It integrates reinforcement learning and rejection sampling to generate diverse, realistic bug scenarios with intermediate reasoning steps. The resulting SWE-Synth dataset surpasses traditional mutation-based methods in scale and authenticity, enabling effective supervised fine-tuning of LLMs for APR. This paradigm shift emphasizes process-awareness and verification, setting new standards for synthetic data generation in software engineering.
Novelty
This is the first work to systematically leverage LLMs to simulate debugging workflows, generating high-quality, process-aware bug datasets with intermediate steps. Unlike mutation or simple synthetic methods, it emphasizes realism, verifiability, and scalability. The integration of test coverage-based component selection and trajectory sampling distinguishes it from prior approaches, providing a new framework for data synthesis that closely mimics human debugging and enhances model training.
Limitations
- The current implementation focuses on Python repositories; adapting to other languages requires additional tooling and testing infrastructure, limiting immediate cross-language applicability.
- Dependence on test coverage data may limit effectiveness in projects with sparse or incomplete test suites, affecting bug realism.
- While more realistic than previous synthetic methods, generated bugs may still be biased toward certain defect types; broader defect diversity needs further validation.
Future Work
Future directions include extending multi-language support, integrating more complex debugging scenarios, and exploring adaptive sampling strategies. Combining reinforcement learning to optimize bug generation, and deploying in industrial environments for real-world validation, are also planned. Further research will focus on enhancing the diversity of synthetic bugs and improving the fidelity of intermediate repair trajectories, aiming to fully replicate human debugging workflows and facilitate broader adoption.
AI Executive Summary
Automated program repair (APR) has become a vital area in software engineering, yet progress remains hindered by the scarcity of high-quality training data. Real bug-fix pairs are costly to curate, noisy, and often lack detailed intermediate reasoning steps, which are crucial for training effective models. Existing benchmarks like SWE-Bench provide some evaluation but fall short in scalability and realism, especially for open-source models. Synthetic data generation offers a promising alternative but faces challenges in realism, diversity, and verification.
This paper introduces SWE-Synth, a novel framework that leverages large language models (LLMs) to simulate human-like debugging workflows at the repository level. The core idea is to use LLMs to generate synthetic bug scenarios by masking components in code, re-implementing them based on context and test feedback, and validating the generated bugs through automated testing. The process involves component selection driven by test coverage, code re-implementation via prompts, and iterative refinement with reinforcement learning and rejection sampling. The resulting dataset contains bug-fix pairs, test cases, and detailed repair trajectories, mimicking real debugging processes.
Experimental results demonstrate that models trained on SWE-Synth data outperform those trained on real-world datasets, achieving a 13% bug fix rate on SWE-Bench Lite, with a 2.3% improvement over baseline. The synthetic data’s realism and process-awareness significantly enhance model generalization and interpretability, especially in complex multi-step repair tasks. This work addresses key bottlenecks in open-source APR, providing scalable, verifiable, and rich training resources.
Looking ahead, the authors plan to extend multi-language support, incorporate more sophisticated debugging scenarios, and validate industrial applicability. The approach paves the way for democratizing high-quality training data, accelerating the development of autonomous software engineering tools, and ultimately transforming how software bugs are diagnosed and fixed automatically.
Deep Dive
Applications
What is the real-world impact?
Limitations & Outlook
What gaps remain?
Abstract
Large language models (LLMs) are transforming automated program repair (APR) through agent-based approaches that localize bugs, generate patches, and verify fixes. However, the lack of high-quality, scalable training datasets, especially those with verifiable outputs and intermediate reasoning traces-limits progress, particularly for open-source models. In this work, we present SWE-Synth, a framework for synthesizing realistic, verifiable, and process-aware bug-fix datasets at the repository level. SWE-Synth leverages LLM agents to simulate debugging workflows, producing not only bug-fix pairs but also test cases and structured repair trajectories. Compared to manually curated datasets, our method scales with minimal human effort while preserving contextual richness and correctness. Experiments show that models trained on SWE-Synth outperform those trained on real-world datasets by 2.3% on SWE-Bench Lite. Our results highlight the potential of synthetic, agent-generated data to advance the state of the art in APR and software engineering automation.