FailForge: Distilling Procedural Competence from Persistent Failures into Code Agents
FailForge extracts procedural skills from failed instances to improve code agent success rate by 6.6 points on SWE benchmarks.
Key Findings
Methodology
FailForge integrates failure diagnosis, leakage filtering, and skill induction to extract high-level procedural strategies from failed trajectories. It guides the model in reattempting tasks, then removes the guidance before training, enabling the model to internalize the behavior. The process involves a fault diagnosis model, a leakage filter to prevent overfitting to solutions, and a guided re-rollout mechanism. This approach leverages failure signals effectively, significantly boosting success rates on complex SWE tasks.
Key Results
- On SWE-bench verified set, FailForge raises pass@1 from 59.6% to 66.2%, with similar improvements at pass@3 and pass@5. The method recovers over 26% of previously failed instances at minimal additional cost. When training Qwen3.5-4B, the augmented data improves the success rate by 6.6 points, especially on the hardest problems, demonstrating the effectiveness of the induced procedural skills.
- Compared to baselines like increased sampling or hints, FailForge achieves higher success rates with comparable or lower token costs. The extracted strategies generalize across different tasks and model scales, indicating robust transferability and practical utility.
- Removing the induced skills during training ensures the model internalizes the strategies rather than relying on external hints. This results in more systematic problem-solving behavior, better generalization, and improved performance on unseen tasks.
Significance
This work addresses a fundamental bottleneck in code agent training—inefficient use of failure data. By transforming failures into valuable procedural knowledge, FailForge enhances the model’s autonomous problem-solving ability. It paves the way for more intelligent, self-improving software agents capable of tackling complex, real-world tasks without extensive human intervention. The approach also offers insights into learning from failure signals, which could influence broader AI research in autonomous reasoning and skill acquisition.
Technical Contribution
FailForge introduces a novel pipeline combining failure diagnosis, leakage filtering, and strategy induction to extract transferable procedural skills from failed trajectories. It innovates by explicitly removing the guidance after training, ensuring learned behaviors are embedded in the model weights. This contrasts with prior methods relying on external hints or partial action mining, offering a more holistic and scalable solution for complex task learning.
Novelty
This is the first systematic approach to induce high-level procedural skills directly from persistent failures in software engineering tasks. Unlike previous work limited to step-level action masking or instance hints, FailForge captures global failure contexts and abstracts strategies that generalize across instances. Its ability to internalize these skills without leakage or external guidance marks a significant advance in autonomous skill learning.
Limitations
- Dependence on accurate fault diagnosis models means that in highly ambiguous or complex failures, the induced skills may be less effective. The leakage filtering process, while robust, cannot fully eliminate all risks of overfitting to solutions.
- The additional training cost and multi-stage pipeline may limit scalability for extremely large datasets. Further optimization is needed for real-time or large-scale deployment.
- Current focus is on software engineering tasks; extending the approach to other domains like robotics or natural language understanding remains future work.
Future Work
Future directions include enhancing fault diagnosis accuracy via multi-modal data, integrating self-supervised learning to improve skill generalization, and applying the framework to broader AI tasks such as automated debugging, testing, and code optimization. Exploring multi-task and transfer learning strategies can further reduce costs and improve adaptability across diverse environments and languages.
AI Executive Summary
In the realm of automated software engineering, large language models have demonstrated remarkable capabilities. However, their performance is often hampered by persistent failures on complex tasks, leading to inefficient training and limited generalization. Traditional rejection fine-tuning (RFT) methods focus solely on successful trajectories, neglecting the rich information embedded in failures. This oversight leaves a significant portion of the training signal untapped, especially at the frontier of model capabilities.
FailForge introduces a paradigm shift by transforming failure instances into valuable procedural knowledge. The framework begins with diagnosing failures through a dedicated agent that analyzes error feedback and execution traces to identify root causes. From this diagnosis, it synthesizes high-level strategies—generalizable procedural skills—that guide the model in reattempting the task. These skills are injected into the teacher’s context during guided re-rollouts, significantly increasing the likelihood of success. Crucially, the skills are only used during training; they are removed before the final model update, ensuring the learned behavior is internalized rather than externally dependent.
Experimental results on SWE benchmarks demonstrate the efficacy of this approach. FailForge improves the success rate by 6.6 points over a strong RFT baseline, with over a quarter of previously failed instances recovered at minimal additional cost. The method’s ability to extract transferable, high-level strategies leads to more systematic problem-solving behaviors, which generalize across different tasks and model scales. This advancement addresses a critical bottleneck in AI-driven code generation, enabling models to learn from their failures in a structured manner.
The broader impact of this work lies in its potential to revolutionize autonomous AI systems. By effectively utilizing failure signals, models can evolve beyond mere pattern matching to strategic reasoning, reducing reliance on human intervention. Although current limitations include dependency on fault diagnosis quality and computational costs, future work aims to enhance robustness, efficiency, and cross-domain applicability. Overall, FailForge marks a significant step toward truly autonomous, self-improving AI agents capable of tackling the most challenging software engineering problems.
Deep Dive
Abstract
Rejection sampling fine-tuning (RFT) is widely used to train code agents by generating trajectories on verifiable software engineering tasks, retaining those that pass the tests, and fine-tuning on the successful rollouts. However, even strong code agents repeatedly fail on a substantial fraction of such tasks, and standard RFT simply discards these failures. The discarded samples are precisely the hardest and most informative ones, drawn from verifiable instances that are costly to curate. Stronger base models may reduce the number of failures, but the remaining hard cases still define the frontier for further improvement. We propose FailForge, an agentic framework that converts failed rollouts into training signal. For each failed instance, an agent diagnoses the failure from error feedback and execution traces, distills the diagnosis into a concise and actionable skill, and injects the skill into the agent context for a guided second attempt. Trajectories that succeed under skill guidance are folded back into the RFT corpus. Crucially, the skill is removed at training time, so the model internalizes the recovered behavior rather than relying on external hints at inference. FailForge recovers over 26% of previously failed instances at marginal additional cost, and training Qwen3.5-4B on the augmented corpus improves the SWE-bench Verified resolve rate by 6.6 points over a strong RFT baseline, with gains concentrated on the hardest problems.