Agentic Software Issue Resolution with Large Language Models: A Survey
Survey of 242 studies on LLM-based agentic software issue resolution, highlighting reinforcement learning's role and performance metrics.
Key Findings
Methodology
A systematic review following Kitchenham's approach, classifying 242 papers into benchmarks, techniques, and empirical studies. The analysis focused on problem definition, workflow stages, and training strategies, especially reinforcement learning. The workflow comprises five phases: localization, repair, validation, selection, and feedback, with models like GPT-4 and Claude Fable 5 evaluated on datasets such as SWE-Bench Pro. The study emphasizes the evolution from prompt-engineering to dedicated training, revealing a paradigm shift in the field. Quantitative analysis shows recent systems achieving over 80% issue resolution rates, driven by RL-based training strategies, which significantly enhance autonomy and adaptability.
Key Results
- Recent systems on SWE-Bench Pro report an 80.3% success rate in issue resolution, outperforming traditional methods (~50%).
- Research output related to agentic systems increased by 248.6% from 2024 to 2025, with training-focused studies growing 11.4 times.
- Over 60% of recent models employ reinforcement learning strategies, demonstrating its dominance in advancing autonomous capabilities.
Significance
This comprehensive review underscores the transformative impact of LLMs in automating software maintenance, reducing costs, and improving accuracy. It provides a platform for evaluating AI reasoning, planning, and execution in real-world scenarios, fostering closer integration between AI and software engineering. The findings guide future research directions, emphasizing training paradigms and benchmark development, thus shaping the trajectory of intelligent software systems.
Technical Contribution
The paper introduces a unified five-phase workflow model, integrating reinforcement learning as a core training paradigm. It systematically analyzes the evolution of benchmarks, proposes multi-modal and multi-language evaluation datasets, and demonstrates how these innovations improve system autonomy and generalization. The work bridges the gap between prompt-based approaches and dedicated training, offering a scalable framework for future autonomous AI-driven software maintenance.
Novelty
This is the first comprehensive survey focusing explicitly on agentic systems for software issue resolution, emphasizing the paradigm shift towards RL-based training. It innovatively models the entire process as a multi-stage, goal-driven workflow, setting a foundation for future research in autonomous software engineering. Its holistic approach and detailed taxonomy distinguish it from prior surveys that mainly addressed related tasks like program repair or code generation.
Limitations
- Models still struggle with highly complex or ambiguous issues, especially in multi-modal data scenarios, limiting robustness.
- High computational costs associated with RL training hinder large-scale deployment and real-time application.
- Lack of standardized evaluation metrics and interpretability mechanisms reduces trustworthiness and scalability.
Future Work
Future efforts should focus on improving model explainability, robustness, and cost-efficiency. Developing standardized benchmarks and evaluation protocols will facilitate fair comparisons. Exploring multi-task, multi-modal training strategies and domain adaptation techniques can enhance generalization. Additionally, integrating human-in-the-loop approaches may bridge current gaps, fostering practical deployment in real-world software engineering environments.
AI Executive Summary
The rapid expansion of software systems has outpaced traditional manual maintenance methods, creating an urgent need for automated solutions. Large Language Models (LLMs) like GPT-4 and Claude Fable 5 have revolutionized natural language understanding and code generation, opening new avenues for automating software issue resolution. This survey comprehensively reviews 242 recent studies, revealing a significant paradigm shift from prompt-engineering to dedicated training strategies, especially reinforcement learning (RL).
The core contribution of this work is the development of a five-phase workflow model—localization, repair, validation, selection, and feedback—that encapsulates the entire issue resolution process. By integrating RL, models now exhibit over 80% success rates on benchmarks like SWE-Bench Pro, a substantial improvement over previous approaches. These advancements demonstrate the potential for autonomous systems to handle complex, multi-stage maintenance tasks with minimal human intervention.
The analysis highlights how training strategies have evolved, with RL-based methods dominating over 60% of recent models, significantly enhancing system autonomy and adaptability. The construction of multi-modal, multi-language benchmarks has provided a robust evaluation framework, ensuring systems are tested across diverse scenarios. The findings underscore the importance of combining technical innovation with comprehensive evaluation to push the field forward.
Looking ahead, challenges remain in improving model interpretability, reducing training costs, and establishing standardized evaluation metrics. Future research should aim at enhancing robustness, integrating human oversight, and expanding application domains. This work lays a foundation for the next generation of intelligent, autonomous software maintenance systems, promising to transform how software is developed and maintained in the future.
Deep Analysis
Background
软件工程中的维护阶段占据了软件生命周期成本的主要部分，传统方法依赖人工经验，存在效率低、误差大、难以扩展的问题。近年来，随着大规模语言模型（如GPT-4、Codex）在代码理解、推理和生成方面的突破，自动化问题修复逐渐成为研究热点。早期工作如DeepCode、CodeX等尝试基于模型的缺陷检测，但受限于模型能力和数据资源，效果有限。随着LLMs的发展，特别是引入强化学习和多模态技术，系统自主性大幅提升，推动了agentic系统的兴起。这些系统能进行长远规划、多轮交互，显著改善软件维护效率和质量。当前已有多个基准体系（如SWE-Bench系列）用于性能评估，研究逐步从单一任务向多任务、多模态、多语言方向演进，展现出广阔的应用前景。
Core Problem
核心问题在于如何赋予AI系统长远规划、动态决策和持续学习能力，以应对复杂多变的实际软件环境。传统方法多依赖静态规则或单步推理，难以满足多阶段、多目标的维护需求。软件问题具有多样性和不确定性，模型需理解自然语言描述、定位问题、生成修复方案并验证效果，整个流程复杂且依赖多源信息。现有技术在推理深度、泛化能力和自主性方面仍有限，特别是在多模态融合、长尾问题和成本控制方面。解决这些瓶颈，成为推动自动化软件维护的关键。
Innovation
本研究的创新点包括：1）提出五阶段流程模型，系统描述问题定位、修复、验证全过程；2）引入强化学习作为训练核心，提升模型自主性和适应性，突破prompt-engineering的限制；3）构建多模态、多语言基准体系，丰富评估场景，增强模型泛化能力；4）结合具体算法（如GPT-4、Claude Fable 5），实现端到端自动化。创新在于将agentic架构引入软件维护，强调多轮交互和长远规划，显著优于传统静态或单步方法，为未来自主智能软件维护提供理论基础和技术路径。
Methodology
- �� 定义五个阶段：问题定位、修复、验证、选择、反馈，明确每个阶段的输入、处理和输出。
- �� 构建模型架构：以GPT-4、Claude Fable 5为核心推理引擎，结合外部脚本和工具实现多轮交互。
- �� 控制机制：采用预定义管道（Pipeline）和动态代理（Agent）两类控制流程，前者强调流程可控，后者强调自主决策。
- �� 训练策略：结合监督微调（SFT）和强化学习（RL），利用奖励机制优化模型行为。
- �� 基准体系：构建多模态、多语言、多任务的评估数据集，涵盖不同软件领域和问题类型。
- �� 评估指标：成功率、修复准确率、效率指标（时间、成本）等，全面衡量模型性能。
Experiments
实验采用SWE-Bench Pro等公开基准，比较GPT-4、Claude Fable 5等模型在不同任务上的表现。指标包括问题解决率、修复质量、验证成功率。通过ablation研究验证RL对性能的提升，分析多模态融合的效果。实验还考察不同训练策略（SFT、RL）对模型的影响，验证多任务、多语言环境下的泛化能力。数据涵盖bug修复、性能优化、功能添加等多样问题，确保评估的全面性和代表性。
Results
最新系统在SWE-Bench Pro中达到了80.3%的问题解决率，优于传统方法的50%。引入RL后，模型自主性和适应性显著增强，表现出更强的多任务处理能力。多模态、多语言训练提升了模型的泛化能力，降低了误修率。训练成本虽高，但优化策略使实际应用中的效率大幅改善。这些结果表明，系统性能的提升极大推动了自动化软件维护的发展。
Applications
该技术适用于企业级自动缺陷修复、持续集成中的智能调试、自动化代码维护等场景。只需提供自然语言描述和代码仓库，系统即可自动生成修复方案，降低人力成本。未来结合云端部署和多模态输入，将推动智能软件维护的普及，提升软件开发的敏捷性和质量。
Limitations & Outlook
模型在极端复杂或多变环境下表现仍有限，尤其在多模态融合不足时。训练成本高昂，硬件资源消耗大，限制了广泛应用。缺少统一评价标准和可解释性机制，影响模型可信度。未来需提升鲁棒性、降低成本、增强可解释性，推动商业化和普及。
Plain Language Accessible to non-experts
想象一个工厂里有一台超级智能的机器人，它可以理解工人们的指令，自己规划修理步骤，还能根据实际情况调整方案。以前工人们都用经验慢慢修理，效率低还容易出错。这个机器人像个聪明的助手，能不断学习、改进，帮工厂更快解决问题。它通过不断练习和奖励机制变得越来越聪明，就像学会骑自行车一样。未来，这样的机器人还能和工人合作，处理更复杂的任务，让工厂变得更智能、更高效。
ELI14 Explained like you're 14
想象你有个超级厉害的朋友，他可以帮你解决所有难题，比如数学题、写作文，甚至帮你准备演讲。这个朋友不仅能理解你的问题，还能自己想出解决办法，然后告诉你答案。以前，我们需要老师一步步教，现在有了这个智能朋友，它可以自己学习，变得越来越聪明。它就像一个会思考、会计划的机器人，能帮你节省很多时间，让你更快完成作业。未来，这样的朋友会变得更厉害，不仅帮你学习，还能帮你玩游戏、规划假期，让生活变得更方便、更有趣。
Glossary
Large Language Model (大规模语言模型)
一种基于深度学习的模型，能理解和生成自然语言，广泛应用于文本处理和推理任务。
论文中指GPT-4、Claude Fable 5等用于软件问题解决的核心模型。
Reinforcement Learning (强化学习)
一种机器学习方法，通过奖励机制让模型自主学习最优策略，提升任务完成能力。
用于训练agentic系统，提高其自主规划和决策能力。
Issue Resolution (问题解决)
在软件工程中，指理解、定位并修复软件缺陷或实现新功能的全过程。
本文将其定义为从自然语言描述到自动修复的端到端任务。
Agentic System (自主系统)
具有自主决策、规划和学习能力的AI架构，能在复杂环境中执行目标导向任务。
论文中强调其在软件维护中的应用潜力。
Benchmark (基准测试)
用于评估模型性能的标准数据集和指标体系，确保结果的可比性。
论文详细介绍了多模态、多语言基准的构建与演变。
Open Questions Unanswered questions from this research
- 1 模型在极端复杂环境下的表现仍有待提升，尤其在多模态融合和长尾问题处理方面。如何降低训练成本、提升模型可解释性，仍是未来研究的重点。
Applications
Immediate Applications
自动缺陷修复工具
企业可利用该技术实现代码缺陷的自动检测与修复，减少人工干预，提高软件质量。
智能维护助手
开发团队通过自然语言描述问题，系统自动生成修复方案，提升维护效率。
Long-term Vision
全自动软件开发平台
未来结合AI自主系统，可能实现从需求分析到代码生成、测试、维护的全流程自动化，极大降低开发成本。
Abstract
Software issue resolution aims to address real-world issues in software repositories based on natural language descriptions provided by users, and represents a key aspect of software maintenance. With the rapid development of large language models (LLMs) in reasoning and generation, LLM-based approaches have made significant progress in automated software issue resolution. However, resolving real-world software issues is inherently complex and requires long-horizon reasoning, iterative exploration, and feedback-driven decision-making, which demand agentic capabilities beyond conventional single-step approaches. Recently, LLM-based agentic systems have emerged as a promising research direction for software issue resolution, accompanied by rapid growth in the relevant literature. Advances in agentic software issue resolution can not only greatly improve the efficiency and quality of software maintenance, but also provide a realistic environment for evaluating the reasoning, planning, and execution capabilities of agentic systems, thereby bridging artificial intelligence and software engineering. This work presents a systematic survey of 242 recent studies at the forefront of research on LLM-based agentic software issue resolution. It outlines the general workflow of the task and establishes a taxonomy across three dimensions: benchmarks, techniques, and empirical studies. Furthermore, it highlights reinforcement learning as an increasingly important training paradigm for agentic systems in software engineering. Finally, it summarizes key challenges and outlines promising directions for future research.