SWE-EVO: Benchmarking Coding Agents in Long-Horizon Software Evolution Scenarios
SWE-EVO benchmark assesses AI coding agents' ability to perform long-term software evolution tasks; top models reach only 25%, exposing significant gaps.
Key Findings
Methodology
This study constructs SWE-EVO by extracting release notes, commit histories, and version snapshots from seven mature open-source Python repositories. The benchmark comprises 48 complex tasks simulating real-world software evolution, involving multi-step modifications across an average of 21 files, validated against test suites with an average of 874 tests per instance. Tasks are framed around version-to-version transitions, where models interpret high-level release notes, plan multi-file changes, and generate patches. Evaluation metrics include Resolved Rate, indicating full task success, and Fix Rate, measuring partial progress. The evaluation setup involves providing models with the pre-release codebase and detailed change descriptions, then assessing the generated patches through automated testing. Results show that even the most advanced models, such as GPT-5.4, achieve only 25% success, highlighting the substantial challenge of long-horizon, multi-file software evolution tasks.
Key Results
- GPT-5.4 achieves a maximum Resolved Rate of only 25% on SWE-EVO, a stark contrast to its 72.8% performance on SWE-Bench Verified, illustrating the increased difficulty of long-term, multi-file tasks.
- Models like GPT-5.2, which perform well on simpler benchmarks, drop from 72.8% on SWE-Bench to 22.92% on SWE-EVO, indicating a significant capability gap in sustained reasoning and planning.
- Failure mode analysis reveals common issues such as syntax errors, incorrect tool usage, and misinterpretation of release notes, which hinder model performance in complex scenarios.
Significance
This work pioneers a comprehensive benchmark for evaluating AI coding agents in the context of real-world software maintenance and evolution. By focusing on version-to-version transitions involving multiple files and steps, SWE-EVO exposes the limitations of current models and emphasizes the need for advanced reasoning, planning, and contextual understanding capabilities. The benchmark provides a critical tool for researchers aiming to develop AI systems capable of autonomous, long-term software evolution, which is vital for industrial applications like automated maintenance, continuous integration, and legacy system modernization. It bridges a significant gap between simplified problem-solving tasks and the complex realities faced in software engineering, pushing the field toward more robust, scalable AI solutions.
Technical Contribution
The paper introduces a novel method for constructing long-horizon software evolution tasks based on real version histories, emphasizing multi-file, multi-step modifications. It develops a comprehensive evaluation framework incorporating both strict success metrics (Resolved Rate) and partial progress indicators (Fix Rate). The benchmark leverages detailed test suites and release notes to simulate authentic development scenarios, challenging models to interpret nuanced requirements and coordinate across many files. Additionally, the study conducts extensive failure mode analysis, identifying key bottlenecks such as instruction understanding and tool interaction, guiding future model improvements. The combination of real-world data, complex task design, and nuanced metrics constitutes a significant advancement over prior benchmarks like SWE-bench, which focus on isolated issues.
Novelty
SWE-EVO is the first benchmark explicitly designed to evaluate AI agents on long-term, multi-file software evolution tasks derived from real open-source projects. Unlike previous benchmarks limited to single Issue fixes or isolated code snippets, SWE-EVO captures the complexity of version-to-version transitions, requiring models to understand high-level release notes, coordinate changes across dozens of files, and pass extensive validation suites. Its emphasis on multi-step, multi-file, and multi-version reasoning marks a fundamental shift toward more realistic and challenging evaluation scenarios, making it a pioneering step in AI-driven software engineering research.
Limitations
- Despite its realism, SWE-EVO still relies on static test suites and does not fully capture the iterative, interactive nature of real-world software development, such as code reviews and incremental testing.
- Current models struggle with nuanced understanding of release notes and complex coordination, indicating that further advances in natural language understanding and reasoning are needed.
- The benchmark focuses on open-source Python projects, which may limit generalization to industrial, multi-language, or legacy systems. Additionally, the computational cost of evaluating large models on heavy test suites remains high.
Future Work
Future research should explore integrating dynamic context management, multi-modal inputs (e.g., code, documentation, diagrams), and reinforcement learning to enhance models' planning and reasoning capabilities. Developing models that can learn continuously from evolving codebases and incorporate feedback from real development workflows will be crucial. Extending SWE-EVO to industrial-scale projects, multi-language environments, and incorporating human-in-the-loop interaction are promising directions. Furthermore, improving interpretability and debugging capabilities of models will help in deploying AI for critical software maintenance tasks, ultimately moving toward autonomous, long-term software evolution systems.
AI Executive Summary
Software engineering has long been characterized by its iterative, multi-faceted nature, involving not only the fixing of bugs but also the continuous evolution of complex codebases across multiple versions. Despite recent advances in large language models (LLMs) such as GPT-4, GPT-5, and various open-source alternatives, their capabilities in handling long-term, multi-file, multi-step software evolution tasks remain limited. Existing benchmarks like SWE-bench primarily evaluate models on isolated issues or small-scale code snippets, which do not adequately reflect the realities faced by developers managing legacy systems and ongoing maintenance.
Recognizing this gap, the authors introduce SWE-EVO, a comprehensive benchmark designed to evaluate AI coding agents' ability to perform long-horizon software evolution tasks. Derived from real-world version histories of seven prominent open-source Python projects, SWE-EVO constructs 48 challenging tasks that simulate version-to-version transitions. Each task involves interpreting high-level release notes, planning multi-step modifications across an average of 21 files, and passing extensive validation tests with an average of 874 test cases. This setup mimics the complexity of real software maintenance, requiring models to understand nuanced requirements, coordinate changes, and preserve functionality.
The evaluation framework employs two key metrics: Resolved Rate, which measures the percentage of tasks fully solved, and Fix Rate, a soft metric capturing partial progress. Results demonstrate that even the most advanced models, such as GPT-5.4, achieve only 25% success on SWE-EVO, a stark contrast to their performance on simpler benchmarks. The analysis reveals common failure modes, including misinterpretation of release notes, syntax errors, and inadequate multi-file reasoning. These findings underscore the significant challenges that remain in developing AI systems capable of autonomous, sustained software evolution.
This work has profound implications for both academia and industry. It highlights the necessity for models with enhanced contextual understanding, planning, and multi-file coordination capabilities. The benchmark serves as a catalyst for future research aimed at bridging the gap between current AI abilities and the demands of real-world software maintenance. By pushing the boundaries of what AI can achieve in long-term, complex tasks, SWE-EVO paves the way toward more autonomous, reliable, and scalable software engineering solutions, ultimately transforming how software is developed, maintained, and evolved in the coming decades.
Deep Analysis
Background
随着大规模预训练语言模型（如GPT系列、Codex等）在代码生成、调试和测试方面取得突破，软件工程中的自动化水平不断提升。早期的评测体系如HumanEval、MBPP和APPS主要关注单文件、单Issue任务，逐渐饱和，难以反映实际开发中的复杂场景。近年来，SWE-bench等基准引入了真实GitHub问题的验证补丁，推动了多Issue、多文件、多任务的评估框架发展。然而，现有评测仍偏重于静态、孤立的问题解决，未能充分模拟软件的持续演化过程。实际开发中，80%以上的工作集中在维护和演化遗留代码，涉及跨模块、版本和规范的协调变更。为弥补这一空白，本文提出了面向长远软件演化的SWE-EVO基准，旨在推动AI编码代理在真实软件生命周期中的应用研究。
Core Problem
现有AI编码代理在单Issue修复任务中表现尚可，但在持续、多文件、多步骤的演化场景中表现极为有限。软件的版本迭代涉及复杂的需求理解、跨文件协调、重构和验证，模型需要具备长远规划和全局理解能力。传统评测未能反映这些挑战，限制了模型在实际软件维护中的应用潜力。如何设计一个能够模拟真实软件演化场景的评测体系，评估模型在多版本、多文件、多步骤任务中的表现，成为亟待解决的问题。本文通过采集真实开源项目的版本变迁，构建了具有代表性的长远演化任务，旨在推动模型在复杂环境中的能力提升。
Innovation
本研究的核心创新在于：1）提出基于真实版本发布说明和提交历史的长远演化任务构建方法，模拟实际软件的版本迭代过程；2）设计了跨越多文件、多步骤的演化任务，显著提升任务复杂性和真实性；3）引入修正率（Fix Rate）指标，衡量模型在部分修复中的表现，避免单一成功率的偏差；4）结合Failure Mode分析，系统性识别模型在指令理解、工具调用和语法正确性等方面的瓶颈，为未来模型优化提供方向。这些创新使得评测体系更贴近实际开发场景，推动AI编码代理从单Issue修复向持续演化能力迈进。
Methodology
- �� 任务构建：采集七个成熟开源Python项目的版本发布说明、提交历史和快照，筛选出符合版本标签的快照作为任务起点和终点。• 任务定义：以版本间的发布说明差异为变更需求，模拟软件演化场景，要求模型在多文件、多步骤中实现变更。• 数据过滤：通过应用测试套件验证变更的有效性，确保任务具有明确的行为变化。• 任务规模：每个任务平均涉及21个文件，包含610行代码变更，测试套件平均874个测试用例。• 评估指标：采用Resolved Rate（完全解决比例）和Fix Rate（部分修复比例），同时引入修正后回归检测。• 模型输入：完整的前版本代码和变更说明，输出为跨文件补丁。• 评估流程：模型生成补丁后，应用于测试环境，验证变更效果和无回归，统计指标得分。
Experiments
- �� 评估平台：使用OpenHands和SWE-agent两个框架，测试18个不同的预训练模型，包括OpenAI的GPT-系列、DeepSeek、Zhipu AI、Qwen和Moonshot AI。• 任务设置：在默认的release note + PR/issue上下文中进行推理，限制模型访问互联网，确保评估的公平性。• 超参数：采用中等推理努力水平，平衡推理成本与准确率。• 性能指标：统计Resolved Rate、Apply Rate（补丁可用性）和Fix Rate，分析不同模型在不同任务难度下的表现差异。• 失败分析：结合Failure Mode分类，识别模型在指令理解、工具调用、语法错误等方面的具体问题。
Results
- �� 最高模型GPT-5.4在SWE-EVO中的Resolved Rate仅为25%，显著低于在SWE-Bench上的72.8%，表明长远演化任务对模型提出了更高的理解和规划要求。• 不同模型表现差异明显，GPT-5.2在SWE-Bench表现优异，但在SWE-EVO中仅为22.92%，反映出模型在持续、多步骤推理中的局限。• Failure Mode分析显示，模型常在指令理解、工具调用和语法正确性方面出现问题，尤其在复杂发布说明中理解偏差明显。• 任务难度随着PR数量、代码变更范围和测试复杂度增加而显著上升，模型在高难度任务中的表现逐步下降。
Plain Language Accessible to non-experts
想象你在管理一个大型工厂，工厂每天都在不断生产新产品、修理旧设备、升级机器。每次升级都需要你协调不同的部门：机械、电子、软件，确保每个环节都配合得当。现在，假设你有一个超级智能助手，它能帮你理解每次升级的需求，规划出详细的操作步骤，并在多个部门之间协调工作。这个助手就像是一个非常聪明的工厂经理，能在长时间内持续管理和优化整个工厂的运作。SWE-EVO这个基准测试就像是模拟这个工厂的升级任务，考察AI助手是否能像人一样，理解复杂的升级需求，协调多个部门，确保工厂顺利运行。它要求助手不仅懂得单个设备的维修，更能规划整个工厂的长远发展，处理各种突发情况，确保每次升级都能顺利完成。
ELI14 Explained like you're 14
想象你在学校里负责组织一个大型活动，比如运动会。每次活动都需要准备很多不同的项目，比如跑步、跳远、接力赛，还要协调不同的老师和学生。每次活动结束后，你还要总结经验，改进下一次的安排。现在，假设你有一个超级聪明的助手，它能帮你理解每次活动的需求，帮你制定详细的计划，并协调所有人一起完成任务。这个助手就像是一个非常聪明的学生会主席，能在长时间内不断学习和改进，确保每次运动会都比上次更成功。SWE-EVO这个基准测试就像是模拟这个过程，看看AI助手是否能像人一样，理解复杂的需求，协调不同的人和部门，完成一场完美的运动会。它不仅要懂得每个项目的细节，还要能规划整个流程，处理突发状况，确保活动顺利进行。
Glossary
Resolved Rate
The proportion of tasks where the AI model successfully passes all relevant tests, indicating complete implementation of the version change.
Used as the primary success metric for evaluating whether the model fully achieves the intended software evolution.
Fix Rate
A soft metric measuring the fraction of failing tests fixed by the model, reflecting partial progress while ensuring no regressions occur.
Provides a nuanced view of model performance beyond binary success/failure outcomes.
Release Note
A document describing the changes, features, and fixes introduced in a new software version, serving as the primary input for the models.
Models interpret release notes to understand the high-level requirements of the evolution task.
Test Suite
A collection of automated tests designed to verify the correctness and stability of the software after modifications.
Used to validate whether the generated patches meet the specified requirements without introducing regressions.
Multi-file Reasoning
The ability of models to understand and modify multiple source files simultaneously, ensuring coordinated changes across the codebase.
A critical capability for long-term software evolution tasks involving complex, multi-file modifications.
Long-Horizon Tasks
Tasks requiring multiple steps, stages, or phases to complete, often spanning across different files, modules, or versions.
SWE-EVO tasks exemplify long-horizon scenarios in software engineering.
Behavioral Delta
The difference in software behavior between versions, used to verify if the evolution meets the specified change.
Test outcomes are compared before and after patch application to assess success.
Version Snapshot
A complete record of a software system at a specific point in time, used as a reference for evolution tasks.
Defines the starting or ending point of a version-to-version transition.
Open Questions Unanswered questions from this research
- 1 Although SWE-EVO exposes the limitations of current models in long-term, multi-file tasks, how to effectively incorporate advanced reasoning, planning, and contextual understanding remains an open challenge. Developing models that can dynamically manage context, learn continuously, and adapt to evolving requirements is crucial. Additionally, integrating human-in-the-loop feedback and real-time interaction could significantly improve performance. Extending the benchmark to industrial-scale, multi-language, and legacy systems will also be necessary to ensure practical applicability and robustness in real-world scenarios.
Applications
Immediate Applications
Automated Software Maintenance
Employ AI agents to interpret release notes, generate multi-file patches, and automate routine updates, reducing manual effort and accelerating deployment pipelines in enterprise environments.
Code Review and Quality Assurance
Use models to assist in reviewing large code changes, detecting inconsistencies, and suggesting improvements, thereby enhancing code quality and reducing review time.
Software Evolution Planning
Leverage AI to simulate and optimize version upgrade strategies, helping developers plan long-term evolution pathways with minimal manual intervention.
Long-term Vision
Autonomous Software Development
Future AI systems could autonomously understand requirements, plan, implement, and verify software evolution, moving toward fully automated software lifecycle management.
Intelligent Maintenance Ecosystems
Build integrated AI-driven ecosystems capable of continuous learning, adaptation, and evolution across diverse software environments, transforming software engineering into a highly automated discipline.
Abstract
Existing benchmarks for AI coding agents focus on isolated, single-issue tasks such as fixing a bug or adding a small feature. However, real-world software engineering is a long-horizon endeavor: developers interpret high-level requirements, coordinate changes across many files, and evolve codebases over multiple iterations while preserving functionality. We introduce SWE-EVO, a benchmark for this long-horizon software evolution challenge. Constructed from release notes of seven mature open-source Python projects, SWE-EVO comprises 48 tasks requiring multi-step modifications spanning an average of 21 files, validated against test suites averaging 874 tests per instance. Experiments reveal a striking capability gap: GPT-5.4 with OpenHands achieves only 25% on SWE-EVO versus 72.80% achieved by GPT-5.2 on SWE-Bench Verified, showing that current agents struggle with sustained, multi-file reasoning. We also propose Fix Rate, a metric capturing partial progress on these complex, long-horizon tasks.
References (20)
SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering
John Yang, Carlos E. Jimenez, Alexander Wettig et al.
OpenHands: An Open Platform for AI Software Developers as Generalist Agents
Xingyao Wang, Boxuan Li, Yufan Song et al.
MemGPT: Towards LLMs as Operating Systems
Charles Packer, Vivian Fang, Shishir G. Patil et al.
Analysis Of Software Maintenance Cost Affecting Factors And Estimation Models
Chamkaur Singh, N. Sharma, Narender Kumar
MemoryBank: Enhancing Large Language Models with Long-Term Memory
Wanjun Zhong, Lianghong Guo, Qi-Fei Gao et al.
CREATOR: Tool Creation for Disentangling Abstract and Concrete Reasoning of Large Language Models
Cheng Qian, Chi Han, Y. Fung et al.
Voyager: An Open-Ended Embodied Agent with Large Language Models
Guanzhi Wang, Yuqi Xie, Yunfan Jiang et al.
Large Language Models as Tool Makers
Tianle Cai, Xuezhi Wang, Tengyu Ma et al.
CodeTF: One-stop Transformer Library for State-of-the-art Code LLM
Nghi D. Q. Bui, Hung Le, Yue Wang et al.
In-context Autoencoder for Context Compression in a Large Language Model
Tao Ge, Jing Hu, Xun Wang et al.
Language Agent Tree Search Unifies Reasoning Acting and Planning in Language Models
Andy Zhou, Kai Yan, Michal Shlapentokh-Rothman et al.
SWE-bench: Can Language Models Resolve Real-World GitHub Issues?
Carlos E. Jimenez, John Yang, Alexander Wettig et al.
Large Language Models for Software Engineering: Survey and Open Problems
Angela Fan, Beliz Gokkaya, Mark Harman et al.
Functional Overlap Reranking for Neural Code Generation
H. To, Minh Huynh Nguyen, Nghi D. Q. Bui
Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection
Akari Asai, Zeqiu Wu, Yizhong Wang et al.
A Survey on Large Language Models for Software Engineering
Quanjun Zhang, Chunrong Fang, Yang Xie et al.
TroVE: Inducing Verifiable and Efficient Toolboxes for Solving Programmatic Tasks
Zhiruo Wang, Daniel Fried, Graham Neubig
RAPTOR: Recursive Abstractive Processing for Tree-Organized Retrieval
Parth Sarthi, Salman Abdullah, Aditi Tuli et al.
Executable Code Actions Elicit Better LLM Agents
Xingyao Wang, Yangyi Chen, Lifan Yuan et al.
LLM-Based Multi-Agent Systems for Software Engineering: Literature Review, Vision, and the Road Ahead
Junda He, Christoph Treude, David Lo
Cited By (20)
SWE-Chain: Benchmarking Coding Agents on Chained Release-Level Package Upgrades
SWE-Bench ProMax: Benchmarking Agents on Large-Scale Multilingual Code Refactoring
The Conversations Beneath the Code: Triadic Data for Long-Horizon Software Engineering Agents
The Kitchen Loop: User-Spec-Driven Development for a Self-Evolving Codebase
PBT-Bench: Benchmarking AI Agents on Property-Based Testing
VibeServe: Can AI Agents Build Bespoke LLM Serving Systems?
Breaking, Stale, or Missing? Benchmarking Coding Agents on Project-Level Test Evolution
ProgramBench: Can Language Models Rebuild Programs From Scratch?
More Is Different: Toward a Theory of Emergence in AI-Native Software Ecosystems
Problem Reductions at Scale: Agentic Integration of Computationally Hard Problems
Needle in the Repo: A Benchmark for Maintainability in AI-Generated Repository Edits
A Multi-agent AI System for Deep Learning Model Migration from TensorFlow to JAX
SWE Refactor Bench: Can Coding Agents Complete a Long-Horizon, Whole-Repository Stack Migration?
SlopCodeBench: Benchmarking How Coding Agents Degrade Over Long-Horizon Iterative Tasks
SWE-Milestone: Evaluating AI Agents on Continuous Software Evolution
SWE-CI: Evaluating Agent Capabilities in Maintaining Codebases via Continuous Integration
SWE-Hub: A Unified Production System for Scalable, Executable Software Engineering Tasks
SWE-AGI: Benchmarking Specification-Driven Software Construction with MoonBit in the Era of Autonomous Agents
Agentic Software Issue Resolution with Large Language Models: A Survey
Towards Autonomous Software Development