SWE-Milestone: Evaluating AI Agents on Continuous Software Evolution
DeepCommit reconstructs verifiable Milestone DAGs for evaluating AI agents on continuous software evolution, revealing performance drops from 80% to 38%.03%.
Key Findings
Methodology
This study introduces DeepCommit, an automated pipeline combining static analysis, LLM-guided milestone construction, and runtime validation to reorganize noisy commit logs into reliable Milestone DAGs. Static analysis extracts commit, symbol, and file change signals, while Claude Opus 4.5 guides the inference of functionally cohesive milestones. Testbed construction and validation ensure executability. The resulting data supports the SWE-Milestone benchmark, which evaluates 12 state-of-the-art models across multiple languages in continuous development scenarios, emphasizing system integrity and error control.
Key Results
- Models achieve over 80% scores on isolated tasks but drop to 38.03% in continuous settings; Gemini 3 Pro’s Resolve Rate is only 13.37%, highlighting the challenge of long-term maintenance.
- Recall increases linearly, indicating feature addition capability, but Precision saturates, reflecting difficulty in preventing regressions. Error accumulation hampers downstream progress, causing development stalls.
- Proactive exploration and verification strategies significantly improve sustained evolution, reducing error propagation and technical debt, demonstrating their critical role.
Significance
This work addresses a critical gap in AI evaluation for software engineering by modeling long-term, dependency-rich development processes. It provides a quantitative framework for assessing model robustness over extended horizons, facilitating progress toward autonomous, reliable software maintenance. The benchmark bridges theoretical research and industrial needs, promoting AI-driven continuous integration and long-term system stability, thus advancing both academia and industry.
Technical Contribution
The paper presents DeepCommit, an end-to-end pipeline that reconstructs dependency-aware Milestone DAGs from noisy commit histories, integrating static analysis, LLM validation, and runtime testing. It introduces the SWE-Milestone benchmark, covering multiple languages and repositories, to evaluate models in realistic continuous evolution scenarios. The analysis of performance degradation and error propagation mechanisms offers new insights into long-term AI-assisted software maintenance, setting a foundation for future improvements.
Novelty
This is the first work to incorporate Milestone-level dependency graphs into continuous software evolution benchmarks, leveraging static analysis and large language models for automated, verifiable reconstruction. Unlike prior isolated or release-based evaluations, it captures the intricate dependency structure of ongoing development, enabling a more faithful assessment of AI agents’ long-term maintenance capabilities, representing a significant methodological advance.
Limitations
- The approach requires substantial computational resources, especially for large repositories, limiting scalability. The pipeline’s accuracy depends on static analysis quality and model inference, which may falter in complex or unconventional workflows.
- Current validation mainly relies on static and limited dynamic checks, potentially missing subtle bugs or regressions. Handling non-standard development practices remains challenging.
- Model performance varies across languages and repository sizes; further robustness and efficiency improvements are needed for industrial deployment.
Future Work
Future research will focus on integrating more powerful multimodal models to enhance Milestone reconstruction accuracy, extending the benchmark to multi-repo and multi-language environments, and developing theoretical models to better understand error propagation. Additionally, efforts will aim to reduce computational costs and improve robustness, facilitating industrial-scale adoption of continuous AI-driven software maintenance tools.
AI Executive Summary
In the rapidly evolving landscape of software development, maintaining long-term system integrity amid continuous updates remains a formidable challenge. Traditional AI evaluation benchmarks focus on isolated tasks or snapshot-based assessments, which fail to capture the intricate dependencies and error accumulation inherent in real-world software evolution. This gap hampers progress toward autonomous systems capable of long-term maintenance. To address this, the authors introduce DeepCommit, a novel pipeline that reconstructs verifiable Milestone DAGs from noisy commit histories using static analysis, large language models, and runtime validation. This approach ensures the generated milestones are executable and testable, forming the backbone of the SWE-Milestone benchmark.
The benchmark evaluates 12 cutting-edge models across multiple programming languages and repositories, simulating real development scenarios with dependency-constrained milestones. Results reveal a stark performance degradation: models achieve over 80% accuracy on isolated tasks but only 38.03% in continuous evolution settings, with the Resolve Rate dropping to 13.37%. The analysis uncovers that while models can add new features effectively, they struggle with system maintenance, error propagation, and regression prevention. These issues lead to a snowball effect, stalling development over time.
The study emphasizes the importance of proactive exploration and rigorous verification strategies, which significantly mitigate error accumulation. This work advances the understanding of AI’s capabilities and limitations in long-term software maintenance, providing a practical evaluation framework aligned with industrial needs. Looking ahead, future efforts will focus on enhancing the efficiency and robustness of Milestone reconstruction, expanding the benchmark’s scope, and integrating multimodal models to better support industrial-scale continuous integration and autonomous software evolution.
Deep Analysis
Background
Software development已从传统的线性流程演变为持续集成、持续部署的复杂体系。近年来，诸如CodeX、GitHub Copilot等AI驱动工具在单一任务中取得突破，但缺乏对长周期、多任务依赖关系的系统评估。现有基准多关注孤立的代码片段或版本快照，未能反映软件演化的连续性与复杂性。随着大模型的发展，自动化评估连续软件演化的需求日益增长，亟需建立能捕捉时间依赖、技术债积累的评估体系。
Core Problem
当前评估方法多集中于单一任务或快照状态，难以反映软件在长时间、多任务、多依赖环境中的维护难题。模型在连续演化中易出现误差累积、系统退化，导致开发停滞。缺乏有效的工具和指标衡量模型在复杂依赖关系中的表现，限制了AI在工业软件维护中的应用推广。
Innovation
提出DeepCommit，结合静态分析、LLM验证与运行时测试，自动重建高可信度的Milestone DAG，确保依赖关系的正确性。设计SWE-Milestone基准，模拟真实连续开发场景，评估模型在多任务、多语言环境中的表现。创新点包括里程碑级别的任务划分、动态验证机制，以及系统误差传播的分析，为长周期软件维护提供新思路。
Methodology
- �� 采集仓库提交历史，利用静态分析提取提交、符号和文件变化信息。• 由LLM模型（Claude Opus 4.5）引导构建Milestone DAG，包括种子发现、合并、依赖推断和拆分。• 结合测试环境验证，确保每个里程碑的可执行性。• 采用多轮验证与修复机制，确保DAG的正确性。• 最终生成可验证的演化轨迹，作为评估基础。
Experiments
在7个开源仓库（涵盖Python、C++等）上，构建98个人工验证的里程碑，评估12个模型。指标包括Score（结合召回率和精确率）和Resolve Rate。对比孤立任务表现，分析连续场景中的性能退化。采用ablation验证不同组件对效果的贡献，测试模型在不同复杂度场景中的鲁棒性。
Results
模型在孤立任务中平均得分超过80%，但连续场景下降至38.03%，Gemini 3 Pro的Resolve Rate仅13.37%。Recall持续增长，反映新功能实现能力，但Precision趋于饱和，说明系统维护难度大。错误积累导致开发停滞，验证主动探索和验证策略能部分缓解问题，验证了误差传播机制的关键作用。
Applications
该评估体系可应用于自动化软件维护、持续集成优化、AI驱动的代码演化管理。为工业界提供量化模型性能的工具，推动自动修复、技术债管理等关键技术的发展。未来可结合大模型，提升长周期演化的自动化能力，助力软件行业实现更高效的持续集成。
Limitations & Outlook
当前方法对大型仓库的计算成本较高，重建过程耗时长。模型在复杂依赖或非标准流程中表现有限，鲁棒性待提升。测试验证主要依赖静态分析，可能漏掉部分潜在错误。未来需优化算法效率，增强适应性，降低成本。
Plain Language Accessible to non-experts
想象你在管理一个大型工厂，每天都有不同的任务和新产品需要生产。为了确保每个环节都顺利进行，你需要知道每个任务的前后关系，不能随意跳过或重复。传统方法就像只看每个工序的快照，不能理解整体流程。本文提出一种智能系统，像是给工厂设计了一套智能调度员，能自动整理所有任务的关系，确保每一步都合理，且能追踪每个任务的变化。这样，工厂就能持续高效地生产新产品，即使遇到问题也能及时修正，不会堆积技术债。这个系统用大模型帮忙分析和验证，确保每个环节都可靠，最终实现工厂的长远稳定运行。
ELI14 Explained like you're 14
想象你在玩一个超级复杂的积木游戏，每次你都要搭建不同的城堡。每次搭完一部分，你都要记住下一步怎么搭，不能乱来。以前的人只看每次搭完的城堡快照，不能理解整个搭建过程，也不能保证每次都搭得稳。现在，这个新方法就像给你一个智能助手，它能帮你整理每个搭建步骤的关系，确保每一块积木都放得对，不会倒塌。它还能检查你搭的城堡是不是牢固，遇到问题还能帮你修正。这样，你就可以不断建出更漂亮、更稳固的城堡，而且不用担心积木堆得乱七八糟。这个助手用大脑一样的模型，帮你分析和验证，保证整个搭建过程顺利进行，城堡越建越好。
Abstract
Real-world software must continuously evolve to meet ever-changing and open-ended requirements. AI agents, increasingly deployed as long-running systems, are now entrusted to drive this evolution. Yet, existing benchmarks evaluate agents on isolated, one-off coding tasks, neglecting the temporal dependencies and technical debt inherent in real-world software evolution. To bridge this gap, we introduce DeepCommit, an agentic pipeline that reconstructs verifiable Milestone DAGs from noisy commit logs, where milestones are defined as functionally cohesive development goals. These executable sequences enable SWE-Milestone, a benchmark that evaluates agents on streams of milestone-level tasks, requiring them to sustain system integrity and limit error accumulation, dimensions of long-term software evolution largely missing from current benchmarks. Our evaluation of 12 frontier models across 4 agent frameworks reveals a critical vulnerability: overall performance scores drop significantly from >80% on isolated tasks to 38.03% in continuous settings, exposing agents' profound struggle with long-term maintenance and error propagation.