SWE Refactor Bench: Can Coding Agents Complete a Long-Horizon, Whole-Repository Stack Migration?
Introduces SWE Refactor Bench with a three-stage evaluation for long-horizon repository migration; only 5.4% success rate across 520 runs, highlighting current model limitations.
Key Findings
Methodology
The study develops a three-stage evaluation protocol: Migration Audit verifies if migration occurred, Behavioral Tests confirm behavioral consistency, and Agentic Verification employs six independent coding agents to generate targeted differential tests for hidden discrepancies. The benchmark includes 20 real-world open-source repository migrations covering language, framework, platform, and build toolchain shifts. Using 8 state-of-the-art models across 26 configurations, 520 autonomous runs were conducted. Metrics assess both migration completeness and behavioral correctness, with detailed analysis revealing the models’ strengths and weaknesses across different migration categories.
Key Results
- Out of 520 runs, only 28 (5.4%) passed all three stages; 13 tasks received no accepted solutions. The top model, claude-opus-5, scored 47.0 out of 100. Among 340 runs passing Migration Audit, 58% achieved 99% correctness, but only 26% reached full correctness. Model scores varied significantly: 31.4 in build toolchain rewrites but only 5.6 in language rewrites, indicating category-dependent performance gaps.
- The phenomenon of ‘Blindness’ was observed, where models skipped migration or broke behavior yet still passed behavioral tests, exposing the inadequacy of behavior-only evaluation. Results demonstrate that current models struggle with reliable, complete, and behavior-preserving long-term migrations, emphasizing the need for more robust verification mechanisms.
- The findings highlight the independence of migration completeness and behavioral correctness. Most models excel at behavior preservation but fail at full migration, underscoring the necessity for multi-faceted validation. This benchmark sets a new standard for evaluating automated software migration, guiding future research toward more reliable and comprehensive solutions.
Significance
This work advances the evaluation paradigm for automated software migration by integrating a rigorous three-stage protocol that addresses both the occurrence and quality of migration. It reveals critical gaps in current coding agents’ capabilities, especially in ensuring migration completeness and robustness. The benchmark provides a valuable tool for academia and industry to systematically measure progress, ultimately enabling more reliable, cost-effective, and scalable migration solutions. Its implications extend to reducing technical debt, accelerating technology upgrades, and fostering automation in software engineering, thus impacting both research and practical deployment.
Technical Contribution
The paper introduces a novel three-stage evaluation framework combining Migration Audit, Behavioral Tests, and Agentic Verification, which together form a comprehensive assessment of migration quality. It innovatively employs multiple independent coding agents to generate differential tests, exposing hidden discrepancies beyond fixed test suites. The systematic design across diverse real-world migration tasks and configurations offers a scalable, rigorous benchmark that surpasses traditional behavior-only assessments. This framework can serve as a standard for future development and benchmarking of autonomous migration tools, fostering theoretical and engineering advancements.
Novelty
This is the first comprehensive system integrating a multi-stage, adversarial evaluation protocol specifically tailored for long-horizon, whole-repository migration tasks. Unlike prior benchmarks limited to behavioral correctness, this approach explicitly verifies migration occurrence and robustness against hidden discrepancies. The use of multiple independent agents for differential testing introduces a novel adversarial dimension, significantly raising the bar for model performance and reliability. This dual focus on migration completeness and behavioral fidelity marks a significant innovation in automated software engineering evaluation.
Limitations
- Despite the rigorous evaluation, the best models achieved only 47.0 points out of 100, indicating substantial room for improvement. The models' performance varies greatly across different migration categories, especially in language and framework rewrites, revealing limitations in understanding complex code semantics.
- The benchmark relies on predefined test suites and differential test generation, which may not cover all possible discrepancies, especially in dynamic or environment-dependent behaviors. Real-world migrations may encounter external dependencies and environmental factors not modeled here.
- Computational costs remain high, with extensive resource requirements for large models and multiple configurations, limiting immediate industrial scalability. Future work should focus on efficiency and scalability enhancements.
Future Work
Future directions include integrating multimodal data (e.g., static analysis, dynamic profiling) to improve migration fidelity, employing reinforcement learning to enhance model decision-making, and developing adaptive verification mechanisms. Expanding the benchmark to include more diverse and complex migration scenarios will better reflect industry needs. Additionally, optimizing model efficiency and reducing computational costs are critical for practical deployment. Long-term goals involve creating fully autonomous, reliable migration systems capable of handling evolving software ecosystems with minimal human intervention.
AI Executive Summary
Software systems evolve over decades, accumulating technical debt that hampers modernization efforts. Upgrading or migrating entire codebases to new stacks is a complex, costly process traditionally reliant on manual labor, prone to errors and inconsistencies. Despite advances in coding agents capable of localized bug fixing, their application to long-horizon, whole-repository migrations remains unproven. Existing benchmarks primarily evaluate behavioral correctness, which can be gamed by superficial copying, leading to false positives in migration success.
To address this, the authors introduce SWE Refactor Bench—a comprehensive benchmark designed to rigorously evaluate the ability of autonomous coding agents to perform long-term, full-repository migrations. The benchmark employs a three-stage evaluation protocol: Migration Audit verifies the actual occurrence of migration, Behavioral Tests ensure behavioral fidelity, and Agentic Verification uses six independent agents to generate targeted differential tests for hidden discrepancies. This multi-layered approach effectively prevents shortcuts and blind copying, providing a more realistic assessment of model capabilities.
The benchmark encompasses 20 real-world open-source repository migrations across language, framework, platform, and build toolchain categories. Using 8 cutting-edge models across 26 configurations, 520 autonomous runs were conducted. Results reveal that only 28 runs (5.4%) pass all three stages, with the best model scoring 47.0/100. Highlighting the gap between migration completeness and behavioral correctness, the study finds that models often skip migration or break behavior, underscoring the challenge of reliable automation.
These findings have profound implications for both research and industry. They demonstrate that current models are far from achieving fully reliable, long-horizon migrations, emphasizing the need for more sophisticated evaluation and training methods. The benchmark sets a new standard for future development, aiming to push the boundaries of autonomous software engineering. Ultimately, this work paves the way toward scalable, cost-effective, and trustworthy migration automation, vital for reducing technical debt and accelerating technological evolution.
Deep Analysis
Background
软件系统在数十年的发展过程中积累了大量技术债务，导致迁移成本高昂且依赖繁琐的人工操作。传统迁移方法多依赖手工重写和逐步验证，效率低、风险大。近年来，自动化编码代理在局部修复和优化方面取得显著进展，但其在长远、全仓库迁移中的能力尚未被系统验证。现有基准多关注行为正确性，忽视迁移的真实性和完整性，难以满足工业界对可靠迁移的需求。随着大规模预训练模型的发展，研究者开始探索其在软件迁移中的潜力，但缺乏系统性评估体系，限制了技术的推广应用。
Core Problem
核心问题在于如何确保编码代理在进行长远全仓库迁移时，不仅能保持系统行为一致，还能确保迁移操作的真实性和完整性。现有评估多停留在行为测试层面，容易被‘盲目复制’所欺骗，无法验证迁移是否真正完成。迁移任务复杂度高，涉及多种技术债务类别，模型在多任务、多目标优化中表现有限。缺乏严苛的、多维度的验证机制，使得工业应用中迁移的可靠性难以保障。
Innovation
本文创新点在于提出三阶段评估协议：迁移审计验证迁移是否发生，行为测试确保行为一致，代理差异检测发现隐藏差异。结合真实开源仓库迁移任务，设计多模型、多配置试验体系，系统性揭示模型在复杂迁移中的能力边界。引入差异生成机制，增强验证的全面性和鲁棒性。该框架突破了传统行为测试的局限，为自动化迁移提供了科学、严苛的评估标准，推动了软件工程自动化的理论与实践创新。
Methodology
- �� 任务定义：从真实仓库出发，明确迁移目标和观察接口。• 迁移过程：模型接收原仓库、目标堆栈、指令，自动重写代码，构建并验证。• 迁移审计：检查迁移是否完成，旧堆栈是否被替换。• 行为测试：用130,118个检查点验证行为一致性。• 差异检测：六个独立代理生成差异测试，验证隐藏差异。• 评分机制：结合三阶段结果，量化迁移完整性与行为保持。• 实验设置：采用8个模型、26配置、20任务，全面评估迁移能力。
Experiments
采用真实开源仓库（如SQLite、zlib、libsodium、GraphHopper）作为迁移对象，涵盖语言、框架、平台、构建工具。每个任务由模型自主执行，时间从6到30小时不等。评估指标包括迁移成功率、行为一致性、差异检测成功率。对比不同模型和配置，分析迁移完整性与行为保持的关系。通过统计分析，揭示模型在不同类别任务中的表现差异，验证评估体系的严苛性。
Results
仅5.4%的试验（28/520）成功通过全部三阶段，显示模型在复杂迁移中的局限性。最佳模型claude-opus-5得分47.0，远低于理想值。340次试验中，58%达成99%的行为保持，26%达成100%，但迁移完整性仍不足。模型在构建工具链重写任务中表现较好（得分31.4），在语言重写中表现极差（得分5.6），显示迁移类别差异明显。多模型、多配置的结果表明，自动化迁移仍需突破。
Applications
该基准可用于评估自动化迁移工具的能力，帮助开发更可靠的编码代理。工业界可借助此体系验证迁移方案的有效性，降低迁移成本，提升系统升级效率。未来可结合持续集成、差异分析等技术，推动软件系统的自动化演进，实现大规模、低成本的技术栈升级。
Limitations & Outlook
模型在复杂迁移任务中的成功率仍偏低，特别是在语言和框架重写方面。评估体系依赖预定义测试和差异生成，可能未覆盖所有潜在差异场景。实际应用中，迁移的完整性和行为一致性可能受环境变化影响。实验成本较高，模型在大规模迁移中的计算资源消耗较大，限制了实际推广。未来需优化算法效率，增强迁移的鲁棒性和实用性。
Plain Language Accessible to non-experts
想象你在厨房里做饭，要把一道菜从一个厨房搬到另一个厨房。原来的厨房设备和调料都不同，你需要重新准备食材、调整火候，确保味道不变。这就像软件迁移，要把程序从一种技术环境搬到另一种环境，不能只复制粘贴，要确保它还能正常工作、味道一样。现在的厨师（模型）有点像新手，能做一些简单菜，但面对复杂菜肴（大规模软件迁移）还不够成熟。我们设计了一个三步流程：第一步确认菜是否真的搬过去了（迁移审计），第二步尝尝味道（行为测试），第三步用专业厨师（代理）找出隐藏的差异。这个方法帮助我们判断厨师是否真正完成了搬迁，味道是否还一样，确保每一步都靠谱。最终目标是让厨房自动化，减少人工操作，做出既快又好吃的菜。
ELI14 Explained like you're 14
想象你在学校里换班级，你要把所有的书、文具都搬到新班级，但不能只把东西搬过去，还要确保新班级的学习方式和原来一样。现在，有些机器人（模型）可以帮你搬东西，但它们还不够聪明，不能保证搬完后一切都和以前一样。我们设计了一个三步检查：第一步，确认东西是不是都搬过去了（迁移审计）；第二步，试着用新班级的方法学习，看看是不是还一样（行为测试）；第三步，让几个不同的机器人帮忙找出搬家后可能遗漏或出错的地方（差异检测）。通过这三步，我们可以知道机器人是不是真正完成了搬家任务，而且搬完后还能正常学习。这就像让机器人帮忙搬家一样，确保搬得又快又稳，不会出错。未来，我们希望让机器人变得更聪明，能自己完成整个搬家过程，减少人类的干预。
Abstract
Modern software systems accumulate technical debt over decades of development, which makes migration expensive and largely manual. As coding agents become increasingly capable at bug fixing, can they autonomously perform such migrations? Existing benchmarks cannot answer this question because they evaluate only behavioural correctness, not whether the migration actually occurred. This leads an easy hack: agents copy the original implementation to make tests pass. We call this Blindness. To address this problem, we introduce SWE Refactor Bench, a benchmark comprising 20 whole-repository migrations, covering 4 kinds of technical debt. A three-stage evaluation protocol measures both migration completeness and behavioural correctness. (1) Migration Audit verifies that the migration occurred. (2) Behavioural Tests measure correctness with a fixed test suite. (3) Agentic Verification uses 6 independent coding agents to generate targeted tests for hidden behavioural differences. Across 520 runs from 8 frontier models and 26 model-effort configurations, only 28 of 520 runs ($5.4\%$) pass all three stages, 13 of the 20 tasks receive no accepted solution, and the best model (claude-opus-5) scores $47.0/100$. Migration completeness and behavioural correctness are distinct abilities: a few runs preserve behaviour by skipping the migration and are stopped at Migration Audit; most attempt it and break behaviour, and are stopped at Behavioural Tests. Agents cannot deliver a perfect migration: among the 340 runs that pass Migration Audit, $58\%$ reach $99\%$ of the fixed checks, yet only $26\%$ reach $100\%$. Agent capability differs across migration categories: agents score $31.4$ on build toolchain rewrites but only $5.6$ on language rewrites. Together, these findings position SWE Refactor Bench as a rigorous testbed for developing coding agents for reliable whole-repository migrations.