BeyondSWE: Can Current Code Agent Survive Beyond Single-Repo Bug Fixing?
BeyondSWE benchmark evaluates code agents on cross-repository, domain-specific, dependency migration, and document generation tasks; top model scores 56.65, still far from saturation.
Key Findings
Methodology
This study introduces BeyondSWE, a benchmark with 246 real GitHub repositories and 500 instances across four complex tasks: cross-repo issue resolution, domain-specific fixes, dependency migration, and doc-to-repo generation. Environment reconstruction uses Docker and manual validation, ensuring realistic settings. SearchSWE is integrated as a diagnostic tool to evaluate search-augmented coding. Models like OpenHands and Codex (GPT-5.4) are tested under different prompts, with performance measured via Resolved Rate, Pass Rate, and (Almost) Correct counts. The evaluation emphasizes environment stability, task authenticity, and expert review, providing a comprehensive assessment of model capabilities in complex scenarios.
Key Results
- The best OpenHands model achieves an average score of 46.12, indicating significant room for improvement; GPT-5.4 (xhigh) with search reaches 56.65, showing search helps but is insufficient for full solution. Across tasks, models struggle with cross-repository reasoning and dependency updates, highlighting the complexity of real-world software engineering. Search-augmented models improve performance notably but still face challenges in transforming retrieved external information into precise, version-compatible code. These findings underscore the need for more robust, integrated reasoning and external knowledge fusion in future models.
Significance
This work exposes critical gaps in current AI-based code agents, especially in handling tasks requiring external knowledge, repository-wide reasoning, and complex migrations. It advances the evaluation paradigm by systematically benchmarking multi-faceted, real-world scenarios, guiding future research toward models capable of holistic understanding and reasoning across repositories. The insights gained are vital for industrial applications such as automated system upgrades, dependency management, and large-scale code synthesis, ultimately pushing AI-assisted software engineering closer to practical deployment.
Technical Contribution
The paper develops BeyondSWE, a comprehensive multi-task benchmark that challenges models beyond traditional bug fixing. It innovates by integrating search tools into the evaluation pipeline, creating a diagnostic framework for search-augmented coding. The environment automation and expert validation ensure high fidelity and reproducibility. The research demonstrates that current models, despite progress, still lack robust multi-source reasoning, highlighting the importance of external knowledge integration and repository-level reasoning for future AI systems in software engineering.
Novelty
This is the first systematic benchmark to evaluate code agents on multi-dimensional, real-world software engineering tasks involving external repositories, domain knowledge, and dependency migrations. It introduces a search-augmented diagnostic framework, providing a new perspective on how external information can enhance code generation. Unlike prior benchmarks focused solely on localized bug fixes, BeyondSWE emphasizes broader knowledge scope and resolution scope, filling a significant gap in the evaluation landscape.
Limitations
- Models still struggle with complex, multi-source reasoning, especially in dependency migration and cross-repository tasks, indicating the need for better knowledge integration. The environment reconstruction process relies heavily on manual expert validation, which may limit scalability. The benchmark primarily focuses on Python repositories, and its applicability to other languages or large-scale industrial codebases remains to be validated.
Future Work
Future research should explore multi-modal knowledge fusion, reinforcement learning strategies for reasoning, and automated environment setup to reduce manual effort. Extending the benchmark to include multi-language and multi-platform scenarios will enhance its relevance. Developing models with stronger external knowledge integration and reasoning capabilities will be crucial for practical deployment in real-world software engineering tasks.
AI Executive Summary
Automated code repair has seen rapid advancements, primarily evaluated within single-repository contexts. However, real-world software development often involves tasks that extend beyond individual repositories, requiring external knowledge, cross-repository reasoning, and system-wide transformations. Recognizing this gap, the present work introduces BeyondSWE, a comprehensive benchmark comprising 246 real-world GitHub repositories and 500 instances across four challenging categories: cross-repo issue resolution, domain-specific fixes, dependency migration, and document-to-repository generation. These tasks simulate practical scenarios where external artifacts, scientific knowledge, or dependency updates are essential, reflecting the complexity of industrial software maintenance.
The benchmark environment is meticulously reconstructed using Docker, with rigorous validation and expert review to ensure realism. To evaluate the potential of search-augmented coding, SearchSWE is incorporated, enabling models to access web search and fetch tools. Experiments with models like OpenHands and GPT-5.4 reveal that the best performing model scores only 56.65, indicating substantial room for improvement. While search tools improve performance, models still face significant challenges in transforming retrieved external information into precise, version-compatible code changes, especially in tasks involving cross-repository reasoning and dependency updates.
These findings highlight that deep search and external knowledge integration remain open problems in AI-assisted software engineering. The study underscores the necessity for future models to reliably combine external evidence with local reasoning and verification, moving toward truly holistic and robust code intelligence systems. This work not only advances benchmarking standards but also provides critical insights for developing next-generation AI tools capable of handling complex, real-world software engineering tasks, ultimately facilitating more autonomous, scalable, and intelligent software maintenance solutions.
Deep Analysis
Background
软件工程中的自动化代码修复技术经历了快速发展，早期如DeepCode、CodeX等主要关注静态分析和局部修复。近年来，预训练模型如GPT-3、Codex极大提升了代码生成能力，推动了自动化水平的提升。然而，现有研究多集中在单仓库环境，缺乏对跨仓库、领域知识和依赖迁移等复杂场景的系统评估。随着软件系统规模扩大，维护成本增加，自动化工具面临更高的挑战，亟需建立多源、多任务的评估体系，以真实反映工业需求。
Core Problem
当前模型在单仓库修复表现尚可，但在处理跨仓库信息、专业领域知识和系统迁移时表现不足。实际开发中，开发者常需参考外部文档、上游项目或进行全局迁移，模型难以充分利用外部资源，导致修复效果有限。这限制了AI在复杂软件维护中的应用潜力。解决这一问题的关键在于设计能融合多源信息、具备跨仓库推理能力的智能系统，以满足实际需求。
Innovation
本研究提出BeyondSWE多任务基准，首次系统性评估模型在跨仓库、领域知识、依赖迁移和文档生成中的能力。引入搜索增强机制，结合Web搜索和网页抓取工具，提升模型对外部信息的利用效率。创新点包括：1）多任务实例设计，模拟真实复杂场景；2）环境自动重建，确保评估公平性；3）搜索-编码融合，突破信息孤岛。这些创新推动模型在复杂场景中的表现，填补多源知识融合的研究空白。
Methodology
- �� 构建BeyondSWE基准，采集246仓库的500个实例，涵盖四类任务。• 采用环境自动重建技术，利用Shell命令和Docker确保环境一致性。• 设计多任务测试，包括跨仓库问题、领域修复、依赖迁移和文档生成。• 集成SearchSWE，结合Web搜索和网页抓取工具，增强模型外部信息访问能力。• 采用多模型评估，包括OpenHands、DeepSeek-V4-Pro和GPT-5.4（xhigh），比较不同提示策略的效果。• 通过严格的测试流程，确保每个实例的环境稳定性和任务真实性。• 结合人工专家审核，过滤掉不符合实际的软件工程场景的实例。
Experiments
在真实环境中进行评估，使用Python仓库，指标包括Resolved Rate、Pass Rate和(Almost) Correct Count。模型参数保持一致，采用不同提示策略，比较无搜索与搜索增强的效果。对比多种模型，分析其在复杂场景中的鲁棒性和知识迁移能力。所有环境经过多轮验证，确保结果的可靠性和可复现性。实验还包括消融研究，验证搜索信息对性能的贡献，确保结论具有说服力。
Results
实验显示，最优OpenHands模型平均得分为46.12，表现尚未饱和；引入搜索后，GPT-5.4（xhigh）在搜索提示下达56.65，明显优于无搜索状态的48.48。不同任务表现差异明显，跨仓库和依赖迁移任务难度较大。搜索增强虽提升部分模型性能，但在将检索信息转化为准确代码方面仍存在瓶颈，说明深度搜索与编码的结合仍需突破。整体来看，模型在复杂多源场景下的能力仍有限，未来需在知识融合和推理能力上持续优化。
Applications
该研究推动了智能化软件维护工具的发展，适用于自动修复跨仓库依赖、迁移和文档生成等场景。企业可利用此技术实现自动化代码迁移、系统升级和知识管理，减少人工成本。未来，结合多模态知识和强化学习，有望实现更智能的系统，支持大规模软件系统的自动化维护与演化。
Limitations & Outlook
模型在复杂任务中的表现仍有限，尤其在跨仓库推理和依赖迁移场景中鲁棒性不足。环境重建依赖大量人工验证，成本较高，自动化程度有待提升。实验主要集中在Python生态，跨语言、多平台场景尚未充分覆盖。未来需增强模型的多源知识融合能力，优化环境自动化流程，扩展多语言支持，提升实际应用的可行性。
Plain Language Accessible to non-experts
想象你在厨房做菜，平时只用一个锅和几种调料就能做出菜肴，但有时候你需要用到不同厨房的工具、调料，甚至参考食谱和其他厨师的经验。现在的智能厨师（模型）就像这样，它们在简单情况下表现不错，但面对复杂任务，比如跨厨房合作、使用不同食谱或迁移食材，就会遇到困难。本文的研究就像是在测试这些厨师在更复杂的厨房环境中的表现，看看他们能否利用外部信息、合作完成大菜。通过设计各种“厨房任务”，研究发现即使最好的厨师也还需要改进，未来的目标是让他们像顶级厨师一样，能灵活应对各种复杂挑战，真正成为厨房里的万能高手。
ELI14 Explained like you're 14
你知道平时做饭的时候，有时候只用一个锅就能搞定菜肴，但有时候需要参考别的厨房的食谱或者用到不同的调料，这样才能做出更好吃的菜。现在的智能厨师（就是那些会写代码的机器人）平时在修复代码时表现还不错，但遇到更复杂的任务，比如要用别的仓库的代码、参考专业领域的知识、迁移依赖或者从零搭建一个新系统，就会变得很困难。这个研究就像是在测试这些机器人在更复杂的厨房环境中能不能用外部的食谱、工具和经验帮忙做菜。结果发现，即使是最厉害的机器人，也还不能完全胜任这些复杂任务。未来，我们希望让它们像顶级厨师一样，能灵活应对各种复杂情况，帮我们更快更好地完成软件开发工作。
Abstract
Current code-agent benchmarks primarily evaluate localized issue resolution within a single target repository, leaving under-tested many software engineering tasks that require external knowledge or broader repository-level changes. We introduce BeyondSWE, a 500-instance benchmark drawn from 246 real-world GitHub repositories to evaluate code agents beyond single-repository bug fixing. BeyondSWE covers four representative settings: cross-repository issue resolution, domain-specific issue resolution, dependency-driven migration, and document-to-repository generation, spanning both broader knowledge scope and broader resolution scope. Our evaluation shows that BeyondSWE remains far from saturated: the best OpenHands-based agent reaches 46.12 average score, while the strongest Codex harness with GPT-5.4 (xhigh) reaches 56.65 under a search-aware prompt. To study whether external information access closes this gap, we use SearchSWE as a controlled diagnostic baseline for search-augmented coding. Search access improves most models and substantially helps some tasks, but the gains remain limited and uneven, showing that current agents still struggle to convert retrieved information into precise, version-compatible, and locally actionable code changes. These results suggest that deep search for coding remains an open problem: progress requires agents that can reliably combine external evidence with repository-local reasoning and execution-based verification.