LoopsBench: From Harness Engineering to Loop Engineering in Coding Agent Evaluation
LOOPSBENCH introduces a long-horizon benchmark for loop engineering, evaluating coding agents on dependency DAGs with flow-aware runtime, revealing persistent challenges.
Key Findings
Methodology
LOOPSBENCH constructs tasks as dependency DAGs where nodes are testable development units, with edges representing prerequisites. It employs a flow-aware runtime that releases tests along the ready frontier, retains completed nodes as regression obligations, and records loop traces for analysis. The benchmark sources data from authentic repositories across 8 languages and 9 domains, totaling 112 tasks with over 5300 units. The evaluation involves models paired with various loop implementations, analyzing their ability to maintain dependency structures, state continuity, and manage regressions through detailed trajectory analysis. Metrics include dependency recovery, patch length, test coverage, and regression frequency, providing a comprehensive assessment of long horizon execution capabilities.
Key Results
- The strongest configuration (Opus-4.7 + Claude Code + outer continuation) resolves only 25% of tasks, highlighting the difficulty of sustained long horizon tasks.
- Most models fail to fully recover source dependencies, produce longer patches, and exhibit frequent regressions, indicating significant room for improvement.
- Closed-source loop implementations better approximate source dependency structures, while open-source implementations tend to follow linear flows, limiting progress.
- Persistent issues in dependency maintenance, state tracking, and regression management suggest these are key bottlenecks for long horizon performance.
Significance
This study pioneers a comprehensive long horizon benchmark tailored for loop engineering, addressing a critical gap in evaluating AI agents’ ability to sustain multi-stage, dependency-rich software development. It advances the field by providing quantifiable metrics and detailed trajectory analysis, enabling targeted improvements in models’ dependency management, state continuity, and regression control. The benchmark’s realistic tasks, sourced from real repositories, make it highly relevant for industrial applications such as automated code maintenance, continuous integration, and AI-assisted development. By exposing fundamental limitations, it guides future research toward more robust, persistent, and autonomous coding agents capable of handling complex, multi-phase projects over extended periods.
Technical Contribution
LOOPSBENCH introduces a dependency DAG-based task structure combined with a flow-aware runtime that dynamically manages test releases and regression obligations. It innovatively integrates source evidence-driven task construction, ensuring authenticity and diversity, and employs trajectory analysis to dissect model performance across planning, implementation, and testing axes. This approach enables detailed diagnostics of dependency recovery, patch complexity, and regression frequency, providing a systematic framework for evaluating long horizon capabilities. The methodology surpasses traditional static benchmarks by emphasizing intermediate obligations and dynamic dependency maintenance, offering new insights into persistent execution challenges and guiding the development of more capable AI coding agents.
Novelty
This is the first benchmark explicitly designed for loop engineering evaluation in long horizon tasks, combining dependency DAGs with flow-aware runtime diagnostics. Unlike prior static or terminal-focused benchmarks, LOOPSBENCH emphasizes intermediate obligation preservation, dependency recovery, and regression management, aligning evaluation more closely with real-world software development processes. Its source evidence-driven task construction and trajectory analysis provide a novel, detailed perspective on model performance, setting a new standard for assessing persistent, multi-stage AI coding systems.
Limitations
- Despite improvements, the best models resolve only 25% of tasks, indicating significant gaps in dependency maintenance and state management capabilities.
- Test coverage remains sparse, and patches tend to be longer than source references, affecting practical efficiency and reliability.
- The current evaluation relies on static dependency DAGs, which do not capture dynamic dependency changes or multi-task interactions, limiting real-world applicability.
Future Work
Future efforts will focus on enhancing models’ global planning, dynamic dependency modeling, and robust state tracking to improve long horizon performance. Incorporating reinforcement learning and adaptive scheduling could enable more autonomous and resilient agents. Expanding the task set to include more diverse domains and dynamic dependencies will increase benchmark relevance. Additionally, integrating real-time dependency updates and multi-agent collaboration scenarios will better simulate complex software environments, pushing the frontier of AI-assisted long horizon software engineering.
AI Executive Summary
As software systems grow increasingly complex, traditional evaluation methods for coding agents fall short in measuring their ability to sustain multi-stage, dependency-rich development over extended periods. Existing benchmarks often focus on isolated tasks or final success metrics, neglecting the intermediate obligations, dependency maintenance, and regression control critical for real-world applications. Recognizing this gap, the authors introduce LOOPSBENCH, a pioneering long horizon benchmark designed explicitly for loop engineering evaluation.
LOOPSBENCH models each task as a dependency DAG, where nodes are testable development units and edges encode prerequisite relations. This structure enables detailed tracking of intermediate obligations and provides source evidence to ensure task authenticity. The flow-aware runtime releases tests along the ready frontier, retains completed nodes as regression obligations, and records loop traces for comprehensive trajectory analysis. The benchmark sources data from authentic repositories across multiple domains and languages, creating a diverse and realistic evaluation environment.
Experimental results reveal that even the most advanced configurations, such as Opus-4.7 with Claude Code and external continuation, resolve only 25% of tasks, underscoring the persistent challenges in long horizon software development. Models struggle to fully recover source dependencies, produce concise patches, and maintain state continuity, with regressions frequently observed. These findings highlight the need for improved global planning, dynamic dependency management, and more robust state tracking mechanisms.
LOOPSBENCH’s introduction marks a significant step toward more realistic and comprehensive evaluation of AI coding agents, fostering innovations that could eventually lead to autonomous, reliable, and scalable software development systems capable of managing complex projects over time. Despite current limitations, this work lays a solid foundation for future research aimed at overcoming the inherent difficulties of long horizon tasks in AI-assisted software engineering.
Deep Analysis
Background
软件工程正逐步向自动化和智能化迈进，编码代理成为关键推动力量。早期如Codex、GPT系列主要解决单一任务或终态成功，缺乏对多阶段、多依赖环境的系统评估。SWE-bench等静态基准虽扩展任务范围，但未充分揭示模型在持续执行中的中间义务维护和回归压力。近年来，长 horizon 任务成为研究热点，旨在模拟真实软件开发中的多阶段协作和状态管理，但缺乏统一的评估平台。本文基于此背景，提出LOOPSBENCH，旨在填补长 horizon 任务评估的空白，推动编码代理在复杂环境中的应用。
Core Problem
现有评估方法难以衡量模型在长 horizon 任务中的持续性和依赖维护能力。传统静态终端任务无法反映中间开发义务的保持与回归压力，导致模型在实际软件开发中表现不足。长 horizon 任务涉及多阶段、多依赖关系，模型需在状态连续、回归控制和任务调度中表现出色，但现有技术缺乏系统性评估工具。这限制了模型在复杂软件工程中的应用潜力，也阻碍了长 horizon 任务的深入研究。
Innovation
本文创新点在于：1）提出基于依赖DAG的长 horizon任务结构，明确中间开发单元和依赖关系；2）结合流感知运行时，动态控制测试释放和状态追踪，提升持续执行能力；3）引入源证据驱动的任务构建流程，确保任务的真实性和多样性；4）通过轨迹分析揭示模型在依赖维护、状态连续和回归管理中的表现差异。这些创新使得评估更贴近实际软件开发场景，填补了长 horizon 任务评估的空白，具有重要的理论和实践意义。
Methodology
- �� 任务采集：从真实源中收集112个任务，涵盖不同领域和语言。
- �� 预处理：将源代码、提交历史和论文资料转化为原子开发单元，建立依赖DAG。
- �� 任务筛选：基于时间跨度和规模阈值筛选出符合长 horizon 要求的任务。
- �� 关系恢复：利用源证据（如代码调用、导入关系）构建依赖边，确保任务的真实性。
- �� 任务构建：在每个任务中，定义开发单元、依赖关系和测试环境，确保可测试性。
- �� 运行时控制：采用流感知机制，逐层释放测试，保持已完成节点作为回归义务，记录轨迹。
- �� 评估指标：包括依赖关系恢复度、补丁长度、测试覆盖和回归事件频率，全面衡量模型表现。
Experiments
实验采用真实源数据，涵盖学术、开源和工业任务，评估不同模型（如Claude、GPT-5.5、Codex）在长 horizon 任务中的表现。指标包括任务解决率、依赖恢复度、补丁长度、测试覆盖率和回归事件。通过对比不同循环实现（开源与闭源）和模型配置，分析模型在维护依赖、状态连续和回归控制上的差异。实验还包括 ablation 研究，验证流感知机制和源证据的重要性。结果显示，最优配置解决率仅达25%，强调长 horizon 任务的挑战性。
Results
模型在长 horizon 任务中的解决率明显偏低，最高仅达25%。依赖关系恢复不足，补丁偏长，测试覆盖有限，回归事件频繁。闭源实现更接近源依赖结构，开源实现多表现为线性流程。模型在状态连续性和回归管理方面表现不足，未来需引入更复杂的全局规划和动态依赖建模。实验验证了流感知机制在提升持续性方面的作用，提供了明确的改进方向。
Applications
该基准可用于评估自动化软件开发工具、持续集成系统和AI辅助编程平台的长 horizon 任务能力。适用于工业界的持续集成、自动化测试和代码维护场景，为模型优化提供量化指标。未来，结合该基准，研发更智能的编码代理，有望实现更高效、可靠的自动化软件开发流程。
Limitations & Outlook
当前模型解决率仍低，显示在复杂依赖和状态管理方面存在明显不足。测试覆盖不足，补丁偏长，影响实际应用效率。评估基于静态依赖DAG，未充分考虑动态变化和多任务交互。未来需引入动态依赖建模和更高效的状态追踪机制，以提升整体性能。
Plain Language Accessible to non-experts
想象你在经营一家大型工厂，工厂里有许多不同的车间，每个车间负责生产不同的零件。这些零件需要按照一定的顺序组装，才能变成完整的产品。工厂的管理系统就像一个大脑，要确保每个车间在正确的时间开始工作，等待前面车间完成任务。现在，如果工厂的管理系统只关注最后的成品，忽略了中间的零件和流程，就可能出现问题。LOOPSBENCH就像是给这个工厂设计了一套智能管理工具，它能追踪每个车间的工作进度，确保每个零件都按顺序生产，避免遗漏或重复。这样，工厂才能高效、连续地生产出复杂的产品，而不是只关注最终的成品。这种方法帮助我们理解复杂任务的每一步，确保整个过程顺利进行。
ELI14 Explained like you're 14
想象你在玩一个超级复杂的拼图游戏，你需要把很多小块拼在一起，才能拼出完整的图片。每一块都依赖前面某几块，不能乱拼。现在，如果你只关心最后拼好的图片，可能会忽略一些重要的步骤，比如某些块必须先拼好才能拼其他的。LOOPSBENCH就像是一个智能拼图助手，它能帮你记住每一块的拼接顺序，确保每一步都正确，避免拼错或漏掉重要的部分。它还能告诉你哪些块已经拼好，哪些还需要拼，帮助你一步步完成整个拼图。这样，你就可以像专业拼图高手一样，把复杂的拼图变得简单又有序，最终拼出漂亮的图片。这种方法让我们理解复杂任务的每个环节，确保每一步都按计划进行，避免出错。
Abstract
Coding agent infrastructure is shifting from harness engineering toward loop engineering as coding agents are deployed for sustained long-horizon software development. Existing benchmarks often center on localized tasks or end-state outcomes, offering limited insight into sustained execution. We introduce LOOPSBENCH, a long-horizon benchmark for loop engineering in coding agent evaluation. Each task is a dependency DAG over separately testable development units with source-evidenced prerequisite edges. LOOPSBENCH comprises 112 tasks from authentic sources spanning 8 programming languages and 9 domains. Its flow-aware runtime releases tests along the ready frontier and retains completed nodes as regression obligations. We evaluate frontier coding agents paired with widely used loop implementations. The strongest configuration, Opus-4.7 with Claude Code and outer continuation, resolves 25.00% of tasks. Recorded plans recover only part of the source-recovered prerequisite DAG, and regression events remain visible across the evaluated loop profiles. We open source the benchmark data and code, including all tasks, more than 5,300 development units, and executable tests, at microsoft/Loopsbench.