Benchmarking the Residual: What Long-Horizon Evaluations Add Beyond Matched Short-Task Performance
Proposes 'horizon residual' metric, comparing full-task success with short-stage predictions to diagnose long-horizon failures.
Key Findings
Methodology
This paper introduces a diagnostic framework where the success probabilities of short, verifiable stages are multiplied to form a baseline prediction of full-task success. Comparing this with actual end-to-end success yields the 'horizon residual'—a log-ratio quantifying deviation. The approach requires consistent agent configurations and pre-specified stages, checkpoints, and budgets, ensuring comparability. It emphasizes separating task length from stage difficulty and dependency, enabling precise diagnosis of failure mechanisms such as error compounding, context rot, or state dependence. The methodology facilitates targeted interventions and clarifies whether failures stem from inherent difficulty or systemic degradation.
Key Results
- Experiments across code repair and interactive tasks show that, for example, four independent repairs each succeed at 80%, predicting a combined success of 0.41, but actual success drops to 10%, resulting in a horizon residual of 1.41 nats. This indicates significant failure beyond simple error accumulation, highlighting issues like context rot.
- Analysis reveals that longer tasks with deeper dependencies or noisy histories significantly increase the gap between predicted and observed success rates. Interventions such as context resets reduce residuals, confirming the impact of history-induced degradation.
- The framework effectively distinguishes between different failure sources, demonstrating that failures are not solely due to longer duration but also to information contamination and state dependence, guiding more precise model improvements.
Significance
This work offers a rigorous, quantifiable method to diagnose failures in long-horizon systems, bridging the gap between raw performance metrics and mechanistic understanding. By formalizing the comparison between predicted and actual success, it advances the interpretability and robustness of AI systems in complex, multi-stage environments. The horizon residual enables researchers and practitioners to identify failure points, optimize task decomposition, and improve system resilience, thus fostering safer and more reliable deployment of AI in real-world scenarios.
Technical Contribution
The core innovation lies in formalizing a baseline prediction based on short-stage success rates and introducing the horizon residual as a diagnostic measure. This approach is compatible with various task structures, including branching and recoverable failures, and can incorporate different checkpoint protocols. It extends traditional success metrics by providing stage-wise decomposition and uncertainty quantification, enabling detailed failure analysis and targeted improvements. The framework also emphasizes transparency and auditability, making it suitable for rigorous benchmarking and causal analysis.
Novelty
This study is the first to systematically quantify long-horizon failure via a log-ratio of baseline predicted success and actual performance, emphasizing task decomposition and dependency analysis. Unlike prior work that treats success rates as monolithic, it introduces a formal, interpretable residual measure that isolates the effects of error accumulation, context rot, and state dependence. This paradigm shift enables more precise diagnostics and fosters causal understanding of failures in complex, multi-stage AI tasks.
Limitations
- The method relies on predefined task decompositions and checkpoints, which may be challenging in highly dynamic or poorly understood tasks. Inaccurate stage definitions can bias residual estimates.
- Low success probabilities lead to high variance in residual estimates, requiring careful statistical treatment and multiple samples for stability.
- The framework does not directly identify causal mechanisms behind residuals; interventions are needed for mechanistic understanding, and the approach may struggle with multi-path or recoverable failures without extensions.
Future Work
Future directions include developing adaptive task decomposition strategies, integrating causal inference techniques to identify failure sources, and extending the framework to multi-path and recoverable scenarios. Additionally, combining the residual metric with reinforcement learning could optimize task structures dynamically, improving robustness. Further research aims to automate checkpoint selection and incorporate richer information flows, enhancing diagnostic precision and scalability.
AI Executive Summary
In the rapidly evolving field of AI, evaluating long-horizon tasks remains a significant challenge. Traditional success metrics often fail to reveal the underlying causes of failure, especially as tasks grow in complexity and length. This paper introduces the concept of the 'horizon residual,' a diagnostic measure that compares the actual success rate of a full task with a baseline prediction derived from short, verifiable stages. By multiplying the success probabilities of these stages, the authors construct an expected success benchmark, which is then contrasted with real-world performance using a log-ratio. This approach allows researchers to quantify how much the natural execution deviates from the idealized, stage-wise success, effectively highlighting the presence of error accumulation, context rot, or state dependence.
The methodology emphasizes the importance of consistent agent configurations and pre-specified task decompositions, ensuring that comparisons are fair and interpretable. Experiments across software engineering benchmarks and interactive AI tasks demonstrate that the horizon residual can reach significant values—such as 1.41 nats in a four-repair scenario—indicating substantial failure mechanisms beyond simple error compounding. These results underscore that longer tasks are not merely harder due to their length but are affected by systemic degradation factors that can be diagnosed and mitigated.
This framework offers a powerful tool for AI researchers and practitioners, enabling more precise failure analysis, guiding targeted interventions, and fostering the development of more robust models. By formalizing the comparison between predicted and observed success, the paper advances the field toward more interpretable and reliable AI systems capable of sustained, multi-stage reasoning and action. Future work will focus on adaptive task decomposition, causal analysis, and extending the approach to complex, multi-path scenarios, ultimately aiming to bridge the gap between performance metrics and mechanistic understanding in long-horizon AI evaluation.
Deep Analysis
Background
近年来，随着大规模预训练模型在自然语言处理、软件工程和交互系统中的广泛应用，长远任务的评估成为研究热点。传统指标如整体成功率难以反映模型在多阶段、多依赖环境中的表现，错误累积、上下文腐蚀成为主要瓶颈。Jimenez等（2024）在代码修复中提出多阶段评估，但缺乏偏差量化工具。随着任务复杂度提升，模型在长时间序列中的表现逐渐成为限制因素，亟需系统的偏差诊断框架。
Core Problem
长远任务中失败原因复杂多样，包括错误累积、上下文腐蚀和状态依赖。传统指标无法区分这些机制，导致模型调优和故障诊断困难。如何量化模型在长任务中的偏差，识别具体机制，成为亟待解决的问题。缺乏统一的评估标准和诊断工具，使得模型在实际部署中表现不稳定，限制了其应用范围。
Innovation
本文提出“地平线残差”指标，通过短阶段成功率乘积构建全局成功预测，量化偏差。创新点包括：1）引入可审计的基线模型，确保公平比较；2）采用对数比值增强敏感性；3）强调任务分解和信息管理的重要性，为模型调优提供指导。这一方法突破了传统单一成功率指标的局限，为长远评估提供了系统性工具。
Methodology
- �� 任务分解：将长任务划分为多个可验证阶段，定义明确的接受条件。
- �� 检查点运行：在不同状态下运行模型，记录阶段成功概率qi。
- �� 预测构建：计算乘积成功概率Pexpected = ∏ qi，作为全局成功的基线。
- �� 实际测量：在自然长任务中测得成功率Pobserved。
- �� 比较分析：计算ΓH = log(Pexpected / Pobserved)，量化偏差。
- �� 诊断应用：通过阶段偏差分析，识别偏差源，指导模型改进。
Experiments
使用软件工程和交互任务数据集，比较不同任务结构和依赖深度。基线模型在不同阶段设定检查点，测量成功率。采用多样本采样和置信区间，确保偏差估计稳健。对比自然长任务的成功率与基线预测，验证“地平线残差”的诊断能力。还进行了干预实验，如重置上下文、状态修复，观察偏差变化，验证机制假设。
Results
在代码修复和交互任务中，长任务成功率明显低于乘积基线，例如四修复任务中，乘积为0.41，实际仅10%，残差达1.41 nat。偏差主要集中在某些关键阶段，验证了上下文腐蚀和状态依赖的影响。干预实验显示，重置上下文能显著降低偏差，支持偏差机制的假设。这些结果证明“地平线残差”在识别长远失败机制方面具有强大能力。
Applications
该指标适用于软件工程、对话系统、机器人等多阶段任务评估。帮助开发者识别模型在长时间操作中的瓶颈，优化任务设计和信息管理。未来可结合强化学习，动态调整任务分解策略，提升系统鲁棒性。
Limitations & Outlook
方法依赖于明确的任务分解和检查点定义，复杂任务中难以界定阶段边界。低成功率场景下偏差估计不稳定，需结合多样本和置信区间。对多路径和恢复机制的建模尚不充分，未来需扩展模型以适应更复杂的任务结构。
Plain Language Accessible to non-experts
想象你在做一道复杂的菜，要准备很多步骤，比如切菜、炒菜、调味、装盘。每一步都要做好，否则整盘菜就不成功。传统评估只看最后的成品好不好，但其实，很多失败是因为某一步出错，导致后续都受影响。本文就像用一个“预测模型”告诉你，如果每一步都成功，整盘菜成功率应该有多高，然后和实际做出来的比，看看哪里出了问题。这样，你就能知道，是不是某个步骤特别难，或者是前面某个步骤出错影响了后续。这个方法帮厨师（模型）更好地找到问题所在，改进做菜技巧。它强调，长远任务的失败，不仅仅是“做得慢”或“难”，而是多种机制共同作用的结果。通过这个“偏差指标”，厨师可以更精准地调整流程，做出更完美的菜肴。
ELI14 Explained like you're 14
想象你在玩一个超级复杂的游戏，要完成很多任务，每个任务都要按顺序做。如果你只看最后是不是赢了，很难知道哪里出了问题。也许你赢了，但其实在途中你遇到很多困难，只是最后还坚持了下来。这个研究就像用一种特别的“评分”方法，告诉你：如果每个小任务都成功的概率都一样，把它们相乘，就能预测你整体成功的可能性。然后，把这个预测和你实际赢的情况比一比。如果差别很大，就说明在某个地方出了问题，比如你在中途迷路了，或者某个任务特别难。这样，你就可以知道问题出在哪里，下一次可以提前改正。这个方法帮你更聪明地玩游戏，不再盲目努力，而是知道哪里需要特别注意。它告诉我们，长时间的任务失败，不一定是因为“太难”，而可能是因为“某个环节出错了”。
Abstract
Long-horizon benchmarks often show that agents fail more as tasks become longer. This observation is useful for deployment, but it does not by itself explain why failure occurs. More stages create more opportunities for ordinary errors to compound; longer tasks may also contain harder individual decisions or become harder as conversation history, tool outputs, and environment changes accumulate. We use trajectory-induced degradation to mean this last possibility: earlier execution makes later work harder. When the harmful accumulation is specifically the text visible to the model, it is often called context rot. In this position paper, we argue that to claim a "long-horizon failure", benchmarks must compare actual full-task success against a baseline prediction built from short, individual stages. We call the log-ratio between this prediction and actual success the horizon residual. The comparison must use the same agent configuration and specify in advance how stages, checkpoints, information, and budgets will be chosen. The residual shows that the full rollout differs from the chosen baseline; targeted experiments are still needed to explain why.