Stop Comparing LLM Agents Without Disclosing the Harness
Proposes 'Binding Constraint Thesis', showing scheduler configuration impacts long-horizon LLM performance more than model upgrades, advocating for disclosure.
Key Findings
Methodology
This paper adopts a control-theoretic framework, modeling the scheduler as a closed-loop controller and the LLM as a stochastic policy. Performance variance is decomposed into model-induced (MV) and scheduler-induced (HV) components via variance analysis. Empirical data from public benchmarks, industry deployments, and controlled experiments demonstrate that scheduler configuration accounts for most performance fluctuations in long-horizon tasks. The authors propose a standardized disclosure protocol and variance decomposition method to improve evaluation transparency, emphasizing that unreported scheduler variations can misattribute gains to models, leading to unreliable cross-model comparisons.
Key Results
- On SWE-bench Pro, the same model (Claude Opus 4.5) under different scheduler configurations shows a performance difference of up to 9.5 percentage points, exceeding typical model upgrade gains. Industry monitoring reports up to 15-point performance swings solely due to scheduler changes. Controlled experiments reveal that scheduler-induced variance (HV) is on average 7.8 times larger than model-induced variance (MV), with multiple instances of model ranking reversals across different scheduler setups.
- Variance decomposition confirms that in long-horizon tasks, the performance variance attributable to scheduler configuration (HV) often surpasses that from the model itself (MV). Experiments with configurations H1, H2, H3 show that changes in scheduler design can account for over 80% of performance fluctuations. These findings challenge the conventional attribution of performance solely to model capability.
- The authors introduce a 'scheduler-aware evaluation framework' comprising a disclosure card, variance decomposition protocol, and trajectory metrics. This framework ensures transparency, enabling fair comparison and attribution of performance gains to either the model or the scheduler, thus addressing the prevalent bias in current leaderboards.
Significance
This work fundamentally shifts the paradigm of LLM evaluation by highlighting the dominant role of system architecture—specifically, the scheduler—in long-horizon tasks. It exposes the risks of attributing performance improvements solely to model scaling or fine-tuning, which can mislead research directions and deployment decisions. By advocating for transparency and standardized protocols, it promotes more reliable benchmarking, fostering progress in system-level AI engineering. The findings also underscore the importance of feedback control mechanisms in ensuring long-term reliability, robustness, and interpretability of AI agents, impacting both academia and industry.
Technical Contribution
The paper introduces a formal control-theoretic model of LLM agents, framing the scheduler as a feedback controller governing the system's stability and context management. It develops a variance decomposition approach to quantify the impact of scheduler configurations, incorporating metrics like Lyapunov stability, context drift, and control lag. The authors propose a structured disclosure protocol and a factorial experimental design to systematically evaluate the influence of scheduler parameters. This approach bridges control theory and AI evaluation, providing a rigorous foundation for system-level performance analysis and benchmarking.
Novelty
This is the first comprehensive study to formalize the role of the scheduler as a closed-loop controller in LLM agent performance, demonstrating that configuration variations can dominate model improvements in long-horizon tasks. The variance decomposition framework and the emphasis on disclosure protocols represent novel contributions, challenging the traditional focus on model-centric evaluation. The work pioneers a systemic perspective, integrating control theory with AI benchmarking, and highlights the critical need for transparency in system components that significantly influence outcomes.
Limitations
- The analysis primarily targets long-horizon tasks with comparable frontier models; applicability to short-horizon or highly disparate models remains limited. The controlled experiments are conducted in simulated environments, which may not fully capture real-world complexities. The proposed disclosure protocol requires industry-wide adoption for maximal impact, which may face practical resistance. Additionally, the framework assumes that the main sources of variance are captured by stability, drift, and lag, potentially overlooking other factors such as hardware variability or user interaction dynamics.
Future Work
Future research will explore adaptive and autonomous scheduler optimization using reinforcement learning, aiming to enhance robustness and reduce manual tuning. Extending the variance analysis to multi-modal and multi-agent systems could further improve system reliability. Developing industry standards for scheduler disclosure and benchmarking will be crucial for widespread adoption. Additionally, integrating real-time monitoring and feedback mechanisms can help dynamically adjust scheduler configurations, ensuring consistent performance in diverse deployment environments.
AI Executive Summary
The evaluation of large language models (LLMs) in long-horizon tasks has traditionally focused on the models themselves, often neglecting the crucial role of the underlying system architecture, particularly the scheduler (or harness). This oversight leads to significant issues: performance scores can vary dramatically depending on the scheduler configuration, sometimes by as much as 15 percentage points, overshadowing the actual improvements in the models. Such variability not only undermines the fairness of benchmarks but also misguides research efforts and deployment strategies.
This paper introduces the 'Binding Constraint Thesis,' asserting that in long-horizon tasks with comparable frontier models, the performance variance attributable to the scheduler (HV) can surpass that of the model (MV). To substantiate this, the authors develop a control-theoretic formalization, modeling the scheduler as a feedback controller that manages context, tool interaction, and error recovery. They employ variance decomposition to quantify how different scheduler configurations influence overall performance, revealing that small changes in the harness can produce performance shifts exceeding those from model upgrades.
Empirical evidence from public benchmarks like SWE-bench and industry reports demonstrates that the variance caused by scheduler differences often exceeds model differences by a factor of 7.8. Controlled experiments with multiple configurations (H1-H3) further confirm that scheduler-induced variance dominates, with performance fluctuations primarily driven by stability, context drift, and control lag. These findings highlight the critical need for transparency: without disclosure of the harness configuration, benchmark scores are incomplete and potentially misleading.
To address this, the authors propose a 'harness-aware evaluation framework,' including a structured disclosure card, a variance decomposition protocol, and trajectory-level metrics. This framework enables fair, interpretable comparisons, ensuring that performance gains are correctly attributed to either the model or the system architecture. Overall, this work advocates for industry-wide standards on harness disclosure, emphasizing system-level control as essential for reliable AI evaluation and deployment. Future directions include automating harness optimization, extending the analysis to multi-modal systems, and establishing transparency standards to improve the robustness and fairness of long-term AI systems.
Deep Analysis
Background
随着LLM在自然语言处理和智能系统中的广泛应用，长远任务的性能评估逐渐成为研究焦点。早期如GPT-3、BERT等模型主要关注模型规模和微调技术，代表性基准包括SuperGLUE、BIG-Bench等。然而，随着系统复杂度增加，调度器（harness）在上下文管理、工具调用、错误处理中的作用逐步凸显。行业实践中，调度器配置差异导致性能波动巨大，传统评估忽视了这一因素，造成模型性能的误导性比较。近年来，研究逐步认识到系统架构对长远任务表现的重要性，但缺乏系统性分析和量化工具。本论文在此背景下，提出调度器作为闭环控制器的理论框架，旨在揭示调度器配置对性能的深远影响，推动评估标准的变革。
Core Problem
当前评估体系普遍将性能归因于模型本身，忽略调度器（harness）对长远任务表现的决定性作用。调度器的配置差异会引起性能高达15个百分点的波动，严重影响模型排名的稳定性和评测的公平性。未披露调度器信息导致跨研究和行业应用中的性能误读，限制了系统优化的方向。解决这一问题，需建立系统性分析调度器配置影响的理论模型，并制定透明披露标准，确保评估的公平性和可比性。
Innovation
本研究创新在于将调度器（harness）作为闭环控制器引入LLM系统评估，提出Variance分解模型，定量分析调度器配置对性能的贡献。引入Lyapunov稳定性、上下文漂移（δt）和控制滞后（τ）指标，系统描述调度器的稳定性与可靠性。设计调度器披露卡和方差分解协议，确保评测透明、公平。不同于传统模型微调和规模扩展的研究，强调系统架构在实际表现中的核心作用，推动系统级AI设计和评估方法创新。
Methodology
- �� 将调度器（H）定义为闭环控制器，模型（πθ）为随机策略，系统状态（st）由调度器管理。• 采用Variance分解，将性能指标（B）拆分为模型引起的方差（MV）和调度器引起的方差（HV），以及交互项。• 设计三类调度器配置（H1、H2、H3），分别代表不同的稳定性、漂移控制和故障恢复能力。• 通过控制实验，比较不同调度器配置对模型性能的影响，验证调度器配置在长远任务中的主导作用。• 引入Lyapunov稳定性、上下文漂移（δt）和控制滞后（τ）指标，量化调度器的可靠性。• 在公开基准（如SWE-bench Verified）上进行大规模测试，验证理论模型的实用性。
Experiments
- �� 选择GPT-5.4、Kimi K2.6、GLM-5.1三款模型，设计三种调度器配置（H1、H2、H3），在受控环境中进行长远任务（如编码任务）测试。• 每个模型-调度器组合进行两次独立运行，保持任务顺序和环境一致。• 评估指标为pass@1性能，计算性能的方差（MV、HV）及其比例。• 通过调度器配置变化，观察模型性能波动，验证调度器对性能的影响是否超过模型升级。• 统计模型排名反转次数，分析调度器配置对排名稳定性的影响。
Results
- �� 实验显示，调度器配置（H1-H3）引起的性能变化平均达13个百分点，远超模型内部变化（约3-5个百分点）。•调度器配置变化引起的性能方差（HV）在平均值上是模型引起的方差（MV）的7.8倍，验证调度器在长远任务中的主导作用。• 多个模型在不同调度器下出现排名反转，说明未披露调度器会严重误导评估结果。• 通过方差分解，明确指出调度器配置在实际系统中的重要性，推动行业采用调度器披露标准。
Applications
- �� 立即应用于行业评测平台，推动调度器配置的透明披露，提升评估公平性。• 在实际部署中，优化调度器设计，提升长远任务的稳定性和可靠性，减少模型升级的盲目性。• 长远来看，推动系统级AI架构设计，结合控制论实现更鲁棒的智能系统，适应复杂多变的应用场景。
Limitations & Outlook
- �� 研究主要集中在长远任务和前沿模型，短期任务或模型差距极大的场景未充分覆盖，可能影响结论的普适性。• 调度器配置的披露依赖于实验设计，实际应用中调度器的复杂性和多样性可能带来评估难题。• 实验环境受控，实际部署中调度器的动态变化和环境干扰可能引入额外不确定性。
Plain Language Accessible to non-experts
想象一个工厂生产线，不仅仅是机器（模型）决定产品质量，还要有一个调度员（调度器）不断监控、调整流程，确保每个环节都在正确轨道上。单纯提升机器的能力（模型）固然重要，但如果调度员不善于管理，生产线还是会出错甚至崩溃。不同调度员的管理方式会让同样的机器产生完全不同的效果。这个调度员就像论文中的“调度器”，它控制着整个系统的运作，决定了最终的成品质量。研究发现，调度员的配置变化，影响比机器升级还大，甚至会让排名出现反转。为了公平评比，必须公开调度员的管理方式，否则就像只看机器不看调度员的工厂，结果会误导人们对工厂实力的判断。
ELI14 Explained like you're 14
想象你在玩一个超级复杂的游戏，不只是你的角色（模型）厉害，还要有个队长（调度器）在旁边指挥。队长负责决定什么时候用特殊技能、什么时候休息、什么时候换装备。即使你的角色很强，如果队长指挥不好，比赛成绩也会很差。反之，如果队长调度得当，即使角色普通，也能赢得比赛。研究发现，队长的指挥方式对比赛结果的影响，比角色的能力还大。不同的队长策略，能让同样的角色表现天差地别，甚至出现排名倒置。为了公平比较，比赛必须公开队长的指挥策略，否则只看角色的强弱是不公平的。这就像论文里强调的，调度器（队长）配置的透明很重要，不能只看模型（角色）本身。
Abstract
This position paper argues that, for long-horizon tasks evaluated across models with comparable frontier capability, the agent execution harness, namely the infrastructure layer that governs context construction, tool interaction, orchestration, and verification around a language model, is often a stronger determinant of agent performance than the model it wraps. We formalize and defend the Binding Constraint Thesis: in this regime, performance variance is governed more by harness configuration than by model choice, and current evaluation protocols therefore systematically misattribute harness-level gains to model improvements. We support this thesis along three lines. First, a control-theoretic formalization treats the harness as the controller of a closed-loop dynamical system and the LLM as the stochastic policy it governs, which explains why small harness changes can produce performance shifts that exceed those obtained by substituting one model for another. Second, published benchmarks, industry deployments, and a controlled variance decomposition show that harness-induced variance can substantially exceed model-induced variance, including cases of model ranking reversal. Third, we propose a harness-aware evaluation framework with a disclosure standard and a variance decomposition protocol. Until harness specifications are disclosed, leaderboard comparisons for long-horizon agents should be treated as incomplete and potentially misleading.