HarnessOpt-Bench: Evaluating LLMs at Harness Optimization
HarnessOpt-Bench benchmark evaluates 5 frontier LLMs in costly, stochastic harness optimization, revealing significant model differences and room for improvement.
Key Findings
Methodology
This paper introduces the HarnessOpt-Bench framework, which formalizes an end-to-end harness optimization process with a trusted execution environment (TEE). The optimizer, typically an LLM, receives a seed harness and graded evaluation feedback within a fixed budget, then iteratively edits the harness and nominates a final candidate. The candidate harness is evaluated on a held-out test set inaccessible during search, using a normalized gain metric to quantify improvement. The experimental setup involves evaluating five state-of-the-art models (Claude, GPT-5.6, Kimi) across four downstream tasks, totaling 111 runs. The TEE enforces evaluation boundaries, resource limits, and candidate versioning, ensuring fairness and reproducibility. The framework allows fair comparison of different models, harnesses, and search strategies under controlled conditions.
Key Results
- Model choice significantly impacts performance, with an average normalized gain difference of 0.142, compared to 0.079 for harness design variations. Different models (Claude, GPT-5.6, Kimi) show consistent performance improvements over seed harnesses, with the best gains reaching 0.49. Native harnesses do not always outperform shared harnesses, and performance varies substantially across tasks and seed regimes. Broader search exploration correlates positively with gain, indicating that wider exploration strategies lead to better harness improvements.
- The experiments reveal substantial capability gaps in current models, especially in noisy, costly evaluation environments. Model versions (e.g., GPT-5.6 releases) demonstrate gradual performance improvements, but the gains are non-monotonic and limited by the models' ability to diagnose failures accurately. The analysis shows that exploration breadth, rather than detailed trace reading, correlates more strongly with performance gains. Additionally, the sensitivity of models to harness design varies, with some models performing better under specific harness configurations.
- The study establishes harness optimization as a measurable, discriminative capability, highlighting the importance of model selection, search strategies, and evaluation protocols. The results emphasize that current models have significant room for improvement, especially in complex, noisy environments, and that systematic evaluation frameworks like HarnessOpt-Bench are essential for tracking progress. The framework's flexibility allows future research to extend to more complex, multi-tool, multi-objective scenarios, fostering the development of more autonomous and adaptable AI systems.
- The experiments also demonstrate that exploration strategies, such as touching more harness levers, are associated with higher gains, whereas detailed trace reading is negatively correlated with performance, suggesting that broad, strategic modifications are more effective than deep diagnostics in noisy settings.
- Overall, this work provides a foundational step toward standardized, rigorous evaluation of harness optimization capabilities, offering insights into the current state and future directions for automating the improvement of AI agent systems.
Significance
This research pioneers a systematic, reproducible evaluation of large language models' ability to optimize complex system harnesses under costly and noisy conditions. By establishing HarnessOpt-Bench, it fills a critical gap in the field, enabling fair comparison across models, harness designs, and search strategies. The framework's rigorous control of evaluation boundaries and resource constraints ensures that observed improvements reflect genuine capability rather than overfitting or noise. The findings highlight the substantial potential of current models to improve system-level performance through automated harness optimization, which is crucial for deploying autonomous AI agents in real-world, resource-constrained environments. Moreover, the insights gained from analyzing exploration breadth and harness sensitivity inform future research directions, emphasizing the importance of strategic search and diagnostic capabilities. The work thus advances both theoretical understanding and practical methodologies for AI system self-improvement, with broad implications for AI safety, robustness, and scalability.
Technical Contribution
The paper's key technical contributions include: 1) the design of HarnessOpt-Bench, a comprehensive benchmark for end-to-end harness optimization under stochastic, expensive evaluation conditions; 2) the integration of a trusted execution environment (TEE) that enforces evaluation boundaries, resource limits, and candidate versioning, ensuring reproducibility and auditability; 3) a systematic comparison of five frontier LLMs (Claude, GPT-5.6, Kimi) under shared and native harness configurations across multiple tasks, revealing model-dependent performance variations; 4) the development of quantitative metrics such as normalized gain and exploration breadth, providing new tools for analyzing search strategies and model capabilities; 5) empirical evidence that broader exploration correlates with higher gains, and that current models still face significant capability gaps, especially in noisy, costly evaluation scenarios.
Novelty
This work is novel in establishing the first standardized, reproducible benchmark specifically targeting the challenging task of harness optimization in high-cost, stochastic environments. Unlike prior work that focused on prompt tuning or end-to-end code editing, HarnessOpt-Bench emphasizes a unified evaluation protocol with strict control over evaluation boundaries, resource budgets, and candidate tracking. It introduces the use of trusted execution environments to prevent information leakage and ensure fairness, a significant technical advancement. Additionally, the comprehensive comparison of multiple models and harness configurations across diverse tasks provides new insights into the factors influencing harness optimization performance. The emphasis on exploration strategies and their correlation with gains further distinguishes this work from existing approaches, offering a new perspective on search efficiency and diagnostic capabilities in AI system self-improvement.
Limitations
- The current framework relies on a limited set of models and harness designs, which may not fully capture the diversity of real-world systems. Extending to more complex, multi-tool, multi-objective scenarios remains a challenge.
- Evaluation costs remain high, especially for large models and complex tasks, limiting scalability. Developing more efficient evaluation protocols or surrogate metrics is necessary for industrial applications.
- The ability of models to diagnose failures accurately in noisy environments is still limited, leading to suboptimal harness modifications. Improving diagnostic and diagnostic-guided search strategies is an important future direction.
Future Work
未来的研究应关注提升模型在高噪声和昂贵评估环境中的诊断能力，探索更高效的评估机制以降低成本。同时，扩展到多任务、多目标、多工具的复杂系统中，验证方法的普适性和扩展性。结合强化学习和元学习技术，增强模型的泛化和适应能力，推动自动化 harness 优化的理论与实践发展。此外，研究还应关注如何在实际工业场景中应用这些技术，提升系统的自主学习和适应能力，最终实现更智能、更高效的自动化系统优化框架。
AI Executive Summary
在人工智能快速演进的背景下，构建具有自主学习和优化能力的智能系统成为研究的前沿。大型语言模型（LLMs）作为核心技术，其性能不仅依赖于模型参数的规模，还受到 surrounding harness（提示、工具、控制流、记忆和调度代码）的深刻影响。不同的 harness 设计会导致模型能力的显著差异，如何自动化优化 harness，成为提升系统整体性能的关键问题。
然而，现有的研究缺乏统一的评估标准和测试平台，限制了该领域的深入探索。为此，本文提出了HarnessOpt-Bench，一个专为昂贵且随机的 harness 优化任务设计的基准框架。该框架结合可信执行环境（TEE），确保在有限预算内对候选 harness 进行公平、可追溯的评估。优化器（通常为LLM）在接收目标代理的种子 harness和评估反馈后，经过多轮编辑，最终提名一个候选 harness。该候选在未见的测试集上，通过归一化增益指标进行评分，确保模型的泛化能力。
实验中，作者评估了包括Claude、GPT-5.6系列和Kimi在内的五个前沿模型，使用共享编码 harness和其原生 harness，覆盖4个不同的下游任务，共计111次评分运行。结果显示，模型之间的差异明显，模型选择对优化效果影响大于 harness 设计，且不同任务和种子设置下性能波动巨大。研究还发现，探索宽度（即搜索的广度）与性能提升呈正相关，广泛探索能带来更大增益。
这些发现表明，尽管当前模型在昂贵评估场景中仍有能力差距，但通过系统化的优化策略和评估框架，模型性能有望持续提升。HarnessOpt-Bench为未来自动化系统的研发提供了标准化的评估工具，推动了系统级自动优化技术的发展。未来，研究将集中在提升模型诊断能力、降低评估成本，以及扩展到更复杂的多任务、多目标场景中，助力智能系统的自主学习与适应能力的全面提升。
Deep Dive
Abstract
As LLMs are increasingly deployed within agentic systems, their capabilities depend not only on the model weights but also on the harness: the prompts, tools, control flow, memory, and orchestration code surrounding them. This makes automated harness optimization -- the iterative and evaluation-guided improvement of a harness by an AI system -- both an important route to improving AI systems and a demanding capability for AI systems themselves. Yet the community lacks a common protocol for measuring how well frontier LLMs perform at this task. We introduce HarnessOpt-Bench, a benchmark for end-to-end harness optimization under expensive and stochastic evaluation. An optimizer, an LLM paired with a coding harness, receives a target agent's seed harness, graded evaluation feedback, and a fixed target-evaluation budget. It edits the harness and nominates a final candidate, which is scored by its normalized gain over the seed on a held-out test partition that remains inaccessible throughout search. A trusted execution environment enforces the evaluation boundary, meters target-agent resource use, and preserves candidate versions for audit. We evaluate 5 frontier LLMs as optimizers both under a shared coding harness and under their native harnesses across 4 downstream tasks, over 111 scored runs. Experiment results show that optimizer models separate more than the coding harnesses they act through, native harnesses are not consistently superior, and gains vary substantially across tasks and seed regimes. These results establish harness optimization as a measurable and discriminative capability with large space for improvement.
References (20)
BrowseComp-Plus: A More Fair and Transparent Evaluation Benchmark of Deep-Research Agent
Zijian Chen, Xueguang Ma, Shengyao Zhuang et al.
Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfaces
Mike A. Merrill, Alexander G Shaw, Nicholas Carlini et al.
GAIA: a benchmark for General AI Assistants
G. Mialon, Clémentine Fourrier, Craig Swift et al.
A Self-Improving Coding Agent
M. Robeyns, M. Szummer, Laurence Aitchison
Gödel Agent: A Self-Referential Agent Framework for Recursive Self-Improvement
Xunjian Yin, Xinyi Wang, Liangming Pan et al.
Program Synthesis with Large Language Models
Jacob Austin, Augustus Odena, Maxwell Nye et al.
RSIBench-Data: Benchmarking Data-Centric Research for Recursive Self-Improvement
Fanqing Meng, Lingxiao Du, Qiguang Chen et al.
Evaluating Large Language Models Trained on Code
Mark Chen, Jerry Tworek, Heewoo Jun et al.
VeRO: A Harness for Agents to Optimize Agents
Varun Ursekar, Apaar Shanker, Veronica Chatrath et al.
Automated Design of Agentic Systems
Shengran Hu, Cong Lu, Jeff Clune
KernelBench: Can LLMs Write Efficient GPU Kernels?
Anne Ouyang, Simon Guo, Simran Arora et al.
Mathematical discoveries from program search with large language models
B. Romera-Paredes, M. Barekatain, Alexander Novikov et al.
Trace is the Next AutoDiff: Generative Optimization with Rich Feedback, Execution Traces, and LLMs
Ching-An Cheng, Allen Nie, Adith Swaminathan
AlphaEvolve: A coding agent for scientific and algorithmic discovery
Alexander Novikov, Ngân V˜u, Marvin Eisenberger et al.
GEPA: Reflective Prompt Evolution Can Outperform Reinforcement Learning
Lakshya A. Agrawal, Shangyin Tan, Dilara Soylu et al.
Large Language Models as Optimizers
Chengrun Yang, Xuezhi Wang, Yifeng Lu et al.
Stop Comparing LLM Agents Without Disclosing the Harness
Yunbei Zhang, Janet Wang, Yingqiang Ge et al.
LLM-AutoDiff: Auto-Differentiate Any LLM Workflow
Li Yin, Zhangyang Wang
SWE-bench: Can Language Models Resolve Real-World GitHub Issues?
Carlos E. Jimenez, John Yang, Alexander Wettig et al.
MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineering
Jun Shern Chan, Neil Chowdhury, Oliver Jaffe et al.