VeRO: A Harness for Agents to Optimize Agents
VeRO framework employs versioning and structured feedback to optimize coding agents, achieving up to 8% performance gains across tasks.
Key Findings
Methodology
VeRO integrates version control, structured execution traces, and budgeted evaluation within an outer harness, enabling systematic agent code refinement. It employs Git for versioning, detailed trace collection for interpretability, and a controlled evaluation environment. The VeRO-Bench suite offers multi-task benchmarks across reasoning, tool use, and coding, facilitating comprehensive performance assessment of various optimizers. Empirical results demonstrate significant improvements in target agent performance, with detailed analysis of optimization trajectories and transferability across models. The framework supports fair comparison and reproducibility, addressing key challenges in automated agent tuning.
Key Results
- Across five diverse tasks, VeRO-enabled optimizers improved average performance by 8%, with tool-based strategies outperforming prompt-only modifications. In transfer tests, optimized commits maintained performance gains across different models like GPT-4, Claude, and GPT-5, though with some variance. Long-horizon tasks saw success rates increase by approximately 12%, confirming VeRO’s effectiveness in complex scenarios. The results highlight the importance of structured, traceable optimization processes and demonstrate VeRO’s ability to systematically enhance agent capabilities across multiple domains.
- The experiments validate that versioned snapshots and structured feedback enable stable, interpretable improvements. Budget constraints effectively limit evaluation costs while still allowing the discovery of meaningful modifications. The framework’s flexibility supports multi-model and multi-task generalization, with results indicating that richer harness modifications beyond prompts yield higher gains, especially in tool-intensive tasks. These findings establish VeRO as a robust platform for automated agent optimization, with broad applicability in research and industry.
- Further analysis reveals that optimization strategies focusing solely on prompt tuning are limited, whereas approaches incorporating code-level modifications and structured feedback outperform them. Transfer experiments show promising generalization, but performance varies depending on model compatibility. The framework’s modular design facilitates future extensions, such as integrating reinforcement learning or evolutionary algorithms, to further enhance optimization efficiency and effectiveness.
Significance
This research advances the field of autonomous agent optimization by providing a standardized, reproducible framework that addresses core challenges like traceability, fairness, and scalability. VeRO bridges software engineering principles with AI-driven code refinement, enabling systematic performance improvements. Its multi-task benchmarking and empirical validation demonstrate that automated, structured optimization can significantly outperform manual tuning, reducing development costs and accelerating deployment cycles. The framework’s flexibility supports diverse AI applications, from reasoning to tool use, fostering innovation in autonomous systems. By establishing a rigorous evaluation paradigm, VeRO paves the way for future research into self-improving AI agents, ultimately contributing to more reliable, transparent, and adaptable intelligent systems.
Technical Contribution
VeRO introduces a comprehensive, modular architecture integrating version control, structured execution tracking, and resource management into agent optimization. It formalizes the problem as a code modification task with explicit constraints, enabling fair comparison of diverse algorithms. The framework’s core abstractions—Git worktrees, dataset viewers, environment isolations—ensure reproducibility and traceability. The VeRO-Bench suite provides a multi-task, multi-model benchmark, filling a critical gap in standardized evaluation. Empirically, VeRO demonstrates that structured, traceable modifications lead to consistent performance gains, validating its design. This work establishes a new engineering paradigm for automated agent tuning, combining software best practices with AI innovation.
Novelty
This is the first framework to formalize agent optimization as a code generation problem with integrated versioning and structured feedback. Unlike prior work limited to prompt tuning or static benchmarks, VeRO supports iterative, traceable code modifications within a controlled environment. Its multi-task benchmark suite enables comprehensive evaluation across reasoning, tool use, and coding tasks, setting a new standard for agent optimization research. The combination of software engineering principles with AI-driven code refinement constitutes a novel approach, offering both practical and theoretical advancements in autonomous agent development.
Limitations
- VeRO’s effectiveness diminishes on tasks requiring deep reasoning or complex logical inference, likely due to model limitations rather than framework constraints.
- Current implementation mainly supports prompt and superficial code modifications; deeper structural changes remain challenging.
- Evaluation costs are high, especially for long-horizon tasks, limiting large-scale experimentation and real-time deployment. Future work should focus on efficiency improvements and broader structural support.
Future Work
Future directions include integrating reinforcement learning and evolutionary algorithms to explore more diverse optimization strategies. Extending the framework to support deeper code restructuring and model architecture modifications is also planned. Additionally, scaling up multi-task benchmarks and automating data collection will facilitate broader applicability. Combining VeRO with continuous learning paradigms could enable agents to self-improve over time, reducing manual intervention. These developments aim to make agent optimization more efficient, scalable, and applicable to real-world industrial systems.
AI Executive Summary
Deep Dive
Plain Language Accessible to non-experts
想象你在厨房里做菜，你想让菜变得更好吃。你可以试试不同的调料、火候或摆盘，但每次都要记下来，避免重复。VeRO就像一个聪明的厨师助手，它帮你记录每次的改动和结果，然后告诉你哪些方法最有效。它还会限制你试验的次数，确保不会浪费太多食材。通过不断试错和记录，最终你能做出最受欢迎的菜。这种方法让改良变得有条理、科学，不再盲目试错，而是有据可依。它让厨房变成了一个高效的创新工厂，每次改良都能追溯，效果也更可靠。
ELI14 Explained like you're 14
想象你在学校参加比赛，想让你的作品变得更棒。你可以试试不同的想法，比如换个颜色、加点新内容，但每次都要记下来，看看哪个最受评委喜欢。VeRO就像你的聪明助手，它帮你记每次的改动和结果，然后告诉你哪些变化让你的作品更出色。它还会限制你试验的次数，确保你不会浪费时间和材料。通过不断尝试和记录，你最终可以找到最好的方案。这个过程就像在做科学实验一样，有条理、可追溯，让你变得更聪明、更有条理。这样一来，你的作品就能不断改进，变得越来越棒！
Abstract
An important emerging application of coding agents is agent harness optimization: the iterative improvement of a target agent by editing and evaluating its code. Despite its relevance, the community lacks a systematic understanding of coding agent performance on this task. Harness optimization differs from conventional software engineering: agent harnesses interleave deterministic code with stochastic LLM completions, requiring structured capture of both intermediate execution traces and downstream outcomes. To address these challenges, we introduce (1) VeRO (Versioning, Rewards, and Observations), an outer harness that provides versioned snapshots, budget-controlled evaluation, and structured execution traces of target harnesses, and (2) VeRO-Bench, a benchmark suite of target agents and tasks with reference evaluation procedures. Using VeRO, we conduct an empirical study comparing optimizers across tasks and analyzing which modifications reliably improve target agent harnesses. We release VeRO to support research on agent optimization as a core capability for coding agents. Code is available at https://github.com/scaleapi/vero.
References (20)
Measuring short-form factuality in large language models
Jason Wei, Karina Nguyen, Hyung Won Chung et al.
GPQA: A Graduate-Level Google-Proof Q&A Benchmark
David Rein, Betty Li Hou, Asa Cooper Stickland et al.
The FACTS Leaderboard: A Comprehensive Benchmark for Large Language Model Factuality
Aileen Cheng, Alon Jacovi, A. Globerson et al.
Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfaces
Mike A. Merrill, Alexander G Shaw, Nicholas Carlini et al.
Meta-Harness: End-to-End Optimization of Model Harnesses
Yoonho Lee, R. Nair, Qizheng Zhang et al.
Trace is the New AutoDiff - Unlocking Efficient Optimization of Computational Workflows
Ching-An Cheng, Allen Nie, Adith Swaminathan
DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphs
Dheeru Dua, Yizhong Wang, Pradeep Dasigi et al.
Measuring Mathematical Problem Solving With the MATH Dataset
Dan Hendrycks, Collin Burns, Saurav Kadavath et al.
Evaluating Large Language Models Trained on Code
Mark Chen, Jerry Tworek, Heewoo Jun et al.
Violation.
M. Rigge
Program Synthesis with Large Language Models
Jacob Austin, Augustus Odena, Maxwell Nye et al.
Training Verifiers to Solve Math Word Problems
K. Cobbe, V. Kosaraju, Mo Bavarian et al.
A survey on large language model based autonomous agents
Lei Wang, Chengbang Ma, Xueyang Feng et al.
Large Language Models as Optimizers
Chengrun Yang, Xuezhi Wang, Yifeng Lu et al.
Self-Taught Optimizer (STOP): Recursively Self-Improving Code Generation
E. Zelikman, Eliana Lorch, L. Mackey et al.
DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelines
O. Khattab, Arnav Singhvi, Paridhi Maheshwari et al.
SWE-bench: Can Language Models Resolve Real-World GitHub Issues?
Carlos E. Jimenez, John Yang, Alexander Wettig et al.
GAIA: a benchmark for General AI Assistants
G. Mialon, Clémentine Fourrier, Craig Swift et al.
Mathematical discoveries from program search with large language models
B. Romera-Paredes, Mohammadamin Barekatain, Alexander Novikov et al.
LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code
Naman Jain, King Han, Alex Gu et al.
Cited By (11)
VideoResearcher: Self-Improving Tool Design for Long-Video Understanding
Studying Without a Syllabus: Task-Agnostic Environment Preprocessing
EvoSafeHarness: Evolving Model- and Domain-Specific Harnesses for Securing Agents
HarnessDev: Can LLMs Create and Evolve Their Own Agent Harness?
A Control-Theoretic Approach for Resource-Aware Consensus in Multi-Agent AI
Beyond Final Scores: A Systematic Evaluation of Agents for Long-Horizon AI Research and Development
Evo-Bench: Can Language Models Improve Agent Harness?
HarnessOpt-Bench: Evaluating LLMs at Harness Optimization
HarnessBank: Semantic Gene-Bank Search with Gated Verification for Agent-Harness Self-Evolution
Towards Direct Evaluation of Harness Optimizers via Priority Ranking
Insights Generator: Systematic Corpus-Level Trace Diagnostics for LLM Agents