HarnessDev: Can LLMs Create and Evolve Their Own Agent Harness?
HarnessDev evaluates LLMs' ability to create and evolve execution infrastructure, showing superior performance in writing and ML experiments compared to human references.
Key Findings
Methodology
HarnessDev evaluates LLMs' ability to develop execution infrastructure through two stages: Creation and Evolution. In Creation, the model starts from a weak seed to build a complete system. In Evolution, it iteratively improves its own infrastructure using downstream feedback. Metrics include task success rate and execution-token cost.
Key Results
- Generated infrastructures outperform human references in writing and ML experiments but lag in code and search tasks.
- Performance gains in the Evolution stage are unstable and only partially transfer to unseen tasks.
- Experiments show performance gains strongly depend on the model executing the infrastructure, indicating limited cross-model transfer.
Significance
This study introduces the HarnessDev benchmark, systematically evaluating LLMs' ability to create and improve execution infrastructure. This capability is crucial for transitioning from research prototypes to practical applications, especially in domains requiring continuous development.
Technical Contribution
HarnessDev provides a new evaluation framework focusing on building and maintaining runnable infrastructure rather than single task outputs. It highlights the model's role in execution systems and reveals performance differences across tasks through two-stage evaluation.
Novelty
HarnessDev is the first to shift evaluation focus from task outputs to runnable infrastructure, filling a gap in existing evaluation methods and offering new insights into LLMs' system-building capabilities.
Limitations
- Generated infrastructures perform poorly in code and search tasks, showing a gap with mature human-engineered systems.
- Performance gains in the Evolution stage are unstable, making consistent performance on unseen tasks challenging.
Future Work
Future research could explore improving stability in the Evolution stage and enhancing cross-model transferability in different execution infrastructures.
AI Executive Summary
As agents transition from research prototypes to practical applications, their capabilities increasingly rely on model-external execution infrastructure, known as the agent harness. HarnessDev evaluates LLMs' ability to create and improve this infrastructure through two stages. In the Creation stage, models start from a weak seed to build a complete execution system. In the Evolution stage, they iteratively improve their own infrastructure using downstream feedback.
Experimental results show that generated infrastructures outperform human references in writing and machine learning experiments but lag in code and search tasks. Performance gains in the Evolution stage are unstable and only partially transfer to unseen tasks. Experiments also indicate that performance gains strongly depend on the model executing the infrastructure, suggesting limited cross-model transfer.
This study systematically evaluates LLMs' ability to create and improve execution infrastructure, offering new insights into the model's role in execution systems. This capability is crucial for transitioning from research prototypes to practical applications, especially in domains requiring continuous development. Future research could explore improving stability in the Evolution stage and enhancing cross-model transferability in different execution infrastructures.
Deep Analysis
Background
As agents transition from research prototypes to practical applications, their capabilities increasingly rely on model-external execution infrastructure. Existing evaluation methods focus on task outputs, neglecting the ability to develop and improve execution infrastructure.
Core Problem
Developing and improving agent harnesses is complex, involving execution loops, tool usage, context management, failure recovery, and result verification.
Innovation
HarnessDev evaluates LLMs' ability to create and improve execution infrastructure through two stages, shifting evaluation focus from task outputs to runnable infrastructure.
Methodology
- �� Creation stage: Models start from a weak seed to build a complete execution system.
- �� Evolution stage: Models iteratively improve their own infrastructure using downstream feedback.
- �� Evaluation metrics: Task success rate and execution-token cost.
Experiments
Experiments involve six creator LLMs, covering four domains and five downstream benchmarks, totaling 2,207 unique downstream instances. Evaluation includes task success rate and execution-token cost.
Results
Generated infrastructures outperform human references in writing and ML experiments but lag in code and search tasks. Performance gains in the Evolution stage are unstable and only partially transfer to unseen tasks.
Applications
This study offers new insights into LLMs' role in execution systems, particularly in domains requiring continuous development.
Limitations & Outlook
Generated infrastructures perform poorly in code and search tasks, showing a gap with mature human-engineered systems. Performance gains in the Evolution stage are unstable, making consistent performance on unseen tasks challenging.
Plain Language Accessible to non-experts
Imagine you're cooking in a kitchen. You have a recipe (agent harness) that tells you how to prepare and cook ingredients (execute tasks). If you change the steps in the recipe (adjust the harness), even if the ingredients (model weights) remain the same, the taste of the dish (task performance) can vary greatly. HarnessDev is like a chef evaluation system, assessing whether you can create a recipe from scratch and continually improve it when trying new dishes.
ELI14 Explained like you're 14
Imagine you're playing a game where you need to build a base to protect yourself. This base is your agent harness. HarnessDev is like a game review system, evaluating whether you can build a base from scratch and improve it as enemies attack. Even if your character (model) doesn't change, the design of the base affects your game performance!
Glossary
LLM (Large Language Model)
A large machine learning model capable of generating and understanding natural language.
Used in the paper to create and improve execution infrastructure.
HarnessDev
A benchmark evaluating LLMs' ability to create and improve execution infrastructure.
Used to assess model performance across different tasks.
Execution Infrastructure
Software system managing execution loops, tool usage, context management, failure recovery, and result verification.
Key component of agent capabilities.
Creation Stage
Models start from a weak seed to build a complete execution system.
First evaluation stage of HarnessDev.
Evolution Stage
Models iteratively improve their own infrastructure using downstream feedback.
Second evaluation stage of HarnessDev.
Open Questions Unanswered questions from this research
- 1 How to improve stability in the Evolution stage, especially in unseen tasks.
- 2 How to enhance cross-model transferability in different execution infrastructures.
Applications
Immediate Applications
Agent Harness Development
Helps engineers create and improve agent harnesses, enhancing task execution efficiency.
Long-term Vision
Automated System Design
Automating the creation and improvement of execution infrastructure, reducing reliance on human engineers.
Abstract
As agents move from research prototypes to deployed tools, their capability increasingly depends on model-external execution infrastructure, commonly termed the agent harness. Changing this harness while holding model weights fixed can substantially alter task performance. Current agent evaluations typically report downstream performance under a chosen harness, leaving a model's ability to develop the harness itself comparatively underexplored. We introduce HarnessDev, a benchmark that shifts the unit of evaluation from task outputs to runnable infrastructure. HarnessDev covers two stages. In Creation, the agent starts from a minimal seed and a small number of cases, then builds a complete execution system. In Evolution, it starts from its own created harness and iteratively revises it using downstream execution feedback, with the goal of improving benchmark performance. We then evaluate each constructed harness on capability (task success on held-out benchmarks) and efficiency (execution-token cost). The reported Creation results cover six creator LLMs, four domains, and five downstream benchmarks totaling 2,207 unique downstream instances, with hidden evaluation tasks withheld from development. We find that generated harnesses remain substantially behind mature human-engineered references on code and on search and research, while matching or exceeding the selected references on writing and machine-learning experimentation, with large variation in execution cost. Evolution produces some performance gains, but they are unstable and transfer only partially to held-out tasks. Experiments with a fixed runtime model further show that the gains depend strongly on the model executing the harness, indicating limited transfer across models.