HarnessBank: Semantic Gene-Bank Search with Gated Verification for Agent-Harness Self-Evolution
HarnessBank achieves agent-harness self-evolution with semantic gene-bank search and gated verification, improving performance by 5.1%-15.4%.
Key Findings
Methodology
HarnessBank is a framework for agent-harness self-evolution, pairing a task agent with an evolver agent for iterative failure diagnosis, harness generation, and evolution verification. It maintains a Harness Gene Bank of high-performing harnesses and introduces a Gated Harness Screening mechanism to efficiently filter high-quality harnesses.
Key Results
- Across seven agent benchmarks, HarnessBank achieves performance improvements ranging from 5.1% to 15.4%, demonstrating model-specific self-evolution capabilities rather than relying on a universally optimal harness.
- Cross-model experiments confirm improvements stem from model-specific self-evolving processes, not a universally optimal harness.
- The Gated Harness Screening mechanism effectively reduces the cost of evaluating numerous offspring harnesses.
Significance
HarnessBank automates agent-harness evolution, addressing the scalability issues of manual engineering. It offers a new pathway for agent performance improvement, reducing risks of search collapse and task-specific overfitting, with broad academic and industrial applications.
Technical Contribution
Compared to existing methods, HarnessBank provides structured exploration and verification mechanisms, avoiding collapse from greedy search. It ensures reliability and diversity in harness evolution through semantic gene-bank and gated screening mechanisms.
Novelty
HarnessBank is the first to propose semantic gene-bank and gated verification mechanisms, differing from greedy search methods by providing a more reliable path for agent-harness evolution.
Limitations
- In some tasks, harness evolution may overly rely on specific training data, limiting performance improvements on test sets.
- The gated screening mechanism may not fully filter out ineffective harnesses in certain scenarios.
Future Work
Future research can explore optimizing screening mechanisms in harness evolution and expanding to more application domains to enhance agent adaptability and robustness.
AI Executive Summary
HarnessBank is an innovative framework for agent-harness self-evolution, addressing issues of search collapse and task-specific overfitting caused by greedy candidate selection and noisy self-feedback in existing methods. By introducing semantic gene-bank and gated harness screening mechanisms, HarnessBank efficiently filters high-quality harnesses, reducing evaluation costs, and achieves performance improvements of 5.1% to 15.4% across seven agent benchmarks.
The framework combines task agents and evolver agents for iterative failure diagnosis, harness generation, and evolution verification. The semantic gene-bank preserves high-performing harnesses across semantic coordinates, supporting reinvention and recombination. The gated harness screening mechanism filters ineffective harnesses through four sequential gates, ensuring reliability in harness evolution.
HarnessBank's innovation lies in its structured exploration and verification mechanisms, avoiding collapse from greedy search. It offers a new pathway for agent performance improvement, reducing risks of search collapse and task-specific overfitting, with broad academic and industrial applications. Future research can explore optimizing screening mechanisms in harness evolution and expanding to more application domains to enhance agent adaptability and robustness.
Deep Analysis
Background
Large Language Models (LLMs) have evolved from passive text generators to reasoning cores of autonomous agents capable of solving long-horizon tasks. Agent harnesses are executable frameworks surrounding LLMs, converting them into task-performing agents. Existing work typically optimizes harness components like prompts, tools, memory, or workflows individually.
Core Problem
Existing agent harness development often relies on manual engineering, challenging scalability as models, tools, and application environments rapidly evolve. Greedy search methods are prone to search collapse and task-specific overfitting, making performance gains difficult to verify.
Innovation
HarnessBank introduces semantic gene-bank and gated harness screening mechanisms, providing structured exploration and verification paths. The semantic gene-bank preserves high-performing harnesses across semantic coordinates, supporting reinvention and recombination. The gated harness screening mechanism filters ineffective harnesses through four sequential gates.
Methodology
- �� Task agent executes environment tasks under current harness.
- �� Evolver agent analyzes execution trajectories, diagnoses failure mechanisms, generates offspring harnesses.
- �� Semantic gene-bank preserves high-performing harnesses, supporting reinvention and recombination.
- �� Gated harness screening mechanism filters ineffective harnesses through four sequential gates.
Experiments
Experiments are conducted across seven domains, including Terminal-Bench-2 and five benchmarks from EvoAgentBench. All experiments evolve the harness only, with the backbone frozen. The primary metric is per-task success (Pass@1), averaged over three attempts per task.
Results
Across seven agent benchmarks, HarnessBank achieves performance improvements ranging from 5.1% to 15.4%. Cross-model experiments confirm improvements stem from model-specific self-evolving processes, not a universally optimal harness. The Gated Harness Screening mechanism effectively reduces the cost of evaluating numerous offspring harnesses.
Applications
HarnessBank can be used for automated agent harness development, reducing manual engineering costs and enhancing agent adaptability and robustness across different application domains. It has broad academic and industrial applications.
Limitations & Outlook
In some tasks, harness evolution may overly rely on specific training data, limiting performance improvements on test sets. The gated screening mechanism may not fully filter out ineffective harnesses in certain scenarios. Future research can explore optimizing screening mechanisms in harness evolution.
Plain Language Accessible to non-experts
Imagine a kitchen where a chef needs to adjust cooking tools based on different ingredients and recipes. Traditional methods are like the chef manually adjusting tools, while HarnessBank is like a smart assistant that automatically analyzes recipes and ingredients to recommend the best tool combination. This not only saves time but also improves the quality of dishes.
ELI14 Explained like you're 14
Imagine you're playing a game where you need to constantly upgrade your gear to defeat stronger enemies. HarnessBank is like a smart assistant that automatically analyzes weaknesses in the game and recommends the best gear combination. This way, you can easily level up and defeat enemies! Isn't that cool?
Glossary
Large Language Model (LLM)
An AI model capable of generating and understanding natural language.
Used as the reasoning core for agents.
Agent Harness
Executable framework surrounding an LLM, converting it into a task-performing agent.
Includes prompts, tool interfaces, control loops, etc.
Semantic Gene Bank
A database preserving high-performing harnesses, supporting reinvention and recombination.
Used in the harness evolution process for selection and preservation.
Gated Harness Screening
Mechanism that filters ineffective harnesses through four sequential gates.
Used to improve efficiency and reliability in harness selection.
Evolver Agent
Agent that analyzes execution trajectories, diagnoses failure mechanisms, and generates offspring harnesses.
Paired with task agent for harness self-evolution.
Open Questions Unanswered questions from this research
- 1 How to further optimize screening mechanisms in harness evolution to enhance agent adaptability and robustness.
- 2 Possibilities for expanding the harness evolution framework to more application domains.
Applications
Immediate Applications
Automated Agent Harness Development
Reduces manual engineering costs and enhances agent adaptability and robustness across different application domains.
Long-term Vision
Broad Academic and Industrial Applications
HarnessBank has broad academic and industrial applications, capable of improving agent performance.
Abstract
Large Language Models (LLMs) have enabled capable agents across diverse applications. Beyond the foundation model, the performance of an agent is governed by the surrounding agent harness, including prompts, tools, control loops, etc. Automatically evolving this harness offers a promising pathway to agent improvement, yet existing approaches typically rely on greedy candidate selection and noisy self-generated feedback, rendering their gains susceptible to search collapse, task-specific overfitting, and poor verifiability. To tackle these challenges, we introduce HarnessBank, a trustworthy agent-harness self-evolution framework that pairs a task agent with a separate evolver agent for iterative failure diagnosis, harness generation, and evolution verification. HarnessBank maintains a Harness Gene Bank composed of high-performing harnesses of different semantic coordinates. Those harnesses are reinvented, recombined, screened, and selected during the self-evolution procedure. Moreover, we propose a Gated Harness Screening mechanism to efficiently filter high-quality harnesses and reduce the cost of evaluating numerous offspring harnesses. Across seven agent benchmarks, HarnessBank produces consistent performance improvements from 5.1% to 15.4%. Cross-model experiments further verify that the improvements come from the model-specific self-evolving process, instead of a universally optimal harness. Our code will be publicly available upon acceptance.