EvoAgentBench: Benchmarking Agent Self-Evolution via Ability Transfer
EvoAgentBench evaluates agent self-evolution via ability transfer across four domains, highlighting limitations of automatic methods.
Key Findings
Methodology
EvoAgentBench extracts abilities from agent executions, constructs domain-specific ability graphs, and evaluates self-evolution across four domains. Each test task is supported by verified training-side abilities, ensuring effective transfer.
Key Results
- Anchor method achieved positive transfer across all models, with an average gain of 10.5% on Qwen3.5-397B.
- Automatic methods like Memento performed poorly in some settings, even degrading performance.
- Ability content is transferable across model families, with Anchor showing positive gains in all test units.
Significance
This study introduces EvoAgentBench, a new framework for evaluating agent self-evolution, addressing gaps in current benchmarks that fail to effectively assess ability transfer. It is significant for improving agent reliability and efficiency.
Technical Contribution
EvoAgentBench provides fine-grained diagnostics of agent self-evolution through ability graphs, distinguishing it from traditional task accuracy comparisons. It offers a diagnostic reference for automatic methods, helping identify issues in ability extraction and routing.
Novelty
EvoAgentBench is the first to focus on ability transfer as the core of evaluating agent self-evolution, differing from previous task or memory benchmarks. It achieves cross-task procedural transfer measurement through ability graphs.
Limitations
- Automatic methods are unstable in some settings, potentially causing negative transfer.
- Current automatic methods fail to achieve consistent positive gain across all settings.
- Accuracy of ability extraction and routing relies on manual annotation.
Future Work
Future research could explore more precise automatic ability extraction and routing methods, and how to apply EvoAgentBench in broader domains.
AI Executive Summary
EvoAgentBench is a new benchmark for evaluating agent self-evolution, focusing on ability transfer. Existing evaluation methods often fail to effectively isolate and measure this ability transfer. EvoAgentBench achieves this by extracting and standardizing abilities across four domains, constructing ability graphs.
In experiments, EvoAgentBench demonstrated the transferability of ability content across model families, particularly excelling in the Anchor method. However, automatic methods like Memento and ReasoningBank performed poorly in some settings, even degrading performance.
The significance of this study lies in providing a new framework for evaluating agent self-evolution, capable of better diagnosing issues in experience encoding and ability routing. This is crucial for improving agent reliability and efficiency in long-horizon tasks. Future research can further optimize automatic ability extraction and routing methods.
Deep Analysis
Background
As large models are increasingly applied to long-horizon tasks, the importance of agent self-evolution is growing. Existing benchmarks often focus on single-task solving or memory retention, neglecting procedural ability transfer. EvoAgentBench provides fine-grained diagnostics of agent self-evolution through ability graphs.
Core Problem
Existing evaluation methods fail to effectively isolate and measure agent ability transfer. Task benchmarks only test single-episode task solving, while memory benchmarks focus on information retention rather than procedural reuse.
Innovation
EvoAgentBench extracts abilities from agent executions and constructs domain-specific ability graphs, achieving cross-task procedural transfer measurement. Each test task is supported by verified training-side abilities.
Methodology
- �� Extract abilities from agent executions and standardize into operational units
- �� Construct domain-specific ability graphs linking tasks with shared procedures
- �� Ensure each test task has training-side ability support
- �� Validate with multiple models and frameworks
Experiments
Experiments were conducted with a 528/267 train/test split, using two scaffolds and three models. Results showed that the Anchor method achieved positive transfer across all models, while automatic methods performed poorly in some settings.
Results
The Anchor method showed positive gains in all test units, with an average gain of 10.5% on Qwen3.5-397B. Automatic methods like Memento performed poorly in some settings, even degrading performance.
Applications
EvoAgentBench can be used to evaluate agent self-evolution in long-horizon tasks, helping develop more reliable and efficient intelligent agents.
Limitations & Outlook
Automatic methods are unstable in some settings, potentially causing negative transfer. The accuracy of ability extraction and routing relies on manual annotation, necessitating exploration of more precise automatic methods.
Plain Language Accessible to non-experts
Imagine a factory where workers are on different production lines. Each line has specific steps and tools. EvoAgentBench is like a new management system that not only records each worker's steps but also extracts which steps are reusable. So, when new orders come in, the factory can complete production faster because they know which steps can be directly used instead of starting from scratch. This system helps the factory increase efficiency and reduce errors.
ELI14 Explained like you're 14
Imagine you're playing a really complex game where you have to relearn skills every time you level up. EvoAgentBench is like a super cheat sheet that helps you remember which skills can be reused in different levels. So you don't have to start from scratch each time, and you can level up faster! But sometimes the cheat sheet gets it wrong, so we're still working on making it smarter.
Glossary
Ability Transfer
The process of applying skills or knowledge learned from one task to another.
Used in EvoAgentBench to evaluate agent self-evolution.
Ability Graph
A graph structure where nodes represent tasks and edges indicate shared reusable abilities.
Used to connect tasks with shared procedures for procedural transfer measurement.
Self-Evolution
The process by which an agent improves future task performance by learning and applying past experiences.
The core evaluation target of EvoAgentBench.
Automatic Methods
Methods for ability extraction and routing that require no manual intervention.
Compared with the Anchor method in experiments.
Negative Transfer
A phenomenon where ability transfer results in decreased task performance.
Observed in some automatic methods.
Open Questions Unanswered questions from this research
- 1 How to improve the accuracy of automatic ability extraction and routing?
- 2 What are the challenges in applying EvoAgentBench to broader domains?
Applications
Immediate Applications
Intelligent Agent Optimization
Evaluate and optimize agent self-evolution capabilities to improve performance in long-horizon tasks.
Long-term Vision
General AI
Achieve more efficient learning and adaptation through ability transfer, advancing the development of general AI.
Abstract
Agent self-evolution in long-horizon LLM systems is largely procedural: useful experience is not merely stored information, but reusable procedures for searching, debugging, and verification. Yet current evaluations do not isolate this form of transfer. Agent benchmarks test single-episode task solving; memory benchmarks target information retention rather than procedural reuse. We introduce EvoAgentBench, a benchmark for agent self-evolution via Ability-guided transfer across four agentic domains: web research, algorithmic reasoning, software engineering, and knowledge work. EvoAgentBench extracts trace-grounded Abilities from agent executions, canonicalizes them into operational units, and builds domain-specific Ability Graphs linking tasks that share procedural overlap. By design, every test task is backed by verified training-side Ability support. Across a 528/267 train/test split, two scaffolds, and three backbones, curated Ability content transfers reliably across model families, but no current automatic method sustains positive gain in all settings. EvoAgentBench shifts self-evolution evaluation from aggregate accuracy comparison to fine-grained diagnosis of experience encoding, routing, and uptake. The benchmark is publicly available at https://huggingface.co/datasets/EverMind-AI/EvoAgentBench.