Dr. Zero: Self-Evolving Search Agents without Training Data
Dr. Zero achieves search capabilities comparable to supervised learning through self-evolution and HRPO without training data.
Key Findings
Methodology
Dr. Zero employs a self-evolution feedback loop using an external search engine as the knowledge environment. The proposer generates diverse questions, and the solver solves them, providing difficulty feedback, forming an automated curriculum. HRPO clusters similar questions to reduce computational costs.
Key Results
- Dr. Zero surpasses supervised agents on several benchmarks, achieving 0.397 EM on NQ, 22.9% higher than supervised baselines.
- In multi-hop tasks, Dr. Zero's 7B model outperforms supervised baselines on 2WikiMQA, demonstrating strong reasoning capabilities.
- HRPO significantly reduces computational requirements, enabling efficient self-evolution without training data.
Significance
Dr. Zero demonstrates the potential for powerful search and reasoning capabilities through self-evolution without human-annotated data. This research breaks data bottlenecks, providing new paths for automated learning in large language models, with significant academic and industrial implications.
Technical Contribution
Dr. Zero introduces Hop-grouped relative policy optimization (HRPO), reducing training computational costs by clustering questions. The self-evolution framework requires no human-annotated data, showcasing new engineering possibilities.
Novelty
Dr. Zero is the first to achieve performance comparable to supervised learning through self-evolution and HRPO without training data, especially in complex multi-hop reasoning tasks.
Limitations
- Dr. Zero may face performance drops in handling extremely complex problems, as these require more computational resources and complex reasoning.
- The method relies on the quality and coverage of the external search engine, which may lead to knowledge gaps.
Future Work
Future work can explore applying Dr. Zero to broader knowledge domains and optimizing its performance in extremely complex tasks. Combining with other unsupervised learning methods may further enhance its capabilities.
AI Executive Summary
As high-quality data becomes increasingly difficult to obtain, self-evolution emerges as a promising paradigm. Dr. Zero employs a self-evolution feedback loop, using an external search engine as the knowledge environment. The proposer generates diverse questions, and the solver solves them, providing difficulty feedback, forming an automated curriculum. To enhance training efficiency, the study introduces Hop-grouped relative policy optimization (HRPO), clustering similar questions to reduce computational costs. Experimental results show that Dr. Zero surpasses supervised agents on several benchmarks, demonstrating the potential for powerful search and reasoning capabilities through self-evolution without human-annotated data. This research breaks data bottlenecks, providing new paths for automated learning in large language models, with significant academic and industrial implications. However, Dr. Zero may face performance drops in handling extremely complex problems, as these require more computational resources and complex reasoning. Future work can explore applying Dr. Zero to broader knowledge domains and optimizing its performance in extremely complex tasks. Combining with other unsupervised learning methods may further enhance its capabilities.
Deep Analysis
Background
In recent years, with the development of large language models (LLMs), achieving automated learning without human-annotated data has become a research hotspot. Traditional methods rely on large amounts of human-annotated data, leading to high data acquisition costs and limited scalability. Self-evolution methods generate and solve problems to achieve automatic improvement in model capabilities, showing progress in mathematical and specific reasoning tasks.
Core Problem
Multi-turn search agents struggle to self-evolve effectively without training data due to limited question diversity and high computational costs for multi-step reasoning. Improving search agents' reasoning capabilities without relying on human-annotated data is an important and challenging problem.
Innovation
Dr. Zero introduces a self-evolution feedback loop where the proposer generates diverse questions, and the solver solves them, providing difficulty feedback, forming an automated curriculum. Additionally, HRPO clusters similar questions to reduce computational costs, enabling efficient self-evolution without training data.
Methodology
- �� The proposer generates diverse questions, using an external search engine as the knowledge environment.
- �� The solver solves questions and provides difficulty feedback, forming an automated curriculum.
- �� HRPO clusters similar questions to reduce computational costs and improve training efficiency.
- �� Through multiple iterations, the capabilities of both the proposer and solver are gradually improved.
Experiments
Experiments were conducted on multiple open-domain question answering benchmarks, including NQ, TriviaQA, PopQA, etc. Using Qwen2.5 3B/7B models as the base, Dr. Zero's performance was compared with supervised learning baselines. Key metrics include exact match scores, with results showing Dr. Zero surpassing several supervised baselines without training data.
Results
Dr. Zero surpasses supervised agents on several benchmarks, achieving 0.397 EM on NQ, 22.9% higher than supervised baselines. In multi-hop tasks, Dr. Zero's 7B model outperforms supervised baselines on 2WikiMQA, demonstrating strong reasoning capabilities.
Applications
Dr. Zero can be used in applications requiring complex reasoning and search capabilities, such as automated question-answering systems and intelligent assistants. These applications do not rely on human-annotated data, reducing development costs and having broad industry impact.
Limitations & Outlook
Dr. Zero may face performance drops in handling extremely complex problems, as these require more computational resources and complex reasoning. Additionally, the method relies on the quality and coverage of the external search engine, which may lead to knowledge gaps.
Plain Language Accessible to non-experts
Imagine a school where the teacher is the proposer and the students are the solvers. The teacher gives students problems of varying difficulty every day, and the students try to solve them and provide feedback on the difficulty. The teacher adjusts the problem difficulty based on the feedback, gradually improving the students' abilities. Dr. Zero is like this school, continuously enhancing its search and reasoning capabilities through self-evolution.
ELI14 Explained like you're 14
Imagine you're playing a game with a robot assistant. This assistant gives you all sorts of puzzles, and you have to search online for the answers. Every time you get it right, the assistant gives you a harder puzzle. This assistant is like Dr. Zero, getting smarter by constantly challenging itself!
Glossary
Self-Evolution
The process where a model continuously improves its capabilities by generating and solving problems.
Dr. Zero achieves unsupervised learning through self-evolution.
Hop-grouped Relative Policy Optimization (HRPO)
An optimization algorithm that reduces training computational costs by clustering similar questions.
HRPO is used in Dr. Zero to enhance training efficiency.
Proposer
The model component responsible for generating diverse questions.
In Dr. Zero, the proposer generates questions to train the solver.
Solver
The model component responsible for solving questions and providing difficulty feedback.
The solver helps the proposer adjust question difficulty by solving problems.
External Search Engine
A tool used to provide a knowledge environment, assisting the model in acquiring external information.
Dr. Zero uses an external search engine as its knowledge environment.
Open Questions Unanswered questions from this research
- 1 How to apply Dr. Zero to broader knowledge domains remains to be explored, especially in handling extremely complex tasks.
- 2 How to further optimize Dr. Zero's performance in unsupervised learning, particularly in multi-hop reasoning tasks.
Applications
Immediate Applications
Automated Question-Answering Systems
Dr. Zero can be used to develop automated question-answering systems without human-annotated data, reducing development costs.
Long-term Vision
Intelligent Assistants
Through self-evolution, Dr. Zero can become a smarter assistant, handling complex search and reasoning tasks.
Abstract
As high-quality data becomes increasingly difficult to obtain, self-evolution without curated training data has emerged as a promising paradigm. This approach allows large language models (LLMs) to autonomously generate and solve complex problems, thereby improving their reasoning capabilities. However, multi-turn search agents struggle in this setting due to limited question diversity and the substantial compute required for multi-step reasoning and tool use. In this work, we introduce Dr. Zero, a framework that enables search agents to effectively self-evolve without human-annotated training data, relying solely on an external search engine as their knowledge environment. In particular, we design a self-evolution feedback loop where a proposer generates structurally diverse questions to train a solver initialized from the same base model. As the solver evolves, it incentivizes the proposer to produce increasingly difficult yet solvable tasks, thus establishing an automated curriculum to refine both agents. To enhance training efficiency, we also introduce hop-grouped relative policy optimization (HRPO). This method clusters structurally similar questions to construct group-level baselines, effectively minimizing the sampling overhead in evaluating each query's individual difficulty and solvability. Consequently, HRPO significantly reduces the compute requirements for proposer training and reward estimation without compromising performance or stability. Extensive experimental results demonstrate that Dr. Zero matches or surpasses fully supervised search agents on several question answering benchmarks, showing that strong agentic search and evidence-grounded reasoning can emerge solely through self-evolution.