DeepSearch-World: Self-Distillation for Deep Search Agents in a Verifiable Environment
DeepSearch-Evolve achieves self-distillation in DeepSearch-World, reaching 93.4% on HotpotQA.
Key Findings
Methodology
DeepSearch-Evolve is a self-distillation framework leveraging the DeepSearch-World environment. This environment offers 420K multi-hop QA tasks constructed through entity-level random walks, supporting key cognitive behaviors like progress verification, reflection, and failure recovery. The framework iteratively trains stronger agents through trajectory generation, filtering, data mixing, and fine-tuning.
Key Results
- DeepSearch-World-9B achieves 93.4% on HotpotQA, 31.2% on BrowseComp, and 61.5% on GAIA, demonstrating competitive performance with open-source agents.
- On BrowseComp-ZH, DeepSearch-World-9B achieved 36.4%, showing significant improvement over the baseline.
- Through trajectory filtering and ReAct conversion, the model excels in tool use and entity hit rates.
Significance
This research demonstrates how verifiable environments can support scalable self-evolution for long-horizon web agents. Through a self-distillation framework, agents learn from their own tool-use experiences without relying on distillation from stronger models, providing a new direction for future research, especially in tasks requiring long-term interactions.
Technical Contribution
DeepSearch-Evolve offers a novel self-distillation approach by training agents through verifiable tool-use experiences. Unlike existing methods, this framework does not rely on sparse rewards or unreliable dense supervision but achieves agent self-improvement through controllable process-level supervision.
Novelty
This is the first implementation of self-distillation in a verifiable environment, utilizing the environment's determinism to provide process-level supervision. Compared to existing methods, DeepSearch-Evolve does not rely on external synthetic pipelines or distillation from stronger models.
Limitations
- The current environment is based on Wikipedia, limiting coverage and domain diversity.
- The update rule relies on self-distillation, which may limit generalization capabilities.
- How to inject higher-level capabilities into RL training remains unexplored.
Future Work
Future work could extend to broader knowledge sources to enhance generalization. Additionally, combining RL-style updates may further improve model flexibility and generalization capabilities.
AI Executive Summary
Training tool-use agents to autonomously improve remains challenging in the deep search domain. Existing methods either rely on fixed teacher-distilled trajectories or sparse-reward reinforcement learning, which fails to provide sufficient supervision. The DeepSearch-Evolve framework achieves self-distillation in a verifiable DeepSearch-World environment, offering a novel solution.
DeepSearch-World is a deterministic and verifiable environment containing 420K multi-hop QA tasks, supporting key cognitive behaviors like progress verification, reflection, and failure recovery. Through trajectory generation, filtering, data mixing, and fine-tuning, DeepSearch-Evolve iteratively trains stronger agents, achieving competitive performance with open-source agents.
Experimental results show that DeepSearch-World-9B performs excellently across multiple benchmarks, such as achieving 93.4% on HotpotQA. This indicates that verifiable environments can support scalable self-evolution for long-horizon web agents. Future research could explore broader knowledge sources and combine RL-style updates to further enhance model flexibility and generalization capabilities.
Deep Analysis
Background
Recent advances in large language models have enabled agents to move beyond passive text generation to plan multi-step tasks, formulate search queries, read documents, browse the web, and refine answers through tool use and iterative reasoning. However, enabling these agents to improve autonomously from their own interactions remains a key challenge toward scalable self-evolving agents.
Core Problem
Existing methods either rely on fixed teacher-distilled trajectories or sparse-reward reinforcement learning, which fails to provide sufficient supervision. Long-horizon agents require a verifiable environment that can expose process-level supervision over intermediate tool-use decisions.
Innovation
The DeepSearch-Evolve framework achieves self-distillation in a verifiable DeepSearch-World environment, offering a novel solution. This environment provides 420K multi-hop QA tasks, supporting key cognitive behaviors like progress verification, reflection, and failure recovery.
Methodology
- �� The DeepSearch-World environment provides deterministic and verifiable tool-use experiences.
- �� The DeepSearch-Evolve framework iteratively trains agents through trajectory generation, filtering, data mixing, and fine-tuning.
- �� Through process-level supervision, agents learn from their own tool-use experiences.
Experiments
Experiments are conducted on multiple benchmarks, including BrowseComp, GAIA, and HotpotQA. Training uses 420K multi-hop QA tasks, with performance evaluated on a validation set.
Results
DeepSearch-World-9B achieves 93.4% on HotpotQA, 31.2% on BrowseComp, and 61.5% on GAIA, demonstrating competitive performance with open-source agents.
Applications
The framework can be directly applied to tasks requiring long-term interactions, such as complex QA systems and information retrieval systems, with broad industry impact.
Limitations & Outlook
The current environment is based on Wikipedia, limiting coverage and domain diversity. The update rule relies on self-distillation, which may limit generalization capabilities. Future work could extend to broader knowledge sources.
Plain Language Accessible to non-experts
Imagine you're in a library with a robot assistant that helps you find answers. This robot can learn on its own and get better over time. DeepSearch-Evolve is like this robot assistant, trained in a virtual, verifiable environment, much like a safe library. By constantly trying and learning, it can become smarter without a teacher. This process is like the robot finding books, reading, and summarizing information, then doing better next time.
ELI14 Explained like you're 14
Hey, imagine you're playing a game with a super-smart assistant that helps you find all the answers! This assistant can learn on its own and get smarter. DeepSearch-Evolve is like this assistant, training itself in a virtual world, just like in a safe game environment. It learns from its attempts and keeps improving, just like you level up in a game. Isn't that cool?
Glossary
Self-Distillation
A training method where a model learns from its own generated data instead of relying on external teacher models.
In DeepSearch-Evolve, agents learn from their own tool-use experiences through self-distillation.
Verifiable Environment
A training environment that provides deterministic and verifiable feedback.
DeepSearch-World is a verifiable environment supporting agent self-evolution.
Multi-hop QA
Questions that require reasoning through multiple steps to answer.
DeepSearch-World contains 420K multi-hop QA tasks for training agents.
ReAct
A standard format for training agents, including planning, memory, and error recovery information.
In DeepSearch-Evolve, trajectories are converted to ReAct format for training.
Trajectory Filtering
A method for selecting high-quality training data by filtering out non-compliant trajectories.
DeepSearch-Evolve uses trajectory filtering to improve training data quality.
Open Questions Unanswered questions from this research
- 1 How to extend verifiable environments to broader knowledge sources to enhance generalization.
- 2 How to inject higher-level capabilities like planning and error recovery into RL training.
Applications
Immediate Applications
Complex QA Systems
DeepSearch-Evolve can be used to train systems capable of handling complex QA tasks, providing more accurate answers.
Long-term Vision
Automated Information Retrieval
This technology could be applied to automated information retrieval systems in the future, improving the efficiency and accuracy of information acquisition.
Abstract
Training tool-use agents to improve from their own experience remains challenging, as supervised fine-tuning relies on fixed teacher-distilled trajectories, while sparse-reward reinforcement learning provides weak supervision for long-horizon interactions. We present DeepSearch-Evolve, a self-distillation framework for web agents built on DeepSearch-World, a deterministic and verifiable environment with reproducible search and page-reading tools. DeepSearch-World contains 420K multi-hop QA tasks constructed from entity-level random walks and supports key agentic cognitive behaviors useful for self-evolving, including progress verification, grounded reflection, and failure recovery. DeepSearch-Evolve iteratively performs trajectory generation, filtering, data mixing, and fine-tuning to train stronger agents. Without distillation from more capable models, DeepSearch-World-9B achieves competitive performance compared with open-source agents, reaching 31.2% on BrowseComp, 61.5% on GAIA, and 93.4% on HotpotQA, showing that verifiable environments enable scalable self-evolution for long-horizon web agents. We will release the environment, 420K training pool, validation set, model, and code to facilitate future research on self-improving deep search agents.