SearchMaster: Grounded and Regulated Self-Play for Search Agents
SearchMaster improves search agent accuracy from 38.19% to 51.52% through self-play.
Key Findings
Methodology
SearchMaster employs a self-play framework to train a single LLM by generating, solving, and verifying search tasks. Its core mechanisms include an Evidence-Chain Generator (ECG), Search-Depth Reward (SDR), and Over-Opening Penalty (OOP) to address pseudo multi-hop issues, evaluate task difficulty by search depth, and limit document openings.
Key Results
- On the BrowseComp-Plus benchmark, SearchMaster improved the Qwen3.5-9B model's accuracy from 30.12% to 60.24%, significantly outperforming other open-source and proprietary models.
- Across six deep-search benchmarks, the average accuracy increased from 38.19% to 51.52%.
- Ablation studies show that ECG, SDR, and OOP mechanisms all contribute to performance improvements.
Significance
SearchMaster demonstrates the potential to generate high-quality search data through self-play without human-labeled data or expert demonstrations. This is significant for academia and industry, particularly in reducing data acquisition costs and enhancing search agent performance.
Technical Contribution
SearchMaster introduces evidence-chain generation, search-depth reward, and over-opening penalty to solve pseudo multi-hop and tool misuse issues in self-play, providing a new technical pathway for search agent training.
Novelty
SearchMaster is the first to combine evidence-chain generation with search-depth reward for self-play training of search agents, significantly improving the quality and difficulty of multi-hop retrieval tasks.
Limitations
- The method may generate overly complex tasks in some scenarios, making them unsolvable by the model.
- Search-depth reward might lead to overly complex search paths.
Future Work
Future work could explore applying SearchMaster in larger search environments and integrating more external knowledge bases to enhance model generalization.
AI Executive Summary
SearchMaster is an innovative self-play framework designed to train large language models (LLMs) as search agents. Traditional search agent training relies on human-labeled data or expert demonstrations, but SearchMaster reduces this dependency by generating, solving, and verifying tasks autonomously.
The core of this method lies in three mechanisms: the Evidence-Chain Generator (ECG) reduces pseudo multi-hop issues, the Search-Depth Reward (SDR) evaluates task difficulty by search depth, and the Over-Opening Penalty (OOP) limits document openings. These mechanisms work together to ensure that the generated tasks are challenging and genuine.
Experimental results show that SearchMaster significantly improves model accuracy across multiple benchmarks, particularly on BrowseComp-Plus, where it achieved a 30.1-point increase. This demonstrates that high-quality data generated through self-play can effectively enhance search agent performance without relying on human annotations or expert guidance.
Deep Analysis
Background
In recent years, search agents based on large language models (LLMs) have excelled in knowledge-intensive question answering. However, these models typically rely on human-labeled data or expert demonstrations for training, which is costly and time-consuming. SearchMaster aims to address this issue through a self-play framework.
Core Problem
Training efficient search agents requires high-quality search data, which demands genuine multi-hop retrieval tasks and effective use of search tools. Existing methods rely on human annotations or expert demonstrations, making large-scale acquisition challenging.
Innovation
SearchMaster's core innovations include:
- �� Evidence-Chain Generator (ECG): reduces pseudo multi-hop issues.
- �� Search-Depth Reward (SDR): evaluates task difficulty by search depth.
- �� Over-Opening Penalty (OOP): limits document openings.
Methodology
SearchMaster's training process includes the following steps:
- �� Evidence-Chain Generator (ECG) generates tasks ensuring cross-document evidence is needed.
- �� Search-Depth Reward (SDR) evaluates task difficulty by successful search depth.
- �� Over-Opening Penalty (OOP) limits document openings to avoid shallow browsing.
- �� Verified Proposer and Solver rollouts are optimized with GRPO.
Experiments
Experiments were conducted on six deep-search benchmarks, including BrowseComp-Plus and other online search benchmarks. Qwen3.5-9B was used as the base model to evaluate SearchMaster's performance improvements across different datasets.
Results
SearchMaster improved accuracy on the BrowseComp-Plus benchmark from 30.12% to 60.24% and performed well on five other online search benchmarks, demonstrating its generalization capability across different environments.
Applications
SearchMaster can be used to train search agents without human-labeled data, suitable for fields requiring efficient information retrieval, such as law, medicine, and academic research.
Limitations & Outlook
Despite its strong performance on multiple benchmarks, SearchMaster may underperform on certain complex tasks. Additionally, reliance on search-depth reward might lead to overly complex search paths.
Plain Language Accessible to non-experts
Imagine you're in a library looking for a specific book. You need to find relevant information from different shelves and then combine this information to find the book you want. This is similar to how SearchMaster works: it generates tasks, finds information from different documents, and combines this information to answer questions. In this way, it can improve search efficiency without relying on human help.
ELI14 Explained like you're 14
Hey there! Imagine you're playing a treasure hunt game. You need to find different clues hidden in different places. Each clue brings you closer to the treasure! SearchMaster is like your treasure hunt helper, it helps you find all the clues and then tells you where the treasure is. Isn't that cool?
Glossary
Self-Play
A method where the model generates tasks and learns from them autonomously.
Used to train search agents to reduce dependency on external data.
Evidence-Chain Generator
A mechanism that generates questions requiring cross-document evidence.
Used to reduce pseudo multi-hop issues.
Search-Depth Reward
A mechanism that evaluates task difficulty by search depth.
Ensures tasks are challenging.
Over-Opening Penalty
A mechanism that limits document openings.
Avoids shallow browsing.
GRPO
A strategy for optimizing model training.
Used to optimize verified Proposer and Solver rollouts.
Open Questions Unanswered questions from this research
- 1 How to apply SearchMaster in larger search environments remains to be explored.
- 2 Integrating more external knowledge bases to enhance model generalization is a future research direction.
Applications
Immediate Applications
Legal Information Retrieval
SearchMaster can be used for information retrieval in the legal field, helping lawyers quickly find relevant cases and legal texts.
Long-term Vision
Medical Research
In medical research, SearchMaster can help researchers quickly retrieve relevant literature and research findings, promoting medical advancements.
Abstract
Training LLM-based search agents requires high-quality search data: tasks that demand genuine multi-hop retrieval and trajectories that use search tools effectively. Existing pipelines often depend on human-written tasks, expert demonstrations, or stronger teacher models. We present SearchMaster, a self-play framework that trains a single LLM from search tasks it generates, solves, and verifies in a local search environment. The key challenge is that self-generated tasks and rollouts can yield misleading signals: pseudo multi-hop questions, success-rate difficulty estimates that ignore search depth, and rollouts with excessive opening but little targeted evidence acquisition. SearchMaster addresses these failure modes with three controls. An Evidence-Chain Generator (ECG) grounds task generation in explicit cross-document evidence chains to reduce pseudo multi-hop questions. A Search-Depth Reward (SDR) scores task difficulty by the search depth of successful rollouts rather than success rate alone, keeping retained tasks search-intensive. An Over-Opening Penalty (OOP) regulates tool use by discouraging excessive document opening, avoiding long but shallow browsing. Verified Proposer and Solver rollouts are then jointly optimized with GRPO. Across six deep-search benchmarks, SearchMaster improves a Qwen3.5-9B backbone from 38.19% to 51.52% average accuracy, with a 30.1-point gain on BrowseComp-Plus. These results show that grounded and regulated self-play can provide effective search-agent training data without human-labeled QA pairs or expert demonstrations. The code is available at https://github.com/WentaoTan/SearchMaster.