SlimSearcher: Training Efficiency-Aware Web Agents via Adaptive Reward Gating
SlimSearcher enhances training efficiency via adaptive reward gating, reducing tool calls by 17%-58%.
Key Findings
Methodology
SlimSearcher integrates supervised fine-tuning and reinforcement learning, optimizing tool and token efficiency through adaptive reward gating. In the SFT stage, Pareto-efficient filtration extracts successful and economical trajectories. During RL, a dynamic reward mechanism evaluates relative efficiency, avoiding brevity bias from absolute penalties.
Key Results
- SlimSearcher reduces tool-call rounds by 17%-58% while maintaining or improving accuracy on GAIA, BrowseComp, and XBenchDeepSearch.
- On GAIA, tool-call rounds decrease by 48.4%, accuracy improves to 70%.
- On BrowseComp, tool-call rounds decrease from 63.70 to 47.63, accuracy improves to 0.447.
Significance
SlimSearcher addresses the computational cost issue in complex information retrieval tasks by optimizing the Pareto frontier between efficiency and accuracy. This method enhances search efficiency while reducing unnecessary tool calls and token consumption.
Technical Contribution
SlimSearcher introduces adaptive efficiency anchoring, optimizing tool and token usage through a dynamic reward structure, significantly reducing computational waste. Compared to existing methods, this framework achieves a better balance between efficiency and accuracy.
Novelty
SlimSearcher is the first to integrate efficiency optimization across the entire training pipeline, avoiding reward hacking issues through adaptive reward gating, ensuring task accuracy while reducing computational redundancy.
Limitations
- In certain complex tasks, the model may still rely on external tools, reducing efficiency.
- The parameter settings of the adaptive reward mechanism may affect model stability.
- Efficiency optimization may be limited in extreme long-horizon tasks.
Future Work
Future work can explore more dynamic reward structures and adaptive parameter settings to further enhance model efficiency and stability.
AI Executive Summary
SlimSearcher is a novel framework that enhances training efficiency for deep research agents through adaptive reward gating. Existing models excel in complex information retrieval tasks but incur high computational costs. SlimSearcher optimizes tool and token efficiency by combining supervised fine-tuning and reinforcement learning, reducing unnecessary calls and consumption.
In experiments, SlimSearcher performs well on multiple long-horizon benchmarks, reducing tool-call rounds by 17%-58% while maintaining or improving accuracy. The method evaluates relative efficiency through a dynamic reward mechanism, avoiding brevity bias from absolute penalties, ensuring task accuracy.
While SlimSearcher makes significant progress in efficiency optimization, it still faces challenges in certain complex tasks where external tools are relied upon. Future research can explore more dynamic reward structures and adaptive parameter settings to further enhance model efficiency and stability.
Deep Analysis
Background
In recent years, deep learning models have made significant progress in information retrieval tasks, but computational costs remain a major issue. Existing methods often adopt brute-force strategies, relying on external tools for redundant reasoning and verification, leading to resource waste. SlimSearcher aims to address this issue by optimizing the Pareto frontier between efficiency and accuracy.
Core Problem
Deep research agents excel in complex information retrieval tasks but incur high computational costs. Existing models often rely on external tools for redundant reasoning and verification, leading to resource waste. How to optimize computational efficiency while maintaining task accuracy is an important and challenging problem.
Innovation
SlimSearcher introduces a framework combining supervised fine-tuning and reinforcement learning, optimizing tool and token efficiency through adaptive reward gating. Unlike existing methods, this framework evaluates relative efficiency through a dynamic reward structure, avoiding brevity bias from absolute penalties.
Methodology
- �� Conduct Pareto-efficient filtration in the SFT stage to extract successful and economical trajectories.
- �� Evaluate relative tool and token efficiency through adaptive reward gating in the RL stage.
- �� Avoid reward hacking issues through strict correctness gating, ensuring task accuracy.
Experiments
Experiments are conducted on long-horizon benchmarks such as GAIA, BrowseComp, and XBenchDeepSearch. We use a multi-dimensional framework to evaluate task success rate, tool-call rounds, and token consumption. Compared to existing methods, SlimSearcher achieves a better balance between efficiency and accuracy.
Results
SlimSearcher reduces tool-call rounds by 17%-58% while maintaining or improving accuracy on multiple benchmarks. On GAIA, tool-call rounds decrease by 48.4%, accuracy improves to 70%. On BrowseComp, tool-call rounds decrease from 63.70 to 47.63, accuracy improves to 0.447.
Applications
SlimSearcher can be used to optimize computational efficiency in complex information retrieval tasks, reducing unnecessary tool calls and token consumption. It can be applied to commercial search engines and open-source agents, enhancing system adaptability and efficiency.
Limitations & Outlook
While SlimSearcher makes significant progress in efficiency optimization, it still faces challenges in certain complex tasks where external tools are relied upon. The parameter settings of the adaptive reward mechanism may affect model stability. Efficiency optimization may be limited in extreme long-horizon tasks.
Plain Language Accessible to non-experts
Imagine you're cooking in a kitchen. You could choose to use a lot of tools to prepare ingredients, like a slicer or blender, but this might waste time and resources. SlimSearcher is like a smart chef who knows when to use tools and when to rely on their skills. By optimizing tool usage and reducing unnecessary steps, it ensures you can quickly and efficiently prepare a delicious meal.
ELI14 Explained like you're 14
Imagine you're playing a game where you need to find hidden treasure. You could choose to use various tools to help you, like maps and compasses, but this might waste time. SlimSearcher is like a smart player who knows when to use tools and when to rely on intuition. By optimizing tool usage and reducing unnecessary steps, it helps you quickly find the treasure!
Glossary
Pareto Frontier
In optimization problems, the Pareto frontier represents the best balance between efficiency and accuracy.
SlimSearcher enhances training efficiency by optimizing the Pareto frontier.
Adaptive Reward Gating
A dynamic reward mechanism for evaluating tool and token efficiency.
Used in the RL stage to optimize efficiency.
Supervised Fine-Tuning
Improves model efficiency awareness by filtering trajectories.
SlimSearcher conducts Pareto-efficient filtration in the SFT stage.
Tool-Call Rounds
The average number of external tool invocations per task.
SlimSearcher reduces tool-call rounds to enhance efficiency.
Token Consumption
The average number of tokens generated by the model per task.
SlimSearcher optimizes token consumption to reduce computational redundancy.
Open Questions Unanswered questions from this research
- 1 How to further optimize efficiency in extreme long-horizon tasks?
- 2 How do parameter settings of adaptive reward mechanisms affect model stability?
Applications
Immediate Applications
Commercial Search Engines
SlimSearcher can optimize computational efficiency in search engines, reducing unnecessary tool calls and token consumption.
Long-term Vision
Open-Source Agents
SlimSearcher can be applied to open-source agents, enhancing system adaptability and efficiency.
Abstract
Deep research agents have demonstrated remarkable capabilities in complex information-seeking tasks, yet this power comes at a steep computational cost. Driven by accuracy-focused training paradigms, current models adopt brute-force strategies characterized by blind tool dependency and performative reasoning-generating long, redundant trajectories that are far from necessary for resolving these tasks, leading to wasteful tool calls and excessive token consumption. To overcome this efficiency trap, we propose SlimSearcher, a principled framework that pushes the Pareto frontier between accuracy and computational cost across both Supervised Fine-Tuning (SFT) and Reinforcement Learning (RL). In the SFT stage, SlimSearcher employs Pareto-efficient filtration to distill trajectories that are both successful and economical, guiding the model toward inherently efficiency-aware search behaviors. During RL, we introduce Adaptive Reward Gating, a dynamic reward-shaping mechanism that evaluates relative tool and token efficiency within a sampled cohort. By cascading these adaptive efficiency metrics with a strict correctness gate, our approach effectively avoids the brevity bias associated with absolute penalties and mitigates reward hacking. Extensive experiments on long-horizon benchmarks, including GAIA, BrowseComp, and XBenchDeepSearch, demonstrate that SlimSearcher reduces average tool-call rounds by 17%-58% while maintaining or improving accuracy.