Beyond Outcome Reward: Decoupling Search and Answering Improves LLM Agents
Proposes DeSA, a two-stage training framework that decouples search and answer generation, significantly improving recall and accuracy metrics.
Key Findings
Methodology
This work systematically analyzes the deficiencies of search agents trained solely with outcome rewards, revealing behaviors like redundant queries and invalid searches. It introduces DeSA, a two-stage framework: first, training with recall-based rewards to enhance search skills; second, fine-tuning with exact match rewards for answer accuracy. Using Qwen2.5-3B/7B models and GRPO algorithm, experiments across seven QA benchmarks show substantial improvements in search recall, defect reduction, and final answer accuracy, validating the decoupling approach.
Key Results
- DeSA reduces deficient search behaviors from 23.36% to 6.96%, increases search recall to 64.5%, and improves average EM accuracy by 8% over baseline methods across datasets.
- On NaturalQuestions and HotpotQA, DeSA outperforms single-stage outcome-only training in both retrieval efficiency and answer correctness, demonstrating the effectiveness of behavior decoupling.
- The approach effectively eliminates redundant and invalid searches, leading to resource savings and higher-quality answers, with significant performance gains verified through ablation studies.
Significance
This research addresses a core limitation in current RL-based search agents by emphasizing the importance of intermediate search behavior optimization. It offers a practical, scalable solution to improve factual accuracy and efficiency in large language models, with broad implications for AI-powered question answering, knowledge retrieval, and autonomous reasoning systems. The decoupling paradigm opens new avenues for designing more robust, interpretable, and adaptable AI agents, fostering progress toward truly autonomous intelligent systems.
Technical Contribution
The paper introduces a novel two-stage training framework, DeSA, which explicitly separates search skill acquisition from answer optimization. It leverages recall-based rewards to guide search behavior, combined with outcome rewards for answer quality, ensuring targeted learning. The use of GRPO enhances training stability. Extensive behavioral analysis and experiments demonstrate that this approach significantly outperforms traditional single-stage methods, providing a new blueprint for reinforcement learning in search-augmented models.
Novelty
This is the first systematic attempt to decouple search behavior optimization from answer generation in RL training for large language models. Unlike prior work that relies solely on outcome rewards, DeSA explicitly guides intermediate search actions, leading to better resource utilization and improved performance. The framework’s modular design and empirical validation across diverse datasets mark a significant advancement in the field.
Limitations
- The method depends on external retrieval systems, which can be computationally intensive and introduce latency. Its scalability to real-time applications needs further exploration.
- In highly complex or multi-modal tasks, the current decoupling strategy may require adaptation to handle richer information sources.
- Generalization to unseen domains or tasks remains a challenge, necessitating further research into transferability and robustness.
Future Work
Future directions include integrating multi-modal data, developing adaptive reward schemes, and reducing reliance on external retrieval systems through self-supervised techniques. Expanding to multi-task and real-time environments will enhance practical deployment. Additionally, exploring unsupervised or semi-supervised approaches for reward design could further improve training efficiency and model robustness.
AI Executive Summary
Large language models (LLMs) have demonstrated remarkable capabilities in natural language understanding and generation, yet their reliance on static knowledge bases and susceptibility to hallucinations remain significant challenges. External search tools have been integrated to mitigate these issues, enabling models to retrieve relevant information dynamically. However, training such search-augmented agents predominantly relies on outcome-based rewards, such as exact match, which implicitly assumes that optimizing the final answer will also improve intermediate search behaviors. This assumption, however, overlooks the complex, multi-step nature of effective search strategies.
In practice, agents trained solely with outcome rewards exhibit systematic deficiencies, including redundant queries, invalid tool calls, and failure to invoke search tools when necessary. These behaviors impair search recall and ultimately degrade answer accuracy, as evidenced by empirical analysis on models like Qwen2.5-3B/7B across multiple QA benchmarks. To address this, the authors propose DeSA, a decoupled two-stage training framework. The first stage emphasizes learning effective search behaviors through recall-based rewards, while the second fine-tunes answer generation using precise outcome rewards.
Experimental results demonstrate that DeSA significantly reduces search defects and enhances overall QA performance. For instance, it lowers the deficient search rate from over 23% to below 7%, while boosting search recall and answer accuracy by substantial margins. These improvements validate the hypothesis that explicit intermediate behavior optimization is crucial for building more reliable and efficient search agents.
The broader impact of this work lies in its potential to transform how reinforcement learning is applied in multi-step, tool-using AI systems. By systematically decoupling search and reasoning, it paves the way for more interpretable, scalable, and robust models capable of complex reasoning tasks. Future research will likely explore multi-modal extensions, adaptive reward schemes, and real-time deployment, further advancing autonomous AI agents in diverse domains.
Deep Dive
Plain Language Accessible to non-experts
想象你在厨房里准备一道复杂的菜肴。你需要先找到各种食材（搜索），确保每样都准备齐全（检索效果），然后再按照食谱（答案生成）把菜做出来。如果只关注最后的成品（最终答案），可能会忽略中间的准备工作，导致菜不够好吃或不完整。这个研究就像教厨师先专注于挑选好食材（搜索技能），再专注于烹饪（答案生成），两步分开训练，最后做出更美味的菜。这种方法让厨师既能快速找到好食材，又能做出完美的菜肴。
ELI14 Explained like you're 14
想象你在玩一个超级复杂的拼图游戏。你首先要找到正确的拼图片（搜索），这样才能拼出完整的图案（答案）。如果你只关心拼好后是不是像原图那样漂亮（最终答案），但没有学会怎么找拼图片，你可能会一直拼错、浪费时间。这个研究就像教你先学会怎么快速找到正确的拼图片（第一步：搜索技能），然后再学习怎么拼出漂亮的图（第二步：答案生成）。把这两件事分开训练，你会变成拼图高手，既快又准，拼出更漂亮的图。这就像让你先变成一个擅长找拼图片的高手，再变成拼图大师。
Abstract
Enabling large language models (LLMs) to utilize search tools offers a promising path to overcoming fundamental limitations such as knowledge cutoffs and hallucinations. Recent work has explored reinforcement learning (RL) for training search-augmented agents that interleave reasoning and retrieval before answering. These approaches usually rely on outcome-based rewards (e.g., exact match), implicitly assuming that optimizing for final answers will also yield effective intermediate search behaviors. Our analysis challenges this assumption: we uncover multiple systematic deficiencies in search that arise under outcome-only training and ultimately degrade final answer quality, including failure to invoke tools, invalid queries, and redundant searches. To address these shortcomings, we introduce DeSA (Decoupling Search-and-Answering), a simple two-stage training framework that explicitly separates search optimization from answer generation. In Stage 1, agents are trained to improve search effectiveness with retrieval recall-based rewards. In Stage 2, outcome rewards are employed to optimize final answer generation. Across seven QA benchmarks, DeSA-trained agents consistently improve search behaviors, delivering substantially higher search recall and answer accuracy than outcome-only baselines. Notably, DeSA outperforms single-stage training approaches that simultaneously optimize recall and outcome rewards, underscoring the necessity of explicitly decoupling the two objectives.