CAFE: Self-Improving Search Agents Need Co-Evolving Feedback
CAFE framework enhances search agents via co-evolving feedback, excelling on seven benchmarks.
Key Findings
Methodology
The CAFE framework integrates search agent and critic roles through a shared-parameter model with alternating optimization. Online RL uses comparative feedback estimates and feedback-aware advantage shaping to optimize request returns. Offline, rollout-derived preference optimization learns feedback.
Key Results
- CAFE outperforms other RL-based search agents on seven benchmarks, with a 2.1 EM and 1.3 F1 improvement.
- It maintains performance gains across six out-of-domain benchmarks and reduces answer-level hallucinations.
- One-sided ablations show that improving only the agent or critic plateaus, while alternating updates continue to enhance performance.
Significance
The CAFE framework addresses early directional errors in long-horizon searches through co-evolving feedback, significantly enhancing self-improvement in search agents. This research offers a new perspective on search agent design, emphasizing the importance of co-evolution between feedback and policy for performance improvement.
Technical Contribution
CAFE fundamentally differs from existing SOTA methods by introducing feedback-aware advantage shaping and rollout-derived preference optimization, offering new theoretical guarantees and engineering possibilities.
Novelty
CAFE is the first to propose the concept of co-evolution between feedback and policy, distinguishing it from previous fixed feedback mechanisms by emphasizing dynamic adaptability.
Limitations
- CAFE may face performance bottlenecks in handling extremely complex search tasks due to increased complexity in feedback generation and policy updates.
- Requires significant computational resources for training, potentially unsuitable for resource-limited environments.
Future Work
Future research can explore CAFE's adaptability across different domains, particularly in optimizing feedback generation and policy updates in resource-constrained environments.
AI Executive Summary
The CAFE framework enhances search agents by addressing early directional errors in long-horizon searches through co-evolving feedback. On seven benchmarks, CAFE outperforms other RL-based search agents with a 2.1 EM and 1.3 F1 improvement, maintaining performance gains across six out-of-domain benchmarks. One-sided ablations show that improving only the agent or critic plateaus, while alternating updates continue to enhance performance. CAFE fundamentally differs from existing SOTA methods by introducing feedback-aware advantage shaping and rollout-derived preference optimization, offering new theoretical guarantees and engineering possibilities. Future research can explore CAFE's adaptability across different domains, particularly in optimizing feedback generation and policy updates in resource-constrained environments.
Deep Analysis
Background
In recent years, search agents have played a crucial role in solving knowledge-intensive problems. However, existing methods often rely on fixed feedback mechanisms, struggling to adapt to dynamically changing search environments. The CAFE framework addresses this issue through co-evolving feedback mechanisms.
Core Problem
Early directional errors in long-horizon searches often lead to cumulative errors in subsequent steps. Existing methods struggle to provide immediate corrective signals, affecting overall performance.
Innovation
CAFE is the first to propose co-evolution between feedback and policy, emphasizing dynamic adaptability. By alternating optimization of search agent and critic roles through a shared-parameter model, it enhances self-improvement in search agents.
Methodology
- �� CAFE integrates search agent and critic roles through a shared-parameter model with alternating optimization. • Online RL uses comparative feedback estimates and feedback-aware advantage shaping to optimize request returns. • Offline, rollout-derived preference optimization learns feedback.
Experiments
Experiments were conducted on seven benchmarks using the Qwen2.5-7/3B-Instruct model, comparing CAFE with other RL-based search agents. Results show CAFE's superior performance across all benchmarks.
Results
CAFE outperforms other RL-based search agents on seven benchmarks, with a 2.1 EM and 1.3 F1 improvement, maintaining performance gains across six out-of-domain benchmarks.
Applications
CAFE can be applied to search tasks requiring high precision and dynamic adaptability, such as complex problem solving and knowledge retrieval.
Limitations & Outlook
CAFE may face performance bottlenecks in handling extremely complex search tasks due to increased complexity in feedback generation and policy updates.
Plain Language Accessible to non-experts
Imagine you're navigating a maze. CAFE acts like a smart assistant that not only tells you which paths are wrong but also adjusts its advice in real-time based on your progress, helping you find the exit faster. By continuously learning and adjusting, CAFE performs better in complex mazes.
ELI14 Explained like you're 14
Imagine playing a super complex maze game, and CAFE is like a super smart helper. It gives you hints when you take a wrong turn and adjusts its strategy based on your progress, helping you find the exit faster. Isn't that cool?
Glossary
CAFE (Co-Evolving Feedback)
A framework that enhances search agent performance through co-evolving feedback and policy.
CAFE is used to improve search agent performance in long-horizon searches.
EM (Exact Match)
A metric that evaluates the degree of match between model output and the standard answer.
Used to evaluate CAFE's performance in benchmark tests.
F1 (F1 Score)
A metric combining precision and recall.
Used to evaluate CAFE's performance in benchmark tests.
RL (Reinforcement Learning)
A machine learning method that learns policies through trial and error to maximize rewards.
CAFE uses RL for online optimization.
SFT (Supervised Fine-Tuning)
A method of model fine-tuning through supervised learning.
Used to initialize the CAFE framework.
Open Questions Unanswered questions from this research
- 1 How to optimize CAFE's feedback generation and policy updates in resource-limited environments?
- 2 How to overcome performance bottlenecks in extremely complex tasks with CAFE?
Applications
Immediate Applications
Complex Problem Solving
CAFE can be applied to complex problem-solving tasks requiring high precision and dynamic adaptability.
Long-term Vision
Knowledge Retrieval
CAFE has the potential to achieve more efficient search and information retrieval in the field of knowledge retrieval.
Abstract
Reliable search requires more than acquiring external evidence. An agent must also recognize and recover from errors as its trajectory unfolds. In-trajectory feedback provides a mechanism for such recovery by diagnosing where the search has drifted and redirecting subsequent reasoning steps. This is particularly important in long-horizon search, where an early directional error may receive no immediate corrective signal and can compound across later steps. Making such feedback learnable, however, creates a coupled problem: the agent must learn when to request and use feedback, while the critic must learn corrections from outcome-confounded rollouts as the agent's failure patterns evolve. We introduce CAFE (Coupled Agent--Feedback Evolution), a framework in which a shared-parameter model alternates between search-agent and critic roles. CAFE initializes feedback-conditioned recovery from trajectories built around the base agent's own failures, then couples online and offline optimization. During online RL, a comparative feedback estimate uses a prompt-level call--skip success gap to shape request returns, while feedback-aware advantage shaping reweights token advantages before and after feedback. Offline, rollout-derived preference optimization learns feedback from matched successful and unsuccessful trajectories. On seven agentic search benchmarks, CAFE outperforms the evaluated RL-based search agents on average, retains its gains across all six out-of-domain benchmarks, and reduces answer-level hallucinations. One-sided ablations show that improving only the agent or only the critic eventually plateaus, whereas alternating the two updates continues to improve performance. These findings suggest that a self-improving search agent needs feedback that co-evolves with the policy it guides.