ECHO: Prune To Act, Trace To Learn With Selective Turn Memory In Agentic RL
ECHO achieves 43.4% accuracy in Agentic RL using selective turn memory, outperforming GRPO and SUPO.
Key Findings
Methodology
ECHO employs a selective turn-memory framework by compressing each environment turn into a source-indexed memory record, selecting useful records to reconstruct policy contexts, and reusing selected source indices to route positive outcome credit to the final trajectory segment. This method excels on the BrowseComp-Plus benchmark.
Key Results
- ECHO achieves 43.4% held-out accuracy on BrowseComp-Plus, significantly outperforming GRPO's 28.9% and SUPO's 36.1%.
- ECHO uses fewer turns and trajectory volume than SUPO, indicating its resource efficiency.
- ECHO demonstrates zero-shot generalization across multi-objective QA, code generation, and deep information-seeking benchmarks.
Significance
ECHO provides an effective context management method for long-horizon language agents, addressing the credit assignment issues caused by context compression in traditional methods. Its generalization capabilities and resource efficiency enhance the feasibility of reinforcement learning in practical applications.
Technical Contribution
ECHO introduces a novel context reconstruction method through selective turn memory and traceable credit routing, distinguishing it from existing collapsed-history methods. It not only improves accuracy but also reduces unnecessary searches and reasoning.
Novelty
ECHO is the first to integrate selective memory with reinforcement learning, achieving unified context reconstruction and credit assignment. Compared to SUPO, ECHO retains source-level traceability.
Limitations
- ECHO may still face challenges when handling extremely long contexts, especially in resource-constrained environments.
- In some complex tasks, the effectiveness of selective memory may not meet expectations.
- Further research is needed to explore its applicability across different tasks.
Future Work
Future research can explore ECHO's application in other complex tasks and optimize its performance in resource-constrained environments. Additionally, further theoretical analysis can help understand its performance differences across tasks.
AI Executive Summary
ECHO is an innovative selective turn-memory framework designed to address context management issues in long-horizon language agents. Traditional methods often lose important source information when compressing history, leading to difficulties in credit assignment. ECHO solves this by compressing each completed environment turn into a compact source-indexed memory record and selecting useful records to reconstruct policy contexts, achieving unified context reconstruction and credit assignment.
In the BrowseComp-Plus benchmark, ECHO achieves 43.4% held-out accuracy, significantly outperforming GRPO and SUPO. This demonstrates ECHO's improvements in both accuracy and resource utilization efficiency. Additionally, ECHO shows strong zero-shot generalization capabilities across multi-objective QA, code generation, and deep information-seeking tasks.
Despite ECHO's strong performance in many areas, it still faces challenges in handling extremely long contexts and resource-constrained environments. Future research can further optimize its performance across different tasks and explore its potential in broader applications.
Deep Analysis
Background
With the rise of large-scale language models, reinforcement learning has become increasingly applied in multi-turn dialogue, tool invocation, and environment feedback. However, as interaction horizons grow, history management becomes a bottleneck. Traditional methods often lose important source information when compressing history, leading to difficulties in credit assignment.
Core Problem
In long-horizon language agents, effectively managing context to retain useful information and achieve accurate credit assignment is a core challenge. Traditional methods often lose important source information when compressing history, leading to difficulties in credit assignment.
Innovation
ECHO achieves unified context reconstruction and credit assignment through a selective turn-memory framework. Its innovations include compressing each completed environment turn into a compact source-indexed memory record and selecting useful records to reconstruct policy contexts.
Methodology
- �� Compress each completed environment turn into a compact source-indexed memory record.
- �� Select useful records to reconstruct policy contexts.
- �� Reuse selected source indices to route positive outcome credit to the final trajectory segment.
Experiments
In the BrowseComp-Plus benchmark, ECHO achieves 43.4% held-out accuracy, significantly outperforming GRPO and SUPO. The experimental design includes using the same search/open-page tool environment, verifier, and 32k-token working context budget.
Results
ECHO achieves 43.4% held-out accuracy on BrowseComp-Plus, significantly outperforming GRPO's 28.9% and SUPO's 36.1%. Additionally, ECHO uses fewer turns and trajectory volume than SUPO, indicating its resource efficiency.
Applications
ECHO demonstrates strong zero-shot generalization capabilities across multi-objective QA, code generation, and deep information-seeking tasks. Its improvements in resource efficiency and accuracy make it highly applicable in practical scenarios.
Limitations & Outlook
Despite ECHO's strong performance in many areas, it still faces challenges in handling extremely long contexts and resource-constrained environments. Future research can further optimize its performance across different tasks and explore its potential in broader applications.
Plain Language Accessible to non-experts
Imagine you're in a library looking for a book. Traditional methods are like compressing all books into one summary, saving space but making it hard to find the information you need. ECHO's approach is more like creating an index card for each book, noting the title, author, and key content. When you need information, you can quickly find the relevant card instead of sifting through the entire summary. This method not only helps you find information quickly but also ensures you know where the information came from.
ELI14 Explained like you're 14
Imagine you're playing a massive multiplayer online game. You need to remember lots of quests and clues, but your memory is limited. Traditional methods are like writing all quests on one long list, saving space but making it hard to find important quests. ECHO's approach is more like creating a small card for each quest, noting the quest name, key clues, and steps to complete. When you need to complete a quest, you can quickly find the relevant card instead of sifting through the entire list. This method not only helps you complete quests quickly but also ensures you know where the quests came from!
Glossary
Selective Turn Memory
A method that compresses each environment turn into a compact source-indexed memory record.
Used for context reconstruction and credit assignment.
Context Reconstruction
The process of rebuilding policy contexts by selecting useful memory records.
Core mechanism of ECHO.
Credit Assignment
The process of routing positive outcome credit to relevant trajectory segments.
Used to improve learning efficiency.
BrowseComp-Plus
A long-horizon tool-use QA benchmark.
Used to evaluate ECHO's performance.
GRPO
A traditional multi-turn reinforcement learning method.
Compared with ECHO for performance.
Open Questions Unanswered questions from this research
- 1 How to optimize ECHO's performance in resource-constrained environments remains to be further studied.
- 2 ECHO's applicability and performance differences across tasks need more experimental validation.
Applications
Immediate Applications
Multi-objective QA
ECHO can be used to improve the accuracy and efficiency of complex QA systems.
Long-term Vision
Intelligent Assistants
ECHO can be used to develop smarter personal assistants, providing more accurate and personalized services.
Abstract
Long-horizon language agents must repeatedly interact with tools, accumulate evidence, and make decisions under bounded context windows. Context-management methods make such rollouts feasible by simplifying past interactions through deletion, folding, or memory editing. However, when useful history is collapsed into compressed states, the reconstructed context may no longer reveal which earlier observations support a successful final answer. This creates a mismatch between bounded-context acting and outcome-based reinforcement learning: the policy acts on reconstructed context, while the learner lacks source-level provenance for assigning credit to the evidence that mattered. We propose ECHO, a selective turn-memory framework for traceable context reconstruction in Agentic RL. ECHO compresses each completed environment turn into a compact source-indexed memory record, reconstructs bounded policy contexts by selecting useful records, and reuses the selected source indices to route positive outcome credit to the final trajectory segment, reused evidence turns, memory findings, and memory-selection actions. On BrowseComp-Plus, ECHO reaches 43.4% held-out accuracy, outperforming GRPO at 28.9% and the rolling-summary baseline SUPO at 36.1%, while using fewer turns and lower trajectory volume than SUPO. The trained policy also improves zero-shot generalization across multi-objective QA, code generation, and deep information-seeking benchmarks on both dense and MoE backbones.