Ask Only When Needed: Proactive Retrieval from Memory and Skills for Experience-Driven Lifelong Agents
ProactAgent combines EXPONEVO and PROACTRL for proactive memory retrieval, achieving 32% higher success rate and 33% fewer rounds on SciWorld.
Key Findings
Methodology
ProactAgent integrates EXPONEVO and PROACTRL. EXPONEVO refines memory and optimizes policies dynamically, organizing past interactions into structured repositories. PROACTRL models retrieval as a policy action, learning when and what to retrieve using paired-branch process rewards.
Key Results
- Achieved 73.5% success rate on SciWorld, 18% higher than GRPO+Reflexion, with 33.2% fewer interaction rounds.
- Achieved 71.28% success rate on AlfWorld with a 22.5% reduction in interaction rounds.
- Approached proprietary model performance on StuLife with a StuGPA of 19.26.
Significance
The research addresses limitations of passive retrieval in lifelong learning, significantly improving efficiency and success rates for long-horizon tasks, and advancing cross-task knowledge utilization.
Technical Contribution
Introduced proactive retrieval as a policy action with paired-branch process rewards, enabling step-level supervision. Unified memory refinement and policy evolution, enhancing lifelong agent performance and efficiency.
Novelty
First to model retrieval as a proactive policy action. Introduced a novel paired-branch reward mechanism, enabling precise retrieval decisions and addressing gaps in prior passive methods.
Limitations
- Requires high-quality initial experience base and supervised cold-start phase.
- Potential retrieval delays in complex tasks.
- High storage and computational demands for the experience base.
Future Work
Future work includes optimizing retrieval efficiency, exploring performance in more complex tasks, and reducing storage and computation costs for the experience base.
AI Executive Summary
ProactAgent is a novel lifelong learning framework addressing the challenge of proactive experience utilization in online long-horizon tasks.
Through its EXPONEVO module, ProactAgent dynamically refines memory and optimizes policies, organizing past interactions into factual, episodic, and skill repositories. The PROACTRL module models retrieval as a policy action, learning when and what to retrieve using paired-branch process rewards to ensure retrieval occurs only when it improves task outcomes or efficiency.
Experiments on SciWorld, AlfWorld, and StuLife demonstrate ProactAgent's superiority over baselines, achieving a 73.5% success rate on SciWorld with a 33% reduction in interaction rounds. This framework represents a significant step forward in proactive knowledge utilization for lifelong agents, with potential for further research in retrieval efficiency and complex task adaptation.
Deep Analysis
Background
Lifelong learning agents must adapt to continuous tasks while accumulating experience. Existing methods rely on passive retrieval, limiting their ability to address dynamic knowledge gaps.
Core Problem
Current methods fail to dynamically retrieve relevant experience when task dynamics change. Memory updates and policy optimization are treated independently, limiting the potential of lifelong learning.
Innovation
ProactAgent addresses these issues with proactive retrieval and joint memory-policy evolution. EXPONEVO dynamically updates memory and policies, while PROACTRL learns retrieval timing and content through paired-branch rewards.
Methodology
- �� EXPONEVO: Organizes experience into factual, episodic, and skill repositories, dynamically refining memory and policies.
- �� PROACTRL: Models retrieval as a policy action, using paired-branch process rewards to learn retrieval timing and content.
- �� Experience Base: Structured storage with type-based decomposition, combining similarity and priority scoring for relevant retrieval.
Experiments
Evaluated on SciWorld, AlfWorld, and StuLife using the Qwen2.5-7B-Instruct model. Compared against multiple baselines with ablation studies to validate contributions.
Results
ProactAgent achieved 73.5% success rate on SciWorld, reducing interaction rounds to 18.38. On AlfWorld, it reached 71.28% success rate with 12.73 rounds. On StuLife, it approached proprietary model performance.
Applications
Applicable to scenarios requiring cross-task knowledge utilization, such as educational assistants, smart customer support, and long-term task planning.
Limitations & Outlook
Relies on high-quality initial experience base, may face retrieval delays in complex tasks, and requires significant storage and computation for the experience base.
Plain Language Accessible to non-experts
Imagine you're cooking in a kitchen with a fridge full of ingredients. ProactAgent acts like a smart assistant that knows exactly when to open the fridge and grab the right ingredient, saving time and avoiding unnecessary effort.
ELI14 Explained like you're 14
Imagine you're playing a puzzle game with a treasure chest full of tools. ProactAgent is like a super-smart teammate who knows exactly when to open the chest and always picks the best tool for the job. Cool, right?
Glossary
ProactAgent
A lifelong learning framework that improves task efficiency and success through proactive memory retrieval.
Used for addressing long-horizon task challenges.
EXPONEVO
A module for refining memory and optimizing policies dynamically.
Organizes experience into structured repositories.
PROACTRL
A reinforcement learning-based retrieval mechanism that learns when and what to retrieve.
Improves retrieval efficiency and accuracy.
Paired-branch rewards
A mechanism comparing retrieval and no-retrieval branches to provide step-level supervision.
Optimizes retrieval decisions.
Experience Base
A structured database storing factual, episodic, and skill-based knowledge.
Supports retrieval for decision-making.
Open Questions Unanswered questions from this research
- 1 How to optimize retrieval efficiency for complex tasks?
- 2 How to reduce storage and computational costs for the experience base?
- 3 How to enable cold-start in unsupervised environments?
Applications
Immediate Applications
Educational Assistant
Helps students dynamically retrieve learning materials, improving study efficiency.
Smart Customer Support
Proactively retrieves relevant information to provide precise answers to user queries.
Long-term Vision
Long-term Task Planning
Supports robots in dynamically adapting and optimizing decisions in complex tasks.
Abstract
Online lifelong learning agents must decide not only how to act but also when to consult prior experience to continually improve on long-horizon tasks. Existing methods typically retrieve memories passively, such as at task initialization or after each step, and therefore miss knowledge gaps that arise during interaction. We propose ProactAgent, an experience-driven lifelong learning framework for proactive retrieval over a structured Experience Base. ProactAgent continually improves through ExpOnEvo, which jointly updates policies and refines memory, organizing past interactions into factual, episodic, and skill repositories. It further introduces ProactRL, which treats retrieval as an explicit policy action and learns when and what to retrieve. By comparing paired continuations from identical interaction prefixes with and without retrieval, ProactRL provides step-level process rewards that encourage retrieval only when it improves task outcomes or efficiency. Experiments on SciWorld, AlfWorld, and StuLife show that ProactAgent consistently outperforms all baselines, achieving up to 32% relative improvement in success rate and over 33% reduction in interaction rounds. Our code will be publicly available at GitHub.