OPD-Evolver: Cultivating Holistic Agent Evolver via On-Policy Distillation
OPD-Evolver uses slow-fast on-policy distillation to learn memory selection, use, writing, and maintenance; its 9B model challenges 397B-scale systems.
Key Findings
Methodology
OPD-Evolver combines a fast test-time evolution loop with a slow training loop. The fast loop operates over trajectory, tip, skill, and tool memories, performing retrieval, selection, execution, writing, and periodic maintenance. The slow loop computes outcome-calibrated memory attribution V(m), then distills privileged hindsight into the same deployable policy. It jointly trains selection, action, writing, and maintenance using full-vocabulary token-level KL on student-generated prefixes.
Key Results
- Across 10 subsets of LifelongAgentBench, MemoryArena, AMA-Bench, and InterCode, OPD-Evolver-9B outperforms same-backbone memory baselines: OS reaches 65.00% versus MemEvolve’s 61.00%, AMA-SA reaches 52.92% versus AWM’s 48.00%, and CTF reaches 57.00% versus MemEvolve’s 53.00%.
- OPD-Evolver-9B exceeds STEP-3.5-FLASH (196B) on 9/10 subsets and QWEN3.5-397B-A17B on 6/10. It scores 64.01% on InterCode SQL versus 62.74% for QWEN3.5-397B-A17B, showing that lifecycle-level evolution can offset a large parameter gap.
- Ablations show that attribution is crucial: removing it reduces InterCode average performance from 38.67% to 32.13%, with Bash/CTF/SQL drops of 4.96/7.31/7.36 points. Removing slow evolution yields 33.10% average performance; removing writing distillation lowers CTF from 34.00% to 29.00%.
Significance
The paper reframes memory from passive storage into an active mechanism for continual behavioral improvement. It addresses a persistent weakness of memory agents: they may retrieve reflections or skills, yet lack reliable judgments about usefulness, reusable writing, and long-term repository health. Academically, the four-capability lifecycle gives self-evolving-agent research a unified target. Industrially, it enables compact models to adapt through external experience while gradually internalizing high-value behavior, potentially reducing dependence on giant models and repeated prompt engineering.
Technical Contribution
The main contribution is converting delayed task return into memory-level supervision. For retrieved candidates, the method compares returns when a memory is selected versus not selected within controlled task groups, then weights the estimate by ρg, tier prior αℓ, and confidence γ to obtain V(m). A privileged teacher additionally observes successful trajectories, future utility of written memories, usage statistics, and redundancy diagnostics. The student samples under deployment conditions and minimizes token-level KL on its own prefixes, unifying selection, action, writing, and maintenance rather than optimizing execution alone.
Novelty
The novelty is not a new memory buffer or an isolated use of OPD; it is the lifecycle-level integration of four coupled evolver abilities into one deployable policy. Unlike ReasoningBank-style prompt augmentation, OPD-Evolver learns which memories deserve selection and preservation. Unlike Skill0, SFT, or GRPO, which primarily improve task execution, it distills privileged hindsight about memory value, reusable writing, and repository operations. The paper therefore shifts the unit of learning from a trajectory to the entire experience lifecycle.
Limitations
- Training uses Agent World Model, Nemotron-Terminal-Corpus, and EnvScaler, all disjoint from evaluation, but the paper does not establish robustness in open-ended, rapidly changing real environments. Performance may depend on the structure of these experience sources.
- Attribution relies on task grouping, retrieval logs, and sufficient comparisons between selected and unselected memories. Sparse rewards, confounded co-occurring memories, or poor grouping can produce incorrect V(m), causing useful memories to be ignored or deleted.
- Privileged teacher contexts, periodic maintenance, and multi-view distillation add training and systems complexity; repository scaling, privacy, and long-horizon cost are not systematically quantified.
Future Work
Promising directions include causal or counterfactual memory attribution, calibrated uncertainty for rare experiences, automatic task grouping, and scalable retrieval for million-item repositories. Future evaluations should test distribution drift, persistent deployment, privacy-preserving memory, multi-agent sharing, and embodied robotics. Combining OPD-Evolver with reinforcement learning, active forgetting, and safety constraints could improve both adaptability and reliability.
AI Executive Summary
Memory is now common in language-model agents, but storage alone does not create self-evolution. Systems such as ReasoningBank can preserve trajectories, reflections, or skills, yet often fail to decide which experience matters, how it should guide multi-turn action, what new knowledge is reusable, and when old entries should be merged or retired. These failures compound: noisy retrieval harms execution, poor writing pollutes future context, and unmanaged repositories degrade over time.
OPD-Evolver addresses this gap through a slow-fast co-evolution framework. Its fast loop performs test-time evolution over four memory levels—trajectories, tips, skills, and tools. For each task it retrieves candidates, selects a compact context, acts, writes new memories, and every 30 tasks performs lookup, merge, and delete maintenance. Its slow loop turns delayed environmental return into supervision. Outcome-calibrated attribution estimates a memory’s value by comparing selected and unselected outcomes within task groups. A privileged teacher then exposes valuable memories, successful trajectories, future writing utility, and repository diagnostics, while the student learns on its own deployment-state prefixes through on-policy token-level KL distillation.
The results suggest a substantial shift in capability. OPD-Evolver-9B reaches 65.00% on LifelongAgentBench OS, 52.92% on AMA-Bench State Abstraction, and 57.00% on InterCode CTF. It beats the 196B STEP-3.5-FLASH on 9/10 subsets and the QWEN3.5-397B-A17B on 6/10, including SQL, where it scores 64.01% versus 62.74%. Ablations confirm that attribution, slow evolution, writing distillation, selection, and maintenance are all consequential. The method points beyond memory-augmented agents toward compact policies that learn how to manage their own improvement, although causal attribution, long-term scaling, safety, and real-world validation remain open challenges.
Deep Analysis
Background
Self-evolving agents have progressed from reflection prompting and trajectory storage to skill libraries and parameter training. Representative systems include ExpeL, AWM, Cheatsheet, MemP, ReasoningBank, EvolveR, MemEvolve, and Skill0. These methods demonstrate that prior experience can improve later behavior, but most optimize retrieval, execution, writing, or training in isolation. The missing object is a policy that manages the complete experience lifecycle while the repository grows and task distributions change.
Core Problem
At round t, the agent receives task xt, retrieves candidates Ct from repository Mt, selects St, produces trajectory τt and return Rt, then creates an update Δt yielding Mt+1. The desired objective is to maximize E[Rt+λU(Mt+1)], where U measures future repository usefulness. The difficulty is credit assignment: environmental reward supervises actions more directly than selection, writing, consolidation, or deletion, and jointly training these coupled behaviors can cause interference.
Innovation
- ��Four-level memory separates episodic trajectories, local tips, reusable skills, and executable tools. •Outcome-calibrated attribution compares selected and unselected memories only among tasks where they were retrieved, reducing irrelevant-task confounding. •Unified OPD treats selection, action, writing, and maintenance as four views of one policy. •Privileged hindsight supplies future memory value and repository diagnostics unavailable at deployment. •The student is trained on its own visited prefixes, reducing the train–inference mismatch of off-policy supervision.
Methodology
- ��Retrieval: form zt=[xt;et] and use QWEN3-EMBEDDING-0.6B with cosine similarity to retrieve Top-K=50 candidates from each memory tier. •Selection: Selθ compresses candidates into St and formats them as context ct. •Execution: πθ samples actions conditioned on xt, interaction history, and ct, producing τt and Rt. •Writing: the policy emits tier-specific Δℓt, with flexible counts. •Maintenance: every Q=30 tasks, it uses lookup, merge, and delete. •Attribution: Â(m)=Σgρg(m)(E[R|selected]−E[R|unselected]); V(m)=αℓγÂ. •Distillation: for k∈{sel,act,write,maint}, the teacher sees privileged hk, the student samples ŷ on zk, and optimization minimizes full-vocabulary token KL on the same prefixes.
Experiments
Training uses heterogeneous data from Agent World Model, nvidia/Nemotron-Terminal-Corpus, and EnvScaler, disjoint from evaluation. Backbones are QWEN3-4B-INSTRUCT-2507 and QWEN3.5-9B; retrieval uses QWEN3-EMBEDDING-0.6B. Benchmarks include LifelongAgentBench, MemoryArena, AMA-Bench, InterCode, and MiniHack. Baselines cover memory systems—ReasoningBank, MemEvolve, EvolveR, MemP, AWM—and training methods including SFT, GRPO, Skill0, MemRL, and Complementary RL. Memory baselines start from empty repositories.
Results
OPD-Evolver-9B achieves 84.50/65.00 on LifelongAgentBench DB/OS, 10.88/11.63 on MemoryArena Math/Physics, 47.32/53.94/52.92 on AMA-Bench CI/SU/SA, and 49.55/57.00/64.01 on InterCode Bash/CTF/SQL. Against GRPO, it improves MiniHack Maze from 23.53% to 27.45% and KeyRoom from 3.92% to 9.80%; against Complementary RL, Maze rises from 20.85% to 27.45%. Selection distillation raises selected-memory median values from 0.66/0.69/0.66 to 0.79/0.76/0.76 on SQL/CTF/Bash.
Applications
The framework is suited to terminal agents, database assistants, coding and debugging systems, browser automation, and embodied navigation. It is most useful where interaction feedback is available and successful procedures recur at different abstraction levels. Deployment requires a retrievable repository, logging, tool permissions, and safeguards around merge/delete operations. External memory can support immediate adaptation, while slow distillation can reduce reliance on repeatedly injecting long contexts.
Limitations & Outlook
The approach assumes useful task grouping, adequate retrieval recall, and enough outcome data to estimate memory value. Sparse rewards, nonstationary tasks, simultaneous memory effects, and rare but critical events can make attribution unreliable. The teacher uses privileged hindsight and successful trajectories, so the quality and diversity of logged experience matter. The paper does not fully measure million-scale repository costs, privacy leakage, catastrophic long-term deletion, or physical-robot deployment. Future work should add causal attribution, uncertainty-aware retention, safety auditing, and evaluations under sustained distribution shift.
Plain Language Accessible to non-experts
Imagine a restaurant whose cooks improve after every customer. A basic memory system is just a storeroom: it keeps old recipes, warnings, cooking records, and ready-made tools. When a new order arrives, the cook grabs recipes that look similar, but similarity does not guarantee usefulness. The kitchen may fill with duplicate, outdated, or misleading instructions.
OPD-Evolver turns the cook into a complete learning manager. Before cooking, it chooses the most helpful recipes. During cooking, it uses them without blindly copying them. Afterwards, it writes down lessons that can help future customers. Every so often, it combines duplicate recipes and throws away material that consistently causes trouble. The system estimates a recipe’s value by asking a practical question: when this recipe was available, did choosing it lead to better meals than leaving it unused?
A special training teacher can look back at the whole cooking process and knows which lessons later proved valuable. The working cook cannot see that hidden answer, but practices using its own normal decisions until it behaves similarly. In experiments, this approach let a 9B model score 64.01% on SQL, above a 397B model’s 62.74%. It is promising, but a real kitchen still needs human checks, because an unusual recipe may be important even if it has rarely succeeded.
ELI14 Explained like you're 14
Think of an AI playing a game with a backpack full of notes. Some notes are full game replays, some are quick warnings, some are step-by-step strategies, and some are tools it can use directly. A normal AI might stuff everything into the backpack and grab the note that looks most similar to the next level. But what if that note is outdated or actually caused players to lose?
OPD-Evolver teaches the AI four skills: choose useful notes, use them while playing, write better notes after each match, and clean the backpack. It checks whether a note helped in similar levels by comparing scores when the note was chosen and when it was ignored. Every 30 tasks it can look up notes, combine duplicates, or delete bad ones. So the AI is not merely remembering; it is learning how to learn.
During training, a teacher gets a secret future view. The teacher knows which notes eventually helped and which new notes became useful. The student does not get this cheat sheet during testing. Instead, it practices from its own normal game situations and learns to make choices that match the teacher’s better judgment. That is why the method is called on-policy distillation.
The results are exciting: the 9B model scored 64.01% on SQL and beat a 397B model’s 62.74%; it also beat a 196B model on 9 of 10 subsets. Remove memory-value scoring, and performance drops sharply. Still, mistakes can happen: deleting a rare note too early might ruin a future level. More long-term and real-world tests are needed!
Glossary
On-Policy Distillation
A student learns from states and prefixes generated by its own current behavior. Technically, a privileged teacher supplies token distributions, and the student minimizes KL divergence on those on-policy prefixes.
OPD-Evolver uses it to train selection, action, writing, and maintenance together.
Outcome-Calibrated Attribution
A memory receives credit according to the return difference between selecting and not selecting it among comparable tasks where it was retrieved. The estimate is weighted for group reliability and confidence.
It produces V(m), the hindsight value used by the teacher.
Four-Level Memory Hierarchy
The repository contains trajectories, tips, skills, and tools. These range from faithful episodic evidence to compact executable artifacts, balancing detail and reusability.
The fast loop retrieves, writes, and maintains all four tiers.
Experience Lifecycle
The complete path from retrieving and selecting experience, through experience-grounded action and reusable writing, to consolidation and deletion. It treats evolution as a closed process rather than a buffer.
The paper maps the lifecycle to selection, act, write, and maint decisions.
Privileged Hindsight
Training-only information about future outcomes, memory utility, successful trajectories, usage, and redundancy. It is unavailable to the deployed agent but enables dense supervision.
The teacher uses it to demonstrate better lifecycle decisions.
Open Questions Unanswered questions from this research
- 1 Attribution remains observational rather than fully causal. When several memories are selected together or rewards are sparse, the method may assign credit to the wrong item; counterfactual interventions are needed.
- 2 The experiments do not establish stability in million-item repositories, persistent distribution shift, or years of interaction. Scaling retrieval, maintenance, and logging may become a major bottleneck.
- 3 Privacy, safe deletion, and multi-agent memory sharing are largely untested. Practical deployment needs auditable retention policies and defenses against poisoned experience.
Applications
Immediate Applications
Terminal and coding agents
Engineering teams can store successful Bash, CTF, and SQL procedures as skills or executable tools, then reuse them on related tasks. Logged outcomes and restricted tool permissions are prerequisites; expected benefits include less repeated trial and error and more consistent debugging.
Enterprise workflow assistants
Customer-support, analytics, and operations assistants can organize cases, warnings, procedures, and templates by reuse level. Organizations need access control, human review, provenance, and deletion policies so an incorrect answer is not repeatedly reinforced.
Long-term Vision
Continually learning embodied robots
A robot could preserve full episodes, local safety warnings, reusable manipulation skills, and tool programs while adapting to homes or factories. Major obstacles are safe exploration, rare-event credit assignment, real-time computation, privacy, and transferring experience across bodies and environments.
Abstract
Memory has become a standard substrate for self-evolving agents, yet retaining experience is not the same as learning how to evolve through it. Existing memory agents can store trajectories, retrieve reflections, or accumulate skills, but often lack the holistic competence to select useful experience, act on it, write reusable knowledge, and maintain a growing repository. We introduce OPD-Evolver, a slow-fast co-evolution framework that cultivates such an agent evolver through on-policy self-distillation. In the fast loop, OPD-Evolver interacts with a four-level memory hierarchy to read, use, write, and maintain experience for rapid test-time evolution. In the slow loop, outcome-calibrated memory attribution and privileged hindsight distill these four abilities into the deployable policy. Across multi-domain benchmarks, OPD-Evolver surpasses memory systems such as ReasoningBank by up to 11.5%, and training-based methods such as Skill0 by ~5.8%. Further analysis shows that OPD-Evolver internalizes high-value experience and memory management, enabling OPD-Evolver-9B to challenge giant counterparts such as Qwen3.5-397B-A17B and Step-3.5-Flash, pointing beyond memory-augmented agents toward genuinely qualified agent evolvers.