Procedural Memory Distillation: Online Reflection for Self-Improving Language Models
PMD method enhances language model performance by converting cross-episode signals into procedural memory, achieving 3.8-13.6% improvement.
Key Findings
Methodology
This study introduces Procedural Memory Distillation (PMD), which converts cross-episode signals into reusable procedural memory and distills it into policy weights during training. PMD organizes memory at three levels of abstraction: raw trajectories, self-reflected strategies and lessons, and higher-level behavioral patterns.
Key Results
- PMD improves over SDPO by 3.8-5.5% on SCIKNOWEVAL and 7.9-13.6% on LIVECODEBENCH.
- Freezing either the memory or the policy results in performance drops of over 10%.
- PMD shows superior performance on both Qwen3-8B and OLMo3-Instruct-7B models.
Significance
The PMD method significantly enhances language model performance by converting cross-episode signals into procedural memory. This approach not only increases verification pass rates but also reduces the occurrence of failure modes, offering a new direction for self-improving language models.
Technical Contribution
PMD distinguishes itself from traditional static or offline memory banks by jointly updating policy and memory. It builds memory online, ensuring the teacher signal remains consistent with the current policy, thereby enhancing learning efficiency.
Novelty
PMD is the first to convert cross-episode signals into procedural memory and internalize it through self-distillation, allowing the model to operate without external memory during inference.
Limitations
- PMD requires substantial computational resources for real-time memory updates.
- Memory abstraction may be too coarse for certain complex problems.
- Important information may be missed during memory construction.
Future Work
Future research could explore optimizing the memory abstraction process to improve performance on complex problems and investigate ways to reduce computational overhead for memory updates.
AI Executive Summary
Procedural Memory Distillation (PMD) is an innovative method that enhances language model performance by converting cross-episode signals into procedural memory. Traditional reinforcement learning methods often focus on single-episode feedback signals, neglecting the rich information across episodes. PMD constructs three levels of abstract memory online, distilling this information into policy, allowing the model to operate without external memory during inference. Experimental results show PMD's superior performance across multiple datasets, significantly outperforming existing self-distillation methods. Although PMD requires high computational resources, its potential to improve model performance is substantial. Future research can further optimize the memory construction process to reduce computational overhead.
Deep Analysis
Background
In recent years, the combination of reinforcement learning and language models has become a research hotspot. Traditional methods like PPO and DPO mainly rely on single-episode feedback signals for policy updates, overlooking the potential of cross-episode information. With the development of self-distillation techniques, researchers are exploring how to leverage the model's own trajectory information to enhance performance.
Core Problem
Existing reinforcement learning methods have limitations in handling cross-episode information, failing to capture which strategies consistently pass verification, which failure modes persist, and which patterns recur. This results in poor performance when dealing with complex problems.
Innovation
PMD constructs procedural memory online, converting cross-episode signals into reusable strategies and behavioral patterns. This innovation allows the model to operate without external memory during inference, improving learning efficiency and verification pass rates.
Methodology
- �� PMD organizes memory into three levels: raw trajectories, self-reflected strategies and lessons, and higher-level behavioral patterns.
- �� Memory is distilled into policy weights during training, allowing the model to operate without external memory during inference.
- �� Jointly updates policy and memory, ensuring the teacher signal remains consistent with the current policy.
Experiments
Experiments were conducted on SCIKNOWEVAL and LIVECODEBENCH datasets using Qwen3-8B and OLMo3-Instruct-7B models. PMD's effectiveness was verified by comparing its performance with SDPO.
Results
PMD improves over SDPO by 3.8-5.5% on SCIKNOWEVAL and 7.9-13.6% on LIVECODEBENCH. Freezing either the memory or the policy results in performance drops of over 10%.
Applications
PMD can be used to enhance language model performance in scientific reasoning and code generation tasks, suitable for scenarios requiring high verification pass rates.
Limitations & Outlook
PMD requires substantial computational resources for real-time memory updates. Memory abstraction may be too coarse for certain complex problems. Future research could explore optimizing the memory abstraction process.
Plain Language Accessible to non-experts
Imagine a chef in a kitchen preparing a dish. Each time they cook, they note which steps were successful and which failed, summarizing some reusable techniques. Next time they cook the same dish, they don't need to consult the recipe but can rely on their experience. PMD is like this chef, recording and summarizing each attempt by the model, converting this information into reusable procedural memory to enhance model performance.
ELI14 Explained like you're 14
Imagine playing a game where each time you pass a level, the game tells you which strategies worked and which didn't. You jot down this information, so next time you play, you can pass the level faster. PMD is like this game assistant, helping language models remember which strategies worked and which didn't, so they can find answers faster when faced with similar problems.
Glossary
Reinforcement Learning
A machine learning method that trains models through rewards and penalties.
Used in the paper to train language models.
Self-Distillation
A technique that optimizes models through feedback from the model itself.
Used to enhance language model performance.
Procedural Memory
A form of memory that records model attempts and strategies.
Core innovation of PMD.
Policy Optimization
Improving performance by adjusting model policies.
Used to train language models.
Cross-Episode Signals
Information from different episodes used to optimize models.
Extracted by PMD.
Open Questions Unanswered questions from this research
- 1 How to optimize the memory abstraction process to improve performance on complex problems?
- 2 How to reduce computational overhead for memory updates?
- 3 What new application scenarios can future research explore?
Applications
Immediate Applications
Scientific Reasoning
PMD can enhance language model performance in scientific reasoning tasks, suitable for scenarios requiring high verification pass rates.
Long-term Vision
Code Generation
PMD can enhance language model performance in code generation tasks, helping developers quickly generate high-quality code.
Abstract
Reinforcement learning with verifiable rewards (RLVR), along with recent selfdistillation variants such as SDPO, evaluates each rollout against a verifier and updates the policy from that episode-level signal. However, the richer procedural information in the rollout is rarely retained or reused. Across episodes and epochs, the model repeatedly encounters related problems under a changing policy, producing cross-episode signals that episode-local updates cannot capture: which strategies consistently pass verification, which failure modes persist, which patterns recur. We propose Procedural Memory Distillation (PMD), which converts these crossepisode signals into reusable procedural memory and distills it into the policy's weights during training. This memory functions as a training scaffold, absorbed into the policy itself, yielding a memory-free model at inference. PMD organizes the memory at three levels of abstraction: raw trajectories, self-reflected strategies and lessons, and higher-level behavioral patterns that recur across problems, all extracted online from the model's own trajectories. A memory-conditioned self-teacher draws on the accumulated experience to supervise the student on its own rollouts, enabling student to progressively internalize procedural knowledge within its parameters. The central design principle is co-evolution: the policy generates rollouts that update the memory, and memory shapes the supervision that updates the policy. Empirically, across Qwen3-8B and OLMo3-Instruct-7B, PMD improves over SDPO by 3.8-5.5% on SCIKNOWEVAL and 7.9-13.6% on LIVECODEBENCH. Co-evolution powers these gains: freezing either the memory or the policy trails PMD by more than 10% across SCIKNOWEVAL domains.