MemoryVAM: Integrating Memory into Video Action Model for Robot Manipulation
MemoryVAM integrates episodic memory into video world models using Recap-Cue, boosting long-horizon robot manipulation success from 5% to 42.5%.
Key Findings
Methodology
MemoryVAM introduces a Perceiver-based Recap Compressor that encodes episode history into compact memory tokens. These tokens are injected into both the visual backbone and the action decoder via dual pathways, enabling the model to condition future predictions and control actions on episodic memory. The Recap-Cue module, trained with video prediction, auxiliary reconstruction loss, and episode boundary supervision, produces memory representations without requiring per-frame labels. The approach is compatible with UNet and Diffusion Transformer architectures by adjusting cross-attention interfaces. Experiments show significant performance gains on LIBERO-Mem, with success rates rising from 5% to 42.5%, and strong results on real robot tasks, including 78.3% success on counting, 80.0% on spatial recall, and 75.0% on sequential tracking.
Key Results
- On LIBERO-Mem, success rate increased from 5% to 42.5%, demonstrating improved long-horizon manipulation capabilities.
- In real robot experiments, the model achieved 78.3% success in counting, 80.0% in spatial recall, and 75.0% in sequential tracking, outperforming memoryless baselines.
- Ablation studies confirmed that dual injection of memory into both future prediction and action decoding is essential for optimal performance.
Significance
This work advances the field by embedding episodic memory directly into video world models, addressing the core challenge of long-term dependency modeling in robotic manipulation. The self-supervised training approach reduces reliance on action labels, making large-scale unsupervised learning feasible. The method enhances robot autonomy in complex, multi-stage tasks, and bridges the gap between visual prediction and control, setting a new standard for long-horizon robotic learning.
Technical Contribution
The paper proposes a dual-pathway memory injection mechanism, utilizing a Recap-Cue module trained with video prediction and auxiliary losses. This design allows episode-level memory to influence both the visual backbone and the control policy, unifying future prediction and action control within a single framework. Compatibility with multiple backbone architectures demonstrates its flexibility. The approach also introduces a task completion estimator (Cue Gate) that leverages episodic memory for autonomous termination, further improving task efficiency.
Novelty
This is the first comprehensive integration of episodic memory into video world models for robot manipulation, where memory is used to condition both future scene prediction and control decisions via a unified, trainable module. Unlike prior work that treats memory as an auxiliary input, MemoryVAM embeds memory as a core component of the predictive model, enabling long-term consistency and improved task success.
Limitations
- Memory capacity is limited by the Recap Compressor, which may truncate or forget information in very long episodes, requiring future online or recurrent extensions.
- Training and inference are computationally intensive due to video prediction and attention mechanisms, posing challenges for real-time deployment.
- The robustness of episodic memory in highly dynamic or cluttered environments remains to be fully validated, especially under severe occlusion or noise.
Future Work
Future directions include developing online recurrent memory modules to support longer episodes, integrating multi-modal data (e.g., language, proprioception) for richer memory representations, and optimizing inference efficiency. Exploring reinforcement learning with episodic memory for adaptive policy improvement and extending the framework to more diverse real-world scenarios are also promising avenues.
AI Executive Summary
Long-horizon robotic manipulation remains a fundamental challenge due to the limited observation window of existing video world models. Traditional approaches rely heavily on current visual input, which hampers performance when tasks involve multiple steps, hidden states, or repeated actions. These limitations become evident in tasks like counting, sequence tracking, or object retrieval, where history and memory are crucial.
MemoryVAM addresses this gap by embedding episodic memory directly into the video prediction framework. The core innovation is the Recap-Cue module, which compresses episode history into a set of memory tokens using a Perceiver architecture. These tokens are injected into both the visual backbone and the action decoder through dual pathways, enabling the model to imagine future states and select actions conditioned on the entire episode history.
Training leverages video prediction, auxiliary reconstruction, and episode boundary supervision, eliminating the need for per-frame labels. Experiments on the LIBERO-Mem benchmark show success rates jump from 5% to 42.5%, demonstrating the method’s effectiveness in complex manipulation tasks. In real robot tests, the model achieves 78.3% success on counting, 80.0% on spatial recall, and 75.0% on sequential tracking, significantly outperforming memoryless baselines.
This work fundamentally shifts how episodic memory is integrated into visual predictive models, making it a core component rather than an auxiliary module. It opens new avenues for scalable, unsupervised long-term robot learning, with potential applications across autonomous systems, assistive robotics, and beyond. Despite promising results, challenges remain in extending memory capacity, reducing computational costs, and ensuring robustness in dynamic environments. Future research will focus on online recurrent memory, multi-modal integration, and real-time deployment, aiming to realize fully autonomous, long-horizon robotic agents.
Deep Dive
Abstract
Video-world-model policies learn action-relevant representations by predicting future observations. However, they condition on only a short observation window, which renders long-horizon manipulation non-Markovian when the correct action depends on earlier events that are no longer visible. We present MemoryVAM, an episodic memory mechanism for video-world-model policies. We employ a Recap-Cue (RC) module, in which a Perceiver-based Recap Compressor maps per-frame CLIP embeddings into compact memory tokens, and a lightweight Cue Gate estimates task completion from memory and language. These tokens are injected into both the video backbone and the action decoder, aligning policy imagination with episode progress and conditioning actions on history. Our model trains the memory module with video prediction, a delta-reconstruction auxiliary loss, and episode-boundary supervision, requiring no per-frame progress labels. The same mechanism applies to UNet and Diffusion Transformer (DiT) backbones by changing only the cross-attention injection interface. On LIBERO-Mem, our model improves average success from 5% to 42.5%. On real robots, it achieves 78.3% success on counting tasks, 80.0% on spatial recall, and 75.0% on sequential tracking. Project page: https://MemoryVAM.github.io/