MEMORA: Embodied Action Memory from Egocentric Videos for Reasoning and Planning
MEMORA employs a formation-consolidation-retrieval lifecycle with typed stores, improving long-term robot planning by up to 20.5 points in memory assessment and 16.6% in planning scores.
Key Findings
Methodology
This paper introduces MEMORA, a framework integrating online editing, four typed memory stores (environment, entity, activity, inferred knowledge), and offline consolidation to build persistent embodied action memory. Using a Segment Encoder, continuous egocentric videos are converted into structured observations. The active Memory Editor dynamically maintains object identities and states through add, update, delete operations, reducing noise and preserving continuity. Offline consolidation abstracts repeated experiences into reusable routines and regularities. Retrieval employs type-aware indices to support experience-grounded reasoning and planning. The system is trained and evaluated on 45 hours of EPIC-KITCHENS-100 extension videos, demonstrating significant improvements over baselines in memory accuracy and robot-grounded planning tasks.
Key Results
- Full MEMORA, combining editing, typed stores, and consolidation, achieved up to 20.5-point improvement in memory assessment accuracy and up to 16.6% higher Robot-Grounded Plan scores across four language models.
- In transfer and composition tasks (Generalize), MEMORA outperformed all baselines, especially when planning for unseen goals, with consistent gains across different model backbones.
- A robot deployment study confirmed that language plans grounded in memory effectively interface with downstream control, demonstrating practical applicability.
Significance
This work advances the field of embodied AI by providing a structured, lifecycle-based memory system capable of handling complex, long-horizon tasks. It addresses the critical challenge of maintaining continuous, structured experience from egocentric videos, enabling robots to reason and plan with context-aware, reusable knowledge. The approach bridges perception, memory, and action, paving the way for more autonomous, adaptable systems capable of long-term operation in dynamic environments. Its modular design and empirical validation set a new standard for integrating structured memory into robotic planning and cognition.
Technical Contribution
The paper formalizes Embodied Action Memory (EAM) as a lifecycle comprising formation, consolidation, and retrieval, instantiated via four specialized memory stores. It introduces the Segment Encoder for converting videos into structured observations, active Memory Editing for dynamic maintenance, and offline consolidation for experience abstraction. The architecture supports multi-scale continuity, semantic-procedural reasoning, and efficient retrieval. These innovations outperform existing static or graph-based memory models, offering a comprehensive, scalable solution for long-term embodied cognition.
Novelty
This research is the first to systematically embed egocentric video experience into a lifecycle-driven, typed memory system tailored for embodied agents. Unlike prior static or single-layer memory models, MEMORA’s active editing and offline consolidation enable continuous, structured, and reusable experience representation, significantly enhancing long-term planning and generalization capabilities in robotics.
Limitations
- The system's performance depends on video quality; noisy or incomplete data can impair memory fidelity and planning accuracy.
- Offline consolidation is computationally intensive, limiting real-time deployment in resource-constrained settings.
- Scalability to highly complex or extended multi-task scenarios remains to be fully validated.
Future Work
Future directions include integrating multimodal sensory data (e.g., audio, tactile), optimizing consolidation efficiency, and enabling autonomous memory updates via reinforcement learning. Extending the framework to real-time applications and more diverse environments will further enhance its robustness and practical utility.
AI Executive Summary
MEMORA introduces a lifecycle-based embodied action memory system designed to enhance long-term robot planning by leveraging egocentric videos. The framework combines online active editing, four specialized memory stores—environment, entity, activity, and inferred knowledge—and offline consolidation to create a persistent, structured memory. This approach addresses the longstanding challenge of maintaining continuous, meaningful experience over extended periods, which is critical for complex, long-horizon tasks.
The core innovation lies in the active Memory Editor, which dynamically updates object identities and states, reducing noise and ensuring continuity. The Segment Encoder converts continuous egocentric video into structured observations, facilitating semantic understanding. Offline consolidation abstracts repeated experiences into reusable routines and regularities, enabling better generalization. Retrieval mechanisms employ type-aware indices to support experience-grounded reasoning, allowing the system to adapt to new goals and environments.
Experimental results on 45 hours of EPIC-KITCHENS-100 extension videos demonstrate that full MEMORA outperforms baseline models, achieving up to 20.5 points in memory assessment accuracy and a 16.6% increase in robot-grounded planning scores across multiple language models. The system's ability to generalize to unseen goals and transfer routines highlights its robustness and potential for real-world deployment.
A robot deployment study further confirms that language plans grounded in structured memory can interface effectively with downstream control, enabling more autonomous and adaptable robotic behavior. This work significantly advances embodied AI by providing a scalable, lifecycle-driven memory architecture that bridges perception, reasoning, and action, opening new avenues for long-term autonomous systems.
Despite these advances, challenges remain in optimizing real-time performance, scaling to more complex scenarios, and integrating additional sensory modalities. Future research will focus on addressing these limitations, refining the efficiency of offline consolidation, and extending the system’s capabilities for broader applications in robotics and AI.
Deep Analysis
Background
机器人在长远任务中面临记忆不足和信息遗忘的挑战。早期工作如LSTM、Transformer在短期记忆中表现优异，但难以应对跨会话、多场景的持续记忆需求。近年来，空间和语义记忆系统（如SLAM、场景图）在导航和环境理解中取得进展，但缺乏对连续体验的系统性建模。Egocentric视频作为丰富的第一人称体验源，为机器人提供了潜在的连续感知基础。已有的模仿学习和跨模态学习方法（如VQA、行为模仿）虽能利用视频信息，但多停留在短期或单次任务中，缺乏持久、结构化的记忆体系。本文在此基础上，提出生命周期驱动的记忆模型，旨在实现连续、多场景、多任务的长期记忆存储与高效利用。
Core Problem
核心问题在于如何将连续的egocentric视频体验转化为结构化、持久的记忆体系，支持复杂的长远规划。传统方法多采用静态存储或简单的图结构，难以维护对象身份、状态变化、空间关系和行为规律的连续性。现有模型在多任务、多会话环境中表现出明显的遗忘和泛化不足，限制了机器人自主认知的深度和广度。解决这一问题需要设计动态维护、多尺度存储和高效检索机制，确保记忆的连续性、可扩展性和实用性。
Innovation
创新点包括：• 引入多类型存储（环境、实体、活动、推断知识）以匹配 embodied experience的多尺度连续性；• 采用Segment Encoder将视频片段转化为结构化观察，增强语义表达；• 实现主动编辑机制，动态维护对象身份和状态历史，减少噪声和误差积累；• 设计离线巩固策略，将重复经验抽象为程序和规律，存入推断知识存储。这些创新共同构建了一个系统性、生命周期驱动的记忆体系，超越了传统静态存储和单一图结构的局限。
Methodology
- �� 利用多模态Segment Encoder将连续视频片段转化为结构化观察，输入视频流；
- �� 在线维护Memory Store，采用ACTIVE MEMORY-EDITING机制，动态添加、更新、删除对象和状态信息，确保信息的时序连续性；
- �� 采用多存储结构（环境、实体、活动、推断知识）分别维护空间、对象、行为和规律的连续性；
- �� 离线进行巩固，将重复出现的经验抽象为程序和规律，存入推断知识存储；
- �� 查询机制通过类型感知索引支持高效检索，支持基于记忆的推理和规划；
- �� 在实验中，利用多模态视频和语言模型（如Qwen系列）实现端到端训练和评估。
Experiments
采用45小时EPIC-KITCHENS-100扩展视频，涵盖18名参与者，进行记忆评估和机器人规划任务。对比模型包括静态存储、无编辑、离线巩固等多种变体，评估指标包括记忆准确率和规划得分。通过ablation研究验证各机制贡献，测试在新目标和跨会话场景中的泛化能力。实验还包括机器人实际部署，验证语言计划的物理执行效果。
Plain Language Accessible to non-experts
想象你在厨房里帮忙做饭，每次你都记得工具放在哪里，步骤怎么走，还会总结出一些自己喜欢的做法。MEMORA就像是你脑袋里的一个超级笔记本，不仅记住了你每次的经验，还能整理出常用的套路。它能帮你记住过去的细节，比如哪个碗用过，哪个步骤重复多了。这样，下次你做饭时，就不用每次都想怎么开始，因为它帮你整理好了所有信息。它就像一个聪明的厨房助手，能让你做饭变得更快、更顺手。
ELI14 Explained like you're 14
你每天在厨房帮忙做饭，会记得哪些东西放在哪里，怎么切菜、炒菜。MEMORA就像你脑袋里的超级记忆宝箱，能把你每天的经验都存起来。每次你做饭，它会帮你记住用过的工具、步骤和小技巧。时间长了，它还能总结出一些规律，比如每次煮汤都要先准备好材料，或者用哪个碗盛米。这样，你以后做饭就不用每次都想怎么开始了，因为它帮你记得一清二楚。它就像你的厨房小助手，让你变得更厉害、更有条理！
Glossary
Embodied Action Memory (EAM)
一种将连续体验转化为结构化、可检索记忆的生命周期模型，支持机器人长远规划。
论文提出的核心记忆概念，用于描述体验到的连续信息的存储与利用。
Segment Encoder
多模态模型，将视频片段转化为结构化观察，提取语义信息。
实现视频到记忆结构的转化关键组件。
ACTIVE MEMORY-EDITING
在线主动编辑机制，动态维护对象身份和状态历史。
确保记忆的连续性和准确性。
离线巩固
将重复经验抽象为可复用程序和规律的过程。
提升记忆的泛化能力和效率。
MEMORA-Bench
评估系统记忆能力的测试平台，包括记忆评估和规划任务。
验证MEMORA在实际场景中的表现。
Open Questions Unanswered questions from this research
- 1 如何进一步提升离线巩固的效率，尤其在实时场景中实现高效抽象和更新。
- 2 系统在极端复杂环境下的表现和扩展性尚未充分验证。
Applications
Immediate Applications
智能家居机器人
利用MEMORA实现家居环境中的连续记忆，提升自主导航和任务执行能力。
工业自动化机器人
在复杂生产线上，存储和利用操作经验，提高效率和适应性。
Long-term Vision
自主认知系统
打造具有长远记忆和推理能力的机器人，能自主学习和适应新环境。
Abstract
Long-horizon robot planning requires more than predicting what actions will do next; it also requires memory of the embodied experience that makes future goals interpretable. People do not plan from the present scene alone: they draw on remembered places, object-state changes, prior procedures, and regularities revealed through repeated action. We formulate Embodied Action Memory (EAM) as the capability to form, maintain, and use such experience as a persistent memory state for later decisions. MEMORA realizes EAM with a formation-consolidation-retrieval lifecycle and four typed stores: Environment Memory, Entity Memory, Activity Memory, and Inferred Knowledge. Online editing maintains object identities and state histories as new observations arrive; offline consolidation abstracts repeated experience into reusable procedures and participant-specific regularities. MEMORA-Bench evaluates this lifecycle on 45 hours of EPIC-KITCHENS-100 extension video across 18 participants through memory-grounded planning, including previously unseen goals, and a complementary memory-assessment task. Across four open-weight language models, full MEMORA--combining editing, typed stores, and consolidation--achieves the strongest aggregate results among the evaluated memory conditions. It improves memory-assessment accuracy by up to 20.5 points over the strongest controlled baseline and improves out-of-distribution Robot-Grounded Plan score by up to 16.6% relative. A qualitative two-task robot deployment study further illustrates how memory-grounded language plans can interface with downstream control, while the overall results show that editable, consolidated memory can supply remembered context for robot planning. Project page: https://yuzihaowashu.github.io/MEMORA/