BrainMem: Brain-Inspired Evolving Memory for Embodied Agent Task Planning
BrainMem, a hierarchical, evolving memory system, improves long-horizon embodied task success by over 20% in benchmarks.
Key Findings
Methodology
This paper introduces BrainMem, a training-free hierarchical memory architecture inspired by human cognition, integrating working, episodic, and semantic memories. It continuously transforms interaction logs into structured knowledge graphs and symbolic summaries, enabling retrieval, reasoning, and behavior adaptation without fine-tuning. Key modules include short-term working memory, long-term episodic graphs (trajectory and spatial graphs), and semantic memory for abstracted knowledge. These components evolve through ongoing interaction, with episodic experiences summarized into semantic guidelines post-task, facilitating progressive improvement. The system seamlessly integrates with multimodal large language models, reducing prompt engineering. Extensive evaluations across EB-ALFRED, EB-Habitat, EB-Navigation, and EB-Manipulation demonstrate significant performance gains, especially on complex, long-horizon tasks.
Key Results
- In EB-ALFRED long-horizon tasks, success rates increased from around 50% to over 75%, a 25% improvement. In EB-Habitat, success rates improved by 4-8%. On EB-Navigation and EB-Manipulation, success rates on long tasks increased by up to 36.7%. Ablation studies confirmed the importance of hierarchical memory and continuous evolution mechanisms.
- Memory modules store trajectories and spatial relations via knowledge graphs, with automatic summarization extracting reusable patterns. Retrieval from these modules guides planning, leading to better task completion across scenarios.
- Compared to prior memory methods, BrainMem outperforms in long-term reasoning and spatial understanding, with success rate improvements validating its effectiveness for general embodied intelligence.
Significance
This work addresses core limitations of current large models in long-term, multi-step embodied tasks by proposing a scalable, hierarchical, and evolving memory system. It mimics human cognition’s multi-layered memory architecture, enabling agents to maintain context, learn from experience, and adapt strategies over time. The approach reduces reliance on extensive training data and task-specific prompts, making embodied AI more robust and generalizable. Its ability to improve task success across diverse benchmarks demonstrates its potential for real-world applications in robotics, virtual assistants, and autonomous systems, pushing the frontier of embodied intelligence.
Technical Contribution
The paper introduces a novel, training-free hierarchical memory system that integrates working, episodic, and semantic memories with continuous interaction-driven evolution. It employs knowledge graphs for structured long-term storage, automated summarization via LLMs for experience abstraction, and multi-level retrieval mechanisms. This design enables experience accumulation, transfer, and reasoning across tasks and environments, surpassing existing static or single-layer memory solutions. The architecture provides a new paradigm for scalable, lifelong learning in embodied agents, with broad applicability to multimodal large models.
Novelty
This is the first work to implement a brain-inspired, multi-layered, evolving memory architecture tailored for embodied AI. Unlike prior approaches that store only successful trajectories or static knowledge, BrainMem dynamically updates and consolidates experiences across multiple timescales, using knowledge graphs and symbolic summaries. Its integration of continuous interaction, automatic experience abstraction, and hierarchical retrieval sets a new standard for scalable, generalizable embodied intelligence.
Limitations
- In highly complex or dynamic environments, knowledge graphs may grow large, impacting retrieval efficiency and scalability. Effective pruning and compression strategies are needed.
- Automatic summarization may sometimes produce biased or overly abstracted knowledge, affecting reasoning accuracy.
- Current design does not explicitly support multi-agent memory sharing or collaboration, which is essential for future multi-robot systems.
Future Work
Future research will focus on multi-agent memory sharing, integrating reinforcement learning to optimize memory evolution, and enhancing spatial perception for more accurate environment modeling. Extending the framework to real-world robotics and multi-agent scenarios will be key to broader deployment.
AI Executive Summary
In complex 3D environments, autonomous agents face significant challenges in maintaining long-term context and learning from experience. Traditional large language models, while powerful in reasoning, are inherently stateless, leading to repeated errors and poor spatial-temporal understanding in embodied tasks. Addressing this, the authors propose BrainMem, a hierarchical, evolving memory system inspired by human cognition. It combines short-term working memory, long-term episodic graphs, and semantic knowledge summaries, enabling agents to continuously learn from interaction histories without additional training.
The core innovation lies in transforming interaction logs into structured knowledge graphs—such as trajectory and spatial graphs—and automatically summarizing experiences into reusable guidelines via large language models. These components are dynamically updated after each action and episode, supporting effective long-horizon planning and adaptation. The system’s plug-and-play design allows seamless integration with diverse multimodal large models, significantly reducing prompt engineering efforts.
Experimental results across four benchmarks—EB-ALFRED, EB-Habitat, EB-Navigation, and EB-Manipulation—demonstrate substantial performance improvements. Success rates on complex, multi-step tasks increased by over 20%, with the most notable gains in spatial reasoning and long-horizon planning. These results highlight the potential of evolving, hierarchical memory architectures to advance embodied AI towards more general, scalable intelligence.
Looking ahead, the authors plan to extend the framework to multi-agent systems, incorporate reinforcement learning for memory optimization, and improve spatial perception. This work paves the way for autonomous agents capable of lifelong learning, robust reasoning, and adaptive behavior in real-world environments, marking a significant step toward human-like embodied intelligence.
Deep Analysis
Background
随着 embodied AI 的快速发展，长时序、多任务场景中的自主规划成为核心难题。早期方法如符号规划（如PDDL）和强化学习在特定任务中取得一定成果，但存在泛化差、样本需求高等问题。近年来，基于大模型的规划方案兴起，利用自然语言理解和推理能力提升任务表现，但缺乏有效的长期记忆机制，导致行为反复、空间理解不足。已有的记忆方案如 Neural Turing Machines、Neural Map 和结构化场景图谱在一定程度上缓解了此问题，但仍难以实现跨任务的经验积累与迁移。认知神经科学的多层次记忆模型启发了新型的AI记忆架构，成为研究热点。
Core Problem
当前大模型在复杂环境中表现受限，主要原因是缺乏持续记忆能力，难以维护长时间的任务上下文和空间关系。缺少系统的经验积累机制，使得代理在多步骤、多场景任务中反复犯错，空间理解和行为协调不足。如何设计一种高效、可扩展的多层次记忆系统，支持动态信息存储、演化和知识总结，成为关键难题。这不仅关系到自主系统的智能水平，也影响其在实际应用中的可靠性和适应性。
Innovation
本文提出分层演化记忆架构BrainMem，模仿人类认知中的多层次记忆体系。创新点包括：
- �� 实时构建知识图谱，存储轨迹和空间关系，支持长时序推理；
- �� 利用大模型自动总结成功经验和失败原因，形成符号化的行为指南；
- �� 无需训练，交互中持续演化，增强经验积累能力；
- �� 支持多任务迁移，知识图谱和符号指南实现跨场景泛化；
- �� 多层次信息融合，提升空间理解和行为规划能力。
Methodology
- �� 交互过程中，短期工作记忆（Mwork）存储近期动作、反馈、状态、位置和时间戳信息。
- �� 长期情节记忆（Mepi）通过轨迹图和空间图编码任务轨迹和空间关系，形成结构化知识。
- �� 任务结束后，利用大模型自动总结轨迹和空间关系，生成符号化的经验指南（Msem），存储成功与失败的关键经验。
- �� 在规划时，从三层记忆中检索相关信息，结合当前任务上下文，生成行为计划。
- �� 记忆持续演化：每个步骤后更新工作记忆和轨迹图，任务结束后更新空间图和语义指南，删除低效信息。
Experiments
在EB-ALFRED、EB-Habitat、EB-Navigation和EB-Manipulation四个基准上，采用成功率作为主要指标。对比无记忆和不同记忆方案，验证BrainMem的有效性。通过不同难度子集和消融实验，分析分层记忆和自动总结机制的贡献。结果显示，BrainMem在长时序和空间复杂任务中成功率提升20%以上，验证其优越性。
Results
在EB-ALFRED长任务中，成功率由50%提升至75%以上，提升约25%。在EB-Habitat中，成功率提升4-8%。在EB-Navigation和EB-Manipulation中，长任务成功率提升达36.7%。消融分析验证了分层记忆和符号总结的关键作用。整体表现优于现有方法，证明其在复杂环境中的适用性。
Applications
该技术可广泛应用于家庭助理、工业机器人、虚拟助手等场景，提升连续任务执行和空间理解能力。未来结合强化学习和多智能体系统，有望实现更复杂的协作和环境适应，为机器人和虚拟代理提供更强认知基础。
Limitations & Outlook
在极端复杂或动态环境中，知识图谱可能膨胀，影响检索效率。自动总结的语义指南可能偏差或抽象不足，影响推理准确性。当前未考虑多智能体协作，未来需扩展多主体记忆交互机制。
Plain Language Accessible to non-experts
想象你在厨房做饭。每次你会记住刚刚做的菜、用过的工具和遇到的问题，比如盐放多了或锅太热。你会把这些经验存起来，下次做菜时会用得更顺手。这个过程就像人脑的记忆系统：短期记忆帮你记住刚刚的操作，情节记忆存储完整的做菜过程，语义记忆总结出哪些调料用得好。这些记忆不断更新，帮助你做得更好。类似的，机器人也可以用这种方式，记住以前的经验，变得越来越聪明，做菜也越来越好吃。
ELI14 Explained like you're 14
想象你在玩一个超级复杂的游戏，每次都要记住很多信息，比如你在哪个房间、拿过什么东西、遇到的难题。可是有时候你会忘记之前的事情，导致重复做一些没用的动作，比如一直按开关却忘记关了灯。这个研究就像给机器人装了一个聪明的记忆系统，它可以像你大脑一样，把玩游戏的每个细节都记下来。它会把这些记忆整理成图表和总结，然后在下一次玩的时候，直接用这些经验来做决定。这样，机器人就能变得越来越聪明，不会再犯以前的错误，也能更快完成任务。就像你玩游戏时学会了怎么过关，变得越来越厉害一样。
Abstract
Embodied task planning requires agents to execute long-horizon, goal-directed actions in complex 3D environments, where success depends on both immediate perception and accumulated experience across tasks. However, most existing LLM-based planners are stateless and reactive, operating without persistent memory and therefore repeating errors and struggling with spatial or temporal dependencies. We propose BrainMem(Brain-Inspired Evolving Memory), a training-free hierarchical memory system that equips embodied agents with working, episodic, and semantic memory inspired by human cognition. BrainMem continuously transforms interaction histories into structured knowledge graphs and distilled symbolic guidelines, enabling planners to retrieve, reason over, and adapt behaviors from past experience without any model fine-tuning or additional training. This plug-and-play design integrates seamlessly with arbitrary multi-modal LLMs and greatly reduces reliance on task-specific prompt engineering. Extensive experiments on four representative benchmarks, including EB-ALFRED, EB-Navigation, EB-Manipulation, and EB-Habitat, demonstrate that BrainMem significantly enhances task success rates across diverse models and difficulty subsets, with the largest gains observed on long-horizon and spatially complex tasks. These results highlight evolving memory as a promising and scalable mechanism for generalizable embodied intelligence.