RMBench: Memory-Dependent Robotic Manipulation Benchmark with Insights into Policy Design
Introduces RMBench benchmark and Mem-0 policy, systematically evaluating memory capabilities in robotic manipulation with success rates up to 42%.
Key Findings
Methodology
This work defines Task Memory Complexity (TMC) to categorize manipulation tasks based on memory demands. Using the RoboTwin 2.0 platform, nine tasks spanning various memory levels were created. The Mem-0 policy employs a modular architecture integrating a vision-language model (VLM) with explicit memory modules—anchor and sliding window—enabling long-term memory modeling. Ablation studies dissect the importance of each component, demonstrating significant performance gains over baseline models. The approach combines structured subtask reasoning with diffusion-based action generation, supporting both simulation and real-world validation.
Key Results
- Mem-0 achieves an average success rate of 42%, outperforming baselines by over 31%. In tasks like 'Rearrange Blocks,' success rates increased from 29% to 89%. 'Cover Blocks' success rose from 0% to 68%. Ablation results show removing anchor memory or task classifiers drops success by ~20%. In real robot tests, Mem-0 generalizes well, confirming the effectiveness of explicit memory modeling.
- In memory-dependent tasks, Mem-0's success rates significantly surpass traditional models, validating the importance of explicit long-term memory. The detailed analysis reveals that components like the task-phase classifier and memory buffers are critical for performance, providing insights for future architecture improvements.
- The results demonstrate that explicit memory mechanisms are essential for complex manipulation tasks requiring reasoning over extended time horizons, advancing the field toward more autonomous and adaptable robots.
Significance
This research addresses a fundamental challenge in robotic manipulation—long-term memory integration—by proposing a systematic benchmark and a modular policy. It bridges the gap between short-horizon, Markovian models and real-world tasks that demand memory of past states. The findings have broad implications for industrial automation, service robotics, and autonomous systems, enabling robots to perform complex, multi-step tasks with higher reliability and flexibility. The introduction of Task Memory Complexity offers a standardized way to evaluate and compare future strategies, fostering progress in long-horizon decision-making.
Technical Contribution
The paper introduces a novel task classification based on memory requirements, creating RMBench, a comprehensive benchmark. The Mem-0 policy features a dual-system architecture with a vision-language model conditioned on explicit memory modules, including anchor and sliding window buffers. The system employs a subtask classifier for structured reasoning, integrated with diffusion-based action generation, enabling efficient long-horizon manipulation. The combination of task-level reasoning and explicit memory modeling constitutes a significant technical advancement over existing short-term models, providing a flexible framework for future research.
Novelty
This is the first work to formalize Task Memory Complexity as a metric for categorizing manipulation tasks by their memory demands. It also pioneers RMBench, a benchmark explicitly designed to evaluate memory-dependent manipulation, and introduces Mem-0, a modular, explicit memory-based policy. Unlike prior approaches relying solely on fixed observation histories or implicit memory, this work emphasizes structured, explicit memory modules integrated with reasoning and control, setting a new standard for systematic evaluation and policy design.
Limitations
- Mem-0 struggles with tasks requiring deep semantic understanding, such as interpreting nuanced instructions, where pretrained models still hold advantages. The subtask classifier may misjudge task completion in fine-grained scenarios, causing inefficiencies. Computational costs increase with longer tasks and larger memory buffers, limiting real-time deployment in resource-constrained systems. Future work should optimize memory management and inference speed, especially for highly complex or extended tasks.
Future Work
Future directions include integrating reinforcement learning to adaptively manage memory, exploring multi-modal data fusion for richer contextual understanding, and extending the benchmark to multi-robot collaboration scenarios. Enhancing the efficiency of the memory modules and reasoning components will be critical for real-world deployment. Additionally, developing more sophisticated task classification and hierarchical planning strategies will further improve long-term autonomy and robustness.
AI Executive Summary
Robotic manipulation has seen rapid advancements, yet most existing policies focus on short-term, Markovian tasks, limiting their ability to handle real-world scenarios that require reasoning over extended histories. To address this, the authors introduce RMBench, a comprehensive benchmark built on the concept of Task Memory Complexity (TMC), which categorizes manipulation tasks based on their memory demands. The benchmark includes nine tasks with varying levels of memory requirements, enabling systematic evaluation of policy capabilities.
Building upon this, the paper proposes Mem-0, a modular, explicit memory policy that integrates a vision-language model with structured memory modules—anchor and sliding window—to facilitate long-term memory retention and reasoning. The architecture includes a subtask classifier for structured decision-making, supporting closed-loop planning and execution. Extensive experiments in simulation and real robots demonstrate that Mem-0 significantly outperforms baseline models, with success rates reaching 42%, a substantial improvement over traditional approaches.
The ablation studies reveal that components such as the anchor memory and task-phase classifier are critical for performance, providing insights into effective architecture design. The results underscore the importance of explicit memory modeling for complex, non-Markovian manipulation tasks, opening new avenues for autonomous robot development.
This work not only advances the theoretical understanding of memory-dependent manipulation but also offers practical tools for benchmarking and policy development. It paves the way for robots capable of long-term reasoning, crucial for industrial automation, service robotics, and beyond. Future research will focus on integrating reinforcement learning, multi-modal data, and multi-robot systems to further enhance capabilities and efficiency.
Deep Analysis
Background
机器人操控技术经历了从传统硬编码到深度学习的演变，早期方法如基于模型的规划和强化学习在简单任务中表现优异，但在复杂、多步骤任务中缺乏长时记忆支持。近年来，视觉语言模型（VLM）和预训练策略推动了操控能力的提升，但大多模型仍依赖短期观察，难以应对非马尔可夫性任务。现有基准如MemoryBench和LIBERO-Long在一定程度上考虑记忆，但缺乏系统的任务分类和多层次评估。本文提出任务记忆复杂度（TMC）指标，旨在系统化划分操控任务的记忆需求，为后续研究提供统一标准。
Core Problem
当前大部分机器人操控策略假设任务为马尔可夫过程，依赖短期观察，难以应对需要长时记忆的任务。这限制了机器人在复杂环境中的自主性和适应性，特别是在需要记忆多轮操作、环境变化或历史状态的场景中表现不足。缺乏系统的任务分类和评估平台，使得不同策略在复杂任务中的性能难以比较，也阻碍了技术的推广应用。解决方案需支持长时记忆建模，提升机器人在实际应用中的表现。
Innovation
提出任务记忆复杂度（TMC）指标，系统划分操控任务的记忆需求，推动操控任务的标准化。设计RMBench，涵盖多层次记忆任务，支持大规模评估。Mem-0策略采用模块化架构，结合视觉语言模型（VLM）与多层记忆（锚点与滑动窗口），实现长时记忆的显式建模。引入任务阶段分类器，支持结构化推理和闭环控制，显著优于传统短期观察模型。这些创新突破了现有模型在非马尔可夫任务中的局限，推动机器人自主决策向更高层次发展。
Methodology
- �� 任务分类：基于任务记忆复杂度（TMC）指标，将任务划分为M(0)、M(1)和M(n)，定义每类任务对记忆的具体需求。• RMBench设计：在RoboTwin 2.0平台上构建九项操控任务，涵盖不同记忆层次，支持大规模评估。• Mem-0架构：包括规划模块（利用视觉语言模型进行子任务推理，结合关键记忆窗口）和执行模块（利用锚点和滑动记忆进行动作生成，采用扩散变换器（DiT）实现低延迟控制）。• 任务阶段分类器：通过MLP检测子任务完成状态，支持闭环交互。• 实验流程：在模拟和真实机器人中，训练多策略模型，比较成功率，进行消融分析，验证架构有效性。
Experiments
采用RMBench中的九项任务，评估包括非预训练模型（DP、ACT）和预训练模型（Pi0.6、X-VLA），训练50个示范，测试100轮。重点在记忆依赖任务中，观察Mem-0的成功率提升。消融实验分析不同记忆模块（锚点、滑动窗口、任务分类器）对性能的影响。真实机器人实验验证系统在实际场景中的适应性。指标包括成功率、任务完成时间和鲁棒性，确保结果的可靠性和可重复性。
Results
Mem-0在记忆任务中的平均成功率达42%，比传统模型提升31%。在“Rearrange Blocks”任务中成功率从29%提升至89%，在“Cover Blocks”中由0%升至68%。消融实验显示，移除锚点记忆或任务分类器导致成功率下降约20%。在真实机器人中，Mem-0表现出优异的泛化能力，验证了显式记忆机制的实用性。这些数据充分证明了设计的有效性和优越性。
Applications
该方法适用于工业自动化中的复杂装配、仓储中的多步骤操作，以及服务机器人中的长时任务管理。通过显式建模长时记忆，机器人能更好地理解环境变化和历史状态，提升自主决策能力。未来还可结合强化学习，优化记忆管理策略，推动智能机器人在复杂环境中的应用。
Limitations & Outlook
当前模型在处理高度语义依赖和超长任务时仍存在性能瓶颈，尤其在极端复杂场景下计算成本较高。任务分类器在某些细粒度任务中误判率较高，影响操作效率。系统在极端复杂场景或超长任务中存在计算成本较高的问题，未来需优化模型效率和记忆管理策略。
Plain Language Accessible to non-experts
想象你在厨房做饭，每次你都需要记住之前放了什么材料、放在哪个架子上，以及什么时候需要添加调料。普通机器人就像只看眼前的食材，不能记住之前的操作，所以只能做简单的菜。而这个研究就像教机器人记住所有操作步骤和材料放置位置，帮助它做出复杂的菜肴。通过设计专门的记忆工具，让机器人可以像人一样回忆起之前的细节，完成更复杂的任务，比如连续做多道菜，甚至根据之前的经验调整做法。这就像给机器人装上了“记忆芯片”，让它变得更聪明、更有耐心，能应对各种厨房挑战。
ELI14 Explained like you're 14
想象你在玩一个需要记住很多步骤的游戏，比如搭积木。刚开始你只记得下一步要放哪个积木，但随着游戏变得复杂，你需要记住之前放的积木位置和顺序。普通机器人就像只看眼前的积木，不能记住之前的操作，所以只能做简单的搭建。而这个研究就像给机器人装上了“记忆宝盒”，让它可以记住之前所有的操作步骤。这样，机器人就能更聪明地完成复杂的搭积木任务，比如搭出高塔或复杂的城堡。研究发现，装上这个“记忆宝盒”的机器人，比普通机器人更快、更稳地完成任务，特别是在需要记住很多信息的情况下。未来，这样的技术还能帮机器人在工厂、医院等地方做更复杂的工作，让它们变得更聪明、更可靠！
Abstract
Robotic manipulation policies have made rapid progress in recent years, yet most existing approaches give limited consideration to memory capabilities. Consequently, they struggle to solve tasks that require reasoning over historical observations and maintaining task-relevant information over time, which are common requirements in real-world manipulation scenarios. Although several memory-aware policies have been proposed, systematic evaluation of memory-dependent manipulation remains underexplored, and the relationship between architectural design choices and memory performance is still not well understood. To address this gap, we introduce RMBench, a simulation benchmark comprising 9 manipulation tasks that span multiple levels of memory complexity, enabling systematic evaluation of policy memory capabilities. We further propose Mem-0, a modular manipulation policy with explicit memory components designed to support controlled ablation studies. Through extensive simulation and real-world experiments, we identify memory-related limitations in existing policies and provide empirical insights into how architectural design choices influence memory performance. The website is available at https://rmbench.github.io/.