Analytic Concept-Centric Memory for Agentic Embodied Manipulation
Concept-centric memory framework improves long-horizon embodied manipulation by explicit object structure, achieving 70% success on RMBench.
Key Findings
Methodology
The approach constructs a structured memory system centered on analytic concepts, representing objects via semantic parts, parametric templates, grounded poses, affordances, and manipulation states. It combines static prior knowledge—semantic and template libraries—with dynamic experience memory, including grounded objects, scene states, transition, and skill memories. During runtime, a coarse-to-fine retrieval mechanism locates relevant objects, states, and skills based on current observations, scene relations, and task instructions. The system performs state inference through scene graphs and action effects, enabling persistent object tracking, state change reasoning, and skill reapplication. This integrated architecture supports efficient, interpretable, and scalable long-term manipulation in complex environments.
Key Results
- On RMBench, the proposed method achieved over 70% success rate across nine manipulation tasks, outperforming baseline approaches like Mem-0 and Mem-VLA by 15-20%. Object re-identification accuracy improved, reducing positional errors to around 3cm, significantly better than 8cm in baseline models. Ablation studies confirmed that explicit object-part modeling and transition memory contributed to a 20% increase in task success and 15% in skill generalization. The system demonstrated robust performance in real-world scenarios, maintaining high accuracy despite perceptual noise.
- In generalization tests on articulated objects from PartNet-Mobility, the system transferred manipulation skills across similar object structures, achieving 65-75% success in unseen configurations, evidencing strong transferability. The hierarchical retrieval and explicit scene reasoning enabled the agent to adapt to novel objects and environments effectively, surpassing prior methods that relied solely on embedding-based or unstructured memories.
- The experiments validated that structured, analytic concept-based memory significantly enhances object persistence, state reasoning, and skill reuse, leading to more reliable and efficient long-horizon manipulation, with potential for real-world deployment in service robots and industrial automation.
Significance
This work addresses fundamental challenges in robotic long-term manipulation by introducing a structured, interpretable memory system that explicitly models object parts, states, and manipulation skills. It bridges perception, reasoning, and action, enabling robots to maintain persistent object identities, infer state changes, and reuse skills across tasks. Such capabilities are crucial for autonomous systems operating in dynamic, unstructured environments, where traditional unstructured or embedding-based memories fall short. The proposed framework paves the way for more intelligent, adaptable robots capable of continuous learning and complex task execution, with broad implications for automation, service robotics, and human-robot collaboration.
Technical Contribution
The paper introduces a novel analytic concept-centric memory architecture that integrates static knowledge bases with dynamic experience, supporting explicit object-part, pose, and affordance modeling. It employs a hierarchical retrieval process combining coarse filtering with fine-grained ranking, and incorporates scene graph-based state inference to predict object state transitions. The design enables persistent object tracking, accurate state change reasoning, and skill reapplication, surpassing existing unstructured or embedding-only methods. The system's modularity and interpretability facilitate scalability and transferability in complex manipulation tasks, representing a significant advancement over prior works like scene graphs, embedding memories, and end-to-end policies.
Novelty
This is the first comprehensive framework to embed analysis-based object representations directly into a hierarchical memory system tailored for long-horizon embodied manipulation. Unlike prior approaches relying solely on unstructured histories or embeddings, this method explicitly models object parts, poses, and affordances, enabling precise retrieval and reasoning. The integration of static knowledge with dynamic experience, along with scene graph-based state inference, provides a new paradigm for persistent, interpretable, and transferable robotic memory, marking a significant step forward in embodied AI research.
Limitations
- The system's performance heavily depends on the accuracy of perception modules like object detection and pose estimation; errors here can propagate through the reasoning process.
- Computational complexity increases with scene size and object count, impacting real-time applicability in highly cluttered environments.
- Generalization to entirely novel object categories or highly deformable objects remains limited; future work should incorporate adaptive learning mechanisms.
Future Work
Future research will focus on integrating reinforcement learning to optimize skill policies within this structured memory framework, enhancing adaptability. Additionally, expanding multimodal perception—such as tactile and auditory cues—could improve scene understanding. Extending the approach to multi-robot systems and lifelong learning scenarios will further push the boundaries of autonomous, persistent manipulation in unstructured environments.
AI Executive Summary
Deep Dive
Plain Language Accessible to non-experts
想象你在厨房里做饭。以前的方法就像把所有食材和工具都放在一个大箱子里，想找什么都得翻半天。而现在，这个新系统像是给每个食材和工具都贴上标签，告诉你它们的部分、位置和状态。比如，面包烤到什么程度了，菜叶是不是洗干净了。每次你用完一个工具或食材，系统都会记住它的变化。这样，你每次做饭时，只要看标签，就能很快找到需要的材料，还知道它们的状态。这个系统让做饭变得像有个聪明的助手帮你整理，效率更高，出错也少了。它就像一个厨房里的超级大脑，帮你记住每个细节，让你轻松做出美味的饭菜。
ELI14 Explained like you're 14
想象你在学校玩拼图游戏。每次你都要记住每块拼图的位置和形状，还要记得上一块拼完后下一块该怎么拼。以前的方法就像把所有拼图碎片都放在一个大袋子里，找起来很麻烦。现在，这个新系统就像用标签把每块拼图都标好，告诉你它属于哪个部分、在哪个位置、拼完后会变成什么样。每次你拼的时候，只要看标签，就能很快找到正确的碎片，还能知道拼完后会变成什么样。这样，你拼图的速度就快多了，也不容易拼错。这就像有个聪明的朋友帮你记住每块拼图的细节，让你更快完成拼图，也更开心。
Abstract
Long-horizon embodied manipulation requires agents to remember persistent objects, track changing scene states, and reuse prior interaction knowledge. However, existing agent memories are often stored as unstructured histories or embedding-based records, making it difficult to retrieve manipulation-relevant object parts, physical states, action effects, and executable skills. We propose an analytic concept-centric memory framework for agentic embodied manipulation. Our memory organizes experience around structured analytic concepts, where objects are represented by semantic parts, parametric templates, grounded poses, affordances, and manipulation states. It further connects object and scene memories with transition memory for action-induced state changes and skill memory for template-grounded and policy-grounded execution. At runtime, the agent performs structured coarse-to-fine retrieval to identify relevant objects, states, transitions, and skills, supporting state-consistent reasoning and skill reuse. Experiments on memory-dependent manipulation, articulated-object generalization, real-world memory evaluation, and ablations show that our approach improves task completion, retrieval accuracy, object re-identification, and cross-object skill generalization over unstructured and embedding-based memory baselines.