SAM2Act: Integrating Visual Foundation Model with A Memory Architecture for Robotic Manipulation
SAM2Act integrates large-scale visual foundation models with multi-view transformers, achieving 86.8% success and enhanced spatial memory.
Key Findings
Methodology
The approach builds upon RVT-2 multi-view transformer, integrating SAM2 visual encoder for multi-resolution feature extraction. It employs a cascaded multi-resolution upsampling mechanism to refine spatial features for precise action heatmaps. A memory bank, encoder, and attention modules are incorporated into SAM2Act+ to enable episodic spatial memory. Virtual cameras generate multi-view images, which are processed by the SAM2 encoder fine-tuned via LoRA. The multi-view transformer fuses these features with language instructions, predicting actions. During training, behavior cloning leverages the memory modules to improve sequential decision-making in complex tasks.
Key Results
- In RLBench, SAM2Act achieves an average success rate of 86.8%, outperforming RVT-2 (81.4%) by 5.4%. It excels particularly in high-precision tasks like peg insertion, with a 44% success rate increase. On MemoryBench, SAM2Act+ reaches 94.3% success, surpassing baselines by 39.3%. Under environmental perturbations, success rate drops only 4.3%, demonstrating strong robustness. Real-world tests with a Franka Panda robot show stable performance across four manipulation tasks, confirming practical viability.
- Ablation studies reveal that multi-resolution upsampling and memory modules contribute significantly, with performance dropping ~20% when used separately. The model maintains high accuracy across diverse scenarios, confirming the effectiveness of combining visual foundation models with episodic memory.
- The approach demonstrates superior generalization in environments with visual and object variations, maintaining high success rates in unseen conditions, and effectively recalling past actions in memory-dependent tasks.
Significance
This work advances robotic manipulation by integrating large-scale visual models and memory mechanisms, addressing core challenges of multi-task learning, generalization, and episodic recall. It provides a scalable framework for robots to operate autonomously in complex, dynamic environments, with broad implications for industrial automation, service robotics, and disaster response. The introduction of MemoryBench offers a standardized way to evaluate spatial memory in robotic systems, fostering further research in this direction.
Technical Contribution
The paper introduces SAM2Act, combining SAM2 visual encoder with a cascaded multi-resolution upsampling network, significantly improving spatial feature extraction. It extends this with SAM2Act+ by adding a memory bank, encoder, and attention modules, enabling episodic spatial memory. The design leverages virtual multi-view images and fine-tuning via LoRA for domain adaptation. The novel MemoryBench benchmark systematically evaluates spatial memory and action recall, filling a critical gap in robotic manipulation research. Empirical results demonstrate state-of-the-art performance across simulated and real-world tasks.
Novelty
This is the first work to combine large-scale visual foundation models with multi-view transformers for robotic manipulation, emphasizing multi-resolution feature refinement and episodic memory integration. The introduction of MemoryBench provides a new standardized benchmark for spatial memory evaluation, addressing a previously overlooked aspect of robotic cognition. The architecture’s ability to generalize across diverse tasks and environments marks a significant step forward in autonomous robotic intelligence.
Limitations
- Despite strong performance, the model's robustness under extreme lighting, occlusion, or highly dynamic scenes remains limited, requiring further improvements in perception robustness.
- Training involves high computational costs due to multi-view image generation and memory module optimization, which may hinder scalability.
- Long-term memory retention over extended tasks could degrade, necessitating more efficient memory encoding and retrieval strategies in future work.
Future Work
Future directions include optimizing memory encoding efficiency, integrating reinforcement learning for adaptive decision-making, and expanding MemoryBench to include more complex, real-world scenarios. Additionally, exploring continual learning and lifelong memory retention will further enhance autonomous capabilities in unstructured environments.
AI Executive Summary
Robotic manipulation in real-world environments demands high adaptability, precision, and memory. Traditional methods often struggle with complex, dynamic scenarios, especially when tasks require recalling past actions or understanding spatial relationships. To address these challenges, this research introduces SAM2Act, a novel policy architecture that leverages large-scale visual foundation models and multi-view transformers to enhance spatial feature extraction and task generalization.
The core innovation lies in integrating the SAM2 visual encoder with a cascaded multi-resolution upsampling network, enabling the robot to generate highly accurate action heatmaps from virtual multi-view images. Building upon this, the authors propose SAM2Act+, which incorporates a memory bank, encoder, and attention modules, allowing episodic recall of past actions and spatial states. This architecture significantly improves performance on complex spatial memory tasks, as demonstrated in the newly designed MemoryBench benchmark, where SAM2Act+ achieves 94.3% success rate, outperforming existing methods by a wide margin.
Extensive experiments in both simulated and real-world settings validate the approach. In RLBench, SAM2Act surpasses prior state-of-the-art methods, achieving an 86.8% success rate across 18 tasks, with notable improvements in high-precision tasks. In the Colosseum benchmark, the model maintains robustness under environmental perturbations, with only a 4.3% performance drop. Real robot experiments further confirm its practical applicability, with success rates exceeding 90% in multiple manipulation tasks.
This work marks a significant step toward autonomous robots capable of complex, memory-dependent manipulation in diverse environments. By combining advanced visual models with episodic memory, it opens new avenues for intelligent, adaptable robotic systems. Future research will focus on improving long-term memory retention, reducing computational costs, and extending capabilities to more dynamic, unstructured scenarios.
Deep Analysis
Background
机器人操控技术经历了从二维感知到三维空间理解的演变。早期方法如Transporter网络和CLIPort主要解决二维抓取和放置任务，受限于空间复杂性。近年来，基于深度学习的多视角变换器（如RVT-2）引入虚拟相机、多视角融合，显著提升空间理解能力。视觉基础模型（如SAM）在目标识别和空间推理中表现优异，为机器人提供丰富的感知特征。尽管如此，现有技术在多任务协作、复杂环境适应和空间记忆方面仍存在不足，亟需结合更强的视觉表达和记忆机制以实现更高自主性。
Core Problem
现有机器人操控系统在多任务、多环境变化中表现有限，尤其在需要空间记忆和动作回忆的任务中表现不佳。传统方法多依赖静态感知或有限的记忆存储，难以应对动态场景和长时序决策。如何结合大规模视觉模型与记忆机制，实现高精度、多任务、鲁棒性强的自主操控，是亟待解决的核心问题。这关系到工业自动化、服务机器人和应急救援等实际应用的广泛需求。
Innovation
提出SAM2Act架构，结合SAM2视觉编码器、多分辨率上采样和多视角变换器，显著提升空间特征表达能力。引入记忆银行、编码器和注意力机制，构建SAM2Act+，实现空间记忆的episodic存储与回忆。设计MemoryBench，系统评估空间记忆能力，填补机器人操控中对记忆依赖任务的评估空白。该方案在多任务和环境扰动中表现优异，推动机器人自主操作迈向更高智能水平。
Methodology
- �� 利用虚拟相机从多视角生成场景图像，输入SAM2编码器提取多分辨率特征。• 采用LoRA微调SAM2编码器以适应特定操控任务。• 多分辨率上采样机制通过级联卷积上采样器，逐步细化特征图，提升空间定位精度。• 构建多视角变换器（RVT-2基础），结合特征和指令预测操作热图。• 引入记忆银行存储历史动作和空间状态，利用注意力机制融合历史信息，增强episodic记忆。• 训练过程中采用行为克隆策略，结合记忆机制优化多任务学习效果。
Experiments
在RLBench和MemoryBench两个基准上进行评估，RLBench涵盖18个多任务场景，MemoryBench专注空间记忆任务。模型在不同环境扰动和真实机器人中测试，比较RVT-2、PerAct等基线。训练使用100个示范，测试25个未见示范。指标包括成功率、泛化能力和记忆表现。消融实验验证多分辨率上采样和记忆机制的贡献，参数调优确保模型稳定性。
Results
SAM2Act在RLBench中成功率达86.8%，优于RVT-2的81.4%，在高精度任务中提升显著。MemoryBench中，SAM2Act+成功率94.3%，比其他方法高出39.3%。在环境扰动下，成功率仅下降4.3%，显示出极强的鲁棒性。真实机器人实验中，模型在多项操控任务中表现稳定，验证了其实际应用潜力。这些结果表明，结合视觉基础模型和记忆机制的策略，极大改善了机器人在复杂环境中的自主能力。
Applications
该方法适用于工业自动化、服务机器人、救援行动等场景，能实现高精度、多任务操作。只需提供示范和环境信息，即可自主完成复杂任务。未来，结合强化学习可进一步提升自主适应和学习能力，推动机器人在未知环境中的自主决策。
Limitations & Outlook
模型在极端光照变化或遮挡情况下表现仍有限，记忆可能在长时间任务中遗忘部分信息。训练成本较高，尤其在多视角虚拟图像生成和记忆机制优化上。复杂场景如多物体交互和动态环境仍需验证，未来需增强鲁棒性和效率。
Plain Language Accessible to non-experts
想象你在厨房做饭，你需要记住放了哪些调料、什么时候放的、以及下一步要做什么。机器人也是这样，它通过摄像头观察环境，学习如何完成任务，但它不像人一样记忆清楚。SAM2Act就像给机器人装了一个聪明的记忆本，能记住之前的操作和空间位置。它还用多角度的“眼睛”看场景，把信息拼在一起，帮助机器人更准确地操作。这样，机器人不仅能完成简单任务，还能记住复杂的步骤，像人一样聪明地工作。
ELI14 Explained like you're 14
嘿，你知道吗？机器人就像一个超级勤快的厨师，但它需要记住很多事情，比如哪个碗在哪个架子上，什么时候放调料。以前的机器人只能看一眼，然后做事，但有时候会忘记之前的步骤。现在，这个新方法就像给机器人装了一个记忆宝盒，它可以记住之前的操作，还能从不同角度观察厨房，把信息拼在一起，帮它做得更快更准。这样一来，机器人就能像人一样记住事情，完成复杂的任务，比如把东西放回原位或者重新组装东西。未来，这样的机器人可以帮我们做更多事情，比如在危险的地方救援或帮忙做家务。
Abstract
Robotic manipulation systems operating in diverse, dynamic environments must exhibit three critical abilities: multitask interaction, generalization to unseen scenarios, and spatial memory. While significant progress has been made in robotic manipulation, existing approaches often fall short in generalization to complex environmental variations and addressing memory-dependent tasks. To bridge this gap, we introduce SAM2Act, a multi-view robotic transformer-based policy that leverages multi-resolution upsampling with visual representations from large-scale foundation model. SAM2Act achieves a state-of-the-art average success rate of 86.8% across 18 tasks in the RLBench benchmark, and demonstrates robust generalization on The Colosseum benchmark, with only a 4.3% performance gap under diverse environmental perturbations. Building on this foundation, we propose SAM2Act+, a memory-based architecture inspired by SAM2, which incorporates a memory bank, an encoder, and an attention mechanism to enhance spatial memory. To address the need for evaluating memory-dependent tasks, we introduce MemoryBench, a novel benchmark designed to assess spatial memory and action recall in robotic manipulation. SAM2Act+ achieves an average success rate of 94.3% on memory-based tasks in MemoryBench, significantly outperforming existing approaches and pushing the boundaries of memory-based robotic systems. Project page: sam2act.github.io.