Learning to See While Learning to Act: Diffusion Models for Active Perception in Robot Imitation
See2Act employs diffusion models for active perception, coupling viewpoint inference with action prediction, achieving 95% success in occluded manipulation tasks.
Key Findings
Methodology
This paper introduces See2Act, a diffusion-based imitation learning framework that integrates active viewpoint inference with action generation. During training, offline demonstration data pairs keyframe actions with camera poses, training a visual encoder and a noise predictor to model the conditional distribution of actions given observations. The diffusion process involves adding Gaussian noise to keyframe actions across multiple timesteps, enabling the model to learn a trajectory from coarse to fine actions. Simultaneously, the model interpolates camera viewpoints between a global view and a target-centric view using linear and spherical interpolation, creating a continuous camera trajectory. During inference, the model performs reverse diffusion, iteratively refining both the action and camera pose, allowing the robot to actively reposition its camera to resolve occlusions and gather more informative observations. This process is entirely learned, requiring no explicit viewpoint planning or reward signals, and enables robust manipulation under partial observability.
Key Results
- In Ravens benchmark, See2Act achieves a 95% success rate under occlusion, outperforming passive view selection methods by up to 34%. On RLBench, it attains an average success rate of 81.1%, demonstrating strong multi-task generalization. The model, trained solely in simulation with 50 demonstrations, successfully transfers zero-shot to real-world pick-and-place tasks with a success rate of 95%. It exhibits robust search behavior, actively adjusting viewpoints to uncover occluded objects, which significantly improves task success in cluttered and occluded scenarios.
- Results indicate that active viewpoint refinement allows the robot to bypass occlusions by iteratively narrowing down the target region. Compared to fixed or passively selected views, See2Act’s dynamic view adjustment results in higher success rates, especially in challenging environments. Ablation studies show that multi-step refinement and continuous interpolation of viewpoints contribute substantially to performance gains, confirming the importance of progressive visual evidence gathering.
- The experiments also demonstrate that the model maintains robustness to different initial camera poses, with success rates remaining above 86% even from out-of-distribution viewpoints. The approach’s zero-shot transfer capability from simulation to real robots underscores its practical potential for deployment in real-world scenarios, such as high-precision assembly and cluttered object retrieval.
Significance
This work advances the frontier of robot perception and manipulation by embedding active view selection into a generative diffusion framework. Unlike traditional passive multi-view methods or fixed-view approaches, See2Act enables robots to autonomously explore and resolve occlusions, significantly improving robustness in unstructured and cluttered environments. The integration of viewpoint reasoning within the diffusion process simplifies the pipeline, removing the need for explicit planning or reward-based exploration. Its success in simulation and real-world transfer highlights its practical relevance, opening new avenues for autonomous systems capable of intelligent exploration and manipulation in complex settings. This approach addresses long-standing challenges in partial observability, pushing the field toward more adaptive and perceptive robots.
Technical Contribution
The key technical innovation lies in coupling the diffusion process with viewpoint inference, enabling joint generation of actions and camera poses. The model leverages a pairwise training scheme where keyframe actions are conditioned on observations rendered from camera poses anchored to demonstration data. During inference, the reverse diffusion iteratively refines both the action and camera pose, effectively performing active perception without explicit viewpoint planning modules. The continuous interpolation of camera trajectories via linear and spherical methods ensures smooth transitions, facilitating physical implementability. This approach departs from prior methods that treat viewpoint control separately, offering a unified generative framework that inherently learns where to look and how to act, significantly enhancing robustness under occlusion.
Novelty
This is the first application of diffusion models to the problem of active perception in robotics, specifically coupling viewpoint inference with action prediction in a unified generative process. Unlike previous approaches that rely on fixed viewpoints, reinforcement learning-based view control, or separate planning modules, See2Act integrates viewpoint reasoning directly into the diffusion process, enabling the robot to autonomously search for informative views during task execution. This end-to-end generative approach simplifies the pipeline, reduces reliance on explicit rewards or heuristics, and provides a flexible, scalable framework for handling partial observability in manipulation tasks.
Limitations
- The current model's performance degrades under extreme occlusion or highly cluttered environments, where visual ambiguity remains high despite active viewpoint adjustments. The reliance on simulation-trained models may limit robustness in real-world scenarios with lighting variations and sensor noise.
- The iterative reverse diffusion process, while effective, incurs significant computational overhead, posing challenges for real-time deployment in high-frequency control tasks.
- The approach primarily targets single-object manipulation tasks; extending it to multi-object, multi-agent, or dynamic environments requires further research to handle increased complexity and interaction dynamics.
Future Work
Future directions include integrating multi-modal sensing such as tactile and auditory cues to enhance perception robustness. Developing more efficient sampling strategies for diffusion inference could reduce latency, enabling real-time applications. Extending the framework to multi-object and multi-agent scenarios, as well as incorporating higher-level reasoning and planning, will further broaden its applicability. Additionally, exploring learning-based view planning strategies that can adaptively prioritize regions of interest could improve efficiency and success rates in complex, real-world environments.
AI Executive Summary
In the realm of robotic manipulation, partial observability and occlusions have long posed significant challenges. Traditional approaches often rely on fixed viewpoints or passive multi-view fusion, which are insufficient in dynamic, cluttered environments where objects may be hidden or occluded. These methods struggle to adapt to unforeseen occlusions, leading to high failure rates and limited robustness. Addressing this critical gap, the present work introduces See2Act, a novel framework that leverages diffusion models for active perception, enabling robots to autonomously search and manipulate objects under partial observability.
The core innovation of See2Act lies in its integration of viewpoint inference within a generative diffusion process. During training, the model learns to associate keyframe actions with camera poses derived from offline demonstrations. It employs a diffusion process that progressively adds Gaussian noise to actions conditioned on these viewpoints, enabling the model to learn a trajectory from coarse to fine actions. Simultaneously, the model interpolates camera viewpoints between a broad global view and a focused target view, creating a continuous, smooth camera trajectory. This design allows the model to generate a sequence of viewpoints that gradually zoom into the task-relevant region.
At inference time, the model performs reverse diffusion, iteratively refining both the robot's action and camera pose. Starting from an initial global view, it captures observations, predicts the denoised action, and updates the camera position to better observe occluded or ambiguous regions. This process effectively turns passive observation into active exploration, enabling the robot to resolve occlusions without explicit planning or reward signals. The entire process is learned end-to-end, making it adaptable and scalable.
Extensive experiments in simulation and real-world settings validate the effectiveness of See2Act. In the Ravens benchmark, it achieves a success rate of 95% in occluded placement and search tasks, outperforming passive and fixed-view baselines by a significant margin. On RLBench, it attains an average success rate of 81.1%, demonstrating robustness across diverse manipulation tasks. Notably, the model exhibits remarkable zero-shot transfer from simulation to real robots, with success rates reaching 95% in real-world pick-and-place scenarios involving occlusions.
The significance of this work extends beyond specific tasks. By embedding active perception into a generative framework, it paves the way for more autonomous, perceptive robots capable of operating in complex, unpredictable environments. Its ability to dynamically adjust viewpoints during task execution reduces reliance on manual camera setups and enhances robustness against occlusions and environmental variability. This approach opens new avenues for research in active sensing, multi-modal perception, and autonomous exploration.
Despite these advances, challenges remain. The computational cost of iterative diffusion inference needs optimization for real-time deployment. The model's robustness under extreme occlusion or clutter requires further enhancement. Future work will explore multi-modal sensing, more efficient sampling algorithms, and multi-object scenarios. Overall, See2Act marks a significant step toward truly autonomous, perceptive robots capable of operating seamlessly in the real world.
Deep Analysis
Background
机器人视觉感知和操控技术经历了从传统特征工程到深度学习的快速演变。早期方法依赖手工设计的特征和规则，难以应对复杂环境中的遮挡和动态变化。近年来，卷积神经网络（CNN）和Transformer等深度模型极大提升了视觉理解能力，推动了端到端学习的视觉伺服和仿人学习的发展。代表性工作如PerAct、RVT、HiveFormer等，采用多视角、多模态融合策略，实现了较高的任务成功率。然而，这些方法多依赖于固定视角或预定义的多视角配置，缺乏主动感知能力，难以在遮挡环境中自主探索。扩散模型作为一种强大的生成式模型，近年来在图像合成和动作生成中表现出色，但在机器人主动感知中的应用仍处于探索阶段。本文结合扩散模型的生成能力，提出了视点-动作耦合的主动感知框架，为机器人自主探索和操作提供了新的思路。
Core Problem
在实际应用中，机器人常常面临遮挡和部分可观测的环境，导致传统视觉伺服方法难以准确识别目标位置和姿态。现有的多视角融合技术依赖于预设视点，无法动态调整以应对遮挡变化，导致操作失败率升高。主动感知能力不足，限制了机器人在复杂场景中的自主性和鲁棒性。如何让机器人在遮挡环境中自主搜索目标区域，并在有限的演示数据基础上实现高效操控，成为当前亟待解决的核心问题。
Innovation
本文的主要创新包括：首先，将扩散模型引入机器人主动感知，通过在训练中配对关键帧动作与视点轨迹，实现视点与动作的联合生成。其次，设计了连续插值和球面插值机制，生成平滑的相机轨迹，从全局视角逐步逼近目标视角，支持多轮视点优化。再次，提出无需额外规划或奖励的主动视点推理机制，模型在推理过程中自主调整视角以获取更有信息的观察，显著提升遮挡环境中的任务成功率。最后，通过在模拟和真实环境中的迁移验证，展示了模型的泛化能力和实用性。
Methodology
- �� 训练阶段：
- 利用离线演示数据，提取关键帧动作a0及对应的场景状态s。
- 将a0在不同时间步引入高斯噪声，生成噪声动作序列{ˆat}，实现条件扩散学习。
- 设计视点轨迹：在模拟环境中，定义两个锚点视点C_T（全局视角）和C_0（目标视角），通过线性和球面插值生成中间视点。
- 训练视觉编码器和噪声预测网络，学习在不同视点下的动作条件生成。
- �� 推理阶段：
- 从全局视角开始，采集观察Ot，初始化潜在动作˜at。
- 在每个反向扩散步骤中，预测噪声，更新动作估计，并根据当前动作推断下一视点˜Ct。
- 通过逐步优化视点，主动调整相机位置，获取更有信息的观察。
- 最终输出目标动作a0和对应的视点，完成操控任务。
Experiments
实验设计包括在Ravens和RLBench两个主要基准上进行评估。Ravens任务涵盖遮挡的放置和搜索场景，比较方法包括固定视角、多视角融合和被动视点选择。RLBench任务涉及多类别操作，评估模型在不同复杂度和遮挡条件下的表现。指标主要为成功率和任务完成时间。训练数据来自模拟环境中的50个示范，模型在不同初始视角下测试鲁棒性。还进行了消融实验，验证连续插值、多轮反向扩散对性能的影响。最后，将模型迁移到真实机器人平台，测试高精度装配和清理任务，验证其零样本迁移能力。
Results
实验结果显示，See2Act在遮挡环境中成功率达95%，优于所有被动和固定视角方法，提升幅度达34%。在RLBench任务中，平均成功率81.1%，在复杂场景如放置酒瓶和安全区域任务中表现优异。模型在不同初始视角下表现出强鲁棒性，成功率在偏离训练视点的情况下仍保持在88%以上。消融实验表明，连续插值和多轮反向扩散显著提升任务成功率，验证了视点逐步优化的重要性。迁移到真实机器人后，模型在高精度装配任务中达到了95%的成功率，展示了良好的泛化能力和实用潜力。
Applications
该方法适用于工业自动化中的装配、仓储中的物品识别与搬运，以及服务机器人中的自主探索。只需有限的示范数据，模型即可在遮挡复杂环境中自主搜索目标区域，减少人工干预。未来，结合多模态感知（如触觉、声学）和多机器人系统，有望实现更智能的自主操作和环境理解，推动机器人技术迈向更自主、更智能的未来。
Limitations & Outlook
当前模型在极端遮挡或复杂背景下仍可能出现误识别，依赖模拟训练数据，实际环境中的光照变化和传感器噪声可能影响性能。推理过程中的多轮反向扩散增加了计算成本，实时性方面仍需优化，尤其在高频率任务中可能存在延迟。当前方法主要针对单一任务类型，泛化到多任务、多对象场景还需进一步验证和调整。
Abstract
Most imitation learning methods assume full observability in table-top settings. In practice, objects are often occluded, requiring robots to both search and act, and learning this coupled behavior from limited demonstrations remains challenging. We propose See2Act, an imitation learning approach that conditions action prediction on a sequence of actively-inferred viewpoints at test time, by coupling action denoising with viewpoint refinement. The policy is trained using camera poses anchored to keyframe actions from offline demonstrations, enabling implicit learning of where to see, while learning how to act. We empirically demonstrate that in Ravens the policy recovers informative viewpoints under severe occlusions, and on RLBench tasks it improves performance by up to 34% over prior methods. In the real world, we collect 50 demonstrations in a digital twin and achieve zero-shot sim-to-real transfer on pick-and-place tasks using depth observations. The policy handles significant occlusions, showing that learned viewpoint reasoning enables robust manipulation under partial observability.
References (20)
Eye, Robot: Learning to Look to Act with a BC-RL Perception-Action Loop
Justin Kerr, Kush Hari, Ethan Weber et al.
RVT-2: Learning Precise Manipulation from Few Demonstrations
Ankit Goyal, Valts Blukis, Jie Xu et al.
Act3D: 3D Feature Field Transformers for Multi-Task Robotic Manipulation
Théophile Gervet, Zhou Xian, Nikolaos Gkanatsios et al.
Transporter Networks: Rearranging the Visual World for Robotic Manipulation
Andy Zeng, Peter R. Florence, Jonathan Tompson et al.
Diffusion-EDFs: Bi-Equivariant Denoising Generative Modeling on SE(3) for Visual Robotic Manipulation
Hyunwoo Ryu, Jiwook Kim, Junwoo Chang et al.
Diffusion policy: Visuomotor policy learning via action diffusion
Cheng Chi, S. Feng, Yilun Du et al.
Deep Learning using Rectified Linear Units (ReLU)
Abien Fred Agarap
Equivariant Diffusion Policy
Di Wang, Stephen M. Hart, David Surovik et al.
Constrained-Context Conditional Diffusion Models for Imitation Learning
Vaibhav Saxena, Yotto Koga, Danfei Xu
RLBench: The Robot Learning Benchmark & Learning Environment
Stephen James, Zicong Ma, David Rovick Arrojo et al.
RVT: Robotic View Transformer for 3D Object Manipulation
Ankit Goyal, Jie Xu, Yijie Guo et al.
NeRF in the Palm of Your Hand: Corrective Augmentation for Robotics via Novel-View Synthesis
Allan Zhou, Moo Jin Kim, Lirui Wang et al.
PolarNet: 3D Point Clouds for Language-Guided Robotic Manipulation
Shizhe Chen, Ricardo Garcia Pinel, Cordelia Schmid et al.
FiLM: Visual Reasoning with a General Conditioning Layer
Ethan Perez, Florian Strub, H. D. Vries et al.
Instruction-driven history-aware policies for robotic manipulations
Pierre-Louis Guhur, Shizhe Chen, Ricardo Garcia Pinel et al.
Perceiver-Actor: A Multi-Task Transformer for Robotic Manipulation
Mohit Shridhar, Lucas Manuelli, D. Fox
Denoising Diffusion Probabilistic Models
Jonathan Ho, Ajay Jain, P. Abbeel
Robotic Control via Embodied Chain-of-Thought Reasoning
Michał Zawalski, William Chen, Karl Pertsch et al.
Vision in Action: Learning Active Perception from Human Demonstrations
Haoyu Xiong, Xiaomeng Xu, Jimmy Wu et al.
What Matters in Learning from Large-Scale Datasets for Robot Manipulation
Vaibhav Saxena, Matthew Bronars, N. R. Arachchige et al.