Attention from Action, for Action: Emergent Visual Bottlenecks for Policy Learning
Seeker learns control-relevant visual regions from action supervision, boosting robot task success from 48.3% to 76.7%.
Key Findings
Methodology
Seeker employs frozen DINOv3 features combined with an iterative, query-based attention mechanism conditioned on task and robot state. It features multiple attention heads with gating to fuse diverse spatial cues. The process involves: β’ Extracting features from pretrained DINOv3; β’ Designing a task- and state-conditioned query vector Q(0); β’ Iteratively updating Q through multi-head self-attention, producing attention maps and context vectors; β’ Using a diffusion-based action head trained with action prediction loss to supervise ROI learning without explicit spatial labels. The learned ROI is then converted into a bounding box and mask, which are used for RGB cropping, mask-guided background augmentation, and point cloud filtering, enhancing policy robustness and data efficiency.
Key Results
- In simulation, Seeker increased average success rate from 42.6% (no crop baseline) to 62.6%, a 52.8% relative gain, with notable improvements in contact-rich tasks like assembly and pick-and-place. In real-world experiments, success rate rose from 48.3% to 76.7%, and under environmental shifts, success increased from 20.0% to 60.0%. The ROI learned by Seeker supports mask-guided augmentation and point cloud filtering, significantly improving robustness against lighting and background variations.
- Seeker's action supervision enables it to autonomously learn task-relevant visual regions, outperforming traditional keypoint or event-based ROI methods. Its multi-head iterative attention adapts to task stages and environment changes, providing a flexible, dynamic focus that improves policy data efficiency and generalization across diverse scenarios.
- Ablation studies confirm that the action-supervised ROI surpasses external labels and heuristic methods like RVT2-Crop and Oracle ROI, demonstrating superior adaptability and robustness. The framework's ability to generalize across multiple tasks and sensing modalities highlights its potential for broad application in robotic manipulation.
Significance
This work addresses a fundamental challenge in robot perception: how to autonomously identify task-critical visual information without relying on costly annotations. By learning control-dependent visual bottlenecks directly from actions, the approach significantly reduces labeling efforts while enhancing robustness and adaptability. It paves the way for scalable, data-efficient robot learning in complex, real-world environments, with implications for industrial automation, service robots, and autonomous systems. The method's ability to dynamically focus on relevant regions aligns well with the goal of creating more autonomous, perceptually aware robots capable of operating in unstructured settings.
Technical Contribution
The paper introduces a novel framework combining pretrained visual features with an iterative, query-based attention mechanism conditioned on task and robot state. Key innovations include: β’ A dynamic, multi-head attention module with gating, enabling flexible fusion of diverse spatial cues; β’ A diffusion-based action prediction head that supervises ROI learning without spatial labels; β’ A multi-step iterative process that refines the attention focus as task progresses, allowing the ROI to move and scale adaptively. These contributions collectively enable the robot to learn a control-relevant visual bottleneck in a fully autonomous manner, improving data efficiency and robustness without requiring external supervision or annotations.
Novelty
Unlike prior works that rely on fixed event detection or semantic labels for ROI extraction, Seeker learns a control-dependent ROI directly from action supervision using a dynamic, iterative attention process. It leverages pretrained visual features and a gating mechanism to adaptively fuse multiple spatial cues, making it highly flexible and applicable across tasks. This is the first approach to combine diffusion-based action prediction with a query-driven attention mechanism for autonomous ROI learning, marking a significant step forward in label-free, control-aware visual perception for robotics.
Limitations
- Seeker's performance may degrade under severe visual occlusion or when visual information is heavily corrupted, as the attention mechanism relies on visual features that can be compromised.
- Training requires a substantial amount of diverse action demonstrations to generalize well across different tasks and environments, which may be costly in some applications.
- The iterative attention process, while flexible, introduces computational overhead, potentially limiting real-time deployment in high-frequency control scenarios.
- Current experiments focus on manipulation tasks with moderate complexity; scaling to highly dynamic or unstructured environments remains a challenge.
- Future work should explore integrating multi-modal sensory data, optimizing computational efficiency, and extending to multi-agent scenarios.
Future Work
Future research could focus on integrating additional sensory modalities such as depth or tactile information to further improve ROI robustness. Developing more efficient, real-time attention mechanisms will enable deployment in high-speed control tasks. Combining this approach with reinforcement learning could optimize ROI adaptation strategies through trial-and-error, enhancing autonomous exploration. Extending the framework to multi-robot systems and unstructured environments will broaden its applicability. Additionally, exploring self-supervised or unsupervised methods to reduce data requirements remains an important direction.
AI Executive Summary
In the realm of robotic manipulation, one persistent challenge is how to efficiently and reliably focus perception on task-relevant regions within complex visual environments. Traditional approaches often depend on external labels, such as human gaze, object annotations, or semantic segmentation, which are costly to acquire and may not generalize well across different tasks or environments. This reliance hampers the scalability and adaptability of robotic systems, especially in unstructured or dynamic settings.
Recent advances in deep learning, particularly the advent of large-scale pretrained visual models like DINO, have opened new avenues for autonomous perception. However, leveraging these models for task-specific attention remains non-trivial. Existing methods that attempt to learn regions of interest (ROI) often rely on fixed heuristics, such as detecting keyframes based on motion or contact events, which are limited by their static nature and inability to adapt to continuous task progress.
Addressing this gap, the paper introduces Seeker, a novel framework that learns control-relevant visual bottlenecks solely from action supervision. Seeker employs frozen DINOv3 features as a foundation, over which it constructs an iterative, query-based attention mechanism conditioned on task and robot state. This mechanism dynamically refines its focus through multiple attention heads with gating, enabling the model to adaptively shift attention as the task unfolds. The core innovation lies in training this attention module using a diffusion-based action prediction head, which provides dense supervision without requiring explicit spatial labels.
Experimental results demonstrate that Seeker significantly outperforms traditional ROI methods and external state-of-the-art baselines. In simulation, it boosts success rates from 42.6% to 62.6%, and in real-world robotic tasks, success rates increase from 48.3% to 76.7%. Moreover, the learned ROI is versatile, supporting background augmentation and point cloud filtering, which further enhances robustness against environmental variations such as lighting and background shifts.
This work's significance lies in its ability to autonomously learn control-dependent visual regions, reducing reliance on costly annotations and enabling more scalable, data-efficient robot learning. Its dynamic attention mechanism allows for continuous adaptation, making it suitable for complex, long-horizon tasks. The approach opens new pathways for integrating visual perception and control, with broad implications for industrial automation, service robotics, and autonomous systems.
Despite these advances, challenges remain. The method's performance under severe occlusion or in highly unstructured environments needs further investigation. Computational costs associated with iterative attention may limit real-time applications, and the reliance on large amounts of demonstration data poses scalability questions. Future work aims to incorporate multi-modal sensing, optimize efficiency, and extend the framework to multi-agent and highly dynamic scenarios, pushing the boundaries of autonomous perception and manipulation.
Deep Dive
Abstract
Visual bottlenecks that focus policy inputs on regions of interest (ROIs) can improve data-efficient visuomotor learning by separating where to look from how to act. Many ROI interfaces rely on external spatial labels, such as gaze, object classes, or affordance annotations. Label-free alternatives often derive crops from trajectories by detecting gripper or motion events and centering a fixed crop at the projected end-effector. Such action-derived crops are useful spatial priors that require no additional labels, but they encode fixed choices about event timing, proxy points, and crop scale. When the visual evidence needed for control lies away from the end-effector or changes continuously with task progress, these crops can become misaligned. We propose Seeker, a task- and state-conditioned readout that learns attention from action. Starting from frozen DINOv3 features, Seeker iteratively updates a query with gathered visual evidence, producing progression-aware ROIs solely from action supervision. The learned ROI serves as a spatial interface for RGB cropping, mask-guided background augmentation, and point-cloud filtering. In simulation and the real world, Seeker improves data efficiency and robustness over no-crop, augmentation, and action-derived crop baselines. On real robots, Seeker raises average in-domain success from the best baseline's 48.3% to 76.7% and success under lighting/background shifts from 20.0% to 60.0%.
References (20)
Open-World Object Manipulation using Pre-trained Vision-Language Models
Austin Stone, Ted Xiao, Yao Lu et al.
RVT-2: Learning Precise Manipulation from Few Demonstrations
Ankit Goyal, Valts Blukis, Jie Xu et al.
ATK: Automatic Task-driven Keypoint Selection for Robust Policy Learning
Yunchu Zhang, Shubham Mittal, Zhengyu Zhang et al.
Learning Robotic Manipulation Policies from Point Clouds with Conditional Flow Matching
Eugenio Chisari, Nick Heppert, Max Argus et al.
R+X: Retrieval and Execution from Everyday Human Videos
Georgios Papagiannis, Norman Di Palo, Pietro Vitiello et al.
Enhancing Reusability of Learned Skills for Robot Manipulation via Gaze Information and Motion Bottlenecks
Ryo Takizawa, Izumi Karino, Koki Nakagawa et al.
Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection
Shilong Liu, Zhaoyang Zeng, Tianhe Ren et al.
Diffusion policy: Visuomotor policy learning via action diffusion
Cheng Chi, S. Feng, Yilun Du et al.
Equivariant Diffusion Policy
Di Wang, Stephen M. Hart, David Surovik et al.
Flow Matching for Generative Modeling
Y. Lipman, Ricky T. Q. Chen, Heli Ben-Hamu et al.
MimicPlay: Long-Horizon Imitation Learning by Watching Human Play
Chen Wang, Linxi (Jim) Fan, Jiankai Sun et al.
ICRT: In-Context Imitation Learning via Next-Token Prediction
Letian Fu, Huang Huang, Gaurav Datta et al.
IMLE Policy: Fast and Sample Efficient Visuomotor Policy Learning via Implicit Maximum Likelihood Estimation
Krishan Rana, Robert Lee, David Pershouse et al.
RVT: Robotic View Transformer for 3D Object Manipulation
Ankit Goyal, Jie Xu, Yijie Guo et al.
Policy Adaptation via Language Optimization: Decomposing Tasks for Few-Shot Imitation
Vivek Myers, B. Zheng, Oier Mees et al.
Revisiting Feature Prediction for Learning Visual Representations from Video
Adrien Bardes, Q. Garrido, Jean Ponce et al.
Ο0: A Vision-Language-Action Flow Model for General Robot Control
Kevin Black, Noah Brown, Danny Driess et al.
Learning Transferable Visual Models From Natural Language Supervision
Alec Radford, Jong Wook Kim, Chris Hallacy et al.
Generalizable Coarse-to-Fine Robot Manipulation via Language-Aligned 3D Keypoints
Jianshu Hu, Lidian Wang, Shujia Li et al.
Q-attention: Enabling Efficient Learning for Vision-based Robotic Manipulation
Stephen James, A. Davison