DUET-DINO: Simultaneous Cross-View World Modeling for Latent Planning in Robot Manipulation
DUET-DINO enables 7-DoF robot planning via cross-view latent world modeling, achieving 92%-60% task success rates.
Key Findings
Methodology
DUET-DINO integrates side- and wrist-camera observations using cross-attention modules for complementary information exchange, enabling 7-DoF action planning with dual-view goal cost optimization.
Key Results
- Achieved 92% success rate on reach tasks, significantly outperforming single-view baselines (79%).
- Achieved 72.5% success rate on angled-reach tasks, reducing final angular error to 22.7°.
- Achieved 60% success rate on grasp-and-lift tasks, outperforming independent dual-view baselines (45%).
Significance
This study addresses the limitations of existing latent world models in fine-grained 7-DoF control, advancing robot manipulation capabilities in complex tasks and paving the way for autonomous operations.
Technical Contribution
Introduced a cross-view latent modeling framework combining DINOv3 encoders and cross-attention modules, enabling dual-view optimization and improving planning accuracy for complex tasks.
Novelty
First to propose a latent world model leveraging both side- and wrist-view information, overcoming single-view limitations in spatial and rotational action prediction.
Limitations
- Generalization to dynamic environments remains unverified.
- High computational cost for training due to large-scale datasets.
- Wrist-camera field-of-view limitations may affect certain tasks.
Future Work
Future work includes extending to dynamic environments and optimizing computational efficiency through lightweight model design.
AI Executive Summary
DUET-DINO is a novel cross-view latent world model designed for 7-DoF robot manipulation planning. Existing latent world models struggle with fine-grained spatial and rotational action predictions, limiting their applicability in complex tasks. DUET-DINO addresses this by jointly modeling side- and wrist-camera observations using cross-attention modules to exchange complementary information, enabling accurate planning across the full 7-DoF action space.
Experimental results demonstrate DUET-DINO's superiority, achieving 92%, 72.5%, and 60% success rates on reach, angled-reach, and grasp-and-lift tasks, respectively. Notably, it reduces final angular error to 22.7° in orientation-intensive tasks, outperforming single-view and independent dual-view baselines.
While DUET-DINO excels in static environments, its performance in dynamic scenarios remains untested. Additionally, the high computational cost of training poses challenges. Future work aims to enhance its adaptability to dynamic environments and reduce computational demands, broadening its potential applications in robotics and automation.
Deep Analysis
Background
Latent world models have gained traction in robotics for goal-directed planning by predicting future latent states. However, most models rely on single-view observations, limiting their ability to capture complex spatial and rotational dynamics, especially in 7-DoF control tasks.
Core Problem
Current latent world models lack precision in fine-grained 7-DoF control, particularly in predicting rotational and gripper-state dynamics, hindering their application in complex manipulation tasks.
Innovation
Key innovations of DUET-DINO include:
- �� Cross-view latent modeling: Combines side- and wrist-camera information using cross-attention for complementary information exchange.
- �� Dual-view goal cost optimization: Jointly optimizes dual-view latent rollouts to enhance 7-DoF planning.
- �� DINOv3 encoder: Leverages dense self-supervised features for capturing fine-grained dynamics.
Methodology
DUET-DINO implementation involves:
- �� Encoding side- and wrist-view images into latent representations using DINOv3 encoders.
- �� Employing cross-attention modules to exchange latent information between views.
- �� Using view-specific predictor heads to forecast future latent states.
- �� Optimizing dual-view goal cost via CEM to generate 7-DoF action sequences.
Experiments
Experiments were conducted on DROID and RoboArena datasets, covering diverse tasks like reach, angled-reach, and grasp-and-lift. Metrics included success rate, final position error, and angular error.
Results
DUET-DINO achieved 92% success in reach tasks, 72.5% in angled-reach, and 60% in grasp-and-lift, significantly outperforming single-view and independent dual-view baselines.
Applications
DUET-DINO is suitable for industrial assembly and warehouse logistics, particularly in tasks requiring precise spatial and rotational control.
Limitations & Outlook
The model's performance in dynamic environments is untested, training is computationally expensive, and wrist-camera field-of-view limitations may affect performance in specific tasks.
Plain Language Accessible to non-experts
Imagine you're cooking in a kitchen. The side camera is like your eyes, giving you a wide view of the entire kitchen, while the wrist camera is like your hand, focusing on chopping or stirring. DUET-DINO acts like a smart assistant that uses both your eyes and hands' perspectives to help you perform each cooking step more accurately.
ELI14 Explained like you're 14
Imagine playing a robot game where the robot has two cameras: one sees the whole room, and the other sees its hand. DUET-DINO is like a super-smart AI in the game that uses both cameras to help the robot make the best moves! It can grab a banana or turn objects perfectly. Cool, right?
Glossary
Latent World Model
A model that predicts future states in a latent space for robot planning.
DUET-DINO uses latent world models for 7-DoF action planning.
Cross-Attention
A mechanism for exchanging information between different latent representations.
DUET-DINO uses cross-attention to combine side- and wrist-view information.
DINOv3
A self-supervised visual encoder that extracts high-quality latent features.
DUET-DINO employs DINOv3 to process side- and wrist-view images.
CEM (Cross-Entropy Method)
An optimization algorithm for generating high-quality action sequences.
DUET-DINO uses CEM to optimize dual-view goal cost.
7-DoF
Seven degrees of freedom for a robot's end-effector, including 3 translations, 3 rotations, and gripper state.
DUET-DINO focuses on planning for 7-DoF actions.
Open Questions Unanswered questions from this research
- 1 How can DUET-DINO generalize to dynamic environments?
- 2 What strategies can reduce the computational cost of training?
- 3 Can this approach be extended to incorporate more camera views?
Applications
Immediate Applications
Industrial Automation
Used in manufacturing for complex assembly tasks, improving efficiency and precision.
Warehouse Logistics
Applicable in sorting and transporting items in warehouses, optimizing operations.
Long-term Vision
Home Robot Assistants
Future applications in home service robots for complex household tasks.
Abstract
Action-conditioned latent world models predict future visual representations, enabling zero-shot goal-conditioned robot planning and control. However, their predictions for fine-grained spatial and rotational actions are unreliable for full 7-DoF end-effector control. To address this gap, we introduce DUET-DINO, a simultaneous cross-view latent world model that jointly learns action-conditioned predictions from static side- and wrist-camera observations through cross-view conditioning. By exploiting complementary global scene and gripper-centric information, DUET-DINO enables latent planning over the full 7-DoF action space. Across spatially diverse reach, orientation-intensive angled-reach, and multi-goal grasp-and-lift tasks, DUET-DINO consistently outperforms single-view and independent dual-view baselines, achieving 92% success on reach, 72.5% on angled-reach, and 60.0% on lift tasks. DUET-DINO is trained from scratch on DROID and RoboArena datasets and generalizes robustly under visual distribution shifts. We further show that while V-JEPA 2 wrist-view predictions underestimate visual dynamics induced by fine-grained actions, DINOv3 predictions better capture action-conditioned scene changes, leading to stronger downstream planning. The code and model checkpoints will be open-sourced. Project page: https://utn-air.github.io/DUET-DINO