Duet: Dual-Robot Understanding via Efficient Teaching
DUET leverages human demonstration pretraining with Action Chunking Transformer to enhance heterogeneous dual-robot collaboration, achieving 20% higher success rates.
Key Findings
Methodology
This paper introduces a unified visuomotor policy framework based on Action Chunking Transformer (ACT), trained via a two-stage process. First, it pretrains on human-human collaborative demonstrations collected through a synchronized VR teleoperation system, utilizing SAM 3D Body for pose extraction and mapping to a shared pose space. Second, it fine-tunes on limited robot trajectories. The system integrates multimodal data—RGB streams, pose keypoints, and robot states—to learn coordinated actions across heterogeneous platforms (Unitree G1 and Dexmate Vega1). The approach emphasizes efficient data collection, leveraging human priors to bootstrap learning, significantly reducing the need for extensive robot data.
Key Results
- Pretrained models achieved success rates exceeding 75% across four tasks, a 20% improvement over models trained solely on robot data. Data collection efficiency increased by 5.4 times, drastically lowering costs. The policies demonstrated smoother, more coordinated motions, with success rates of 80% in Trash Collection, 78% in Box Organization, and over 70% in Board Tilting and Doll Passing. Fine-tuning with limited robot data maintained or improved performance, confirming the effectiveness of human priors.
- Ablation studies showed that models without human pretraining scored around 40-60%, while the proposed approach consistently outperformed baselines trained on larger robot datasets, confirming the value of human demonstrations in reducing data requirements and enhancing generalization.
Significance
This work addresses the critical bottleneck of data scarcity in multi-robot visuomotor learning. By integrating human demonstrations into a scalable pretraining pipeline, it enables rapid, cost-effective training of complex heterogeneous systems. The approach bridges the gap between human dexterity and robotic execution, paving the way for scalable, autonomous multi-robot systems capable of complex, contact-rich tasks in unstructured environments. Its success demonstrates a promising direction for deploying robots in real-world scenarios with minimal data collection overhead.
Technical Contribution
The key technical innovation is the integration of SAM-based human pose extraction with a shared pose space, enabling cross-embodiment action understanding. The use of Action Chunking Transformer allows modeling long-horizon, contact-rich behaviors by decomposing actions into chunks, improving coordination. The dual-stage training paradigm—pretraining on human data, then fine-tuning on robot data—significantly reduces data requirements. The synchronized VR teleoperation system enhances data collection efficiency, while the multimodal fusion within ACT improves robustness and generalization across platforms.
Novelty
This is the first work to incorporate large-scale human collaborative demonstrations into a heterogeneous dual-robot visuomotor policy training pipeline. It uniquely combines SAM pose extraction, cross-embodiment pose mapping, and transformer-based action chunking to address multi-robot coordination challenges. Unlike prior works focusing on single-robot imitation or classical control, this approach leverages human priors to bootstrap learning, reducing data needs and improving transferability across platforms.
Limitations
- The approach relies heavily on accurate pose extraction; in complex or occluded environments, pose estimation errors may degrade performance.
- Currently validated only on two heterogeneous robots; scalability to larger teams or different platforms requires further adaptation.
- Training still demands significant computational resources, especially for multimodal data processing and transformer training, which may limit real-time deployment.
Future Work
Future research will explore extending the framework to multi-robot teams beyond two units, incorporating self-supervised learning to further reduce supervision needs, and improving robustness in dynamic, cluttered environments. Additionally, integrating reinforcement learning could enable adaptive behaviors, and expanding platform compatibility will facilitate broader deployment in industrial and service domains.
AI Executive Summary
The rapid advancement of robotics has brought about increasingly complex multi-robot systems capable of performing tasks beyond the reach of single units. However, training these heterogeneous systems remains a significant challenge due to the high costs and difficulty of data collection. Traditional approaches rely heavily on explicit modeling and extensive robot-specific demonstrations, which are often impractical at scale.
This paper introduces DUET, a novel framework that leverages human demonstrations to bootstrap the learning process of dual-robot systems. By employing a synchronized VR teleoperation system, two human operators control heterogeneous robots—Unitree G1 and Dexmate Vega1—simultaneously, capturing high-fidelity collaborative behaviors efficiently. The key innovation lies in the use of SAM 3D Body for extracting human pose keypoints, which are then mapped into a shared, embodiment-agnostic pose space. This enables the model to learn cross-embodiment action representations.
At the core of DUET is an Action Chunking Transformer (ACT), a deep neural network architecture that models long-horizon, contact-rich behaviors by decomposing actions into manageable chunks. The training process is staged: first, the model is pretrained on large-scale human-human demonstration data, then fine-tuned on a limited set of robot trajectories. This approach significantly reduces data requirements, achieving success rates over 75% across four contact-rich tasks, including Trash Collection, Box Organization, Board Tilting, and Doll Passing.
Experimental results demonstrate that models pretrained on human data outperform or match those trained solely on robot data, while requiring 5.4 times less data collection effort. The policies exhibit smoother, more coordinated motions, and generalize well across platforms. These findings suggest that human priors can effectively bridge the gap between human dexterity and robotic control, enabling scalable, contact-rich multi-robot collaboration.
Overall, DUET offers a scalable, efficient pathway toward deploying autonomous multi-robot systems in real-world environments. Its combination of human demonstration pretraining, multimodal data fusion, and transformer-based action modeling marks a significant step forward in end-to-end robot learning. Future work will expand to larger teams, incorporate reinforcement learning, and enhance robustness, ultimately bringing intelligent, cooperative robots closer to everyday applications.
Deep Dive
Abstract
Dual-robot collaboration enables tasks that exceed the reach and payload of a single robot, such as collaboratively transporting objects across environments and executing coordinated handovers. Data acquisition is the primary bottleneck for training these systems. To this end, we introduce DUET, a dual-robot learning framework for mobile manipulation. For efficient data collection, we create a unified dual-embodiment synchronized VR-based teleoperation system for in-domain heterogeneous robot data collection. We further develop a complementary tracking pipeline that records human-human coordination and collaborative mobile manipulation priors. To allow efficient learning, we introduce an Action Chunking Transformer based architecture that first pretrains collaborative policies on efficient human-human demonstrations, before finetuning them on a minimal set of real-robot teleoperation trajectories. We develop a benchmark of four collaborative tasks to evaluate our framework using a Unitree G1 humanoid and a Dexmate Vega1 mobile manipulator. The results demonstrate that harnessing human priors not only yields superior task performance compared to baselines trained only on robot data, but also reduces the total human effort required for data collection. Our human data collection pipeline achieves 5.4x acceleration on average from teleoperation, but we perform equally or better than robot-only data trained policies across all tasks. Our project page is available at https://zhaoy37.github.io/Duet/.