ARMimic: Learning Robotic Manipulation from Passive Human Demonstrations in Augmented Reality
ARMimic uses XR headsets for passive demonstrations, combining hand tracking and virtual robots to improve data collection efficiency and generalization.
Key Findings
Methodology
ARMimic integrates native XR hand tracking, AR robot overlays, and real-time depth sensing to create a hardware-minimal passive data collection system. It maps human hand poses to virtual robot configurations via inverse kinematics, ensuring collision-free, kinematically feasible demonstrations. The system employs a unified imitation learning pipeline using Transformer-based Action Chunking with Transformers (ACT), fusing visual and proprioceptive data to learn manipulation policies. Cross-modal data alignment is achieved through segmentation models like SAM and hand segmentation, enabling interchangeable human and robot trajectories for cross-embodiment generalization.
Key Results
- In long-horizon bowl stacking, ARMimic reduces demonstration time by 50% compared to teleoperation and improves task success rate by 11%, outperforming the baseline ACT trained on teleoperated data.
- On pick-and-place tasks, success rate reaches 75%, 10% higher than ACT; average task completion time drops from 47.7s to 30.8s.
- Scaling data size enhances performance; with only 50 human demonstrations, success rate exceeds previous results by 10%, and combining human and robot data further boosts success by 17%.
Significance
This work addresses the hardware and calibration bottlenecks of traditional demonstration methods, enabling scalable, safe, and natural data collection in real-world environments. By leveraging XR devices for passive, high-fidelity demonstrations, it paves the way for large-scale robot skill learning, reducing costs and increasing accessibility, with broad implications for industry and research.
Technical Contribution
ARMimic introduces a low-hardware, portable framework combining native hand tracking, virtual embodiment, and real-time collision detection. It employs cross-modal data alignment and Transformer-based policy learning, facilitating cross-embodiment generalization. The system’s design emphasizes safety, simplicity, and robustness, enabling scalable passive demonstration collection without external motion capture or complex calibration.
Novelty
This is the first framework to utilize XR headsets for passive demonstration collection with virtual robot overlays, eliminating the need for external hardware. Its innovative integration of real-time collision handling, virtual embodiment, and cross-modal alignment distinguishes it from prior work relying on complex calibration or external sensors, offering a practical, scalable solution.
Limitations
- Current system mainly supports planar or limited 3D environments; complex spatial scenarios require further development.
- Discrepancies between virtual and real robot motions may affect high-precision tasks, especially in dynamic or cluttered scenes.
- User operation relies on intuitive gestures, which may induce fatigue or errors over long sessions; robustness needs enhancement.
Future Work
Future efforts include expanding multi-robot adaptability, improving collision detection algorithms, and integrating reinforcement learning for autonomous policy refinement. Additionally, exploring more complex environments and longer-horizon tasks will be key to broadening applicability.
AI Executive Summary
The development of autonomous manipulation skills in robots has long been hindered by the high costs and complexity of demonstration data collection. Traditional methods like teleoperation or kinesthetic teaching require specialized hardware, calibration, and often disrupt workflows. Recent advances in augmented reality (AR) devices have opened new avenues for passive, egocentric demonstration collection, but existing solutions still depend on external sensors or complex setups. ARMimic presents a novel framework that leverages consumer-grade XR headsets, such as Meta Quest 3, combined with a stationary RGB-D camera, to enable safe, scalable, and hardware-light data collection.
This system employs native hand tracking to infer human gestures, which are mapped onto a virtual robot through inverse kinematics, creating a virtual overlay aligned with the user’s movements. Real-time collision detection ensures demonstrations are feasible and safe, even in cluttered environments. The core of ARMimic is a unified imitation learning pipeline based on Transformer architectures like ACT, which processes visual and proprioceptive inputs to learn manipulation policies. The approach treats human and robot trajectories as interchangeable, fostering cross-embodiment generalization.
Experimental results demonstrate that ARMimic significantly reduces demonstration time—by 50% compared to teleoperation—and improves task success rates by 11% over the baseline ACT trained on teleoperated data. In complex tasks such as long-horizon bowl stacking, the system maintains high performance and robustness, even with limited demonstrations. As dataset size grows, policy performance continues to improve, highlighting the scalability of the approach.
The broader impact of ARMimic lies in its practical, low-cost, and safe data collection method, which can be deployed in diverse real-world settings without disrupting existing workflows. It opens new pathways for large-scale, generalizable robot learning, reducing reliance on expensive hardware and complex calibration. Future work will focus on extending multi-robot capabilities, enhancing environmental robustness, and integrating reinforcement learning techniques to enable autonomous policy refinement, ultimately accelerating the deployment of autonomous manipulation systems.
Deep Dive
Abstract
Imitation learning is a powerful paradigm for robot skill acquisition, yet conventional demonstration methods--such as kinesthetic teaching and teleoperation--are cumbersome, hardware-heavy, and disruptive to workflows. Recently, passive observation using extended reality (XR) headsets has shown promise for egocentric demonstration collection, yet current approaches require additional hardware, complex calibration, or constrained recording conditions that limit scalability and usability. We present ARMimic, a novel framework that overcomes these limitations with a lightweight and hardware-minimal setup for scalable, robot-free data collection using only a consumer XR headset and a stationary workplace camera. ARMimic integrates egocentric hand tracking, augmented reality (AR) robot overlays, and real-time depth sensing to ensure collision-aware, kinematically feasible demonstrations. A unified imitation learning pipeline is at the core of our method, treating both human and virtual robot trajectories as interchangeable, which enables policies that generalize across different embodiments and environments. We validate ARMimic on two manipulation tasks, including challenging long-horizon bowl stacking. In our experiments, ARMimic reduces demonstration time by 50% compared to teleoperation and improves task success by 11% over ACT, a state-of-the-art baseline trained on teleoperated data. Our results demonstrate that ARMimic enables safe, seamless, and in-the-wild data collection, offering great potential for scalable robot learning in diverse real-world settings.