Open-TeleVision: Teleoperation with Immersive Active Visual Feedback
Open-TeleVision integrates active stereo vision and immersive VR to enable high-precision remote teleoperation, improving data collection for imitation learning.
Key Findings
Methodology
The system combines VR headsets with active stereo RGB cameras mounted on robots, enabling operators to perceive the environment stereoscopically through head movements. It employs inverse kinematics (IK) and motion retargeting (dex-retargeting) to map human gestures onto multi-finger or gripper robots. A Transformer encoder-decoder architecture with action chunking facilitates imitation learning, trained on four long-horizon tasks: can sorting, insertion, folding, and unloading, across two humanoid robots. The system supports remote operation via internet with real-time stereo streaming, significantly enhancing spatial awareness and control accuracy.
Key Results
- Models utilizing DinoV2 backbone achieved success rates of 92%, 87%, 90%, and 100% across four tasks on H1 and GR-1 robots, outperforming ResNet18 and mono-input variants. Remote teleoperation enabled cross-continental collaboration, improving data quality and training efficiency.
- Active stereo perception and streaming stereo video substantially improved intuitive control and task success. Imitation policies demonstrated high robustness, with success rates reaching 100% and average task completion times reduced by approximately 30%.
- Ablation studies confirmed that stereo input and active sensing significantly boost generalization and precision, especially in fine manipulation and long-horizon tasks.
Significance
This work addresses core limitations of traditional teleoperation systems by integrating immersive stereo vision, enabling operators to perceive and manipulate environments remotely with high fidelity. It provides a scalable framework for collecting high-quality demonstration data, advancing robot learning, and enabling collaborative multi-site operations. The open-source nature accelerates research and industrial adoption, promising transformative impacts in manufacturing, healthcare, and hazardous environment applications. The ability to perform long-distance, high-precision tasks remotely opens new horizons for robotics deployment in real-world scenarios.
Technical Contribution
The key innovations include the integration of active stereo vision with VR for immersive perception, combined with a Transformer-based imitation learning framework that handles long sequences via action chunking. The system employs inverse kinematics and motion retargeting for dexterous control of multi-finger and gripper hands, supporting complex soft and rigid object manipulation. It also enables robust remote operation over the internet, overcoming occlusion and spatial perception challenges typical of static camera setups. These contributions collectively push the boundary of high-DOF teleoperation and learning-based robot control.
Novelty
This is the first integration of active stereo vision with immersive VR for high-DOF remote teleoperation, enabling operators to perceive and control environments with natural head and hand movements. Unlike prior work relying on static cameras or limited gesture tracking, this system dynamically adjusts viewpoints and streams stereoscopic video, significantly enhancing spatial understanding. The use of a Transformer architecture with action chunking for long-horizon imitation learning in remote settings is also novel, addressing key challenges in scalable robot training.
Limitations
- System relies on high-performance GPUs and precise mechanical structures, which may limit widespread adoption. Cost and complexity remain barriers.
- Environmental factors like occlusion, lighting, and network latency can impair perception and control accuracy, especially in cluttered or dynamic scenes.
- Long-distance communication introduces delays that may affect real-time responsiveness, requiring further optimization.
Future Work
Future efforts will focus on multi-camera and multi-modal sensing to improve robustness, integrating autonomous perception modules for target recognition. Reducing hardware costs and computational demands will be prioritized to facilitate broader deployment. Exploring reinforcement learning for autonomous adaptation and expanding to multi-robot coordination are promising directions. Additionally, enhancing system resilience to environmental variability and network conditions will be critical for real-world applications.
AI Executive Summary
Remote teleoperation has long been hampered by limited spatial perception and control fidelity, especially for complex, long-horizon tasks. Traditional systems rely on static cameras or simplified gesture tracking, which restrict operator immersion and precision. To overcome these challenges, this work introduces Open-TeleVision, a novel framework that combines immersive virtual reality with active stereo vision. By integrating head-controlled stereo cameras and a Transformer-based imitation learning architecture, the system enables operators to perceive their environment stereoscopically and manipulate robots with high degrees of freedom.
The core innovation lies in active perception: the robot’s stereo camera dynamically follows the operator’s head movements, providing ego-centric, real-time 3D views. This enhances spatial awareness, allowing operators to focus on task-relevant regions intuitively. Simultaneously, the system maps human gestures to robot manipulators via inverse kinematics and motion retargeting, supporting dexterous multi-finger and gripper control. The Transformer encoder-decoder, with action chunking, effectively models long sequences, enabling the robot to perform complex tasks such as can sorting, insertion, towel folding, and tube unloading with success rates exceeding 90%.
Experimental results demonstrate that models trained with this setup outperform baselines based on ResNet18 and mono-input configurations, especially in precision-critical tasks. The system’s ability to facilitate remote, multi-site collaboration was validated through coast-to-coast teleoperation, achieving real-time control at 60Hz over the internet. The integration of active stereo sensing and immersive feedback significantly improves task success, generalization, and user experience.
This approach paves the way for scalable, high-fidelity remote robot operation in industrial, medical, and hazardous environments. While hardware costs and environmental factors pose challenges, ongoing work aims to enhance robustness, reduce costs, and expand autonomous capabilities. Overall, Open-TeleVision marks a substantial step toward intelligent, immersive teleoperation, transforming how humans interact with robots across distances.
Deep Dive
Abstract
Teleoperation serves as a powerful method for collecting on-robot data essential for robot learning from demonstrations. The intuitiveness and ease of use of the teleoperation system are crucial for ensuring high-quality, diverse, and scalable data. To achieve this, we propose an immersive teleoperation system Open-TeleVision that allows operators to actively perceive the robot's surroundings in a stereoscopic manner. Additionally, the system mirrors the operator's arm and hand movements on the robot, creating an immersive experience as if the operator's mind is transmitted to a robot embodiment. We validate the effectiveness of our system by collecting data and training imitation learning policies on four long-horizon, precise tasks (Can Sorting, Can Insertion, Folding, and Unloading) for 2 different humanoid robots and deploy them in the real world. The system is open-sourced at: https://robot-tv.github.io/