Learning Human-to-Humanoid Real-Time Whole-Body Teleoperation
RL-based H2O enables real-time humanoid teleoperation via RGB camera, achieving dynamic motion imitation with success rate over 72.5%.
Key Findings
Methodology
This work introduces a goal-conditioned RL framework combining large-scale human motion datasets with a ‘sim-to-data’ filtering process. Using inverse kinematics, human motions are retargeted to humanoid-compatible sequences. A privileged motion imitator trained in simulation filters out infeasible motions, creating a high-quality dataset. The RL policy, trained with domain randomization, learns to track diverse motions using RGB keypoints, enabling zero-shot transfer to real robots. The system integrates a multi-layer reward design and a goal space based on key joint positions, ensuring robustness and generalization across tasks.
Key Results
- In simulation, H2O successfully mimics complex actions like walking, jumping, and kicking, with success rates exceeding 72.5%. The root-relative MPJPE (Empjpe) is about 91.7mm, and global MPJPE (Eg-mpjpe) is 166.7mm, demonstrating precise motion tracking. The system generalizes well to real-world scenarios, performing tasks such as pushing, waving, and boxing with minimal error.
- Filtering via the ‘sim-to-data’ process significantly improves performance, with success rates rising from 53.2% on unfiltered data to over 72.5% on filtered datasets. Larger training datasets further enhance accuracy, confirming the importance of data quality and quantity.
- Real-world tests show the robot can perform complex motions in real-time using only RGB input, with stable execution in diverse environments, validating the approach’s robustness and practical viability.
Significance
This research pioneers the integration of deep RL, motion retargeting, and vision-based perception for full-body humanoid teleoperation. It reduces hardware complexity by eliminating external sensors, enabling scalable data collection and autonomous learning. The ability to perform dynamic, multi-task motions in real-world settings marks a significant step toward intelligent, adaptable robots for household, medical, and rescue applications. The approach addresses longstanding challenges in motion transfer, perception, and control, opening new avenues for human-robot collaboration and autonomous operation.
Technical Contribution
The paper introduces a comprehensive pipeline combining inverse kinematics, privileged RL training, and domain randomization to achieve robust sim-to-real transfer. It innovates with a goal-conditioned RL policy that uses RGB keypoints as input, along with a multi-layer reward structure to ensure motion fidelity. The ‘sim-to-data’ filtering mechanism, based on a privileged imitator, enhances data quality, enabling scalable training. These contributions collectively push forward the state-of-the-art in vision-based humanoid control, offering a practical framework for real-time, dynamic motion imitation.
Novelty
This is the first work to realize real-time, learning-based whole-body humanoid teleoperation solely from RGB camera input. It uniquely combines large-scale motion retargeting, a ‘sim-to-data’ filtering process, and goal-conditioned RL, surpassing prior efforts limited to upper-body or offline imitation. The end-to-end pipeline allows dynamic, multi-task motions to be transferred seamlessly from simulation to real hardware, representing a breakthrough in autonomous humanoid control.
Limitations
- The system relies heavily on accurate 3D pose estimation; errors in keypoint detection can degrade motion fidelity, especially in cluttered or poorly lit environments.
- Real-time stability in highly dynamic or complex interactions remains challenging, requiring further robustness enhancements.
- Training involves significant computational resources due to large-scale simulation and domain randomization, which may limit deployment in resource-constrained settings.
Future Work
Future directions include integrating multi-modal sensors like LiDAR and IMUs to improve perception accuracy, developing online learning mechanisms for continual adaptation, and optimizing model architectures for efficiency. Extending the framework to more complex tasks, such as collaborative manipulation and high-risk rescue operations, will further broaden its impact.
AI Executive Summary
This paper introduces a groundbreaking system, Human to Humanoid (H2O), which leverages deep reinforcement learning to enable real-time, vision-based teleoperation of full-sized humanoid robots. Traditional control methods often depend on external sensors and simplified models, limiting their ability to perform dynamic, natural motions. In contrast, H2O combines a large-scale human motion dataset with a ‘sim-to-data’ filtering process, ensuring only feasible motions are used for training. Using inverse kinematics, human actions are retargeted to humanoid-compatible sequences, which are then refined through simulation-based filtering with a privileged motion imitator.
The core of the system is a goal-conditioned RL policy trained in simulation with extensive domain randomization, making it robust enough to transfer directly to real hardware without additional fine-tuning. The input to the policy is derived solely from RGB keypoints detected by a standard camera, simplifying hardware requirements and enabling scalable data collection. The policy learns to imitate a wide range of motions, from walking and jumping to complex interactions like boxing and pushing objects.
Experimental results demonstrate that the robot can perform these motions in real-world scenarios with high accuracy, achieving success rates over 72.5% and low positional errors. The approach significantly advances the field by removing reliance on external sensors, reducing hardware complexity, and enabling large-scale data-driven learning. It opens new possibilities for autonomous humanoid robots in household, medical, and rescue environments, where dynamic, adaptive motion is essential. Future work will focus on multi-modal perception, online adaptation, and expanding task complexity, aiming for even more versatile and resilient robotic systems.
Deep Dive
Abstract
We present Human to Humanoid (H2O), a reinforcement learning (RL) based framework that enables real-time whole-body teleoperation of a full-sized humanoid robot with only an RGB camera. To create a large-scale retargeted motion dataset of human movements for humanoid robots, we propose a scalable "sim-to-data" process to filter and pick feasible motions using a privileged motion imitator. Afterwards, we train a robust real-time humanoid motion imitator in simulation using these refined motions and transfer it to the real humanoid robot in a zero-shot manner. We successfully achieve teleoperation of dynamic whole-body motions in real-world scenarios, including walking, back jumping, kicking, turning, waving, pushing, boxing, etc. To the best of our knowledge, this is the first demonstration to achieve learning-based real-time whole-body humanoid teleoperation.