Real-Time Whole-Body Teleoperation of a Humanoid Robot Using IMU-Based Motion Capture with Sim2Sim and Sim2Real Validation

In recent years, humanoid robots have advanced rapidly, particularly in performing complex operational tasks. However, achieving stable, natural whole-body motion remains a challenge. Traditional methods, such as pre-scripted motion playback and offline trajectory optimization, perform well in controlled environments but often require extensive tuning and data in practical applications. Learning-based controllers, such as reinforcement learning, can achieve robust motion control in some cases but involve complex training processes and high demands for data and computational resources.

Motion capture technology offers a direct and intuitive way to teleoperate humanoid robots by retargeting human natural motion onto robots in real-time. However, due to kinematic discrepancies between humans and robots, inherent high-frequency noise in IMU sensors, and challenges in transferring from simulation to physical hardware, achieving this goal is not straightforward.

This paper proposes an innovative solution by developing a learning-free real-time whole-body teleoperation system using a Virdyn IMU motion capture suit and a Unitree G1 robot. The system operates seamlessly on both simulation and real hardware, demonstrating its potential in complex operational tasks.

Achieving stable, low-latency whole-body teleoperation is an open research challenge in the field of humanoid robots. The main difficulties include: kinematic discrepancies between human and robot structures leading to motion retargeting issues; high-frequency noise inherent in IMU sensors affecting the accuracy of pose estimation; the need for joint-limit-safe motion at robot control rates; and the risk of instability when transitioning from simulation to physical hardware.

These issues pose challenges for existing teleoperation systems in terms of stability and responsiveness, particularly in scenarios requiring rapid response and complex actions. Therefore, developing a system capable of stable, low-latency whole-body teleoperation without relying on learning or offline processing is of significant importance.

The core innovations of this paper include the development of a learning-free real-time whole-body teleoperation system that operates seamlessly on both simulation and real hardware.

• �� Motion Retargeting Algorithm: Computes equivalent angles via geometric projection to preserve intended motion intent while remaining within the robot's physical range. This approach avoids complex learning processes, enabling rapid deployment.

• �� Real-Time Smoothing Filter: Applies a lightweight exponential moving average filter to smooth high-frequency noise in IMU estimates, ensuring stability and responsiveness of motion.

• �� Synchronized Motion Control: Synchronizes upper-body, lower-body, and torso motions within a single retargeting step to maintain the robot's center-of-mass trajectory consistent with the operator's posture. This synchronization ensures overall stability of the robot.

These innovations allow the system to transfer directly from simulation to real hardware without additional domain adaptation or parameter tuning.

The paper presents a lightweight real-time whole-body teleoperation pipeline with the following methodology:

• �� Motion Capture: Uses a Virdyn IMU motion capture suit to record human full-body motion data in real-time. The suit is equipped with inertial sensors distributed across major body segments, estimating segment orientations and joint angles without relying on external cameras or optical markers.

• �� Motion Retargeting: Maps human skeleton joints to their closest functional counterparts on the Unitree G1 kinematic tree. For structural differences that prevent direct correspondence (e.g., human hip complex vs. robot's three-DoF hip joint), computes equivalent angles via geometric projection.

• �� Joint Limit Enforcement: Clips every mapped joint command to the robot's hardware joint limits before transmission. Soft limits are applied to prevent actuator saturation, especially for high-velocity motions.

• �� Real-Time Smoothing: Applies a lightweight exponential moving average filter per joint to smooth high-frequency noise in IMU estimates, ensuring stability and responsiveness.

• �� Synchronization: Synchronizes upper-body, lower-body, and torso motions within a single retargeting step, ensuring the robot's center-of-mass trajectory remains consistent with the operator's posture. The entire pipeline operates in a tight loop with no buffering or batch processing.

The experimental design includes Sim2Sim validation in the MuJoCo physics simulator and Sim2Real validation on the physical Unitree G1 robot.

• �� Datasets: Real-time full-body motion data recorded using the Virdyn IMU motion capture suit.

• �� Baselines: Compared with traditional learning-based whole-body controllers to evaluate the system's performance in terms of stability and responsiveness.

• �� Metrics: Key evaluation metrics include motion stability, responsiveness, and retargeting accuracy.

• �� Hyperparameters: The time constant of the lightweight exponential moving average filter is tuned to balance noise attenuation and motion responsiveness.

• �� Ablation Studies: The system's performance is tested across different motion categories (e.g., walking, sitting, turning) to validate its effectiveness in various scenarios.

Experimental results demonstrate that the system can stably reproduce a wide range of motions, including walking, standing, sitting, turning, bowing, and coordinated gestures.

• �� Sim2Sim validation in the MuJoCo simulator shows that the system produces physically plausible configurations with no joint limit violations, self-collisions, or abrupt discontinuities in joint velocity.

• �� Sim2Real validation on the physical Unitree G1 robot achieves real-time motion reproduction, with no noticeable delay between the operator's motion and the robot's execution, confirming Sim2Real effectiveness.

• �� Experiments show the system can transfer directly from simulation to real hardware without additional domain adaptation or parameter tuning.

The system's application scenarios include:

• �� Remote Operation: Enables stable, low-latency remote operation in scenarios requiring rapid response and complex actions, such as disaster recovery and hazardous environment operations.

• �� Human-Robot Interaction: Provides a more intuitive operation experience in scenarios requiring natural, fluid human-robot interaction, such as entertainment and education.

• �� Industrial Automation: Improves production efficiency and operational safety in industrial scenarios requiring precise control and rapid response, such as assembly lines and quality inspection.

Despite its impressive performance, the system has some limitations.

• �� The system relies on the Unitree G1's onboard servo controllers for low-level stability. In highly dynamic motions, such as rapid direction changes or large-amplitude arm swings, the robot's balance could be improved.

• �� The EMA filter introduces a small, motion-speed-dependent phase lag, potentially affecting fast motion responsiveness. Future work could explore using an adaptive Kalman filter to further reduce this effect.

• �� The current system does not integrate hand manipulation, which could be expanded using finger-level IMU data or vision-based hand tracking.