Make Tracking Easy: Neural Motion Retargeting for Humanoid Whole-body Control

Humanoid robots play an increasingly important role in modern technology, especially in applications requiring complex motor skills, such as medical assistance, entertainment, and manufacturing. Traditionally, researchers have relied on large-scale human motion data, such as video recordings or motion-capture databases, to train robot motor control policies through imitation learning or reinforcement learning. In this process, motion retargeting serves as a critical bridge between human demonstrations and robotic execution. Conventional retargeting methods, including inverse kinematics (IK)-based approaches and differential optimization schemes like GMR, primarily seek optimal joint configurations at the geometric level. However, this conventional decoupled architecture of 'retargeting first, tracking later' suffers from two major bottlenecks: mathematical non-convexity and lack of physical feasibility.

The core problem of motion retargeting is how to accurately transfer human motion data to humanoid robots while accounting for their distinct kinematic structures and physical constraints. Traditional optimization-based methods are inherently non-convex and prone to local optima, leading to physical artifacts like joint jumps and self-penetration. These methods are highly sensitive to initialization and require tedious parameter tuning. Geometric optimization methods lack awareness of physical plausibility and therefore merely propagate these errors mechanically, leading to a classic 'garbage in, garbage out' dilemma.

The core innovation of this paper is the introduction of a Neural Motion Retargeting (NMR) framework that transforms the motion retargeting problem from static optimization to dynamic distribution mapping. NMR utilizes a hierarchical data pipeline called Clustered-Expert Physics Refinement (CEPR), leveraging a Variational Autoencoder (VAE) for motion clustering to group heterogeneous movements into latent motifs. This strategy significantly reduces computational overhead for massively parallel reinforcement learning experts, which project and repair noisy human demonstrations onto the robot's feasible motion manifold. By incorporating physics simulation and reinforcement learning, NMR generates high-fidelity, physically consistent motion data, enhancing the model's generalization ability and physical feasibility.

• �� NMR framework addresses humanoid motion retargeting via dynamic mapping, significantly reducing joint jumps and self-collisions.

• �� Uses a Variational Autoencoder (VAE) for motion clustering to group heterogeneous movements into latent motifs.

• �� Leverages massively parallel reinforcement learning experts to project and repair noisy human demonstrations onto the robot's feasible motion manifold.

• �� Employs a non-autoregressive CNN-Transformer architecture for global temporal context reasoning, suppressing reconstruction noise and bypassing geometric traps.

The experimental design includes testing on the Unitree G1 humanoid across diverse dynamic tasks, such as martial arts and dancing. Baseline methods include GMR and PHUMA. Evaluation metrics include joint jumps, self-collisions, and joint limit violations. With 30,000 physics-validated motion pairs, NMR suppresses physically infeasible components in upstream SMPL-X noise. Experiments also include testing the convergence of downstream whole-body control policies using NMR-generated references.

Experimental results show that NMR eliminates joint jumps and significantly reduces self-collisions across diverse dynamic tasks on the Unitree G1 humanoid. Compared to state-of-the-art baselines, NMR achieves a 54% reduction in self-collisions and reduces joint limit violations to 16.80%. Furthermore, NMR-generated references accelerate the convergence of downstream whole-body control policies, establishing a scalable path for bridging the human-robot embodiment gap.

The NMR framework can be directly applied in fields requiring high-precision motion control, such as medical assistance and complex manufacturing. Its advantages in eliminating joint jumps and self-collisions make it more reliable for real-world applications. Additionally, the high-fidelity motion data generated by NMR can be used to train more complex robot control strategies, improving robot performance in dynamic tasks.

Despite NMR's excellent performance in various tasks, it may still have limitations in handling very complex or irregular motion sequences. Additionally, the requirement for large-scale computational resources and training time may limit NMR's deployment in practical applications. Future research directions include further optimizing the NMR framework to handle more complex dynamic tasks, exploring more efficient training strategies to reduce computational resource requirements, and applying NMR to more diverse robotic platforms.