HumanTracker: Towards Comprehensive and Human-Aligned Motion Tracking Benchmark

The evolution of humanoid motion control has transitioned from rule-based methods to data-driven deep learning approaches. Early techniques relied on predefined motion templates and inverse kinematics, which limited adaptability to complex tasks. The advent of large-scale datasets like AMASS, PHUMA, and Motion-X provided rich resources for training neural networks capable of generating and tracking realistic human motions. Deep mimicry methods such as DeepMimic, PHC, and UHM further advanced the field by enabling robots to imitate human-like movements through reinforcement learning and imitation learning frameworks. Despite these developments, evaluation metrics remained largely geometric, focusing on joint position errors, which do not fully capture the perceptual qualities of motion. As applications expanded into virtual reality, animation, and teleoperation, the need for perceptually aligned evaluation metrics became evident. Prior datasets, while large, lacked detailed categorization and did not explicitly address the physical artifacts that influence human perception of motion quality. This gap motivated the development of more comprehensive benchmarks and metrics that consider both physical stability and perceptual naturalness.

Traditional evaluation metrics such as MPJPE and success rate primarily measure geometric accuracy, neglecting the physical artifacts and stability issues that significantly impact perceived motion quality. These metrics often produce misleadingly positive evaluations for motions that, while geometrically accurate, exhibit foot sliding, unstable support, or unnatural contact transitions. Moreover, existing datasets are limited in diversity, often focusing on simple locomotion or short sequences, which fail to challenge current algorithms in complex, contact-rich scenarios. This disconnect hampers progress in developing humanoid controllers that produce truly human-like movements. The core challenge lies in designing an evaluation framework that aligns closely with human perception, accurately reflecting the stability, contact realism, and overall naturalness of motion, especially in long-horizon and dynamic tasks.

The paper introduces several key innovations: 1) Construction of a large, categorized motion dataset with detailed labels and high-fidelity optical capture data, covering four distinct motion families; 2) Development of HumanScore, a preference-based metric trained on human judgments, utilizing a Transformer architecture to process multi-modal features; 3) Integration of a pairwise preference learning framework with Bradley–Terry loss, enabling the model to detect subtle artifacts like foot sliding and unstable support; 4) Standardized evaluation protocol that combines traditional metrics with HumanScore, facilitating comprehensive and human-aligned assessment of humanoid motion tracking. These innovations collectively enable a shift from purely geometric evaluation to perceptually meaningful metrics, fostering more natural and stable robot behaviors.

• �� Data collection: 24 professional performers record diverse motions using optical motion capture systems, covering daily, dynamic, interactive, and ground-level activities; • Data preprocessing: segments with artifacts are removed, and each clip is labeled with category and textual descriptions; • Feature extraction: multi-modal features including joint positions, velocities, contact states, and root motion are derived for each frame; • Preference data collection: experts compare synchronized motion pairs, indicating which is better in terms of stability, contact, and naturalness; • Model training: a Transformer encoder processes sequences of features, with the Bradley–Terry loss optimizing the model to predict human preferences; • Evaluation: all trackers are tested under a standardized pipeline, with metrics including success rate, MPJPE, and HumanScore, computed over fixed-length windows.

The experimental setup involves evaluating four state-of-the-art humanoid trackers—GMT, TWIST2, SONIC, and Humanoid-GPT—on the HumanTracker benchmark. The dataset includes approximately 153 hours of categorized motion sequences, split into training and testing sets with balanced distribution. Each tracker generates motion rollouts in a simulated environment, with reference motions retargeted via GMR. The evaluation metrics include traditional success rate (Succ), MPJPE, and the proposed HumanScore, which reflects perceptual quality. Preference data for training HumanScore is collected from expert annotations on paired motion segments, ensuring alignment with human judgments. Ablation studies examine the impact of contact features and temporal context on model performance. The experiments demonstrate that HumanScore correlates strongly with human preferences and reveals failure modes missed by conventional metrics, especially in complex, contact-rich motions.

Results show that HumanScore achieves over 90% agreement with human preferences across all motion categories, outperforming MPJPE and success rate in capturing perceptual quality. Humanoid-GPT consistently ranks highest in success rate (94.4%) and preference scores (54.7), indicating superior stability and naturalness. In high-dynamic and ground-level motions, traditional metrics often overestimate performance, whereas HumanScore accurately reflects stability issues like foot sliding and unstable support. Ablation experiments reveal that incorporating contact features and longer temporal context improves preference prediction accuracy by approximately 10%, emphasizing the importance of physical artifacts in perceptual evaluation. The dataset’s categorization allows detailed failure analysis, guiding future improvements in humanoid control algorithms.

This evaluation framework can be directly applied to improve humanoid robot controllers, virtual avatar animation, and teleoperation systems by providing a perceptually aligned performance measure. It enables researchers to diagnose specific failure modes, such as contact instability or support loss, and optimize controllers accordingly. The dataset and metrics also facilitate benchmarking in virtual reality and gaming industries, where motion realism is critical. Long-term, integrating HumanScore into reinforcement learning loops could lead to autonomous systems that learn to produce motions more aligned with human preferences, enhancing trust and interaction quality in real-world applications.

The current model relies on optical motion capture data, which may not fully generalize to real-world sensors or noisy environments. The preference annotations are subjective and may vary across annotators, potentially affecting model robustness. The approach primarily addresses static and semi-dynamic motions; extreme or highly unpredictable motions remain challenging. Additionally, computational costs for real-time evaluation need further optimization. Future work should focus on expanding the dataset with more diverse and challenging motions, incorporating multi-sensor data, and developing adaptive models capable of online learning and real-time inference.