ZeroWBC: Learning Natural Whole-Body Humanoid Interaction from Human Egocentric Data
ZeroWBC leverages egocentric videos and vision-language models to generate and execute whole-body humanoid behaviors without teleoperation, achieving diverse scene-aware actions.
Key Findings
Methodology
ZeroWBC employs a two-stage framework: first, a VQ-VAE encodes human motions into discrete tokens; second, a fine-tuned Qwen-VL-3B model autoregressively predicts future motion tokens conditioned on egocentric images and instructions. The tokens are decoded into continuous motions and retargeted onto a humanoid robot. A reinforcement learning-based interactive tracking policy then executes these motions, guided by a novel interaction-oriented reward emphasizing global root and key body-part trajectory alignment. This approach eliminates the need for large-scale robot teleoperation data, relying instead on human egocentric videos paired with synchronized motion and text annotations, significantly reducing data collection costs while enabling diverse, scene-aware behaviors.
Key Results
- On Nymeria dataset, ZeroWBC achieved a FID of 0.298, R-Precision of 0.847, outperforming previous methods in semantic consistency and spatial accuracy. In real-world tests on Unitree G1, it successfully performed obstacle avoidance, ball kicking, and sitting tasks with success rates exceeding 80%, demonstrating strong zero-shot generalization to unseen tasks and environments.
- Ablation studies confirmed that the interaction-oriented reward and future motion encoding significantly improved naturalness and task precision. The system maintained high performance across static scenes and various difficulty levels, validating the effectiveness of the generation-then-tracking paradigm.
- The framework's ability to generate diverse, scene-aware behaviors from minimal input showcases its potential for scalable, autonomous humanoid control, reducing reliance on costly teleoperation datasets and enabling real-time deployment in complex environments.
Significance
This work addresses longstanding challenges in humanoid control by shifting from costly, task-specific teleoperation data to scalable learning from human egocentric videos. It bridges the gap between high-level semantic understanding and low-level motion execution, enabling robots to perform complex, scene-aware interactions naturally. The integration of multimodal models and discrete motion tokens offers a new pathway for scalable, flexible robot learning, with broad implications for service, assistive, and collaborative robotics. It paves the way for autonomous systems capable of understanding and acting in unstructured environments, significantly advancing the field of robot autonomy and human-robot interaction.
Technical Contribution
ZeroWBC introduces several key innovations: 1) a VQ-VAE-based motion tokenizer that compresses high-dimensional motion data into discrete tokens, facilitating efficient generation; 2) a vision-language model fine-tuned for autoregressive prediction of motion tokens conditioned on egocentric visual and textual inputs, enhancing semantic grounding; 3) a novel interaction-oriented reward function that emphasizes global root and key body-part trajectory alignment during tracking, improving task relevance and naturalness; 4) a curriculum learning strategy with progressive difficulty exposure and future motion encoding, boosting model robustness and generalization. These contributions collectively enable scalable, flexible, and natural humanoid interaction learning from human demonstrations.
Novelty
This paper is the first to leverage egocentric videos paired with synchronized motion and text annotations for large-scale, scene-aware humanoid behavior generation. Unlike prior works relying on teleoperation datasets or simple visual control, ZeroWBC combines discrete motion tokens, multimodal prediction, and interaction-focused tracking, establishing a new paradigm for scalable, zero-shot humanoid control. Its generation-then-tracking architecture and reward design are novel contributions that significantly advance the state-of-the-art in autonomous humanoid behavior synthesis.
Limitations
- The current system mainly handles static scenes; dynamic environments with moving objects or humans pose challenges for real-time replanning and adaptation.
- Open-loop generation introduces latency (~400 ms), leading to phase lag and jitter in dynamic tasks, limiting responsiveness.
- Lack of tactile and force feedback restricts contact-rich task performance, such as grasping or force modulation, requiring future integration of multi-sensory perception.
Future Work
Future efforts will focus on developing closed-loop control strategies for dynamic scene adaptation, reducing latency through model acceleration or distillation, and incorporating tactile and force sensing for contact-rich tasks. Extending the framework to multi-agent scenarios and multi-modal perception will further enhance autonomous capabilities, enabling more robust, real-time, and contact-aware humanoid interactions in complex, unstructured environments.
AI Executive Summary
ZeroWBC represents a significant leap forward in humanoid robot control, addressing the high costs and limited scalability of traditional teleoperation-based learning. By harnessing human egocentric videos paired with synchronized motion and text annotations, it constructs a scalable, data-efficient framework for scene-aware, natural whole-body behaviors. The core innovation lies in a generation-then-tracking architecture: a VQ-VAE compresses human motions into discrete tokens, which are predicted autoregressively by a fine-tuned vision-language model conditioned on egocentric images and instructions. These tokens are decoded into continuous motions and retargeted onto a humanoid robot, which then executes them via a reinforcement learning-based interactive tracker guided by a novel interaction-oriented reward. This reward emphasizes the alignment of global root and key body-part trajectories, ensuring task relevance and motion naturalness. Extensive experiments on Nymeria, HumanML3D, and real-world Unitree G1 robot tasks demonstrate the system’s ability to generate diverse, scene-aware behaviors with high fidelity and zero-shot generalization. Notably, the robot successfully performs obstacle avoidance, ball kicking, and sitting tasks, with success rates exceeding 80%, even on unseen tasks. These results highlight the potential of combining multimodal learning and discrete motion representations to enable scalable, autonomous humanoid control. The approach reduces reliance on costly teleoperation datasets, lowers data collection barriers, and opens new avenues for deploying robots in unstructured, real-world environments. Future work aims to incorporate closed-loop control, tactile sensing, and multi-agent capabilities, further advancing autonomous, contact-rich human-robot interaction.
Deep Dive
Abstract
Achieving versatile and natural whole-body humanoid interaction control remains challenging due to the high cost of whole-body teleoperation data. We present ZeroWBC, a teleoperation-free framework that learns humanoid whole-body interaction from human egocentric videos paired with synchronized whole-body motion and text annotations. ZeroWBC adopts a generation-then-tracking formulation to tackle the static scene whole-body interaction control problem. Given an initial egocentric image and a language instruction, a fine-tuned Vision-Language Model generates future human whole-body motion tokens, which are decoded into continuous motions and retargeted to the humanoid. The resulting reference motions, together with root and key body-part trajectories, are then executed by a general interactive motion tracking policy. To improve interaction performance, we introduce an interaction-oriented tracking reward that prioritizes global root and key body-part trajectory alignment while preserving natural whole-body motion. Experiments on the Unitree G1 humanoid robot show that ZeroWBC enables diverse scene-aware behaviors without robot teleoperation demonstrations. These results suggest a scalable paradigm for learning natural humanoid whole-body interaction from human egocentric data.