MotionWAM: Towards Foundation World Action Models for Real-Time Humanoid Loco-Manipulation
MotionWAM integrates video diffusion and motion models for real-time humanoid control, achieving over 30% success rate improvement.
Key Findings
Methodology
MotionWAM combines a Video Diffusion Transformer (Video DiT) with a Motion Diffusion Transformer (Motion DiT), leveraging intermediate denoising features for end-to-end whole-body action prediction. The training involves three stages: large-scale egocentric video pretraining for dynamic prior learning, cross-embodiment fine-tuning on diverse Unitree G1 datasets, and supervised teleoperation fine-tuning. The model predicts unified whole-body motion tokens, covering locomotion, torso, height, foot, and hand actions, within a single action space. Flow-matching loss enforces temporal coherence and physical plausibility. The approach enables real-time inference at 4.9Hz, suitable for closed-loop humanoid control.
Key Results
- On nine real-world tasks, MotionWAM achieves an average success rate exceeding 76%, over 30% higher than strong baselines like VLA. Notably, in complex coordination tasks, success rates jump from 43.9% to 76.1%, demonstrating the benefit of unified action representation for active leg and torso behaviors.
- The model maintains real-time performance, with inference speed 7 times faster than iterative denoising models, validating its practicality for dynamic balance and interaction tasks.
- Ablation studies confirm that each training stage—pretraining, cross-embodiment fine-tuning, and teleoperation—contributes significantly, with performance dropping sharply when any stage is omitted, highlighting the importance of staged transfer learning.
Significance
This work advances the application of large-scale video generative models in robotics, enabling autonomous humanoid control with complex whole-body coordination. It bridges the gap between static perception and dynamic physical interaction, pushing toward robots that can perform human-like tasks in unstructured environments. The end-to-end design and real-time capability open new avenues for industrial automation, service robots, and rescue operations, marking a significant step in visual-driven robot intelligence.
Technical Contribution
The core innovation lies in unifying the control space with a single motion latent, replacing hierarchical upper-lower control schemes. The integration of Video DiT and Motion DiT via intermediate denoising features enables end-to-end, physically plausible whole-body motion prediction. The three-stage training strategy ensures effective transfer from large-scale egocentric videos to specific robot embodiments, maintaining temporal coherence through flow-matching. This architecture offers a new paradigm for visuomotor control, combining generative modeling with robotics.
Novelty
This is the first work to combine a video diffusion model with a unified motion diffusion model for real-time whole-body humanoid control. Unlike prior methods limited to arm or simple locomotion, MotionWAM supports complex, task-driven foot and hand interactions within a single predictive framework. Its staged training approach ensures robust transfer from large-scale video data to physical robot control, setting a new standard for visual-based humanoid locomotion.
Limitations
- The system has only been validated on the Unitree G1 platform; its transferability to other humanoid robots remains untested, possibly requiring adaptation for different hardware configurations.
- Performance drops when encountering objects or scenarios outside the training distribution, especially with partial or occluded visual input, indicating reliance on visual fidelity.
- While inference speed is sufficient for current tasks, more complex multi-robot or multi-agent scenarios may demand further optimization to sustain real-time operation.
Future Work
Future directions include integrating multimodal perception such as tactile and auditory cues to enhance environmental understanding, extending the framework to multi-robot collaboration, and improving model efficiency for deployment on edge devices. Additionally, exploring unsupervised or reinforcement learning paradigms could further improve adaptability and robustness in unstructured environments.
AI Executive Summary
Achieving autonomous humanoid control capable of complex whole-body tasks remains a significant challenge in robotics. Traditional hierarchical systems often restrict leg and torso actions, limiting the robot’s ability to perform dynamic interactions like kicking or stepping on objects. Existing visual models excel at perception but struggle with real-time, physically grounded control. To address this, the authors introduce MotionWAM, a novel framework that leverages large-scale egocentric videos to train a unified, end-to-end control policy.
MotionWAM integrates a Video Diffusion Transformer (Video DiT) with a Motion Diffusion Transformer (Motion DiT), enabling the system to predict comprehensive, coordinated whole-body motions directly from a single camera input. The key innovation is the use of intermediate denoising features, which allow the model to generate physically plausible actions efficiently. The training process involves three stages: large-scale egocentric video pretraining, cross-embodiment fine-tuning on diverse datasets, and targeted teleoperation fine-tuning on the specific humanoid platform.
Experimental results demonstrate that MotionWAM outperforms state-of-the-art baselines by over 30% success rate across nine real-world tasks, including complex foot and hand interactions. The model operates at 4.9Hz, making it suitable for closed-loop control, and achieves behaviors such as kicking, stepping, and object manipulation that previous hierarchical policies cannot perform. This work signifies a major step toward human-like, visually grounded autonomous robots capable of versatile, coordinated actions.
Despite its success, the approach faces limitations in hardware generalization and scene complexity. Future work aims to incorporate multimodal sensing, multi-robot coordination, and efficiency improvements, paving the way for more adaptable and intelligent robotic systems in real-world applications.
Deep Dive
Abstract
World Action Models (WAMs) couple a video dynamics prior to the policy and have shown encouraging results on tabletop manipulation, but iterative denoising over high-dimensional video-action latents leaves them too slow for real-time humanoid loco-manipulation. The problem is compounded by the dominant hierarchical paradigm, in which a high-level manipulation policy controls only the upper body while a low-level controller tracks coarse base commands -- placing upper and lower body in inconsistent action spaces and reducing the legs to balance-preserving locomotion. We present MotionWAM, a real-time WAM that drives autonomous humanoid loco-manipulation from a single egocentric camera by conditioning the policy on the intermediate denoising features of a video world model. MotionWAM replaces the upper-lower split with a unified motion latent and predicts whole-body motion tokens that jointly cover locomotion, torso motion, height regulation, foot interaction, and hand manipulation in a single action space. A three-stage learning framework progressively adapts the video world model to egocentric visual dynamics and to the target humanoid embodiment. On nine real-world Unitree G1 tasks, MotionWAM runs in real time, substantially outperforms Vision-Language-Action (VLA) baselines fine-tuned on the same demonstrations by over 30% in overall success rate, and executes task-driven foot interaction that decoupled upper-lower policies cannot reach. Our results suggest that video-pretrained WAMs can be lifted from tabletop manipulation to coordinated, human-like whole-body humanoid control.