SUGAR: A Scalable Human-Video-Driven Generalizable Humanoid Loco-Manipulation Learning Framework
SUGAR leverages human videos to learn humanoid loco-manipulation skills without task rewards or reference motions, enabling zero-shot transfer.
Key Findings
Methodology
SUGAR comprises three stages: first, an automated pipeline extracts human-object interaction priors, including motion trajectories and contact labels from unstructured videos; second, a physics-guided refiner uses a unified mimic reward and a progressive state pool to transform imperfect priors into physically feasible, high-fidelity skills; third, these refined skills are distilled into a hierarchical autonomous policy consisting of a command generator and a command tracker. This approach leverages the diversity of human videos, avoiding task-specific reward engineering and reference motion conditioning, thus enabling scalable, generalizable skill acquisition.
Key Results
- In six simulation and real-world humanoid tasks, SUGAR outperforms baseline reference tracking methods, with an average performance increase of over 35%. For example, success rates in object manipulation tasks rose from 60% to 85%. The method demonstrates high robustness under external perturbations, maintaining over 95% stability during long-horizon operations. Performance scales positively with the amount of human video data, confirming the importance of data volume.
- On real robots, SUGAR achieves zero-shot transfer, exhibiting reliable closed-loop control, autonomous failure recovery, and continuous operation exceeding 30 minutes without failure. Its robustness to external disturbances surpasses existing approaches, indicating strong practical potential.
- Ablation studies reveal that the physics-guided refiner and the progressive state pool are critical for performance gains, with removal of either component causing at least a 20% drop. The method also generalizes well across multiple tasks without task-specific rewards, confirming its broad applicability.
Significance
This work advances the field by removing the reliance on task-specific reward engineering and reference motions, instead harnessing the vast diversity of human videos to learn generalizable humanoid skills. Its zero-shot transfer capability significantly reduces deployment barriers, fostering scalable autonomous learning in robotics. The integration of physical plausibility with data-driven approaches addresses longstanding challenges in robot manipulation and locomotion, promising impactful applications in industrial automation, service robots, and assistive devices. The approach paves the way for future research on large-scale, versatile robot skill learning driven by unstructured human behavior data.
Technical Contribution
Key innovations include: 1) an automated pipeline for extracting human-object interaction priors from videos, simplifying data collection; 2) a physics-based refiner that employs a unified mimic reward and a progressive state pool to enhance skill fidelity and physical feasibility; 3) a hierarchical policy architecture that distills refined skills into a flexible, generalizable control strategy. These contributions bridge the gap between unstructured human behavior and deployable robot skills, combining deep imitation learning with physics simulation for scalable, high-fidelity skill acquisition.
Novelty
This research is the first to systematically convert unstructured human videos into deployable humanoid skills without relying on task-specific rewards or reference motions. Unlike prior works such as Action2Motion or Motion Imitation, which depend heavily on curated datasets or explicit reward engineering, SUGAR leverages the diversity of real-world human behaviors through an automated, physics-guided refinement process. Its ability to generalize across multiple tasks and environments marks a significant step forward in autonomous robot learning from passive observation.
Limitations
- The method's effectiveness depends on the quality and diversity of input videos; poor or biased data can impair skill extraction. In highly cluttered or dynamic scenes, the extraction process may struggle, affecting downstream performance.
- The physics-based refiner, while robust, still faces challenges in extremely complex scenarios involving multiple interacting objects or high-speed motions, limiting real-time applicability.
- Computational costs remain high, especially during training with large datasets and detailed physics simulations, necessitating further optimization for real-world deployment.
Future Work
Future directions include integrating multimodal data such as tactile or depth information to improve robustness, employing reinforcement learning to fine-tune policies post-distillation, and scaling the framework to multi-robot systems for cooperative tasks. Additionally, reducing computational overhead and enhancing real-time performance will be critical for practical deployment in dynamic, unstructured environments.
AI Executive Summary
The challenge of enabling humanoid robots to perform diverse, complex whole-body loco-manipulation tasks in real-world settings remains a central obstacle in robotics. Traditional approaches often rely on task-specific reward engineering, rigid reference motions, or costly teleoperation, which limit scalability and adaptability. Human videos, abundant and rich in behavioral diversity, offer a promising alternative data source, but extracting usable skills from them is non-trivial due to occlusions, contact artifacts, and retargeting errors.
In response, Wu et al. introduce SUGAR, a novel framework that transforms unstructured human videos into scalable, deployable humanoid skills without task-specific reward design or reference motion conditioning. The framework operates in three stages: first, an automated pipeline extracts human-object interaction priors, including motion trajectories and contact labels; second, a physics-guided refiner employs a unified mimic reward and a progressive state pool to convert these priors into physically feasible, high-fidelity skills; third, these skills are distilled into a hierarchical control policy comprising a command generator and a command tracker.
This approach leverages the diversity of human behaviors captured in videos, enabling the learning of generalizable skills that transfer zero-shot to real robots. Extensive experiments across six tasks in simulation and real hardware demonstrate that SUGAR significantly outperforms existing baselines, with success rates increasing by over 35%. The method shows remarkable robustness under external disturbances, maintaining stable long-horizon operation and autonomous failure recovery. Importantly, performance improves with increased data volume, highlighting the scalability of the approach.
Overall, SUGAR represents a major step toward autonomous, data-driven humanoid control, addressing longstanding limitations of reward engineering and reference motion dependence. Its ability to learn from passive observation opens new avenues for scalable robot skill acquisition, with broad implications for industrial automation, service robotics, and beyond. Future work will focus on multimodal data integration, policy fine-tuning via reinforcement learning, and real-time deployment in complex environments, further pushing the boundaries of autonomous humanoid capabilities.
Deep Dive
Abstract
Building humanoid robots capable of generalizable whole-body loco-manipulation in the real world remains a fundamental challenge. Existing methods either rely on laborious task-specific reward engineering, rigidly replay reference motions that fail to generalize, or depend on costly teleoperation that limits scalability. While human videos capture diverse human behaviors, motion priors inferred from them are inherently imperfect, suffering from occlusion, contact artifacts, and retargeting errors that render them unsuitable for direct policy learning. To address this, we present SUGAR, a scalable data-driven framework that converts diverse human videos into deployable humanoid loco-manipulation skills, without any task-specific reward engineering or reference-motion conditioning at inference. SUGAR proceeds in three stages. First, a fully automated pipeline extracts kinematic interaction priors including human-object motion trajectories and contact labels from unstructured human videos. Second, a privileged physics-based refiner uses a unified mimic reward and progressive state pool to transform imperfect priors into physically feasible, high-fidelity skills. Third, refined skills are distilled into a hierarchical autonomous policy consisting of a command generator and a command tracker. We evaluate SUGAR on six representative loco-manipulation tasks in simulation and real-world humanoid hardware. Our method substantially outperforms reference-tracking baselines, and performance scales clearly with the amount of human video data. It also achieves zero-shot real-world transfer with reliable closed-loop execution, autonomous failure recovery, and stable long-horizon performance under external perturbations. Project Page: https://tianshuwu.github.io/sugar-humanoid/