RuN: Residual Policy for Natural Humanoid Locomotion
RuN framework uses residual policy for natural humanoid locomotion, achieving 0-2.5m/s velocity range with improved training efficiency.
Key Findings
Methodology
RuN employs a Conditional Motion Generator (CMG) trained on AMASS to produce natural motion trajectories, paired with a residual policy optimized via PPO to handle dynamic corrections. This decoupled design simplifies exploration space.
Key Results
- Result 1: RuN achieves natural gaits across 0-2.5m/s velocity range on Unitree G1, reducing FID to 0.8753, outperforming GMP's 1.1874.
- Result 2: Training time reduced to 8 hours, 30% faster than baselines.
- Result 3: Ablation studies confirm residual policy and CMG significantly improve motion naturalness and stability.
Significance
This work addresses the long-standing conflict between motion naturalness and dynamic stability in humanoid robots, enabling broader applications in complex environments. RuN bridges motion generation and control, advancing both academia and industry.
Technical Contribution
Contributions include introducing a residual learning framework to simplify control, developing an efficient CMG model avoiding VAE pitfalls, and leveraging AAC architecture for robust policy training.
Novelty
RuN is the first to decouple motion generation and dynamic correction, avoiding multi-objective conflicts in traditional methods. Compared to GMP, RuN's CMG design is more efficient and stable.
Limitations
- Limitation 1: CMG struggles to generate stable trajectories under extreme speeds or complex terrains.
- Limitation 2: High computational requirements for training may limit accessibility for smaller labs.
Future Work
Future work includes extending RuN to multi-terrain environments and optimizing CMG for broader motion types like jumping or crawling.
AI Executive Summary
RuN framework addresses the challenge of natural humanoid locomotion by introducing residual learning. Existing methods require policies to simultaneously perform motion imitation, velocity tracking, and stability maintenance, leading to complexity and limited performance.
RuN decouples these tasks using a Conditional Motion Generator (CMG) for natural motion trajectories and a residual policy for dynamic corrections. CMG is trained on AMASS, while the residual policy is optimized via PPO, significantly simplifying exploration space.
Experiments on the Unitree G1 robot demonstrate RuN achieves natural gaits across 0-2.5m/s velocity range, with training efficiency improved by 30% and FID reduced to 0.8753. The framework offers new perspectives for motion control while highlighting future optimization directions.
Deep Analysis
Background
Humanoid robots require natural locomotion in complex environments, but existing methods like ZMP and MPC often result in rigid gaits unsuitable for dynamic tasks. Deep reinforcement learning (DRL) enables complex motion but struggles to balance naturalness and stability.
Core Problem
Current methods force policies to simultaneously perform motion imitation, velocity tracking, and stability maintenance, creating a complex control problem. Decoupling these tasks is key to improving efficiency and performance.
Innovation
RuN decouples motion generation and dynamic correction using residual learning. CMG generates natural motion trajectories, while the residual policy focuses on dynamic corrections, simplifying exploration space and improving training efficiency.
Methodology
- �� Train CMG on AMASS dataset to produce natural motion trajectories.
- �� Optimize residual policy via PPO to correct CMG outputs for dynamic interactions.
- �� Employ AAC architecture for robust policy training.
- �� Use domain randomization to reduce sim-to-real gap.
Experiments
Experiments conducted on Unitree G1 robot, with training time ~8 hours using PPO. Compared against baselines Humanoid-Gym, AMP, and GMP, evaluating metrics like motion naturalness (FID) and velocity tracking error (Evel).
Results
RuN achieves natural gaits across 0-2.5m/s velocity range, reducing FID to 0.8753 and training time by 30%. Ablation studies confirm residual policy and CMG significantly improve motion naturalness and stability.
Applications
RuN is applicable to service robots, rehabilitation robots, and tasks requiring natural motion and dynamic stability.
Limitations & Outlook
CMG struggles with extreme speeds or complex terrains. Training requires significant computational resources, limiting accessibility for smaller labs.
Plain Language Accessible to non-experts
Imagine RuN as a smart chef in a kitchen. The CMG is like a master chef creating recipes, while the residual policy is an assistant making real-time adjustments to suit diners' preferences. This teamwork ensures the cooking process is both efficient and adaptive.
ELI14 Explained like you're 14
Think of playing a robot game where your robot needs to walk and run smoothly. RuN gives your robot two 'power-ups': a super map generator for planning steps and a real-time adjuster to avoid tripping! Cool, right?
Glossary
Residual Learning
A method that learns small corrections to optimize existing model outputs.
Used to refine CMG-generated motion trajectories.
Conditional Motion Generator
A model generating natural motion trajectories based on input conditions.
Provides reference trajectories for residual policy corrections.
AMASS Dataset
A large-scale dataset with 40 hours of human motion capture data.
Used to train CMG for natural motion generation.
PPO Algorithm
A reinforcement learning algorithm optimizing policies to maximize rewards.
Used to train the residual policy network.
FID Metric
Measures similarity between generated and real motion distributions.
Evaluates motion naturalness.
Open Questions Unanswered questions from this research
- 1 How to extend RuN to multi-terrain environments?
- 2 How to optimize CMG for complex motions like jumping?
Applications
Immediate Applications
Service Robots
Enhance naturalness and stability in complex environments.
Rehabilitation
Enable robots to provide smoother gait assistance for patients.
Long-term Vision
All-Terrain Robots
Develop robots capable of adapting to diverse terrains for complex tasks.
Abstract
Enabling humanoid robots to achieve natural and dynamic locomotion across a wide range of speeds, including smooth transitions from walking to running, presents a significant challenge. Existing deep reinforcement learning methods typically require the policy to directly track a reference motion, forcing a single policy to simultaneously learn motion imitation, velocity tracking, and stability maintenance. To address this, we introduce RuN, a novel decoupled residual learning framework. RuN decomposes the control task by pairing a pre-trained Conditional Motion Generator, which provides a kinematically natural motion prior, with a reinforcement learning policy that learns a lightweight residual correction to handle dynamical interactions. Experiments in simulation and reality on the Unitree G1 humanoid robot demonstrate that RuN achieves stable, natural gaits and smooth walk-run transitions across a broad velocity range (0-2.5 m/s), outperforming state-of-the-art methods in both training efficiency and final performance.