MulDP: Multimodal Diffusion Policy for Autonomous Quadruped Parkour Navigation across Complex Terrains
MulDP employs multimodal diffusion models to enable autonomous quadruped navigation across complex terrains, achieving 89.7% success in diverse scenarios.
Key Findings
Methodology
This paper introduces MulDP, a conditional diffusion-based policy that integrates historical depth images, proprioception, and goal information. It employs a multimodal temporal encoder, spatial encoder, and decision memory to generate temporally coherent velocity commands. The model is trained end-to-end on the QPND dataset, which includes diverse terrains and behaviors. During inference, the model iteratively denoises future commands, enabling long-horizon, anticipatory navigation. Extensive simulation and real-world tests demonstrate its robustness and generalization in complex environments.
Key Results
- In both simulation and real-world experiments, MulDP achieved an average success rate of 89.7%, path efficiency of 71.9%, and an average time to goal of 4.9 seconds across various terrains such as gaps, hurdles, and stairs. It outperformed baselines like NavDP, ViNT, and PointNav, especially in obstacle traversal and dynamic target following, confirming its superior long-range planning and anticipatory control capabilities.
- Ablation studies revealed that proprioception and data augmentation are critical; removing proprioception decreased success rate from 89.7% to 69.5%, and omitting data augmentation led to severe overfitting, reducing success to 32.5%. These findings highlight the importance of multimodal perception and robust training data for real-world deployment.
- The model demonstrated effective long-distance navigation in unseen large-scale environments, with velocity profiles showing proactive acceleration before gaps and obstacle avoidance behaviors, indicating strong predictive and anticipatory abilities.
Significance
This work advances autonomous quadruped navigation by integrating generative diffusion models with multimodal perception, addressing the challenge of long-horizon, dynamic environment traversal. It shifts the paradigm from traditional mapping-based or reactive methods to a probabilistic, anticipatory control framework, enabling robots to operate more safely and efficiently in unstructured terrains. The creation of the QPND dataset further supports scalable training and transfer, fostering broader research in autonomous legged locomotion. The approach paves the way for deploying robots in real-world scenarios like search and rescue, inspection, and exploration, where adaptability and robustness are paramount.
Technical Contribution
The paper introduces a novel multimodal diffusion policy that models action distributions conditioned on visual, proprioceptive, and goal inputs. It employs a hierarchical encoding architecture, combining a multimodal transformer for temporal context, a CNN for immediate perception, and a decision memory module. The diffusion process iteratively refines velocity commands, ensuring smooth, anticipatory control. The dataset QPND supports training with noise injection and data augmentation, facilitating sim-to-real transfer. This integrated framework offers a new avenue for long-horizon, multimodal robot control, surpassing existing deterministic or sampling-based methods.
Novelty
This is the first application of conditional diffusion models for autonomous quadruped navigation in complex terrains, integrating multimodal perception for long-term, anticipatory velocity control. Unlike prior works focusing on waypoint prediction or reactive obstacle avoidance, MulDP directly generates velocity commands conditioned on rich sensory inputs, enabling more fluid and proactive behaviors. The combination of hierarchical encoding, diffusion-based sampling, and a dedicated dataset represents a significant innovation in robot control strategies.
Limitations
- The model's performance degrades in environments with extremely high sensor noise or drastic environmental changes, indicating robustness limitations under adverse conditions.
- Training relies heavily on simulated data, and despite transfer strategies, real-world deployment still faces domain gap issues, particularly in highly dynamic or cluttered environments.
- Real-time inference is constrained by hardware capabilities; higher update frequencies or more complex environments may require optimization or hardware acceleration.
Future Work
Future directions include integrating reinforcement learning for adaptive decision-making, enhancing perception robustness via sensor fusion, and extending to multi-robot systems. Additionally, optimizing inference speed and energy efficiency will be crucial for deployment on resource-constrained platforms. Exploring multi-task learning to handle diverse navigation objectives and environments will further broaden applicability.
AI Executive Summary
Autonomous navigation of quadruped robots in complex terrains remains a significant challenge, especially in unstructured environments with obstacles, gaps, and dynamic elements. Traditional methods relying on mapping and planning often struggle with long-horizon, anticipatory behaviors necessary for safe traversal. Addressing this gap, the present work introduces MulDP, a novel multimodal diffusion policy that directly generates velocity commands conditioned on visual, proprioceptive, and goal information.
This approach leverages a hierarchical encoding architecture, including a multimodal transformer for temporal context, a spatial encoder for immediate perception, and a decision memory module, to produce smooth, anticipatory control signals. The core innovation lies in modeling action distributions via a generative diffusion process, which iteratively refines velocity commands, enabling long-term planning and robust obstacle traversal.
The authors also constructed the Quadruped Parkour Navigation Dataset (QPND), a large-scale multimodal dataset capturing diverse terrains and behaviors, facilitating training and sim-to-real transfer. Extensive experiments in simulation and real-world settings demonstrate that MulDP achieves an average success rate of 89.7%, outperforming baseline methods significantly. Its velocity profiles show proactive acceleration before gaps and obstacle avoidance, indicating strong predictive capabilities.
This research marks a substantial step forward in autonomous legged locomotion, providing a flexible, probabilistic framework that can adapt to complex, dynamic environments. Future work will focus on improving real-time inference, robustness, multi-robot coordination, and broader application scenarios, promising a new era of intelligent, autonomous quadruped systems.
Deep Dive
Abstract
Quadruped robots have demonstrated impressive agility in parkour locomotion across complex terrains. However, most systems still rely on human intervention for high-level planning, and autonomous parkour navigation remains underexplored. The key challenges include fine-grained velocity regulation, long-horizon anticipatory behaviors, and tight coupling between perception and embodied execution. To address these challenges, we propose a Multimodal Diffusion Policy (MulDP) that integrates visual perception with robot proprioception and goal information to generate temporally coherent and anticipatory navigation velocity commands, tightly coupling perception with embodied control to enable robust autonomous navigation. To support the training of MulDP, we construct the first Quadruped Parkour Navigation Dataset (QPND), a multimodal dataset that encompasses diverse navigation behaviors and complex terrains. Extensive simulation and real-world experiments demonstrate that MulDP enables robust long-horizon autonomous navigation and effective traversal across complex terrains.