DiffusionDrive: Truncated Diffusion Model for End-to-End Autonomous Driving
Proposes truncated diffusion with multi-mode anchors, reducing steps from 20 to 2, achieving 88.1 PDMS at 45FPS for autonomous driving.
Key Findings
Methodology
This paper introduces a truncated diffusion strategy that incorporates prior multi-mode anchors, truncating the diffusion process to only 2 steps. A transformer-based cascade decoder enhances scene interaction, enabling efficient multi-modal trajectory generation. The model trains on clustered anchors, learning to denoise from anchored Gaussian distributions to realistic trajectories. During inference, it samples from the anchored Gaussian, performs minimal denoising, and selects the highest confidence trajectory. The approach significantly reduces computational overhead while maintaining diverse, high-quality outputs, validated on NAVSIM with 88.1 PDMS and 45FPS, outperforming state-of-the-art methods.
Key Results
- On NAVSIM, DiffusionDrive achieves 88.1 PDMS with only 2 denoising steps, surpassing VADv2 (81.0) and Hydra-MDP (86.5). It runs at 45FPS on NVIDIA 4090, 4x faster than vanilla diffusion.
- In nuScenes, it reduces L2 error by 20.8% and collision rate by 63.6%, demonstrating superior robustness and accuracy across datasets.
- Ablation studies confirm that truncating to 2 steps and cascade decoding improve trajectory diversity and quality, with minimal computational cost.
Significance
This work advances the application of diffusion models in real-time autonomous driving by addressing computational bottlenecks and mode collapse issues. It enables efficient, diverse, and plausible multi-modal trajectory generation, crucial for safe navigation in complex traffic environments. The approach bridges the gap between generative modeling and practical deployment, offering a scalable solution for end-to-end autonomous systems, with potential industry impact on safety and efficiency.
Technical Contribution
The key technical innovation is the integration of multi-mode anchors into a truncated diffusion process, drastically reducing inference steps. The transformer cascade decoder enhances scene interaction, improving trajectory quality. The model combines probabilistic priors with efficient denoising, enabling high-speed, multi-modal predictions. This approach departs from traditional vocabulary-based or single-modal methods, offering a flexible, scalable framework for real-time applications.
Novelty
This is the first work to embed multi-mode prior anchors into a diffusion process for autonomous driving, employing a truncated diffusion schedule to achieve rapid inference. The combination of anchored Gaussian distributions with cascade transformer decoders is novel, enabling high-diversity, high-quality trajectory generation in dynamic traffic scenes, surpassing existing methods that rely on fixed vocabularies or rule-based heuristics.
Limitations
- The approach depends heavily on the quality and representativeness of the anchor set; poor anchors may limit diversity or accuracy.
- In highly unpredictable or extreme scenarios, the model may still produce suboptimal trajectories, requiring further robustness enhancements.
- Training and inference demand significant computational resources, posing challenges for deployment on resource-constrained platforms.
Future Work
Future research will focus on adaptive anchor generation to improve scene generalization, integrating reinforcement learning to optimize trajectory diversity, and model compression techniques to reduce computational costs. Extending the framework to multi-agent interactions and more complex traffic scenarios will further enhance its practical applicability.
AI Executive Summary
Autonomous driving has seen rapid progress with perception modules reaching high accuracy, yet trajectory prediction remains a bottleneck, especially in complex urban environments. Traditional rule-based or single-modal models struggle to generate diverse, plausible trajectories in real-time, limiting safety and flexibility. Recent advances in generative models, particularly diffusion-based approaches, offer promising avenues for multi-modal prediction but are hampered by high computational costs and mode collapse issues.
This paper introduces DiffusionDrive, a novel truncated diffusion framework that leverages prior multi-mode anchors to drastically reduce inference steps from 20 to just 2. By integrating a transformer-based cascade decoder, the system enhances scene interaction, enabling it to generate diverse, high-quality trajectories efficiently. The core idea is to initialize the diffusion process from anchored Gaussian distributions, which encode prior driving patterns, and truncate the denoising schedule during training and inference.
Experimental results on the NAVSIM dataset demonstrate that DiffusionDrive achieves an unprecedented 88.1 PDMS score at 45FPS, outperforming existing methods like VADv2 and Hydra-MDP. The model also shows superior performance on nuScenes, with a 20.8% reduction in L2 error and a 63.6% decrease in collision rate, validating its robustness and generalization.
The significance of this work lies in its ability to produce diverse, plausible trajectories in real-time, addressing critical challenges in autonomous driving. It opens new pathways for integrating generative diffusion models into safety-critical systems, promising safer and more adaptable autonomous vehicles.
Despite these advances, limitations remain, including dependence on anchor quality and computational demands. Future directions involve adaptive anchor learning, reinforcement learning integration, and model compression to facilitate deployment in resource-constrained environments. Overall, DiffusionDrive marks a substantial step forward in scalable, multi-modal trajectory prediction for autonomous driving, with broad implications for industry and research.
Deep Dive
Abstract
Recently, the diffusion model has emerged as a powerful generative technique for robotic policy learning, capable of modeling multi-mode action distributions. Leveraging its capability for end-to-end autonomous driving is a promising direction. However, the numerous denoising steps in the robotic diffusion policy and the more dynamic, open-world nature of traffic scenes pose substantial challenges for generating diverse driving actions at a real-time speed. To address these challenges, we propose a novel truncated diffusion policy that incorporates prior multi-mode anchors and truncates the diffusion schedule, enabling the model to learn denoising from anchored Gaussian distribution to the multi-mode driving action distribution. Additionally, we design an efficient cascade diffusion decoder for enhanced interaction with conditional scene context. The proposed model, DiffusionDrive, demonstrates 10$\times$ reduction in denoising steps compared to vanilla diffusion policy, delivering superior diversity and quality in just 2 steps. On the planning-oriented NAVSIM dataset, with the aligned ResNet-34 backbone, DiffusionDrive achieves 88.1 PDMS without bells and whistles, setting a new record, while running at a real-time speed of 45 FPS on an NVIDIA 4090. Qualitative results on challenging scenarios further confirm that DiffusionDrive can robustly generate diverse plausible driving actions. Code and model will be available at https://github.com/hustvl/DiffusionDrive.