UniTeD: Unified Temporal Diffusion for Joint Perception and Planning in Autonomous Driving
UniTeD introduces a joint temporal diffusion model for perception and planning, achieving state-of-the-art results with 90.24 PDMS on NAVSIM v1.
Key Findings
Methodology
UniTeD employs a unified temporal diffusion decoder that performs iterative denoising on perception (dynamic agents, static maps) and planning (trajectory prediction) within a shared generative space. It introduces a Temporal Transition Module (TTM) to align historical and current frame noise levels, and an Anchor Refresh Strategy (ARS) to mitigate training-inference distribution shifts. The framework processes multi-view visual features, conditioned on multi-task noisy anchors, trained with noise-conditioned multi-task loss. This design enables bidirectional task interaction, robust multi-modal outputs, and temporal consistency, significantly improving end-to-end autonomous driving performance.
Key Results
- On NAVSIM v1 and v2 benchmarks, UniTeD achieves PDMS scores of 90.24 and 90.13 respectively, surpassing recent diffusion-based and discriminative models. In Bench2Drive, it attains a Driving Score of 87.25, demonstrating superior handling of complex, interactive scenarios. Ablation studies confirm that TTM enhances temporal coherence, while ARS stabilizes training-inference alignment. The multi-task joint training reduces perception errors propagation, leading to more accurate and robust trajectory predictions across diverse environments.
- The temporal modeling via TTM effectively reduces noise-level mismatch across frames, improving trajectory consistency. ARS dynamically refreshes low-confidence queries, maintaining distribution alignment during inference. Multi-view feature fusion and noise-conditioned training bolster generalization, enabling the model to adapt to real-world complexities. Results indicate that the unified diffusion approach outperforms existing state-of-the-art methods in both accuracy and robustness.
- Extensive experiments validate UniTeD’s capability to generate multi-modal trajectories and enhance perception tasks simultaneously, outperforming prior discriminative and generative models. Its architecture supports scalable multi-task learning, making it suitable for real-world autonomous driving systems with high safety and reliability demands.
Significance
This work advances autonomous driving by integrating perception and planning into a single, unified generative framework, overcoming limitations of decoupled modules. The joint modeling approach leverages diffusion models’ uncertainty quantification, enabling multi-modal trajectory generation and robust perception. The incorporation of temporal context and distribution shift mitigation strategies addresses critical challenges in real-world deployment. As a result, UniTeD paves the way for more reliable, safe, and adaptable autonomous systems, bridging the gap between academic research and practical application in complex traffic environments.
Technical Contribution
UniTeD’s primary innovation lies in extending diffusion models to a multi-task, temporally-aware framework that jointly models perception and planning. The introduction of TTM ensures temporal consistency, while ARS stabilizes training-inference distribution alignment. The shared generative space facilitates deep bidirectional interaction between perception and planning queries, enabling mutual refinement. The multi-task noise-conditioned training enhances robustness against uncertainty, setting a new standard for end-to-end autonomous driving models that combine generative uncertainty modeling with temporal dynamics.
Novelty
UniTeD is the first to unify perception and planning within a single diffusion-based generative model that explicitly incorporates temporal context. Unlike prior works that treat perception as fixed input or focus solely on planning, this approach allows joint, iterative refinement of all tasks, significantly improving multi-modal outputs. The combination of TTM and ARS addresses key issues of temporal mismatch and training-inference shift, representing a novel paradigm shift in end-to-end autonomous driving modeling.
Limitations
- Despite its strengths, UniTeD still faces challenges in extreme weather or sensor failure scenarios where perception quality drops significantly. The computational cost of iterative diffusion and multi-task training is high, limiting real-time deployment without optimization. Additionally, the model’s performance in highly dynamic or unpredictable environments needs further validation. Future work should focus on efficiency improvements and robustness under adverse conditions.
Future Work
Future directions include optimizing the diffusion process for real-time inference, possibly via accelerated sampling techniques. Incorporating reinforcement learning could further enhance decision-making robustness. Extending temporal modeling to longer horizons and multi-modal sensor fusion will improve long-term stability. Additionally, deploying and testing UniTeD on real autonomous vehicles will be crucial to validate its practical utility and safety in diverse traffic scenarios.
AI Executive Summary
Autonomous driving has long sought a seamless integration of perception and planning, yet traditional modular systems suffer from error propagation and limited adaptability. Recent advances in deep learning have pushed toward end-to-end solutions, but these often treat perception and planning as separate or loosely coupled components. Diffusion models, known for their success in generative tasks, have been primarily confined to planning modules, leaving perception less integrated. This gap hampers the ability to handle multi-modal uncertainties and temporal dynamics effectively.
UniTeD introduces a groundbreaking unified temporal diffusion framework that jointly models perception and planning. By performing iterative denoising within a shared generative space, the system enables bidirectional information exchange, mutual refinement, and robust multi-task learning. The Temporal Transition Module (TTM) addresses the challenge of aligning historical and current frame noise levels, ensuring temporal consistency. Meanwhile, the Anchor Refresh Strategy (ARS) mitigates training-inference distribution shifts, maintaining model stability during deployment.
Experimental results on NAVSIM and Bench2Drive benchmarks demonstrate that UniTeD surpasses existing state-of-the-art methods, achieving PDMS scores of over 90 and a Driving Score of 87.25. These metrics reflect its ability to generate accurate, multi-modal trajectories and robust perception outputs in complex, dynamic environments. The model’s multi-task training approach reduces error propagation, enhances generalization, and improves safety margins.
This work signifies a major step forward in autonomous driving research, combining the strengths of diffusion models with temporal and multi-task modeling. It opens new avenues for scalable, reliable, and safe self-driving systems capable of operating in real-world scenarios. Future efforts will focus on efficiency, real-world deployment, and extending the framework to longer temporal horizons, aiming to bring fully autonomous vehicles closer to everyday reality.
Deep Dive
Plain Language Accessible to non-experts
想象你在厨房里准备一顿大餐。不同的食材、厨具和调料代表了感知、地图和路径规划。传统的方法就像是你先准备好所有食材，然后逐步按照菜谱做菜，容易出错。而UniTeD像是有个聪明的厨师，他可以同时考虑所有食材和菜谱，边做边调整，确保每一步都完美。这个厨师会记住之前的操作，知道什么时候需要多放点盐，什么时候要多搅拌。它不断学习和改进，确保每次做的菜都比上次更好。就像自动驾驶系统一样，UniTeD在不断观察环境、调整策略，确保车辆在复杂的路况中安全行驶。这个系统像个超级厨师，既能快速反应，又能不断优化，帮你做出最棒的“菜”——安全、智能的自动驾驶。
Abstract
Diffusion models have shown strong potential for multi-modal planning in end-to-end autonomous driving. However, most existing methods confine diffusion to the planning module, conditioning on fixed outputs from separate discriminative perception networks. This decoupled design propagates perception errors to the planner, increasing optimization difficulty and reducing robustness. To overcome these limitations, we propose UniTeD, a Unified Temporal Diffusion framework that jointly models perception and planning through iterative denoising in a shared generative space. By enabling bidirectional information exchange, the framework facilitates mutual refinement between tasks and improves robustness via noise-conditioned multi-task training. We further extend this unified diffusion paradigm to a streaming setting by incorporating temporal context. A Temporal Transition Module (TTM) is introduced to resolve the noise-level mismatch between historical and current frames. In addition, we propose an Anchor Refresh Strategy (ARS) to alleviate the training-inference distribution shift commonly observed in sparse diffusion-based end-to-end driving frameworks. Without bells and whistles, UniTeD achieves state-of-the-art performance across multiple benchmarks, surpassing both recent discriminative end-to-end methods and diffusion-based planning approaches.