DrivingGPT: Unifying Driving World Modeling and Planning with Multi-modal Autoregressive Transformers
DrivingGPT employs a multimodal autoregressive Transformer to unify world modeling and path planning, outperforming baselines with superior video quality and planning scores.
Key Findings
Methodology
This work introduces a multimodal driving language by discretizing images via pretrained VQ-VAE and relative actions into tokens, interleaving them to form a unified sequence. A Llama-like Transformer with frame-wise rotary embeddings models this sequence through next-token prediction, enabling joint world modeling and planning. The model is trained end-to-end on large datasets (nuPlan, NAVSIM), optimizing both visual generation (FID, FVD) and planning metrics (PDMS). The approach effectively captures temporal and multimodal dependencies, surpassing diffusion-based methods in long-term coherence and diversity, while maintaining real-time inference capabilities.
Key Results
- On nuPlan, DrivingGPT achieved a FID of 12.78 and FVD of 278.11, outperforming SVD's 24.03 and 418.93, respectively, indicating higher visual quality and temporal consistency.
- In planning tasks on NAVSIM, it scored 82.4% PDMS, surpassing the ResNet+MLP baseline at 77.8%, demonstrating effective end-to-end trajectory prediction.
- Long video generation experiments with 64 frames showed a FVD of 506.95, significantly better than SVD's 1079.28, confirming the model's ability to generate coherent, diverse, and extended sequences.
Significance
This research bridges the gap between visual scene understanding and trajectory planning by leveraging a unified autoregressive sequence model. It addresses the limitations of diffusion models in multi-modality fusion and temporal coherence, advancing the state-of-the-art in end-to-end autonomous driving systems. The ability to generate high-quality long-term videos and accurate paths in complex environments has profound implications for real-world deployment, safety, and robustness of autonomous vehicles.
Technical Contribution
The paper proposes a novel multi-modal driving language combining discretized visual and action tokens, modeled via a Transformer with frame-level rotary embeddings. This design enables simultaneous learning of scene prediction and path planning within a single model, facilitating end-to-end training. It departs from traditional diffusion-based visual generation by adopting autoregressive sequence prediction, offering improved temporal consistency, diversity, and multimodal fusion capabilities. The approach also introduces a unified vocabulary and tokenization scheme, simplifying multi-task learning and inference.
Novelty
This is the first work to unify driving world modeling and trajectory planning into a single autoregressive sequence prediction framework using discrete tokens. Unlike prior methods relying on diffusion models for visual synthesis, it innovatively employs a Transformer architecture for both visual and action sequence generation, enabling joint optimization and long-horizon prediction. The integration of multi-modal tokens and the end-to-end training paradigm constitute a significant leap in autonomous driving research.
Limitations
- The model's performance in highly unstructured or unseen environments is limited by the discretization granularity of tokens and training data diversity.
- Computational complexity increases with sequence length, impacting real-time deployment in resource-constrained scenarios.
- Discretization may lead to loss of fine-grained details, affecting the fidelity of generated scenes and trajectories. Future work should focus on adaptive tokenization and model efficiency improvements.
Future Work
Future directions include enhancing multi-sensor fusion, refining tokenization strategies for richer scene representation, and integrating reinforcement learning to improve decision-making robustness. Additionally, deploying the model on real autonomous vehicles for real-world testing and safety validation remains a key goal.
AI Executive Summary
DrivingGPT introduces a groundbreaking approach to autonomous driving by unifying world modeling and trajectory planning within a single sequence prediction framework. Leveraging a multimodal driving language, the model discretizes visual and action data into tokens, which are then processed by a Transformer architecture inspired by Llama. This design allows the system to generate both high-quality videos of driving scenes and accurate future trajectories simultaneously.
Traditional methods often rely on separate modules or diffusion-based visual generators, which face challenges in temporal coherence and multi-modality fusion. In contrast, DrivingGPT's autoregressive sequence modeling captures long-term dependencies and multimodal correlations effectively, resulting in superior performance on large-scale datasets like nuPlan and NAVSIM. Quantitative metrics such as FID, FVD, and PDMS demonstrate clear improvements over existing baselines, with the model producing more diverse, coherent, and realistic long-horizon videos and trajectories.
The significance of this work lies in its potential to revolutionize end-to-end autonomous driving systems. By integrating scene understanding and planning into a unified model, it simplifies system architecture, reduces error propagation, and enhances robustness. The approach also opens avenues for future multi-sensor fusion, adaptive tokenization, and reinforcement learning integration, pushing the boundaries of what autonomous vehicles can achieve in complex real-world environments.
Deep Analysis
Background
自动驾驶技术经历了从基于规则的系统到深度学习驱动的端到端模型的演变。早期方法依赖手工设计的感知、决策与规划模块，代表性工作如Behavior Cloning和End-to-End学习。近年来，世界模型（如Video Prediction和3D空间建模）逐渐成为研究热点，旨在实现场景的未来预测与决策支持。扩散模型在视觉生成中表现出色，但在时间一致性和多模态融合方面存在瓶颈。多模态学习结合视觉、动作和语义信息，提升场景理解能力，但多模态信息的异质性和模型复杂度仍是难点。本文提出将多模态信息离散化、统一编码，利用Transformer实现端到端的联合建模，填补了视觉生成与路径规划融合的空白，为自动驾驶系统提供了新的技术路径。
Core Problem
现有自动驾驶模型多依赖单一模态或分离子系统，难以实现连续、鲁棒的场景理解与路径规划。扩散模型虽能生成高质量视觉内容，但在多模态融合和长序列时间一致性方面表现不足。多模态信息的异质性和时间依赖性使得模型难以同时优化视觉和动作预测，限制了系统在复杂环境中的泛化能力和实时性。如何设计一个统一的序列模型，融合视觉和动作信息，进行长时间跨度的场景预测和路径规划，是当前的核心难题。
Innovation
本文提出多模态驾驶语言，将图像和动作离散化为交错Token，形成统一词汇表，创新性地将世界建模与路径规划融合为序列预测任务。采用帧级旋转嵌入的Transformer架构，增强模型对时间和空间信息的表达能力，解决了传统扩散模型在连续性和多模态融合上的局限。模型实现端到端训练，兼顾视频生成和路径规划，显著提升多模态场景理解的效率和效果。这一创新突破了多模态融合的瓶颈，为自动驾驶的端到端系统提供了新思路。
Methodology
- �� 图像编码：利用预训练VQ-VAE将连续视频帧转化为离散Token，减少信息损失。
- �� 动作离散化：将相对位姿（∆x, ∆y, ∆θ）通过分箱离散化，形成动作Token。
- �� 统一多模态词汇：将图像Token和动作Token交错编码，形成多模态驾驶语言。
- �� 模型架构：采用Llama-like的Transformer模型，加入帧级旋转嵌入，进行下一Token预测。
- �� 训练目标：同时预测未来图像Token和动作Token，实现世界建模与路径规划的联合学习。
- �� 推理过程：通过采样生成连续视频和路径，结合离散Token解码还原场景。
Experiments
使用nuPlan和NAVSIM两个大规模数据集，训练模型以最大化下一Token的预测概率。评估指标包括视频质量（FID、FVD）和路径规划（PDMS）。在视频生成中，模型在长序列中表现出更好的连续性和多样性。在路径规划方面，模型在复杂场景中实现了较高的安全性和效率。对比不同视觉Tokenizers，选择LlamaGen作为最优方案。还进行了消融实验验证多模态融合的有效性和模型的泛化能力。
Results
DrivingGPT在nuPlan上FID值为12.78，优于SVD的24.03，FVD值为278.11，优于SVD的418.93。在路径规划中，模型在NAVSIM上实现了82.4%的PDMS得分，优于ResNet+MLP基线的77.8%。长序列视频生成中，模型在64帧视频中FVD为506.95，优于SVD的1079.28，表现出更强的时间一致性和内容丰富性。模型还有效缓解了对象幻觉问题，表现出更强的场景连续性和多样性。
Applications
该模型适用于自动驾驶中的场景预测、路径规划和虚拟仿真，为未来自主系统提供端到端解决方案。其多模态建模能力也可拓展至智能交通、虚拟现实等领域，提升系统的场景理解和决策能力。未来结合多传感器信息，将进一步增强模型的鲁棒性和实用性。
Limitations & Outlook
模型在极端复杂或未见场景下仍存在生成偏差，离散Token表达能力有限，可能导致细节缺失。训练依赖大量标注数据，泛化能力受数据偏差影响。推理时计算成本较高，长序列生成存在效率瓶颈。未来需优化模型结构，提升实时性和泛化能力。
Plain Language Accessible to non-experts
想象你在玩一个超级复杂的拼图游戏，每一块拼图代表一个场景的画面，每一步移动代表你的驾驶动作。传统方法就像只用一块拼图拼拼图，难以看到全局，也难以规划下一步。DrivingGPT就像有一个聪明的拼图助手，它能同时记住每一块拼图的细节和你每次的移动，把所有信息变成一种特殊的语言，然后用一个超级智能的机器人（Transformer）来预测下一块拼图应该放在哪里，以及你下一步该怎么走。这样，你不仅能拼出漂亮的画面，还能提前规划好未来的路线。它能在复杂的街景中，连续不断地拼出合理的场景和安全的路线，就像一个非常聪明的拼图大师一样。
Abstract
World model-based searching and planning are widely recognized as a promising path toward human-level physical intelligence. However, current driving world models primarily rely on video diffusion models, which specialize in visual generation but lack the flexibility to incorporate other modalities like action. In contrast, autoregressive transformers have demonstrated exceptional capability in modeling multimodal data. Our work aims to unify both driving model simulation and trajectory planning into a single sequence modeling problem. We introduce a multimodal driving language based on interleaved image and action tokens, and develop DrivingGPT to learn joint world modeling and planning through standard next-token prediction. Our DrivingGPT demonstrates strong performance in both action-conditioned video generation and end-to-end planning, outperforming strong baselines on large-scale nuPlan and NAVSIM benchmarks.