DriveGPT4: Interpretable End-to-end Autonomous Driving via Large Language Model
DriveGPT4 leverages multimodal large language models for end-to-end interpretable autonomous driving, trained on BDD-X dataset, outperforming existing methods with significant improvements in control and explanation tasks.
Key Findings
Methodology
DriveGPT4 adopts a multimodal architecture based on LLaMA2, integrating a dedicated video tokenizer (Valley) and CLIP visual encoder to extract comprehensive features from multi-frame videos. The model fuses temporal and spatial features through concatenation and pooling, then projects these features into the text domain using a specially designed projector. It employs a two-stage training process: initial large-scale pretraining on WebVid-2M and CC3M datasets for general video-text alignment, followed by domain-specific fine-tuning on a combined dataset of 56K autonomous driving instruction-following samples (BDD-X + ChatGPT generated) and 223K general instruction data (LLaVA, Valley). The model generates both natural language explanations (vehicle actions, reasons) and low-level control signals (speed, steering angle) in a unified framework, leveraging the same tokenizer for text and control signals, which are decoded into human-readable formats. The training emphasizes diversity in instruction data to improve robustness and interpretability.
Key Results
- On the BDD-X test set, DriveGPT4 achieves a CIDEr score of 113.20 for description and explanation tasks, surpassing the baseline ADAPT (100.93) and Video-LLaMA (10.31). For control signal prediction, the speed RMSE drops to 1.30, significantly better than ADAPT’s 3.02, demonstrating high accuracy in complex scenarios. In the 'Hard' split, the CIDEr score reaches 57.29, indicating robust understanding in challenging environments.
- By integrating extensive instruction tuning data, including 56K autonomous driving samples and 223K general instruction-following data, DriveGPT4 demonstrates superior question-answering accuracy, with BLEU4 scores averaging 18.32, outperforming baselines. The model maintains high performance across different scene complexities, validating its generalization and robustness.
- The model's multi-task capability enables it to generate detailed vehicle behavior descriptions, justifications, and precise control signals simultaneously, facilitating transparent and trustworthy autonomous driving systems. Ablation studies confirm that multimodal feature fusion and instruction tuning are crucial for performance gains.
Significance
This work pioneers the integration of large multimodal language models into end-to-end autonomous driving systems with explicit interpretability. By enabling vehicles to explain their actions and predict controls in natural language, DriveGPT4 addresses critical issues of trust, safety, and transparency that have hindered commercial deployment of autonomous vehicles. Its ability to handle complex traffic scenarios with high accuracy and provide human-understandable reasoning marks a significant step forward in making autonomous driving more accessible, reliable, and ethically sound. The approach also opens new avenues for human-AI interaction in intelligent transportation, fostering broader acceptance and regulatory support.
Technical Contribution
DriveGPT4 introduces a novel multimodal architecture that combines a dedicated video tokenizer, a pre-trained LLaMA2 model, and a visual instruction tuning dataset specifically crafted for autonomous driving. The key innovations include the unified handling of visual features and control signals within a language modeling framework, the use of a text-based control signal representation for interpretability, and a two-stage training strategy that balances general visual understanding with domain-specific knowledge. The model’s ability to generate both natural language explanations and precise control signals in a single inference pass represents a significant technical advancement, enabling end-to-end interpretability and interaction.
Novelty
This research is the first to embed large multimodal language models directly into an end-to-end autonomous driving pipeline with explicit interpretability. Unlike prior works limited to visual explanations or rule-based systems, DriveGPT4 leverages a unified text-based representation for vehicle control signals, enabling natural language explanations of actions and reasoning. Its training on a combination of large-scale general data and domain-specific instruction data, along with the innovative use of visual tokenization, sets a new standard for transparency and robustness in autonomous driving AI. This approach bridges the gap between high-performance control and human-understandable reasoning, marking a pioneering step in the field.
Limitations
- Despite its advances, DriveGPT4's performance may degrade in scenarios involving extreme weather conditions (e.g., heavy fog, snow) or highly unpredictable traffic behaviors, due to limited training data in such environments.
- The computational cost of processing multi-frame videos with large models remains high, potentially limiting real-time deployment on resource-constrained platforms.
- The reliance on synthetic instruction data generated via ChatGPT, while effective, may introduce biases or gaps that could affect robustness in real-world scenarios. Further efforts are needed to incorporate more diverse and realistic data sources.
Future Work
未来将扩展多模态数据源，融合激光雷达、雷达等传感器信息，以增强在复杂环境中的感知能力。计划引入强化学习和模拟平台（如CARLA）进行自主决策训练，提升模型的鲁棒性和适应性。同时，将探索模型的可解释性增强机制，使其不仅能描述行为，还能提供更深入的决策依据。未来还将关注模型在极端天气和未见场景下的表现，优化其泛化能力和安全性，推动其在实际交通中的应用落地。
AI Executive Summary
自动驾驶技术近年来取得了巨大突破，但其核心难题之一在于系统的透明性和可解释性。传统深度学习模型虽然在控制精度上表现优异，但作为黑箱系统，难以让人理解其决策过程，限制了其在实际应用中的信任度。DriveGPT4的提出，正是为了填补这一空白。该系统基于最新的多模态大语言模型（如LLaMA2），结合专门设计的视觉指令调优数据集，实现了车辆行为的自然语言描述、推理和控制预测的统一。通过引入视频编码和控制信号文本化技术，DriveGPT4不仅在BDD-X数据集上取得了优异的性能，还在复杂交通场景中表现出强大的理解和控制能力。
模型的训练分为两个阶段：首先利用WebVid-2M和CC3M等大规模通用视频-文本数据进行预训练，确保模型具备广泛的视觉和语言理解能力；随后，结合BDD-X和ChatGPT生成的指令调优数据进行微调，强化模型在自主驾驶场景中的表现。实验结果显示，DriveGPT4在车辆动作描述、理由生成和控制信号预测方面均优于现有的最先进方法（如ADAPT），特别是在“Hard”场景中表现尤为出色。这一突破不仅验证了多模态融合和指令调优的有效性，也为未来智能驾驶系统的透明化和人机交互提供了新思路。
从应用角度看，DriveGPT4的可解释性极大增强了用户信任，有助于解决自动驾驶的伦理和安全问题。未来，模型将融合更多传感器信息，结合强化学习和模拟训练，进一步提升在极端环境下的表现。随着技术的不断演进，DriveGPT4有望成为实现安全、透明、智能自动驾驶的关键技术之一，推动自动驾驶从实验室走向商业化，开启智慧交通新时代。
Deep Analysis
Background
自动驾驶技术经历了从传统规则驱动到深度学习的演变。早期系统依赖于复杂的感知、规划和控制模块，面临环境复杂性带来的局限。近年来，端到端深度学习模型（如Behavior Cloning、End-to-End CNN）逐渐崭露头角，显著提升了控制精度，但缺乏可解释性，难以满足行业对安全和透明的需求。多模态学习和大语言模型（如GPT、LLaMA）在自然语言处理和视觉理解中取得突破，为自主驾驶带来新的可能。已有研究尝试结合视觉和语言，生成车辆状态描述和行为推理（如BDD-X、LLaVA），但多为单模态或局部解释，缺乏端到端的整体解决方案。DriveGPT4正是在此背景下，融合多模态视频编码、预训练大模型和指令调优，提出了一种具有强泛化能力和良好交互性的端到端可解释自主驾驶系统。
Core Problem
现有自主驾驶系统多为模块化设计，系统复杂、集成难度大、缺乏全局理解。端到端模型虽简化流程，但黑箱特性限制了其在实际中的信任度和调试能力。如何实现系统的透明化，既保证高性能，又能提供自然语言的行为解释，是行业亟待解决的核心难题。此外，现有解释方法多为视觉化或预定义规则，难以满足多样化用户需求。DriveGPT4试图通过大语言模型实现多模态数据的理解、行为的自然语言描述和预测，解决这一瓶颈。
Innovation
DriveGPT4的创新点包括：1）引入多模态视频编码技术（Valley视频Tokenizer + CLIP视觉编码器），实现多帧视频的高效特征提取；2）结合大规模预训练模型（LLaMA2）和自主驾驶视觉指令调优数据集，增强模型的领域适应性和交互能力；3）将控制信号文本化，使车辆控制预测成为自然语言输出，极大提升可解释性；4）采用混合微调策略，结合通用指令调优数据和自主驾驶特定数据，增强模型的泛化能力和鲁棒性。这些创新共同推动多模态大语言模型在自主驾驶中的应用边界，突破传统黑箱模型的局限。
Methodology
- �� 视频输入：采样8帧视频，利用Valley视频Tokenizer将每帧转换为文本域的特征序列，提取全局和局部特征。
- �� 特征融合：将所有时间帧的全局特征拼接（T = F G0 ⊕ F G1 ⊕ ... ⊕ F G N），局部特征通过池化（Pooling）整合为空间特征（S）。
- �� 特征投影：将时间和空间特征通过专门设计的投影器（Projector）映射到文本域，结合车辆速度和转向角信息，形成模型输入。
- �� 语言解码：利用LLaMA2模型对融合特征进行解码，生成车辆行为描述、理由和控制信号。
- �� 训练流程：第一阶段为大规模视频-文本对齐预训练（利用WebVid-2M和CC3M），保持模型的通用视觉和语言理解能力；第二阶段为结合BDD-X和ChatGPT生成的指令调优数据的混合微调，强化模型在自主驾驶场景中的表现。
- �� 交互问答：模型能同时回答用户关于车辆状态、行为理由的自然语言问题，并预测下一步控制信号，实现端到端的可解释自主驾驶。
Experiments
实验采用BDD-X数据集，包含约2万条视频片段，分为训练和测试集。模型性能通过多项指标评估，包括CIDEr、BLEU4、ROUGE-L以及控制信号的RMSE和AUC。对比基线包括ADAPT、Video-LLaMA等，DriveGPT4在描述、理由和控制预测任务中均取得优异成绩，特别是在“Hard”场景中表现出色。模型还进行了消融实验，验证多模态特征融合和指令调优的贡献。通过不同场景的分组，评估模型在复杂交通环境中的鲁棒性和泛化能力。所有实验均在高性能GPU集群上完成，确保推理速度满足实际应用需求。
Results
DriveGPT4在BDD-X测试集上，描述和解释任务的CIDEr得分达113.20，明显优于ADAPT（100.93）和Video-LLaMA（10.31）；在车辆控制信号预测中，速度的RMSE降至1.30，优于基线的3.02，显示出更高的预测精度。模型在“Hard”场景下表现尤为突出，CIDEr达57.29，验证其在复杂交通环境中的理解和控制能力。问答任务中，BLEU4得分达18.32，反映出模型在多样化提问中的适应性和准确性。消融实验显示，结合多模态特征和指令调优显著提升了模型性能，验证了设计策略的有效性。
Applications
DriveGPT4可应用于智能驾驶辅助、自动驾驶决策支持和交通管理系统。其自然语言交互能力使得驾驶员或调度员可以实时获取车辆状态、行为解释和未来控制预测，提升系统透明度和信任度。模型还可作为自动驾驶测试平台的核心组件，用于验证和优化车辆行为。未来，结合多传感器信息和强化学习，DriveGPT4有望实现更高层次的自主决策，推动智能交通的普及。
Limitations & Outlook
模型在处理极端复杂交通场景（如突发事故、多源传感器融合不足）时仍存在一定的误判和控制偏差，主要由于训练数据的局限性和模型容量限制。当前模型对多帧视频的处理受限于硬件资源，长时间或高帧率视频输入会导致推理速度下降，影响实时性。模型在面对未见过的交通环境或特殊天气条件（如大雾、暴雨）时，表现尚不稳定，需进一步增强鲁棒性和泛化能力。
Plain Language Accessible to non-experts
想象你在一个大型厨房里准备一道复杂的菜肴。每次你都需要看着食材，判断下一步怎么做，比如什么时候放盐、什么时候炒菜。传统的厨师就像是一本死板的食谱，只告诉你步骤，但你不知道为什么要这么做，也不能根据现场情况灵活调整。而DriveGPT4就像是一个聪明的厨师助手，它可以观察厨房里的所有食材和火候，用普通话告诉你：‘你需要多放一点盐，因为味道还不够’；还能预测下一步应该加什么调料，甚至解释为什么要这样做。它还能回答你各种问题，比如‘为什么要先炒葱？’或者‘下一步我该怎么做？’用简单的话语让你明白整个烹饪过程。这样一来，你不仅能做出更好吃的菜，还能学到为什么要这样做，就像有了一个会讲故事、会解释的聪明厨师助手，让你在厨房里既开心又懂得道理。
ELI14 Explained like you're 14
想象你在玩一个超级复杂的积木游戏，你不仅要把积木搭得稳，还要知道每块积木为什么要放在那儿，以及下一块应该怎么放。这就像开车一样，传统的方法就像是告诉你每一步怎么做，但你不知道为什么这么做，也不知道下一步会发生什么。而DriveGPT4就像是一个聪明的朋友，它可以看着你搭积木的过程，用简单的话告诉你：‘你把红色积木放在这里，是因为它可以让整体更稳固’，还可以预测下一块积木应该放在哪里，帮你搭得更快更稳。它还能听你问问题，比如‘为什么我需要这样放积木？’或者‘下一步我应该怎么做？’，用自然的语言回答你。这样一来，你就能更懂得为什么要这样做，也能更自信地完成搭建任务。这就像是给你一个会讲故事、会解释的超级助手，让你在玩积木时既开心又明白所有的道理。
Abstract
Multimodal large language models (MLLMs) have emerged as a prominent area of interest within the research community, given their proficiency in handling and reasoning with non-textual data, including images and videos. This study seeks to extend the application of MLLMs to the realm of autonomous driving by introducing DriveGPT4, a novel interpretable end-to-end autonomous driving system based on LLMs. Capable of processing multi-frame video inputs and textual queries, DriveGPT4 facilitates the interpretation of vehicle actions, offers pertinent reasoning, and effectively addresses a diverse range of questions posed by users. Furthermore, DriveGPT4 predicts low-level vehicle control signals in an end-to-end fashion.These advanced capabilities are achieved through the utilization of a bespoke visual instruction tuning dataset, specifically tailored for autonomous driving applications, in conjunction with a mix-finetuning training strategy. DriveGPT4 represents the pioneering effort to leverage LLMs for the development of an interpretable end-to-end autonomous driving solution. Evaluations conducted on the BDD-X dataset showcase the superior qualitative and quantitative performance of DriveGPT4. Additionally, the fine-tuning of domain-specific data enables DriveGPT4 to yield close or even improved results in terms of autonomous driving grounding when contrasted with GPT4-V.
References (20)
ADAPT: Action-aware Driving Caption Transformer
Bu Jin, Xinyi Liu, Yupeng Zheng et al.
Valley: Video Assistant with Large Language Model Enhanced Ability
Ruipu Luo, Ziwang Zhao, Min Yang et al.
Visual Instruction Tuning
Haotian Liu, Chunyuan Li, Qingyang Wu et al.
Multimodal End-to-End Autonomous Driving
Yi Xiao, Felipe Codevilla, A. Gurram et al.
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Jacob Devlin, Ming-Wei Chang, Kenton Lee et al.
CenterLineDet: Road Lane CenterLine Graph Detection With Vehicle-Mounted Sensors by Transformer for High-definition Map Creation
Zhenhua Xu, Yuxuan Liu, Yuxiang Sun et al.
InsightMapper: A Closer Look at Inner-instance Information for Vectorized High-Definition Mapping
Zhenhua Xu, K. Wong, Hengshuang Zhao
Textual Explanations for Self-Driving Vehicles
Jinkyu Kim, Anna Rohrbach, Trevor Darrell et al.
Learning Attraction Field Representation for Robust Line Segment Detection
Nan Xue, S. Bai, Fudong Wang et al.
nuScenes: A Multimodal Dataset for Autonomous Driving
Holger Caesar, Varun Bankiti, Alex H. Lang et al.
Grounding Human-To-Vehicle Advice for Self-Driving Vehicles
Jinkyu Kim, Teruhisa Misu, Yi-Ting Chen et al.
Pyramid Scene Parsing Network
Hengshuang Zhao, Jianping Shi, Xiaojuan Qi et al.
Talk2Car: Taking Control of Your Self-Driving Car
Thierry Deruyttere, Simon Vandenhende, Dusan Grujicic et al.
5分で分かる!? 有名論文ナナメ読み：Jacob Devlin et al. : BERT : Pre-training of Deep Bidirectional Transformers for Language Understanding
知秀 柴田
Language
Autonomous cars: Recent developments, challenges, and possible solutions
Sehaj P. Singh, B. Saini
Learning Transferable Visual Models From Natural Language Supervision
Alec Radford, Jong Wook Kim, Chris Hallacy et al.
The Role of the Hercules Autonomous Vehicle During the COVID-19 Pandemic: An Autonomous Logistic Vehicle for Contactless Goods Transportation
Tianyu Liu, Qinghai Liao, Lu Gan et al.
Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrieval
Max Bain, Arsha Nagrani, Gül Varol et al.
Learning Interpretable End-to-End Vision-Based Motion Planning for Autonomous Driving with Optical Flow Distillation
Hengli Wang, Peide Cai, Yuxiang Sun et al.
Cited By (20)
Event-Triggered MPC With Linear Inter-Event Control for AV Path Tracking
A Large Language Model-Based Game Equilibrium Selection Approach for Human-Machine Shared Driving
PriorEye: Geospatial Visual Priors for End-to-End Autonomous Driving
Long-term Traffic Simulation via Structured Autoregressive Modeling
OWMDrive: Causality-Aware End-to-End Autonomous Driving via 4D Occupancy World Model
EVLA: An Electro-Aware Multimodal Assistant for Physically-Grounded Driving Reasoning and Control
X-Mind: Efficient Visual Chain-of-Thought via Predictive World Model for End-to-End Driving
Reasoning-aware Speculative Decoding for Efficient Vision-Language-Action Models in Autonomous Driving
Overthink-Triggered Slowdown Attacks on LVLM-Based Robotic Systems
Camera–Radar Data-Based Vision Language Model for Automotive Driving Scene Description
GLERO: Graph-LLM Enhanced Reward Optimization for mixed-traffic CAV control
Foundation models for autonomous driving: A comprehensive survey
Benchmarking the Robustness of Autonomous Driving to Environmental Illusions: A Lane Perception Perspective.
AnchorVLA: Bridging Discrete Decisions and Continuous Trajectories for Vision-Language-Action Planning
PixelPilot: Scalable Vision-Language-Action Models for End-to-End Autonomous Driving
EECA: a design-oriented explainability framework for end-to-end cockpit agents
WCog-VLA: A Dual-Level World-Cognitive Vision-Language-Action Model for End-to-End Autonomous Driving
OpenLongTail: Generative Scaling of Long-Tail Driving Data
Can the Cloud Drive? Infrastructure Feasibility of Offloading Autonomous Driving Across 5G and 6G
STGraphVQA: spatial-temporal graph reasoning with hierarchical cognition for interpretable driving scene understanding