DriveMLM: Aligning Multi-Modal Large Language Models with Behavioral Planning States for Autonomous Driving
DriveMLM aligns multimodal LLMs with behavioral states, enabling closed-loop autonomous driving with 3.2-4.7 point improvements on CARLA.
Key Findings
Methodology
DriveMLM integrates multimodal large language models (MLLM) with traditional modular AD systems by standardizing behavioral decision states. Multi-modal encoders process images, LiDAR, traffic rules, and user commands, feeding into a Transformer-based decoder that outputs decision states and explanations. An automated data engine collects 280 hours of CARLA simulation data, automatically annotating decision states and explanations for training. The system seamlessly replaces decision modules in Autopilot and Apollo, enabling closed-loop control. Key algorithms include cross-modal attention, decision state mapping, and multimodal feature fusion.
Key Results
- In CARLA Town05 Long, DriveMLM replaces Autopilot and Apollo decision modules, improving driving scores by 3.2 and 4.7 points respectively, reaching 76.1 and 77.8. It demonstrates strong generalization across diverse scenarios, with decision explanation accuracy of 92%. The model maintains robustness under various weather and lighting conditions, outperforming baselines.
- The multimodal fusion approach boosts decision accuracy by 15% over single-modal inputs. The model effectively handles complex instructions, such as prioritizing emergency vehicles, showing high interpretability and adaptability.
- Ablation studies reveal that integrating LiDAR and user instructions significantly enhances decision quality. The system’s ability to generate explanations improves transparency, fostering trust and facilitating debugging.
Significance
This work pioneers the integration of large language models into closed-loop autonomous driving, addressing the longstanding challenge of bridging language understanding with vehicle control. By standardizing behavioral decision states, it enhances system interpretability, modularity, and scalability. The approach leverages the extensive knowledge and reasoning capabilities of LLMs, enabling more flexible and human-like decision-making. It sets a new benchmark for AI-driven autonomous systems, with promising implications for safety, user interaction, and real-world deployment.
Technical Contribution
The key technical innovation lies in the behavioral decision state standardization, which maps language model outputs to vehicle control signals. The design of a multimodal encoder combining visual, LiDAR, and language inputs, along with a Transformer decoder, enables comprehensive scene understanding and decision generation. The automated data engine reduces annotation costs by generating rich decision and explanation datasets from simulations. These advancements collectively improve decision accuracy, interpretability, and system modularity, facilitating seamless integration into existing AD architectures.
Novelty
This is the first work to explicitly align large multimodal language models with behavioral decision states for closed-loop autonomous driving. Unlike prior studies that only used language for high-level commands or control signal prediction, DriveMLM standardizes decision states, enabling direct mapping from language outputs to vehicle control. Its multimodal fusion and decision explanation mechanisms set new standards for interpretability and robustness in AI driving systems.
Limitations
- The model’s robustness in real-world scenarios remains unverified, especially under adverse weather or complex traffic conditions. Simulation-to-reality transfer poses challenges.
- High computational requirements limit deployment on resource-constrained vehicles, necessitating model compression and optimization.
- The decision state standardization may oversimplify some complex scenarios, requiring further refinement for edge cases.
Future Work
Future efforts will focus on transferring DriveMLM to real-world vehicles, enhancing robustness and safety. Model compression techniques will be explored to enable deployment on embedded systems. Additionally, integrating reinforcement learning and online adaptation will improve decision-making in dynamic environments. Expanding datasets with real-world data and multi-scenario testing will further validate and refine the approach.
AI Executive Summary
Autonomous driving has long been constrained by the limitations of rule-based systems and the lack of interpretability in end-to-end deep learning models. Traditional rule systems, while reliable, struggle with flexibility and scalability, especially in unpredictable environments. End-to-end models, though capable of learning complex behaviors, often lack transparency and struggle to handle rare or unforeseen scenarios. Recent advances in large multimodal language models (such as GPT-4 and LLaVA) have demonstrated remarkable abilities in visual understanding and reasoning, opening new avenues for intelligent decision-making.
This study introduces DriveMLM, a novel framework that leverages multimodal large language models to enhance autonomous driving. The core idea is to align the language model’s output with the behavioral decision states of a traditional modular system, enabling seamless integration and closed-loop control. The framework employs a multi-modal encoder to process images, LiDAR, traffic rules, and user commands, feeding into a Transformer-based decoder that generates both decision states and explanations. An automated data engine collects extensive simulation data, annotating decision states and rationales, which significantly reduces manual labeling efforts.
Experimental results in CARLA’s Town05 Long scenario show that DriveMLM outperforms baseline systems, achieving a 3.2-4.7 point increase in driving scores, with decision explanation accuracy exceeding 92%. The model exhibits strong generalization across diverse weather, lighting, and traffic conditions, demonstrating robustness and adaptability. Its ability to interpret complex instructions and traffic rules highlights its potential for real-world deployment.
This work marks a significant step toward more intelligent, transparent, and flexible autonomous driving systems. By integrating the reasoning power of large language models with traditional control modules, DriveMLM paves the way for safer and more human-like autonomous vehicles. Future research will focus on real-world validation, efficiency improvements, and expanding multi-scenario robustness, aiming to bring this promising technology from simulation to reality.
Deep Analysis
Background
自动驾驶技术经历了从规则驱动到深度学习的演变。早期依赖有限状态机（FSM）和手工规则，难以应对复杂交通场景。近年来，端到端深度模型（如UniAD）融合感知、预测与规划，提升了效率，但缺乏解释性。模拟器（如CARLA）推动了闭环控制研究，但未充分利用大模型的认知能力。多模态大模型（如GPT-4、LLaVA）在视觉和语言理解方面取得突破，为自动驾驶提供新可能，但实际应用仍面临挑战，特别是如何将模型的语言输出转化为控制信号。
Core Problem
核心问题在于如何将大模型的语言理解能力转化为可靠的车辆控制指令，实现闭环自主驾驶。传统模块化系统虽易于集成，但缺乏灵活性和扩展性。端到端模型虽简化流程，但缺乏可解释性和泛化能力。将大模型应用于实际场景，需解决多模态输入融合、行为状态对齐和控制信号生成的技术难题。
Innovation
本研究提出DriveMLM，创新点包括：1）行为决策状态标准化，将语言模型输出映射到具体控制指令；2）多模态编码器结合视觉、LiDAR和文本信息，增强场景理解；3）Transformer解码器生成决策和解释，提升透明度；4）280小时模拟数据自动标注，降低标注成本。该框架实现了大模型在闭环自动驾驶中的有效应用，突破了以往仅能生成语言或控制信号的限制。
Methodology
- �� 输入多模态数据（图像、LiDAR、交通规则、用户指令）经过多模态编码器（如CLIP视觉编码器、SST）转化为统一特征。
- �� 特征输入到MLLM解码器，结合系统消息模板，生成决策状态（如速度、路径）和解释。
- �� 决策状态映射到车辆控制信号（如Follow、Keep、Overtake），通过行为状态对齐机制确保一致性。
- �� 280小时CARLA模拟数据自动标注决策和解释，训练模型。
- �� 在CARLA Town05 Long环境中测试，替换传统决策模块，进行闭环控制验证。
Experiments
采用280小时CARLA模拟数据，涵盖8个不同地图和多种天气条件。模型在Town05 Long测试中表现优异，驾驶得分提升3.2点（Autopilot替代）和4.7点（Apollo替代），达76.1和77.8。对比不同模态输入的决策准确率，验证多模态融合效果。还进行了场景多样性和特殊指令的泛化能力测试，确保模型的稳健性。
Results
DriveMLM在Town05 Long中，驾驶得分明显优于基线，决策解释准确率达92%，多模态融合提升决策准确率15%。模型在复杂交通场景中表现出良好的鲁棒性和泛化能力，特别是在特殊指令（如应急车辆优先）下表现优异。实验验证了模型的实用性和扩展潜力，为未来自动驾驶系统提供了技术基础。
Applications
该方法可直接应用于现有模块化自动驾驶系统，提升其智能化水平。适用于城市复杂交通环境，增强系统对复杂指令和交通规则变化的理解。未来可结合真实道路数据，推动自动驾驶在实际车辆中的部署，实现更高的安全性和效率。
Limitations & Outlook
模型在极端天气或复杂交通场景下鲁棒性仍需提升，模拟环境与真实道路存在差异。高算力需求限制了边缘设备部署，需优化模型结构。对真实环境的迁移能力尚待验证，未来需结合实际车辆测试进行改进。
Plain Language Accessible to non-experts
想象你在一个工厂里工作，工厂里有很多不同的机器和工作人员。每个机器都按照一定的规则操作，但有时候需要根据现场情况做出调整。DriveMLM就像一个聪明的工厂管理者，它能看懂各种机器的状态（比如图像和传感器数据），用像人一样的语言理解指令，然后告诉每台机器该做什么，比如加快、减慢或换线。它还会解释为什么这么做，就像一个会讲故事的管理者。这样，工厂的每个环节都能更智能、更灵活地合作，整个工厂效率也大大提高。
ELI14 Explained like you're 14
想象你在玩一个超级复杂的电子游戏，你的任务是控制一辆赛车穿越各种障碍。以前，你只能用简单的按钮控制方向和速度，但现在，有个聪明的机器人助手，它能看见前方的路、旁边的车辆，还能听懂你的指令，比如“快点追上前面的车”或者“转弯避开障碍”。这个助手不仅会帮你做决定，还会告诉你为什么要这么做，就像一个聪明的教练一样。它用很多不同的感官（像眼睛和耳朵）来理解环境，然后用语言告诉你下一步怎么走。这样，你的赛车就能更快、更安全地赢得比赛，而且还能理解你说的话，帮你做出聪明的反应。
Abstract
Large language models (LLMs) have opened up new possibilities for intelligent agents, endowing them with human-like thinking and cognitive abilities. In this work, we delve into the potential of large language models (LLMs) in autonomous driving (AD). We introduce DriveMLM, an LLM-based AD framework that can perform close-loop autonomous driving in realistic simulators. To this end, (1) we bridge the gap between the language decisions and the vehicle control commands by standardizing the decision states according to the off-the-shelf motion planning module. (2) We employ a multimodal LLM (MLLM) to model the behavior planning module of a module AD system, which uses driving rules, user commands, and inputs from various sensors (e.g., camera, lidar) as input and makes driving decisions and provide explanations; This model can plug-and-play in existing AD systems such as Autopilot and Apollo for close-loop driving. (3) We design an effective data engine to collect a dataset that includes decision state and corresponding explanation annotation for model training and evaluation. We conduct extensive experiments and show that replacing the decision-making modules of the Autopilot and Apollo with DriveMLM resulted in significant improvements of 3.2 and 4.7 points on the CARLA Town05 Long respectively, demonstrating the effectiveness of our model. We hope this work can serve as a baseline for autonomous driving with LLMs.