DriveWorld-VLA: Unified Latent-Space World Modeling with Vision-Language-Action for Autonomous Driving
DriveWorld-VLA unifies vision-language-action with world models in latent space, achieving 91.3 PDMS on NAVSIMv1 and 0.16 collision rate on nuScenes.
Key Findings
Methodology
DriveWorld-VLA employs multi-modal inputs—BEV features, text prompts, and historical actions—processed through a Vision-Language Model (VLM) to produce shared latent representations. The framework integrates a three-stage training: joint learning of scene prediction and action forecasting, action-conditioned future imagination via Diffusion Transformer, and reward-guided refinement. It models future scene evolution entirely in latent space, reducing pixel-level computations. The core innovation is using large language model (LLM) hidden states as a shared latent space, enabling causal “what-if” reasoning and controllable imagination, which supports proactive long-term planning. The system leverages a denoising diffusion process for scene prediction conditioned on actions, facilitating explicit causal inference and scenario simulation.
Key Results
- On NAVSIMv1, DriveWorld-VLA achieves 91.3 PDMS, outperforming previous methods like LAW and Epona, demonstrating superior scene prediction and planning accuracy.
- On NAVSIMv2, it reaches 86.8 EPDMS, with high scores in DAC (96.8%) and Lane Keeping (97.0%), confirming its long-term reasoning capabilities.
- On nuScenes, it attains a 0.16% collision rate, surpassing state-of-the-art approaches, indicating robustness and safety in complex real-world scenarios.
Significance
This work advances autonomous driving by integrating visual, linguistic, and action-based reasoning within a unified latent space, enabling long-term causal scene understanding and imagination. It addresses critical limitations of existing models that lack holistic scene modeling and causal inference, paving the way for safer, more reliable autonomous vehicles. The approach enhances interpretability, safety, and decision-making efficiency, marking a significant step toward fully autonomous, intelligent driving systems that can anticipate and adapt to complex traffic environments.
Technical Contribution
The paper introduces a novel architecture that embeds multi-modal information into a shared latent space using LLM hidden states, enabling end-to-end causal reasoning. It employs a Diffusion Transformer for future scene imagination conditioned on actions, supporting proactive planning. The three-stage training paradigm ensures stable joint optimization, integrating multi-modal perception, controllable scene generation, and reward-based refinement. This approach reduces pixel-level simulation costs and enhances long-term decision-making, representing a fundamental shift from traditional pixel-based models to efficient latent space reasoning.
Novelty
This is the first work to deeply unify vision-language-action models with world models in a latent space for autonomous driving. It leverages LLM hidden states as a shared representation, enabling causal “what-if” reasoning and controllable scene imagination. Unlike prior methods that treat scene prediction and planning separately, this approach integrates them into a single, end-to-end framework, significantly improving long-term planning and safety.
Limitations
- The model's performance degrades in highly complex or unseen scenarios, especially under adverse weather or dense traffic, due to limited generalization of learned causal relationships.
- Training requires large-scale multi-modal datasets and significant computational resources, which may hinder real-time deployment.
- Further work is needed to improve robustness, reduce computational costs, and extend reasoning capabilities to more diverse environments.
Future Work
Future research will focus on enhancing model robustness through self-supervised learning, reducing computational demands, and integrating reinforcement learning for adaptive decision-making. Expanding causal modeling to incorporate richer environmental factors and multi-agent interactions will be key. Additionally, efforts to improve real-time inference and deployment efficiency will accelerate practical adoption in autonomous vehicles.
AI Executive Summary
DriveWorld-VLA represents a significant leap in autonomous driving by unifying vision-language-action reasoning with world models within a shared latent space. Traditional approaches often rely on pixel-level scene simulation, which is computationally expensive and limited in long-term reasoning. This framework innovatively embeds multi-modal inputs—visual, linguistic, and historical actions—into a common latent representation via a large language model (LLM). Through a three-stage training process, it progressively enhances the model’s ability to predict future scenes, simulate multiple action outcomes, and refine decisions based on reward feedback.
The core technical breakthrough lies in employing a Diffusion Transformer to perform causal “what-if” reasoning, enabling the system to imagine future scenarios conditioned on candidate actions. This approach allows the vehicle to evaluate the long-term consequences of its decisions proactively, rather than reactively. Extensive evaluations on datasets such as NAVSIMv1, NAVSIMv2, and nuScenes demonstrate the model’s superiority, achieving 91.3 PDMS, 86.8 EPDMS, and a collision rate of only 0.16%. These results surpass existing state-of-the-art methods, confirming the effectiveness of the latent-space causal reasoning framework.
This research not only advances the theoretical understanding of scene modeling and decision-making but also has profound practical implications. It paves the way for safer, more reliable autonomous vehicles capable of complex reasoning and anticipation. Despite its achievements, challenges remain—particularly in handling highly complex scenarios and reducing computational costs. Future work aims to incorporate reinforcement learning, improve robustness, and extend reasoning to multi-agent environments, ultimately bringing fully autonomous, intelligent driving closer to reality.
Deep Analysis
Background
自主驾驶技术经历了从传感器感知到深度学习端到端系统的快速发展。早期如DriveNet、PilotNet等专注于传感器到控制的映射，但缺乏对未来场景的长远推理能力。近年来，世界模型（WM）成为研究热点，代表作如DriveWorld、Vista、LiDARDM等，旨在模拟环境动态和未来状态。多模态感知、场景理解和预测成为核心，推动复杂交通场景的理解与决策。尽管如此，现有模型多局限于短期预测或单一模态，难以实现长远、因果推理，限制了自主驾驶的安全性和鲁棒性。
Core Problem
当前自主驾驶模型在场景预测与决策中存在信息孤岛，缺乏对未来场景的整体理解。多模态信息未能充分融合，导致长远规划和因果推理能力不足。像素级模拟成本高，难以实现实时性。模型在复杂交互和未见场景中的泛化能力有限，影响安全性和鲁棒性。解决这一瓶颈，需在潜在空间中实现多模态融合与因果推理的统一架构，提升长远决策能力。
Innovation
提出DriveWorld-VLA架构，利用大规模语言模型（LLM）隐藏状态作为多模态共享潜在空间，增强信息融合。引入Diffusion Transformer进行未来场景的因果“假设”推理，支持多行动方案的场景模拟。采用三阶段训练：联合学习、动作微调和未来引导优化，确保模型稳定性和长远推理能力。创新点在于在潜在空间中实现因果关系的端到端建模，减少像素级滚动，提升效率与可控性。这一架构突破了传统的外部模拟和特征共享限制，开启自主驾驶中潜在空间建模的新范式。
Methodology
- �� 输入多模态信息：BEV特征、文本提示、历史动作，经过专用tokenizer编码。
- �� 利用VLM模型融合多模态信息，提取潜在表示H_t。
- �� 联合训练阶段：在潜在空间中同时学习未来场景的BEV预测和动作预测。
- �� 未来场景预测：采用Denoiser和Diffusion Transformer，基于潜在表示生成未来BEV状态。
- �� 动作预测：通过动作解码器预测未来轨迹，结合监督学习。
- �� 微调阶段：引入行动条件的未来“假设”推理，学习因果关系。
- �� 未来引导阶段：利用奖励函数评估未来场景，优化行动生成。
- �� 最终模型实现多模态融合、因果推理和长远规划的统一。
Experiments
在NAVSIMv1、NAVSIMv2和nuScenes上进行评估。使用PDMS、EPDMS和碰撞率作为主要指标。训练采用AdamW优化器，分别在8块GPU上进行约120和93小时。对比多种最先进方法，验证DriveWorld-VLA在长远推理、场景预测和安全性方面的优越性。还进行了消融实验，验证三阶段训练策略的重要性。
Results
模型在NAVSIMv1达91.3 PDMS，优于Law和Epona等方法。在NAVSIMv2取得86.8 EPDMS，表现优异，特别在DAC（96.8%）和LK（97.0%）指标上。nuScenes上实现0.16%的碰撞率，优于现有主流方法。这些结果验证了潜在空间融合和因果推理的有效性，显著提升自主驾驶的安全性和效率。
Applications
该模型适用于自动驾驶车辆的长远决策、场景理解和安全控制。需要多模态传感器输入和高性能计算平台。未来可扩展至自动驾驶辅助系统，提高交通安全和效率，推动智能交通系统发展。
Limitations & Outlook
模型在极端复杂场景和极端天气条件下表现尚不理想，推理偏差可能导致安全风险。训练成本高，部署难度大。未来需增强模型的泛化能力和鲁棒性，降低计算资源需求。
Plain Language Accessible to non-experts
想象你在一个工厂工作，工厂每天都在生产不同的产品。工厂里的机器人需要知道未来会发生什么，才能提前准备。传统的方法就像让机器人只看一眼就做决定，不能预测未来的变化。而这项新技术就像给机器人装上了一个“未来预知”装置，它可以在脑海中模拟未来的场景，并根据这些想象做出更聪明的决定。它在潜在空间里建立了一个“工厂模型”，可以想象不同的生产线变化，提前预料到可能的问题，从而让工厂运转得更顺畅、更安全。这就像你在玩一款策略游戏，提前模拟几步，知道哪个策略最有效，然后再去执行。这样，机器人就能更好地应对复杂的交通环境，避免碰撞，提前做出反应，确保安全。这个技术让自动驾驶变得更聪明、更可靠，就像给汽车装上了“未来预知”的大脑一样。
ELI14 Explained like you're 14
想象你在玩一款超级复杂的策略游戏，你不仅要看眼前的情况，还要预测未来会发生什么，然后提前准备。传统的自动驾驶系统就像只看眼前，反应慢，容易出错。而这项新技术就像给汽车装上了一个“未来预知器”，它可以在脑海里模拟未来的场景，想象不同的交通情况，看看哪个行动会更安全、更快。它用一种特别的“潜在空间”来存放所有的场景信息，就像你在脑海中建了一个虚拟的城市模型。通过不断学习和训练，这个模型可以理解交通规则、预测其他车辆的行为，还能想象各种可能的未来。这样，汽车就能提前知道危险，提前做出反应，就像你在游戏中提前布局一样。这让自动驾驶变得更聪明、更安全，就像拥有了未来预知的超级大脑！
Abstract
End-to-end (E2E) autonomous driving has recently attracted increasing interest in unifying Vision-Language-Action (VLA) with World Models to enhance decision-making and forward-looking imagination. However, existing methods fail to effectively unify future scene evolution and action planning within a single architecture due to inadequate sharing of latent states, limiting the impact of visual imagination on action decisions. To address this limitation, we propose DriveWorld-VLA, a novel framework that unifies world modeling and planning within a latent space by tightly integrating VLA and world models at the representation level, which enables the VLA planner to benefit directly from holistic scene-evolution modeling and reducing reliance on dense annotated supervision. Additionally, DriveWorld-VLA incorporates the latent states of the world model as core decision-making states for the VLA planner, facilitating the planner to assess how candidate actions impact future scene evolution. By conducting world modeling entirely in the latent space, DriveWorld-VLA supports controllable, action-conditioned imagination at the feature level, avoiding expensive pixel-level rollouts. Extensive open-loop and closed-loop evaluations demonstrate the effectiveness of DriveWorld-VLA, which achieves state-of-the-art performance with 91.3 PDMS on NAVSIMv1, 86.8 EPDMS on NAVSIMv2, and 0.16 3-second average collision rate on nuScenes. Code and models will be released in https://github.com/liulin815/DriveWorld-VLA.git.