UNIVERSE: Unified Video Action Models for Autonomous Driving with Flexible Mask-Modulated Modality Generation
UNIVERSE employs a single mask-modulated Diffusion Transformer for joint video and trajectory prediction, achieving 4.3× inference speedup and state-of-the-art generalization.
Key Findings
Methodology
UNIVERSE utilizes a unified Diffusion Transformer (DiT) with mask modulation to jointly train future video latents and ego-trajectory tokens. Dense supervision from video guides trajectory denoising directly. A Modality-Decoupling Visibility Mask blocks mutual attention between future video and trajectory tokens, ensuring causality. During inference, removing future video denoising enables trajectory-only prediction, increasing speed by 4.3×. The training employs Flow Matching loss for both modalities, promoting dynamic scene understanding.
Key Results
- On NAVSIM, UNIVERSE achieves 91.0 PDMS, surpassing Two-DiT's 89.6, and demonstrates strong zero-shot transfer to nuScenes and Bench2Drive with no fine-tuning. It maintains high planning accuracy and generates consistent future videos, validated by FID and FVD metrics.
- Inference speed is improved by 4.3× when performing trajectory-only rollout, with minimal performance loss. Ablation studies confirm the importance of shared DiT, co-training, and mask decoupling for robust generalization.
- The model exhibits superior zero-shot transfer, reducing L2 displacement errors to 0.83m on nuScenes and 1.33m on Bench2Drive, with collision rates as low as 0.03%, demonstrating strong cross-domain adaptability.
Significance
This work advances autonomous driving by integrating dense scene dynamics modeling directly into trajectory prediction, enabling better generalization to unseen environments. The unified architecture simplifies deployment, reduces inference latency, and enhances safety. Its zero-shot capabilities significantly lower adaptation barriers, facilitating real-world deployment. The approach bridges the gap between scene understanding and action planning, setting a new standard for scalable, interpretable, and efficient autonomous systems.
Technical Contribution
The core innovation is a single shared DiT architecture with mask-based causal decoupling, allowing joint training of video and trajectory modalities. This design ensures causal integrity and efficient inference, supporting trajectory-only and video-only rollouts. The use of Flow Matching loss for dense supervision enhances dynamic scene modeling. The model’s ability to remove future video denoising at inference without performance degradation is a key engineering breakthrough, enabling real-time deployment.
Novelty
This is the first work to unify future video prediction and action generation within a single diffusion transformer, employing a mask-based decoupling mechanism to prevent information leakage. Unlike previous dual-branch models, UNIVERSE achieves high efficiency, strong generalization, and interpretability, especially in autonomous driving contexts. Its ability to perform zero-shot transfer across datasets underscores its novelty and practical significance.
Limitations
- The model relies on extensive high-quality video datasets for training, which demands significant computational resources. Future improvements are needed to reduce training costs.
- The mask design, while effective, may not fully prevent leakage in highly complex or unpredictable scenarios, potentially affecting robustness.
- Performance under adverse weather or sensor failures remains untested, requiring further validation for real-world deployment.
Future Work
Future research will focus on integrating multi-modal data such as LiDAR and high-definition maps to enhance scene understanding. Developing more efficient training algorithms and model compression techniques will reduce resource demands. Additionally, incorporating reinforcement learning could optimize decision-making policies, moving towards fully end-to-end autonomous driving systems.
AI Executive Summary
Autonomous driving systems face persistent challenges in achieving robust action generalization across diverse environments. Traditional models often overfit to dataset-specific priors, limiting their adaptability. Recent advances leverage future video prediction to encode scene dynamics, but architectural choices—such as separate or cascaded models—restrict the transferability of learned representations.
This paper introduces UNIVERSE, a novel unified video-action model built upon a single mask-modulated Diffusion Transformer. By jointly training future video latents and ego-trajectory tokens within one shared architecture, UNIVERSE enables dense visual supervision to directly influence trajectory denoising. The key innovation lies in the Modality-Decoupling Visibility Mask, which shares historical context across modalities while blocking mutual attention between future video and trajectory tokens, ensuring causal integrity. During inference, the model can perform trajectory-only predictions by removing future video denoising, achieving a 4.3× speedup without sacrificing accuracy.
Experimental results on NAVSIM demonstrate a PDMS of 91.0, outperforming previous methods like Two-DiT. The model also exhibits excellent zero-shot transfer to nuScenes and Bench2Drive, with significantly reduced L2 displacement errors and collision rates, validating its cross-domain robustness. The ability to generate consistent future videos further supports its scene understanding capabilities.
This work marks a significant step toward scalable, interpretable, and efficient autonomous driving. By integrating dense scene dynamics directly into action prediction, UNIVERSE addresses core limitations of existing architectures. Its design facilitates deployment in real-time systems, with potential extensions to multi-modal sensor fusion and reinforcement learning. Despite current resource demands, the approach opens new avenues for research in unified perception and control, promising safer and more adaptable autonomous vehicles in the future.
Deep Analysis
Background
自主驾驶技术经历了从规则基础到深度学习的演变，早期依赖手工规则，难以应对复杂场景。近年来，深度神经网络如端到端规划模型（如VADv2、TransFuser）显著提升了感知与决策能力。视频生成与世界建模技术（如DriveGAN、GAIA系列）不断推进场景逼真模拟，但多为场景生成或数据增强，缺乏对动作泛化的系统性研究。现有模型多采用双分支架构，视频预测与轨迹生成相互独立，限制了动态信息的深度融合。本文基于此背景，提出统一模型以解决泛化与效率问题。
Core Problem
自主驾驶中，模型在新环境下的动作泛化能力不足，主要因现有架构将视频理解与轨迹预测分离，导致场景动态信息未能有效传递，且推理速度受限。传统模型在复杂交通环境中容易出现误判或过拟合，难以实现高效、鲁棒的自主决策。如何在保证因果性和推理效率的基础上，增强模型对未知场景的适应性，成为关键难题。
Innovation
提出单一掩码调制的Diffusion Transformer，将未来视频潜变量与轨迹标记联合训练，实现密集监督引导轨迹去噪，突破双分支架构限制。引入模态解耦可视掩码，有效阻断未来目标泄露，保证因果性。模型支持轨迹单独推断，推理速度提升4.3倍，兼顾效率与泛化。采用Flow Matching作为训练目标，增强模型的密集动态理解能力。
Methodology
- �� 输入历史观测、当前速度和指令，通过文本编码器和VAE提取特征。• 构建单一的Diffusion Transformer（DiT），将未来视频潜变量与轨迹标记投影到同一空间。• 采用共享的自注意力和前馈网络进行联合训练，利用Flow Matching优化未来预测。• 引入模态解耦可视掩码，阻断未来视频与轨迹的互注意，确保因果关系。• 在推理阶段，根据需求移除未来视频去噪或轨迹去噪，实现轨迹单推或视频单推。• 训练目标结合密集的未来视频和轨迹的Flow Matching损失，提升动态理解。
Experiments
采用NAVSIM、nuScenes和Bench2Drive三大数据集，评估模型的闭环规划性能、零样本迁移能力和未来视频质量。对比多种变体，验证联合训练、掩码设计和参数共享的效果。超参数包括：训练步数、学习率、批次大小等。采用PDMS、L2误差、碰撞率等指标，确保全面评价。通过消融实验确认模型设计的关键因素。
Results
UNIVERSE在NAVSIM上达成91.0 PDMS，优于Two-DiT的89.6，且在nuScenes和Bench2Drive实现零样本迁移，L2误差分别为0.83米和1.33米，碰撞率低至0.03%。推理速度比联合模型快4.3倍，且在视频生成指标FID和FVD中表现优异，验证了模型在场景理解和动作泛化方面的优势。
Applications
模型可用于自主驾驶中的路径规划、场景模拟和安全评估，依赖于丰富的历史观测和指令输入。其高效推理能力适合实时部署，支持多模态信息融合，提升系统的鲁棒性和泛化能力。未来可结合激光雷达和地图信息，拓展应用场景。
Limitations & Outlook
模型训练依赖大量高质量视频潜变量，计算成本高，硬件需求大。掩码设计在复杂交通环境中仍可能出现信息泄露或误差。模型在极端天气或传感器异常情况下的鲁棒性不足，未来需增强对异常场景的适应能力。
Plain Language Accessible to non-experts
想象你在一个工厂里工作，工厂每天都要生产不同的产品。工人们需要根据之前的生产情况预测未来的需求，然后提前准备材料和工序。传统的方法就像每个工序都自己猜测下一步，容易出错，也不够快。而现在，工厂引入了一台智能机器，它可以同时观察过去的生产数据和未来的需求预测，利用复杂的算法（就像大脑一样）同时做出决策。这台机器不仅能预测未来的生产计划，还能根据这些预测调整自己的操作，确保生产效率最大化。它还设计了特殊的“屏蔽”机制，确保在预测未来时不会泄露不该知道的信息。这样一来，工厂的生产变得更快、更准确，也能应对各种突发情况。这个例子就像UNIVERSE一样，把多个任务融合在一个智能系统中，通过学习过去和预测未来，帮助自动驾驶车辆更安全、更智能地行驶。
ELI14 Explained like you're 14
想象你在玩一款赛车游戏，你的目标是让赛车跑得更快、更安全。以前的游戏里，你只能看到眼前的路面，然后根据经验控制方向和速度，但有时候会撞到障碍物或者偏离路线。现在，有一款新游戏，它可以提前“预测”未来几秒的路况，告诉你哪里可能有障碍，帮你提前做出反应。这就像你有个超级助手，不仅能看到现在，还能预知未来的路况。这个助手用了一种特别的“魔法”——它通过学习大量的赛车视频，掌握了道路的变化规律。它还能在你需要时，只告诉你当前的路线，不用每次都预测未来，从而让游戏变得更快、更流畅。这个助手就是UNIVERSE的核心思想：用智能预测未来，帮助车辆更聪明地行驶，避免碰撞，提升安全性。就像你在玩游戏时，有个聪明的伙伴帮你提前规划路线，既快又稳，真是太酷了！
Abstract
World Action Models (WAMs) have shown strong potential for improving action generalization in autonomous driving by using future video prediction as dense supervision for scene dynamics and temporal causality. However, it remains unclear which architecture better transfers video-modeling benefits to trajectory generation. Existing cascaded or dual-DiT designs separate video imagination from action prediction, weakening the transfer of video-learned world dynamics to the trajectory branch: the action model may still overfit dataset-specific driving priors, while the video model only indirectly regularizes planning. We propose UNIVERSE, a unified video-action model built upon a single mask-modulated Diffusion Transformer. By co-training future video latents and ego-trajectory tokens within shared generative parameters, UNIVERSE allows dense video supervision to directly shape trajectory denoising, leading to stronger cross-domain action generalization. To ensure causal validity and efficient deployment, we introduce a Modality-Decoupling Visibility Mask, which shares historical context across modalities while blocking mutual attention between future video and trajectory tokens. This prevents future-target leakage and enables trajectory-only inference by removing future-video denoising at test time, achieving a $4.3\times$ speedup over joint video-action rollout while maintaining comparable planning accuracy. The same model also supports video-only and joint video-action rollouts. Experiments show that UNIVERSE achieves 91.0 PDMS on NAVSIM (vs. 89.6 for the Two-DiT variant), and demonstrates strong zero-shot transfer to nuScenes and Bench2Drive without fine-tuning, while ablations confirm the importance of single-DiT unification, video co-training, and mask-based modality decoupling.