Metis: A Generalizable and Efficient World-Action Model for Autonomous Driving and Urban Navigation
Metis employs decoupled video generation and action prediction with Mixture-of-Transformers, achieving state-of-the-art autonomous driving performance.
Key Findings
Methodology
Metis utilizes a Mixture-of-Transformers architecture, with dedicated video generation (VGE) and action prediction (AE) experts, sharing a latent space for interaction. An asymmetric attention mask is introduced, allowing the action model to bypass explicit future observation generation during inference, significantly reducing computational overhead. The training employs flow matching to jointly optimize both experts, ensuring consistency. The approach combines pre-trained large-scale video models (Wan2.2-5B) with diffusion transformers for trajectory prediction. Experiments on NAVSIM-v2 and CityWalker datasets demonstrate superior performance, with metrics like EPDMS reaching 32.2 and 90.3, and real-time deployment confirms efficiency gains.
Key Results
- On NAVSIM-v2 navhard benchmark, Metis achieves an EPDMS score of 32.2, outperforming all baselines, with a 50% reduction in inference time. On navtest, it maintains top performance with a 0.64m L2 error and 9.8° MAOE. In CityWalker, average L2 error drops to 0.64 meters, MAOE to 9.8°, surpassing VLA methods. Real robot tests validate robustness and real-time inference, with computational costs halved compared to prior models.
Significance
This work addresses fundamental limitations of existing world action models by decoupling visual prediction from action planning, enabling faster, more generalizable autonomous navigation. It bridges the gap between high-fidelity environment modeling and real-time decision-making, paving the way for safer, more efficient autonomous vehicles. The approach’s scalability and robustness open new avenues for deploying intelligent systems in complex urban environments, significantly advancing the state of the art in autonomous systems engineering.
Technical Contribution
The primary innovation lies in the asymmetric attention mechanism that allows training of separate experts while enabling inference to skip explicit future video generation. The Mixture-of-Transformers structure maintains task-specific distributional properties, improving generalization. The flow matching training ensures joint optimization, and the shared latent space facilitates effective interaction. These contributions collectively enable high-performance, efficient, and flexible autonomous navigation models that outperform existing methods on multiple benchmarks.
Novelty
This is the first work to decouple video generation from action prediction via structured asymmetric attention, allowing training-time joint optimization with inference-time efficiency. Unlike prior tightly coupled models, Metis achieves a significant reduction in inference latency while maintaining or improving performance. Its innovative architecture and training strategy set a new benchmark for real-time, generalizable world action modeling in autonomous driving.
Limitations
- Despite efficiency gains, the model still relies on large-scale pretraining, which is computationally expensive. Its performance may degrade in highly dynamic or unseen environments due to limited adaptation capacity. The current architecture assumes static sensor inputs and may struggle with sensor noise or occlusions. Future work should focus on reducing training costs, enhancing robustness, and integrating multi-modal data for broader applicability.
Future Work
Future directions include integrating reinforcement learning for adaptive policy refinement, extending the model to multi-modal sensor inputs, and scaling training on larger, more diverse datasets. Improving robustness in highly dynamic scenarios and reducing pretraining dependency are also key goals. Additionally, exploring multi-agent coordination and real-world deployment in varied urban environments will further validate and enhance the system’s practicality.
AI Executive Summary
Metis introduces a novel framework for autonomous driving and urban navigation by decoupling environment prediction from action planning through a Mixture-of-Transformers architecture. Traditional world action models (WAMs) often face high inference latency due to explicit future observation generation and tightly coupled visual and action modules, limiting their real-time applicability. To overcome these challenges, Metis employs dedicated video generation and action prediction experts, sharing a latent space for interaction, and utilizes an asymmetric attention mask that allows the action model to operate without explicit future scene generation during inference.
During training, the model jointly optimizes both experts using flow matching, ensuring consistency and leveraging large-scale pre-trained video models like Wan2.2-5B. The asymmetric attention mechanism enforces unidirectional information flow, enabling the model to predict physically plausible environment dynamics conditioned on planned trajectories while maintaining high efficiency. Extensive experiments on NAVSIM-v2 and CityWalker datasets demonstrate that Metis achieves state-of-the-art results, with EPDM scores of 32.2 and 90.3, and reduces inference time by approximately 50% compared to previous methods.
Real-world robot deployments further confirm the model’s robustness, real-time performance, and practical feasibility. The core innovation—decoupling visual environment modeling from action prediction—addresses longstanding bottlenecks in autonomous navigation, offering a scalable, generalizable solution that can adapt to complex urban scenarios. Despite current limitations such as reliance on large pretraining and challenges in highly dynamic environments, future work aims to incorporate reinforcement learning, multi-modal data, and broader dataset scales to push the boundaries of autonomous system capabilities. Overall, Metis marks a significant step toward smarter, faster, and safer autonomous navigation systems, with promising implications for industry and research alike.
Deep Analysis
Background
随着自动驾驶和城市导航的快速发展，环境感知与路径规划技术不断演进。早期依赖规则和地图的方案逐渐被深度学习模型取代，但仍存在泛化差、计算成本高等问题。近年来，视觉语言模型（VLM）和视频生成技术（如扩散模型）在环境理解中展现出巨大潜力，推动了世界动作模型（WAM）的出现。WAM结合环境预测与动作生成，试图实现更符合物理规律的路径规划。代表性工作包括基于Transformer的ReCogDrive、SGDrive，以及结合强化学习的端到端系统。这些模型在环境动态建模方面取得一定成功，但在推理效率和泛化能力方面仍有待提升。
Core Problem
现有WAM在自主驾驶和城市导航中面临两个核心难题：一是推理时需要生成未来观察，导致延迟高、计算成本大；二是视频生成与动作预测紧密耦合，影响模型的泛化能力和适应性。具体表现为：在复杂环境中，模型难以实现实时响应；而紧耦合架构限制了对不同场景的适应能力。这些问题严重制约了WAM在实际应用中的推广，亟需创新架构以提升效率和泛化。
Innovation
Metis的创新点主要包括：
- �� 引入Mixture-of-Transformers架构，分别训练视频生成专家（VGE）和动作预测专家（AE），保持各自的分布特性。
- �� 在训练中，通过流匹配实现两个专家的联合优化，确保潜在空间中的信息一致性。
- �� 采用非对称注意掩码，限制动作模型只关注当前观察，推理时跳过未来视频生成，显著提升效率。
- �� 设计结构化注意机制，使未来视频信息在训练中引导动作优化，但在推理中实现快速跳过。
- �� 结合预训练视频模型（如Wan2.2-5B）和扩散Transformer，增强模型对环境动态的理解和路径预测能力。
Methodology
- �� 输入：当前视觉观察（ot）、文本指令（l）。
- �� 训练阶段：
- 视觉信息通过VAE编码，文本信息通过T5编码。
- 训练视频生成专家（VGE）捕获时空动态，动作预测专家（AE）进行低维轨迹预测。
- 通过流匹配优化两个专家的输出，确保潜在空间一致。
- 采用非对称注意掩码，限制动作模型只关注当前观察，未来视频信息在训练中引导动作。
- �� 推理阶段：
- 仅利用当前观察的潜在表示，预测动作。
- 跳过未来视频生成，减少计算。
- 通过共享潜在空间，保持环境动态的合理性。
- 结合动作流和环境演变的速度场预测，优化路径规划。
Experiments
模型在NAVSIM-v2和CityWalker两个公开数据集上进行评估，指标包括EPDMS、TTC、L2误差和MAOE。采用不同的超参数（如视频长度、扩散步数）进行消融分析，验证非对称注意掩码的效果。在硬件平台上实现实时推理，验证效率提升。还在真实机器人平台上进行零样本部署，测试环境适应性和鲁棒性。对比多种基线方法，验证模型在复杂场景中的优越表现。
Results
Metis在NAVSIM-v2 navhard基准上EPDMS达32.2，优于所有比较方法，推理速度提升约50%。在navtest中，保持领先，L2误差为0.64米，MAOE为9.8度。CityWalker数据集上，平均L2误差为0.64米，MAOE低至9.8°，表现优异。实车测试验证了模型的鲁棒性和实时性，显示出强大的泛化能力。消融实验表明非对称注意掩码在性能和效率上均优于对称方案。
Applications
该模型适用于自动驾驶车辆路径规划、复杂城市环境中的导航系统，以及无人机自主飞行。模型在训练阶段依赖大规模环境数据，部署时只需当前观察信息，适合边缘计算设备。未来结合多模态感知和强化学习，有望实现更安全、更智能的自动驾驶方案，推动智能交通发展。
Limitations & Outlook
模型在极端复杂或未见环境中仍存在泛化不足，尤其在高速动态场景下表现有限。训练成本高，依赖大量预训练模型，硬件要求高。推理时虽跳过视频生成，但动作预测在高动态场景中仍可能偏差。未来需优化模型结构，增强鲁棒性，降低训练成本，并扩展多模态融合能力。
Plain Language Accessible to non-experts
想象你在操控一辆遥控车。以前的遥控车每次都要看一堆动画，等待它完成所有动作，才能决定下一步，太慢了。现在，有两个助手，一个专门观察当前环境，帮你判断下一步怎么走；另一个会提前模拟未来的场景，但只在训练时用。在实际操作时，你只看当前环境的提示，助手不用模拟未来，就能快速决定动作。这样，遥控车反应快多了，也更聪明，能应对突然出现的障碍。Metis就像这个助手系统，把环境预测和动作规划拆开，既快又稳。
ELI14 Explained like you're 14
你玩一个超级酷的游戏，你要控制一个角色在城市里跑来跑去。以前的游戏每次都要看一堆动画，等动画结束才能决定下一步，太慢了。而Metis就像是给你准备了两个帮手，一个帮你看未来的场景（比如交通、路况），另一个帮你规划下一步（比如转弯、加速）。但在你行动时，你只告诉助手你现在的想法，助手会帮你快速决定下一步，而不用每次都重新设计整个场景。这样，你的动作变得更快、更聪明，也更能应对突发状况。是不是很酷？
Glossary
World Action Model (世界动作模型)
一种结合环境预测与动作规划的模型，能模拟未来场景以辅助决策。
本文提出的核心框架，用于自主导航和驾驶。
Mixture-of-Transformers (变换器混合架构)
由多个专门的Transformer专家组成的架构，用于不同任务的解耦学习。
实现视频生成与动作预测的分离。
非对称注意掩码 (Asymmetric Attention Mask)
一种限制信息流向的机制，确保模型在推理时只关注必要信息。
提升推理效率，保持训练-推理一致性。
流匹配 (Flow Matching)
一种优化模型输出连续变化的技术，用于动作和环境的拟合。
训练目标之一，确保模型能准确预测未来状态。
扩散Transformer (Diffusion Transformer)
结合扩散模型的Transformer，用于高质量动作预测。
实现复杂环境中的路径规划。
Open Questions Unanswered questions from this research
- 1 如何进一步降低大规模预训练模型的训练成本？
- 2 模型在极端动态环境中的泛化能力如何提升？
- 3 多模态信息融合对路径规划的具体影响尚待研究。
Applications
Immediate Applications
自动驾驶路径规划
在实际车辆中应用Metis，实现高效、鲁棒的实时路径规划，提升交通安全与效率。
城市智能导航系统
为智能导航设备提供更快、更准确的路径预测，改善用户体验。
Long-term Vision
无人机自主飞行
利用Metis的环境预测能力，实现复杂城市环境中的自主飞行与避障。
Abstract
World action models~(WAMs) have shown great promise for autonomous driving and urban navigation. Built upon Vision-Language-Action models or video generation models, existing approaches suffer key limitations: (1) High inference latency due to future observation prediction at test time, and (2) tightly coupled video and action modeling leading to representational mismatch and degraded generalization. To address both issues, we propose Metis, an end-to-end WAM framework that decouples video generation and action prediction. Specifically, Metis employs a Mixture-of-Transformers architecture with dedicated experts for video generation and action prediction, preserving the intrinsic distributional properties of each task. To enhance efficiency, we introduce an asymmetric attention mask that enables joint training of both experts while allowing the action model to bypass explicit video generation during inference. This design ensures training-inference consistency and significantly reduces computational costs without compromising planning performance. Extensive experiments demonstrate state-of-the-art performance on the NAVSIM navhard and navtest benchmarks and the CityWalker navigation benchmark, validating both the generalizability and efficiency across diverse tasks. Real-robot deployments further confirm the practical feasibility of our approach.