End-to-End Driving with Online Trajectory Evaluation via BEV World Model
Proposes WoTE framework using BEV world model for online trajectory evaluation, achieving state-of-the-art results.
Key Findings
Methodology
The proposed WoTE framework integrates multi-modal sensor inputs processed via ResNet34 into BEV features, then employs a transformer encoder-based world model for future state prediction. It constructs a differentiable, single-step BEV prediction mechanism that efficiently forecasts multiple future BEV states based on refined trajectories. The reward model combines imitation and simulation rewards, supervised through traffic simulators like nuPlan, to evaluate trajectory safety end-to-end. Training involves multi-objective loss functions including BEV semantic map supervision, reward prediction, and trajectory regression, enabling fully differentiable optimization. Experiments on NAVSIM and CARLA Bench2Drive demonstrate superior performance, validating the effectiveness of future state integration in trajectory evaluation.
Key Results
- On NAVSIM, the method achieves a PDMS of 87.1, surpassing previous model-free approaches by over 6%, with a success rate of 98.5%, indicating robust safety performance in challenging scenarios.
- In CARLA's Bench2Drive, the approach attains a Driving Score of 61.71, with a success rate of 31.36%, significantly outperforming rule-based and model-free baselines, confirming strong generalization and robustness.
- Ablation studies reveal that incorporating future BEV states improves trajectory safety and evaluation accuracy, especially in complex traffic environments, demonstrating the critical role of future prediction in end-to-end autonomous driving.
Significance
This work advances autonomous driving by integrating future environment predictions directly into trajectory evaluation, addressing the critical challenge of safety assurance in dynamic, uncertain traffic scenarios. By leveraging traffic simulators for supervision, it overcomes the scarcity of multi-future annotations, enabling scalable, end-to-end training. The approach enhances the interpretability and safety of autonomous systems, facilitating deployment in real-world settings. Its ability to perform real-time, accurate future state forecasting marks a significant step toward more reliable, intelligent autonomous vehicles, with broad implications for industry and research.
Technical Contribution
The paper introduces a novel BEV space-based world model that predicts future states with high efficiency via single-step forward passes, contrasting with computationally intensive image-based models. It employs a transformer encoder architecture for recurrent future prediction, combined with a differentiable reward prediction module that fuses multiple reward signals. The supervision strategy leverages traffic simulators to generate realistic BEV semantic maps and reward labels, enabling end-to-end training. The integration of multi-modal sensor fusion, attention mechanisms, and traffic simulation supervision constitutes a comprehensive, scalable framework that significantly outperforms existing methods in both open-loop and closed-loop evaluations.
Novelty
This is the first work to implement a BEV space-centric, single-step future state prediction model combined with traffic simulator-based supervision for trajectory evaluation in autonomous driving. Unlike prior methods relying on image-space prediction or rule-based evaluation, it introduces a fully differentiable, end-to-end trainable system that explicitly models environmental dynamics in BEV space, enabling real-time, accurate, and safe trajectory assessment. This innovative approach bridges the gap between environment modeling and trajectory optimization, setting a new benchmark in the field.
Limitations
- The model's robustness in extreme weather conditions or sensor failures remains untested, potentially affecting future state accuracy and safety.
- While BEV prediction is efficient, high-density traffic scenarios still pose computational challenges for real-time deployment.
- The reliance on traffic simulators for supervision introduces dependency on simulation fidelity; discrepancies between simulated and real-world scenarios may impact transferability.
Future Work
Future research will focus on enhancing robustness under adverse weather and sensor anomalies, integrating reinforcement learning for adaptive decision-making, and expanding training datasets with more diverse real-world scenarios. Additionally, improving traffic simulator realism and scalability will be prioritized to better support supervision and transfer to real-world deployment.
AI Executive Summary
Autonomous driving has made significant strides with end-to-end learning approaches, yet ensuring safety remains a core challenge. Traditional systems often rely on rule-based or offline models for trajectory evaluation, which struggle to adapt to complex, dynamic environments. This paper introduces WoTE, a novel framework that leverages a BEV space-based world model to predict future environmental states efficiently and accurately. By integrating these predictions into the trajectory evaluation process, WoTE enables the vehicle to anticipate future scenarios, assess safety, and select optimal paths in real time.
The core innovation lies in constructing a differentiable, single-step BEV world model that forecasts multiple future states based on refined trajectory proposals. This model employs transformer encoders for recurrent prediction, coupled with a reward prediction module that combines imitation and simulation rewards. Traffic simulators like nuPlan provide supervision, generating realistic BEV semantic maps and reward signals, thus overcoming the scarcity of multi-future annotations. The entire system is trained end-to-end, ensuring seamless optimization.
Extensive experiments on NAVSIM and CARLA's Bench2Drive demonstrate that incorporating future environment states significantly improves safety metrics, with PDMS exceeding 87% and success rates surpassing 98% in challenging scenarios. The approach outperforms existing rule-based and model-free methods, confirming its robustness and generalization. These results highlight the potential of future state-aware evaluation to revolutionize autonomous driving safety and reliability.
Overall, this work marks a substantial step toward deploying safer, more intelligent autonomous vehicles. By explicitly modeling environmental dynamics in BEV space and integrating simulation-based supervision, it opens new avenues for scalable, real-time trajectory evaluation, promising broad impact across industry and academia. Future efforts will aim to enhance robustness, computational efficiency, and real-world transferability, pushing autonomous driving closer to widespread adoption.
Deep Analysis
Background
自主驾驶技术经历了从基于规则的系统到深度学习模型的演变。早期方法依赖手工设计的规则和静态地图，难以应对复杂的交通环境。近年来，端到端深度学习模型如TransFuser、UniAD等，通过融合多模态传感器信息，实现感知、预测和规划的联合优化，显著提升了系统性能。然而，轨迹评估仍多依赖离线规则或图像空间预测，难以保证动态环境下的安全性和鲁棒性。多模态融合、交通模拟器的发展为训练提供了支持，但缺乏高效、端到端的未来状态预测机制，限制了模型的泛化能力。本文在此背景下，提出在BEV空间中构建未来状态预测模型，结合交通模拟器进行监督，旨在解决多未来场景缺乏标注、实时性不足等关键难题。
Core Problem
当前自主驾驶系统在轨迹预测和安全评估方面面临两大瓶颈：一是未来环境状态的高效、准确预测困难，二是缺乏端到端、可微的评估机制。传统方法多基于静态地图或规则，难以动态适应复杂交通环境。图像空间中的未来预测计算成本高，难以满足实时需求。此外，缺少多场景、多未来路径的标注，导致模型在实际复杂场景中表现不佳。如何在保证计算效率的同时，融合未来环境信息进行轨迹评估，成为亟待突破的核心问题。
Innovation
本文的创新点主要包括：
1) 在BEV空间中构建未来状态预测模型，采用单步前向机制，大幅提升预测效率，适合实时应用；
2) 结合交通模拟器（如nuPlan）进行监督，解决多未来场景缺乏标注的问题，增强模型泛化能力；
3) 设计端到端的奖励预测机制，将未来状态信息融入轨迹评价，实现完全可微的端到端训练；
4) 采用多模态融合和交叉注意机制，提升模型对复杂交通环境的适应性。这些创新突破了传统基于图像预测和规则评价的限制，为自主驾驶提供了更科学的决策依据。
Methodology
- �� 输入：多模态传感器（LiDAR、RGB图像）经过ResNet34提取BEV特征，形成BEV状态。
- �� 轨迹锚点：利用K-Means聚类生成多模态轨迹锚点。
- �� 轨迹细化：通过交叉注意机制和MLP对锚点进行优化，得到多条候选轨迹。
- �� 未来状态预测：将当前BEV状态和轨迹输入世界模型（Transformer编码器）进行多步预测，输出未来多时刻的BEV状态。
- �� 奖励模型：结合未来状态和轨迹信息，预测模仿奖励和仿真奖励（如碰撞、车道偏离、碰撞时间等），用以评价轨迹。
- �� 监督：利用交通模拟器生成真实未来BEV语义图，采用Focal Loss和二元交叉熵进行监督。
- �� 训练：端到端优化BEV预测、奖励预测和轨迹回归，确保模型整体性能。
Experiments
在NAVSIM和CARLA的Bench2Drive上进行验证，使用PDMS、成功率和Driving Score等指标。超参数包括256个轨迹锚点，训练轮次分别为30和27。采用Adam优化器，学习率1e-4。对比多种模型（如Hydra-MDP、TransFuser），验证未来状态预测和轨迹评价的贡献。消融实验显示引入未来状态显著提升安全性和鲁棒性。模型在复杂交通场景中表现优异，验证了方法的有效性和实用性。
Results
在NAVSIM数据集上，PDMS达87.1，比基线提升6%以上，成功率达98.5%，优于Hydra-MDP等模型。在CARLA Bench2Drive中，Driving Score提升至61.71，成功率达31.36%。消融实验表明，未来状态预测和端到端奖励机制是性能提升的关键因素。这些结果验证了在复杂、多变环境中引入未来状态的有效性和必要性。
Applications
该方法适用于自动驾驶车辆的实时路径规划和安全评估，特别是在复杂交通环境中。通过结合交通模拟器进行监督，能在缺乏标注的场景下实现高效训练。未来可扩展到无人驾驶出租车、物流车等行业，提升系统安全性和自主性。还可结合强化学习进一步优化路径选择，实现自主适应复杂场景。
Limitations & Outlook
模型在极端天气或传感器异常情况下表现尚不理想，未来需增强鲁棒性。实时性虽优，但在高密度场景中仍存在计算瓶颈。交通模拟器的逼真度影响监督效果，需提升模拟的真实性。未来应结合更多真实场景数据，优化模型泛化能力。
Plain Language Accessible to non-experts
想象你在开车时，除了看前方的路，还会提前猜测前面可能出现的情况，比如前面有人突然转弯或停车。这个过程就像你在脑海里模拟未来几秒的场景，然后决定怎么走。现在，自动驾驶汽车也在做类似的事情，它们用传感器收集周围环境信息，然后用电脑模拟未来几秒的交通情况，提前判断潜在的危险。本文提出的方法就像给汽车装上了一个“未来预言机”，它可以快速预测未来的交通环境，帮助汽车做出更安全、更合理的决策。通过模拟未来的交通场景，汽车可以提前避开危险，确保乘客安全。这就像你在玩一款赛车游戏，提前预判对手的动作，然后选择最佳路线，赢得比赛。它让自动驾驶变得更聪明、更安全，也更像人类司机会“预判未来”。
ELI14 Explained like you're 14
想象你在玩一款赛车游戏，你不仅要控制赛车，还要猜测前面会发生什么，比如弯道或者障碍物。你会提前想象几秒后可能的场景，然后决定怎么操作，才能最快、最安全地到达终点。自动驾驶汽车也在用类似的方法，它们用传感器收集周围环境信息，然后用电脑模拟未来几秒的交通情况。这样，汽车就能提前知道前面可能发生的危险，提前做出反应。本文介绍的技术就像给汽车装上了一个“未来预言机”，它可以快速模拟未来的交通环境，帮助汽车选择最安全的路线。通过提前预判，汽车可以避免碰撞和交通堵塞，就像你在游戏中提前规划路线一样。这让自动驾驶变得更聪明、更安全，也更像人类司机会“预判未来”。
Abstract
End-to-end autonomous driving has achieved remarkable progress by integrating perception, prediction, and planning into a fully differentiable framework. Yet, to fully realize its potential, an effective online trajectory evaluation is indispensable to ensure safety. By forecasting the future outcomes of a given trajectory, trajectory evaluation becomes much more effective. This goal can be achieved by employing a world model to capture environmental dynamics and predict future states. Therefore, we propose an end-to-end driving framework WoTE, which leverages a BEV World model to predict future BEV states for Trajectory Evaluation. The proposed BEV world model is latency-efficient compared to image-level world models and can be seamlessly supervised using off-the-shelf BEV-space traffic simulators. We validate our framework on both the NAVSIM benchmark and the closed-loop Bench2Drive benchmark based on the CARLA simulator, achieving state-of-the-art performance. Code is released at https://github.com/liyingyanUCAS/WoTE.