Plug-and-Play Traffic Element Awareness for End-to-End Autonomous Driving
Proposes a plug-and-play traffic element awareness framework that significantly improves end-to-end autonomous driving across various models, with detailed 3D detection and topological encoding.
Key Findings
Methodology
This work augments multiple public datasets (nuScenes, NAVSIM-v1, NAVSIM-v2) with comprehensive traffic element annotations, including 3D positions and categories. It employs lightweight 2D detection (YOLO), monocular depth estimation (UniDepthv2), and LiDAR fusion to localize traffic elements in 3D. A topology prediction module estimates adjacency matrices linking traffic elements and lane centerlines, encoded into structured language via a pretrained BERT. These signals are integrated into various end-to-end models—regression, diffusion, scoring—by concatenation with vision queries, enabling rule-aware trajectory planning. Auxiliary supervision with L1 and focal losses guides the detection, while topology conditioning refines decision-making.
Key Results
- On nuScenes, traffic element supervision reduces L2 trajectory error by 0.09 meters, collision rate by 0.05%. In NAVSIM-v2, models with traffic element awareness improve EP-DMS scores by 47%, setting new state-of-the-art. Closed-loop Bench2Drive tests show a 2.5-point increase in driving score and 4% higher success rate. Across models, the improvements are consistent, especially in complex intersections and rule adherence scenarios.
- Ablation studies reveal 3D traffic representations outperform 2D cues, traffic signs outperform lights, and topological encoding complements local rule signals. The approach generalizes well across architectures, demonstrating robustness and broad applicability.
- The method introduces negligible runtime overhead, making it practical for real-time deployment. It effectively enhances safety and rule compliance, addressing key limitations of prior perception-centric models.
Significance
This research addresses a critical gap in autonomous driving by systematically integrating traffic rules and topology into end-to-end models. It demonstrates that traffic elements serve as vital signals for safe, rule-compliant navigation, especially in complex scenarios. The lightweight, universal framework bridges the gap between perception and decision-making, offering a scalable solution for industry deployment. By elevating the importance of traffic signals beyond mere geometric cues, it paves the way for safer, more reliable autonomous vehicles that can better handle real-world complexities and regulatory constraints.
Technical Contribution
The paper introduces a novel plug-and-play architecture that combines 3D traffic element detection, topological relation prediction, and language-based encoding to embed rule signals into various end-to-end models. It innovates by using lightweight detection modules, structured language conditioning, and ego-centric topology filtering, enabling seamless integration without extensive architectural changes. This approach enhances the models’ understanding of traffic rules and improves their decision-making robustness, validated across multiple datasets and model types. The framework’s modularity and efficiency represent a significant step forward in scalable autonomous driving solutions.
Novelty
This work is the first to systematically incorporate traffic elements as explicit, structured signals into end-to-end autonomous driving systems in a plug-and-play manner. Unlike prior methods focusing solely on dynamic agents or dense geometric cues, it emphasizes rule-critical signals like traffic lights, signs, and lane topology. The combination of lightweight 3D detection, topological language encoding, and multi-model validation establishes a new paradigm for rule-aware perception and planning, demonstrating broad generalization and practical deployment potential.
Limitations
- The approach relies on accurate traffic element detection; detection failures or misclassification can impair decision quality, especially in adverse conditions.
- In highly sparse or occluded environments, the sparse traffic signals may limit the effectiveness of rule encoding.
- Additional computational overhead, though minimal, may pose challenges for resource-constrained platforms, requiring further optimization.
Future Work
Future directions include enhancing detection robustness via self-supervised learning, extending to more complex traffic scenarios, and integrating reinforcement learning for policy optimization. Exploring unsupervised or semi-supervised methods to reduce annotation dependency and deploying in real-world multi-agent environments are also promising avenues. Additionally, refining topology encoding for dynamic scenes and incorporating multi-modal sensor fusion will further improve system reliability and safety.
AI Executive Summary
Autonomous driving has rapidly evolved from rule-based systems to deep end-to-end models, yet a critical gap remains: the underutilization of traffic signals and rules. Most current systems prioritize dynamic agents like vehicles and pedestrians, neglecting the regulatory cues provided by traffic lights, signs, and lane topology. This oversight leads to suboptimal decision-making, especially in complex intersections where rule compliance is vital for safety.
This paper introduces a novel plug-and-play framework that embeds traffic element awareness into diverse end-to-end driving models. By augmenting datasets such as nuScenes and NAVSIM with structured 3D annotations for traffic lights, signs, and lane topology, the authors develop lightweight detection modules combined with topological relation prediction. These signals are encoded into natural language descriptions using a pretrained BERT model, then integrated into existing perception and planning pipelines via concatenation with vision queries. This design ensures broad compatibility with various architectures, including regression, diffusion, and scoring-based models.
The core technical innovation lies in the multi-modal fusion of 3D traffic signals and topological cues, which serve as explicit rule signals guiding trajectory planning. The system is trained with auxiliary losses—L1 for position, focal loss for categories—and leverages ego-centric filtering to focus on relevant signals. Extensive experiments across datasets demonstrate consistent performance gains: in nuScenes, trajectory errors decrease by 0.09 meters, collision rates drop by 0.05%, and in NAVSIM-v2, the models achieve a 47% improvement in safety metrics, setting new state-of-the-art results. Closed-loop tests on Bench2Drive further confirm the robustness, with a 2.5-point increase in overall driving score.
The significance of this work is profound: it elevates the role of traffic signals from geometric cues to vital rule signals, enabling autonomous vehicles to better interpret and obey traffic laws. The lightweight, modular design ensures practical deployment, making it a promising step toward safer, rule-compliant autonomous driving in complex environments. Despite some limitations—such as detection dependency and sparse signals—the framework opens new avenues for integrating high-level traffic understanding into end-to-end systems, promising safer roads and more reliable autonomous vehicles in the near future.
Deep Analysis
Background
自动驾驶技术经历了从基于规则到深度学习的演变。早期系统依赖高精地图和手工规则，代表如Waymo、Tesla，强调感知与规则遵守。近年来，端到端学习（如Wayve、Nuro）逐渐兴起，试图用深度模型直接映射传感器到控制指令，但多忽视交通规则和拓扑关系。当前研究多关注动态交通参与者（车辆、行人），而对交通灯、标志等交通元素的系统性研究不足。虽然部分工作引入地图或几何信息，但缺乏对其影响的深入分析。随着交通环境复杂化，理解交通元素的重要性日益凸显，但缺乏统一、轻量的集成方案。
Core Problem
核心问题在于交通元素在端到端自主驾驶中的作用未被充分挖掘。现有模型多依赖几何和动态目标，忽视交通信号，导致在复杂场景中易违规或误判。公共数据集缺乏结构化交通元素标注，限制了系统性研究。不同模型架构的异质性也使得单一方案难以推广。如何高效、普适地将交通元素信息融入多模型体系，提升规则理解和场景适应性，成为亟待解决的难题。
Innovation
本研究的创新点包括：1）在多个数据集上扩展交通元素标注体系，解决数据不足问题；2）提出轻量级的3D交通元素检测和拓扑关系预测机制，增强场景理解；3）引入结构化语言编码，作为规则信号融入模型；4）实现“即插即用”的集成方式，兼容多模型架构。这些创新突破了传统几何或动态目标导向的限制，强调规则和拓扑关系在安全性中的核心作用，为端到端自主驾驶提供了新思路。
Methodology
- �� 数据扩展：在nuScenes、NAVSIM等数据集上加入交通元素标注，包括交通灯、标志的3D位置和类别。
- �� 交通元素检测：采用YOLO等模型进行2D检测，结合深度估计（UniDepthv2）和LiDAR点云，推算3D中心点。
- �� 拓扑关系预测：利用轻量级MLP预测交通元素与车道中心线的关系，形成邻接矩阵。
- �� 结构化编码：将交通元素类别、拓扑关系转为自然语言描述，输入预训练BERT编码器，生成规则条件。
- �� 融合策略：将交通元素的3D表示和拓扑编码与视觉特征拼接，作为解码器输入，指导轨迹规划。
- �� 训练目标：在主任务损失基础上加入交通元素检测（L1、焦点损失）和拓扑条件损失，整体优化。
Experiments
在nuScenes、NAVSIM-v1、NAVSIM-v2和Bench2Drive上进行评估，比较引入交通元素前后模型的性能变化。采用L2轨迹误差、碰撞率、EPDMS等指标，验证不同模型（回归、扩散、评分）在多场景中的表现。通过消融实验分析3D表示、拓扑编码和交通标志的贡献。参数设置遵循各模型原始训练方案，确保公平性。结果显示交通元素显著提升模型安全性和规则遵守能力，验证其作为通用信号的有效性。
Results
引入交通元素后，nuScenes模型L2误差平均降低0.09米，碰撞率降低0.05%，在NAVSIM-v2中EPDMS提升47%，刷新SOTA。闭环测试中，驾驶得分提升2.5分，成功率提高4%。ablation验证3D表示优于2D，交通标志比灯光更关键，拓扑关系增强规则理解。多模型表现一致，验证了方法的普适性。整体结果表明，交通元素作为“规则信号”极大改善自主驾驶模型的安全性和合规性。
Applications
该方法适用于自动驾驶车辆的感知与决策模块，特别在复杂交叉口和交通法规严格区域。只需少量交通元素标注和轻量检测器，即可提升模型规则理解能力。未来可结合自动标注和自监督技术，降低标注成本，推动行业应用。长远来看，融合交通元素与拓扑关系，有望实现更智能、更安全的自动驾驶系统，适应多变交通环境。
Limitations & Outlook
当前方法依赖高质量交通元素检测，检测失误会影响整体性能。复杂场景下，稀疏或遮挡的交通元素可能导致信息缺失。拓扑关系预测在极端复杂环境中可能不准确。系统引入增加了计算负担，需进一步优化模型效率。未来需增强鲁棒性，减少对标注的依赖，提升在极端环境中的表现。
Plain Language Accessible to non-experts
想象你在开车时，除了看前方的路，还会注意交通灯、路标、指示牌。这些交通元素告诉你该停、走、转弯，帮助你遵守交通规则，避免违规。自动驾驶系统也是这样，但它的“眼睛”和“脑袋”是传感器和算法。过去的系统主要关注动态的车辆和行人，就像只关注路上的行驶车辆。而交通灯和标志就像交通警察的手势，指引车辆正确行驶。本文提出一种方法，让自动驾驶的“眼睛”不仅看到车辆，还能识别交通灯、标志，并理解它们的规则关系。这样，车辆就能像人一样，提前知道该停还是走，转弯还是直行。研究中，他们用特殊的技术把交通元素的位置信息转成数字，再用语言描述交通规则，把这些信息融入到车辆的决策中。结果显示，这样的改进让自动驾驶车辆在复杂交叉口表现更安全、更守规，减少了碰撞和违规。就像给车辆装上了“交通规则的指南针”，让它们在复杂的道路环境中也能自如应对。
Abstract
Traffic elements such as traffic lights and road signs play a fundamental role in human driving decisions and should naturally influence end-to-end driving performance. However, existing end-to-end driving research predominantly focuses on dynamic road participants (e.g., vehicles and pedestrians), while the role of traffic elements remains largely unexplored. The community still lacks a systematic study quantifying their impact, largely because public datasets rarely provide structured traffic-element annotations and modern driving systems vary widely in architecture and training paradigm. In this work, we present the first systematic investigation of traffic element awareness for end-to-end autonomous driving. We construct a unified research infrastructure by augmenting multiple public driving datasets with comprehensive traffic-element annotations. To support diverse model families, we adopt a minimal and universal integration design that incorporates traffic-element signals into existing pipelines in a plug-and-play manner with negligible architectural modification. We evaluate this design across modern paradigms, including perception-prediction-planning pipelines, vision-language-action models (VLA), regression-based planners, diffusion-based policies, and trajectory-scoring frameworks, on nuScenes, NAVSIM-v1, NAVSIM-v2, and Bench2Drive. Across all paradigms and datasets, this simple integration consistently improves driving performance, demonstrating that traffic element awareness provides a robust and generalizable signal for end-to-end driving systems. Notably, on the challenging NAVSIM-v2 benchmark, our approach significantly improves state-of-the-art architectures and data pipelines, establishing a new state of the art.