Safety-Enhanced Autonomous Driving Using Interpretable Sensor Fusion Transformer
InterFuser uses Transformer-based sensor fusion to improve autonomous driving safety, achieving top CARLA leaderboard score of 76.18.
Key Findings
Methodology
This paper introduces InterFuser, a Transformer-based framework that fuses multi-modal, multi-view sensor data including LiDAR and cameras. It employs a multi-stage encoder-decoder architecture with multi-head self-attention for cross-modal interactions. Intermediate interpretable features—such as object density maps and traffic rule signals—are generated and used as safety heuristics. The system is trained with multi-task loss functions balancing path prediction, safety features, and traffic signals. A safety controller constrains low-level actions via linear programming, ensuring safety in complex scenarios.
Key Results
- InterFuser achieves a driving score of 76.18 on CARLA leaderboard, outperforming prior methods like TransFuser and LAV. It significantly reduces infractions and collisions, especially in adversarial urban scenarios, with an 80% reduction in red light violations. Ablation studies confirm the importance of multi-view, multi-modal fusion, and intermediate features. The model generalizes well across Town05 and CARLA 42 Routes benchmarks, demonstrating robustness and safety improvements over state-of-the-art approaches.
Significance
This work advances autonomous driving by integrating global context perception with explainability. The use of Transformer enables comprehensive scene understanding from multiple sensors, addressing limitations of prior fusion methods. The intermediate features provide transparent decision bases, crucial for safety verification and debugging. The approach bridges the gap between high-performance perception and safety assurance, paving the way for real-world deployment of reliable autonomous vehicles. Its success on CARLA benchmarks underscores its potential for real urban environments.
Technical Contribution
The paper introduces a novel single-stage multi-view, multi-modal Transformer architecture that scales better than previous multi-stage models. It innovatively incorporates interpretable features as safety heuristics, enabling explicit action constraints. The safety controller leverages these features with linear programming to ensure actions stay within safe sets. The model's design enhances scalability, interpretability, and safety, setting new standards in end-to-end autonomous driving. Empirical results validate its superiority over existing models in complex urban scenarios.
Novelty
This is the first work to unify multi-view, multi-modal sensor fusion within a single Transformer framework tailored for autonomous driving. It uniquely integrates intermediate interpretable features as safety heuristics, offering transparency and verifiability. Unlike prior methods limited to fewer sensors or simple concatenation, this approach achieves comprehensive scene understanding and safety constraints simultaneously, representing a significant innovation in perception and safety integration.
Limitations
- The model depends heavily on high-quality sensor data; performance may degrade under adverse weather or sensor failure. While effective in simulation, real-world deployment requires addressing domain gaps and computational costs. The current safety controller, based on linear programming, may struggle with highly dynamic or unpredictable scenarios. Further, the approach's scalability to ultra-dense urban environments with numerous agents needs validation.
Future Work
Future efforts will focus on integrating more advanced trajectory prediction and reinforcement learning-based safety control to handle highly dynamic scenarios. Enhancing robustness against sensor failures and adverse weather is also a priority. Exploring adaptive fusion strategies and lightweight model architectures will facilitate real-world deployment. Additionally, incorporating learning-based safety guarantees and formal verification methods could further improve system reliability for commercial autonomous vehicles.
AI Executive Summary
Autonomous driving has made remarkable progress, yet safety remains a critical barrier to widespread deployment. Existing systems often rely on single sensors or simplistic fusion strategies, which struggle to handle complex urban scenarios with unpredictable events. The inability to interpret model decisions further hampers verification and debugging, limiting trust and safety guarantees. Addressing these challenges, this study introduces InterFuser, a Transformer-based sensor fusion framework that integrates multi-view RGB cameras and LiDAR data into a unified perception system.
InterFuser employs a multi-stage encoder-decoder architecture with multi-head self-attention mechanisms, enabling comprehensive scene understanding from diverse sensors. A key innovation is the generation of intermediate interpretable features—such as object density maps and traffic rule signals—that serve as safety heuristics. These features are used to constrain low-level control actions via linear programming, ensuring safety even in adversarial or complex scenarios.
Experimental results on CARLA benchmarks demonstrate the effectiveness of InterFuser, achieving a top driving score of 76.18, surpassing all prior methods. The model exhibits superior performance in reducing infractions and collisions, especially under challenging urban conditions. Ablation studies confirm the importance of multi-view fusion, positional encoding, and safety constraints, providing insights into the model’s robustness.
This approach not only advances perception accuracy but also enhances transparency and safety, addressing a long-standing gap in end-to-end autonomous driving. Its ability to produce explainable decision bases paves the way for safer, more reliable autonomous vehicles. Looking ahead, integrating more sophisticated trajectory prediction and adaptive safety mechanisms will further push the boundaries of autonomous driving, bringing us closer to real-world deployment.
Deep Analysis
Background
近年来，自动驾驶技术迅速发展，深度学习与多模态传感器融合成为核心。早期多采用激光雷达或摄像头单一传感器，难以应对复杂交通环境。TransFuser等模型引入Transformer架构，实现多模态融合，但多局限于少数视角，扩展性不足。随着多视角、多模态传感器的普及，如何高效融合信息、实现全局感知成为研究热点。传统方法多缺乏可解释性，难以验证决策过程。近年来，强调安全性和透明度的研究逐渐兴起，推动端到端模型向可解释化发展。
Core Problem
现有自主驾驶系统在复杂城市环境中表现不足，尤其在突发事件和长尾场景中存在安全漏洞。多模态、多视角信息融合面临扩展性和效率瓶颈，单一模型难以兼顾性能和安全。缺乏可解释性导致系统难以验证和调试，限制实际部署。动态目标预测不足，难以满足高安全标准。如何实现全局感知、保证决策透明、安全可靠，是当前亟待解决的核心问题。
Innovation
提出基于Transformer的InterFuser，创新点包括：1）单阶段多模态多视角融合架构，提升信息交互效率；2）引入中间可解释特征（目标密度图、交通规则信号），作为安全约束，增强模型透明度；3）结合线性规划实现安全控制，有效约束车辆行动。该架构突破了传统多模态融合的扩展瓶颈，显著提升场景理解和安全性。模型在CARLA排行榜中获第一，验证其优越性能。引入中间特征机制，为模型提供可追溯的决策依据，增强系统可信度。
Methodology
- �� 输入：激光雷达点云和多视角RGB图像。• 特征提取：采用ResNet提取基础特征，经过1×1卷积降维。• 编码：将空间特征转为Token，加入位置编码和传感器嵌入，输入Transformer编码器。• 解码：利用多头自注意力机制，生成路径点、目标密度图和交通规则信号。• 预测：路径由GRU自回归预测，密度图通过MLP输出，交通规则由线性层预测。• 损失：多任务损失结合路径、密度图和交通规则误差。• 安全控制：利用中间特征，通过线性规划约束车辆行为，确保安全。
Experiments
在CARLA 0.9.10.1环境中，采集3百万帧专家数据，涵盖8个城镇和多种天气。模型在排行榜中优于TransFuser和LAV，达成最高得分。通过消融实验验证多视角、多模态融合、位置编码和安全控制的贡献。Town05和CARLA 42 Routes测试中也表现优异。指标包括驾驶得分、违规率和碰撞率，模型在复杂场景中表现出色，特别在突发事件和交通违规检测方面优势明显。
Results
InterFuser在CARLA排行榜中得分76.18，优于第二名的75.14，且在碰撞率和交通违规方面显著降低。消融实验显示多模态融合和中间特征对性能提升至关重要。模型在Town05和CARLA 42 Routes测试中也保持领先，验证了其泛化能力。性能提升主要得益于多视角信息的充分利用和安全控制机制的引入。
Plain Language Accessible to non-experts
想象你在开车时，眼睛不仅看前方，还能看到左右两边和远处的交通灯。传统的自动驾驶就像只用一只眼睛看路，容易漏掉侧面或远处的危险。而这项技术像是给车装了多个“眼睛”，还能把这些“眼睛”看到的内容融合在一起，形成一幅完整的交通场景图。它还会告诉你哪些地方可能有危险，比如突然出现的行人或闯红灯的车辆。更厉害的是，它还能解释为什么要这么开，像是给你一份“安全地图”，让你知道哪些行为是安全的，哪些可能出问题。这样，车子不仅会开得更快，还能更安全，像个聪明又可靠的司机助手。
ELI14 Explained like you're 14
想象你在玩一款赛车游戏，你的车可以用很多不同的摄像头看路，比如正前方、左右两边，还能用雷达看到远处的障碍物。以前的自动驾驶就像只用一只眼睛看路，容易漏掉侧面或远处的危险。而这项新技术就像给车装上了好多“眼睛”，还能把这些“眼睛”看到的内容拼在一起，帮车子更好地理解周围的环境。它还能告诉你，前面是不是有红绿灯、路上有没有行人或其他车。更酷的是，它还能解释为什么要这么开，比如“前面有行人，减速”或“红灯亮了，停车”。这样，车子就变得像一个聪明又安全的司机伙伴，不仅跑得快，还能保证安全！
Abstract
Large-scale deployment of autonomous vehicles has been continually delayed due to safety concerns. On the one hand, comprehensive scene understanding is indispensable, a lack of which would result in vulnerability to rare but complex traffic situations, such as the sudden emergence of unknown objects. However, reasoning from a global context requires access to sensors of multiple types and adequate fusion of multi-modal sensor signals, which is difficult to achieve. On the other hand, the lack of interpretability in learning models also hampers the safety with unverifiable failure causes. In this paper, we propose a safety-enhanced autonomous driving framework, named Interpretable Sensor Fusion Transformer(InterFuser), to fully process and fuse information from multi-modal multi-view sensors for achieving comprehensive scene understanding and adversarial event detection. Besides, intermediate interpretable features are generated from our framework, which provide more semantics and are exploited to better constrain actions to be within the safe sets. We conducted extensive experiments on CARLA benchmarks, where our model outperforms prior methods, ranking the first on the public CARLA Leaderboard. Our code will be made available at https://github.com/opendilab/InterFuser