PlanT: Explainable Planning Transformers via Object-Level Representations
PlanT employs object-level sparse features with Transformer, achieving expert-level driving scores on CARLA Longest6, 5.3× faster inference.
Key Findings
Methodology
This paper introduces PlanT, a Transformer-based planner trained via imitation learning, using sparse object features including vehicles and route segments. The model employs self-attention to fuse scene information, outputting future trajectories. An object relevance metric RFDS quantifies the model’s ability to identify critical objects. Experiments on CARLA’s Longest6 benchmark show superior performance over baselines like Roach and AIM-BEV, with inference speed increased by 5.3×, reaching expert-level scores. The approach combines a perception module for sensor input, enabling end-to-end perception-planning, demonstrating robustness and efficiency.
Key Results
- PlanT achieved a driving score of 76.91 on Longest6, surpassing the average expert score, with an inference time of only 5.46 ms—5.3× faster than pixel-based methods like PlanCNN. When combined with perception, it outperformed existing methods by over 10 points, confirming its practical viability.
- In sensor-based setups, PlanT outperformed TransFuser by 10.36 points and achieved 2.7× faster inference. RFDS analysis showed the model’s high accuracy in identifying key objects, enhancing interpretability and safety.
- Ablation studies confirmed the importance of full 360° view, vehicle speed, and route representation. The model demonstrated excellent scalability with data and model size, maintaining high data efficiency and outperforming baselines in various configurations.
Significance
This work advances autonomous driving by replacing dense pixel-level scene understanding with sparse object-level features, significantly improving decision speed and transparency. The approach addresses longstanding challenges of computational cost and explainability, offering a scalable, interpretable solution that aligns with human reasoning. Its success in simulation paves the way for safer, more trustworthy real-world deployment, with the RFDS metric providing a new standard for evaluating model interpretability. Overall, it marks a significant step toward practical, explainable autonomous vehicles.
Technical Contribution
Technically, the paper introduces a simple yet effective Transformer architecture that directly processes sparse object features, eliminating the need for dense scene encoding. The model leverages self-attention for scene fusion, combined with imitation learning to train on expert trajectories, and introduces RFDS for interpretability. The architecture achieves high inference speed and performance, outperforming pixel-based methods. This work bridges the gap between scene understanding and decision transparency, offering a lightweight, scalable solution for autonomous driving.
Novelty
This is the first systematic validation of object-level sparse representations in dense traffic scenarios, demonstrating their effectiveness over traditional pixel-based BEV inputs. The use of a standard Transformer with a novel scene encoding, combined with a new interpretability metric RFDS, provides a fresh perspective on scene understanding and decision explanation in autonomous driving. Unlike prior works relying on dense semantic maps, this approach simplifies the input, improves speed, and enhances explainability, representing a significant innovation.
Limitations
- The model’s robustness in extreme weather or highly congested scenarios remains untested; perception errors could impact safety. Further validation in real-world conditions is needed.
- Dependence on perception modules means that sensor failures or inaccuracies could degrade overall performance, highlighting the need for more resilient perception strategies.
- Handling rare or abnormal traffic situations, such as unexpected obstacles or unusual behaviors, remains challenging and requires further research.
Future Work
Future directions include integrating multimodal sensors for richer scene understanding, applying reinforcement learning for adaptive decision-making, and expanding object relationship modeling to improve interpretability. Enhancing robustness in adverse conditions and deploying in real-world environments are key goals. Additionally, developing more comprehensive scene representations and explainability tools will further bridge the gap between research and practical autonomous driving systems.
AI Executive Summary
Autonomous driving faces the challenge of balancing scene understanding, decision speed, and transparency. Traditional pixel-level scene encoding methods, while accurate, are computationally intensive and lack interpretability. This study introduces PlanT, a novel planning framework that leverages sparse object-level features processed through a standard Transformer architecture. By focusing on vehicles and route segments, the model captures essential scene information efficiently, enabling rapid inference and high-quality decision-making. In the CARLA Longest6 benchmark, PlanT achieved a driving score of 76.91, surpassing the expert level, with an inference time of just 5.46 ms—5.3 times faster than pixel-based baselines like PlanCNN. The model was trained via imitation learning, using expert trajectories, and incorporated an auxiliary task to predict other vehicles' future states, further enhancing its understanding.
A key innovation is the introduction of RFDS, a metric that quantifies the model’s ability to identify the most relevant objects influencing safety decisions. Results show that PlanT’s attention weights reliably highlight critical scene elements, providing transparency and interpretability. When combined with a perception module, the system outperformed existing sensor-based methods such as TransFuser by over 10 points in driving score, demonstrating its practical potential.
The approach emphasizes the importance of a full 360° view, vehicle speed, and route information, validated through extensive ablation studies. Its scalability across data and model sizes suggests strong applicability in real-world scenarios. While promising, challenges remain in handling adverse weather, sensor failures, and rare traffic situations. Future work aims to incorporate multimodal perception, reinforcement learning, and richer scene modeling to improve robustness and explainability further. Overall, PlanT represents a significant step toward efficient, transparent, and safe autonomous driving systems, bridging the gap between high-performance decision-making and human-like scene reasoning.
Deep Analysis
Background
自主驾驶技术经历了从规则驱动到深度学习的演变。早期依赖手工设计规则，难以应对复杂环境。近年来，卷积神经网络（CNN）在感知任务中表现突出，如Baidu Apollo和Tesla Autopilot，推动了端到端学习的发展。使用像素级BEV图像作为输入，虽然提升了场景理解，但计算成本高，且缺乏可解释性。近年来，研究逐渐转向对象级表示和Transformer架构，旨在提升效率和透明度。相关工作如Roach、AIM-BEV验证了对象级表示的潜力，但缺乏系统性分析。本文提出结合Transformer的稀疏对象特征，突破了现有瓶颈。
Core Problem
现有模型多依赖密集像素级场景编码，导致推理慢、解释性差，难以满足实际需求。复杂交通中，关键对象识别和优先级排序不足，影响安全决策。如何在保证性能的同时提升模型的可解释性，成为行业难题。此外，模型在多模态感知融合、场景理解和决策透明度方面仍有提升空间。
Innovation
提出对象级稀疏表示，简化场景输入，降低计算负担。采用标准Transformer架构，通过自注意力机制融合信息。引入RFDS指标，量化关键对象识别能力，增强可解释性。结合模仿学习，避免复杂标注，模型在速度和性能上优于像素基础方法。创新点还包括全景视野和对象关系建模，为自主驾驶提供新思路。
Methodology
- �� 场景表示：以车辆和路线段为对象，提取位置、朝向、速度等属性。• 输入编码：线性投影对象特征，加入对象类型向量。• Transformer编码：采用BERT架构的自注意力机制，融合所有对象信息。• 轨迹预测：利用[CLS] token输出，通过GRU解码器预测未来路径。• 训练目标：模仿专家轨迹，结合辅助任务预测其他车辆未来状态，优化L1和交叉熵损失。• 相关性分析：通过注意力权重和RFDS指标，评估关键对象识别。
Experiments
在CARLA Longest6基准上，使用228k帧数据，比较不同模型（如PlanCNN、Roach、AIM-BEV）和不同输入配置（全景视野、速度信息）。采用驾驶得分（DS）、路径完成率（RC）和违规惩罚（IS）作为评价指标。模型在不同数据规模和模型容量下训练，验证其数据效率和泛化能力。引入感知模块后，系统表现优异。消融实验验证全景视野、速度信息的重要性，模型在推理时间和性能上均优于基线。
Results
PlanT在Longest6达到了76.91的驾驶得分，超过专家水平，推理时间仅为5.46毫秒，比像素基础方法快5.3倍。结合感知模块后，性能提升10点以上，显著优于TransFuser。RFDS指标显示模型能准确识别最关键的对象，提升决策透明度。不同数据和模型规模的实验验证了模型的良好扩展性和效率，展现出实际部署潜力。
Applications
该模型适用于自动驾驶车辆的感知与决策系统，尤其在复杂交通环境中实现高效、安全的路径规划。依赖对象级稀疏表示，减少对高成本像素级场景理解的依赖，适合实时应用。未来可结合多模态感知和强化学习，提升系统的鲁棒性和自主性，推动自动驾驶产业的广泛落地。
Limitations & Outlook
模型在极端天气或交通密集场景中的鲁棒性仍需验证，感知误差可能影响整体性能。对复杂异常场景的适应能力有限，未来需增强模型的泛化能力。模型对感知模块的依赖较大，感知性能下降会影响决策效果。计算成本虽低，但在更大规模场景中仍需优化。
Plain Language Accessible to non-experts
想象你在开车时，司机会专注于前方重要的车和路标，而忽略一些不那么重要的细节。自动驾驶系统也一样，传统的方法会把所有场景信息都装进电脑里，像拼图一样拼成一幅大图，然后再做决定。这不仅很慢，还不容易理解为什么会这么做。这个研究提出一种新方法，就像司机只关注最重要的几辆车和路段，用简单的标签描述它们。然后，利用一种叫Transformer的技术，让电脑像人一样专注于这些关键对象，快速做出反应。结果显示，这种方法不仅比传统的拼图方式快5倍，还能像专家一样安全驾驶，还能告诉你它为什么关注某个对象，就像司机告诉你“我注意到前面那辆车很快，要小心”。这让自动驾驶变得更聪明、更透明，也更适合实际应用。
ELI14 Explained like you're 14
想象你在玩一款赛车游戏，你只需要关注前面的几辆车，知道它们的速度和位置，然后决定什么时候加速或刹车。你不用记住每一辆车的详细信息，只关注那些会影响你安全的车。这就像你在开车时会专注于最重要的东西，而忽略掉一些无关紧要的细节。这个研究也是一样，科学家们让电脑只关注那些关键的车辆和路线段，而不是把整个场景都装进去。用一种叫Transformer的技术，电脑可以像你一样，专注于最重要的对象，快速做出反应。这样，电脑不仅可以更快地做决定，还能告诉你它为什么关注某个对象，比如“我注意到那辆车很快，可能会变道”。这让自动驾驶变得更聪明、更可靠，也更容易理解它的决策过程。
Glossary
Transformer（变换器）
一种基于自注意力机制的深度学习模型，用于融合序列信息，提升模型理解能力。
在本文中，Transformer用于场景信息的融合与决策生成。
对象级表示（Object-Level Representation）
用稀疏的对象特征集描述场景，避免密集像素信息，提升效率与可解释性。
作为输入特征，替代传统的像素级场景编码。
RFDS（相对过滤驾驶得分）
衡量模型识别关键对象能力的指标，通过限制观察对象验证其对安全决策的贡献。
用于评估模型的可解释性和对象识别能力。
模仿学习（Imitation Learning）
通过模仿专家行为训练模型，使其学习到类似的决策策略。
训练PlanT模型的主要策略。
CARLA（模拟平台）
一个用于自动驾驶研究的高仿真环境，支持多种传感器和场景模拟。
本文中的数据集和评估基准。
Open Questions Unanswered questions from this research
- 1 在极端复杂或非规则交通环境中，模型的鲁棒性和泛化能力仍需验证，特别是在感知误差较大的情况下。
- 2 如何进一步提升模型对异常场景的适应能力，以及在多模态感知融合中的优化策略，仍是未来研究方向。
Applications
Immediate Applications
自动驾驶路径规划系统
结合对象级表示和Transformer架构，提升车辆在复杂交通中的决策速度和安全性，适用于未来自动驾驶车辆的核心控制模块。
智能交通管理与监控
利用模型对关键交通对象的识别能力，优化交通流调度和事故预警，提升城市交通智能化水平。
Long-term Vision
全自动无人驾驶生态系统
实现端到端自主驾驶，从感知、理解到决策的全流程自动化，推动无人驾驶商业化和普及。
Abstract
Planning an optimal route in a complex environment requires efficient reasoning about the surrounding scene. While human drivers prioritize important objects and ignore details not relevant to the decision, learning-based planners typically extract features from dense, high-dimensional grid representations containing all vehicle and road context information. In this paper, we propose PlanT, a novel approach for planning in the context of self-driving that uses a standard transformer architecture. PlanT is based on imitation learning with a compact object-level input representation. On the Longest6 benchmark for CARLA, PlanT outperforms all prior methods (matching the driving score of the expert) while being 5.3x faster than equivalent pixel-based planning baselines during inference. Combining PlanT with an off-the-shelf perception module provides a sensor-based driving system that is more than 10 points better in terms of driving score than the existing state of the art. Furthermore, we propose an evaluation protocol to quantify the ability of planners to identify relevant objects, providing insights regarding their decision-making. Our results indicate that PlanT can focus on the most relevant object in the scene, even when this object is geometrically distant.