Driving on Registers
DrivoR uses pretrained ViT and register tokens to compress multi-camera features, enabling efficient end-to-end autonomous driving with superior performance.
Key Findings
Methodology
This paper introduces DrivoR, an end-to-end autonomous driving architecture based on pretrained Vision Transformers (ViT). The key innovation is the use of camera-aware register tokens that compress multi-camera visual features into a compact scene representation, significantly reducing downstream computational load. The model comprises three Transformer modules: perception encoder, trajectory decoder, and scoring decoder. The perception encoder fine-tunes ViT-S with LoRA, extracting visual features and generating scene tokens via register tokens. The trajectory decoder produces candidate trajectories from learned queries, while the scoring decoder evaluates and scores these trajectories by mimicking oracle scores, predicting sub-scores for safety, comfort, and efficiency, enabling behavior-conditioned driving. Training employs Winner-Takes-All loss and sub-score binary cross-entropy, with inference interpreting scores as rewards for diverse behavior modulation.
Key Results
- On NAVSIM-v1 and NAVSIM-v2 datasets, DrivoR surpasses or matches state-of-the-art methods, achieving PDMS scores of 94.6 on NAVSIM-v1 and 54.6 on NAVSIM-v2’s EPDMS, with only ~40M parameters and 3× faster inference. It demonstrates superior safety and efficiency metrics, with robust generalization in HUGSIM zero-shot tests showing 49.8 road completion and 35.7 HD-Score.
- Ablation studies confirm that register tokens significantly improve efficiency and performance, and the decoupled scoring module enhances behavior control. The model maintains robustness across various data augmentations and fine-tuning strategies.
- The approach achieves a 3× throughput increase over baseline ViT-L models, with reduced GFLOPS and memory, validating its suitability for real-time deployment in multi-camera scenarios.
Significance
This work demonstrates that a pure Transformer-based architecture with targeted token compression can effectively address the computational bottleneck in visual perception for autonomous driving. It simplifies the system by avoiding complex intermediate representations or large trajectory vocabularies, making end-to-end models more practical for real-world deployment. The behavior-conditioned scoring mechanism offers a flexible way to personalize and enhance safety, marking a significant step toward scalable, adaptive autonomous driving systems with broad industry impact.
Technical Contribution
The paper pioneers the application of ViT register tokens for multi-camera scene compression in autonomous driving, reducing input sequence length and computational complexity. The decoupled decoder architecture separates trajectory generation from scoring, enabling multi-modality and controllability. The sub-score prediction module allows behavior modulation without retraining, enhancing flexibility. The overall design is simple, parameter-efficient, and achieves state-of-the-art results, advancing the theoretical understanding and engineering practice of vision-based end-to-end driving models.
Novelty
This is the first work to leverage ViT register tokens specifically for visual feature compression in multi-camera end-to-end driving. It introduces a decoupled trajectory generation and scoring framework, with a novel behavior modulation mechanism via sub-score weighting. Unlike prior methods relying on large models or complex intermediate representations, this approach emphasizes simplicity, efficiency, and interpretability, filling a critical gap in the literature.
Limitations
- The model's performance may degrade under adverse weather or occlusion conditions, as it heavily depends on visual input quality. Its reliance on high-quality cameras limits robustness in real-world scenarios.
- Pure Transformer architecture, while efficient, still faces computational challenges at very high resolutions or with increasing sensor counts, requiring further optimization.
- Behavior modulation via sub-score weighting, though flexible, needs extensive validation in safety-critical applications to prevent unintended behaviors.
Future Work
Future directions include integrating multi-modal sensors such as LiDAR and high-definition maps to improve perception robustness. Developing online adaptation and reinforcement learning strategies could enhance real-world deployment. Further research on multi-objective optimization and safety guarantees will be essential to ensure reliable and trustworthy autonomous driving in diverse environments.
AI Executive Summary
Autonomous driving has long sought a balance between perception accuracy and computational efficiency. Traditional methods often rely on complex perception pipelines with heavy feature extraction, leading to high latency and limited scalability. Recent advances in Vision Transformers (ViT) have shown promise for visual understanding, but their application in real-time driving remains constrained by the enormous sequence length and computational cost. Addressing this challenge, the authors propose DrivoR, a novel architecture that leverages pretrained ViT models combined with camera-aware register tokens to compress multi-camera visual features into a compact scene representation.
The core idea is to replace traditional pooling or dense feature maps with a small set of learned tokens that encapsulate scene context efficiently. These tokens serve as inputs to two lightweight Transformer decoders: one generates multiple candidate trajectories, and the other scores these candidates by predicting sub-scores related to safety, comfort, and efficiency. The scoring module is trained to mimic oracle scores, enabling the system to evaluate and select trajectories conditioned on desired behaviors. This design allows the model to adapt dynamically during inference, supporting personalized driving styles.
Extensive experiments on NAVSIM-v1, NAVSIM-v2, and HUGSIM benchmarks demonstrate that DrivoR outperforms or matches state-of-the-art methods, with a parameter count of only 40 million and inference speed three times faster than comparable ViT-based models. The results highlight the effectiveness of targeted token compression and the decoupled generation-scoring architecture. The approach not only achieves high accuracy but also significantly reduces computational overhead, making real-time deployment feasible.
The significance of this work lies in its demonstration that a simple, pure Transformer architecture can meet the demanding requirements of autonomous driving. By focusing on efficient scene representation and behavior-conditioned decision-making, the authors open new avenues for scalable, adaptable, and interpretable autonomous systems. Future work will explore multi-modal sensor fusion, online learning, and safety guarantees, aiming to bring this technology closer to widespread industry adoption.
Deep Analysis
Background
自动驾驶技术经历了从规则基础到深度学习的演变，卷积神经网络（如ResNet、VoV-Net）在感知任务中取得了显著成功，但在多摄像头、多模态融合和实时性方面仍存在瓶颈。Transformer架构（如ViT、EVA、DINO）逐渐展现出优越的特征表达能力，推动端到端学习的发展。现有方法多依赖复杂的中间表示（如鸟瞰图BEV）或庞大的轨迹词典，导致系统复杂、计算成本高。近年来，场景压缩和特征融合成为研究热点，旨在在保证感知精度的同时降低计算负担。尽管如此，如何在保证高性能的同时实现高效、可控的端到端规划仍是未解难题。
Core Problem
核心问题在于多摄像头视觉信息的高维特征带来的计算瓶颈。传统的池化方法虽然减缓了负担，但会丢失关键空间信息，影响路径规划的准确性。现有模型在保持高准确率的同时难以实现实时性，特别是在复杂场景和高分辨率下表现不佳。此外，缺乏有效的场景压缩机制限制了模型的扩展性和行为调控能力，亟需一种既能压缩信息又能保留关键上下文的方案，以满足实际部署需求。
Innovation
本文提出注册令牌机制，将每个摄像头的特征通过微调的ViT生成一组紧凑的场景令牌，有效压缩多视角信息。引入的感知编码器无需复杂中间表示，简洁高效。轨迹生成和评分两个解码器相互解耦，增强多样性和可控性。子分数预测机制允许根据不同偏好调节驾驶行为，提升个性化和安全性。整体架构简洁，参数少，训练快，性能优越，突破了视觉特征压缩在端到端自主驾驶中的应用瓶颈。
Methodology
- �� 感知编码器：采用预训练ViT-S提取视觉特征，通过注册令牌实现多摄像头信息的压缩，微调采用LoRA技术。• 轨迹解码器：输入学习的轨迹查询，结合ego状态，生成多候选轨迹，采用Winner-Takes-All损失训练。• 评分解码器：将轨迹转为查询，结合场景令牌进行评分，学习子分数预测，支持行为调节。• 训练目标：结合轨迹和评分的损失函数，优化模型参数。• 推理阶段：将评分视为奖励函数，支持多样化行为调节。• 关键技术：注册令牌压缩、多解码器解耦、子分数调节，确保模型高效、灵活。
Experiments
使用NAVSIM-v1、NAVSIM-v2和HUGSIM数据集，比较多种基线模型，指标包括PDMS、EPDMS和道路完成率。训练采用Adam优化器，微调ViT采用LoRA，模型参数约40M。消融实验验证注册令牌的有效性，模型在不同场景下表现出优越的鲁棒性和泛化能力。通过对比不同数据增强策略，分析模型在复杂环境中的适应性。
Results
在NAVSIM-v1测试集，DrivoR达94.6的PDMS，优于大部分对比方法，参数少且计算快。在NAVSIM-v2中，EPDMS达54.6，领先于GTRS和RAP。HUGSIM闭环测试中，零样本迁移表现优异，道路完成率达49.8，HD-Score 35.7。消融实验显示注册令牌显著提升效率和性能，行为调控机制增强了模型的调节能力。
Applications
该模型适用于多摄像头自动驾驶系统，特别在城市复杂场景中实现实时感知和决策。只需摄像头输入，无需额外传感器，便于部署。可用于无人驾驶、辅助驾驶等场景，提升安全性和效率。未来结合激光雷达和地图信息，将进一步增强系统鲁棒性。
Limitations & Outlook
模型对极端天气和遮挡敏感，感知性能可能下降。纯Transformer架构在高分辨率场景中仍存在计算瓶颈。训练依赖模拟数据，实际环境适应性待验证。行为调控机制在多目标优化中尚需完善，未来需考虑多模态融合和强化学习以提升鲁棒性。
Plain Language Accessible to non-experts
想象你在厨房做饭，厨房里有很多不同的厨具和食材。每次做饭时，你需要看清所有食材的状态、位置和用量，然后决定用哪些厨具、放在哪个锅里。传统的方法就像用大锅把所有食材都搅在一起，虽然简单，但会丢失很多细节，做出来的菜可能不够好。现在，科学家们设计了一种聪明的“厨师助手”，它用一种特殊的“魔法眼镜”——叫做ViT，把每个摄像头看到的场景变成一组紧凑的“魔法符号”。这些符号像是厨房的简洁地图，告诉你哪些食材在哪、需要注意什么。这个助手还能根据不同的需求，比如快点做完或做得更安全，调整做菜的策略。它可以快速地帮你规划出一份菜谱，并判断哪条路线最安全、最舒服。这个系统就像一个聪明的厨房助手，既节省时间，又能做出高品质的菜肴，未来还能根据你的偏好，帮你做出个性化的美味佳肴。
ELI14 Explained like you're 14
想象你在玩一个超级复杂的游戏，你的任务是带着一辆车在城市里跑。以前的方法就像用一堆大地图，把所有路、车、行人都画在一起，然后试图找到最好的路线，但地图太大，计算起来很慢。现在，这个新方法像是给你准备了一份特别的简洁地图，只用几个神奇的符号代表不同的视角，比如前面、左边、右边的路况。这些符号是用一种叫ViT的“魔法眼镜”生成的，能快速帮你理解整个场景。然后，系统会帮你画出几条可能的路线，像是“我可以走这条、那条”，并给每条路线打分，比如安全、舒服、快。你还可以告诉它，你更关心安全还是速度，它就会帮你调整选择。这样一来，你的车就能更快、更聪明地在城市里跑，避免危险，还能根据你的偏好做出不同的决策。就像你有一个超级智能的导航助手，帮你安全又快速地到达目的地！
Glossary
Vision Transformer（ViT，视觉变换器）
一种基于Transformer架构的视觉特征提取模型，能有效捕捉图像中的长距离依赖关系。技术上，它将图像划分为块，像处理文本一样进行注意力计算。
在本文中，ViT用于提取多摄像头的视觉特征，并通过注册令牌实现场景压缩。
注册令牌（Register Tokens）
一种特殊的Token，用于编码每个摄像头的感知信息，压缩多视角特征为紧凑的场景表示。它们在Transformer中作为场景的“摘要”。
本文引入注册令牌，用于替代传统池化操作，提升信息压缩效率。
Winner-Takes-All（WTA，赢家通吃）损失
一种训练策略，只对最接近真实轨迹的候选轨迹进行监督，鼓励模型输出多样化的轨迹候选。
用于训练轨迹生成解码器，增强模型的多模态表达能力。
EPDMS（扩展预测驾驶模型评分）
一种综合衡量自主驾驶行为的指标，考虑安全、舒适、效率等多个子分数，用于评估模型性能。
在NAVSIM-v2中作为主要评估指标。
行为调控（Behavior Conditioning）
通过调整评分子分数的权重，实现对驾驶行为偏好的控制，比如偏重安全或速度。
本文模型支持根据不同偏好调节轨迹选择。
Open Questions Unanswered questions from this research
- 1 如何在极端复杂环境（如极端天气、夜间）下保持感知和决策的鲁棒性仍需研究，尤其是多模态融合策略的优化。
- 2 模型在实际部署中的在线学习和适应能力尚未充分验证，未来需结合强化学习或在线微调技术提升适应性。
- 3 在多目标、多约束场景下的行为调控机制还需完善，确保安全性与效率的平衡。
Applications
Immediate Applications
城市自动驾驶导航
可在城市复杂交通环境中实现实时路径规划和行为调节，降低硬件成本，提升安全性。
辅助驾驶系统
为高级驾驶辅助提供高效感知和决策支持，增强驾驶体验与安全保障。
Long-term Vision
全自动无人驾驶
结合多模态感知和强化学习，打造自主、智能、个性化的无人驾驶生态系统，逐步实现商业化。
Abstract
We present DrivoR, a simple and efficient transformer-based architecture for end-to-end autonomous driving. Our approach builds on pretrained Vision Transformers (ViTs) and introduces camera-aware register tokens that compress multi-camera features into a compact scene representation, significantly reducing downstream computation without sacrificing accuracy. These tokens drive two lightweight transformer decoders that generate and then score candidate trajectories. The scoring decoder learns to mimic an oracle and predicts interpretable sub-scores representing aspects such as safety, comfort, and efficiency, enabling behavior-conditioned driving at inference. Despite its minimal design, DrivoR outperforms or matches strong contemporary baselines across NAVSIM-v1, NAVSIM-v2, and the photorealistic closed-loop HUGSIM benchmark. Our results show that a pure-transformer architecture, combined with targeted token compression, is sufficient for accurate, efficient, and adaptive end-to-end driving. Code and checkpoints will be made available via the project page.