NTR: Neural Token Reconstruction for Scene Token Bottleneck in End-to-End Driving
Introduces Neural Token Reconstruction (NTR) with masked latent reconstruction to enhance scene token representations, achieving 8.0461 RFS on Waymo E2E.
Key Findings
Methodology
NTR employs a self-distillation framework with masked latent reconstruction, where an online ViT encoder processes masked images to generate scene tokens, while an EMA teacher provides target latent features. During training, a lightweight decoder reconstructs masked patch-level features solely from scene tokens, guided by semantic priors derived from foundation models. This process ensures the scene tokens encode richer, less redundant visual information critical for planning. The entire setup is training-only, with no inference overhead, directly constraining the bottleneck to improve representation quality and planning performance.
Key Results
- On Waymo E2E, NTR achieves a state-of-the-art RFS of 8.0461, surpassing previous methods. On NavSim datasets, PDMS reaches 94.1, and EPDMS reaches 90.9, demonstrating consistent improvements. Scene token analysis shows lower redundancy and higher effective rank, confirming richer representations. Ablation studies reveal that combining masked latent reconstruction, EMA targets, and semantic priors yields the best performance, with RFS improving from 7.652 to 7.974.
- The method reduces scene token similarity and increases effective dimensionality, leading to more robust planning across diverse scenarios. Semantic priors focus supervision on critical regions, further enhancing structure preservation and safety-critical information. These results validate the effectiveness of direct bottleneck supervision in perception-free planning.
Significance
This work addresses a fundamental bottleneck in perception-free end-to-end autonomous driving: how to ensure compact scene tokens retain sufficient visual information for accurate planning. By directly supervising the representation through masked latent reconstruction, NTR significantly improves planning accuracy and robustness without increasing inference costs. It bridges the gap between dense visual features and compact planning inputs, enabling more reliable deployment in real-world scenarios. The approach offers a scalable, training-only solution that enhances the interpretability and efficiency of autonomous systems, paving the way for safer and more adaptable self-driving vehicles.
Technical Contribution
The key technical innovation is the integration of masked latent reconstruction within a self-distillation framework, which directly constrains the quality of scene token representations. This contrasts with prior works that rely solely on planning loss supervision. The use of foundation-model-derived semantic priors guides the reconstruction process toward structurally important regions, improving the semantic richness of the tokens. Additionally, the method maintains inference efficiency by removing auxiliary modules at deployment, making it practical for real-world systems. The combination of these techniques advances the state-of-the-art in representation learning for perception-free autonomous driving.
Novelty
This is the first work to introduce masked latent reconstruction as a supervision signal for compact scene tokens in perception-free driving. Unlike previous approaches that supervise only through planning loss, NTR explicitly constrains the information content of the scene tokens, leading to richer, less redundant representations. The integration of foundation-model-based semantic priors for target allocation further distinguishes this work, enabling structure-aware supervision without additional perception modules. These innovations collectively push the boundary of how compact visual representations can be learned and utilized for autonomous driving.
Limitations
- The method relies on the quality of foundation-model annotations for semantic priors, which may vary across domains and scenarios, potentially limiting generalization.
- The training process introduces additional complexity and computational overhead due to the reconstruction and self-distillation components.
- The approach primarily focuses on the scene token compression stage, leaving room for improvements in the underlying visual feature extraction backbone.
Future Work
Future directions include integrating more advanced pretraining techniques for the visual backbone, exploring multi-modal fusion to enhance robustness, and developing adaptive semantic priors for diverse environments. Additionally, extending the framework to handle rare or unseen scenarios and optimizing training efficiency are promising avenues to further improve real-world deployment.
AI Executive Summary
Autonomous driving has rapidly evolved, with end-to-end systems promising streamlined decision-making by directly mapping visual inputs to control commands. However, a persistent challenge remains: how to effectively compress rich visual information into a compact, planning-friendly representation without losing critical details. Existing methods often rely solely on planning objectives to supervise this compression, resulting in redundant and less informative scene tokens that hinder performance.
This paper introduces Neural Token Reconstruction (NTR), a novel framework that directly supervises the scene token bottleneck through masked latent reconstruction. During training, a lightweight decoder reconstructs masked patch-level features using only the compact scene tokens as memory, guided by semantic priors derived from foundation models. This process ensures the scene tokens encode richer, less redundant visual information, crucial for accurate planning.
The core of NTR lies in its self-distillation mechanism, where an online encoder and an EMA teacher collaborate to generate and refine latent targets. By focusing supervision on structured, driving-related regions, NTR effectively enhances the semantic quality of the tokens. Importantly, all auxiliary modules are removed at inference, ensuring no additional computational cost.
Extensive experiments on the Waymo E2E and NavSim datasets demonstrate that NTR achieves state-of-the-art results, with an RFS of 8.0461 on Waymo and PDMS of 94.1. The learned scene tokens exhibit lower redundancy and higher effective rank, indicating more efficient and meaningful representations. These improvements translate into more accurate, stable, and safer planning in complex traffic scenarios.
Overall, NTR offers a significant step forward in perception-free autonomous driving, combining representation learning with practical deployment considerations. Its ability to produce richer scene understanding without extra inference overhead makes it a promising approach for future autonomous systems, paving the way toward safer and more reliable self-driving vehicles.
Deep Dive
Plain Language Accessible to non-experts
想象你在整理一堆照片，要挑出最重要的部分放在一个小盒子里，方便以后快速找到关键内容。以前的方法就像只用一句话告诉你“这是街道”，但没有告诉你具体有什么车、行人、交通灯。现在，这个新方法像是用一种聪明的方式，不仅把重要的内容放进去，还能自己反复检查，确保没有遗漏关键细节。它用一种特殊的“记忆”机制，让这个小盒子装的内容更丰富、更有用。这样，无论你走到哪里，都能快速找到重要信息，做出正确的决定。这就像有个聪明的助手，帮你把复杂的场景变得简单又可靠。
ELI14 Explained like you're 14
嘿，你知道吗？开车时，车子要看很多东西，比如路标、车、行人、交通灯，但如果只用少少几句话就能知道路况，那就厉害了！这篇论文就像发明了一种超级聪明的记忆系统，让车子用很少的“关键词”就知道路上的重要细节。它会自己反复练习，确保这些关键词能记住车道、交通灯、行人在哪儿。这样，车子就能更快、更安全地开到目的地，就像你用一句话总结出一篇文章一样厉害。未来，这种技术还能让自动驾驶变得更聪明、更可靠！
Abstract
Recent perception-free end-to-end (E2E) autonomous driving methods bypass explicit perception outputs by compressing dense image patch tokens into compact scene tokens for downstream trajectory generation and scoring. While these scene tokens form a compact visual bottleneck for the planner, they receive supervision solely from the planning objective, providing limited constraints on the encoded visual information. To address this limitation, we introduce Neural Token Reconstruction (NTR), a representation learning framework to directly constrain the compact scene-token bottleneck in perception-free driving. NTR introduces a self-distillation masked latent reconstruction objective that reconstructs masked patch-level latent features using only compact scene tokens as reconstruction memory. This forces reconstruction gradients to pass exclusively through the scene-token bottleneck, encouraging scene tokens to preserve richer and less redundant visual representations for planning. We further introduce semantic priors derived from foundation-model annotations as a weak semantic interface biasing reconstruction targets toward driving-related structures without introducing explicit perception heads. All auxiliary reconstruction components are removed at inference time, leaving the deployed planner unchanged. NTR achieves state-of-the-art performance on three public autonomous driving benchmarks, including 8.0461 RFS on Waymo E2E and 94.1 PDMS / 90.9 EPDMS on NavSim1&2. The learned scene tokens exhibit lower pairwise redundancy and higher effective rank, indicating that effective bottleneck supervision improves both compact visual representation learning and planning performance.