Unified Driving Tokens: Representation- and Geometry-Guided Discrete Tokenizer for Driving World Models and Planning
Proposes a unified, representation- and geometry-guided discrete tokenizer for autonomous driving, improving scene reconstruction and planning performance.
Key Findings
Methodology
This work introduces a representation-guided and geometry-enhanced discrete tokenizer based on VQ-VAE. It leverages frozen DINO features for semantic alignment, employs cross-frame depth and relative pose supervision, and uses multi-codebook quantization for stability. The model jointly reconstructs RGB and semantic features, ensuring appearance fidelity and semantic consistency. Experiments on NAVSIM demonstrate high-fidelity scene reconstruction, semantic alignment, and effective geometric encoding. The learned tokens support lightweight planning and autoregressive world modeling, outperforming baselines in both generation quality and planning accuracy.
Key Results
- On NAVSIM, the proposed model achieves a reconstruction rFID of 4.15, PSNR of 26.51, and SSIM of 0.774, surpassing existing methods. Depth prediction errors (AbsRel) are reduced to 0.0556, with pose errors (Trans: 0.647m, Rot: 2.01°). In planning tasks, the discrete tokens enable a lightweight trajectory predictor to improve path accuracy and safety metrics. The autoregressive transformer attains over 85% token prediction accuracy, producing continuous and diverse scene sequences, validating the multi-task capability of the tokens.
- Multi-codebook quantization effectively alleviates capacity conflicts, maintaining high-quality appearance reconstruction while encoding semantic and geometric cues. The autoregressive world model demonstrates superior scene generation and prediction, with improved continuity and diversity compared to baselines. These results confirm the tokens' versatility across multiple tasks, supporting their role as a unified scene representation.
- Reusing the same discrete token interface across scene understanding, planning, and generation tasks simplifies system design and enhances efficiency. The approach paves the way for integrated multi-task autonomous systems, with potential extensions to multi-modal data and reinforcement learning for safer, more robust driving.
Significance
This research addresses the longstanding challenge of designing a unified scene representation that supports both high-quality scene reconstruction and effective planning in autonomous driving. By integrating semantic and geometric cues into a discrete token framework, it bridges the gap between generative modeling and decision-making modules. The multi-codebook mechanism and cross-frame supervision ensure stable training and continuous scene understanding, which are crucial for real-world deployment. The approach not only advances theoretical understanding of discrete scene representations but also offers practical pathways for safer, more reliable autonomous vehicles. Its ability to unify perception, prediction, and planning within a single discrete interface marks a significant step toward holistic autonomous systems, potentially transforming industry standards and safety protocols.
Technical Contribution
The paper introduces a novel discrete tokenizer that combines semantic alignment via frozen DINO features with geometric supervision from adjacent frames, utilizing multi-codebook quantization for capacity enhancement. It extends VQ-VAE with joint RGB, semantic, and geometric reconstruction, ensuring appearance fidelity and scene structure preservation. The framework supports both lightweight planning decoders and autoregressive world models, demonstrating versatility. The multi-task training stabilizes codebook utilization and improves scene continuity, offering a comprehensive solution for multi-task scene understanding and generation in autonomous driving. This work bridges the gap between generative and discriminative scene representations, providing a scalable, unified approach.
Novelty
This is the first work to explicitly integrate semantic representation alignment and geometric supervision within a discrete tokenization framework tailored for autonomous driving. The multi-codebook quantization mechanism addresses capacity limitations, enabling simultaneous encoding of appearance, semantics, and geometry. Unlike prior methods that focus solely on pixel fidelity or semantic features, this approach creates a unified, multi-faceted scene representation that supports diverse downstream tasks, including planning and scene generation, with demonstrated superior performance.
Limitations
- The model's performance may degrade in highly dynamic or cluttered environments where geometric and semantic cues are ambiguous or noisy. Its reliance on accurate depth and pose supervision limits applicability in scenarios with sparse or unreliable sensor data.
- Increased model complexity due to multi-codebook quantization and cross-frame supervision results in higher computational costs, posing challenges for real-time deployment on resource-constrained platforms.
- Generalization to real-world, diverse driving conditions remains to be validated, as the current evaluation is primarily on the NAVSIM dataset. Further robustness testing and domain adaptation are necessary.
Future Work
Future directions include integrating multi-modal data such as LiDAR and radar to enhance scene understanding, optimizing model architectures for real-time deployment, and exploring reinforcement learning strategies to improve decision-making. Extending the framework to handle more complex, dynamic environments and validating on diverse real-world datasets will be crucial for practical deployment.
AI Executive Summary
Autonomous driving systems require a comprehensive understanding of complex, dynamic scenes to ensure safety and efficiency. Traditional visual representations often focus solely on pixel-level fidelity, which limits their ability to support high-level scene understanding and planning. Existing methods struggle to unify scene reconstruction, semantic comprehension, and geometric reasoning within a single framework, hindering the development of robust, scalable autonomous systems.
This paper introduces a novel discrete scene tokenizer that is both representation-guided and geometry-enhanced. Built upon the VQ-VAE architecture, it leverages frozen DINO features for semantic alignment, ensuring that the tokens encode meaningful scene semantics. Simultaneously, the model incorporates cross-frame depth and relative pose supervision, capturing the scene's geometric structure and temporal continuity. To address capacity limitations and stabilize training, the authors employ a multi-codebook quantization mechanism, which allows the tokens to encode appearance, semantics, and geometry simultaneously.
Experimental results on the NAVSIM dataset demonstrate the effectiveness of this approach. The proposed tokenizer achieves high-fidelity scene reconstruction with a rFID of 4.15, PSNR of 26.51, and SSIM of 0.774, outperforming baseline methods. Depth and pose predictions further validate the geometric encoding capabilities, with errors significantly lower than previous models. In downstream tasks, the tokens enable a lightweight trajectory planner to improve path accuracy and safety metrics, while a GPT-style autoregressive world model generates diverse, continuous scene sequences, confirming the tokens' versatility.
The unified discrete interface simplifies multi-task learning, bridging perception, prediction, and generation. It offers a promising pathway toward more integrated, efficient autonomous driving systems capable of understanding and predicting complex environments. Despite current limitations in dynamic scenes and computational costs, this work lays a solid foundation for future multi-modal, real-world autonomous systems, with potential to revolutionize industry standards and safety protocols.
Deep Dive
Plain Language Accessible to non-experts
想象你在一个大厨房里做饭。每次做菜，你都需要知道每个食材的状态、位置和用量。以前，只是简单地看食材的颜色和形状，不能完全理解它们的关系，也不知道它们未来会变成什么样。现在，厨师们发明了一种神奇的工具，可以把每个食材变成一串特殊的符号，这些符号不仅告诉你食材的外观，还能告诉你它们在厨房里的位置、相互关系，甚至未来会变成什么样。这样，你就可以更快、更好地准备菜肴，避免食材混乱或做错菜。这就像论文中的离散标记器，不仅能帮汽车“看清”周围的场景，还能理解场景的结构和运动，为安全驾驶提供了强有力的支持。
ELI14 Explained like you're 14
想象你在玩一个超级复杂的积木游戏，你要搭建一个城市模型。以前，你只知道每个积木的颜色和形状，但不知道它们之间的关系。现在，想象你有一种神奇的魔法，可以让你不仅看到积木的外表，还能知道它们在空间中的位置、相互的关系，甚至未来会变成什么样。这就像论文里说的用特殊的“魔法”把场景变成一串特殊的符号，这些符号既能帮你快速搭建，也能让你预测未来的场景。这样，你就可以更快、更安全地驾驶汽车，像玩积木一样搭建未来的道路。是不是很酷？这就是这项新技术的神奇之处！
Abstract
Discrete visual tokens should provide a compact representation for both token-based world modeling and planning in autonomous driving. However, most tokenizers are inherited from image generation and are optimized mainly for pixel reconstruction, which may leave a gap between what is easy to generate and what is useful to decode for driving decisions. We present a representation-guided and geometry-enhanced tokenizer that learns discrete tokens under joint supervision. The tokenizer aligns its discrete bottleneck with a frozen DINO feature space through feature decoding, while preserving appearance via RGB reconstruction with perceptual and adversarial losses. To inject geometric state-related cues, we add adjacent-frame depth and relative-pose supervision during training and stabilize joint objectives with multi-codebook quantization. We evaluate the same learned tokens with a lightweight planning readout and a GPT-style next-token world model. Experiments on NAVSIM show improved reconstruction fidelity and representation consistency, competitive planning performance under a fixed decoder, and better generative quality under matched settings.