Teaching Vision-Language-Action Models What to See and Where to Look
DriveTeach-VLA enhances VLA models using DVD and 2D-TGP, achieving 90.4 PDMS on Navsim.
Key Findings
Methodology
DriveTeach-VLA employs Driving-aware Vision Distillation (DVD) for visual feature learning and 2D Trajectory-Guided Prompts (2D-TGP) for spatial guidance. DVD uses self-distillation with traffic-critical bounding boxes, while 2D-TGP projects trajectories onto the image plane.
Key Results
- On Navsim, DriveTeach-VLA achieves 90.4 PDMS, outperforming AutoVLA (89.1).
- On nuScenes, it achieves 0.30 L2 error and 0.12 collision rate, the best among benchmarks.
- Ablation studies confirm that DVD and 2D-TGP significantly improve spatial attention and trajectory prediction accuracy.
Significance
This work addresses the lack of spatial dependency in existing VLA models by introducing driving-specific visual and spatial priors, significantly improving trajectory prediction reliability. It opens new directions for multimodal learning in autonomous driving.
Technical Contribution
Introduced DVD and 2D-TGP, enabling tight coupling between perception and planning. DVD enhances visual encoders via self-distillation, while 2D-TGP aligns trajectory planning with spatial reasoning.
Novelty
First to apply 2D-TGP to VLA models, combined with DVD to improve spatial understanding and trajectory prediction.
Limitations
- 2D-TGP relies on accurate camera calibration, which may fail under poor sensor conditions.
- High computational cost limits deployment in resource-constrained environments.
- Performance in highly dynamic scenes remains to be validated.
Future Work
Future work could explore more efficient DVD methods to reduce computational overhead and adapt 2D-TGP for multi-sensor fusion scenarios.
AI Executive Summary
Existing Vision-Language-Action (VLA) models for autonomous driving struggle with spatial dependencies, leading to unreliable trajectory predictions. DriveTeach-VLA addresses this by introducing Driving-aware Vision Distillation (DVD) and 2D Trajectory-Guided Prompts (2D-TGP). DVD enhances visual encoders with traffic-critical features, while 2D-TGP projects trajectories onto the image plane for precise spatial guidance.
On the Navsim benchmark, DriveTeach-VLA achieves 90.4 PDMS, surpassing the prior best AutoVLA (89.1). On nuScenes, it achieves 0.30 L2 error and 0.12 collision rate, setting new records. Ablation studies confirm the critical role of DVD and 2D-TGP in improving spatial attention and trajectory prediction.
Despite its advancements, the method faces challenges like high computational costs and dependence on accurate sensor calibration. Future research could focus on optimizing DVD and adapting 2D-TGP for multi-sensor scenarios, paving the way for more reliable autonomous driving solutions.
Deep Analysis
Background
Vision-Language Models (VLMs) have shown promise in multimodal learning but face challenges in autonomous driving. Existing methods rely heavily on text-driven Visual Question Answering (VQA) data, lacking spatial and behavioral modeling.
Core Problem
Current VLA models underperform in trajectory prediction due to text-centric training that fails to capture critical spatial dependencies, making reliable planning difficult.
Innovation
DriveTeach-VLA introduces two innovations: DVD, which uses self-distillation to teach visual encoders traffic-critical features, and 2D-TGP, which projects trajectories onto the image plane for spatial guidance.
Methodology
- �� DVD: Uses Grounding DINO to detect critical objects and applies self-distillation to optimize ViT encoders.
- �� 2D-TGP: Projects expert trajectories onto the image plane to generate spatial prompts.
- �� Training: Combines DVD pretraining, SFT fine-tuning, and GRPO reinforcement learning.
Experiments
Experiments were conducted on Navsim and nuScenes datasets, using PDMS and L2 error as metrics. Ablation studies validated the impact of DVD and 2D-TGP.
Results
DriveTeach-VLA achieves 90.4 PDMS on Navsim and 0.30 L2 error on nuScenes, outperforming all baselines.
Applications
Applicable to trajectory planning in autonomous driving, especially in complex dynamic scenarios.
Limitations & Outlook
The model's reliance on accurate camera parameters and high computational demands may limit its deployment in real-world scenarios.
Plain Language Accessible to non-experts
Think of DriveTeach-VLA as a driving coach. DVD is like teaching you to notice important things on the road, like traffic lights or pedestrians. 2D-TGP is like drawing a clear route on a map, showing exactly where to go. Together, they help the model not just see but also understand how to act, making driving safer and more reliable.
ELI14 Explained like you're 14
Imagine you're playing a racing game, and there's a super-smart assistant. It tells you where obstacles are (like other cars or traffic lights) and draws the best route on the screen. DriveTeach-VLA is like that assistant, but for real self-driving cars! Cool, right?
Glossary
Vision-Language Model
A model combining visual and textual data for tasks like image captioning.
Used to extract multimodal information from images and text.
Driving-aware Vision Distillation
A method to teach visual encoders traffic-critical features via self-distillation.
Optimizes the model's visual encoder for driving tasks.
2D Trajectory-Guided Prompt
Projects trajectories onto the image plane to provide spatial guidance.
Guides trajectory prediction with precise spatial cues.
Self-Distillation
A training method where a teacher model guides a student model to improve.
Enhances the visual perception of the model.
PDMS (Planning-Driven Metric Score)
A composite metric for evaluating trajectory planning performance.
Used in Navsim benchmarks to compare model performance.
Open Questions Unanswered questions from this research
- 1 How can 2D-TGP be adapted for multi-sensor fusion scenarios?
- 2 How can DVD's computational cost be reduced for resource-constrained environments?
Applications
Immediate Applications
Autonomous Driving Trajectory Planning
Optimizes vehicle trajectories in urban traffic, reducing collisions and improving safety.
Advanced Driver Assistance Systems
Provides precise spatial guidance for ADAS in complex scenarios.
Long-term Vision
Fully Autonomous Driving
Enables highly reliable autonomous systems for dynamic and complex environments.
Abstract
Vision-Language-Action (VLA) models have emerged as a promising paradigm for end-to-end autonomous driving. However, existing VLAs' training relies heavily on text-centric visual question answering and chain-of-thought reasoning data, which emphasizes linguistic reasoning rather than action-grounded planning. As a result, the learned representations capture semantic knowledge but lack spatial dependencies crucial for reliable trajectory prediction. We propose DriveTeach-VLA, a framework that explicitly teaches VLAs what to see and where to look. Driving-aware Vision Distillation (DVD) injects driving-specific perceptual priors into the vision encoder, while 2D Trajectory-Guided Prompts (2D-TGP) provide spatial conditioning aligned with feasible driving trajectories. Together, they form a vision-guided learning pipeline: what to see (DVD pretraining) - where to look (TGP-guided SFT) - how to act (TGP-guided GRPO). DriveTeach-VLA achieves the state-of-the-art performance on NAVSIM and nuScenes. Our code is available at: https://github.com/ShivaTeam/DriveTeach-VLA.