RT-VLA: Real-Time Vision-Language-Action Models via Knowledge Distillation
RT-VLA employs multi-level knowledge distillation to compress SimLingo's capabilities into a real-time, efficient model, reducing inference time by 44.8× while maintaining performance.
Key Findings
Methodology
RT-VLA utilizes a multi-layer supervised distillation framework, transferring visual features, query representations, waypoint predictions, and language logits from the large teacher model SimLingo to a compact student. The architecture employs EVA-02 as the visual encoder and lightweight language modules, with feature alignment modules to match different dimensions. During training, multi-task loss functions optimize driving accuracy and language reasoning, while offline language analysis enables post-hoc explanations. The approach combines visual feature distillation, query and waypoint supervision, and language logit matching, with on-policy fine-tuning to enhance explanation consistency without increasing inference latency.
Key Results
- RT-VLA achieves a 44.8× speedup in vision-only mode (from 1544ms to 34.48ms per frame) and 7.9× in vision+language mode (from 1544ms to 196ms), while maintaining a driving score of 85.19, close to the teacher's 85.07. On Bench2Drive, it demonstrates real-time responsiveness and comparable decision quality.
- In safety-critical scenarios, RT-VLA responds earlier and executes maneuvers faster than SimLingo, significantly improving reaction times in urban dense traffic. Offline language explanations provide detailed scene descriptions post-incident, enhancing interpretability.
- The multi-level distillation combined with on-policy fine-tuning results in a lightweight model that balances high driving performance with low latency, making it suitable for real-world deployment. Ablation studies confirm each component's contribution to overall performance.
Significance
This work addresses the critical challenge of high inference latency in vision-language autonomous driving models. By compressing complex models into efficient, explainable systems, it paves the way for practical deployment in real traffic environments. The innovative multi-level distillation strategy offers a new paradigm for model compression, enabling high-performance, real-time decision-making with interpretability—key for safety and user trust. The approach bridges the gap between research and industry, facilitating scalable, safe autonomous vehicles.
Technical Contribution
The paper introduces a novel multi-layer supervised distillation framework that transfers visual, query, waypoint, and language logits from a large teacher to a lightweight student. It employs feature alignment modules for cross-model compatibility and integrates offline language analysis for post-hoc explanations. The two-stage training—initial driving optimization followed by language fine-tuning—ensures the model retains both high driving accuracy and interpretability. This method advances the state-of-the-art in model compression for multimodal autonomous driving, providing theoretical guarantees of performance retention and practical efficiency gains.
Novelty
This is the first application of multi-level supervision distillation in a vision-language autonomous driving context, combining visual features, path predictions, and language logits. Unlike prior works that focus solely on output distillation, this approach preserves deep feature representations and reasoning capabilities. The integration of offline language explanation further distinguishes it, enabling post-hoc interpretability without sacrificing real-time performance. This comprehensive strategy sets a new benchmark for efficient, explainable autonomous driving models.
Limitations
- The model relies heavily on simulation data, which may not fully capture real-world variability, potentially affecting robustness. Its performance under adverse weather or sensor failure remains untested.
- Lacking multimodal sensors like LiDAR, the model's perception in complex environments could be limited, especially in low visibility conditions. Safety guarantees are not explicitly incorporated, posing risks in critical scenarios.
- The approach assumes the availability of high-quality training data and accurate annotations. Long-tail and rare events may still challenge the model's reliability, necessitating further robustness enhancements.
Future Work
Future research will focus on integrating multimodal sensors such as LiDAR and radar to improve perception robustness. Developing safety-aware training objectives and incorporating online learning strategies will enhance reliability in diverse environments. Additionally, extending the framework to real-world datasets and scenarios will facilitate deployment, while further refining explanation mechanisms to support transparency and user trust.
AI Executive Summary
Autonomous driving has rapidly evolved, with vision-language models offering promising capabilities for high-level reasoning and interpretability. However, their deployment in real-world scenarios is hindered by high inference latency caused by large model sizes and complex reasoning modules. To address this, the present work introduces RT-VLA, a lightweight, knowledge-distilled model that effectively condenses the capabilities of the large SimLingo teacher into a compact form suitable for real-time operation.
RT-VLA employs a multi-level supervision strategy, distilling visual features, query representations, waypoint predictions, and language logits from the teacher. The architecture uses an efficient visual encoder (EVA-02) and lightweight language modules, with feature alignment modules ensuring cross-model compatibility. During training, a combination of multi-task loss functions guides the model to retain high driving accuracy and reasoning ability. An innovative offline language analysis component enables post-hoc explanations, providing safety-critical insights without adding runtime latency.
Experimental results on the Bench2Drive dataset demonstrate that RT-VLA achieves a 44.8× reduction in inference time in vision-only mode and 7.9× in vision+language mode, while maintaining a driving score close to the original teacher model. The model responds faster in dense urban scenarios, improving safety and responsiveness. Additionally, the offline explanation capability offers valuable interpretability, crucial for safety validation and user trust.
This research marks a significant step toward practical, scalable autonomous driving systems that combine high performance, efficiency, and transparency. Future directions include integrating multimodal sensors, enhancing safety robustness, and deploying in real-world environments to realize fully autonomous, explainable vehicles.
Deep Analysis
Background
近年来，端到端自主驾驶模型逐渐成为研究热点，代表性方法包括DriveCoT、TransFuser和SimLingo等。这些模型通过深度学习实现感知、推理与控制的统一，提升了系统的整体性能。然而，复杂的模型结构带来高延迟，限制了在实际道路中的应用。尤其是VLA模型，结合视觉与语言的推理能力，虽提升了决策的解释性，但推理速度成为瓶颈，亟需高效的模型压缩方案。
Core Problem
现有VLA模型在保证推理能力的同时，推理延迟过高，难以满足实时驾驶需求。复杂的模型架构导致推理时间长，影响反应速度，增加安全风险。如何在保持推理性能的基础上大幅降低延迟，是当前的核心难题。此外，模型的可解释性不足，也限制了其在安全关键场景的应用。
Innovation
本研究提出多层次知识蒸馏策略，将教师模型的视觉特征、路径点和语言 logits逐层迁移到轻量学生模型中，显著提升推理速度。引入离线后验解释机制，实现安全场景的事后分析，增强模型的可解释性。不同于传统单一输出蒸馏方法，本方案结合多模态特征的深层迁移，突破了模型压缩与解释的双重瓶颈，推动了自主驾驶模型的实用化。
Methodology
- �� 设计多层次蒸馏框架，将教师模型的视觉特征、查询表示、路径点和语言 logits逐层迁移到学生模型。• 采用EVA-02作为视觉编码器，简化模型结构，提升效率。• 构建对齐模块，将不同模型的特征和序列长度统一，为蒸馏提供基础。• 在训练中结合路径点、视觉特征和语言的多任务损失，确保模型在推理和解释方面的能力。• 引入离线语言微调，通过模拟推理场景，提升模型的后验解释能力。• 采用两阶段训练策略，先优化驾驶性能，再专注于语言解释，确保两者兼得。
Experiments
在Bench2Drive数据集上进行评估，模型在220条路线中测试，采用驾驶得分、推理时间和评论质量三项指标。训练使用SimLingo的训练数据，验证集占比5%。对比基线包括原始SimLingo和简化版本，重点考察推理速度与性能的折中。通过消融实验验证多层次蒸馏和微调的效果。模型在不同场景中表现出优异的响应速度和决策能力，验证其实用性。
Results
RT-VLA在推理速度上实现44.8倍提升（从1544ms降至34.48ms），在Vision+Language模式下提升7.9倍（从1544ms降至196ms），同时保持85.19的驾驶得分，几乎与教师模型持平。评论质量也接近原模型，验证了知识蒸馏在性能和效率上的平衡。实验证明，蒸馏和微调显著改善模型的鲁棒性和解释能力。
Applications
该模型适用于自动驾驶系统中的实时决策，尤其在复杂城市环境中，能快速响应交通变化。其低延迟特性保证了车辆的安全性和响应速度，适合部署在智能交通管理和自动驾驶车辆中。未来还可结合多模态传感器，提升环境感知能力，推动行业应用落地。
Limitations & Outlook
模型依赖模拟数据训练，可能在真实环境中表现不佳。未结合LiDAR等多模态信息，面对恶劣天气或复杂光照条件时鲁棒性不足。安全性方面，未引入鲁棒性优化措施，存在潜在风险。未来需加强安全保障和多模态融合，提升模型的实际适应性。
Plain Language Accessible to non-experts
想象你在厨房做饭，厨师（模型）需要准备各种食材（信息），并按照食谱（任务）做出美味的菜肴（决策）。传统厨师用很多时间逐一检查每个食材，等待所有准备工作完成后再开始烹饪，这样速度很慢。而RT-VLA就像一个聪明的助手，提前学习厨师的做菜技巧，把重要的步骤和食材信息压缩成小本子（模型参数），在厨房里快速反应，只在需要详细解释时才翻开本子，提供详细的菜谱说明。这样既保证了菜的质量，又大幅节省了时间，能在繁忙的厨房里快速出菜，确保每道菜都美味可口。
ELI14 Explained like you're 14
想象你在玩一个超级复杂的电子游戏，里面有很多任务和提示。以前的游戏AI需要花很多时间思考每个动作，反应很慢，玩起来不顺畅。现在，科学家们发明了一种聪明的助手（RT-VLA），它提前学会了很多技巧，把复杂的思考过程变得很快。它可以在你刚遇到问题时迅速做出反应，还能在你需要解释为什么这么做时，告诉你详细的理由。这个助手不仅快，还能解释自己的动作，帮助你更好理解游戏规则。这样，你在游戏中就能更顺利、更安全地完成任务，体验也更好。
Abstract
Vision-Language-Action (VLA) models have shown strong potential for end-to-end autonomous driving by jointly modeling visual perception, language reasoning, explainability and action prediction. However, their large vision-language backbones and reasoning modules introduce substantial inference latency and thereby prevent their deployment in the unforgiving reality of the road networks. We propose RT-VLA, a lightweight, distilled VLA model that transfers the driving and reasoning capabilities of the state-of-the-art SimLingo model into a compact student through multi-level supervised distillation. RT-VLA preserves language-based reasoning and supports post-hoc explanation through offline language analysis of safety-critical driving moments without adding latency to real-time control. Compared to the SimLingo teacher, RT-VLA maintains competitive closed-loop driving and language reasoning performance while reducing inference time by 44.8X in vision-only mode and 7.9X in vision+language mode. These results suggest that supervised distillation is a practical approach for building real-time, explainable VLA-style autonomous driving models.