Unifying Language-Action Understanding and Generation for Autonomous Driving
LinkVLA improves autonomous driving instruction accuracy to 91.01 by unifying language and action tokens.
Key Findings
Methodology
LinkVLA unifies language and action tokens into a shared discrete codebook, enhancing cross-modal consistency. It introduces an auxiliary action understanding objective to train the model to generate descriptive captions from trajectories, fostering bidirectional language-action mapping. The two-step coarse-to-fine generation method C2F improves decoding efficiency.
Key Results
- LinkVLA achieved a driving score of 91.01 in Bench2Drive tests, a 6.98% improvement over SimLingo. The success rate reached 74.55%, a 10.82% increase.
- In instruction-following evaluations, LinkVLA demonstrated significant performance improvements, particularly in complex scenarios.
- Ablation studies confirmed the effectiveness of LinkVLA's structural design and generation strategy, especially in reducing inference latency.
Significance
LinkVLA addresses the misalignment between language instructions and action outputs, significantly enhancing the efficiency of autonomous driving systems. This research offers a novel solution for cross-modal consistency, advancing the field of autonomous driving.
Technical Contribution
LinkVLA significantly improves instruction-following capability and efficiency in autonomous driving systems through a unified tokenization framework and bidirectional language-action mapping. Its innovative coarse-to-fine generation strategy reduces inference latency, offering new engineering possibilities.
Novelty
LinkVLA is the first to unify language and action tokens into a shared codebook and enhance semantic consistency through a bidirectional learning objective. This approach is unprecedented in the autonomous driving domain, providing a fresh perspective.
Limitations
- Performance may degrade under extreme weather conditions due to compromised visual input quality.
- The system may require higher computational resources in complex urban environments.
- Understanding of language instructions still depends on the quality of training data.
Future Work
Future research could explore LinkVLA's adaptability in various environments and optimize its performance on resource-constrained devices. Additionally, further studies could enhance the robustness of language instruction understanding.
AI Executive Summary
LinkVLA is a novel vision-language-action model designed to address the misalignment between language instructions and action outputs in autonomous driving. Existing methods often perform poorly in complex scenarios and have inefficient action generation. LinkVLA unifies language and action tokens into a shared discrete codebook, enhancing cross-modal consistency. Its innovative two-step coarse-to-fine generation method significantly improves action sequence decoding efficiency, saving 86% inference time.
Experimental results show that LinkVLA achieved a driving score of 91.01 in Bench2Drive tests, with a success rate of 74.55%. In instruction-following evaluations, LinkVLA demonstrated significant performance improvements, particularly in complex scenarios. This research offers a novel solution for cross-modal consistency, advancing the field of autonomous driving.
Despite LinkVLA's outstanding performance in several aspects, its performance under extreme weather conditions needs further validation. Additionally, the system may require higher computational resources in complex urban environments. Future research could explore LinkVLA's adaptability in various environments and optimize its performance on resource-constrained devices.
Deep Dive
Abstract
Vision-Language-Action (VLA) models are emerging as a promising paradigm for end-to-end autonomous driving, valued for their potential to leverage world knowledge and reason about complex driving scenes. However, existing methods suffer from two critical limitations: a persistent misalignment between language instructions and action outputs, and the inherent inefficiency of typical auto-regressive action generation. In this paper, we introduce LinkVLA, a novel architecture that directly addresses these challenges to enhance both alignment and efficiency. First, we establish a structural link by unifying language and action tokens into a shared discrete codebook, processed within a single multi-modal model. This structurally enforces cross-modal consistency from the ground up. Second, to create a deep semantic link, we introduce an auxiliary action understanding objective that trains the model to generate descriptive captions from trajectories, fostering a bidirectional language-action mapping. Finally, we replace the slow, step-by-step generation with a two-step coarse-to-fine generation method C2F that efficiently decodes the action sequence, saving 86% inference time. Experiments on closed-loop driving benchmarks show consistent gains in instruction following accuracy and driving performance, alongside reduced inference latency.