ETA-VLA: Efficient Token Adaptation via Temporal Fusion and Intra-LLM Sparsification for Vision-Language-Action Models
ETA-VLA reduces computation by 32% via temporal fusion and sparsification, retaining 94% accuracy.
Key Findings
Methodology
ETA-VLA introduces a Temporal Fusion Module (TFM) and an Intra-LLM Sparse Aggregator (ILSA) to enhance the efficiency of vision-language-action models. TFM operates on visual encoder features, adaptively fusing multi-frame historical features via transformers and learnable weights, compressing the temporal dimension while preserving critical motion cues. ILSA performs fine-grained token sparsification inside the LLM, using a RoPE-free semantic scoring mechanism to ensure visual tokens are scored purely based on their semantic relevance to the driving instruction.
Key Results
- ETA-VLA achieves driving performance comparable to state-of-the-art baselines on NAVSIM v2 while reducing computational FLOPs by approximately 32%.
- The method prunes 85% of visual tokens, reducing inference FLOPs by 61%, yet retains 94% of the original accuracy on the NAVSIM v2 benchmark.
- On the Navtest benchmark, ETA-VLA achieves an EPDMS score of 85.0, surpassing many existing methods.
Significance
ETA-VLA represents a significant advancement in vision-language-action models by reducing computational complexity and enhancing efficiency in the autonomous driving domain. It addresses the computational bottlenecks in processing historical multi-view frames, making it feasible to run on embedded hardware. The success of this method demonstrates the potential of sparsification and temporal fusion to optimize model performance.
Technical Contribution
ETA-VLA achieves significant computational efficiency improvements in vision-language-action models through temporal fusion and sparsification strategies. Unlike existing methods, it performs layer-adaptive sparse processing inside the LLM and introduces a RoPE-free semantic scoring mechanism, ensuring the semantic relevance of visual token selection.
Novelty
ETA-VLA is the first to combine temporal fusion and intra-LLM sparsification strategies in vision-language-action models. Unlike previous methods, it performs sparsification inside the LLM, significantly reducing computational complexity while retaining the model's semantic understanding capabilities.
Limitations
- ETA-VLA may experience performance degradation in extremely complex scenarios as the sparsification strategy might lose important visual information.
- The method may require further optimization for real-time applications to meet high-frequency update demands.
Future Work
Future research could explore optimization of sparsification strategies in more complex scenarios and adaptability on different hardware platforms. Additionally, multi-modal fusion with other sensing modalities like radar or LiDAR is a promising direction.
AI Executive Summary
In autonomous driving systems, integrating vision-language-action (VLA) models offers a unified framework for interpreting complex scenes and executing control commands. However, accurate temporal reasoning requires incorporating historical multi-view frames, posing significant computational challenges. ETA-VLA introduces a Temporal Fusion Module (TFM) and an Intra-LLM Sparse Aggregator (ILSA) to effectively reduce computational complexity. TFM operates on visual encoder features, adaptively fusing multi-frame historical features via transformers and learnable weights, compressing the temporal dimension while preserving critical motion cues. ILSA performs fine-grained token sparsification inside the LLM, using a RoPE-free semantic scoring mechanism to ensure visual tokens are scored purely based on their semantic relevance to the driving instruction. Experimental results show that ETA-VLA achieves driving performance comparable to state-of-the-art baselines on NAVSIM v2 while reducing computational FLOPs by approximately 32%. The method prunes 85% of visual tokens, reducing inference FLOPs by 61%, yet retains 94% of the original accuracy on the NAVSIM v2 benchmark. Future research could explore optimization of sparsification strategies in more complex scenarios and adaptability on different hardware platforms.
Deep Analysis
Background
In the field of autonomous driving, vision-language-action (VLA) models have gained attention for their ability to map high-dimensional visual inputs and linguistic instructions into executable driving actions. Traditional VLA architectures typically flatten multi-view historical frames and concatenate them with text prompts, leading to a quadratic explosion in FLOPs as the number of frames increases. This computational complexity makes it infeasible to run on vehicle-embedded hardware.
Core Problem
VLA models face a trade-off between efficiency and accuracy when processing high-dimensional spatiotemporal data. Driving is inherently dynamic; accurate trajectory prediction requires understanding motion history, often necessitating processing past frames. Standard VLA architectures experience a quadratic increase in computational complexity with the number of frames and views, making real-time application challenging.
Innovation
ETA-VLA introduces a Temporal Fusion Module (TFM) and an Intra-LLM Sparse Aggregator (ILSA) to enhance efficiency. TFM operates on visual encoder features, adaptively fusing multi-frame historical features via transformers and learnable weights, compressing the temporal dimension. ILSA performs fine-grained token sparsification inside the LLM, using a RoPE-free semantic scoring mechanism to ensure visual tokens are scored purely based on their semantic relevance to the driving instruction.
Methodology
- �� Temporal Fusion Module (TFM): Operates on visual encoder features, adaptively fusing multi-frame historical features via transformers and learnable weights.
- �� Intra-LLM Sparse Aggregator (ILSA): Performs fine-grained token sparsification inside the LLM, using a RoPE-free semantic scoring mechanism.
- �� Semantic Scoring: Ensures visual tokens are scored purely based on their semantic relevance to the driving instruction.
Experiments
Experiments were conducted on the NAVSIM v2 benchmark using LLaVA-v1.5-7B and Vicuna-7B LLM. The model was pre-trained on the trainval split for 1 epoch and fine-tuned on navtrain for 1 epoch. The performance of ETA-VLA was evaluated on the Navtest and Navhard benchmarks, with the primary metric being EPDMS.
Results
ETA-VLA achieves driving performance comparable to state-of-the-art baselines on NAVSIM v2 while reducing computational FLOPs by approximately 32%. The method prunes 85% of visual tokens, reducing inference FLOPs by 61%, yet retains 94% of the original accuracy on the NAVSIM v2 benchmark.
Applications
ETA-VLA can be directly applied in autonomous driving systems, particularly in scenarios requiring efficient processing of multi-view historical frames. By reducing computational complexity, this method makes it feasible to run on embedded hardware.
Limitations & Outlook
ETA-VLA may experience performance degradation in extremely complex scenarios as the sparsification strategy might lose important visual information. Additionally, the method may require further optimization for real-time applications to meet high-frequency update demands.
Plain Language Accessible to non-experts
Imagine you're driving a car, and your eyes and brain need to process visual information from multiple directions simultaneously. ETA-VLA acts like a smart assistant, helping you filter out unimportant information and retain only the parts crucial for driving decisions. Just as you focus on the road ahead while driving, ignoring roadside billboards, ETA-VLA uses sparsification strategies to reduce the amount of information that needs processing, thereby improving efficiency.
ELI14 Explained like you're 14
Imagine you're playing a racing game, and you need to pay attention to the track, opponents, and speedometer all at once. ETA-VLA is like a super assistant in the game, helping you automatically filter out unimportant details and keep only the information crucial for winning the race. This way, you can react faster and win the race! It's like having a study guide in school that highlights the key points, making your study time more efficient.
Glossary
ETA-VLA (Efficient Token Adaptation)
A framework that enhances the efficiency of vision-language-action models through temporal fusion and sparsification.
Used to reduce computational complexity and improve efficiency in autonomous driving models.
TFM (Temporal Fusion Module)
Operates on visual encoder features, adaptively fusing multi-frame historical features via transformers and learnable weights.
Used to compress the temporal dimension and preserve critical motion cues.
ILSA (Intra-LLM Sparse Aggregator)
Performs fine-grained token sparsification inside the LLM, using a RoPE-free semantic scoring mechanism.
Ensures visual tokens are scored purely based on their semantic relevance to the driving instruction.
RoPE (Rotary Position Embeddings)
A method for encoding relative positions in embeddings.
Replaced in ETA-VLA to eliminate distance bias.
EPDMS (Extended Predictive Driver Model Score)
A comprehensive score assessing the safety, efficiency, and comfort of autonomous driving systems.
Used to validate the driving performance of ETA-VLA.
Open Questions Unanswered questions from this research
- 1 How can sparsification strategies be optimized in more complex scenarios to ensure no loss of critical visual information?
- 2 How can ETA-VLA's adaptability and real-time performance be further enhanced on different hardware platforms?
Applications
Immediate Applications
Autonomous Driving Systems
ETA-VLA can be directly applied in autonomous driving systems, particularly in scenarios requiring efficient processing of multi-view historical frames.
Long-term Vision
Multi-modal Fusion
Future work could involve combining other sensing modalities like radar or LiDAR for robust autonomous driving systems.
Abstract
The integration of Vision-Language-Action (VLA) models into autonomous driving systems offers a unified framework for interpreting complex scenes and executing control commands. However, the necessity to incorporate historical multi-view frames for accurate temporal reasoning imposes a severe computational burden, primarily driven by the quadratic complexity of self-attention mechanisms in Large Language Models (LLMs). To alleviate this bottleneck, we propose ETA-VLA, an Efficient Token Adaptation framework for VLA models. ETA-VLA processes the past $n$ frames of multi-view images and introduces a novel Intra-LLM Sparse Aggregator (ILSA). Drawing inspiration from human driver attention allocation, ILSA dynamically identifies and prunes redundant visual tokens guided by textual queries and temporal consistency. Specifically, we utilize a text-guided scoring mechanism alongside a diversity-preserving sparsification strategy to select a sparse subset of critical tokens, ensuring comprehensive awareness of the driving scene. Extensive experiments on the NAVSIM v2 demonstrate that ETA-VLA achieves driving performance comparable to state-of-the-art baselines while reducing computational FLOPs by approximately 32\%. Notably, our method prunes 85% of visual tokens and reduces inference FLOPs by 61\%, but still retaining 94% of the original accuracy on the NAVSIM v2 benchmark.