Spec-VLA: Speculative Decoding for Vision-Language-Action Models with Relaxed Acceptance

Recent progress in VLA models, exemplified by OpenVLA and RT-2, has enabled robots to interpret multimodal inputs and generate actions. These models leverage large-scale pretraining but face significant computational challenges due to their size and autoregressive decoding. Traditional acceleration methods—architecture redesign, task-specific tuning, quantization, pruning—often require retraining or complex modifications, limiting flexibility. The need for efficient, scalable inference solutions remains urgent, especially for real-time applications in robotics and human-AI interaction. Inspired by large language models, speculative decoding offers a promising avenue, but its application to multimodal models is unexplored, primarily due to the complexity of action prediction and modal integration.

The core challenge lies in balancing decoding speed and prediction accuracy in large VLA models. Autoregressive decoding, while accurate, is slow, impeding real-time control. Existing acceleration methods either demand retraining or compromise performance. Direct application of speculative decoding is hindered by the intricate multimodal understanding and the greedy decoding strategy, which demands exact token matches. Consequently, achieving significant speedup without sacrificing success rate remains an open problem, especially in complex, multi-step action sequences.

The paper introduces several innovations:

1) Relative distance-based relaxation: allowing tokens within a certain distance to be accepted, extending acceptance length.

2) Tree-structured multi-path prediction: enabling parallel hypothesis verification.

3) No retraining needed: compatibility with existing models, low-cost deployment.

These innovations collectively enable longer, faster, and more robust decoding, addressing the bottleneck in current VLA inference pipelines.

• �� Build a draft generator based on Llama, integrating visual and textual features for multi-path prediction.
• �� Calculate relative distances between candidate tokens using bin ID differences, set a relaxation threshold r.
• �� During decoding, accept candidate tokens if their distance D(ai, ˆai) ≤ r, broadening acceptance.
• �� Organize top-K predictions into a tree structure for parallel verification.
• �� Use a verification model to confirm or reject tokens, dynamically adjusting acceptance based on the threshold.
• �� Conduct multi-task training with data augmentation to improve robustness.
• �� Fine-tune thresholds to balance speed and success rate, validated through extensive experiments.

The framework was tested on the LIBERO benchmark, covering four task types: Goal, Object, Spatial, and Long sequences. Each task involved 10 scenarios with 500 demonstrations. The models used a fine-tuned verification model and a draft generator trained on dataset-generated data. Experiments compared traditional autoregressive decoding with the speculative approach, measuring success rate, acceptance length, and speedup. Multiple thresholds were tested, with 50 trials per task, using 4 Tesla A100 GPUs. Ablation studies analyzed the impact of relaxation thresholds, acceptance lengths, and multi-path decoding, confirming the effectiveness of the proposed strategies.

Results show that relaxing acceptance thresholds increases average acceptance length from 2.10 to 2.94 tokens, boosting decoding speed by 42%. Success rates remained above 74%, demonstrating robustness. The acceptance length increased by 26%-44% across tasks, with minimal impact on accuracy. The method effectively balances speed and fidelity, especially in longer sequences, and generalizes well across diverse scenarios. Ablation studies confirmed the importance of threshold tuning, with optimal thresholds providing the best trade-off.

This acceleration technique is directly applicable to real-time robot control, autonomous systems, and multimodal AI interfaces, where rapid decision-making is crucial. It can be integrated into existing VLA models without retraining, making it cost-effective. Long-term, the approach could enable scalable, low-latency multimodal inference for complex tasks like autonomous driving, industrial automation, and human-robot collaboration, especially when combined with hardware acceleration and adaptive thresholding.

The method's effectiveness depends on appropriate threshold tuning; overly relaxed thresholds may reduce accuracy in noisy or complex environments. Its robustness in highly dynamic or unseen scenarios needs further validation. The current experiments are limited to LIBERO; broader testing across different datasets and modalities is necessary. Future work should focus on adaptive threshold mechanisms and hardware integration to address these limitations.