From Pixels to Tokens: A Systematic Study of Latent Action Supervision for Vision-Language-Action Models
This paper systematically studies latent action supervision in VLA models, comparing four strategies with specific improvements: image-based and action-based formulations.
Key Findings
Methodology
The authors propose two complementary perspectives: regularizing trajectories via image-based latent actions, and unifying target spaces with action-based latent tokens. Four strategies—LA-Align, LA-Direct, LA-Cond, and LA-Tok—are instantiated within a unified VLA baseline using shared backbone and action head. Experiments on LIBERO, RoboTwin 2.0, and real-world JAKA robot validate their effectiveness. Results indicate image-based latent actions excel in long-horizon reasoning, while action tokens are superior for complex motor tasks. Discrete supervision outperforms continuous signals, and joint training enhances robustness across heterogeneous datasets.
Key Results
- On LIBERO-Long, image-based strategies improved performance by 8.4%-10.8%, demonstrating their advantage in planning over extended horizons. In RoboTwin 2.0, the action token approach increased success rate by 17.5%, especially in high-dimensional manipulation tasks. Discrete latent token supervision significantly outperformed continuous representations, confirming its effectiveness. Multi-task training with latent supervision consistently improved overall performance, indicating robustness in diverse scenarios.
- Different strategies show task-dependent strengths: image-based methods favor scene understanding and long-term planning, while action tokens excel at precise motor control. Real-world experiments on JAKA further validated these findings, with the proposed methods outperforming baselines in success scores. The results underscore the importance of choosing the right latent supervision based on task characteristics.
- This work is the first comprehensive comparison of latent action strategies in VLA, providing valuable insights for future multimodal robot learning. It demonstrates that integrating discrete latent tokens into VLA training significantly enhances performance, especially in complex, real-world tasks.
Significance
This research addresses the challenge of heterogeneous data in vision-language-action models, proposing latent actions as an effective intermediate representation to bridge semantic gaps across diverse datasets. The findings highlight that latent supervision improves long-term reasoning and motor coordination, crucial for autonomous robots operating in complex environments. By systematically comparing multiple strategies, the work offers a clear framework for selecting optimal approaches in different scenarios, advancing the field of multimodal robotic learning. The demonstrated robustness and scalability suggest broad applicability in real-world autonomous systems, paving the way for more adaptive and intelligent robots.
Technical Contribution
The paper introduces two main approaches: trajectory regularization via image-based latent actions and target space unification through action-based tokens. It designs four strategies—LA-Align, LA-Direct, LA-Cond, and LA-Tok—integrating VQ-VAE and Transformer modules to encode, decode, and supervise latent representations. The novel use of discrete tokens for action supervision, combined with a unified baseline, enables direct training of VLMs with these representations. Extensive experiments validate that discrete supervision yields superior performance, establishing a new standard for multimodal policy learning. The work also provides a comprehensive analysis of task-specific benefits and robustness under multi-task training.
Novelty
This is the first systematic comparison of image-based and action-based latent action strategies within a unified VLA framework. The introduction of four distinct integration strategies, especially the action-to-token mapping, represents a significant innovation. Unlike prior work that focused on single-form latent representations, this study evaluates their relative effectiveness across diverse tasks and datasets, providing a comprehensive understanding of their applicability. The explicit focus on discrete supervision and its benefits in complex, real-world scenarios marks a notable advancement in multimodal robotic learning.
Limitations
- The current approach relies heavily on large-scale annotated datasets and high computational resources, limiting scalability in resource-constrained settings. Its generalization to highly dynamic or unstructured environments remains uncertain, especially where visual or action spaces are extremely high-dimensional. The discretization process may cause information loss, impacting fine control tasks. Moreover, the models' performance in zero-shot or unseen environments needs further validation, and real-time deployment efficiency still poses challenges.
Future Work
Future directions include developing more efficient latent encoding schemes to reduce computational costs, exploring adaptive discretization techniques to preserve control fidelity, and extending the framework to multi-robot systems. Integrating reinforcement learning to optimize latent representations dynamically could further enhance adaptability. Additionally, investigating unsupervised or semi-supervised approaches for latent action learning may reduce data dependency, broadening applicability in real-world scenarios. Cross-modal transfer learning and zero-shot generalization are promising avenues to make these models more versatile and scalable.
AI Executive Summary
Vision-language-action (VLA) models have become a promising paradigm for autonomous robotic systems, capable of understanding complex instructions and executing diverse tasks. However, the heterogeneity of training data—ranging from robotic manipulation videos to human demonstrations—poses significant challenges to model generalization. Traditional supervised approaches often struggle with inconsistent action semantics across datasets, leading to performance degradation.
To address this, the authors propose leveraging latent actions as an intermediate representation that can unify heterogeneous data sources. They introduce two core perspectives: one uses image-based latent actions to regularize the trajectory, providing high-level visual plans; the other employs action-based latent tokens to unify the target space, simplifying semantic alignment. Building on these, four strategies are designed: LA-Align (implicit representation alignment), LA-Direct (explicit latent decoding), LA-Cond (conditional decoding), and LA-Tok (action-to-token mapping). All strategies are integrated into a shared VLA baseline based on Qwen-VL-2B, utilizing VQ-VAE and Transformer modules.
Extensive experiments on datasets like LIBERO and RoboTwin 2.0, along with real-world robot tests, demonstrate the effectiveness of these strategies. Results show that image-based latent actions significantly improve long-horizon planning, boosting performance by up to 10.8%. Conversely, the action token approach excels in complex motor tasks, increasing success rates by 17.5%. Discrete supervision consistently outperforms continuous signals, validating the hypothesis that structured, quantized representations facilitate learning.
This work advances the understanding of how latent actions can enhance multimodal robotic policies, especially in heterogeneous data environments. It provides a comprehensive framework for selecting and designing latent supervision strategies, with broad implications for future autonomous systems. Despite some limitations in scalability and real-time deployment, the findings open new avenues for robust, adaptable, and scalable robot learning, promising a future where robots can seamlessly operate across diverse tasks and settings.
Deep Dive
Abstract
Latent actions serve as an intermediate representation that enables consistent modeling of vision-language-action (VLA) models across heterogeneous datasets. However, approaches to supervising VLAs with latent actions are fragmented and lack a systematic comparison. This work structures the study of latent action supervision from two perspectives: (i) regularizing the trajectory via image-based latent actions, and (ii) unifying the target space with action-based latent actions. Under a unified VLA baseline, we instantiate and compare four representative integration strategies. Our results reveal a formulation-task correspondence: image-based latent actions benefit long-horizon reasoning and scene-level generalization, whereas action-based latent actions excel at complex motor coordination. Furthermore, we find that directly supervising the VLM with discrete latent action tokens yields the most effective performance. Finally, our experiments offer initial insights into the benefits of latent action supervision in mixed-data, suggesting a promising direction for VLA training. Code is available at https://github.com/RUCKBReasoning/From_Pixels_to_Tokens.