LaWAM: Latent World Action Models for Efficient Dynamics-Aware Robot Policies
LaWAM uses latent visual subgoals for efficient scene prediction, achieving 98.6% success with 187ms inference.
Key Findings
Methodology
LaWAM integrates pretrained vision foundation models with a latent action model (LaWM). It employs an inverse dynamics encoder to infer latent actions from visual transitions, and a decoder to predict future scene features, forming latent visual subgoals. This setup allows the policy to generate actions conditioned on predicted scene evolution without pixel-level video generation. Training involves latent action distillation and knowledge insulation to ensure stability across tasks. During inference, the policy predicts latent actions, which LaWM decodes into future scene features, guiding action generation efficiently. The approach reduces computational costs and latency compared to pixel-space WAMs.
Key Results
- On LIBERO, success rate reaches 98.6%, with inference time of 187ms, 24× faster than pixel-based WAMs, while parameters are reduced by 95%.
- In RoboTwin and real-world tasks, success rates exceed 90%, demonstrating strong generalization and robustness in dynamic environments.
- Latent visual subgoals improve scene prediction accuracy, enabling more precise and stable control, especially in deformable object manipulation like towel folding.
Significance
This work addresses the bottleneck of pixel-level future prediction by shifting to a compact latent space, enabling real-time, scalable robot control. It demonstrates that scene dynamics can be effectively modeled with minimal computational overhead, broadening the applicability of predictive control in complex, real-world scenarios. The approach bridges semantic instruction understanding with physically grounded control, paving the way for more autonomous and adaptable robots.
Technical Contribution
The paper introduces LaWM, a latent action-conditioned world model that repurposes the decoder of a latent action model as a scene predictor. It combines inverse dynamics inference with a decoder to generate future scene features, forming latent visual subgoals. The method employs latent action distillation and knowledge insulation to improve stability and transferability. The entire pipeline operates with significantly fewer parameters and lower latency, offering a scalable alternative to pixel-space WAMs. This innovation advances the state-of-the-art in dynamic scene prediction for robotic control.
Novelty
This is the first work to leverage the decoder of a latent action model directly as a scene predictor, avoiding pixel-level generation. The use of latent visual subgoals as explicit scene evolution cues for action generation is novel, providing a compact, efficient, and effective alternative to traditional pixel-space models. This approach achieves high success rates with minimal latency and parameter overhead, marking a significant step forward in predictive robotics.
Limitations
- The model struggles with scenes involving rapid camera motion or large viewpoint shifts, limiting its deployment in mobile or self-moving robots.
- Fine-grained deformable object dynamics, such as cloth deformation, are underrepresented in training data, affecting modeling accuracy.
- Robustness in highly dynamic or unpredictable environments remains a challenge, requiring further data and multi-modal integration to improve performance.
Future Work
Future efforts will focus on integrating multi-modal sensory data, such as tactile and proprioceptive inputs, to enhance scene understanding. Expanding training datasets to include more complex physical interactions will improve robustness. Additionally, exploring explainability and interpretability of latent representations will facilitate deployment in safety-critical applications. Extending LaWAM to mobile and humanoid robots is also a promising direction.
AI Executive Summary
Robotic manipulation and autonomous control demand accurate scene prediction and dynamic understanding. Traditional models rely heavily on pixel-level video generation, which is computationally expensive and introduces high latency, limiting real-time application. To overcome this, LaWAM introduces a novel paradigm: leveraging a latent space of visual features to predict scene evolution efficiently. By integrating a pretrained vision foundation model with a latent action model, LaWAM infers latent actions from visual transitions and decodes them into future scene features, forming latent visual subgoals. This approach bypasses pixel-level generation, drastically reducing inference time to 187ms—over 24 times faster than pixel-space counterparts—while maintaining high success rates across diverse tasks.
The core innovation lies in repurposing the decoder of a latent action model as a scene predictor, enabling the policy to condition actions on predicted scene changes without heavy computational overhead. Extensive experiments on benchmarks such as LIBERO and RoboTwin demonstrate that LaWAM achieves success rates exceeding 98%, outperforming existing methods in both efficiency and accuracy. In real-world experiments, LaWAM effectively handles manipulation tasks like towel folding, pick-and-place, and drawer opening, with success rates above 90%. Its ability to generalize across different embodiments and environments highlights its robustness and scalability.
This work significantly advances the field of predictive robot control by demonstrating that scene dynamics can be captured in a compact latent space, facilitating real-time, scalable, and physically grounded decision-making. The reduced parameter count and inference latency open new possibilities for deploying sophisticated control policies on resource-constrained robots. Looking ahead, integrating multi-modal sensory inputs and expanding training data will further enhance LaWAM’s robustness, paving the way for more autonomous, adaptable robots in complex real-world scenarios.
Deep Dive
Abstract
Vision-Language-Action models (VLAs) leverage large-scale vision-language pretraining for semantic robot control, but often lack explicit foresight into how robot actions change the scene. World-Action Models (WAMs) address this limitation by conditioning policies on predicted futures, yet existing approaches typically rely on computationally expensive video generation with substantial pixel-level redundancy. We present LaWAM, a Latent World Action Model that exposes predictive dynamics to robot policies through compact latent visual subgoals instead of reconstructed future video. At the core of LaWAM is a latent-action-conditioned Latent World Model (LaWM). We obtain LaWM by training a latent action model in the latent space of a pretrained vision foundation model and repurposing its forward decoder to predict future observation features for scene evolution. LaWAM then conditions action generation on these predicted latent visual subgoals to enable dynamics-aware robot control. LaWAM achieves state-of-the-art or competitive success rates (SRs) across LIBERO (98.6% SR), RoboTwin (91.22% SR), and real-world manipulation tasks while retaining low-latency inference. LaWAM runs in 187 ms per action-chunk prediction and achieves up to 24x lower wall-clock latency than pixel-space WAMs.