Latent Action as Intention Enables Efficient Future Imagination for World Action Models
LAWA encodes future intentions as compact latent actions, reducing inference latency by 42.9%, achieving 80.8% success on RoboCasa with superior generalization.
Key Findings
Methodology
The paper introduces LAWA, a novel World Action Model (WAM) architecture that replaces explicit future observation generation with a discrete latent action sequence representing future intentions. It employs a pre-trained action-free video encoder to produce manipulation-centric codebook targets, which are jointly denoised with continuous latent states and executable action chunks via multi-model joint attention. During training, a mask-supervised latent action tokenizer compresses visual transitions into discrete tokens biased towards interaction regions. At inference, the model discards the future-video branch, performing only latent action and action chunk denoising, enabling efficient future imagination. Extensive experiments on RoboCasa demonstrate state-of-the-art success rates of 65.6% in few-shot and 80.8% in full data settings, outperforming Fast-WAM by 9.6 and 4.5 points respectively, while reducing inference latency by 42.9%. The model also exhibits strong zero-shot robustness on LIBERO-Plus and real-world tasks.
Key Results
- On RoboCasa, LAWA achieves success rates of 65.6% (few-shot) and 80.8% (full data), surpassing Fast-WAM (56.0%, 76.3%) by 9.6 and 4.5 points, respectively, and matching Joint-WAM performance (64.1%, 78.8%) while reducing latency by 42.9%.
- In zero-shot transfer on LIBERO-Plus, LAWA reaches 74.4%, outperforming Fast-WAM (60.0%) and matching Joint-WAM (70.4%), especially under viewpoint and sensor noise perturbations.
- Ablation studies confirm that the structured latent action sequence and egocentric pre-training significantly enhance generalization, with performance improving as training data scales, demonstrating scalability.
Significance
This work addresses the bottleneck of high inference latency in traditional WAMs by proposing a compact latent action representation of future intentions. It effectively balances performance, generalization, and efficiency, enabling robots to perform complex tasks faster and more robustly. The approach opens new avenues for scalable, real-time robotic decision-making, with broad implications for autonomous systems in dynamic environments.
Technical Contribution
The key technical innovation lies in replacing explicit future observation generation with a structured, discrete latent action sequence that encodes future intentions. The model integrates a mask-supervised latent action tokenizer trained on action-free videos, combined with multi-model joint attention to control information flow. This design reduces computational costs, enhances robustness, and maintains high task success rates, setting a new standard for efficient future-oriented robot control.
Novelty
This is the first work to encode future intentions solely as a sequence of compact latent actions, avoiding the expensive process of visual future prediction. It combines discrete tokenization with mask supervision and pretraining on egocentric videos, establishing a new paradigm that outperforms existing methods like Fast-WAM and Joint-WAM in both efficiency and generalization, marking a significant step forward in robot world modeling.
Limitations
- The current approach may struggle in highly dynamic or unpredictable environments where latent action sequences cannot sufficiently capture complex future states, limiting robustness.
- Dependence on pretraining data diversity means that biases or gaps in training videos could impair generalization to unseen scenarios.
- Real-time performance still requires further optimization, especially in high-speed, long-horizon tasks, necessitating hardware acceleration or model compression.
Future Work
Future research will focus on enriching latent action representations to handle more complex, long-term planning, integrating reinforcement learning for adaptive future intention generation, and expanding multi-modal perception. Additionally, optimizing inference speed through hardware-aware design and extending the framework to multi-agent scenarios are promising directions.
AI Executive Summary
Robotic manipulation increasingly demands predictive capabilities to plan and execute complex tasks efficiently. Traditional world action models (WAMs) leverage visual prediction to model environment dynamics, but this process incurs significant latency, hindering real-time control. Fast-WAM attempted to address this by removing future observation generation during inference, yet it suffered from reduced generalization, especially with limited training data.
This paper introduces LAWA, a novel architecture that redefines future imagination by encoding future intentions as a sequence of compact latent actions. Instead of generating full future observations, LAWA predicts a structured, discrete sequence representing the robot's future goals. This approach employs a pre-trained action-free video encoder to produce manipulation-centric codebook targets, which are jointly denoised with continuous latent states and executable action chunks through a multi-model joint attention mechanism. During inference, the model discards the future-video branch, focusing solely on denoising latent intentions and actions, resulting in a significant reduction in latency.
Extensive experiments on the RoboCasa dataset demonstrate that LAWA achieves a success rate of 65.6% in few-shot and 80.8% in full-data settings, outperforming Fast-WAM by 9.6 and 4.5 points respectively, while reducing inference latency by 42.9%. The model also exhibits strong zero-shot robustness on LIBERO-Plus, with a success rate of 74.4%, surpassing existing models under various perturbations. Ablation studies confirm that structured latent actions and egocentric pretraining are key to its superior generalization and scalability.
Overall, LAWA offers a compelling solution to the latency-performance trade-off in robot world modeling. By representing future intentions in a compact, manipulable latent space, it enhances the robot's ability to plan and adapt swiftly in complex environments. This work paves the way for more efficient, robust, and scalable autonomous systems, with broad implications for industrial automation, service robots, and beyond. Despite current limitations in highly dynamic scenarios, ongoing research aims to extend the model's capacity for long-term planning and multi-modal perception, promising a transformative impact on robotic intelligence.
Deep Dive
Abstract
World action models (WAMs) improve robot control by modeling how observations evolve, but generating future observations at test time incurs substantial latency. Fast-WAM removes this process for efficiency; however, our matched implementations show lower generalization for Fast-WAM than for future-aware alternatives, especially with scarce robot demonstrations and in out-of-distribution scenarios. To bridge this gap, we introduce **LAWA**, a WAM architecture that uses compact latent actions as an operational representation of future intentions, enabling efficient test-time future imagination without generating future observations. Specifically, a discrete tokenizer enhanced by action-free pre-training produces manipulation-centric codebook targets. LAWA jointly denoises a continuous latent state anchored to these targets with executable action chunks while omitting the future-video branch at inference. On RoboCasa, LAWA achieves state-of-the-art average success rates of 65.6% and 80.8% in the few-shot and full data settings, improving over the matched Fast-WAM baseline by 9.6 and 4.5 points, respectively. It also preserves the performance level of the matched Joint-WAM variant while requiring 42.9% lower inference latency. LAWA also demonstrates competitive zero-shot robustness on LIBERO-Plus and superior performance on real-world tasks. These results show that future imagination need not be discarded: retaining it with compact latent actions yields an effective trade-off among performance, generalization, and latency. Code and models will be released.