WALA Learning Executable Latent Actions from Action-Labeled Demonstrations and Action-Free Videos
WALA learns executable latent actions from action-labeled demonstrations and action-free videos, achieving 75.2% success on RoboCasa.
Key Findings
Methodology
WALA pretrains a semantic-geometric latent action model from videos without action annotations to learn action-relevant representations. It predicts future deltas in DINOv3 feature space and dense depth space, preserving task-relevant semantic and geometric structures while avoiding raw pixel reconstruction.
Key Results
- WALA achieves a 75.2% average success rate on RoboCasa, outperforming the strongest baseline, DIAL, by 5 percentage points.
- In RoboTwin 2.0's random setting, WALA achieves a 92.8% success rate, leading other methods.
- Ablation studies confirm the contribution of semantic-geometric world supervision to performance gains.
Significance
WALA offers a novel approach to robot policy learning by combining action-labeled demonstrations and action-free videos. It addresses the challenge of relying on expensive action-labeled data and demonstrates strong generalization across various manipulation tasks.
Technical Contribution
WALA introduces semantic-geometric future prediction in latent action learning, providing a new training signal. By predicting in DINOv3 and dense depth spaces, it avoids pixel-level reconstruction, enhancing model efficiency and accuracy.
Novelty
WALA is the first to utilize action-free videos for robot policy learning by providing effective training interfaces through latent action target matching and future dynamics prediction.
Limitations
- WALA may be limited by video quality and depth estimation accuracy when handling complex scenes.
- The method may face challenges in real-time application in resource-constrained environments.
Future Work
Future research could explore applying WALA in more complex scenes and optimizing its performance in resource-limited environments.
AI Executive Summary
WALA is a novel framework that learns executable latent actions by combining action-labeled demonstrations and action-free videos. Traditional robot policy learning relies on expensive and hard-to-scale action-labeled data, whereas WALA provides effective training signals by predicting future changes in DINOv3 feature space and dense depth space.
In experiments, WALA achieved a 75.2% average success rate on RoboCasa, significantly outperforming existing state-of-the-art baselines. Ablation studies confirmed the contribution of semantic-geometric world supervision to performance gains. Additionally, WALA excelled in RoboTwin 2.0's random setting, achieving a 92.8% success rate.
While WALA demonstrates strong generalization across various manipulation tasks, it may be limited by video quality and depth estimation accuracy when handling complex scenes. Future research could explore applying WALA in more complex scenes and optimizing its performance in resource-limited environments.
Deep Analysis
Background
Robot policy learning typically relies on action-labeled demonstrations, which are expensive and hard to scale. Recent advances in vision-language-action models have shown significant progress in language-conditioned manipulation tasks but still face challenges in data collection and insufficient supervision of future scene evolution.
Core Problem
Existing methods mainly rely on action-labeled demonstrations, which are insufficient to cover the rich physical interactions in the real world. Additionally, standard action-supervised training objectives provide only weak supervision about the future consequences of actions, making it difficult to handle long-horizon, contact-rich, or spatially precise manipulation tasks.
Innovation
WALA offers a novel approach to robot policy learning by combining action-labeled demonstrations and action-free videos. Its core innovation lies in utilizing action-free videos for latent action learning and predicting future changes in DINOv3 feature space and dense depth space.
Methodology
- �� Pretrain a semantic-geometric latent action model from videos without action annotations to learn action-relevant representations.
- �� Predict future deltas in DINOv3 feature space and dense depth space, avoiding raw pixel reconstruction.
- �� During policy training, freeze the pretrained encoder to provide stable latent action targets.
Experiments
Experiments were conducted on RoboTwin 2.0 and RoboCasa-GR1-Tabletop, containing 50 and 24 manipulation tasks, respectively. Training and evaluation were performed on PPU, with average success rate reported as the primary metric.
Results
On RoboCasa, WALA achieved a 75.2% average success rate, outperforming the strongest baseline, DIAL, by 5 percentage points. In RoboTwin 2.0's random setting, WALA achieved a 92.8% success rate, leading other methods.
Applications
WALA can be applied to robotic manipulation tasks, especially in scenarios requiring generalization capabilities and low data dependency. Its method is suitable for various robotic platforms and task settings.
Limitations & Outlook
WALA may be limited by video quality and depth estimation accuracy when handling complex scenes. Additionally, the method may face challenges in real-time application in resource-constrained environments.
Plain Language Accessible to non-experts
Imagine a robot as a smart apprentice learning by watching workers operate machines. WALA is like this apprentice, learning not only from action-labeled demonstrations but also from unlabeled worker videos. This way, the robot can understand how to operate without explicit instructions. It's like a novice chef learning to cook by watching videos rather than relying solely on detailed recipes.
ELI14 Explained like you're 14
Imagine you're playing a game where your character is a robot. This robot needs to learn how to complete tasks without clear instructions. WALA is like a super helper in the game, learning how to operate by watching other players' videos, not just relying on a coach's guidance. It's like in Minecraft, where you learn to build complex structures by watching YouTube videos instead of just reading manuals.
Glossary
Latent Action
In machine learning, latent actions are abstract representations learned by models to simplify complex action prediction problems.
In WALA, latent actions are used to extract action-relevant dynamics from videos.
DINOv3 Feature Space
A deep learning model for image feature extraction that captures semantic information in images.
WALA uses DINOv3 feature space for future change prediction.
Dense Depth Space
A space representing depth information for each pixel in an image, capturing the geometric structure of a scene.
WALA predicts future geometric changes in dense depth space.
RoboCasa
A benchmark platform for robotic manipulation tasks with various complex tasks.
WALA achieved new state-of-the-art results on RoboCasa.
Action-Labeled Demonstration
Robot demonstration data with explicit action labels used for training machine learning models.
WALA combines action-labeled demonstrations with action-free videos for learning.
Open Questions Unanswered questions from this research
- 1 How to apply WALA in more complex scenes? Current methods may be limited by video quality.
- 2 How to optimize WALA's performance in resource-limited environments? Limited computational resources may affect real-time application.
Applications
Immediate Applications
Robotic Manipulation
WALA can enhance robot performance in various manipulation tasks, especially in scenarios requiring generalization capabilities.
Long-term Vision
Automated Production
WALA has the potential to achieve more efficient robot learning and operation in industrial automation, reducing reliance on expensive data.
Abstract
Generalizable robot policies typically rely on action-labeled robot demonstrations, which are expensive to collect and difficult to scale. In contrast, large-scale human and robot videos contain rich physical interactions but often lack executable robot action labels. We present WALA, a framework for learning executable latent actions from both action-labeled demonstrations and action-free videos. WALA first pretrains a semantic-geometric latent action model from videos by modeling the evolution between current observations and sparsely sampled future observations. Instead of reconstructing raw pixels, WALA predicts future deltas in the DINOv3 feature space and dense depth space, preserving task-relevant semantic and geometric structure while reducing sensitivity to appearance details. During policy training, the pretrained encoder provides stable latent action targets, and the decoder serves as a trainable latent world model. The latent actions generated by the vision-language backbone are jointly supervised by robot action prediction, latent action target matching, and future dynamics prediction. This enables action-labeled demonstrations to provide executable control supervision, while action-free videos contribute dynamics supervision without requiring robot action annotations. Experiments show that WALA achieves strong performance on RoboTwin, sets a new state-of-the-art result on RoboCasa with 75.2% average success, and improves both policy performance and generalization in real-world manipulation tasks.