SpatialVAM:Spatial-Aware Multi-View Video Diffusion as a Data-Efficient Robot Policy
SpatialVAM achieves data-efficient robot policy learning via 3D video diffusion, improving Meta-World success rates by 22%.
Key Findings
Methodology
SpatialVAM transforms point clouds into multi-view RGB and heatmap videos, leveraging a video diffusion model to predict future frames and actions. Key components include a multi-view video diffusion transformer and a lightweight action decoder.
Key Results
- On Meta-World tasks, SpatialVAM achieved 89.1% success with only 10 demonstrations, outperforming baselines by 22%.
- On RoboCasa kitchen tasks, SpatialVAM surpassed 3D Diffuser Actor with a 15% success rate improvement.
- In real-world tasks, SpatialVAM achieved an average success rate of 57.1% across seven tasks, significantly outperforming other methods.
Significance
This research reduces data dependency in robotic manipulation while enhancing generalization and robustness through multi-view and 3D structural modeling. It opens new possibilities for multi-task manipulation and real-world deployment.
Technical Contribution
SpatialVAM is the first framework to use video foundation models for 3D world-action modeling, introducing a diffusion-based framework for joint multi-view heatmap and RGB video prediction.
Novelty
This is the first framework combining video diffusion with multi-view 3D modeling, distinct from traditional 2D video prediction or point-cloud-based methods.
Limitations
- Relies heavily on camera calibration and multi-view setups, which may limit deployment.
- Performance in highly dynamic scenes may require more training data.
- Does not explore adaptive view selection, leaving potential improvements untapped.
Future Work
Future directions include adaptive view selection, extending to dynamic scenes, and integrating multi-modal inputs like language instructions.
AI Executive Summary
Robotic manipulation requires understanding both 3D spatial structures and temporal dynamics, yet existing methods often neglect one or both aspects, leading to high data demands and limited generalization. SpatialVAM addresses this by introducing a multi-view video diffusion model that naturally integrates 3D information into video foundation models while aligning representation formats between video pretraining and action finetuning.
Experiments on Meta-World, RoboCasa, and real-world platforms demonstrate SpatialVAM's effectiveness, achieving 22%, 15%, and 16% improvements in success rates, respectively. The method learns complex long-horizon tasks with just 10 demonstrations and generalizes well to out-of-distribution scenarios.
However, SpatialVAM's reliance on multi-view setups may limit practical deployment. Future work could explore adaptive view selection and extensions to dynamic environments. Overall, this research sets a new benchmark for data-efficient multi-task robotic manipulation.
Deep Analysis
Background
Recent advances in Vision-Language-Action (VLA) and Video Action Models (VAM) have driven progress in robotic manipulation. However, these methods often rely on 2D visual observations or static image-text pretraining, lacking unified modeling of 3D spatial structures and temporal dynamics.
Core Problem
Existing methods face a representation gap between perception and control: observations are 2D videos, while actions occur in 3D physical space. This gap increases data requirements and limits generalization.
Innovation
SpatialVAM introduces the first 3D video action model, leveraging multi-view projections to encode 3D structures and a video diffusion model to jointly predict future RGB and heatmap videos, bridging the perception-control gap.
Methodology
- �� Multi-view projection: Converts point clouds into multi-view RGB images and heatmaps.
- �� Video diffusion model: Extends Wan2.2 with a multi-view video diffusion transformer for future video prediction.
- �� Action decoding: Back-projects heatmaps to recover 3D trajectories and predicts end-effector rotation and gripper states.
Experiments
Experiments span Meta-World, RoboCasa, and real-world platforms, evaluating data efficiency, generalization, and robustness. Baselines include UniPi, DreamZero, and 3D Diffuser Actor.
Results
SpatialVAM achieved 89.1% success on Meta-World, a 22% improvement; 15% higher success on RoboCasa; and 57.1% average success across real-world tasks.
Applications
SpatialVAM is suited for high-precision, data-efficient robotic tasks like industrial automation and household service robots.
Limitations & Outlook
Relies on multi-view setups, limiting deployment; performance in dynamic scenes needs further validation; adaptive view selection remains unexplored.
Plain Language Accessible to non-experts
Imagine you're cooking in a kitchen. You need to see the stove from the front to check the flame and from the side to monitor the pot. SpatialVAM acts like a smart kitchen assistant, observing from multiple angles and predicting the next steps, like stirring or adding water. Its uniqueness lies in learning complex tasks quickly from just a few demonstrations.
ELI14 Explained like you're 14
Think of a robot game where the robot has to pick up a toy and place it on a shelf. Regular robots see from one angle, like looking through a keyhole. SpatialVAM is like a robot with super vision—it sees from multiple angles and predicts the next move. Even cooler, it learns by watching you just a few times! Awesome, right?
Glossary
Video Diffusion Model
A model that generates videos by iteratively denoising, used for future frame prediction.
Used to jointly predict future RGB and heatmap videos.
Multi-View Projection
Technique to project 3D point clouds into 2D images from different angles.
Encodes 3D spatial structures.
Heatmap
A 2D image representing the probability distribution of a target location.
Represents end-effector target positions.
Video Foundation Model
A pretrained large-scale video model for feature extraction.
SpatialVAM extends Wan2.2.
Back-Projection
Technique to recover 3D positions from 2D heatmaps.
Used to generate 3D end-effector trajectories.
Open Questions Unanswered questions from this research
- 1 How to maintain efficiency and robustness in dynamic scenes?
- 2 Can adaptive view selection further improve performance?
- 3 How to integrate language instructions for multi-modal extensions?
Applications
Immediate Applications
Industrial Automation
Enables efficient multi-task manipulation in production lines, such as assembly and sorting.
Household Robots
Assists robots in performing complex tasks in home environments, like cleaning and object handling.
Long-term Vision
General-Purpose Robotics
Develops robots capable of autonomously learning and adapting to diverse tasks.
Abstract
Robotic manipulation requires understanding both the 3D spatial structure of the environment and its temporal evolution, yet most existing policies neglect one or both aspects. They often rely on 2D visual observations or backbones pretrained on static image--text pairs, which leads to high data requirements and limited comprehension of environment dynamics. To address this, we introduce SpatialVAM, the first 3D Video Action Model that simultaneously predict spatial-aware multi-view heatmap videos and RGB videos. Our key insight is that this design naturally injects 3D information into video foundation models while aligning the representation format between video pretraining and action finetuning. Extensive experiments demonstrate that SpatialVAM enables data-efficient, robust, generalizable, and interpretable manipulation. With only ten demonstration trajectories and no additional pretraining, SpatialVAM handles challenging long-horizon and contact-rich tasks, generalizes to out-of-distribution settings, and predicts realistic future videos. Evaluations on Meta-World (22\%$\uparrow$), RoboCasa (15\%$\uparrow$) and real-world robotic platforms (16\%$\uparrow$) show that SpatialVAM consistently outperforms other video action models, vision language action models and 3D-based policies, establishing a new state-of-the-art in data-efficient multi-task manipulation.