WSA$_1$: a 3D-Centric World-Spatial-Action Model for Generalizable Robot Control
WSA1 employs 3D world-spatial-action joint modeling with only 6K hours of demonstration data, achieving 93% success in RoboTwin2.0 and +20% over SOTA in real tasks.
Key Findings
Methodology
WSA1 utilizes a unified transformer-based multi-modal architecture with three expert modules: 2D spatial, 3D spatial, and 3D action experts. It employs a bidirectional causal attention mechanism to jointly learn predictive 3D world dynamics, visual reasoning, and inverse dynamics. The training targets include 3D world prediction, 2D visual thinking, and inverse dynamics, leveraging limited demonstration data (6000 hours). Key algorithms involve diffusion transformers for action prediction and multi-modal causal attention to ensure generalization. The model integrates these components to capture causal relationships between actions and world states.
Key Results
- In RoboTwin2.0 simulation benchmark, WSA1 achieved a 93% success rate, outperforming current SOTA models, with only 6,000 hours of demonstration data, including 1,000 hours of real robot data. In real-world tasks, performance improved by over 20%, demonstrating strong generalization.
- The model exhibited stable multi-task and multi-scene performance, validated by ablation studies showing the importance of the bidirectional causal mechanism. It adapts well to environmental changes, outperforming baselines significantly.
- Limited data pre-training yielded high performance, breaking the dependency on large-scale real robot data, and providing a new paradigm for data-efficient robot learning.
Significance
This work advances robot foundation models by integrating 3D causal world modeling, reducing reliance on large real-world datasets. It significantly enhances the generalization and robustness of robots in complex environments, bridging the gap between simulation and reality. The approach offers a practical pathway toward autonomous, adaptable robots capable of complex manipulation and decision-making, impacting both academia and industry.
Technical Contribution
The key technical innovation is the introduction of a 3D causal joint modeling framework within a multi-modal transformer architecture, enabling bidirectional learning of world dynamics and actions. The model employs a mixture-of-experts design and a bidirectional causal attention mechanism, which together improve data efficiency and generalization. This approach differs from prior unidirectional prediction models by explicitly modeling causal relationships, opening new possibilities for autonomous learning and reasoning.
Novelty
This is the first work to propose a 3D world-spatial-action joint modeling paradigm that explicitly captures causal relationships between robot actions and 3D environment evolution. Unlike existing models that focus on unidirectional prediction or semantic understanding, this method emphasizes bidirectional causal inference, significantly improving real-world applicability and robustness.
Limitations
- The model's generalization still faces challenges in highly dynamic or unseen environments due to limited training data diversity. Its reliance on pre-trained multi-modal models may limit transferability to radically different tasks.
- Computational complexity and inference latency are high, restricting real-time deployment in resource-constrained settings.
- Current validation is primarily in simulation and limited real-world scenarios; industrial-scale deployment requires further robustness and scalability improvements.
Future Work
Future directions include expanding multi-source data integration to cover more diverse environments, optimizing model efficiency for real-time applications, and incorporating reinforcement learning to enhance autonomous decision-making. Further research will focus on scaling the approach to industrial robotics and autonomous systems in unstructured, unpredictable environments.
AI Executive Summary
Robotic control research has long sought models capable of generalizing across diverse tasks and environments. Traditional approaches rely heavily on large-scale real-world data, which is costly and difficult to scale. This paper introduces WSA1, a novel robot foundation model built upon a 3D world-spatial-action joint modeling paradigm. By leveraging a sophisticated multi-modal transformer architecture with three expert modules, WSA1 explicitly models the causal relationships between robot actions and 3D world dynamics. The model employs a bidirectional causal attention mechanism, enabling it to predict how actions influence the environment and vice versa, fostering a deeper understanding of physical interactions.
The core innovation lies in unifying predictive 3D world modeling, visual reasoning, and inverse dynamics within a single framework. This integration allows the robot to generate goal-directed actions, predict their causal effects, and adapt behaviors dynamically. Training on only 6,000 hours of heterogeneous demonstration data, including just 1,000 hours of real robot data, WSA1 achieves state-of-the-art performance on the RoboTwin2.0 benchmark with a 93% success rate. In real-world tasks, it outperforms existing models by over 20%, demonstrating remarkable data efficiency and generalization.
These results highlight the potential of causal, 3D-centric modeling for advancing autonomous robots capable of complex manipulation in unstructured environments. The approach offers a practical pathway to reducing data dependency and improving robustness, paving the way for scalable, intelligent robotic systems. Future work will focus on expanding data sources, optimizing computational efficiency, and deploying in industrial settings, ultimately bringing autonomous, adaptable robots closer to widespread real-world application.
Deep Dive
Abstract
Recent advances in embodied AI have established robot foundation models (RFMs) as the dominant approach for generalist robotic systems to date. By leveraging imitation learning on extensive robot demonstrations, RFMs have achieved impressive capabilities in mapping visual observations and language instructions to continuous robotic actions. However, current RFMs lack an inherent ability to reason about physical dynamics and the causal effects of robot behaviors on the 3D physical world. This creates a fundamental mismatch between 2D-centric visual perception and 3D-centric embodied interaction, severely limiting the generalization ability of RFMs in real-world tasks.To address this gap, we present WSA$_1$, a novel RFM built upon proposed 3D-Centric World-Spatial-Action modeling paradigm. It not only learns 3D world-aware visual thought for future robot behaviors, but also models mutual constraints between 3D world state transitions and robotic actions to enhance behavior generalization. Notably, WSA$_1$ achieves highly data-efficient pre-training with 6k hours of expert demonstration data (only 1k hours from real robot), while delivering competitive manipulation performance (93% success rate) on RoboTwin2.0 simulation benchmark and achieving +20% average boosted performance over state-of-the-art RFMs on real-world robot control tasks. These results reveal that generalizable RFM can be attained without large-scale real robot data when paired with 3D-centric world-action joint modeling, which offers a practical and affordable pathway to generalist robotic systems.