Xiaomi-Robotics-U0: Unified Embodied Synthesis with World Foundation Model
Xiaomi-Robotics-U0 is a 38-billion-parameter multimodal autoregressive model enabling multi-robot, multi-view scene generation and fine-grained embodied transfer.
Key Findings
Methodology
The model employs a multimodal autoregressive framework based on Transformer architecture, integrating ViT and GPT components. It leverages large-scale pretraining on a World Foundation Model, extending it to embodied scenarios through multi-task joint optimization. Core tasks include text-to-image synthesis, image editing, embodied scene generation, transfer, and video synthesis. The training utilizes datasets like Robotics Scene Dataset, incorporating multi-view, multi-robot data, with structured control mechanisms to ensure geometric coherence and interaction fidelity. Loss functions combine cross-entropy, contrastive, and geometric consistency terms, facilitating multi-task learning and fine-grained control.
Key Results
- The model outperforms GPT-Image-2.0 in both single-step and sequential embodied scene generation, with human evaluation scores improving by over 20%. In real-world manipulation tasks, success rates increased from 36.9% to 63.2%, demonstrating enhanced robustness. It achieved first place in the World Arena embodied video generation challenge, showcasing superior dynamic scene synthesis. Ablation studies confirmed the importance of structured control mechanisms and multi-view consistency strategies for performance gains.
- Across diverse benchmarks, the model exhibits strong generalization in out-of-distribution scenarios, with significant improvements in transfer and editing capabilities. Its multi-robot multi-view scene generation maintains high fidelity and interaction realism, surpassing existing state-of-the-art methods. The results validate the model’s potential as a unified embodied world model and scalable data engine for embodied intelligence.
- Further experiments highlight the model’s ability to support fine-grained, controllable editing and dynamic interaction, with ablation results indicating that structured control contributes over 15% performance boost. The model’s robustness across tasks and environments underscores its versatility and scalability.
Significance
This work bridges the gap between large-scale foundation models and embodied robotics, demonstrating that pretraining on broad visual and textual data can be effectively transferred to complex embodied tasks. It addresses longstanding challenges in multi-view consistency, geometric coherence, and task generalization, paving the way for more autonomous, perceptive, and interactive robots. The approach offers a scalable pathway to integrate large multimodal models into real-world robotic systems, fostering advancements in autonomous manipulation, scene understanding, and multi-agent collaboration, with broad implications for both academia and industry.
Technical Contribution
The paper introduces a unified multimodal autoregressive architecture that extends the capabilities of large foundation models into embodied scenarios. It innovates with structured, controllable transfer mechanisms, multi-task joint training, and multi-view consistency strategies, enabling high-quality scene synthesis across multiple robots and perspectives. The integration of geometric and interaction constraints within a Transformer-based framework provides theoretical guarantees for multi-view coherence. The model’s design facilitates scalable training and inference, opening new avenues for embodied AI research and applications.
Novelty
This is the first model supporting high-fidelity multi-view scene generation across multiple robot embodiments within a unified framework. Unlike prior work limited to single-view or task-specific models, it achieves geometric consistency and interaction fidelity through structured control mechanisms. Its multi-task training paradigm and multi-view coherence strategies set new standards for embodied scene synthesis, making it a pioneering effort in scalable, controllable embodied AI.
Limitations
- Despite its strengths, the model struggles with highly dynamic or cluttered scenes where geometric and interaction consistency degrade. Its high computational cost limits real-time deployment on edge devices. The reliance on large-scale datasets raises concerns about data bias and generalization to unseen environments. Future work should focus on model compression, online adaptation, and incorporating reinforcement learning for more robust, real-time embodied interaction.
Future Work
Future directions include integrating online learning and reinforcement learning to enable adaptive, real-time interaction. Expanding multimodal inputs like audio and tactile data could enrich scene understanding. Efforts to optimize model efficiency will facilitate deployment on resource-constrained platforms. Additionally, exploring multi-agent collaboration and autonomous task planning will further advance embodied AI capabilities, bringing closer to autonomous robots capable of complex, real-world tasks.
AI Executive Summary
The quest to endow robots with human-like perception and interaction remains a central challenge in artificial intelligence. Traditional approaches often rely on task-specific models, which lack generalization and scalability. Recent advances in foundation models, exemplified by systems like DALL·E and GPT, have demonstrated remarkable capabilities in image and video synthesis, but their direct application to embodied robotics faces hurdles such as geometric consistency, multi-view coordination, and dynamic interaction.
Addressing these challenges, Xiaomi’s team introduces Xiaomi-Robotics-U0, a groundbreaking 38-billion-parameter multimodal autoregressive model designed for embodied scene synthesis. This model extends the principles of large-scale pretraining into the robotics domain, integrating text, image, and video modalities within a unified Transformer-based architecture. Its core innovation lies in combining multi-task learning with structured control mechanisms, ensuring multi-view geometric coherence and interaction fidelity across multiple robot embodiments.
The model is trained on extensive datasets like Robotics Scene Dataset, which include multi-view, multi-robot data, enabling it to generate high-quality, consistent scenes from various perspectives and control granular edits. Experimental results demonstrate its superiority over existing models such as GPT-Image-2.0, with significant improvements in scene quality, transfer success rates, and dynamic interaction realism. Notably, the model achieved first place in the World Arena embodied video generation challenge, underscoring its state-of-the-art performance.
These advances have profound implications for embodied AI, offering a scalable pathway to autonomous robots capable of perception, manipulation, and collaboration in complex environments. The research paves the way for future innovations in online learning, multimodal integration, and real-time deployment, ultimately bringing robots closer to human-like intelligence and adaptability. Despite current limitations in computational cost and scene complexity, ongoing work aims to address these challenges, promising a transformative impact on robotics and AI.
Deep Dive
Abstract
Recent foundation image and video generation models offer strong generalization and controllability, but their direct application to embodied scenarios is limited by requirements for multi-view consistency, geometric coherence, and robot embodiment constraints. Existing methods typically adapt foundation models with limited robot data, often sacrificing visual knowledge acquired during large-scale pre-training. We present Xiaomi-Robotics-U0, a 38-billion-parameter multimodal autoregressive model for unified embodied synthesis. It treats embodied generation as an extension of foundation image and video generation and jointly optimizes text-to-image generation, image editing, embodied scene generation, embodied transfer, and embodied video generation. This unified framework preserves the generalization of the pre-trained world foundation model while adapting it to embodied settings. Xiaomi-Robotics-U0 is the first model to support high-quality multi-view scene generation across multiple robot embodiments and to introduce structured, controllable embodied transfer for fine-grained editing while preserving multi-view consistency and interaction dynamics. It achieves state-of-the-art results on single-step and sequential generation tasks, outperforming GPT-Image-2.0 in human evaluations of embodied scene generation and transfer, ranking first on World Arena for embodied video generation, and improving the out-of-distribution success rate of pi_0.5 from 36.9% to 63.2% on challenging real-world manipulation tasks. These results show that foundation world models can serve both as embodied world models and scalable data engines for embodied intelligence. Code and checkpoints are available at https://robotics.xiaomi.com/xiaomi-robotics-u0.html.