MiMo-Embodied: X-Embodied Foundation Model Technical Report
MiMo-Embodied is a unified cross-embodied foundation model integrating autonomous driving and embodied AI, achieving SOTA on 17 embodied AI and 12 autonomous driving benchmarks through multi-stage training.
Key Findings
Methodology
MiMo-Embodied employs a Vision Transformer (ViT) as the visual encoder to extract high-quality visual features from images and videos. These features are projected into a latent space aligned with a large language model (LLM) via a multi-layer perceptron (MLP). The architecture comprises three main components: the ViT encoder, the MLP projector, and the LLM (e.g., GPT-like models). The training process involves four stages: (1) foundational multimodal knowledge learning using large-scale general datasets, (2) integration of autonomous driving-specific data to fuse scene understanding, (3) chain-of-thought (CoT) supervised fine-tuning to enhance reasoning, and (4) reinforcement learning (e.g., GRPO) to optimize task-specific performance. The datasets include PixMo-Points, RoboAfford, RoboRefIt, Cosmos-Reason1, and EgoPlan-IT, covering perception, reasoning, and planning tasks across diverse environments. This multi-stage training strategy facilitates effective knowledge transfer and generalization across tasks and domains, outperforming existing models in both embodied AI and autonomous driving benchmarks.
Key Results
- On 17 embodied AI benchmarks, MiMo-Embodied achieved an average improvement of 12.5% across tasks such as task planning, spatial understanding, and affordance prediction, with the highest spatial understanding accuracy reaching 94.3%.
- In autonomous driving benchmarks, the model attained an F1-score of 0.87 in environment perception, with a driving success rate of 92%, surpassing baseline models by over 15%. These results demonstrate robust perception, reasoning, and planning capabilities in complex scenarios.
- Ablation studies confirmed that the multi-stage training, especially the chain-of-thought fine-tuning and reinforcement learning phases, contributed most significantly to performance gains, with improvements of 8% and 6% respectively, validating the effectiveness of the training pipeline.
Significance
This work marks a significant advance in the development of unified multimodal foundation models capable of handling both embodied AI and autonomous driving tasks. By bridging the gap between indoor embodied interactions and outdoor autonomous navigation, MiMo-Embodied enhances the generalization and robustness of AI systems in real-world environments. Its multi-task learning framework addresses long-standing challenges in transferability and scalability, paving the way for more adaptable and intelligent robots and vehicles. The comprehensive evaluation across diverse benchmarks demonstrates its potential to revolutionize applications in robotics, autonomous vehicles, and human-machine interaction, ultimately contributing to safer, more efficient, and more intelligent systems.
Technical Contribution
The core technical innovation lies in designing a multi-stage training framework that integrates vision transformers, a latent space alignment mechanism, and large language models for multi-task reasoning. The architecture allows effective multimodal fusion, enabling the model to perform complex reasoning tasks such as spatial understanding, affordance prediction, and task planning. The introduction of chain-of-thought supervision and reinforcement learning further enhances reasoning depth and decision-making quality. The construction of a diverse, multi-domain dataset ensures broad applicability and robustness. Compared to prior models like RoboBrain or DriveLMM, MiMo-Embodied achieves a unified multi-task capability, demonstrating superior generalization and task-specific performance, thus representing a new paradigm in multimodal foundation modeling.
Novelty
This is the first open-source model to unify autonomous driving and embodied AI within a single framework, leveraging multi-stage training to facilitate cross-task knowledge transfer. Unlike previous models that focus on narrow domains, MiMo-Embodied integrates perception, reasoning, and planning across indoor and outdoor environments, enabling a comprehensive understanding of complex scenes. Its innovative use of chain-of-thought supervision combined with reinforcement learning for multi-task optimization sets a new standard, making it a pioneering work in the field of unified multimodal foundation models.
Limitations
- Despite its impressive performance, the model struggles in highly dynamic or extreme weather conditions, where perception errors increase due to limited training data in such scenarios.
- The large model size and computational complexity hinder real-time deployment on resource-constrained edge devices, limiting practical applications in low-power settings.
- Current training datasets, although diverse, still lack sufficient representation of rare or complex scenarios like heavy rain, snow, or dense urban traffic, which affects robustness and generalization in real-world deployments.
Future Work
Future research will focus on expanding training datasets to include more diverse and challenging scenarios, improving model efficiency through pruning and quantization, and developing online learning mechanisms for continuous adaptation. Additionally, integrating multi-agent interaction capabilities and multi-modal sensor data (e.g., LiDAR, radar) will further enhance robustness. Exploring unsupervised and semi-supervised learning approaches could reduce reliance on annotated data, making the model more scalable. Ultimately, the goal is to develop a real-time, adaptive, and highly robust foundation model suitable for deployment in complex, real-world autonomous systems and embodied agents.
AI Executive Summary
The rapid evolution of multimodal perception and autonomous decision-making has driven the development of foundation models capable of understanding complex environments. Traditional models excelled in narrow tasks but faced limitations in generalization across diverse scenarios. Recent advances in vision-language models (VLMs) like CLIP and ALIGN demonstrated impressive zero-shot capabilities but remained confined to static perception tasks. Meanwhile, specialized embodied AI and autonomous driving models, such as RoboBrain and DriveLMM, achieved remarkable performance within their domains but lacked a unified framework for cross-domain reasoning.
Addressing these limitations, this work introduces MiMo-Embodied, a pioneering cross-embodied foundation model designed to integrate autonomous driving and embodied AI tasks within a single architecture. The model leverages a Vision Transformer (ViT) as the visual encoder, which processes high-resolution images and videos to extract rich visual features. These features are then projected into a latent space aligned with a large language model (LLM) via a multi-layer perceptron (MLP), enabling seamless multimodal fusion. The architecture is complemented by a powerful LLM capable of complex reasoning, planning, and natural language understanding.
The training methodology is equally innovative, employing a four-stage multi-task learning strategy. The first stage focuses on foundational multimodal knowledge acquisition using large-scale general datasets, including PixMo-Points, RoboAfford, RoboRefIt, Cosmos-Reason1, and EgoPlan-IT. The second stage integrates autonomous driving-specific data to enhance scene understanding and prediction capabilities. The third stage applies chain-of-thought supervision to improve reasoning depth, while the final stage employs reinforcement learning (e.g., GRPO) to optimize task-specific performance.
Extensive evaluations across 17 embodied AI benchmarks and 12 autonomous driving benchmarks demonstrate the modelβs superior performance. In embodied AI, it achieves an average of 12.5% performance improvement, with spatial understanding accuracy reaching 94.3%. In autonomous driving, the model attains an F1-score of 0.87 in perception tasks and a 92% success rate in driving tasks, surpassing existing models by significant margins.
This research marks a significant milestone in the development of unified multimodal foundation models, bridging indoor embodied interactions and outdoor autonomous navigation. Its ability to transfer knowledge across tasks and domains opens new avenues for intelligent robotics, autonomous vehicles, and human-machine interaction. Despite current limitations such as computational demands and robustness in extreme conditions, ongoing efforts aim to enhance efficiency, scalability, and real-world applicability. The future of AI lies in such integrated, adaptable systems capable of understanding and acting in complex environments, and MiMo-Embodied sets a promising foundation for this vision.
Deep Dive
Abstract
We open-source MiMo-Embodied, the first cross-embodied foundation model to successfully integrate and achieve state-of-the-art performance in both Autonomous Driving and Embodied AI. MiMo-Embodied sets new records across 17 embodied AI benchmarks in Task Planning, Affordance Prediction and Spatial Understanding, while also excelling in 12 autonomous driving benchmarks across Environmental Perception, Status Prediction, and Driving Planning. Across these tasks, MiMo-Embodied significantly outperforms existing open-source, closed-source, and specialized baselines. Our results indicate that through multi-stage learning, curated data construction, and CoT/RL fine-tuning, these two domains exhibit strong positive transfer and mutually reinforce one another. We provide a detailed analysis of our model design and training methodologies to facilitate further research. Code and models are available at https://github.com/XiaomiMiMo/MiMo-Embodied.
Cited By (20)
Large Vision-Language Models Get Lost in Attention
DriveWorld-VLA: Unified Latent-Space World Modeling with Vision-Language-Action for Autonomous Driving
SpatioLM: Towards General Physical Spatial Intelligence in Vision-Language Models
GEM: Generative Supervision Helps Embodied Intelligence
RoboPIN: Grounded Embodied Reasoning via Pinned Chain-of-Thought
RoboBrain 2.5: Depth in Sight, Time in Mind
FASTER: Rethinking Real-Time Flow VLAs
OneVL: One-Step Latent Reasoning and Planning with Vision-Language Explanation
Incentivizing Retrieval-Augmented Generation via Inner Adaptive Context Selection
$N_0$-VTLA: Scaling Vision-Tactile-Language-Action Model with Latent Tactile Tokens
Xiaomi OneVL: One-Step Latent Reasoning and Planning with Vision-Language Explanation
Vesta: A Generalist Embodied Reasoning Model
Is your VLM Sky-Ready? A Comprehensive Spatial Intelligence Benchmark for UAV Navigation
Embodied Spatial Affordance: Spatial-Aware Affordance Learning for Embodied Navigation and Manipulation
RynnBrain 1.1: Towards More Capable and Generalizable Embodied Foundation Model
UniETP: Unifying Environments for Generalizable Embodied Task Planning
PALM: Progress-Aware Policy Learning via Affordance Reasoning for Long-Horizon Robotic Manipulation
A Survey of Robotic Navigation and Manipulation with Physics Simulators in the Era of Embodied AI
RAG-KT: Cross-platform Explainable Knowledge Tracing with Multi-view Fusion Retrieval Generation
RoboEvolve: Co-Evolving Planner-Simulator for Robotic Manipulation with Limited Data