Xiaomi-GUI-0 Technical Report
Xiaomi-GUI-0 achieves 72.0% success on real devices, significantly improving stability.
Key Findings
Methodology
Xiaomi-GUI-0 employs a real-device-dominant hybrid infrastructure, supported by sandboxes, ensuring execution distribution close to real deployment. The model is trained through a three-stage pipeline: supervised fine-tuning, step-level reinforcement learning, and agentic reinforcement learning, progressively enhancing task execution capabilities.
Key Results
- On RealMobile, Xiaomi-GUI-0 achieved a 72.0% success rate, while on AndroidWorld, it reached 78.9%, significantly enhancing execution stability and abnormal-state recognition in real-world tasks.
- The error-driven data flywheel mechanism allows the model to convert failure trajectories into corrective actions and recovery demonstrations, improving self-recovery in abnormal states.
- In experiments, the model demonstrated high task completion rates and robustness across various application scenarios, particularly excelling in handling abnormal pages.
Significance
This research narrows the gap between benchmark scores and real usability by training and evaluating GUI agents on real devices. It addresses shortcomings in interface layout, interaction logic, and abnormal state distribution, providing a solution more aligned with real-world applications for academia and industry.
Technical Contribution
Technical contributions include constructing a real-device-dominant hybrid infrastructure for task execution and trajectory collection, and introducing an error-driven data flywheel that extracts supervisory signals from error patterns exposed during real execution, enhancing model stability and robustness.
Novelty
Xiaomi-GUI-0 is the first to achieve end-to-end training and evaluation of multimodal GUI agents in real mobile environments, utilizing a real-device-dominant hybrid infrastructure and an error-driven data flywheel, significantly improving execution stability in abnormal states.
Limitations
- The model may still fail in handling extreme abnormal states, such as complex payment authentication or multi-layered permission dialogs.
- The reliance on physical devices limits the scale of training and evaluation.
Future Work
Future work could explore extending this approach to more device types and operating systems, and further optimize the error-driven data flywheel mechanism to enhance model performance in more complex tasks.
AI Executive Summary
Xiaomi-GUI-0 is a multimodal GUI agent designed for real mobile environments, addressing shortcomings in interface layout, interaction logic, and abnormal state distribution. By training and evaluating on real devices, the model achieves higher task completion rates and execution stability.
The model employs a real-device-dominant hybrid infrastructure, supported by sandboxes, ensuring execution distribution close to real deployment. Through a three-stage training pipeline, the model progressively enhances task execution capabilities, excelling in handling abnormal states.
Experimental results show that Xiaomi-GUI-0 achieved a 72.0% success rate on RealMobile and 78.9% on AndroidWorld. This research provides a solution more aligned with real-world applications for academia and industry, narrowing the gap between benchmark scores and real usability.
Deep Analysis
Background
With the proliferation of mobile devices, graphical user interface (GUI) agents are increasingly applied in automating task execution. However, existing GUI agents largely rely on offline trajectories, simulated environments, and standardized benchmarks, which differ significantly from real application scenarios. Researchers are exploring methods to train and evaluate GUI agents on real devices to better adapt to real application environments.
Core Problem
Existing GUI agents exhibit significant discrepancies in interface layout, interaction logic, and abnormal state distribution compared to real applications, leading to a persistent gap between high benchmark scores and real usability. Addressing this issue is crucial for improving execution stability and abnormal state recognition in real tasks.
Innovation
Xiaomi-GUI-0 achieves end-to-end training and evaluation of multimodal GUI agents in real mobile environments by constructing a real-device-dominant hybrid infrastructure and introducing an error-driven data flywheel, significantly enhancing execution stability in abnormal states.
Methodology
- �� Employ a real-device-dominant hybrid infrastructure, supported by sandboxes.
- �� Construct multi-source training data, covering high-frequency task data, high-generalization data, and capability enhancement data.
- �� Introduce an error-driven data flywheel to convert failure trajectories into corrective actions and recovery demonstrations.
- �� Train through a three-stage pipeline to progressively enhance task execution capabilities.
Experiments
Experiments were conducted on RealMobile and AndroidWorld, evaluating task completion rates on real devices and simulated environments. The design included various application scenarios and abnormal states to ensure robustness and stability under different conditions.
Results
Experimental results show that Xiaomi-GUI-0 achieved a 72.0% success rate on RealMobile and 78.9% on AndroidWorld. The model demonstrated high task completion rates and robustness across various application scenarios, particularly excelling in handling abnormal pages.
Applications
Xiaomi-GUI-0 can be directly applied to automate complex tasks on mobile devices, especially in scenarios requiring handling of abnormal states. Its high success rate and stability on real devices make it highly applicable in the industry.
Limitations & Outlook
Despite its strong performance on real devices, Xiaomi-GUI-0's reliance on physical devices limits its scale. Additionally, the model may still fail in handling extreme abnormal states.
Plain Language Accessible to non-experts
Imagine you're navigating a complex maze, and Xiaomi-GUI-0 is like a smart guide helping you find the exit. It knows every turn and can quickly adjust the route when obstacles appear. Trained in real environments, it recognizes complex situations, like sudden obstacles or wrong paths, and makes timely adjustments. It's like an experienced driver finding the fastest route in heavy traffic, ensuring you reach your destination on time.
ELI14 Explained like you're 14
Imagine playing a mobile game where you need to complete various tasks, like finding hidden treasures or solving puzzles. Xiaomi-GUI-0 is like a super helper, quickly completing these tasks for you. It understands your commands and offers solutions when you face challenges. For example, if you encounter a locked door needing a password, it finds the right key for you. It's like having a smart friend always ready to help, making you unbeatable in the game!
Glossary
GUI Agent
An intelligent agent capable of executing tasks on graphical user interfaces, typically trained using vision-language models.
Xiaomi-GUI-0, as a multimodal GUI agent, completes user tasks on real devices.
Multimodal
Technology combining multiple data modalities (e.g., vision and language) to enhance model understanding and execution capabilities.
Xiaomi-GUI-0 uses multimodal technology to achieve higher task completion rates.
Reinforcement Learning
A machine learning method training agents to optimize decisions through reward mechanisms.
Xiaomi-GUI-0 uses reinforcement learning to improve execution stability.
Abnormal State
Unexpected states during task execution, such as permission dialogs or payment authentication.
The model improves task completion by recognizing and handling abnormal states.
Data Flywheel
A feedback mechanism continuously optimizing data quality and model performance.
Xiaomi-GUI-0 uses an error-driven data flywheel to enhance execution stability in abnormal states.
Open Questions Unanswered questions from this research
- 1 How to extend this approach to a wider range of devices and operating systems?
- 2 How to further optimize the error-driven data flywheel mechanism to improve model performance in complex tasks?
Applications
Immediate Applications
Mobile Device Automation
Xiaomi-GUI-0 can be used to automate complex tasks on mobile devices, especially in scenarios requiring handling of abnormal states.
Long-term Vision
Intelligent Assistant
In the future, Xiaomi-GUI-0 could evolve into a more intelligent assistant, helping users complete more complex tasks across various devices.
Abstract
Graphical user interface (GUI) agents build on vision-language models to complete user tasks end-to-end in real applications through interface actions such as tapping, swiping, text entry, and navigation. However, existing GUI agents are trained and evaluated largely on offline trajectories, simulated environments, and standardized benchmarks. These differ substantially from real applications in interface layout, interaction logic, and abnormal-state distribution, and cannot faithfully characterize execution stability in real-world use, where account states, permission dialogs, payment authentication, and risk control continually reshape the state distribution and open a persistent gap between benchmark scores and real usability. To close this gap, we propose Xiaomi-GUI-0, a native multimodal GUI agent for real mobile environments, trained and evaluated within a real-device closed loop. At its core is a real-device-dominant hybrid infrastructure, where physical devices are the primary execution environment and sandboxes provide auxiliary support, so that data collection, training, rollout, and evaluation share an execution distribution close to real deployment. We construct multi-source training data spanning high-frequency head tasks, high-generalization data for long-tail intents, and capability-enhancement data for reflection and memory, and introduce an error-driven data flywheel that turns failure trajectories into corrected actions, reflective explanations, and recovery demonstrations. The model is trained through a progressive three-stage pipeline of supervised fine-tuning, step-level reinforcement learning, and agentic reinforcement learning. Evaluated on public benchmarks and our in-house RealMobile, Xiaomi-GUI-0 achieves 72.0% success on RealMobile and 78.9% on AndroidWorld, while substantially improving execution stability and abnormal-state recognition in real-world tasks.