Mobile-Agent: Autonomous Multi-Modal Mobile Device Agent with Visual Perception
Mobile-Agent uses visual perception with DINO, OCR, and GPT-4V to autonomously operate mobile apps, achieving 91% success in Mobile-Eval.
Key Findings
Methodology
Mobile-Agent integrates visual detection models (Grounding DINO, OCR) with GPT-4V for element localization and task planning. The system comprises a visual perception module, operation decomposition, and self-reflection. It uses only screenshots, avoiding system metadata, to identify UI elements and generate step-by-step actions. The approach employs multi-modal fusion, iterative planning, and error correction, validated on Mobile-Eval benchmark, demonstrating high accuracy (over 90%) and robustness across multi-application scenarios.
Key Results
- On Mobile-Eval, Mobile-Agent achieved success rates of 91%, 82%, and 82% across three instruction types, averaging over 90%. The step accuracy (PS) was around 80%, indicating reliable operation. Efficiency reached approximately 80% of human steps, showing high autonomy. The self-reflection mechanism enabled error detection and correction, ensuring task completion even with invalid actions. These results confirm the system’s effectiveness in real-world-like multi-task environments.
- The experiments demonstrated strong performance in multi-application, multi-step tasks, with high success and completion rates. The model effectively handled complex instructions, including cross-application operations and Chinese interface scenarios. Its ability to self-correct errors contributed significantly to robustness. The approach surpasses traditional XML-based localization, offering a universal, permission-free solution for mobile automation.
- Case studies illustrated the model’s understanding of user instructions, ability to plan multi-step operations, and correct errors through self-reflection. It successfully managed multi-application workflows, maintained high accuracy, and supported multilingual interfaces. Overall, Mobile-Agent proved capable of autonomous, flexible, and reliable mobile operation, paving the way for future intelligent assistants.
Significance
This work advances mobile automation by eliminating reliance on system files, enabling universal, permission-free operation via pure visual perception. It addresses key challenges like element localization, task decomposition, and error correction, which are critical for scalable intelligent assistants. The high success and robustness demonstrate its potential for real-world deployment, reducing dependence on platform-specific tools. Such a vision-centric approach opens new avenues for developing adaptive, cross-platform mobile AI systems, significantly impacting both academia and industry by enabling more natural, flexible human-device interactions.
Technical Contribution
The core innovation lies in combining advanced visual detection (Grounding DINO, OCR) with GPT-4V’s multimodal reasoning to achieve system-agnostic UI element localization and task planning. The introduction of a self-reflection mechanism allows the system to identify and correct errors dynamically. The architecture supports multi-step, multi-application workflows without system-specific metadata, representing a significant leap over prior XML/HTML-dependent methods. This framework broadens the scope of autonomous mobile agents, enabling more flexible, scalable automation solutions.
Novelty
This is the first work to propose a purely vision-based, system-independent mobile agent capable of multi-step, multi-application task execution. Unlike previous approaches relying on app-specific files, it leverages visual detection and large language models for dynamic, adaptable operation. The integration of self-reflection for error correction further distinguishes it, providing a robust, self-improving system that can operate across diverse environments without platform-specific customization.
Limitations
- Despite high accuracy, the system occasionally mislocalizes UI elements in highly cluttered or similar interfaces, impacting reliability. Its performance on non-Latin scripts like Chinese needs further enhancement. The current speed is limited by detection and planning modules, affecting real-time responsiveness. Scalability to more complex tasks or longer workflows may require optimization. Future work should address these issues to improve robustness and efficiency.
Future Work
Future directions include integrating reinforcement learning to enhance decision-making, expanding support for more diverse operating systems, and improving real-time performance. Additionally, exploring multi-modal inputs like voice commands and haptic feedback could further enrich interaction capabilities. Developing adaptive models that learn user preferences and interface variations will make the system more personalized and scalable, pushing mobile AI towards truly autonomous, intelligent assistants.
AI Executive Summary
Deep Dive
Plain Language Accessible to non-experts
想象你在厨房做饭，手机就像厨房的操作台。以前的方法像是用菜谱（XML文件）告诉你每一步该做什么，但如果菜谱丢了或看不懂，就很难做菜。现在，这个新方法不用菜谱，只用眼睛观察厨房里的食材和工具（视觉信息），就像用直觉和观察去找食材、做菜。它能自己判断需要什么、在哪里找，还能在做错时自己发现并改正，就像一个聪明的厨师一样。这样，无论厨房怎么变，它都能灵活应对，帮你快速做出美味的菜肴（完成任务）。
ELI14 Explained like you're 14
想象你在玩一个超级复杂的游戏，里面有很多房间和任务。以前，你需要一本攻略（XML文件）告诉你每个房间在哪里、怎么操作，但如果攻略丢了或看不懂，就很麻烦。现在，这个新助手就像用你的眼睛和耳朵观察房间里的东西，然后自己想办法完成任务。它可以看图片、识别文字和图标，知道哪里可以点、怎么点，还能自己发现操作错了会改正，就像一个聪明的朋友帮你探索、学习，帮你完成各种挑战。这样，不管房间怎么变，它都能帮你应对，变得更聪明、更灵活！
Abstract
Mobile device agent based on Multimodal Large Language Models (MLLM) is becoming a popular application. In this paper, we introduce Mobile-Agent, an autonomous multi-modal mobile device agent. Mobile-Agent first leverages visual perception tools to accurately identify and locate both the visual and textual elements within the app's front-end interface. Based on the perceived vision context, it then autonomously plans and decomposes the complex operation task, and navigates the mobile Apps through operations step by step. Different from previous solutions that rely on XML files of Apps or mobile system metadata, Mobile-Agent allows for greater adaptability across diverse mobile operating environments in a vision-centric way, thereby eliminating the necessity for system-specific customizations. To assess the performance of Mobile-Agent, we introduced Mobile-Eval, a benchmark for evaluating mobile device operations. Based on Mobile-Eval, we conducted a comprehensive evaluation of Mobile-Agent. The experimental results indicate that Mobile-Agent achieved remarkable accuracy and completion rates. Even with challenging instructions, such as multi-app operations, Mobile-Agent can still complete the requirements. Code and model will be open-sourced at https://github.com/X-PLUG/MobileAgent.