AMEX: Android Multi-annotation Expo Dataset for Mobile GUI Agents
AMEX dataset enhances mobile GUI agent performance with 104K screenshots and multi-level annotations.
Key Findings
Methodology
AMEX dataset improves mobile GUI agent performance through three-level annotations: GUI interactive element grounding, screen and element functionality descriptions, complex natural language instructions. SPHINX Agent is fine-tuned to validate dataset effectiveness.
Key Results
- In ScreenSpot test, SphAgent excelled in icon/widget subset with 72.6% accuracy.
- In ANDROIDCONTROL test, adding AMEX data improved click and long press actions by 14%.
- In AITW test, overall accuracy increased by 2.5%.
Significance
AMEX dataset enhances mobile GUI agents' environment understanding, addressing existing datasets' shortcomings in task diversity and annotation accuracy, providing new research directions for academia and industry.
Technical Contribution
AMEX offers detailed GUI element functionality descriptions, surpassing traditional icon classification methods, providing comprehensive environment understanding for mobile GUI agents. SPHINX Agent fine-tuning validates dataset effectiveness.
Novelty
AMEX is the first to provide multi-level GUI annotations, particularly defining scrollable elements and functionality descriptions, filling gaps in existing datasets.
Limitations
- Dataset mainly limited to English, with limited support for other languages.
- SPHINX Agent not evaluated online, real-world application effects unknown.
Future Work
Future work should expand to multilingual support, conduct online dynamic evaluations, and explore other operating system platforms.
AI Executive Summary
AI agents on mobile devices require better environment understanding to improve task completion efficiency. Existing datasets have shortcomings in annotation accuracy and task diversity, limiting agent performance.
The AMEX dataset enhances agents' environment understanding through multi-level GUI annotations, including interactive element grounding, functionality descriptions, and complex instruction chains. SPHINX Agent fine-tuning results validate dataset effectiveness.
Future work should expand to multilingual support and conduct online dynamic evaluations to further enhance agents' real-world application effects.
Deep Analysis
Background
With the proliferation of mobile devices, AI agents have become crucial tools for human-computer interaction. However, existing virtual assistants have limited functionality, unable to handle complex GUI tasks. Researchers are exploring new mobile interaction paradigms, developing GUI agents capable of directly interacting with screen elements.
Core Problem
Existing GUI agents face challenges in real-world applications, mainly due to a lack of understanding of GUI layouts and interactive element functionalities. These limitations stem from inadequate datasets that fail to reflect the complexity and diversity of mobile GUI environments.
Innovation
The AMEX dataset enhances mobile GUI agents' environment understanding through three-level annotations. First, interactive elements are defined as clickable and scrollable elements. Second, detailed screen and element functionality descriptions are provided. Finally, complex instruction chains are collected to simulate real-world application scenarios.
Methodology
- �� Data collection: Screenshots and XML data collected through manual and automated tools.
- �� Annotation: Interactive element grounding manually reviewed, GPT and Gemini generate functionality descriptions.
- �� Fine-tuning: SPHINX Agent fine-tuned using AMEX dataset.
Experiments
Experiments conducted using ScreenSpot, AITW, and ANDROIDCONTROL datasets for evaluation. SphAgent fine-tuning validates AMEX dataset's effectiveness in enhancing agent performance.
Results
In ScreenSpot test, SphAgent excelled in icon/widget subset. Adding AMEX data significantly improved click and long press actions in ANDROIDCONTROL test. Overall accuracy increased in AITW test.
Applications
AMEX dataset can be used to train smarter mobile GUI agents, improving task completion efficiency, applicable to various mobile application scenarios.
Limitations & Outlook
Dataset mainly limited to English, with limited support for other languages. SPHINX Agent not evaluated online, real-world application effects unknown. Future work should expand to multilingual support and conduct online dynamic evaluations.
Plain Language Accessible to non-experts
Imagine using a smart assistant on your phone that not only sets alarms but understands every icon and button on the screen. Like a super assistant, it can read every detail on the screen, knowing which button to click and which area to scroll. This way, it can help you complete various complex tasks, like finding the lowest price in a shopping app or creating a new playlist in a music app. This assistant is like an expert who knows all the rules of apps, quickly helping you complete tasks.
ELI14 Explained like you're 14
Imagine playing a mobile game with lots of buttons and menus. Now, there's a super assistant that can read these buttons and menus, knowing what each button does. For example, it knows which button upgrades your character and which menu shows tasks. This assistant is like a guide in the game, helping you quickly find what you need and complete tasks. It understands all the information on the screen, helping you make the right choices.
Glossary
GUI (Graphical User Interface)
Interface where users interact with devices through graphical elements.
Refers to application interfaces on mobile devices in the paper.
SPHINX Agent
Baseline model used for fine-tuning and validating AMEX dataset.
Used to evaluate dataset's enhancement of agent performance.
ScreenSpot
Benchmark dataset for evaluating model's element grounding capability.
Used in experiments to test SphAgent's performance.
AITW
Large-scale dataset for evaluating agent's task completion ability.
Used to test SphAgent's GUI control ability.
ANDROIDCONTROL
Dataset for evaluating agent's task completion ability on general apps.
Used to test SphAgent's performance improvement.
Open Questions Unanswered questions from this research
- 1 How to enhance agent performance in multilingual environments?
- 2 How to validate agent's real-world application effects through online dynamic evaluations?
Applications
Immediate Applications
Mobile App Optimization
Agents trained with AMEX dataset can be used to optimize mobile app user experience.
Long-term Vision
Cross-Platform Smart Assistant
Develop smart assistants supporting multiple operating systems for broader application scenarios.
Abstract
AI agents have drawn increasing attention mostly on their ability to perceive environments, understand tasks, and autonomously achieve goals. To advance research on AI agents in mobile scenarios, we introduce the Android Multi-annotation EXpo (AMEX), a comprehensive, large-scale dataset designed for generalist mobile GUI-control agents which are capable of completing tasks by directly interacting with the graphical user interface (GUI) on mobile devices. AMEX comprises over 104K high-resolution screenshots from popular mobile applications, which are annotated at multiple levels. Unlike existing GUI-related datasets, e.g., Rico, AitW, etc., AMEX includes three levels of annotations: GUI interactive element grounding, GUI screen and element functionality descriptions, and complex natural language instructions with stepwise GUI-action chains. We develop this dataset from a more instructive and detailed perspective, complementing the general settings of existing datasets. Additionally, we finetune a baseline model SPHINX Agent and illustrate the effectiveness of AMEX.The project is available at https://yxchai.com/AMEX/.