GUIOdyssey: A Comprehensive Dataset for Cross-App GUI Navigation on Mobile Devices
GUIOdyssey dataset and OdysseyAgent model enhance cross-app navigation performance.
Key Findings
Methodology
GUIOdyssey dataset includes 8,334 cross-app navigation episodes, averaging 15.3 steps each, covering 6 devices and 212 apps. OdysseyAgent is a multimodal agent with a history resampler module optimizing performance and inference speed.
Key Results
- OdysseyAgent achieves the highest accuracy on the GUIOdyssey dataset, surpassing Claude3.5-Sonnet and GPT-4o, especially in complex cross-app tasks.
- Experiments show that historical information (actions, screenshots, context) significantly enhances OdysseyAgent's performance.
- OdysseyAgent performs well in both in-domain and out-of-domain scenarios.
Significance
This research addresses the limitations of existing datasets and models in cross-app navigation tasks by providing a comprehensive dataset and an efficient multimodal agent, advancing intelligent agent development for mobile devices.
Technical Contribution
Introduces the GUIOdyssey dataset and OdysseyAgent model, the former offering rich cross-app navigation instances, the latter effectively balancing performance and inference speed through a history resampler module.
Novelty
GUIOdyssey is the first dataset focused on cross-app navigation, and OdysseyAgent innovates with a multimodal approach and history resampler module.
Limitations
- OdysseyAgent may face performance bottlenecks when handling extremely long sequences.
- The dataset's diversity may not cover all possible application scenarios.
Future Work
Future research can expand dataset diversity, optimize the model for longer sequences, and explore more multimodal fusion strategies.
AI Executive Summary
With the proliferation of smartphone applications, users often need to navigate across multiple apps to complete tasks. Existing GUI navigation datasets and models are limited to single-app scenarios, failing to meet real-world demands. To address this, the research team developed GUIOdyssey, a dataset comprising 8,334 cross-app navigation episodes, averaging 15.3 steps each, involving 6 devices and 212 apps. Based on this dataset, they also developed OdysseyAgent, a multimodal agent using a history resampler module to optimize performance and inference speed. Experimental results show that OdysseyAgent excels in both in-domain and out-of-domain scenarios, particularly in complex cross-app tasks. This research not only provides new datasets and models for cross-app navigation but also points the way for future intelligent agent research.
Deep Analysis
Background
With the widespread use of smartphones, users need to switch between multiple apps to complete complex tasks. However, existing GUI navigation datasets and models are limited to single-app scenarios, failing to meet real-world demands. In recent years, researchers have attempted to enhance navigation capabilities of intelligent agents through large-scale language models and multimodal methods, but a dataset focused on cross-app navigation is still lacking.
Core Problem
Existing GUI navigation datasets are limited to single-app scenarios, unable to capture the complex interactions of cross-app tasks. This leads to poor performance of existing models in handling cross-app navigation tasks, necessitating a comprehensive cross-app navigation dataset to advance research.
Innovation
GUIOdyssey is the first dataset focused on cross-app navigation, covering a wide range of apps and devices. OdysseyAgent effectively balances performance and inference speed through a history resampler module, making it an innovative multimodal agent.
Methodology
- �� GUIOdyssey dataset includes 8,334 episodes, averaging 15.3 steps each.
- �� OdysseyAgent uses a multimodal approach, combining visual and language information.
- �� The history resampler module optimizes performance and inference speed.
Experiments
Experiments were conducted on the GUIOdyssey dataset, evaluating OdysseyAgent's performance in both in-domain and out-of-domain scenarios. Action Matching Score was used as the evaluation metric, showing that OdysseyAgent excels in complex cross-app tasks.
Results
OdysseyAgent achieves the highest accuracy on the GUIOdyssey dataset, especially in complex cross-app tasks. The use of historical information significantly enhances model performance.
Applications
The research outcomes can be used to develop smarter mobile device assistants, enhancing user experience, especially in scenarios requiring cross-app task completion.
Limitations & Outlook
OdysseyAgent may face performance bottlenecks when handling extremely long sequences, and the dataset's diversity may not cover all possible application scenarios. Future research can expand dataset diversity and optimize the model for longer sequences.
Plain Language Accessible to non-experts
Imagine having a smart assistant that helps you quickly complete tasks on your phone, like switching between different apps to perform complex operations. GUIOdyssey is like a large training ground with various tasks and app combinations, teaching this assistant how to work efficiently. OdysseyAgent is the brain of this assistant, remembering past actions and making quick decisions. Through continuous learning and optimization, it performs better in complex tasks.
ELI14 Explained like you're 14
Imagine you're playing a phone game that requires switching between multiple apps, like finding information in one app and using it in another. GUIOdyssey is the collection of game levels, and OdysseyAgent is your game character, remembering your past actions and helping you quickly pass levels. With continuous practice, it becomes smarter, helping you complete tasks faster!
Glossary
GUIOdyssey
A comprehensive dataset focused on cross-app navigation, containing 8,334 episodes.
Used to train and evaluate cross-app navigation agents.
OdysseyAgent
A multimodal navigation agent using a history resampler module to optimize performance.
Trained and tested on the GUIOdyssey dataset.
History Resampler Module
A module for compressing historical screenshot information to improve inference efficiency.
A key component in OdysseyAgent for handling long sequences.
Multimodal
A method combining visual and language information to enhance model understanding and decision-making.
OdysseyAgent uses multimodal methods to improve navigation performance.
Action Matching Score
A metric for evaluating navigation agent performance, measuring the match between predicted and actual actions.
Used to evaluate OdysseyAgent's performance on the GUIOdyssey dataset.
Open Questions Unanswered questions from this research
- 1 How to further enhance OdysseyAgent's performance on extremely long sequences?
- 2 How to expand the GUIOdyssey dataset to cover more application scenarios?
Applications
Immediate Applications
Smartphone Assistant
Enhance user experience in mobile device cross-app operations, reducing manual actions.
Long-term Vision
Fully Automated Workflows
Achieve more complex automated tasks, enhancing productivity and user experience.
Abstract
Autonomous Graphical User Interface (GUI) navigation agents can enhance user experience in communication, entertainment, and productivity by streamlining workflows and reducing manual intervention. However, prior GUI agents often trained with datasets comprising tasks that can be completed within a single app, leading to poor performance in cross-app navigation. To address this problem, we present GUIOdyssey, a comprehensive dataset for cross-app mobile GUI navigation. GUIOdyssey comprises 8,334 episodes with an average of 15.3 steps per episode, covering 6 mobile devices, 212 distinct apps, and 1,357 app combinations. Each step is enriched with detailed semantic reasoning annotations, which aid the model in building cognitive processes and enhancing its reasoning abilities for complex cross-app tasks. Building on GUIOdyssey, we develop OdysseyAgent, an exploratory multimodal agent for long-step cross-app navigation equipped with a history resampler module that efficiently attends to historical screenshot tokens, balancing performance and inference speed. Extensive experiments conducted in both in-domain and out-of-domain scenarios validate the effectiveness of our approach. Moreover, we demonstrate that historial information involving actions, screenshots and context in our dataset can significantly enhances OdysseyAgent's performance on complex cross-app tasks.