JarvisGUI: Towards Cross-Device GUI Agents with Dynamic Task Composition
JarvisGUI evaluates cross-device GUI agents with dynamic task composition, revealing capability gaps in real-world workflows.
Key Findings
Methodology
JarvisGUI models GUI tasks as input-output transformations using a lightweight type system, enabling automatic composition of multi-step cross-device workflows and dynamic evaluation of agent performance within a unified framework. Evaluations on Android, Windows, and Ubuntu reveal shortcomings in state-transfer awareness, cross-platform contextual reasoning, and long-horizon dependency management.
Key Results
- Experiments show that existing open-source GUI agents have significantly lower success rates on cross-device tasks compared to single-device tasks, particularly in long-horizon dependency management.
- In cross-platform tasks, agents struggle with state transfer and context retention, achieving only half the success rate of single-device tasks.
- In multi-step tasks, agents show low completion rates for intermediate steps, indicating deficiencies in complex task planning.
Significance
JarvisGUI provides a new benchmark for evaluating cross-device GUI agents, filling the gap in existing evaluations for multi-device interactions. By revealing capability gaps in real-world workflows, this research offers crucial guidance for future agent development.
Technical Contribution
This study is the first to systematically explore cross-device GUI agent tasks, proposing a slot-typing-based atomic task modeling framework that reliably and extensibly composes complex cross-device tasks. JarvisGUI reveals bottlenecks in cross-device coordination.
Novelty
JarvisGUI is the first benchmark specifically designed to evaluate cross-device GUI agents, breaking the limitations of existing single-device evaluations by providing a multi-platform ecosystem requiring coordination across devices.
Limitations
- JarvisGUI primarily evaluates in virtual environments, which may differ from real device settings.
- The evaluated agents are mostly open-source projects, which may not represent the capabilities of commercial agents.
Future Work
Future work could expand JarvisGUI to support more devices and operating systems and introduce real device environments for evaluation. Additionally, smarter agents could be developed to improve cross-device task success rates.
AI Executive Summary
In modern digital environments, users often need to coordinate tasks across multiple devices and platforms. However, existing GUI agent benchmarks are mostly limited to single-device tasks, failing to reflect real-world complexity. JarvisGUI addresses this gap by introducing dynamic task composition and cross-device evaluation.
JarvisGUI uses a lightweight type system to model GUI tasks as input-output transformations, enabling automatic composition of multi-step cross-device workflows. Evaluations on Android, Windows, and Ubuntu reveal shortcomings in state-transfer awareness, cross-platform contextual reasoning, and long-horizon dependency management.
Experimental results show that existing agents have significantly lower success rates on cross-device tasks compared to single-device tasks, particularly in long-horizon dependency management. This finding provides crucial guidance for future agent development and highlights the importance of evaluating in real device environments.
Deep Analysis
Background
With the rapid advancement of multimodal large language models, GUI agents can perceive complex visual environments and autonomously execute tasks. However, existing benchmarks focus on single-device tasks, neglecting the complexity of cross-device interactions. In practice, users often coordinate actions across smartphones, personal computers, and servers, a cross-device dependency not captured in existing benchmarks.
Core Problem
Existing GUI agent benchmarks are limited to single-device tasks, failing to reflect the complexity of real-world cross-device workflows. This leads to an overly optimistic assessment of agents' readiness for real-world use, unable to reveal their capability gaps in state-transfer awareness, cross-platform contextual reasoning, and long-horizon dependency management.
Innovation
JarvisGUI introduces a lightweight type system to model GUI tasks as input-output transformations, enabling automatic composition of multi-step cross-device workflows. Unlike existing benchmarks, JarvisGUI evaluates on Android, Windows, and Ubuntu, requiring agents to coordinate actions across multiple devices.
Methodology
- �� Model GUI tasks as input-output transformations using a lightweight type system.
- �� Automatically compose multi-step cross-device workflows.
- �� Evaluate on Android, Windows, and Ubuntu platforms.
- �� Dynamically assess agent performance, revealing shortcomings in state-transfer awareness, cross-platform contextual reasoning, and long-horizon dependency management.
Experiments
Experiments are conducted in virtual environments covering Android, Windows, and Ubuntu platforms. Various open-source GUI agents are evaluated using a two-stage planner-grounder architecture. The experimental design includes assessments of state transfer, context retention, and long-horizon dependency management in cross-device tasks.
Results
Experimental results show that existing agents have significantly lower success rates on cross-device tasks compared to single-device tasks, particularly in long-horizon dependency management. In cross-platform tasks, agents struggle with state transfer and context retention, achieving only half the success rate of single-device tasks.
Applications
JarvisGUI provides a new benchmark for evaluating cross-device GUI agents, applicable to scenarios requiring coordination across multiple devices and platforms. It helps developers identify capability gaps in real-world workflows and guides future development efforts.
Limitations & Outlook
JarvisGUI primarily evaluates in virtual environments, which may differ from real device settings. The evaluated agents are mostly open-source projects, which may not represent the capabilities of commercial agents. Future work could expand JarvisGUI to support more devices and operating systems and introduce real device environments for evaluation.
Plain Language Accessible to non-experts
Imagine you're in a kitchen, cooking different dishes on different stoves. Each stove represents a device, and you are the GUI agent. You need to move between stoves, ensuring each dish is cooked on time. JarvisGUI is like a cooking competition, evaluating your ability to coordinate tasks across multiple stoves. Existing benchmarks only focus on your performance on one stove, while JarvisGUI requires you to switch between multiple stoves, ensuring each dish is completed on time.
ELI14 Explained like you're 14
Imagine you're playing a game where you control multiple characters at once. Each character has different tasks, and you need to switch between them quickly to make sure each one completes their task. JarvisGUI is like this game, evaluating your ability to switch and coordinate between multiple characters. Existing benchmarks only focus on your performance with one character, while JarvisGUI requires you to switch between multiple characters, ensuring each one completes their task on time.
Glossary
GUI Agent
An intelligent program capable of executing tasks on a graphical user interface through simulated human interactions.
Used in the paper to evaluate cross-device task execution capabilities.
Cross-Device
Involves interactions and coordination across multiple devices.
Refers to tasks requiring coordination across multiple platforms in the paper.
Dynamic Task Composition
The process of automatically composing multiple steps to create complex tasks.
Used in the paper to generate cross-device workflows.
Lightweight Type System
A system for modeling task input-output transformations, supporting automatic task composition.
Used in the paper for modeling GUI tasks.
State-Transfer Awareness
The agent's ability to perceive state changes during task execution.
Used in the paper to evaluate agent performance in cross-device tasks.
Open Questions Unanswered questions from this research
- 1 How to evaluate GUI agents' cross-device capabilities in real device environments?
- 2 How to improve agents' performance in long-horizon dependency management?
- 3 How to expand JarvisGUI to support more devices and operating systems?
Applications
Immediate Applications
Cross-Device Workflow Evaluation
JarvisGUI can be used to evaluate scenarios requiring coordination across multiple devices and platforms, helping developers identify capability gaps in real-world workflows.
Long-term Vision
Intelligent Agent Development
By revealing existing agents' capability gaps, JarvisGUI provides crucial guidance for future intelligent agent development, driving the emergence of smarter agents.
Abstract
Real-world GUI usage frequently involves workflows that span multiple devices and platforms, requiring the transfer of intermediate results, maintenance of shared state, and coordination across heterogeneous environments. However, existing GUI benchmarks overwhelmingly evaluate agents on single-device, statically defined tasks, thus leaving such cross-device capabilities largely unexamined, resulting in an overly optimistic assessment of agents' readiness for real-world usage. We introduce JarvisGUI, a dynamic benchmark that evaluates GUI agents on cross-device workflows requiring coordinated interaction across heterogeneous platforms, including Android, Windows, and Ubuntu. Specifically, JarvisGUI formulates GUI tasks as input-output transformations under a lightweight type system, which allows us to automatically compose multi-step, cross-device workflows and dynamically evaluate agent performance within a unified framework. By evaluating agents in virtual environments spanning multiple operating systems, JarvisGUI reveals that state-of-the-art open-source GUI agents struggle with the state-transfer awareness, cross-platform contextual reasoning, and long-horizon dependency management required for real-world workflows, exposing a critical capability gap invisible to existing benchmarks.