VenusBench-Mobile: A Challenging and User-Centric Benchmark for Mobile GUI Agents with Capability Diagnostics
VenusBench-Mobile evaluates mobile GUI agents through user-intent-driven tasks and capability-oriented annotations, revealing their shortcomings in real environments.
Key Findings
Methodology
VenusBench-Mobile evaluates using user-intent-driven task design and capability-oriented annotation, covering 10 user intent categories and 80 environment variations. This approach allows for fine-grained behavior analysis, revealing deficiencies in perception and memory.
Key Results
- On VenusBench-Mobile, current SOTA agents achieve only a 36.9% success rate, significantly lower than traditional benchmarks, indicating higher task difficulty.
- Diagnostic analysis shows failures are primarily due to deficiencies in perception and memory, often obscured by coarse evaluations.
- Even the strongest agents achieve near-zero success under environment variations, exposing their brittleness to realistic distribution shifts.
Significance
VenusBench-Mobile reveals the shortcomings of current mobile GUI agents in real-world deployment through more challenging and realistic tasks, providing a crucial evaluation benchmark to advance more general and robust agents.
Technical Contribution
The study introduces a user-intent-driven task design and capability-oriented annotation scheme, enabling fine-grained failure attribution analysis, distinct from existing coarse success rate evaluations.
Novelty
VenusBench-Mobile is the first benchmark to design tasks from a user-intent perspective, breaking away from previous app-centric evaluation paradigms.
Limitations
- Current benchmarking is primarily conducted in simulated environments, which may not fully reflect real-world complexity.
- The types of environment variations are limited, potentially not covering all possible real scenarios.
Future Work
Future research can expand the types and complexity of environment variations and conduct tests on real devices to better assess agent performance.
AI Executive Summary
VenusBench-Mobile introduces a novel benchmarking approach by evaluating mobile GUI agents in realistic environments through user-intent-driven task design and capability-oriented annotation. Existing benchmarks are largely app-centric and task-homogeneous, failing to reflect the diversity and instability of real-world usage. VenusBench-Mobile defines what and how to evaluate, revealing deficiencies in perception and memory capabilities of current agents, even under environment variations.
Experimental results show that current agents achieve significantly lower success rates on VenusBench-Mobile compared to traditional benchmarks, indicating higher task difficulty and realism. Diagnostic analysis further reveals deficiencies in perception and memory, often obscured by coarse evaluations.
Based on these findings, VenusBench-Mobile provides a crucial evaluation benchmark for robust deployment of mobile GUI agents, advancing more general and robust agents. Future research can expand environment variations and conduct tests on real devices to better assess agent performance.
Deep Analysis
Background
Recent advances in visual language models enable mobile GUI agents to interact with graphical user interfaces through visual perception and natural language instructions. Existing benchmarks are largely app-centric and task-homogeneous, failing to reflect real-world diversity and instability, limiting agents' reliability and deployment.
Core Problem
Existing benchmarks fail to adequately evaluate mobile GUI agents in realistic environments, particularly under diverse user intents and environment variations. Deficiencies in perception, memory, and decision-making are often obscured by coarse evaluations.
Innovation
VenusBench-Mobile introduces a user-intent-driven task design and capability-oriented annotation scheme, providing a more challenging and realistic evaluation framework. It covers 10 user intent categories and 80 environment variations, allowing fine-grained behavior analysis.
Methodology
- �� User-intent-driven task design covering 10 categories and 149 tasks.
- �� Capability-oriented annotation scheme for fine-grained behavior analysis.
- �� Systematic environment variations to test agent robustness under realistic distribution shifts.
Experiments
Experiments were conducted on VenusBench-Mobile, evaluating various SOTA mobile GUI agents, including UI-Venus-72B and Gemini-3-Pro. Metrics used include task success rate, capability dimension accuracy, and stability evaluation.
Results
Results show that current agents achieve significantly lower success rates on VenusBench-Mobile compared to traditional benchmarks, indicating higher task difficulty. Diagnostic analysis reveals deficiencies in perception and memory.
Applications
VenusBench-Mobile can be used to evaluate and improve mobile GUI agents' performance, particularly under diverse user intents and environment variations. It provides a crucial evaluation benchmark for robust deployment.
Limitations & Outlook
Current benchmarking is primarily conducted in simulated environments, which may not fully reflect real-world complexity. The types of environment variations are limited, potentially not covering all possible real scenarios. Future research can expand environment variations and conduct tests on real devices.
Plain Language Accessible to non-experts
Imagine using a smartphone app where an assistant understands what you want to do and helps you complete tasks. VenusBench-Mobile is like a rigorous exam testing these assistants' performance in various situations. It not only checks if the assistant can complete tasks but also analyzes their stability and capabilities under different environments. This way, we can identify shortcomings and improve their performance.
ELI14 Explained like you're 14
Imagine playing a game where the assistant helps you complete various tasks. VenusBench-Mobile is like a super hard level testing these assistants in different situations. It not only checks if the assistant can complete tasks but also analyzes their stability and capabilities under different environments. This way, we can find out where they fall short and make them better. Cool, right?
Glossary
GUI Agent
Graphical User Interface agents are automated systems capable of interacting with user interfaces.
Used to execute user instructions and complete tasks.
User Intent-Driven
Tasks designed based on actual user needs and intents.
Used to design more realistic evaluation tasks.
Capability-Oriented Annotation
Fine-grained task failure analysis based on agent capabilities.
Used to identify specific capability deficiencies of agents.
Environment Variations
Testing agent robustness under different language, layout, and interface conditions.
Used to evaluate agent performance under realistic distribution shifts.
Success Rate
The proportion of tasks successfully completed by an agent in the benchmark.
Used to evaluate overall agent performance.
Open Questions Unanswered questions from this research
- 1 How to test agent performance on real devices to better reflect real-world complexity.
- 2 Expand the types of environment variations to cover more possible real scenarios.
Applications
Immediate Applications
Mobile App Testing
Developers can use VenusBench-Mobile to evaluate GUI agent performance in their apps, ensuring stability under diverse user needs.
Long-term Vision
Smart Assistant Improvement
Identify and improve agent shortcomings, advancing more general and robust smart assistants.
Abstract
Existing online benchmarks for mobile GUI agents remain largely app-centric and task-homogeneous, failing to reflect the diversity and instability of real-world mobile usage. To this end, we introduce VenusBench-Mobile, a challenging online benchmark for evaluating general-purpose mobile GUI agents under realistic, user-centric conditions. VenusBench-Mobile builds two core evaluation pillars: defining what to evaluate via user-intent-driven task design that reflects real mobile usage, and how to evaluate through a capability-oriented annotation scheme for fine-grained agent behavior analysis. Extensive evaluation of state-of-the-art mobile GUI agents reveals large performance gaps relative to prior benchmarks, indicating that VenusBench-Mobile poses substantially more challenging and realistic tasks and that current agents remain far from reliable real-world deployment. Diagnostic analysis further shows that failures are dominated by deficiencies in perception and memory, which are largely obscured by coarse-grained evaluations. Moreover, even the strongest agents exhibit near-zero success under environment variations, highlighting their brittleness in realistic settings. Based on these insights, we believe VenusBench-Mobile provides an important stepping stone toward robust real-world deployment of mobile GUI agents. Code and data are available at https://github.com/inclusionAI/UI-Venus/tree/VenusBench-Mobile.