AndroidDaily: A Verifiable Benchmark for Mobile GUI Agents on Real-World Closed-Source Applications
Introduces AndroidDaily benchmark and GRADE evaluator, enabling automatic verification of GUI agents in real-world closed-source apps.
Key Findings
Methodology
AndroidDaily comprises 350 tasks across 94 high-frequency closed-source Android apps, designed based on empirical user behavior data. The three-tier external guidelines—operational obligations, output quality, negative constraints—standardize task completion criteria. GRADE employs a vision-language model (VLM) to track agent trajectories, match them against guidelines, and produce step-by-step diagnostics without internal state access. Large-scale experiments show 87.37% agreement with human judgments, revealing current models' performance gaps, especially in multi-constraint and cross-application tasks, with success rates around 62%. Analysis highlights latency, memory, and protocol issues as key bottlenecks.
Key Results
- The strongest model achieves only 62.0% success on AndroidDaily, indicating significant challenges in real-world scenarios.
- GRADE's automatic evaluation accuracy reaches 87.37%, validating its reliability in closed-source environments.
- Analysis identifies latency-induced misalignment, memory loops, and protocol degradation as primary failure modes, guiding future improvements.
Significance
This work addresses the critical challenge of evaluating GUI agents in commercial closed-source apps, providing a scalable, verifiable framework. AndroidDaily and GRADE bridge the gap between academic research and practical deployment, facilitating development of robust, real-world capable mobile AI assistants. The benchmark's diversity and realism push forward the understanding of multi-task, multi-constraint, cross-application interactions, vital for industry adoption and user experience enhancement.
Technical Contribution
The paper introduces a novel process-aware, multi-tier external guideline system combined with vision-language trajectory tracking, enabling step-level, verifiable evaluation without internal state access. It innovates by integrating multimodal models for real-time, long-horizon assessment, and constructing a large-scale, realistic task set grounded in empirical user data. These contributions significantly extend the capabilities of current GUI evaluation methods, offering interpretability and robustness in complex environments.
Novelty
This is the first work to implement a guideline-based, step-level diagnostic evaluation framework tailored for closed-source mobile applications. Unlike prior approaches relying on code assertions or binary success metrics, this method leverages external observable cues and multimodal models, enabling scalable, verifiable assessment of long-horizon, multi-constraint tasks in real-world scenarios. It represents a fundamental advance over existing static or synthetic benchmarks.
Limitations
- Models still struggle with multi-constraint, multi-task scenarios, primarily due to latency, memory, and protocol issues, limiting real-world applicability.
- Evaluation relies heavily on visual cues, which can be affected by interface changes or visual noise, challenging robustness.
- Current experiments focus on a subset of app categories; broader coverage and multi-modal integration are needed for comprehensive assessment.
Future Work
Future directions include optimizing inference speed via reinforcement learning, enhancing cross-application coordination, and integrating user feedback for adaptive guideline refinement. Expanding benchmark scope to more diverse app categories and multi-modal signals will further improve robustness and generalization, accelerating the deployment of intelligent mobile assistants in complex, real-world environments.
AI Executive Summary
As mobile devices become ubiquitous, the deployment of autonomous GUI agents promises to revolutionize user interaction, automating routine tasks and improving accessibility. However, evaluating these agents in real-world commercial applications remains a significant challenge. Most existing benchmarks rely on simulated environments or open-source apps, which do not reflect the complexity and opacity of closed-source apps that dominate daily mobile use. These apps do not expose internal states, making traditional code-based verification infeasible. Consequently, there is a pressing need for evaluation frameworks that can operate solely on observable external cues.
In response, this study introduces AndroidDaily, a large-scale benchmark comprising 350 realistic tasks across 94 high-frequency, closed-source Android applications. The tasks are designed to mirror real user behaviors in scenarios such as transportation, shopping, content creation, and social media. The benchmark emphasizes cross-application workflows, multi-constraint decision-making, and feedback-driven interactions, making it highly representative of practical use cases. To evaluate agents in such opaque environments, the authors propose GRADE, a process-aware, multi-tier external guideline system. GRADE leverages vision-language models (like CLIP and GPT-4) to track visual trajectories, replay actions, and match them against explicit external rules—covering operational obligations, output quality, and negative constraints.
Experimental results demonstrate that GRADE achieves 87.37% agreement with human judgments, validating its effectiveness. However, the best models only reach a 62% success rate on the benchmark, highlighting substantial gaps in reasoning, speed, and robustness. Analysis reveals that latency, memory management issues, and protocol degradation are primary bottlenecks. These findings underscore the importance of improving inference efficiency, long-term memory, and protocol stability for real-world deployment.
This work significantly advances the evaluation of mobile GUI agents, providing a scalable, verifiable, and realistic framework. It bridges the gap between academic research and practical applications, guiding future development towards more robust, efficient, and user-aligned AI assistants. The combination of large-scale real-world tasks and step-level diagnostics sets a new standard for benchmarking in this domain, with broad implications for industry and academia alike.
Deep Analysis
Background
近年来，移动GUI代理技术快速发展，代表性工作包括CogAgent、UI-TARS、Ferret-UI等。这些方法在静态界面理解和自动操作方面取得一定突破，但在商业闭源应用中难以验证其性能。传统评估多依赖模拟环境或代码断言，无法反映真实场景的复杂性。随着多任务、多约束、多应用场景的出现，迫切需要面向实际应用的动态、可验证评估体系。现有研究多关注单一任务或开源环境，缺乏对商业闭源应用的系统性评估，限制了技术的实际推广。
Core Problem
核心问题在于如何在无法访问应用内部状态的情况下，客观、自动地评估移动GUI代理的性能。现有方法依赖代码断言或系统变量，难以适应商业闭源应用的限制。实际任务具有多目标、多路径、多约束的特性，单一指标难以全面反映代理能力。长时序、多任务环境中的误差累积和状态偏差，进一步加剧了评估难度。解决这一问题对于推动移动智能助手的实用化具有重要意义，但缺乏有效的评估工具。
Innovation
本研究的创新点包括：1）提出基于外部指南的多层次任务定义体系，将任务转化为可观察的操作义务、输出质量和负面约束，避免依赖内部状态；2）设计视觉-语言模型（如CLIP、GPT-4）作为评估器，结合轨迹重放实现无状态验证；3）构建大规模真实场景任务集，模拟用户多样行为，提升评估的实用性。这些创新突破了传统依赖代码断言的限制，为闭源应用中的自动评估提供了新思路。
Methodology
- �� 任务构建：基于用户行为数据筛选94个高频应用，设计350个多场景任务，涵盖信息检索、内容生成和操作执行。
- �� 指南定义：每个任务配备三层外部指南，包括操作义务（如完成特定操作）、输出质量（如内容准确性）和负面约束（如避免违规行为）。
- �� 评估流程：利用视觉-语言模型（如CLIP、GPT-4）追踪代理轨迹，重放动作，提取关键视觉信息。
- �� 轨迹分析：匹配指南，检测任务完成情况，生成逐步诊断报告。
- �� 实验验证：在多个模型上测试，包括Auto-UI、Ferret-UI，比较与人类评判的一致性，分析失败原因。
Experiments
采用94个应用中350个任务作为评估集，使用不同规模的GUI代理模型（如Auto-UI、Ferret-UI）进行测试。指标包括成功率、与人类评判的一致性（87.37%），以及在多约束任务中的性能表现。通过消融实验验证指南体系的有效性，分析推理延迟、记忆管理和协议退化对性能的影响。设置不同的任务难度和约束密度，确保评估的全面性和代表性。
Results
结果显示，最强模型在AndroidDaily上的成功率为62.0%，远低于理想水平，反映出实际应用中的复杂性。GRADE的评估准确率达87.37%，验证其可靠性。分析发现，模型在多任务、多约束场景中表现不佳，主要受限于推理延迟（平均延迟超过1秒）、记忆管理不足导致的循环行为，以及协议能力退化。这些发现为未来模型优化提供了明确方向。
Applications
该评估框架适用于开发更强大的移动GUI代理，推动智能助手在实际商业场景中的应用。可用于企业测试、模型调优和用户体验优化，特别是在多任务、多应用协同的复杂环境中。未来还可结合用户反馈，动态调整任务指南，提升系统的适应性和鲁棒性。
Limitations & Outlook
当前模型在多任务、多约束环境中仍表现有限，推理延迟和记忆不足导致误差累积。视觉信息的依赖使得界面变化和视觉噪声成为鲁棒性挑战。此外，实验范围主要集中在部分应用类别，未来需扩展到更多场景和多模态信息融合，以提升整体性能。
Plain Language Accessible to non-experts
想象你在一家大型工厂工作，工厂里有许多不同的机器，每个机器都有自己的操作规程。有时候，你需要让不同的机器合作完成一项复杂的任务，比如制造一台新机器。你不知道每台机器的内部运作细节，只能通过观察它们的外部表现，比如声音、动作和显示屏上的信息。为了确保任务顺利完成，你制定了一套外部规则，比如必须先装配好零件，再进行测试，不能跳过步骤，也不能用错误的零件。这些规则帮助你判断每一步是否正确完成，而不用知道机器内部的详细流程。AndroidDaily和GRADE就像这样一套外部规则和观察系统，帮助我们评估手机上的智能助手在没有内部信息的情况下，是否完成了用户的任务。它们通过观察界面变化、操作轨迹，确保每一步都符合预设的规则，像工厂的检验员一样，确保每个环节都达标。这种方法让我们可以在真实、复杂的环境中，科学、客观地评估智能助手的表现，而不用依赖那些难以获取的内部数据。
ELI14 Explained like you're 14
想象你在学校里参加一个比赛，但你不能直接看老师的评分细节，只能通过观察老师的动作和你自己的表现来判断自己是不是做得好。比如，老师让你做一道数学题，你不知道老师心里在想什么，但你可以看他是否点头、是否写了评分纸、是否给了你提示。你要根据这些外部的线索，判断自己是不是完成得好。这就像AndroidDaily和GRADE一样，它们没有直接看到手机里的秘密数据，只能通过屏幕上的画面和操作轨迹，判断智能助手是否完成了任务。它们制定了一些规则，比如“必须点击这个按钮”、“不能跳过步骤”、“输出的内容要正确”。然后，系统会观察助手的每一步操作，看看是不是符合这些规则。这样，即使没有内部信息，也能公平、客观地评估助手的表现。就像老师用外部线索打分一样，这个方法让我们可以在真实的手机环境中，检测智能助手是不是帮我们完成了想做的事情。
Abstract
The rapid development of GUI foundation models and mobile GUI agents has spurred numerous evaluation benchmarks, yet most rely on simulated environments or open-source applications, leaving real-world closed-source applications largely unevaluated. The core difficulty is that closed-source applications do not expose internal states, making traditional automatic verification inapplicable. To bridge this gap, we introduce AndroidDaily, a large-scale benchmark comprising 350 realistic daily-use tasks across 94 high-frequency Android applications spanning transportation, shopping, local services, entertainment, content creation, social media, and everyday utilities. To enable automatic and verifiable assessment in these opaque environments, we propose Guideline-grounded Reviewer for Automatic Diagnostic Evaluation (GRADE), a process-aware evaluator built on a three-tiered system of observable external guidelines: operational obligations, output quality, and negative constraints. GRADE tracks the agent's visual trajectory against these criteria and produces step-level diagnostic judgments, turning long-horizon, open-ended mobile interactions into verifiable evaluation without relying on hidden internal states. Experiments show that GRADE achieves 87.37\% agreement with human evaluators. The strongest model reaches a 62.0\% success rate on AndroidDaily, highlighting a substantial gap between current reasoning capabilities and practical execution in realistic mobile workflows.