Say One Thing, Do Another? Diagnosing Reasoning-Execution Gaps in VLM-Powered Mobile-Use Agents
Introduces Ground-Truth Alignment framework to diagnose reasoning-execution gaps in VLM mobile agents.
Key Findings
Methodology
The paper proposes a new evaluation framework called Ground-Truth Alignment (GTA) to measure the alignment of reasoning chains with ground-truth actions. By combining GTA with the standard Exact Match (EM) metric, it diagnoses reasoning accuracy and execution accuracy gaps. The four-quadrant diagnostic space categorizes model outputs into ideal, execution gap, both wrong, and reasoning gap.
Key Results
- Reasoning-execution gaps are prevalent across various mobile interaction tasks, with execution gaps occurring more frequently than reasoning gaps. Even in the largest models, execution gaps remain significant.
- Scaling up model size reduces the overall gap, but sizable execution gaps persist.
- Experiments show that the GTA framework reliably reflects systematic EG/RG patterns in state-of-the-art models.
Significance
This research provides concrete tools and insights for diagnosing and reducing reasoning-execution gaps in mobile agents, supporting the development of more trustworthy agents. It is significant for improving user trust and reducing potentially harmful action authorizations.
Technical Contribution
Technical contributions include introducing the GTA evaluation framework, providing a new method to separate reasoning accuracy from execution accuracy. This method reveals systematic error patterns in existing models and offers directions for future model improvements.
Novelty
This is the first systematic diagnosis of reasoning-execution gaps in VLM-powered mobile agents. Unlike traditional execution accuracy evaluations, the GTA framework provides more granular diagnostic capabilities.
Limitations
- The GTA framework's evaluation relies on accurate CoT generation, and incorrect reasoning chains may lead to erroneous GTA scores.
- In complex tasks, models may overly rely on the superficial plausibility of reasoning chains.
Future Work
Future work could explore further narrowing execution gaps, especially in complex and dynamic environments. Enhancing model transparency and interpretability in reasoning processes is also a potential direction.
AI Executive Summary
Vision-language models (VLMs) powered mobile agents show great potential in interpreting natural language instructions and executing corresponding actions on mobile devices. However, existing evaluation methods often overlook the consistency between reasoning processes and actual execution, potentially leading to over-trust by users and authorization of harmful actions.
This paper introduces a new evaluation framework called Ground-Truth Alignment (GTA) to diagnose reasoning-execution gaps. By combining GTA with the standard Exact Match (EM) metric, the study reveals two main types of reasoning-execution gaps: Execution Gap (EG) and Reasoning Gap (RG). Experimental results show that these gaps are prevalent across various mobile interaction tasks, with execution gaps occurring more frequently.
While scaling up model size can reduce the overall gap, significant execution gaps remain even in the most advanced models. The study demonstrates that the GTA framework reliably reflects systematic error patterns, providing concrete diagnostic tools and support for developing more trustworthy mobile agents.
Deep Analysis
Background
In recent years, vision-language models (VLMs) have made significant progress in mobile agents. These agents can execute actions on mobile devices by interpreting screenshots and natural language instructions. However, existing evaluation methods primarily focus on execution accuracy, neglecting the consistency between reasoning processes and actual execution.
Core Problem
The core problem is the existence of reasoning-execution gaps, which may lead to over-trust by users and authorization of potentially harmful actions. Existing evaluation methods fail to effectively diagnose these gaps, limiting a deeper understanding of model behavior.
Innovation
The core innovation of this paper is the introduction of the Ground-Truth Alignment (GTA) evaluation framework. GTA measures the alignment of reasoning chains with ground-truth actions, providing a new method to diagnose reasoning-execution gaps.
Methodology
- �� Introduce the GTA evaluation framework, combined with the Exact Match (EM) metric.
- �� Design an automated GTA evaluator to map free-form reasoning chains into GTA scores.
- �� Conduct extensive experiments to systematically quantify and characterize execution gaps (EG) and reasoning gaps (RG).
Experiments
Experiments were conducted on multiple mobile interaction benchmarks, including AITZ, CAGUI, and AndroidControl. The models' reasoning accuracy (GTA) and execution accuracy (EM) were evaluated, with extensive experiments to quantify and characterize execution gaps (EG) and reasoning gaps (RG).
Results
Experimental results show that reasoning-execution gaps are prevalent across various mobile interaction tasks, with execution gaps occurring more frequently. Even in the most advanced models, execution gaps remain significant.
Applications
The GTA framework can be used to diagnose and reduce reasoning-execution gaps in mobile agents, supporting the development of more trustworthy agents. This is significant for improving user trust and reducing potentially harmful action authorizations.
Limitations & Outlook
The GTA framework's evaluation relies on accurate CoT generation, and incorrect reasoning chains may lead to erroneous GTA scores. In complex tasks, models may overly rely on the superficial plausibility of reasoning chains.
Plain Language Accessible to non-experts
Imagine you're cooking in a kitchen. You have a recipe (reasoning chain) that tells you how to make a dish (execution action). If you follow the recipe but the dish turns out wrong, that's an execution gap. If you make the right dish but the recipe and what you actually did don't match, that's a reasoning gap. This research is like checking if your recipe and actual cooking steps align to ensure you make the perfect dish every time.
ELI14 Explained like you're 14
Imagine you're playing a mobile game, and the game gives you instructions like tapping a button on the screen. Now, imagine your phone assistant is also playing the game, but sometimes it messes up. It thinks right but presses the wrong button—that's an execution gap. Sometimes it presses the right button but thinks differently from what it does—that's a reasoning gap. This research helps make your phone assistant smarter so it doesn't make these mistakes!
Glossary
Vision-Language Model (VLM)
A model that combines visual and language information to understand and generate multimodal data.
Used in mobile agents to interpret screenshots and natural language instructions.
Chain-of-Thought (CoT)
A sequence of intermediate reasoning steps generated by the model during task execution.
Used to improve execution accuracy and alignment with user intent.
Ground-Truth Alignment (GTA)
A metric to evaluate the alignment of reasoning chains with ground-truth actions.
Core metric for diagnosing reasoning-execution gaps.
Execution Gap (EG)
A scenario where reasoning is correct but execution fails.
Identifies execution issues in GTA evaluation.
Reasoning Gap (RG)
A scenario where execution succeeds but reasoning process conflicts with actual execution.
Identifies reasoning issues in GTA evaluation.
Open Questions Unanswered questions from this research
- 1 How to further narrow execution gaps in complex and dynamic environments?
- 2 How to enhance model transparency and interpretability in reasoning processes?
Applications
Immediate Applications
Mobile App Navigation
Improving reasoning-execution consistency to enhance navigation capabilities in mobile apps and reduce user errors.
Automated Task Execution
Applying the GTA framework in automated tasks to improve execution accuracy and task completion efficiency.
Long-term Vision
Trustworthy Smart Assistants
Developing more trustworthy smart assistants capable of accurately executing user instructions in complex environments.
Abstract
Mobile-use agents powered by vision-language models (VLMs) have shown great potential in interpreting natural language instructions and generating corresponding actions based on mobile graphical user interface. Recent studies suggest that incorporating chain-of-thought (CoT) reasoning tends to improve the execution accuracy. However, existing evaluations emphasize execution accuracy while neglecting whether CoT reasoning aligns with ground-truth actions. This oversight fails to assess potential reasoning-execution gaps, which in turn foster over-trust: users relying on seemingly plausible CoTs may unknowingly authorize harmful actions, potentially resulting in financial loss or trust crisis. In this work, we introduce a new evaluation framework to diagnose reasoning-execution gaps. At its core lies Ground-Truth Alignment (GTA), which measures whether the action implied by a CoT matches the ground-truth action. By combining GTA with the standard Exact Match (EM) metric, we jointly assess both the reasoning accuracy and execution accuracy. This joint perspective reveals two types of reasoning-execution gaps: (i) Execution Gap (EG), where the reasoning correctly identifies the correct action but execution fails, and (ii) Reasoning Gap (RG), where execution succeeds but reasoning process conflicts with the actual execution. Experimental results across a wide range of mobile interaction tasks reveal that reasoning-execution gaps are prevalent, with execution gaps occurring more frequently than reasoning gaps. Moreover, while scaling up model size reduces the overall gap, sizable execution gaps persist even in the largest models. Further analysis shows that our framework reliably reflects systematic EG/RG patterns in state-of-the-art models. These findings offer concrete diagnostics and support the development of more trustworthy mobile-use agents.