Do GUI Agents Believe Their Eyes? Diagnosing State-Belief Reliance on Pixels versus Structure
The study explores GUI agents' reliance on pixels vs. structure, introducing the Perception-Fusion Gap metric.
Key Findings
Methodology
The study tests multimodal GUI agents' reliance on pixels and structure using 735 probes, employing single-channel interventions to measure visual state reliance, and introduces the Perception-Fusion Gap (PFG) metric to analyze decision tendencies under conflict.
Key Results
- For textual states, all models, even when visually correct, rely on outdated structures in up to 0.88 of probes.
- White-box ablation shows textual state beliefs are influenced by a single structural value, with gradient attribution indicating visual evidence is processed but overridden.
- In multi-step environments, a single mis-sourced belief leads to task failure with a self-recovery rate of only 0.03.
Significance
The study reveals decision mechanisms of multimodal GUI agents under visual and structural information conflict. The proposed PFG metric provides a new method for assessing agents' visual reliance, significantly impacting the reliability of human-computer interaction systems.
Technical Contribution
Introduces the PFG metric, offering a novel method to measure decision tendencies of agents under visual and structural information conflict, revealing structural dependence of textual state beliefs, and validating this phenomenon through white-box ablation.
Novelty
First systematic measurement of GUI agents' reliance on visual vs. structural information under conflict, introducing the PFG metric, filling the gap left by existing benchmarks that fail to assess belief origins.
Limitations
- The study focuses mainly on the structural dependence of textual state beliefs, not fully exploring other belief types.
- The experimental environment is simulated, which may differ from real-world applications.
Future Work
Future research could extend to more types of belief reliance analysis and validate the PFG metric's effectiveness in real-world applications.
AI Executive Summary
Multimodal GUI agents read interface states through pixels and structured data. However, existing benchmarks fail to assess the source of agents' beliefs. This study tests 735 probes, formalizing visual state reliance and introducing the Perception-Fusion Gap (PFG) metric to measure decision tendencies under visual and structural information conflict.
The study finds that textual state beliefs, even when visually correct, tend to rely on structural information, particularly evident in outdated structural snapshots. White-box ablation reveals the mechanism by which textual state beliefs are influenced by a single structural value, with gradient attribution showing visual evidence is processed but overridden.
The significance of this study lies in revealing the decision mechanisms of multimodal GUI agents under visual and structural information conflict, providing a new perspective for improving the reliability of human-computer interaction systems. Future research could extend to more types of belief reliance analysis and validate the PFG metric's effectiveness in real-world applications.
Deep Analysis
Background
Multimodal GUI agents read interface states through pixels and structured data. However, existing benchmarks fail to assess the source of agents' beliefs. This study tests 735 probes, formalizing visual state reliance and introducing the Perception-Fusion Gap (PFG) metric to measure decision tendencies under visual and structural information conflict.
Core Problem
The decision mechanisms of multimodal GUI agents under visual and structural information conflict remain unclear, with existing benchmarks failing to assess belief origins, leading to potential erroneous decisions during task execution.
Innovation
Introduces the PFG metric, offering a novel method to measure decision tendencies of agents under visual and structural information conflict, revealing structural dependence of textual state beliefs, and validating this phenomenon through white-box ablation.
Methodology
- �� Use 735 probes to test multimodal GUI agents' reliance on pixels and structure
- �� Employ single-channel interventions to measure visual state reliance
- �� Introduce the Perception-Fusion Gap (PFG) metric
- �� Analyze structural dependence of textual state beliefs through white-box ablation
Experiments
The experimental design includes 735 probes covering real web, mobile, and desktop interfaces. Single-channel interventions measure visual state reliance, analyzing decision tendencies under visual and structural information conflict.
Results
The study finds that textual state beliefs, even when visually correct, tend to rely on structural information, particularly evident in outdated structural snapshots. White-box ablation reveals the mechanism by which textual state beliefs are influenced by a single structural value.
Applications
The findings can be used to improve the reliability of human-computer interaction systems, particularly in the design and optimization of multimodal GUI agents, ensuring correct decisions under visual and structural information conflict.
Limitations & Outlook
The study focuses mainly on the structural dependence of textual state beliefs, not fully exploring other belief types. The experimental environment is simulated, which may differ from real-world applications.
Plain Language Accessible to non-experts
Imagine using a smartphone app with buttons and text. A multimodal GUI agent is like your assistant, understanding the interface state through screenshots and backend data. The study finds that when screen information conflicts with backend data, the agent might rely on outdated backend data to make incorrect decisions. It's like seeing one thing and remembering another, but choosing to trust your memory over the present reality.
ELI14 Explained like you're 14
Hey there! Imagine you're playing a mobile game with lots of buttons and text. Your phone assistant is like a little robot, helping you play by looking at the screen and reading backend data. But sometimes, the screen info and backend data don't match, and the robot might choose to believe the outdated data instead of what's right in front of it. It's like remembering an answer differently from what the teacher says in class, but you choose to trust your memory. Isn't that interesting?
Glossary
Perception-Fusion Gap
Measures the decision tendency of models under visual and structural information conflict.
Used to assess agents' belief sources during information conflict.
Multimodal GUI Agent
An intelligent agent that reads interface states through pixels and structured data.
The study's subject, analyzing decision mechanisms under information conflict.
Single-Channel Intervention
Measures model reliance on a single information channel by altering it.
Used to measure visual state reliance in experiments.
White-Box Ablation
Analyzes decision mechanisms by removing specific information.
Used to validate structural dependence of textual state beliefs.
Gradient Attribution
Analyzes the importance of input features in model decisions.
Shows how visual evidence is processed in decision-making.
Open Questions Unanswered questions from this research
- 1 How to enhance agents' visual reliance in multimodal environments?
- 2 How to validate the PFG metric's effectiveness in real-world applications?
Applications
Immediate Applications
Human-Computer Interaction Optimization
Use the PFG metric to assess agents' visual reliance, improving system reliability.
Long-term Vision
Intelligent Agent Design
Design more reliable intelligent agents in multimodal environments, ensuring correct decisions during information conflict.
Abstract
Multimodal GUI agents read an interface through two redundant channels: the rendered pixels of a screenshot and a serialized structure such as a document object model or accessibility tree. Before acting, an agent forms a belief about the current interface state, but existing benchmarks score task success, element grounding, or attack resistance and do not ask whether that belief is drawn from the pixels. We formalize visual state reliance, the attribution of a state belief to pixels, structure, or priors, and measure it with paired single-channel interventions over 735 probes spanning real web, mobile, and desktop interfaces, of which 225 are zero-edit divergences mined from live production websites, all scored by deterministic forced choice with no model judge. Our central metric is the Perception-Fusion Gap (PFG), the fraction of probes a model perceives correctly yet resolves toward structure under conflict; a stricter variant that re-verifies perception on a tight crop of the target region leaves the gap intact. Across models from four vendors, textual state beliefs defer to structure while image-only accuracy stays near ceiling, and on unedited stale snapshots from live pages the same models follow the outdated structure on up to 0.88 of probes. A white-box ablation traces the textual effect to a single copied structural value, and gradient attribution shows the visual evidence is processed yet overridden. In live multi-step environments, one mis-sourced belief at the first step compounds into task failure with a self-recovery rate of at most 0.03. Comparing four mitigations on identical probes, prompt-level cues fail at the action level, certificate checks buy safety with refusals, and a training-free consistency gate is alone in reducing both hijack and task error. Visual state reliance thus gives a measurable diagnostic of whether agent state beliefs are visually grounded.