One Forward Beats Two: InnerZoom for Accurate and Efficient GUI Grounding
InnerZoom achieves efficient GUI grounding with a single forward pass, significantly improving benchmark scores.
Key Findings
Methodology
InnerZoom employs a single forward pass to bridge cross-layer evidence, avoiding the extra computational cost of traditional ZoomIn methods. It transforms intermediate layer target-region evidence into a compact cross-layer state, preserving, refining, and reinjecting this state in later decoding layers to guide coordinate prediction.
Key Results
- InnerZoom achieves outstanding performance on six GUI grounding benchmarks, scoring 64.7 on OSWorld-G, 40.2 on UI-Vision, 73.1 on OSWorld-GR, and 87.6 on MMBench-GUI, surpassing previous best results by 4.1, 3.2, 2.9, and 2.3 points, respectively.
- Under a 4B setting, InnerZoom improves the same SFT+RL baseline by 5.3 points on average, outperforming two-pass ZoomIn by 1.3 points while reducing latency by 31.8% and TFLOPs by about 29%.
- Eliminates the loss of target-region evidence in final coordinate prediction by maintaining and refining cross-layer evidence states, achieving higher grounding accuracy.
Significance
InnerZoom addresses the bottleneck in GUI grounding by efficiently bridging the region-to-point gap, significantly reducing computational cost and latency. This method not only improves grounding accuracy but also offers a more efficient solution for interactive GUI agents, with significant academic and industrial implications.
Technical Contribution
InnerZoom technically bridges cross-layer evidence with a single forward pass, avoiding the extra computational overhead of traditional methods. It introduces iterative evidence adapters and evidence-guided decoding mechanisms, providing new engineering possibilities and theoretical guarantees.
Novelty
InnerZoom is the first to effectively bridge cross-layer evidence within a single forward pass, offering a more efficient solution compared to existing ZoomIn methods by avoiding additional inference steps.
Limitations
- In complex interfaces, target region identification may still be imprecise, affecting final coordinate prediction.
- Adaptability and scalability to large-scale models require further validation.
Future Work
Future research could explore InnerZoom's performance on larger datasets and further optimize cross-layer evidence extraction and utilization.
AI Executive Summary
In GUI grounding, traditional methods often require multiple forward passes to improve accuracy, leading to increased computational cost and latency. InnerZoom achieves efficient GUI grounding with a single forward pass by bridging cross-layer evidence. This method transforms intermediate layer target-region evidence into a compact cross-layer state, preserving, refining, and reinjecting this state in later decoding layers to guide coordinate prediction.
Experimental results show that InnerZoom significantly improves performance across multiple benchmarks, notably surpassing existing best results on datasets like OSWorld-G and UI-Vision. Additionally, InnerZoom significantly reduces computational latency and resource consumption, providing a more efficient solution for interactive GUI agents.
While InnerZoom achieves breakthroughs in accuracy and efficiency, its adaptability to complex interfaces remains an area for further research. Future work could explore its performance on larger datasets and further optimize cross-layer evidence extraction and utilization.
Deep Analysis
Background
With the advancement of human-computer interaction technology, GUI grounding has become an important research area. Traditional methods often rely on multiple forward passes to improve grounding accuracy, but this increases computational cost and latency. Recently, large model methods based on machine learning have become mainstream, but bottlenecks still exist in the region-to-point conversion process.
Core Problem
The core problem in GUI grounding is how to improve grounding accuracy without increasing computational cost. Traditional ZoomIn methods improve accuracy but require additional inference steps, leading to wasted computational resources.
Innovation
InnerZoom's core innovation lies in bridging cross-layer evidence with a single forward pass. This method avoids the extra computational overhead of traditional ZoomIn methods by maintaining and refining cross-layer evidence states for more efficient coordinate prediction.
Methodology
- �� Extract target-region evidence: Extract target-region cues from intermediate decoder responses.
- �� Evidence state optimization: Optimize cross-layer evidence states through iterative evidence adapters.
- �� Evidence-guided decoding: Inject optimized evidence states into key/value projections at target-region positions.
Experiments
Experiments used multiple public datasets, including OS-Atlas and OmniAct, to evaluate InnerZoom's performance across different benchmarks. Comparisons with existing methods verified InnerZoom's advantages in accuracy and efficiency.
Results
InnerZoom surpasses existing best results across multiple benchmarks, notably improving scores by 4.1 and 3.2 points on OSWorld-G and UI-Vision datasets, respectively. Additionally, InnerZoom significantly reduces computational latency and resource consumption.
Applications
InnerZoom can be directly applied to interactive GUI agents, improving their response speed and accuracy. Its efficient computation makes it widely applicable in resource-constrained environments.
Limitations & Outlook
While InnerZoom achieves breakthroughs in accuracy and efficiency, its adaptability to complex interfaces remains an area for further research. Additionally, its adaptability and scalability to large-scale models require further validation.
Plain Language Accessible to non-experts
Imagine you're cooking in a kitchen. Traditional methods are like preparing all ingredients for each dish before cooking, ensuring each dish is perfect but consuming a lot of time and effort. InnerZoom's approach is like having some basic ingredients ready and adjusting them flexibly while cooking each dish, saving time while ensuring quality. This way, InnerZoom improves GUI grounding accuracy and efficiency without increasing computational cost.
ELI14 Explained like you're 14
Imagine you're playing a game where you need to quickly click targets on the screen. Traditional methods are like zooming in on the screen each time to find the target before clicking, which is accurate but slow. InnerZoom's method is like marking the target's position in the game, allowing you to click quickly without zooming in each time. Isn't that cool?
Glossary
GUI Grounding
GUI grounding refers to the process of finding and clicking the target position in a user interface based on instructions.
Used in the paper to describe the application scenario of InnerZoom.
Cross-Layer Evidence
Cross-layer evidence refers to the target-region information transmitted and optimized between different layers of a neural network.
Core mechanism in InnerZoom for guiding coordinate prediction.
ZoomIn Method
A method that improves GUI grounding accuracy through multiple forward passes.
Traditional method compared with InnerZoom.
Evidence Adapter
An evidence adapter is a mechanism for optimizing and transmitting cross-layer evidence states.
Used in InnerZoom to optimize target-region evidence.
Target Region
The target region refers to the specific area in the GUI that needs to be located and clicked.
Extracted through intermediate decoder responses in InnerZoom.
Open Questions Unanswered questions from this research
- 1 How to improve target region identification accuracy in more complex interfaces?
- 2 How adaptable is InnerZoom to large-scale models?
Applications
Immediate Applications
Interactive GUI Agents
InnerZoom can be applied to interactive GUI agents, improving their response speed and accuracy, suitable for resource-constrained environments.
Long-term Vision
Intelligent Human-Computer Interaction Systems
InnerZoom can provide more efficient solutions for intelligent human-computer interaction systems, driving development in this field.
Abstract
MLLM-based GUI grounding methods commonly formulate target localization as autoregressive coordinate generation, enabling models to leverage the strong instruction-following and semantic understanding capabilities of MLLMs. However, this formulation requires the model to retain region-level target evidence while decoding coordinate tokens with the spatial precision demanded by GUI clicking. Our diagnostic analysis reveals that target-region awareness emerges in intermediate decoder layers but is neither retained nor translated into the final coordinate prediction. Existing ZoomIn-style methods address this issue through an external crop-and-rerun pass, which improves localization but increases end-to-end latency and computational cost. To retain the accuracy benefits of two-pass zooming without this extra cost, we propose InnerZoom, a single-forward framework for cross-layer evidence bridging. InnerZoom transforms target-related cues from the original forward pass into a compact cross-layer evidence state, then preserves, refines, and reinjects this state throughout later decoding layers to guide coordinate prediction. Extensive experimental results suggest that InnerZoom-4B achieves state-of-the-art performance on all six GUI grounding benchmarks, obtaining 64.7 on OSWorld-G, 40.2 on UI-Vision, 73.1 on OSWorld-GR, and 87.6 on MMBench-GUI, surpassing the previous best results by 4.1, 3.2, 2.9, and 2.3 points, respectively. Under a controlled 4B setting, InnerZoom improves the same SFT+RL baseline by 5.3 points on average and outperforms two-pass ZoomIn by 1.3 points on average, while reducing end-to-end latency by up to 31.8% and TFLOPs by about 29%. Code and models will be publicly available.