GUI-Eyes: Tool-Augmented Perception for Visual Grounding in GUI Agents
GUI-Eyes achieves 44.8% grounding accuracy with tool-augmented perception using only 3k samples.
Key Findings
Methodology
GUI-Eyes employs a reinforcement learning framework with visual tools like cropping and zooming in a two-stage reasoning process. The first stage involves coarse exploration, and the second stage focuses on fine-grained grounding. A spatially continuous reward function is designed to integrate location proximity and region overlap, providing dense supervision.
Key Results
- On the ScreenSpot-Pro benchmark, GUI-Eyes-3B achieves 44.8% grounding accuracy using only 3,000 labeled samples, significantly outperforming supervised and RL baselines.
- It excels in CAD, development tools, and scientific software, achieving 48.2%, 70.8%, and 69.4% accuracy, respectively.
- Ablation studies confirm the effectiveness of the tool reward design, with the full reward function significantly improving grounding accuracy.
Significance
GUI-Eyes significantly enhances the robustness and data efficiency of GUI agents through tool-augmented active perception, addressing the high data annotation costs and generalization issues of traditional methods. This framework offers new insights into the integration of perception and decision-making, advancing human-computer interaction.
Technical Contribution
GUI-Eyes breaks new ground in active visual tool invocation and reward function design, proposing a two-stage reasoning strategy and spatially continuous reward function, significantly enhancing perception and decision-making capabilities.
Novelty
GUI-Eyes is the first to introduce tool-augmented active perception in GUI tasks, achieving deep integration of visual and decision-making processes, distinct from existing static input methods.
Limitations
- In complex interfaces, the tool invocation strategy may increase computational overhead.
- Performance on low-contrast or blurred interfaces still needs improvement.
Future Work
Future work could explore integrating more visual tools, such as dynamic zoom and multi-view fusion, to further enhance adaptability and generalization.
AI Executive Summary
Recent advances in vision-language models and reinforcement learning have driven progress in GUI automation. However, most existing methods rely on static visual inputs, lacking active perception capabilities. GUI-Eyes introduces a tool-augmented active perception framework, significantly improving interface grounding accuracy. This framework employs a two-stage reasoning strategy, with the first stage involving coarse exploration and the second stage focusing on fine-grained grounding. A spatially continuous reward function is designed, integrating location proximity and region overlap to provide dense supervision, alleviating the reward sparsity common in GUI environments.
On the ScreenSpot-Pro benchmark, GUI-Eyes-3B achieves 44.8% grounding accuracy using only 3,000 labeled samples, significantly outperforming existing supervised and RL baselines. This result highlights the critical role of tool-aware active perception in building robust and data-efficient GUI agents.
Despite significant progress, GUI-Eyes still faces challenges in computational overhead in complex interfaces and performance in low-contrast environments. Future work could explore integrating more visual tools to enhance adaptability and generalization.
Deep Analysis
Background
With the development of large language models and vision-language models, GUI automation has become a focus in human-computer interaction research. Existing methods mainly rely on supervised fine-tuning, requiring large annotated datasets and lacking robustness in unfamiliar environments. Reinforcement learning reduces supervision demands and improves adaptability and generalization.
Core Problem
Existing GUI agents mainly rely on static visual inputs, lacking active perception capabilities, making it difficult to handle complex interfaces and ambiguous instructions. Models need to integrate perception and decision-making, learning not only what to see but how often to observe, to support robust visual-grounded interactions.
Innovation
GUI-Eyes introduces a tool-augmented active perception framework, employing a two-stage reasoning strategy and spatially continuous reward function, achieving deep integration of visual and decision-making processes. Compared to traditional methods, it significantly enhances perception and decision-making capabilities.
Methodology
- �� Introduce a tool-augmented active perception framework with visual tools like cropping and zooming.
- �� Employ a two-stage reasoning strategy, with the first stage involving coarse exploration and the second stage focusing on fine-grained grounding.
- �� Design a spatially continuous reward function, integrating location proximity and region overlap to provide dense supervision.
Experiments
Evaluated on ScreenSpot, ScreenSpot-v2, and ScreenSpot-Pro benchmarks, using 3,000 labeled samples for training. Compared against existing supervised and RL baselines, demonstrating robustness and data efficiency.
Results
GUI-Eyes-3B achieves 44.8% grounding accuracy on the ScreenSpot-Pro benchmark, significantly outperforming existing supervised and RL baselines. It excels in CAD, development tools, and scientific software, achieving 48.2%, 70.8%, and 69.4% accuracy, respectively.
Applications
GUI-Eyes can be used for automated testing, user interface design, and human-computer interaction research, significantly improving interface grounding accuracy and efficiency.
Limitations & Outlook
In complex interfaces, the tool invocation strategy may increase computational overhead. Performance on low-contrast or blurred interfaces still needs improvement. Future work could explore integrating more visual tools to enhance adaptability and generalization.
Plain Language Accessible to non-experts
Imagine you're in a kitchen cooking, and GUI-Eyes is like a smart assistant helping you decide when to use tools like knives or blenders to better complete tasks. It first observes the kitchen broadly, then selects the right tool for detailed processing as needed. This not only saves time but ensures each step is done better. This approach is similar to intelligently choosing when to use visual tools in complex user interfaces to improve interface grounding accuracy.
ELI14 Explained like you're 14
Imagine you're playing a game where you need to find hidden objects. GUI-Eyes is like a super helper that not only tells you where to look but also helps you zoom in or crop the screen to make it easier to find the target. It's like a smart detective that knows when to look more closely and when to find the answer directly. This way, you can complete tasks faster and with fewer mistakes!
Glossary
GUI Agents
Graphical User Interface agents used for automating interface interaction tasks.
Used in the paper to implement active perception and decision-making models.
Active Perception
Dynamically adjusting observation strategies to acquire more useful information.
A key capability of GUI-Eyes in the paper.
Reinforcement Learning
A machine learning method that optimizes behavior strategies through reward signals.
The framework used to train GUI-Eyes.
Visual Tools
Tools used to enhance visual input, such as cropping and zooming.
Used in the paper to improve interface grounding accuracy.
Spatially Continuous Reward Function
A reward design integrating location proximity and region overlap.
Provides dense supervision, alleviating reward sparsity issues.
Open Questions Unanswered questions from this research
- 1 How to improve tool invocation strategy accuracy on low-contrast interfaces?
- 2 How to reduce computational overhead in complex interfaces?
- 3 How to integrate more visual tools to enhance adaptability?
Applications
Immediate Applications
Automated Testing
GUI-Eyes can be used for software automated testing, improving testing efficiency and accuracy.
User Interface Design
Enhances interface grounding capabilities, helping designers better understand and optimize user interfaces.
Long-term Vision
Intelligent Human-Computer Interaction
Could be used in more complex human-computer interaction scenarios in the future, improving user experience and system intelligence.
Abstract
Recent advances in vision-language models (VLMs) and reinforcement learning (RL) have driven progress in GUI automation. However, most existing methods rely on static, one-shot visual inputs and passive perception, lacking the ability to adaptively determine when, whether, and how to observe the interface. We present GUI-Eyes, a reinforcement learning framework for active visual perception in GUI tasks. To acquire more informative observations, the agent learns to make strategic decisions on both whether and how to invoke visual tools, such as cropping or zooming, within a two-stage reasoning process. To support this behavior, we introduce a progressive perception strategy that decomposes decision-making into coarse exploration and fine-grained grounding, coordinated by a two-level policy. In addition, we design a spatially continuous reward function tailored to tool usage, which integrates both location proximity and region overlap to provide dense supervision and alleviate the reward sparsity common in GUI environments. On the ScreenSpot-Pro benchmark, GUI-Eyes-3B achieves 44.8% grounding accuracy using only 3k labeled samples, significantly outperforming both supervised and RL-based baselines. These results highlight that tool-aware active perception, enabled by staged policy reasoning and fine-grained reward feedback, is critical for building robust and data-efficient GUI agents.