Enhancing Visual Grounding for GUI Agents via Self-Evolutionary Reinforcement Learning
Proposes Self-Evolutionary Reinforcement Learning for GUI grounding, achieving 47.3% accuracy with only 3K samples on a 7B model.
Key Findings
Methodology
This work introduces a framework combining seed data filtering, dense policy gradients, and self-evolution via attention maps. First, high-quality seed data is curated by filtering noisy samples. Second, a dense point reward mechanism is designed to address sparse rewards, evaluating proximity between predicted and true positions. Third, the model iteratively refines itself using attention maps as self-supervision, guiding the localization process. The approach employs Group Relative Policy Optimization (GRPO), integrating layer-wise attention to progressively calibrate interface element localization. Experiments show that with only 3K samples, the 7B model surpasses larger models across three benchmarks, notably achieving 47.3% accuracy on ScreenSpot-Pro, outperforming UI-TARS-72B by 24.2%.
Key Results
- On three GUI grounding benchmarks, SE-GUI-7B with 3K samples achieves 47.3% accuracy, significantly outperforming traditional supervised fine-tuning (SFT) models and larger models, demonstrating the efficiency of reinforcement learning combined with self-evolution.
- Across multi-platform environments, the model reaches 88.2% and 90.3% accuracy on ScreenSpot and ScreenSpot-v2 datasets, respectively, indicating strong generalization.
- Ablation studies confirm that high-quality data filtering and high-resolution inputs are critical for performance gains, emphasizing the importance of visual detail and noise reduction.
Significance
This research addresses longstanding challenges in GUI understanding, notably data scarcity and poor generalization in high-resolution environments. By integrating reinforcement learning with self-supervised attention maps, it offers a scalable, data-efficient solution that can operate effectively with minimal supervision. The approach paves the way for more autonomous, adaptable GUI agents, reducing reliance on extensive labeled datasets and enabling deployment in real-world, complex scenarios such as industrial automation, assistive technologies, and cross-platform automation. Its ability to leverage small datasets while maintaining high accuracy marks a significant step forward in intelligent visual grounding.
Technical Contribution
The paper introduces a novel reinforcement learning framework that combines dense point rewards with attention-guided self-supervision, enabling models to iteratively refine their focus on interface elements. The key technical innovation lies in using layer-wise attention maps as intermediate supervision signals, which are projected back onto the visual space to guide localization. The method also employs a filtering strategy to ensure data quality and a KL divergence constraint to stabilize training. These contributions collectively enable high performance in few-shot settings, setting a new benchmark for GUI visual grounding.
Novelty
This work is the first to integrate self-evolution attention maps into reinforcement learning for GUI element localization, effectively addressing reward sparsity and data noise issues. Unlike prior SFT-based methods that require vast labeled datasets, this approach emphasizes model self-supervision and iterative refinement, demonstrating superior performance with minimal data. Its combination of dense rewards and attention-based feedback constitutes a new paradigm in visual grounding, especially in high-resolution, complex environments.
Limitations
- The current approach may struggle with highly dynamic or extremely cluttered interfaces where attention maps cannot reliably focus on target elements. Its performance in real-time or highly interactive scenarios remains to be validated.
- Computational overhead increases due to attention map generation and iterative training, limiting scalability in resource-constrained settings.
- The method's effectiveness in ultra-high-resolution environments (>5 million pixels) needs further exploration, as current experiments are limited to this resolution range.
Future Work
Future research will explore multi-modal fusion, integrating textual and contextual cues to enhance localization robustness. Efforts will also focus on optimizing attention map computation for efficiency, enabling real-time deployment. Extending the framework to handle dynamic, multi-task, and multi-lingual interfaces will further broaden its applicability. Additionally, integrating unsupervised pretraining and domain adaptation techniques could improve performance in unseen or evolving environments, advancing towards truly autonomous GUI understanding.
AI Executive Summary
Graphical User Interface (GUI) agents have become increasingly capable of understanding and executing user commands across diverse platforms. Despite these advances, accurately grounding instructions to specific interface elements remains a significant challenge, especially in high-resolution, complex environments. Traditional supervised fine-tuning (SFT) methods, while effective in simple scenarios, demand vast amounts of labeled data and often suffer from weak generalization. To address these limitations, this paper introduces a novel reinforcement learning (RL) framework that leverages self-evolution through attention maps. The core idea is to iteratively refine the model by using its own attention mechanisms as supervisory signals, guiding the localization process more precisely. This approach incorporates three key strategies: seed data filtering to ensure high-quality training samples, a dense point reward mechanism to provide continuous feedback, and a self-evolutionary fine-tuning loop that progressively improves the model's focus on relevant interface elements.
The experimental results are compelling. Using only 3,018 high-quality samples, a 7B-parameter model achieves state-of-the-art performance across three benchmarks. Notably, it reaches 47.3% accuracy on the challenging ScreenSpot-Pro dataset, surpassing much larger models like UI-TARS-72B by 24.2%. The model demonstrates excellent generalization across desktop, mobile, and web environments, with accuracy rates exceeding 88%. Ablation studies confirm that data quality and resolution are critical factors, emphasizing the importance of noise filtering and visual detail.
This work signifies a major step forward in GUI understanding, showing that reinforcement learning combined with self-supervised attention can effectively address the challenges of sparse rewards and limited data. Its implications extend to industrial automation, assistive technologies, and cross-platform automation, where data efficiency and robustness are paramount. Future directions include multi-modal fusion, real-time adaptation, and scaling to more dynamic interfaces, promising a broad impact on intelligent visual systems.
Deep Dive
Abstract
Graphical User Interface (GUI) agents have made substantial strides in understanding and executing user instructions across diverse platforms. Yet, grounding these instructions to precise interface elements remains challenging, especially in complex, high-resolution, professional environments. Traditional supervised finetuning (SFT) methods often require large volumes of diverse data and exhibit weak generalization. To overcome these limitations, we introduce a reinforcement learning (RL) based framework that incorporates three core strategies: (1) seed data curation to ensure high quality training samples, (2) a dense policy gradient that provides continuous feedback based on prediction accuracy, and (3) a self evolutionary reinforcement finetuning mechanism that iteratively refines the model using attention maps. With only 3k training samples, our 7B-parameter model achieves state-of-the-art results among similarly sized models on three grounding benchmarks. Notably, it attains 47.3\% accuracy on the ScreenSpot-Pro dataset, outperforming much larger models, such as UI-TARS-72B, by a margin of 24.2\%. These findings underscore the effectiveness of RL-based approaches in enhancing GUI agent performance, particularly in high-resolution, complex environments.