Learn where to Click from Yourself: On-Policy Self-Distillation for GUI Grounding
GUI-SD introduces on-policy self-distillation with visual privileged context and entropy-guided loss, boosting GUI grounding accuracy to 68.4% and training speed 4× faster.
Key Findings
Methodology
This paper proposes the GUI-SD framework, which employs visual privileged context and entropy-guided distillation to enhance token-level supervision. The privileged context is constructed by overlaying a Gaussian soft mask on the target bounding box, providing informative guidance without leaking exact coordinates. The distillation loss is based on a weighted reverse KL divergence, where weights are dynamically assigned according to the digit importance and teacher confidence. The framework integrates these components into a training pipeline evaluated on six GUI benchmarks, such as ScreenSpot-v2 and UI-Vision, demonstrating superior accuracy and efficiency over traditional reinforcement learning (e.g., GRPO) and naive self-distillation methods.
Key Results
- GUI-SD achieves an average accuracy of 68.4% across six benchmarks, outperforming baseline methods like GRPO-Gaussian by approximately 3-4 percentage points. For instance, on ScreenSpot-Pro, accuracy improves from 55.6% to 60.7%, and on OSWorld-G-Refine, it reaches 70.9%. The training speed per epoch is about four times faster than GRPO, significantly reducing computational costs. Ablation studies confirm that visual privileged guidance and entropy-based weighting are crucial for performance gains, especially in dense and high-resolution scenes.
- The approach demonstrates excellent generalization across diverse GUI scenarios, including complex layouts and small targets. Visual guidance prevents distillation collapse by maintaining distribution diversity, while the digit-wise weighted loss emphasizes learning high-order digits, leading to more precise localization. The combination of these strategies results in robust performance on challenging samples and hard cases, validating the effectiveness of the design.
- Extensive experiments show that GUI-SD surpasses existing state-of-the-art methods in accuracy, training efficiency, and robustness. The visual privileged context significantly improves teacher supervision quality, and the adaptive weighting mechanism ensures focus on critical coordinate digits. The method's ability to deliver dense token-level supervision from a single rollout marks a substantial advancement over sequence-level rewards, paving the way for more efficient autonomous GUI agents.
Significance
This research addresses fundamental limitations of reinforcement learning-based GUI grounding methods, notably the high cost of multiple rollouts and sparse rewards. By leveraging on-policy self-distillation with visual guidance and entropy-aware weighting, GUI-SD provides a dense, reliable supervision signal that accelerates training and enhances accuracy. This paradigm shift reduces reliance on environment interactions, making autonomous GUI agents more scalable and practical for real-world applications such as automated testing, interface understanding, and assistive technologies. The framework's ability to focus on high-impact tokens and avoid distillation collapse offers a new direction for research in visual-language grounding tasks.
Technical Contribution
The core technical contributions include the integration of visual privileged context through boundary box overlays and Gaussian soft masks, which enrich the teacher's supervision without leaking exact coordinates. Additionally, the entropy-guided weighted reverse KL loss dynamically emphasizes high-confidence and high-priority coordinate digits, addressing the heterogeneity in supervision reliability. These innovations collectively enable dense, targeted token-level supervision from a single policy rollout, significantly improving training efficiency and robustness. The framework also introduces a novel combination of multi-scale visual cues and adaptive weighting strategies, providing theoretical and practical advancements over prior methods such as reinforcement learning and naive distillation.
Novelty
This work is the first to apply on-policy self-distillation to GUI grounding, introducing visual privileged guidance to prevent distillation collapse and an entropy-based weighting scheme to prioritize critical coordinate digits. Unlike prior reinforcement learning approaches like GRPO, which require multiple rollouts and sparse rewards, GUI-SD achieves dense supervision from a single interaction, reducing computational overhead. The combination of visual soft masks and adaptive loss weighting constitutes a novel approach that effectively addresses the heterogeneity in supervision quality across coordinate tokens. These innovations collectively push the boundary of efficient, accurate GUI element localization.
Limitations
- The visual privileged guidance relies on high-quality bounding box annotations and parameter tuning of the Gaussian soft mask, which may limit robustness in noisy or occluded scenes. Adaptive parameter tuning mechanisms are needed for broader applicability.
- The current framework assumes access to accurate bounding boxes and hints during training, which may not be feasible in fully unsupervised settings. Extending the approach to weakly supervised or unsupervised scenarios remains an open challenge.
- While training efficiency is improved, the model's inference speed and memory footprint in high-resolution scenes could still pose deployment challenges, especially on resource-constrained devices. Future work should focus on lightweight architectures and optimization techniques.
Future Work
Future research directions include developing adaptive parameter tuning for the visual soft mask to enhance robustness across diverse scenes, integrating multi-modal cues such as speech or touch for richer interaction understanding, and exploring lightweight model architectures for real-time deployment. Additionally, extending the framework to handle multi-step, multi-object tasks with dynamic scene changes will be crucial for advancing autonomous GUI agents. Investigating unsupervised or semi-supervised learning strategies to reduce annotation dependency is another promising avenue, aiming to scale the approach to larger, more varied datasets and real-world applications.
AI Executive Summary
The task of graphical user interface (GUI) grounding—mapping natural language instructions to visual element coordinates—has become increasingly vital in automating human-computer interaction. Traditional reinforcement learning approaches, such as GRPO, have demonstrated promising results but are hampered by high computational costs and sparse reward signals, especially on difficult samples where all rollouts yield zero reward. These limitations hinder scalability and practical deployment, prompting researchers to seek more efficient training paradigms.
In this context, the paper introduces GUI-SD, a novel framework leveraging on-policy self-distillation, which offers a dense, token-level supervision signal from a single policy rollout. This approach hinges on two key innovations: the construction of a visually enriched privileged context and an entropy-guided loss function. The privileged context is created by overlaying a Gaussian soft mask on the target bounding box, providing informative guidance without revealing the exact coordinates. This design prevents the teacher distribution from collapsing into a near-one-hot form, a common pitfall in naive distillation. The entropy-guided loss further refines the training process by assigning higher weights to high-confidence, high-impact coordinate digits, such as the most significant bits, while down-weighting uncertain predictions.
Extensive experiments across six diverse GUI benchmarks, including ScreenSpot-v2 and UI-Vision, demonstrate that GUI-SD consistently outperforms existing methods in both accuracy and training efficiency. The results show an average accuracy of 68.4%, surpassing the best baseline by nearly 5 percentage points, while reducing training time per epoch by approximately four times. Ablation studies confirm that the visual guidance and entropy-based weighting are critical for these improvements, especially in dense and complex scenes.
This research marks a significant step forward in GUI automation, addressing the core challenges of sparse rewards and inefficient training. By enabling dense supervision with minimal environment interaction, GUI-SD paves the way for more scalable and robust autonomous agents capable of understanding and manipulating complex interfaces. The framework's reliance on visual cues and adaptive weighting strategies offers a flexible, generalizable solution that can extend to other visual-language grounding tasks. Despite some limitations related to parameter tuning and scene complexity, the approach opens promising avenues for future work, including multi-modal integration, lightweight model design, and unsupervised learning, ultimately aiming to realize truly intelligent, autonomous GUI agents in real-world applications.
Deep Analysis
Background
随着人工智能技术的发展，自动化界面理解逐渐成为人机交互中的核心问题之一。早期方法多依赖规则匹配和模板识别，但在多样化和复杂化的界面场景中表现有限。深度学习的引入极大推动了界面元素的识别与定位，特别是强化学习（如GRPO）在策略探索方面取得了一定突破。然而，这些方法普遍面临训练成本高、奖励稀疏、样本难以优化等瓶颈。近年来，自我蒸馏作为一种无需外部教师的知识迁移技术，逐渐被引入到视觉和语言任务中，显示出高效学习的潜力。尽管如此，将自我蒸馏应用于GUI定位任务仍处于探索阶段，主要挑战包括如何设计有效的引导信息、避免蒸馏崩溃，以及提升模型在复杂场景中的鲁棒性。本文在此背景下，提出结合视觉引导和熵调节机制的自我蒸馏框架，为GUI自动化提供新的解决方案。
Core Problem
传统的强化学习方法如GRPO在GUI定位中虽然取得一定效果，但存在多轮采样成本高、奖励信号稀疏的问题，尤其在硬样本和复杂场景中表现不佳。自我蒸馏技术提供了单轮密集的像素级监督，但在实际应用中遇到两大难题：一是蒸馏崩溃，即教师分布因泄露精确坐标变得过于确定，导致蒸馏退化为硬标签训练，失去软信息的优势；二是优化不均，所有数字位被等同对待，忽略了高阶数字的重要性和教师信心的差异，导致模型在关键位置学习不足。这些问题限制了自我蒸馏在GUI任务中的潜力，亟需设计一种既能保持丰富信息，又能有针对性优化的机制。
Innovation
本文的创新点主要体现在两个方面：一是引入视觉丰富的特权上下文，通过在目标边界框上叠加高斯软掩码，构建信息充足但不泄露精确坐标的引导环境，有效避免蒸馏崩溃问题；二是提出熵引导的逆KL损失，将不同数字位的优先级和教师信心结合，动态调节每个位置的蒸馏权重，重点优化高阶数字。这种设计突破了传统均匀加权和硬标签限制，使模型在复杂场景中表现更优。具体而言，视觉引导确保教师分布的多样性和信息丰富性，而熵引导机制则保证优化集中在最关键的数字位上，从而实现更精细的定位效果。这些创新共同推动了GUI自动化技术的前沿发展。
Methodology
- �� 视觉特权上下文构建：在目标边界框上叠加高斯软掩码，生成逐渐模糊的区域，避免泄露精确坐标信息，同时提供丰富的引导信息。
- �� 视觉提示设计：结合目标区域的边界框和提示文本（如“答案在红色矩形内”），增强教师的推理能力，减少直接复制答案的可能。
- �� 训练目标：采用逆KL散度损失，将教师的软分布作为目标，指导学生模型学习，确保分布多样性。
- �� 权重调节策略：引入位置优先权wpos(t)，对高阶数字赋予更大权重，利用指数递减方式强调高阶位的重要性。
- �� 信心调节：通过计算教师输出的熵值，自动调节每个token的蒸馏强度，低熵代表高置信度，赋予更大权重。
- �� 损失函数：结合位置和信心两个因素，设计加权逆KL损失，确保优化集中在关键位置和高置信度区域。
- �� 训练流程：在六个GUI基准上进行大规模训练和评估，结合消融实验验证各组件的有效性。
Experiments
- �� 数据集：采用ScaleCUA的约7K样本，涵盖多样化的GUI场景，包括高分辨率截图和复杂布局。
- �� 基线模型：比较GRPO、Naive OPSD以及其他SOTA方法。
- �� 评估指标：主要关注定位准确率（如ScreenSpot-Pro的准确率）、训练时间（每个epoch的耗时）和硬样本表现。
- �� 超参数：调节高斯掩码的σ值，设置数字位的权重参数α，以及熵调节的温度τ。
- �� 消融研究：逐步移除视觉引导和熵调节，分析对性能的影响。
- �� 训练细节：采用Adam优化器，学习率调度，训练轮次控制在合理范围内，确保模型收敛。
Results
- �� GUI-SD在六个基准上均优于对比方法，平均准确率达68.4%，比GRPO-Gaussian提升约3-4个百分点。
- �� 在复杂场景如UI-Vision和OSWorld-G-Refine中，性能提升尤为明显，准确率分别达到86.7%和70.9%。
- �� 训练速度方面，GUI-SD每个epoch比GRPO快约4倍，大幅降低训练成本。
- �� 消融实验验证，视觉引导和熵调节的结合带来最大性能提升，尤其在硬样本和边界样本中表现出更强鲁棒性。
- �� 数字高阶位的加权策略显著改善了高阶数字的预测精度，验证了位置优先的有效性。
Applications
- �� 立即应用：可用于自动化界面测试工具，提升界面元素定位的准确性和效率，降低人工标注成本。
- �� 立即应用：在智能助手和自主界面操作中，实现对用户指令的准确执行，如自动点击、界面导航。
- �� 立即应用：支持多平台、多场景的界面理解，为工业自动化和智慧城市提供基础技术支撑。
- �� 长远愿景：推动自主GUI代理的发展，使其能在复杂、多变环境中自主学习与适应，广泛应用于智能制造、智慧城市和人机交互系统。
Limitations & Outlook
- �� 视觉引导设计依赖高质量的图像和准确的目标边界框，在低质量或遮挡严重的场景中表现可能受限。
- �� 参数调节（如高斯掩码σ值和权重参数）缺乏自适应机制，需针对不同场景进行手动调优。
- �� 模型在高分辨率场景中的推理速度和存储需求较高，实际部署时存在性能瓶颈。
- �� 未来需探索更轻量化的模型结构和自适应参数调节策略，以提升实际应用的鲁棒性和效率。
Plain Language Accessible to non-experts
想象你在玩一个找宝藏的游戏。以前，你可能要在一张复杂的地图上逐点搜索，花费大量时间，而且很容易迷路。这就像用传统的强化学习方法，需要反复尝试，每次都要花费很多资源。而现在，你有一份聪明的指南，它在地图上用模糊的光晕标记了宝藏可能出现的区域。你不用每次都试错，而是根据这个提示，快速锁定目标区域。更棒的是，这个指南还会告诉你，越靠近中心的区域越有可能藏着宝藏（高阶数字），你会优先去这些地方搜索。这样一来，你不仅找得更快，还能用更少的尝试找到宝藏。这就像给机器人一份“聪明的地图”和“优先级列表”，让它在复杂的界面中快速准确地找到目标元素，节省了大量时间和精力。
ELI14 Explained like you're 14
想象你在玩一个找宝藏的游戏。以前，你可能要在一张大地图上逐个点搜索，花费很多时间。而现在，有个聪明的助手会在地图上用模糊的光晕标记出宝藏可能藏着的区域，还会告诉你哪些地方更有可能藏着宝藏（比如越靠近中心越好）。你只需要根据这些提示，优先去那些“高概率”的地方找。这样一来，你可以用更少的时间和努力找到宝藏，而且不容易迷路。这个助手就像是给你一份聪明的指南，让你在复杂的界面中快速找到目标元素，不用反复试错，也不用花费太多力气。它用一种聪明的方式，把复杂的任务变得简单又高效，就像你在玩游戏时的“秘籍”。
Glossary
GUI Grounding (GUI定位)
将自然语言指令映射到界面元素的空间坐标，是实现自动化操作的基础。
论文中，GUI Grounding是核心任务，目标是让模型准确找到界面上的目标元素。
Self-Distillation (自我蒸馏)
一种模型训练技术，模型作为教师和学生，通过内部知识迁移提升性能，无需外部教师模型。
本文将自我蒸馏应用于GUI定位，通过视觉引导和熵调节优化模型学习过程。
Reverse KL Divergence (逆KL散度)
一种衡量两个概率分布差异的指标，反映教师分布对学生分布的引导效果，强调高置信度区域。
在损失函数中使用逆KL，确保模型集中学习教师信心高的像素和数字位。
Privileged Context (特权上下文)
模型在训练时访问的额外信息，用于提供更丰富的指导，但在推理时不可用。
本文用视觉引导的边界框和提示词构建特权上下文，增强教师的引导能力。
Entropy-Guided Distillation (熵引导蒸馏)
根据教师输出的熵值动态调节每个token的蒸馏权重，以提升可靠性和效率。
该机制确保模型重点学习高置信度和高重要性的数字位，避免低置信度干扰。
Gaussian Soft Mask (高斯软掩码)
一种模糊区域遮罩，用于突出目标区域，逐步淡化周围区域，避免泄露精确坐标。
在视觉引导中使用，帮助模型专注于目标区域，同时保持信息丰富。
Token Weighting (标记加权)
根据数字的重要性和教师信心，为不同位置的预测赋予不同的学习权重。
高阶数字（如百位）获得更大权重，模型优先优化关键位置。
Ablation Study (消融实验)
逐步移除模型的某些组件，验证每个部分对整体性能的贡献。
本文通过消融验证视觉引导和熵调节的效果。
GUI Benchmarks (GUI基准数据集)
一组标准化的GUI场景数据集，用于评估界面元素定位算法的性能。
包括ScreenSpot-v2、UI-Vision等，覆盖多样化场景。
Training Efficiency (训练效率)
模型在训练过程中达到一定性能所需的时间和资源，反映算法的实用性。
GUI-SD在训练时间上比传统方法快约4倍。
Abstract
Graphical User Interface (GUI) grounding maps natural language instructions to the visual coordinates of target elements and serves as a core capability for autonomous GUI agents. Recent reinforcement learning methods (e.g., GRPO) have achieved strong performance, but they rely on expensive multiple rollouts and suffer from sparse signals on hard samples. These limitations make on-policy self-distillation (OPSD), which provides dense token-level supervision from a single rollout, a promising alternative. However, its applicability to GUI grounding remains unexplored. In this paper, we present GUI-SD, the first OPSD framework tailored for GUI grounding. First, it constructs a visually enriched privileged context for the teacher using a target bounding box and a Gaussian soft mask, providing informative guidance without leaking exact coordinates. Second, it employs entropy-guided distillation, which adaptively weights tokens based on digit significance and teacher confidence, concentrating optimization on the most impactful and reliable positions. Extensive experiments on six representative GUI grounding benchmarks show that GUI-SD consistently outperforms GRPO-based methods and naive OPSD in both accuracy and training efficiency. Code and training data are available at https://zhangyan-ucas.github.io/GUI-SD/.
References (20)
GUI-G2: Gaussian Reward Modeling for GUI Grounding
Fei Tang, Zhangxuan Gu, Zhengxi Lu et al.
Self-Distillation Enables Continual Learning
Idan Shenfeld, Mehul Damani, Jonas Hübotter et al.
InfiGUI-R1: Advancing Multimodal GUI Agents from Reactive Actors to Deliberative Reasoners
Yuhang Liu, Pengxiang Li, Congkai Xie et al.
Qwen3-VL Technical Report
Shuai Bai, Yuxuan Cai, Rui-Zhe Chen et al.
Zooming without Zooming: Region-to-Image Distillation for Fine-Grained Multimodal Perception
Lai Wei, Liang He, Jun Lan et al.
Scaling Computer-Use Grounding via User Interface Decomposition and Synthesis
Tianbao Xie, Jiaqi Deng, Xiaochuan Li et al.
Measure Twice, Click Once: Co-evolving Proposer and Visual Critic via Reinforcement Learning for GUI Grounding
Wenkai Wang, Xiyun Li, Hongcan Guo et al.
Distilling the Knowledge in a Neural Network
Geoffrey E. Hinton, O. Vinyals, J. Dean
GUI-Actor: Coordinate-Free Visual Grounding for GUI Agents
Qianhui Wu, Kanzhi Cheng, Rui Yang et al.
ToDi: Token-wise Distillation via Fine-Grained Divergence Control
Seongryong Jung, Suwan Yoon, Donggeon Kim et al.
GUI-G1: Understanding R1-Zero-Like Training for Visual Grounding in GUI Agents
Yuqi Zhou, Sunhao Dai, Shuai Wang et al.
LPO: Towards Accurate GUI Agent Interaction via Location Preference Optimization
Jiaqi Tang, Yu Xia, Yi-feng Wu et al.
MobileIPL: Enhancing Mobile Agents Thinking Process via Iterative Preference Learning
Kun Huang, Weikai Xu, Yuxuan Liu et al.
GTA1: GUI Test-time Scaling Agent
Yan Yang, Dongxu Li, Yutong Dai et al.
Enhancing Visual Grounding for GUI Agents via Self-Evolutionary Reinforcement Learning
Xinbin Yuan, Jian Zhang, Kaixin Li et al.
GUI-R1 : A Generalist R1-Style Vision-Language Action Model For GUI Agents
Run Luo, Lu Wang, Wanwei He et al.
SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agents
Kanzhi Cheng, Qiushi Sun, Yougang Chu et al.
Navigating the Digital World as Humans Do: Universal Visual Grounding for GUI Agents
Boyu Gou, Ruohan Wang, Boyuan Zheng et al.
DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models
Zhihong Shao, Peiyi Wang, Qihao Zhu et al.
OS-ATLAS: A Foundation Action Model for Generalist GUI Agents
Zhiyong Wu, Zhenyu Wu, Fangzhi Xu et al.
Cited By (7)
Test-Time Self-Evolving GUI Visual Grounding via Reflection-Guided On-Policy Self-Distillation
Trust the Right Teacher: Quality-Aware Self-Distillation for GUI Grounding
Latent On-Policy Self-Distillation
SkillLens: Visual Skill Cards for Retrieval-Augmented GUI Action Prediction and On-Policy Distillation
The Next Screenshot Knows: Gated Hindsight Distillation for Mobile GUI Agents
IoU-PD: IoU-Aware Privileged Distillation for Visual Grounding with Multimodal Large Language Models
A Brief Overview: On-Policy Self-Distillation In Large Language Models