Trust the Right Teacher: Quality-Aware Self-Distillation for GUI Grounding
Proposes quality-aware self-distillation with correctness gating and confidence scaling, boosting GUI coordinate accuracy by 2.16%.
Key Findings
Methodology
This paper introduces a quality-aware self-distillation framework combining soft correctness gating and teacher probability scaling. It verifies teacher predictions against ground-truth bounding boxes to assess spatial correctness, adjusting signal strength accordingly. Teacher confidence is used to calibrate supervision weights, all integrated into a weighted reverse-KL loss. Experiments across six GUI benchmarks demonstrate significant performance gains, with the combined mechanism outperforming single components.
Key Results
- Achieved 72.23% macro-average accuracy across six benchmarks, surpassing the baseline GUI-SD by 2.16%. The ablation shows that combining correctness gating and probability scaling yields the best results, confirming their complementary roles. The method improves coordinate prediction accuracy by 4-5% in complex scenes like UIEG and MMG, validating robustness.
- Parameter tuning identified optimal λ=3 and α=0.5, balancing supervision strength and reliability. The approach outperforms prior methods like SFT and GRPO, leveraging dense token-level supervision and soft teacher signals.
- The experimental results highlight the importance of signal quality control, with the proposed method reducing unreliable teacher guidance and emphasizing high-confidence signals, leading to more accurate GUI localization.
Significance
This work addresses the critical challenge of teacher signal reliability in self-distillation for GUI grounding. By directly verifying spatial consistency, it mitigates the risk of misleading supervision, enabling models to learn more precise spatial representations. The approach advances the state-of-the-art in vision-language models, with broad implications for automated interface understanding, intelligent assistants, and human-computer interaction. Its robustness across diverse datasets demonstrates practical potential for real-world deployment, reducing reliance on manual annotations and improving accuracy in complex scenarios.
Technical Contribution
The core technical innovation lies in integrating spatial verification into teacher signal quality assessment, combining soft correctness gating with confidence-based scaling. This allows dynamic filtering of teacher predictions based on their spatial plausibility and certainty, improving supervision quality. The design of a weighted reverse-KL loss that incorporates these mechanisms ensures effective suppression of unreliable signals while preserving useful knowledge. This approach offers a principled way to enhance teacher-student training in autoregressive, coordinate-sensitive tasks.
Novelty
This is the first work to incorporate direct spatial verification as a teacher signal quality measure in GUI grounding. Unlike prior methods relying solely on indirect proxies like entropy or probability, this approach uses the ground-truth bounding box to verify teacher predictions, enabling precise, context-aware filtering. The combination of soft gating and confidence scaling introduces a novel, effective mechanism for improving teacher signal reliability in autoregressive models, setting a new benchmark in fine-grained spatial localization.
Limitations
- The method depends on accurate ground-truth bounding boxes; errors in annotations can reduce the effectiveness of spatial verification. In highly cluttered or occluded scenes, the verification may still pass unreliable signals. Additional computational overhead from gating and scaling may limit real-time applications. Future work should explore adaptive parameter tuning and multi-modal verification to address these issues.
Future Work
Future directions include integrating multi-modal cues (e.g., visual context, textual cues) for more robust spatial verification, developing adaptive gating strategies learned end-to-end, and extending the framework to dynamic, real-time interface environments. Exploring reinforcement learning to optimize signal filtering policies could further enhance model robustness and generalization.
AI Executive Summary
GUI grounding is a fundamental task in vision-language modeling, requiring precise localization of interface elements from high-resolution screenshots. Traditional supervised methods rely heavily on annotated coordinates, limiting their ability to leverage rich spatial information. Self-distillation offers a promising alternative by providing dense, token-level teacher signals derived from the model itself, but its effectiveness hinges on the reliability of these signals.
This paper introduces a novel quality-aware self-distillation framework that addresses the core challenge of teacher signal reliability. The key innovation involves a soft correctness-aware gating mechanism that verifies whether the teacher’s current coordinate prediction remains within the true bounding box, dynamically down-weighting unreliable signals. Complementing this, teacher probability scaling uses the teacher’s confidence score to further calibrate supervision strength, ensuring that high-confidence signals are emphasized while uncertain ones are softened.
The combined approach effectively filters out misleading teacher guidance, leading to more accurate coordinate predictions. Extensive experiments across six GUI benchmarks, including SSP, ScreenSpot-v2, UIEG, and MMG, demonstrate consistent improvements, with the average accuracy reaching 72.23%, outperforming strong baselines such as GUI-SD by over 2 points. Ablation studies confirm that the synergy of gating and scaling is essential for optimal performance.
This work significantly advances the state-of-the-art in GUI grounding, providing a robust framework for reliable teacher signal utilization. Its implications extend to broader vision-language tasks requiring fine-grained spatial understanding, such as robotic manipulation, augmented reality, and intelligent interface design. Future research may explore multi-modal verification, adaptive parameter learning, and real-time deployment, further broadening its impact.
Deep Analysis
Background
GUI定位作为视觉-语言模型中的关键任务，近年来随着深度学习的发展取得了显著进步。早期方法主要依赖于监督学习，使用标注的坐标信息进行训练，如SeeClick、CogA-gent等。随着模型复杂度提升，研究者开始引入强化学习和自蒸馏技术，旨在利用模型自身的预测信息增强训练效果。自蒸馏尤其受到关注，因为它能提供丰富的令牌级监督信号，有助于提升空间理解能力。然而，教师信号的可靠性成为制约性能提升的瓶颈，尤其在复杂场景中，偏离目标的预测会误导模型学习。
Core Problem
核心问题在于教师信号的质量控制。自回归生成过程中，偏离目标的预测会导致教师提供误导性信息，影响模型训练效果。传统自蒸馏未考虑空间验证，容易引入噪声，限制了模型在复杂GUI场景中的表现。如何有效筛选、校准教师信号，确保其空间合理性，成为提升定位精度的关键瓶颈。
Innovation
本研究提出空间验证的软正确性门控机制，结合教师置信度进行动态调节，首次将空间一致性作为教师信号质量的直接指标。具体创新包括：• 利用目标边界框验证教师预测的空间合理性；• 设计软门控策略，避免信号硬性舍弃；• 结合教师概率进行细粒度校准，提升信号质量。这些创新显著改善了教师信号的可靠性，推动GUI定位技术向更高精度发展。
Methodology
- �� 输入：用户指令、截图、目标边界框。• 构建：在训练中引入布局保持区域遮挡的教师专用输入。• 预测：学生生成响应，教师基于空间验证评估预测合理性。• 门控：判断教师预测是否仍在目标边界内，软调节信号强度。• 置信度：利用教师预测概率进行信号校准。• 损失：采用加权反向KL散度，动态调节令牌的蒸馏权重。• 训练：端到端优化，结合空间验证和置信度校准，提升模型鲁棒性。
Experiments
在六个GUI基准（如SSP、UIEG、MMG等）上，采用Qwen-3.5-9B作为基础模型，比较不同方法的性能。设置不同门控强度和缩放系数，进行消融分析。指标包括宏平均准确率，验证信号筛选效果。通过多轮调优，确定最优参数λ=3和α=0.5，确保性能最优。
Results
实验显示，结合空间验证的质量感知机制显著优于传统自蒸馏和基线方法，平均准确率提升至72.23%。 Ablation验证了门控和概率缩放的互补性，单独使用效果有限。模型在复杂场景中的鲁棒性增强，验证了信号筛选策略的有效性。参数调优确保了模型在多场景中的适应性和稳定性。
Applications
该方法适用于自动界面识别、智能助手、自动化测试等场景，能显著提升目标元素定位的准确性。依赖高质量边界框和空间验证机制，适合多模态交互系统部署，推动人机交互智能化。未来可结合多模态信息，拓展到动态变化的界面环境，提升适应性。
Limitations & Outlook
依赖边界框的准确性，误差会影响验证效果。在极端复杂或遮挡场景中，空间验证可能不足以排除所有误导信号。引入的门控和缩放机制增加计算成本，限制在资源有限环境中的应用。未来需优化验证机制和算法效率，增强鲁棒性。
Plain Language Accessible to non-experts
想象你在厨房做饭，每次放调料都要尝试一下味道是否合适。老师的信号就像调料的味道，有时候太咸或太淡，会误导你做菜。为了做出好菜，你会用味觉判断调料是否合适，还会根据经验调整用量。这篇论文就像是给厨师设计了一套聪明的味觉检测系统，能判断调料是否放得恰当，并根据味道的自信程度调整用量。这样，厨师就能做出更美味的菜肴，而不用担心调料放错了。
ELI14 Explained like you're 14
想象你在玩一个游戏，要找到屏幕上的宝藏位置。游戏里的助手会告诉你宝藏在哪，但有时候它的提示不准，可能会误导你。为了变得更聪明，你可以用一些聪明的方法来判断助手的提示是否靠谱，比如：它说的范围是不是符合地图上的标记？如果它说的范围太模糊或不在正确区域，你会少听它的话。这个论文就像是给助手装了一个聪明的“眼睛”，可以判断它的提示是否靠谱，然后只听那些“眼睛”确认的提示。这样，你就能更快找到宝藏，避免被误导。
Glossary
Self-Distillation (自蒸馏)
一种模型训练方法，模型用自己生成的预测作为教师信号，提升自身性能。
论文中用来提供丰富的令牌级监督信息。
Teacher Probability (教师概率)
教师模型对某个预测的置信度，反映预测的确定性。
用作信号校准的重要指标。
Soft Correctness-Aware Gating (软正确性门控)
根据空间验证结果动态调节教师信号的强度，避免误导。
核心机制之一，用于筛选可靠的教师预测。
Spatial Verification (空间验证)
利用目标边界框判断教师预测是否在目标区域内，是信号可靠性的重要依据。
论文的关键技术之一。
Reverse KL Divergence (反向KL散度)
衡量两个概率分布差异的指标，用于模型输出的对齐。
作为训练中的损失函数，调节学生与教师的预测差异。
Open Questions Unanswered questions from this research
- 1 如何在极端复杂场景中进一步提升空间验证的鲁棒性？
- 2 能否结合多模态信息增强信号筛选？
- 3 在动态变化界面中验证机制如何适应？
Applications
Immediate Applications
自动界面元素定位
提升界面元素识别的准确性，减少人工标注需求，适用于自动化测试和界面分析。
智能交互助手
增强虚拟助手对界面元素的理解，实现更自然的操作和反馈，提升用户体验。
Long-term Vision
全面自动化界面理解
实现跨平台、多场景的界面自动识别与交互，推动智能系统自主操作，减少人工干预。
Abstract
Graphical user interface (GUI) grounding requires vision-language models (VLMs) to identify small target elements in high-resolution screenshots and predict precise screen coordinates. On-policy self-distillation (OPSD) is a promising post-training approach for this coordinate-sensitive task, since it provides dense token-level teacher signals beyond hard coordinate labels. However, naive OPSD is not well suited to GUI grounding: OPSD evaluates the teacher on student-generated prefixes, the quality of coordinate-token teacher signals can degrade when the prefix has already deviated from the target coordinate, leading to unreliable teacher signal. To mitigate this, We propose quality-aware self-distillation for VLM-based GUI grounding, which improves coordinate-token teacher-signal quality through soft correctness-aware gating and teacher-probability scaling. The soft correctness-aware gate checks whether the teacher's current coordinate-token prediction can still be completed into the ground-truth box under the student-generated prefix. If not, the corresponding teacher signal is down-weighted. Teacher-probability scaling then uses the teacher's confidence as a lightweight factor to further calibrate the strength of the gated supervision. A key empirical finding is that neither component alone improves overall performance, whereas combining them consistently improves performance. This suggests that the two mechanisms play complementary roles: correctness-aware gating suppresses unreliable coordinate-token supervision, while teacher-probability scaling calibrates the strength of the remaining signals. Experiments across six GUI grounding benchmarks show that our method consistently improves the base model and outperforms strong baselines.