GUI-Perturbed: Domain Randomization Reveals Systematic Brittleness in GUI Grounding Models
GUI-Perturbed employs domain randomization to reveal systematic weaknesses in GUI grounding models, especially in spatial reasoning and robustness, with accuracy drops of 27-56%.
Key Findings
Methodology
This study introduces GUI-Perturbed, a framework that applies independent domain randomization to GUI scenes and instructions using Mind2Web MHTML archives and Playwright. It perturbs visual styles, zoom levels, and instruction types (direct vs. relational) along multiple axes, enabling detailed analysis of model robustness. The evaluation involves three 7B models from the same architecture lineage, assessing their performance under various controlled perturbations. The framework isolates specific capability axes—spatial reasoning, visual robustness, reasoning calibration—by measuring accuracy drops and prediction consistency. Additionally, the study explores the effects of LoRA fine-tuning and data augmentation, providing insights into training limitations.
Key Results
- All models exhibit significant accuracy degradation (27-56 percentage points) on relational instructions, highlighting a fundamental deficiency in spatial reasoning. A 70% browser zoom causes a statistically significant performance drop (~4-8 points), indicating reliance on absolute position encoding. Data augmentation and LoRA fine-tuning, contrary to expectations, worsen robustness, suggesting current training strategies are insufficient. Models show relative resilience to style changes but are highly sensitive to spatial and zoom perturbations. The framework effectively isolates the specific axes of performance degradation, offering a granular diagnostic tool.
- Models trained on millions of screenshots still struggle with relational instructions, with accuracy dropping sharply under perturbations. Chain-of-thought reasoning partially mitigates spatial reasoning gaps but does not fully resolve them. The results reveal a systemic over-reliance on memorized absolute positions rather than relational understanding, emphasizing the need for models to learn more flexible spatial representations. The experiments demonstrate that current benchmarks fail to expose these weaknesses, underscoring the importance of domain-randomized evaluation.
- The study confirms that visual and spatial robustness are critical for real-world deployment. The findings suggest that improving spatial reasoning and environment adaptability requires new training paradigms beyond standard supervised fine-tuning. The open-source dataset, augmentation pipeline, and fine-tuned models facilitate further research, encouraging community efforts to develop more resilient GUI understanding models.
Significance
This research exposes the systematic vulnerabilities of state-of-the-art GUI grounding models, especially in spatial reasoning and environmental robustness. By employing domain randomization along multiple axes, it provides a detailed diagnostic framework that surpasses traditional benchmarks. The insights gained highlight the gap between benchmark performance and real-world reliability, emphasizing the need for models that understand spatial relationships and adapt to environmental changes. These findings have broad implications for deploying GUI agents in dynamic settings like web automation, enterprise workflows, and assistive technologies. The open-source tools and datasets foster community-driven improvements, aiming to develop models with genuine robustness and generalization capabilities, thus advancing the field toward more practical and trustworthy AI systems.
Technical Contribution
The paper introduces GUI-Perturbed, a novel evaluation framework that leverages multi-axis domain randomization at the DOM level to systematically diagnose GUI grounding model weaknesses. Unlike traditional static benchmarks, it independently perturbs visual styles, zoom levels, and instruction types, enabling precise attribution of performance drops to specific capability axes. The framework integrates an open-source augmentation pipeline (GUI-DR) that generates diverse perturbation variants from MHTML archives, facilitating large-scale robustness testing. Experiments with three 7B models demonstrate that current training strategies, including LoRA fine-tuning and data augmentation, do not sufficiently improve spatial reasoning or visual robustness. The approach provides a new diagnostic paradigm that guides future model training and architecture design, emphasizing the importance of learning invariant spatial representations.
Novelty
This work is the first to systematically apply multi-axis domain randomization at the DOM level for GUI grounding evaluation, revealing fundamental weaknesses in spatial reasoning and visual robustness that are hidden in standard benchmarks. The independent perturbation of visual styles, zoom, and instruction types allows for precise attribution of failure modes, a significant advancement over prior static or uncontrolled scene variation methods. The framework’s ability to isolate specific capability axes offers a new diagnostic tool for the community, enabling targeted improvements in model design and training. This approach bridges the gap between benchmark performance and real-world robustness, marking a substantial step forward in GUI understanding research.
Limitations
- 目前仅在Web环境中验证，尚未扩展到移动端或桌面应用，未来需考虑多平台适应性；扰动机制虽能揭示性能退化，但未提供直接的模型改进方案，仍需结合训练策略优化；极端扰动条件下模型性能仍显著下降，说明鲁棒性提升仍面临挑战。
Future Work
未来将结合强化学习和自监督学习，增强模型在多轴扰动环境中的适应能力。计划扩展GUI-Perturbed框架到移动端和桌面场景，丰富扰动类型，提升模型在实际环境中的鲁棒性。同时，结合模型解释技术，深入理解空间推理机制，推动GUI理解模型的稳健性和泛化能力，最终实现更具环境适应性的智能GUI代理。
AI Executive Summary
Deep Dive
Plain Language Accessible to non-experts
想象你在一家工厂工作，工厂里有许多不同的机器和操作流程。每次你需要找到某个特定的机器，比如一台装配机器人，但工厂的布置经常变，比如灯光变暗、机器位置微调，甚至工人换了位置。以前，你只记住了每台机器的具体位置和外观，但当环境变化时，你就找不到它了。研究人员就像在工厂里不断改变机器的外观和位置，测试一个智能机器人能不能在不同环境下找到目标。结果发现，很多机器人其实只记住了机器的绝对位置，而没有理解它们之间的关系。这个发现提醒我们，要让机器人变得更聪明，就得教它理解空间关系，而不是死记硬背。这样，无论环境怎么变，它都能找到目标，就像你学会了看书架上的书是怎么摆放的，而不是只记住某一本书的具体位置。
ELI14 Explained like you're 14
想象你在学校的图书馆找一本书，可是每次你去，书架上的书都换了位置。有时候，书的颜色变了，有时候，书架变得更大或更小。以前，你只记得那本书在左边第几排，但现在，书架变得不一样了，你就找不到那本书了。这就像电脑里的模型一样，它们学会了只记住了书的具体位置，而没有理解它们之间的关系。研究发现，这些模型在面对环境变化，比如放大或缩小屏幕、改变界面样式时，表现得很差。于是，研究人员设计了一种新方法，就像不断变换书架的样子，测试模型是否还能找到目标。结果显示，模型只会死记硬背，不能理解空间关系。这个研究告诉我们，要让电脑变得更聪明，就得教它理解事物之间的关系，而不是只记住它们的具体位置。这样，无论书架怎么变，你都能找到那本书！
Abstract
GUI grounding models report over 85% accuracy on standard benchmarks, yet drop 27-56 percentage points when instructions require spatial reasoning rather than direct element naming. Current benchmarks miss this because they evaluate each screenshot once with a single fixed instruction. We introduce GUI-Perturbed, a controlled perturbation framework that independently varies visual scenes and instructions to measure grounding robustness. Evaluating three 7B models from the same architecture lineage, we find that relational instructions cause systematic accuracy collapse across all models, a 70% browser zoom produces statistically significant degradation, and rank-8 LoRA fine-tuning with augmented data degrades performance rather than improving it. By perturbing along independent axes, GUI-Perturbed isolates which specific capability axes are affected-spatial reasoning, visual robustness, reasoning calibration-providing diagnostic signal that aggregate benchmarks cannot. We release the dataset, augmentation pipeline, and a fine-tuned model.