GTA1: GUI Test-time Scaling Agent
GTA1 employs test-time scaling and RL-based coordinate prediction, achieving state-of-the-art GUI interaction accuracy with 83.1% on ScreenSpot-Pro.
Key Findings
Methodology
GTA1 integrates a test-time scaling strategy with a reinforcement learning (RL) model for direct coordinate prediction. During inference, multiple candidate actions are sampled from a planner, then evaluated by a multimodal large language model judge to select the most appropriate. If the action involves coordinate interaction, the grounding model predicts the precise point. The training employs reward signals for successful clicks, with data cleaned via bounding box consistency checks. This approach enhances robustness in complex, high-resolution GUIs by exploring short-term alternatives without full sequence rollout.
Key Results
- GTA1-7B achieves 83.1% accuracy on ScreenSpot-Pro and 89.3% on OSWorld, outperforming previous methods by 5-15 percentage points. Task success rate on OSWorld-Verified reaches 82.5%. Ablation studies confirm that test-time sampling improves planning robustness, and RL coordinate prediction reduces localization errors. The model demonstrates strong generalization across diverse high-resolution interfaces.
Significance
This work addresses longstanding challenges in GUI automation: effective planning in large action spaces and precise visual grounding. By combining sampling-based decision-making with direct coordinate regression, GTA1 significantly advances autonomous interface interaction. Its robustness and accuracy open pathways for practical applications in automated testing, intelligent assistants, and human-computer interaction, pushing AI towards more reliable and adaptable visual reasoning in complex environments.
Technical Contribution
The key technical innovations include: 1) a test-time sampling strategy that explores multiple candidate actions and uses a large language model judge for selection, reducing planning uncertainty; 2) a reinforcement learning model that directly predicts interaction coordinates, bypassing boundary box reliance; 3) a reward mechanism that emphasizes successful clicks, improving model robustness. The architecture balances decision exploration and precise localization, setting a new standard for GUI agents.
Novelty
This study is the first to combine test-time sampling with RL-based coordinate prediction in GUI agents, effectively addressing the limitations of prior supervised methods that rely on center or boundary box annotations. Unlike existing approaches, GTA1 explores multiple short-term alternatives without full sequence rollout, enabling high-precision interaction in complex, dynamic interfaces. Its direct coordinate prediction and sampling-based planning constitute a novel paradigm in GUI automation research.
Limitations
- Despite improvements, the model struggles in highly dynamic or occluded interfaces due to limited training data diversity and visual ambiguity, leading to localization errors.
- The sampling process, while robust, introduces computational overhead, making real-time deployment challenging in resource-constrained environments.
- Current focus on static high-resolution screens limits adaptability to multi-modal or multi-task scenarios, requiring further extension for broader applicability.
Future Work
Future research will focus on integrating multi-modal cues, such as audio or external knowledge, to enhance robustness. Developing more efficient sampling and judgment mechanisms will reduce computational costs, enabling real-time applications. Extending the framework to multi-task and multi-modal environments, as well as exploring unsupervised or semi-supervised training, will further improve generalization and practical deployment.
AI Executive Summary
Automating graphical user interface (GUI) tasks remains a core challenge in advancing artificial intelligence towards general-purpose systems. Traditional approaches often rely on supervised learning, which struggles with high-resolution, complex, and dynamic interfaces. These methods tend to be brittle, especially when faced with diverse layouts, visual clutter, or unpredictable changes. To address these issues, this work introduces GTA1, a novel GUI agent that combines a test-time scaling strategy with reinforcement learning (RL) for direct coordinate prediction.
The core idea behind GTA1 is to explore multiple candidate actions at each decision point, evaluate their contextual appropriateness using a multimodal large language model judge, and select the best. This sampling-based approach mitigates the risk of early errors cascading into task failure. When an action involves clicking or interacting at a specific point, the grounding model predicts the precise location, guided by reward signals for successful interactions. This combination allows the agent to navigate complex, high-resolution GUIs with high accuracy and robustness.
Experimental results demonstrate that GTA1 surpasses existing state-of-the-art methods across multiple benchmarks, including ScreenSpot-Pro and OSWorld. The 7B-parameter model achieves an accuracy of 83.1% in GUI grounding and a task success rate of 82.5%, outperforming previous models by significant margins. Ablation studies confirm the effectiveness of test-time sampling and RL coordinate prediction, highlighting their roles in improving performance.
This research significantly advances the field of autonomous GUI interaction, providing a lightweight, scalable, and highly accurate framework. Its potential applications span automated testing, intelligent virtual assistants, and human-computer interaction, promising more reliable and adaptable AI systems in real-world environments. Future directions include multi-modal integration, efficiency optimization, and extension to multi-task scenarios, aiming for broader deployment and higher-level reasoning capabilities.
Deep Analysis
Background
GUI自动化技术经历了从规则脚本到深度学习模型的演变。早期方法如基于模板匹配和规则引擎，难以适应复杂多变的界面。近年来，深度监督学习模型如UI-TARS、UI-R1等，通过图像识别和文本理解实现了较好的性能，但在高分辨率和动态场景中仍存在鲁棒性不足的问题。多模态大模型的出现，为界面理解和交互提供了新的可能性，结合视觉和语言信息，推动了自主界面代理的发展。然而，面对庞大的动作空间和复杂的视觉环境，现有方法在规划和定位方面仍难以兼顾效率与精度，亟需创新的模型架构和训练策略。
Core Problem
核心问题在于：如何在庞大的动作空间中实现高效且鲁棒的任务规划，以及在高分辨率、复杂界面中实现精准的目标定位。传统方法多依赖单一监督信号，难以应对界面多样性和动态变化，导致误差积累和任务失败。特别是在没有“前瞻”能力的情况下，如何避免早期错误影响整体任务执行，是制约GUI自主代理发展的关键瓶颈。这些问题的解决对于实现真正的自主界面操作具有重要意义。
Innovation
本研究的创新点包括：
1）引入测试时尺度采样策略，通过多候选方案探索，结合判别模型选择最优，增强规划鲁棒性；
2）采用RL模型直接预测交互坐标，避免边界框依赖，提高泛化能力；
3）奖励机制强化成功点击行为，提升模型在复杂界面中的表现。这些创新结合，显著改善了高分辨率界面中的任务执行效果，突破了传统方法的局限。
Methodology
- �� 输入用户指令、当前界面截图和已执行轨迹到规划器，采样K个候选动作提案。
- �� 利用多模态大模型作为判别器，评估每个提案的合理性，选择最优方案。
- �� 若为坐标型操作，Grounding模型预测具体交互点。
- �� 通过奖励机制，强化成功点击，训练模型直接输出坐标。
- �� 采用数据清洗策略，确保训练数据中界面元素标注的准确性。
- �� 在训练中结合GRPO算法，优化模型参数，提升泛化能力。
- �� 测试时，结合采样、判别和定位，逐步完成任务，避免早期错误累积。
Experiments
采用公开的高分辨率GUI数据集如ScreenSpot-Pro和OSWorld，评估模型的定位准确率和任务成功率。对比多种SOTA方法，验证GTA1在复杂界面中的优越性。设置不同模型规模（7B、32B参数），进行消融实验，验证采样策略和RL模型的贡献。指标包括准确率、成功率和响应时间，确保模型在真实环境中的实用性。实验还包括在动态变化场景中的鲁棒性测试，验证模型的泛化能力。
Results
GTA1-7B模型在ScreenSpot-Pro上达到83.1%的定位准确率，优于现有方法5-15个百分点。在OSWorld-Verified任务中，成功率达82.5%。消融实验显示，测试时尺度采样显著提升鲁棒性，RL直接坐标预测减少误差。模型在高分辨率、多任务环境中表现出优异的适应性，验证了其在复杂场景中的实用性和高精度。
Applications
该技术可应用于自动化软件测试、智能虚拟助手、界面优化等场景。用户只需提供界面截图和任务指令，系统即可自主完成操作，减少人工干预。特别适合高分辨率、多任务、多场景的工业应用，推动智能界面交互的普及与发展。
Limitations & Outlook
模型在极端动态或遮挡严重的界面中仍存在定位误差，主要由于训练数据多样性不足和视觉模糊。RL训练成本较高，限制实时性。未来需优化模型结构，提升多模态融合能力，降低计算成本，增强适应性和实用性。
Plain Language Accessible to non-experts
想象你在厨房做菜，菜单上写着各种菜肴，但每次你都要根据实际情况选择合适的步骤。传统的方法就像只记住了菜单上的步骤，遇到特殊情况就容易出错。而GTA1就像有个聪明的助手，它会在你做菜时，帮你同时考虑多个可能的做法，挑选出最合适的那一个。它还能根据你之前的操作，预测下一步应该放在哪个碗里，确保每次都做得又快又准。这样一来，无论厨房多复杂，它都能帮你顺利完成菜肴。这种方法让自动化操作变得更智能、更稳健，就像有个厨艺高手在帮忙一样。
ELI14 Explained like you're 14
你知道做饭的时候，有时候菜单上的步骤不一定适合当下的厨房情况？比如说，锅太热或者调料不够。以前的机器人就像只会照着菜单走，遇到问题就会乱。现在，GTA1就像一个聪明的厨师助手，它会同时考虑好几个做法，然后帮你挑出最合适的那个。它还能根据你之前的操作，预测下一步应该放在哪个碗里，确保每次都做得又快又好。这就像你有个超级助手，总能帮你应对厨房里的各种突发状况，让做饭变得轻松又有趣。未来，这样的技术还能帮我们自动化更多复杂的任务，就像有个智能厨师一样，帮我们搞定各种挑战！
Abstract
Graphical user interface (GUI) agents autonomously complete tasks across platforms (\eg, Linux) by sequentially decomposing user instructions into action proposals that iteratively interact with visual elements in the evolving environment. However, two main challenges arise: i) planning (\ie, the action proposal sequence) under expansive action space, where selecting an appropriate plan is non-trivial, as many valid ones may exist; ii) accurately grounding actions in complex and high-resolution interfaces, \ie, precisely interacting with visual targets. This paper investigates the aforementioned challenges with our \textbf{G}UI \textbf{T}est-time Scaling \textbf{A}gent, namely GTA1. First, we conduct test-time scaling to select the most appropriate action proposal: at each step, multiple candidate proposals are sampled and evaluated and selected by a judge model. It trades off computation for better decision quality by concurrent sampling. Second, we propose a model that improves grounding of the selected action proposals to its corresponding visual elements. Our key insight is that reinforcement learning (RL) facilitates grounding through inherent objective alignments, rewarding successful clicks on interface elements. Experimentally, GTA1 achieves state-of-the-art performance on both grounding and agent task execution benchmarks. The code and models are released here.