Scaling GUI Agents with Visual State Transitions
Proposes State Transition Pretraining (STP) using joint inverse and forward dynamics to enhance GUI agents, improving success rates by up to 6.2%.
Key Findings
Methodology
This paper introduces STP, leveraging a unified multimodal model (like BAGEL) trained on visual state transitions. It jointly optimizes inverse dynamics (predict actions from state changes) and forward dynamics (predict next states from current states and actions). Using large-scale, annotation-free transition data extracted from existing trajectories, the model learns to understand GUI dynamics without explicit task labels. After pretraining, the model is fine-tuned on task-specific trajectories, showing consistent improvements over baseline models trained solely via trajectory fine-tuning. Experiments across desktop and mobile benchmarks demonstrate that joint optimization yields stable performance gains, which scale with the amount of transition data.
Key Results
- On AgentNetBench, models with STP improved average success rate by 2.3% to 6.2%, with coordinate accuracy gains of 1.6% to 6.8%. In mobile environments, success rates increased by 0.9% on AndroidControl and 0.6% on GUIOdyssey. These improvements are consistent across different data scales and evaluation metrics, confirming the effectiveness of the pretraining stage.
- Joint inverse and forward dynamics optimization provides more stable and scalable improvements than single-objective training. Performance continues to improve as transition data volume increases, demonstrating the benefit of structured self-supervised signals.
- Even when the transition data overlaps with fine-tuning data, performance gains persist, indicating that the benefits stem from better internal representations and understanding of GUI dynamics, not just data exposure.
Significance
This work addresses the critical challenge of data efficiency in scaling GUI agents. By introducing a self-supervised pretraining stage based on visual state transitions, it reduces reliance on costly annotations and enhances model generalization. The approach bridges the gap between static trajectory-based learning and dynamic understanding, enabling more scalable and adaptable GUI automation systems. Its implications extend to industrial automation, intelligent assistants, and autonomous interface interaction, marking a significant step toward more autonomous and robust GUI agents.
Technical Contribution
The core innovation lies in combining inverse and forward dynamics objectives within a single multimodal model, trained on abundant, annotation-free transition data. This structured pretraining enhances the model’s internal representation of GUI dynamics, facilitating better transfer to downstream tasks. The use of a unified multimodal architecture allows seamless integration of visual and textual modalities, enabling the model to predict actions and states effectively. This approach differs from prior methods that rely solely on supervised trajectory data or reinforcement learning, offering a more scalable and data-efficient solution with theoretical stability and empirical robustness.
Novelty
This is the first work to leverage large-scale, unlabeled visual state transitions for pretraining GUI agents. By jointly optimizing inverse and forward dynamics, it introduces a new paradigm that captures the causal relationships in GUI interactions without explicit task annotations. Unlike previous approaches limited to supervised fine-tuning or reinforcement learning, this method exploits self-supervised signals from raw transition data, providing a novel, scalable pathway for GUI agent development.
Limitations
- The current model primarily captures local state changes and may struggle with long-term dependencies or highly complex interactions. Its robustness in noisy or highly dynamic environments remains to be validated.
- Pretraining costs are substantial, requiring significant computational resources, which may limit accessibility for some applications.
- The approach relies on high-quality transition data; in scenarios with sparse or noisy interactions, performance may degrade. Future work should explore integrating reinforcement learning for better exploration and robustness.
Future Work
Future directions include combining this pretraining with reinforcement learning to improve long-term planning and exploration capabilities. Expanding to multimodal data sources, such as audio or haptic feedback, could further enhance understanding. Additionally, developing more efficient training algorithms to reduce computational costs and extending to more complex, real-world GUI environments are promising avenues.
AI Executive Summary
The rapid proliferation of graphical user interfaces (GUIs) has driven the need for intelligent agents capable of automating complex interactions across diverse platforms. Traditional approaches rely heavily on supervised trajectory data, which is costly and limited in scale. To address this, recent research has explored self-supervised learning methods, but effectively modeling GUI dynamics remains challenging.
This paper introduces State Transition Pretraining (STP), a novel framework that leverages large-scale, annotation-free visual state transition data to enhance GUI agent performance. By jointly optimizing inverse dynamics (predicting actions from state changes) and forward dynamics (predicting next states from current states and actions), the model internalizes a structured understanding of GUI behavior. Using a unified multimodal architecture, such as BAGEL, the pretraining process captures causal relationships in interface interactions, providing a robust initialization for subsequent task-specific fine-tuning.
Extensive experiments across desktop and mobile benchmarks demonstrate that models pretrained with STP outperform baseline models trained solely on trajectories. Success rates improve by up to 6.2%, and coordinate accuracy gains are observed across datasets like AgentNetBench, AndroidControl, and GUIOdyssey. The results show that joint optimization of dynamics objectives yields more stable and scalable improvements, especially as the amount of transition data increases. Notably, even in domain-shifted scenarios where transition data overlaps with fine-tuning data, performance gains persist, indicating that the benefits are rooted in better internal representations.
This work significantly advances the scalability and robustness of GUI agents, reducing dependency on costly annotations and enabling more autonomous, adaptable interface automation. Its implications span industrial automation, intelligent assistants, and beyond. Future research will explore integrating reinforcement learning, expanding multimodal inputs, and optimizing training efficiency, paving the way for truly autonomous GUI systems.
Deep Analysis
Background
GUI作为人机交互的核心，近年来伴随深度学习的发展，出现了多种基于视觉和语言的自动化代理。早期方法依赖手工规则或强化学习，面临数据稀缺和泛化不足的问题。近年来，视觉-语言模型（如CLIP、ALIGN）推动多模态理解，但在动态建模方面仍存在挑战。传统轨迹微调虽有效，但成本高昂且难以扩展。为突破这一瓶颈，研究者开始利用无标注的状态转移数据，旨在通过自监督学习提升模型动态理解能力，为大规模应用提供可能。
Core Problem
核心问题在于如何高效利用大量无标注的GUI状态转移数据，提升模型对界面动态的理解和预测能力。现有方法多依赖昂贵的轨迹标注，难以扩展到大规模场景。同时，单一目标优化（如轨迹微调）难以兼顾视觉、动作和任务规划，导致模型泛化能力不足。设计一种结构化、可扩展的预训练策略，成为制约GUI代理发展的关键难题。
Innovation
本文创新点在于引入无标注状态转移的预训练任务，结合逆动力学（预测动作）与正动力学（预测下一状态）两个目标，构建结构化学习信号。采用多模态模型（如BAGEL）实现跨模态理解，增强模型对GUI动态的感知能力。区别于传统依赖轨迹标注的策略，该方法利用大规模无标注数据进行预训练，显著提升模型的迁移和泛化能力，为规模化GUI代理提供新思路。
Methodology
- �� 构建无标注状态转移数据集，从现有轨迹中提取连续截图和动作元组。
- �� 设计联合优化目标：逆动力学（st, st+1 → at）和正动力学（st, at → st+1），通过多模态模型（如BAGEL）实现。
- �� 训练过程中，模型同时学习预测动作和下一状态，形成内在的GUI世界模型。
- �� 预训练完成后，结合任务指令进行轨迹微调，提升任务执行能力。
- �� 实验中采用多尺度数据集，比较不同预训练目标和数据规模对性能的影响。
Experiments
采用AgentNet、AndroidControl和GUIOdyssey等公开数据集，分别在桌面和移动场景中测试。模型在不同轨迹数据量（如2K、5K、18K轨迹）基础上进行微调，评估指标包括成功率、定位精度和任务完成率。对比仅用轨迹微调的模型，验证STP的有效性。还进行了不同目标组合和数据规模的消融实验，分析预训练的贡献。
Results
在AgentNetBench上，STP模型成功率提升2.3%至6.2%，在AndroidControl和GUIOdyssey中，成功率分别提升0.9%和0.6%。联合优化目标比单一目标更稳定，性能随转移数据规模增长持续改善。即使在数据重叠条件下，预训练仍带来性能提升，验证了其结构化学习信号的有效性。整体结果表明，STP显著增强模型的动态理解和泛化能力。
Applications
该方法适用于自动化测试、界面交互优化和智能助手等场景。只需无标注的状态转移数据即可预训练模型，降低数据采集成本。未来可结合强化学习实现自主探索，应用于复杂交互环境，推动工业界的自动化和智能化升级。
Limitations & Outlook
当前模型主要依赖局部状态转移信息，难以捕获长时依赖。复杂场景中的鲁棒性和泛化能力仍需验证。预训练成本较高，实际部署时需优化模型效率。未来应结合强化学习和多任务训练，增强模型适应性。
Plain Language Accessible to non-experts
想象一个工厂里，工人每天都在不同岗位操作机器。每次操作后，机器状态会发生变化，比如开关被打开或关闭。工厂管理者希望机器人学会这些操作背后的规律，提前知道操作后会发生什么。于是，他们让机器人观察很多操作记录，学习每次操作带来的变化。这样，机器人就能在没有指示的情况下，自己预测未来的状态，知道哪些操作会带来什么结果。这就像我们教孩子玩拼图游戏，让他们通过观察拼图的变化，学会下一步怎么拼。最终，机器人变得更聪明，可以自主完成复杂任务，不再需要每次都告诉它具体怎么做。这就是本文提出的状态转移预训练的核心思想，用大量无标注的操作变化数据，让模型理解界面动态，从而更好地完成任务。
ELI14 Explained like you're 14
想象你在玩一个超级复杂的游戏，每次你点击屏幕上的按钮，界面会发生变化。有时候你会忘记哪些操作会带来什么结果，但如果你能提前知道下一步会发生什么，就能更快完成任务。这个研究就像让机器人看很多你做的操作记录，然后学会预测每次点击后界面会变成什么样。这样，机器人就能自己猜到下一步该怎么做，而不用每次都告诉它具体操作。它就像一个聪明的助手，能提前知道界面会变成什么样，帮你节省很多时间。研究中，科学家用一种特别的方法，让机器人通过观察界面变化，学习动作和界面之间的关系。结果显示，这样的学习方式让机器人变得更聪明，能更好地完成各种任务。未来，这种技术还能用在手机、电脑等设备上，让我们的生活变得更方便。
Abstract
We introduce State Transition Pretraining (STP) as a new scaling axis for GUI agents. During the STP stage, we continually pretrain a unified multimodal model on visual state transitions by jointly optimizing inverse dynamics (predicting actions from state changes) and forward dynamics (predicting next states from current states and actions). This optimization equips the model with better action-grounded visual representations and an internal world model of GUI dynamics. When subsequently fine-tuned on trajectories with task instructions, our STP-trained models consistently outperform baselines trained solely via direct trajectory fine-tuning across agent benchmarks in both desktop and mobile GUI scenarios (AgentNetBench, AndroidControl, and GUIOdyssey). Further empirical studies show that joint dynamics optimization yields stable improvements over single-objective training, and downstream performance scales steadily with the volume of transition data.