GSAR: Goal-State-Anchor Rewards for Mobile GUI Agents with Self-Evolving Data Synthesis
GSAR combines self-evolving data synthesis and goal-state anchoring to improve GUI agent training with over 90% accuracy in trajectory verification.
Key Findings
Methodology
The GSAR framework integrates automatic data generation via self-evolving interactions with mobile environments and a goal-state anchor mechanism that automatically annotates key UI elements in successful trajectories. This process involves: • Environment exploration through random interaction and task complexification; • Trajectory collection and filtering with GPT-4o; • Automatic extraction of goal states and UI element identification via accessibility trees; • Construction of reward signals based on anchored UI elements and action history. This design ensures scalable, accurate reward supervision for reinforcement learning.
Key Results
- 在AndroidControl和GUI-Odyssey基准测试中，GSAR实现轨迹验证准确率超过90%，显著优于传统模型评估方法，逼近规则基方法的性能。
- 在实际训练中，使用GSAR奖励的代理在AndroidWorld和自建任务集上表现出任务成功率提升8-23%，训练速度加快，泛化能力增强。
- 消融实验显示，结合目标状态锚点和动作历史的奖励机制，能有效降低误判率，提升奖励信噪比，验证机制的鲁棒性。
Significance
本研究解决了GUI强化学习中的环境多样性不足和奖励信号不稳定的问题，提供了自动化、可扩展的任务生成和奖励评价体系。通过自我演化和目标锚定技术，减少了人工干预，提高了训练效率和模型泛化能力，为移动端自动化操作和复杂任务学习开辟新途径。这一框架推动了自主交互系统的研究，具有深远的学术和产业价值。
Technical Contribution
创新点包括：• 提出自我演化数据合成机制，实现环境和任务的自动多样化生成，突破人工配置瓶颈；• 引入目标状态锚定机制，自动识别UI关键元素作为奖励参考，提升奖励的稳定性和准确性；• 结合动作历史与锚点信息，设计高效奖励评估模型，改善奖励信号质量。该体系支持多模型、多任务的自动化训练，具有良好的扩展性。
Novelty
本工作首次将自我演化数据合成与目标状态锚定结合应用于GUI强化学习，解决环境多样性和奖励不可靠的核心难题。与传统基于规则或模型的奖励方法相比，GSAR实现了完全自动化的环境和奖励生成，极大提升训练效率和泛化能力。这一创新为复杂交互场景中的自主学习提供了新思路，具有重要的学术突破意义。
Limitations
- 在极端复杂或多步骤任务中，目标状态识别仍存在误差，影响奖励的绝对准确性，需优化UI元素识别算法。
- 自我演化依赖探索策略的效率，探索不足可能导致环境多样性不足，影响训练效果。
- 在大规模环境中，状态锚点自动标注的计算成本较高，需提升算法效率。
Future Work
未来将结合多模态信息增强目标状态识别的鲁棒性，探索多任务、多智能体协同训练，扩展到更复杂场景。同时引入迁移学习和元学习策略，提升模型在新环境中的适应性，推动GUI代理向更高自主性发展。
AI Executive Summary
As mobile interfaces become increasingly complex, developing efficient and stable GUI agents remains a key challenge. Traditional approaches rely heavily on manual environment setup and rule-based rewards, which limit scalability and adaptability. This hampers the deployment of autonomous systems capable of handling diverse tasks across different applications. To address these issues, the paper introduces GSAR, a novel reinforcement learning framework that combines self-evolving data synthesis with goal-state anchoring.
The core idea is to enable the agent to generate a wide variety of environments and tasks automatically. Through interaction-driven evolution, the system explores app states, creates new tasks via a vision-language model, and filters trajectories with GPT-4o. Successful trajectories are then used to extract goal states, where key UI elements are automatically identified and anchored as reward references. This process ensures that reward signals are both accurate and scalable, overcoming the limitations of rule-based or model-based evaluators.
Experimental results demonstrate that agents trained with GSAR achieve over 90% accuracy in offline trajectory verification, outperforming existing reward models. In online reinforcement learning, agents using GSAR rewards show significant improvements in task success rates—up to 23% higher—compared to baseline methods. Ablation studies confirm that combining goal-state anchors with action history yields the most robust reward signals. The framework's scalability and effectiveness suggest broad applicability in mobile automation, interface understanding, and beyond.
Overall, GSAR represents a significant step toward fully autonomous GUI agents, reducing manual effort and enhancing training stability. Future work will explore multi-modal integration, multi-task learning, and transferability to more complex, real-world scenarios, pushing the frontier of intelligent interface automation.
Deep Analysis
Background
移动端界面自动化操作的研究经历了从规则驱动到深度学习的演变。早期依赖手工规则，效率低且难以适应复杂场景。近年来，基于视觉和语言模型的GUI理解逐渐兴起，如GUITAR、Rico数据集等推动了端到端学习方法的发展。强化学习在界面操作中的应用也逐步展开，但受限于环境多样性和奖励信号的准确性，训练效果不稳定，泛化能力不足。现有研究多依赖人工配置环境或规则验证，难以实现大规模自动化训练，限制了技术的推广。
Core Problem
核心问题在于环境多样性不足和奖励信号不可靠。手动环境配置繁琐，难以自动生成丰富的任务样本，限制了大规模强化学习的实现。同时，现有奖励机制如模型评估或规则验证，存在准确率低、扩展性差的问题，导致训练过程不稳定、效果不佳。解决这些瓶颈对于实现自主、通用的GUI代理至关重要。
Innovation
本研究提出两大创新：• 自我演化数据合成：通过环境交互自动生成多样任务和轨迹，突破人工配置限制；• 目标状态锚定：自动识别成功轨迹终态中的UI元素作为奖励参考，提升奖励的准确性和稳定性。这两项创新结合，极大改善了环境多样性和奖励信号的质量，为大规模自主训练提供了技术基础。
Methodology
- �� 任务环境的自我演化：通过随机探索和任务复杂化，自动扩展环境和任务空间；• 轨迹采集与筛选：利用GPT-4o执行生成任务，筛除不合理轨迹；• 目标状态提取：从成功轨迹中自动识别终态屏幕；• UI元素锚定：利用可访问性树自动标注关键UI元素，构建奖励参考；• 奖励评估：结合动作历史和锚点信息，训练高效的奖励模型。
Experiments
采用AndroidControl、GUI-Odyssey两个公开基准，结合自建任务集，评估数据合成质量和奖励机制效果。对比传统模型和基于规则的奖励，验证GSAR在轨迹验证和强化学习中的优越性。通过离线指标（准确率、F1）和在线训练表现（任务成功率、收敛速度）进行全面评估。消融实验验证锚点和动作历史的贡献，分析不同组件对性能的影响。
Results
在AndroidControl和GUI-Odyssey中，GSAR实现轨迹验证准确率超过90%，优于模型评估方法。训练中，代理任务成功率提升8-23%，收敛速度明显加快。消融实验显示，结合锚点和动作历史的奖励设计，误判率最低，奖励信噪比最高。在自建基准中，训练任务的成功率也有显著提升，验证了方法的实用性和鲁棒性。
Applications
该技术适用于移动端自动化测试、界面交互学习、智能助手等场景。只需提供初始环境和任务描述，系统即可自动生成多样任务和环境，训练高效、稳定。未来可扩展到多模态交互、复杂多步骤任务，推动自主交互系统的商业化应用。
Limitations & Outlook
目前方法在极端复杂任务中，目标状态识别仍存在误差，影响奖励的绝对准确性。环境探索依赖探索策略，可能导致多样性不足。大规模环境中，状态锚点自动标注的计算成本较高，需优化算法效率。未来需结合多模态信息和迁移学习，提升鲁棒性和适应性。
Plain Language Accessible to non-experts
想象你在厨房做菜。每次做菜都需要准备不同的食材、调料和步骤。以前，厨师会写详细的食谱，告诉你每一步怎么做，但每次都要手工准备，费时又不灵活。现在，有个智能厨师，它能自己试验不同的食谱，学习哪些组合最好，还能自动识别做完的菜是否成功。它会观察每次做菜的最后状态，记住哪些步骤成功了，然后用这些经验指导下一次做菜。它还会根据之前的成功经验，调整食谱，变得越来越聪明。这就像论文里的方法：它自己探索不同的环境，识别成功的关键点，用这些点作为奖励的依据，逐步学会在不同厨房里做出美味菜肴。
ELI14 Explained like you're 14
想象你在玩一个游戏，你要完成各种任务，比如找到宝藏或打败敌人。以前，游戏设计师会告诉你每个任务怎么做，但每次都要手工设计关卡，太麻烦了。现在，有个聪明的机器人，它可以自己试着完成任务，然后记住成功的关键步骤，比如哪个按钮按了，哪个位置的宝藏被找到。它会观察自己完成任务后，最后的画面和操作，把重要的细节记下来，作为以后判断自己是否完成任务的标准。这样，它就可以自己不断练习，变得越来越厉害，不用每次都有人指导。这个方法就像论文里的GSAR，通过让机器人自己探索、识别成功的关键UI元素，自动生成任务和奖励，帮助它更快学会操作各种不同的手机界面。
Abstract
Vision-Language Models (VLMs) based GUI agents stand to benefit significantly from online reinforcement learning (RL). However, their training is bottlenecked by two fundamental issues: current data synthesis methods for GUI Agents rely on specific environments and struggle to generate diverse data, while existing evaluators either suffer from limited scalability or provide inaccurate and unreliable reward signals. To overcome these challenges, we introduce GSAR (Goal-State-Anchor Reward), a RL reward framework that supports scalable task generation and delivers reliable reward signals for stable and efficient policy optimization. Our approach features self-evolving data synthesis, which produces multiple environments through task execution and generates diverse tasks and goal states. Complementing this, a state-anchor mechanism automatically annotates task-relevant UI elements in successful goal states as reference anchors. During RL training, these reference anchors provide accurate, scalable reward signals that substantially enhance efficiency. Extensive evaluations demonstrate that our framework achieves over 90% accuracy on offline trajectory verification and performs closest to rule-based methods. Furthermore, agents trained using our reward framework exhibit strong performance on both AndroidWorld and our constructed benchmark, establishing a scalable approach for GUI agent training.