HATS: Hardness-Aware Trajectory Synthesis for GUI Agents
HATS employs hardness-aware trajectory synthesis, enhancing GUI agents' generalization in ambiguous interactions.
Key Findings
Methodology
HATS integrates hardness-driven exploration with alignment-guided refinement via HD-MCTS, guiding sampling toward semantically complex actions. The hardness metric, based on the reconstruction recall R(A,B), quantifies action ambiguity. The framework iteratively explores, validates, and refines trajectories, propagating hardness rewards to bias future search. This closed-loop mechanism significantly improves data diversity and semantic fidelity, outperforming state-of-the-art baselines across benchmark GUI environments.
Key Results
- On ANDROIDWORLD, HATS achieved an overall success rate of 22.6%, surpassing OS-GENESIS by over 11%, with notable improvements in complex interaction categories. In WEBARENA, success rates reached 24.87%, demonstrating robustness across web domains.
- The introduction of the hardness metric effectively prioritized ambiguous yet informative actions, leading to richer training data. Multi-round validation reduced instruction-trajectory misalignment, with the recall metric correlating strongly with task success.
- Ablation studies confirmed that both hardness-driven exploration and iterative refinement contributed significantly to performance gains, especially in challenging scenarios with semantic ambiguity.
Significance
This work advances GUI trajectory synthesis by explicitly modeling semantic ambiguity, addressing the core challenge of capturing complex, context-dependent actions. It enhances the quality and diversity of training data, leading to more robust and generalizable GUI agents. The approach bridges the gap between simple interactions and real-world complexity, opening pathways for scalable autonomous interface understanding. Its implications extend to automated testing, intelligent assistants, and interface design, fostering more adaptive and resilient AI systems.
Technical Contribution
HATS introduces a novel hardness-aware framework combining exploration and refinement within HD-MCTS, leveraging a semantic ambiguity metric based on verification recall. This dual mechanism ensures sampling of high-value, complex interactions and rigorous instruction-trajectory alignment. The method departs from prior random or one-shot approaches, providing a scalable, data-efficient pipeline that significantly improves the fidelity and diversity of synthesized trajectories, thus enabling better downstream training of GUI agents.
Novelty
This is the first work to formalize the notion of semantic hardness in GUI trajectory synthesis, integrating it into a closed-loop exploration-refinement process. Unlike existing methods that rely on random sampling or single-pass instruction generation, HATS systematically targets ambiguous actions and iteratively corrects instruction-trajectory pairs, resulting in richer, more accurate datasets. This innovation effectively tackles the long-standing challenge of modeling complex, context-dependent GUI interactions.
Limitations
- The framework depends on pre-trained vision-language models, which may limit understanding in highly dynamic or unseen interface scenarios. The hardness metric and parameters require careful tuning across different environments.
- The multi-round validation process, while effective, introduces additional computational overhead, potentially impacting scalability in real-time applications.
- Handling highly complex or rapidly changing interfaces remains challenging; future work should explore integrating reinforcement learning or meta-learning to further improve exploration efficiency.
Future Work
Future directions include integrating multi-modal signals such as audio or haptic feedback, extending the framework to multi-task and multi-environment settings, and developing adaptive hardness metrics that learn from user interactions. Combining reinforcement learning strategies could further optimize exploration efficiency, enabling autonomous agents to better handle real-world, dynamic interfaces. Additionally, exploring transfer learning across domains may enhance generalization capabilities.
AI Executive Summary
HATS introduces a novel hardness-aware trajectory synthesis framework designed to address the persistent challenge of capturing semantic-ambiguous actions in GUI environments. Traditional data collection methods often favor simple, intuitive interactions, resulting in datasets that lack the complexity necessary for training robust, generalizable GUI agents. To overcome this, HATS defines a hardness metric based on the verification recall R(A,B), quantifying the semantic ambiguity of actions. This metric guides exploration toward underrepresented, complex interactions via the HD-MCTS algorithm, which balances exploration and exploitation through a UCB policy, expanding the search tree with actions that are both rare and semantically challenging.
The core innovation lies in the multi-round alignment-guided refinement process, which iteratively verifies and corrects synthesized instructions. Unlike one-shot approaches, this process ensures high fidelity and semantic consistency, significantly reducing instruction-trajectory misalignment. The entire system operates in a closed-loop, propagating hardness rewards along the search tree to bias future exploration toward difficult yet valuable interactions. Extensive experiments on datasets such as ANDROIDWORLD and WEBARENA demonstrate that HATS outperforms existing methods, achieving success rate improvements of over 30% in complex scenarios. These results highlight the framework’s ability to generate diverse, high-quality trajectories that enhance the training of GUI agents.
This work marks a significant step forward in automating high-fidelity GUI data synthesis, enabling agents to better understand and execute complex, context-dependent actions. Its implications extend beyond interface automation to broader fields like autonomous robotics, intelligent assistants, and interface design. By explicitly modeling semantic ambiguity and systematically addressing it through a closed-loop process, HATS offers a scalable, effective solution for advancing AI understanding of complex human-computer interactions. Future research will focus on integrating multi-modal signals, reducing computational costs, and expanding to multi-task, multi-environment scenarios, further pushing the boundaries of autonomous interface understanding.
Deep Analysis
Background
随着大规模视觉-语言模型（VLMs）的发展，GUI代理在自动化任务中的应用逐渐成熟。早期方法依赖手工规则或外部工具，存在数据稀缺和泛化能力不足的问题。近年来，逆向任务合成和自动轨迹生成成为研究热点，显著降低了数据采集成本，但仍面临语义模糊交互难以捕获的问题。复杂场景中的隐晦动作、上下文依赖和视觉歧义，导致训练数据质量下降，模型泛化受限。现有方法多忽视动作的语义模糊性，导致代理在实际应用中表现不稳。为解决这一难题，研究者开始关注动作的语义难度和交互的多样性，提出硬度感知机制，旨在提升轨迹合成的语义丰富性和鲁棒性。
Core Problem
核心问题在于现有轨迹合成方法难以捕获语义模糊的动作，导致数据多样性不足和模型泛化能力差。随机探索偏向简单直观的交互，忽视复杂上下文依赖的隐晦动作，限制了模型在真实场景中的表现。此外，指令生成的单次验证机制难以保证语义一致性，存在偏差和噪声，影响训练效果。如何在保证数据多样性的同时，确保指令与动作的语义对齐，成为提升GUI代理性能的关键难题。
Innovation
HATS的创新点在于引入硬度指标，定义动作的语义模糊度，结合硬度驱动探索和多轮指令修正，形成闭环优化机制。具体包括：
- �� 硬度指标：基于反向验证指标R(A,B)，量化动作的语义模糊程度。
- �� 硬度驱动探索：利用HD-MCTS，优先采样高硬度（模糊）区域，丰富数据多样性。
- �� 多轮修正：通过多轮指令验证与修正，确保指令与动作的语义一致性。
- �� 软硬结合：将硬度作为探索和修正的反馈信号，动态调整采样策略，提升轨迹质量。
Methodology
- �� 采用HD-MCTS算法，结合UCB策略，平衡探索与利用，优先采样高硬度区域。
- �� 在选择阶段，从根节点开始递归选择动作，利用硬度奖励引导偏向复杂交互。
- �� 扩展阶段，随机采样未探索动作，扩展搜索树。
- �� 模拟阶段，执行动作生成轨迹，进行多轮指令反向验证。
- �� 反向传播，将硬度奖励沿路径反向传播，调整未来探索偏向。
- �� 在修正阶段，利用多轮验证机制，反复修正指令，确保语义一致。
- �� 最终筛选出高质量、语义丰富且可执行的轨迹，用于训练。
Experiments
在ANDROIDWORLD和WEBARENA两个数据集上，采用多模态视觉-语言模型（InternVL2-4B/8B、Qwen2-VL-7B）进行训练，比较基线包括OS-GENESIS、任务驱动和自我指令方法。指标包括任务成功率和指令-动作对齐度，硬度参数调优通过参数敏感性分析验证。采用不同任务类别，进行 ablation 和多场景测试，确保结果的全面性。实验还评估硬度指标的有效性和多轮验证的贡献。
Results
HATS在ANDROIDWORLD中整体成功率达22.6%，在WEBARENA中达24.87%，均优于OS-GENESIS，尤其在复杂模糊交互场景中提升显著。硬度指标的引入使得高难度区域探索更有效，模型能够捕获更多隐晦动作，泛化能力增强。多轮验证显著降低指令偏差，提升了轨迹的语义一致性。消融实验显示，硬度驱动探索和多轮修正是性能提升的关键因素，验证了方法的有效性。
Applications
该技术可应用于自动化测试、智能助手、界面设计优化等场景，尤其适合复杂交互环境中的任务自动化。通过自动生成高质量轨迹，减少人工标注成本，提升模型在实际场景中的鲁棒性。未来还可结合用户反馈，动态调整硬度定义，增强系统自主学习能力，推动人机交互的智能化发展。
Limitations & Outlook
目前方法依赖预训练模型，受模型表达能力限制，复杂场景中仍存在理解不足的问题。硬度参数调优具有一定经验依赖，可能在不同应用中需要调整。极端复杂或动态界面中，探索效率可能下降，未来需结合强化学习优化策略。
Plain Language Accessible to non-experts
想象你在一家工厂工作，工厂里有很多不同的机器，每个机器都能做不同的事情。有些机器操作简单，比如按一下按钮就能开关；有些机器则很复杂，需要同时考虑很多因素，比如温度、压力、时间等。工厂的目标是让所有机器都能顺利完成任务，但有些操作很难理解，比如某些按钮的作用依赖于前面做过的事情，或者看起来一样但功能不同。为了让工厂更高效，工人需要知道哪些操作最难理解，特别是那些容易出错或隐藏在细节里的操作。HATS就像是给工厂装上了一个智能助手，它可以判断哪些操作最复杂，帮助工人多练习这些难点，确保每个操作都能正确完成。这样，工厂的生产效率就会大大提高，机器也能更好地协作，整个流程变得更顺畅。
ELI14 Explained like you're 14
想象你在学校里玩一个游戏，你需要完成各种任务，比如找到隐藏的宝藏、解谜或者操作一些复杂的机器。每个任务都需要你做不同的动作，有些动作很容易，比如按按钮；但有些动作很难理解，比如为什么要在特定顺序做某些事情，或者看起来一样的按钮其实有不同的作用。这个游戏里，很多时候你会遇到一些隐藏的难点，比如你不知道哪个按钮会打开门，或者需要先做某个步骤才能成功。HATS就像是给你一个聪明的朋友，他能告诉你哪些动作最难，帮你多练习那些难点，让你变得更厉害。最终，你就能更快、更准确地完成任务，游戏也变得更有趣、更有挑战性。这个方法让机器人也能像你一样，学会应对复杂的任务，不会被隐藏的难点难倒。
Glossary
Hardness Metric (硬度指标)
衡量动作语义模糊程度的指标，基于验证指标R(A,B)，反映动作理解难度。
引导探索偏向复杂交互区域。
HD-MCTS (Hardness-Driven Monte Carlo Tree Search)
结合硬度指标的蒙特卡洛树搜索算法，用于系统性探索复杂交互。
核心算法，用于轨迹采样。
Reconstruction Recall (反向验证指标)
衡量指令与动作匹配程度的指标，反映语义一致性。
用于硬度评估和修正。
Semantic Ambiguous Actions (语义模糊动作)
在特定上下文中具有多重意义或易混淆的界面交互动作。
研究重点，影响泛化。
Closed-loop Optimization (闭环优化)
通过探索、验证、修正不断反馈，提升轨迹质量的机制。
方法核心思想。
Open Questions Unanswered questions from this research
- 1 如何在更复杂的动态界面中保持探索效率？硬度指标在多模态、多任务环境中的适应性如何？未来是否可以结合强化学习进一步提升探索策略？
Applications
Immediate Applications
自动化测试
利用HATS自动生成高质量测试轨迹，提升软件界面测试的效率和覆盖率，减少人工成本，适应多样化界面设计。
智能助手
训练更鲁棒的GUI代理，支持复杂任务执行，提升智能助手在多场景下的交互能力。
Long-term Vision
自主学习系统
实现机器人自主理解复杂界面动作，逐步减少对人工标注的依赖，推动人机交互向更自然、更智能方向发展。
Abstract
Graphical user interface (GUI) agents powered by large vision-language models (VLMs) have shown remarkable potential in automating digital tasks, highlighting the need for high-quality trajectory data to support effective agent training. Yet existing trajectory synthesis pipelines often yield agents that fail to generalize beyond simple interactions. We identify this limitation as stemming from the neglect of semantically ambiguous actions, whose meanings are context-dependent, sequentially dependent, or visually ambiguous. Such actions are crucial for real-world robustness but are under-represented and poorly processed in current datasets, leading to semantic misalignment between task instructions and execution. To address these issues, we propose HATS, a Hardness-Aware Trajectory Synthesis framework designed to mitigate the impact of semantic ambiguity. We define hardness as the degree of semantic ambiguity associated with an action and develop two complementary modules: (1) hardness-driven exploration, which guides data collection toward ambiguous yet informative interactions, and (2) alignment-guided refinement, which iteratively validates and repairs instruction-execution alignment. The two modules operate in a closed loop: exploration supplies refinement with challenging trajectories, while refinement feedback updates the hardness signal to guide future exploration. Extensive experiments show that agents trained with HATS consistently outperform state-of-the-art baselines across benchmark GUI environments.