Executable Agentic Memory for GUI Agent
Proposes Executable Agentic Memory (EAM) using knowledge graphs to enhance long-horizon GUI planning, achieving 19.6% success improvement and 6× cost reduction.
Key Findings
Methodology
This work introduces a structured knowledge graph (KG) as persistent memory for GUI agents, built via state-aware DFS exploration and action-group mining to compress multi-step routines. A lightweight Q-function guides Monte Carlo Tree Search (MCTS) over the KG, enabling robust path planning with theoretical bias-consistency guarantees and sample complexity bounds. The offline pipeline autonomously explores GUI states, merges trajectories using Byte Pair Encoding (BPE) for action grouping, and enriches the KG with semantic descriptions. Online, the agent performs value-guided MCTS, leveraging the Q-model for efficient path extraction, with iterative self-training refining the Q-function. The approach reduces token costs and latency, outperforming SOTA baselines like UI-TARS-7B by up to 19.6% success rate on Android benchmarks.
Key Results
- On AndroidWorld, success rate reaches 52.6%, surpassing UI-TARS-7B's 33%. API token costs are reduced 6-fold, with an average latency of 2.8 seconds. The method demonstrates strong generalization across multiple benchmarks, maintaining robustness in complex, long-horizon tasks.
- Theoretical analysis guarantees bias consistency of the Q-model on critical states and provides finite-sample bounds for path recovery. Empirical ablations confirm that action grouping and value-guided search significantly improve efficiency and success rates.
- The framework's integration of structured knowledge, reinforcement learning, and efficient search offers a scalable solution for reliable GUI automation, addressing key limitations of prior generative or memory-based methods.
Significance
This research advances GUI automation by grounding decision-making in structured knowledge graphs, enabling long-term planning with high reliability and efficiency. It addresses the core challenge of error propagation in stepwise models, providing a theoretically sound and practically effective framework. The approach bridges the gap between symbolic reasoning and neural models, paving the way for more autonomous, scalable, and robust intelligent agents in real-world applications such as mobile automation, testing, and assistive technologies. Its success demonstrates the potential of combining structured knowledge with reinforcement learning to solve complex, long-horizon tasks that were previously intractable.
Technical Contribution
The paper's key technical contributions include: 1) constructing a compact, executable GUI knowledge graph via state-aware DFS and Byte Pair Encoding-based action grouping; 2) developing a Q-guided MCTS framework with bias guarantees for path planning; 3) providing theoretical analysis of bias consistency and sample complexity for path recovery; 4) integrating self-training for Q-model refinement, ensuring robustness and generalization. These innovations collectively enable efficient, reliable long-term planning in GUI environments, significantly reducing computational costs compared to generative models.
Novelty
This work is the first to embed a structured, executable knowledge graph as the core memory for GUI agents, combined with a theoretically grounded Q-guided search strategy. Unlike prior approaches relying solely on LLMs or unstructured memory, it explicitly models environment dynamics and guarantees path correctness through bias analysis. The action grouping mechanism further distinguishes it by compressing multi-step routines into reusable high-level actions, greatly reducing search complexity and enabling scalable long-horizon planning.
Limitations
- The approach depends on comprehensive offline exploration; incomplete coverage may lead to suboptimal paths. Dynamic or highly stochastic environments pose challenges for maintaining an accurate, up-to-date knowledge graph. The reliance on offline construction limits adaptability to environment changes. Computational costs, while reduced, still require optimization for large-scale deployment. Future work should focus on online updating and adaptive learning mechanisms.
Future Work
Future directions include integrating online learning to dynamically update the knowledge graph, extending the framework to multi-modal inputs for richer environment understanding, and applying the approach to more complex, real-world scenarios such as multi-user systems or adaptive interfaces. Additionally, exploring more scalable search algorithms and强化学习策略，将进一步提升长远规划的效率和鲁棒性。
AI Executive Summary
Modern GUI agents often struggle with long-horizon tasks due to their reliance on step-by-step interpretation and decision-making, which are prone to error accumulation and inefficiency. Traditional approaches, such as large language model (LLM)-based generation, lack robustness in complex environments, leading to failures in sustained automation. To address these issues, this research introduces Executable Agentic Memory (EAM), a structured knowledge graph (KG) that encapsulates GUI environment dynamics and task routines. The core innovation lies in constructing this KG through state-aware DFS exploration and multi-step routine compression via Byte Pair Encoding, resulting in a compact, executable representation of GUI states and actions.
At inference, a lightweight Q-function guides Monte Carlo Tree Search (MCTS) over the KG, enabling efficient and reliable path planning. Theoretical analysis guarantees bias consistency of the Q-model and provides finite-sample bounds for path recovery, ensuring the method's robustness. Empirical results on benchmarks like AndroidWorld demonstrate a success rate of 52.6%, outperforming the previous SOTA by 19.6%, while reducing token costs sixfold and maintaining an average latency of 2.8 seconds. These improvements highlight the method’s potential for real-time, long-horizon GUI automation.
The approach effectively bridges symbolic reasoning and neural models, offering a scalable solution for complex automation tasks. Future work aims to incorporate online updates, multi-modal inputs, and broader application scenarios, pushing GUI automation toward higher reliability and adaptability in real-world settings.
Deep Analysis
Background
随着大型语言模型（LLMs）在多模态GUI任务中的应用逐步成熟，早期研究多依赖模型中心的逐步生成策略，存在长远规划能力不足、误差累积严重的问题。代表性工作如Wen等（2024）将GPT-4V引入GUI操作，Zhang等（2025）加入记忆模块，Wang等（2024a）实现模块化架构，但都面临高API调用成本和鲁棒性不足的难题。近年来，基于知识图谱和强化学习的方法逐步兴起，试图通过外部知识增强决策能力，但多依赖LLM的上下文注入，缺乏结构化路径保证。整体来看，现有方案在长距离、多步骤任务中表现有限，亟需一种既能保证路径可执行性，又能高效搜索的解决方案。
Core Problem
核心问题在于如何构建一个结构化、可持续的GUI知识存储，并在长远任务中实现高效、鲁棒的路径规划。传统逐步生成策略易受误差影响，导致路径偏离或失败。长距离任务对记忆的完整性和推理能力提出更高要求，现有方法难以在保证效率的同时实现路径的可靠重现。如何结合强化学习的理论保证与知识图谱的结构优势，设计一种能在复杂环境中持续学习、动态更新的长远规划系统，是当前亟待解决的难题。
Innovation
创新点包括：1）提出结构化知识图谱作为GUI代理的核心记忆，明确状态与操作关系；2）利用状态感知DFS和动作分组压缩多步操作，提升记忆效率；3）引入偏差保证的Q模型，指导MCTS在KG上高效搜索路径，确保路径的可行性与最优性；4）结合理论分析，提供偏差一致性和样本复杂度保证，为长距离规划提供理论支撑。这些创新有效突破了传统逐步生成和模型依赖的限制，显著提升长远任务的鲁棒性和效率。
Methodology
- �� 离线阶段：采用状态感知DFS系统性探索GUI环境，收集轨迹数据，构建任务相关的知识图谱。• 轨迹合并：通过动作分组（基于Byte Pair Encoding）压缩多步操作，形成高层次动作节点，优化搜索空间。• 语义丰富：从状态转移中提取页面描述和功能描述，丰富知识图谱的语义信息。• 在线推理：利用偏差保证的Q模型引导MCTS在知识图谱上进行路径搜索，选择高奖励路径。• Q模型训练：通过自训练流程，结合偏好学习和贝尔曼备份不断优化Q值预测能力。• 理论分析：证明Q模型偏差一致性及路径恢复的样本复杂度界，确保路径最优性。• 实验验证：在多个GUI基准上测试，比较成功率、延迟和成本，验证方法的有效性。
Experiments
采用AndroidWorld、MobileMiniWob++和DroidTask三大基准，评估成功率、延迟和API调用成本。对比SOTA方法如UI-TARS-7B和GPT-4o，验证EAM的优越性。设置不同的模型规模和搜索参数，进行消融实验，分析动作分组、偏差保证和自训练的贡献。通过多轮训练验证模型的收敛性和泛化能力，确保在不同环境下的适应性。
Results
在AndroidWorld上，EAM成功率达52.6%，超越UI-TARS-7B的33%，提升19.6%；在MobileMiniWob++和DroidTask中表现同样优异。API调用成本降低6倍，平均延迟缩短至2.8秒。模型在多任务环境中展现出良好的鲁棒性和泛化能力，验证了理论分析的有效性。消融实验显示动作分组和偏差引导显著提升搜索效率，训练自适应能力增强路径恢复的准确性。
Applications
该方法适用于复杂的移动端自动化、智能助手和自动测试等场景，能显著降低人工干预，提高效率。未来可结合多模态信息实现更智能的环境感知，推动自动化在工业、医疗和教育中的应用变革。
Limitations & Outlook
当前方法依赖于充分的离线探索和知识图谱的完整性，面对动态环境或未覆盖场景时仍可能出现路径偏差。模型训练和搜索过程存在一定的计算成本，需优化算法以适应大规模应用。未来需结合在线学习和环境自适应机制，增强系统的鲁棒性和扩展性。
Plain Language Accessible to non-experts
想象你在一家工厂工作，工厂里有很多不同的机器和流程。每台机器代表一个状态，每个操作代表一项任务。以前，我们需要逐个操作，手动记住每个步骤，容易出错，也很慢。现在，工厂里有一本详细的流程图（知识图谱），告诉你每个状态可以做什么，下一步会到哪里。你可以用这本流程图快速找到最合适的路径，不用每次都重新思考。这个方法就像有个聪明的助手，记住所有流程，还能帮你快速找到最优路线，确保每个任务都能顺利完成。这样，工厂的工作变得更快、更可靠，也更容易管理。
ELI14 Explained like you're 14
想象你在玩一个超级复杂的游戏，比如冒险游戏。以前，你每次都要自己猜下一步该怎么走，容易迷路或者走错路。现在，有个智能助手，它记住所有你走过的路和遇到的宝藏，还能帮你规划最短、最安全的路线。它用一种特别的地图（知识图谱）把所有地点和任务连接起来，然后用聪明的算法（像玩棋一样）帮你找出最佳路径。这个助手还会不断学习你的偏好，变得越来越聪明。结果，你可以更快完成任务，不会迷路，也能应对更难的挑战。这就像有个超级聪明的朋友一直在帮你规划冒险路线！
Abstract
Modern GUI agents typically rely on a model-centric and step-wise interaction paradigm, where LLMs must re-interpret the UI and re-decide actions at every screen, which is fragile in long-horizon tasks. In this paper, we propose Executable Agentic Memory (EAM), a structured Knowledge Graph (KG) that shifts GUI planning from free-form generation to a robust retrieval-and-execution process. Our approach includes a sample-efficient memory construction pipeline using state-aware DFS and action-group mining to compress multi-step routines. To ensure efficient planning, we introduce a value-guided graph search where a lightweight Q-function model steers Monte Carlo Tree Search (MCTS) over the KG. We theoretically establish bias-consistency for the Q-model and derive sample complexity bounds for path recovery. Empirically, EAM outperforms state-of-the-art baselines like UI-TARS-7B by up to $19.6\%$ on AndroidWorld, while reducing token costs $6\times$ relative to GPT-4o. With a $2.8$s average latency, EAM enables reliable, quick, and long-horizon GUI automation.