SE-GA: Memory-Augmented Self-Evolution for GUI Agents
SE-GA integrates hierarchical memory and self-evolution to boost GUI task success to 89.0%, enabling long-term planning and continuous learning.
Key Findings
Methodology
SE-GA employs a hierarchical memory system comprising episodic, semantic, and experiential memories, combined with Test-Time Memory Extension (TTME) for long-term planning. TTME dynamically retrieves relevant memories during inference, enhancing reasoning over extended horizons. The Memory-Augmented Self-Evolution (MASE) framework uses data collected via TTME to stabilize and improve the base policy iteratively. Extensive offline and online evaluations show success rates of 89.0% on ScreenSpot and 75.8% on AndroidControl-High, demonstrating superior generalization and robustness across complex, dynamic environments.
Key Results
- On the ScreenSpot benchmark, SE-GA achieved a success rate of 89.0%, outperforming existing methods by over 10%, confirming its effectiveness in multi-step, complex tasks.
- In AndroidControl-High, success rate reached 75.8%, indicating strong adaptability to environment changes and long-term planning.
- Across multiple datasets, including AndroidWorld, SE-GA demonstrated significant improvements, validating the benefits of hierarchical memory and self-evolution in generalization to dynamic scenarios.
Significance
This work addresses fundamental challenges in GUI agent development, notably information retention over long horizons and adaptation to environmental variability. By integrating layered memory and continuous self-improvement, SE-GA advances autonomous agents capable of robust, scalable, and flexible operation. It paves the way for more intelligent human-computer interaction systems, reducing manual intervention and enabling smarter automation in diverse fields such as mobile app testing, assistive technologies, and industrial automation.
Technical Contribution
The core technical innovations include the hierarchical memory architecture that captures short-term, domain-general, and long-term experiences, and the TTME mechanism that dynamically retrieves relevant memories during inference. Additionally, the MASE training pipeline combines supervised fine-tuning with iterative self-evolution, enabling the model to adapt without extensive retraining. These contributions collectively enhance the stability, efficiency, and scalability of GUI agents, setting new benchmarks in multi-step task performance.
Novelty
This is the first work to unify hierarchical memory retrieval with test-time memory extension and self-evolution training in GUI agents. Unlike prior approaches relying on static datasets or simple memory buffers, SE-GA dynamically manages and leverages rich, multi-modal memories for long-term planning and continuous learning, representing a significant step forward in autonomous GUI navigation.
Limitations
- Despite improvements, the model may still face challenges in environments with rapid, unpredictable changes, risking memory saturation or outdated information.
- Training requires large-scale interaction data, leading to high computational costs and data collection bottlenecks.
- Scalability to multi-modal, multi-agent, or real-time industrial scenarios remains to be validated, requiring further research.
Future Work
Future directions include optimizing memory management to prevent saturation, integrating multi-modal data (e.g., audio, text) for richer context understanding, and exploring meta-learning techniques for faster adaptation. Additionally, scaling the framework to multi-agent systems and deploying in real-world industrial environments will be key to realizing broader impacts.
AI Executive Summary
Autonomous GUI agents are essential for automating complex human-computer interactions, yet they face significant hurdles in maintaining long-term contextual understanding and adapting to dynamic environments. Existing methods often rely on limited context windows and static policies, resulting in error accumulation and poor generalization in real-world scenarios.
This paper introduces SE-GA, a novel framework that combines hierarchical memory structures with self-evolution mechanisms to address these challenges. At its core, TTME dynamically retrieves episodic, semantic, and experiential memories during inference, enabling the agent to perform long-horizon planning and decision-making. Complementing this, MASE provides a training pipeline that leverages data collected via TTME to iteratively stabilize and improve the agent’s foundational policy.
Experimental results across multiple benchmarks demonstrate that SE-GA achieves state-of-the-art success rates—89.0% on ScreenSpot and 75.8% on AndroidControl-High—significantly outperforming prior methods. These results highlight the framework’s robustness and ability to generalize across diverse, dynamic environments. The hierarchical memory design allows the agent to recall relevant past experiences and domain knowledge, reducing error propagation and enhancing decision quality.
The broader impact of this work lies in advancing autonomous, continuously learning GUI agents capable of handling real-world variability. Future work will focus on optimizing memory management, incorporating multi-modal data, and scaling to industrial applications, promising a new era of intelligent human-computer interaction systems.
Deep Analysis
Background
近年来，GUI代理技术快速发展，结合视觉语言模型（VLMs）实现了部分自动化操作（如Chen et al., 2024b），但多步任务中的信息遗忘和环境变化仍是难点。早期方法多采用行为克隆（Sun et al., 2025）或静态策略（Wang et al., 2025b），难以应对动态布局和长远规划。层次化记忆机制（Wang & Liu, 2024）虽提升了短期记忆能力，但缺乏对长远信息的系统管理。近年来，强化学习（Guo et al., 2025）和自我演化（Fang et al., 2025）成为研究热点，但在复杂、多变环境中仍存在稳定性不足的问题。
Core Problem
现有GUI代理多依赖有限的上下文窗口，难以保持完整的交互历史，导致错误累积和任务失败。同时，缺乏有效的机制重用过去成功经验，限制了在动态环境中的泛化能力。这些问题严重制约了GUI代理在实际场景中的应用效果，亟需引入更丰富的记忆结构和持续学习机制。
Innovation
本研究提出层次化记忆体系（episodic、semantic、experiential）结合TTME实现长远规划，突破了传统静态策略的局限。引入MASE训练流程，利用交互数据实现模型自我演化，增强模型的持续学习能力。创新点还在于动态检索机制和多模态记忆融合，提升了记忆的准确性和泛化能力。这些创新为GUI代理在复杂环境中的表现提供了新路径。
Methodology
- �� 构建层次化记忆库：episodic存储近期交互，semantic存储抽象规则，experiential存储成功轨迹。
- �� TTME实现：在任务执行中动态检索记忆，增强长远推理能力。
- �� MASE训练：分为基础微调（SFT）和自我演化两个阶段，利用交互数据不断优化模型。
- �� 记忆检索：采用相似度匹配（余弦、融合）实现多模态信息的高效检索。
- �� 训练策略：引入重要性比率、动态剪裁和分层奖励设计，确保训练稳定性和任务导向性。
Experiments
采用ScreenSpot、AndroidControl-High、GUIOdyssey和AndroidWorld等数据集，比较多种基线模型（如UI-TARS、Qwen2.5-VL等）。指标包括成功率、动作类型准确率和空间定位精度。训练中使用4A800 GPU，采用分阶段学习率和批次调节，进行消融实验验证各组件贡献。结果显示，SE-GA在复杂多步任务中表现优异，成功率提升显著。
Results
在ScreenSpot上，成功率达89.0%，比传统方法提升超过10%；在AndroidControl-High上，成功率达75.8%，表现出良好的环境适应性。消融实验表明，层次记忆和自我演化机制对性能提升起到关键作用。跨场景测试验证了模型的泛化能力，显示其在多样化环境中的优越表现。
Applications
可广泛应用于自动化测试、智能助理、自动化办公等场景，尤其适合需要长远规划和环境适应的复杂任务。未来可结合多模态信息，推动智能代理在工业自动化、智能家居等领域的落地。
Limitations & Outlook
当前模型在极端动态环境中仍存在记忆溢出和信息遗漏问题，训练成本较高，数据采集量大，模型在超大规模场景中的扩展性待验证。未来需优化记忆管理和模型效率，以适应更复杂的实际应用。
Plain Language Accessible to non-experts
想象你在厨房做饭。每次你做菜时，会记住之前用过的调料、菜谱和做饭的经验。偶尔，你会尝试不同的调料组合，创造新菜。这就像给厨房装上了不同的“记忆库”：一个记住你最近用过的调料（episodic），一个存放菜谱和技巧（semantic），还有一个总结你做菜经验的“经验库”。当你再次做菜时，你会从这些记忆中快速找到合适的调料和方法，做出更好吃的菜。SE-GA让电脑也像你一样不断记忆、学习、改进，变得越来越聪明，能应对各种复杂的“厨房任务”。
ELI14 Explained like you're 14
想象你在玩一个超级复杂的游戏，比如冒险游戏。每次遇到新任务或难题时，你会记住之前的经验，比如怎么打败某个怪物或找到隐藏的宝藏。你还会记住一些通用规则，比如“不要轻信陌生人”或“多探索环境”。每次遇到新挑战，你会从你的“记忆库”里找出相关的经验和规则，帮你做出更聪明的决定。这个“记忆库”就像你的脑袋，帮你记住过去的经验，学会更好应对未来的挑战。SE-GA就像让电脑变得像你一样，能不断学习、记忆和改进，变得越来越厉害，能应付各种复杂的任务，就像你在游戏中变得更强一样。
Abstract
Autonomous Graphical User Interface (GUI) agents often struggle with multi-step tasks due to constrained context windows and static policies that fail to adapt to dynamic environments. To address these limitations, this work proposes the Self-Evolving GUI Agent (SE-GA), a novel framework that integrates hierarchical memory structures with an iterative self-improvement mechanism. At the core of our approach is Test-Time Memory Extension (TTME), which facilitates long-term planning by dynamically retrieving episodic, semantic, and experiential memories to provide salient contexts during inference. To ensure continuous learning, we introduce Memory-Augmented Self-Evolution (MASE), which is a training pipeline that adopts the data collected by TTME to stabilize and enhance the agent's foundational policy. Extensive evaluations across both offline and online benchmarks demonstrate SE-GA achieves state-of-the-art performance, reaching success rates of 89.0\% on ScreenSpot and 75.8\% on the challenging AndroidControl-High dataset. Furthermore, significant improvements on the AndroidWorld benchmark highlight the superior generalization to dynamic environments. Open source code: https://github.com/jinshilong-dev/SE-GA