MementoGUI: Learning Agentic Multimodal Memory Control for Long-Horizon GUI Agents
MementoGUI enhances long-horizon GUI agents with learned multimodal memory control, improving task success rates by over 20% on benchmarks.
Key Findings
Methodology
This paper introduces MementoGUI, a framework that integrates a frozen GUI backbone with a learned memory controller, MementoCore, which manages online multimodal memory through selective writing, compression, and retrieval. MementoCore modularizes these operations into four operators: step processing, memory compression, episodic writing, and relevance-based episodic selection, trained via large-scale automated data curation from computer-use trajectories. The system maintains a working memory for transient task states and episodic memory for reusable past experiences, enabling dynamic, relevance-driven memory management without fine-tuning the backbone. Experiments on GUI-Odyssey, MM-Mind2Web, and MementoGUI-Bench demonstrate consistent improvements over baselines, with larger memory modules further boosting performance.
Key Results
- On GUI-Odyssey with UI-Venus-1.5-8B, integrating working memory increased action match from 54.58% to 67.69%, and trajectory success from 1.29 to 2.69; adding episodic memory further improved these metrics to 68.32% and 3.57, outperforming no-history and text-only baselines.
- In MM-Mind2Web, the memory-augmented models achieved significant gains in step success rate, indicating better long-term task management.
- Across all benchmarks, memory control improved semantic action matching (VAM), task progress (TPS), and memory consistency (MCS), confirming the robustness and generality of the approach.
Significance
This work addresses the core challenge of long-horizon GUI control—effective management of multimodal, long-term state information. By shifting from passive history replay to active, relevance-based memory control, it advances autonomous GUI agents capable of handling complex, multi-step tasks. The approach offers a scalable, plug-in solution that enhances existing models without retraining their core policies, opening new avenues for scalable automation and intelligent interface interaction in industry and research.
Technical Contribution
The key technical innovation lies in the design of a trainable, modular memory controller that manages multi-scale, multimodal memory streams. It introduces a relevance-based episodic retrieval mechanism, combined with structured working memory, all integrated via a unified input serialization. The training pipeline leverages automated data curation and preference-based fine-tuning (DPO), enabling scalable, task-specific optimization without backbone finetuning. This design offers a flexible, extensible architecture for long-horizon decision-making in multimodal environments.
Novelty
This is the first work to explicitly formulate long-horizon GUI control as an active, multimodal memory management problem, rather than relying solely on raw history or unstructured text memory. The integration of relevance-based episodic retrieval with structured working memory, trained via automated supervision, represents a significant departure from prior static or rule-based memory systems, providing a scalable, generalizable solution for complex interactive tasks.
Limitations
- The system heavily depends on large-scale automated data collection, which may limit adaptability to unseen environments or novel interface designs.
- Memory compression and relevance filtering might omit critical details in extremely long or complex trajectories, potentially affecting decision accuracy.
- Computational overhead for large memory modules and retrieval operations could impact real-time performance in resource-constrained settings.
Future Work
Future directions include developing adaptive, reinforcement learning-based memory relevance policies, enhancing transferability to unseen tasks, and integrating meta-learning for rapid adaptation. Exploring more efficient memory compression techniques and extending the framework to multi-agent scenarios are also promising avenues.
AI Executive Summary
Managing long-term, multimodal information in GUI agents remains a fundamental challenge. Existing approaches often rely on passive history replay or simple text-based memory, which are insufficient for complex, multi-step tasks requiring persistent state awareness. This paper introduces MementoGUI, a novel framework that actively manages multimodal memory through a trainable controller, MementoCore. By modularizing memory operations into four operators—step processing, compression, episodic writing, and relevance-based episodic retrieval—the system dynamically preserves only task-relevant interface events and visual cues. This relevance-driven approach enables the agent to maintain a concise, yet comprehensive, multimodal context that supports long-horizon decision-making.
The core innovation is the automated, scalable data curation pipeline that converts raw computer-use trajectories into supervised training data for the memory controller. This pipeline leverages rule-based corruption and vision-language model filtering to generate preference pairs, facilitating effective learning of memory operations without fine-tuning the backbone GUI model. The experimental results across multiple benchmarks demonstrate that integrating this active memory control significantly boosts performance metrics such as action matching and task success rate. For example, on GUI-Odyssey with UI-Venus-1.5-8B, the method improves action match from 54.58% to 68.32% and trajectory success from 1.29 to 3.57.
These findings underscore the importance of active, relevance-based memory management in long-horizon GUI tasks. The approach not only advances the state-of-the-art in GUI automation but also provides a scalable, plug-in solution adaptable to various models and environments. Looking ahead, future work will focus on enhancing the adaptability and efficiency of memory relevance policies, aiming to realize fully autonomous, long-term interactive agents capable of complex multi-modal reasoning and decision-making in real-world scenarios.
Deep Analysis
Background
随着深度学习和多模态模型的发展，GUI自动化已成为研究热点。早期方法多依赖规则或模板，后续引入视觉理解与语言模型，显著提升了交互能力。代表性工作包括UI-Venus、GUI-Owl和MAI-UI等，解决了单步理解与动作预测问题。然而，长时序任务中的信息遗失、重复与决策困难仍未根本解决。近年来，记忆机制如知识库、技能库和强化学习记忆被引入，但多模态、结构化、长时序管理仍是挑战。
Core Problem
长轨迹GUI任务中，关键界面事件和视觉证据容易被遗忘或被冗余信息淹没，导致代理在复杂场景下表现不稳定。传统方法多依赖固定历史或文本记忆，难以动态筛选有用信息，限制了长时序任务的完成能力。如何主动管理多模态记忆，提升信息的相关性和紧凑性，成为核心难题。
Innovation
提出MementoGUI框架，结合结构化多模态记忆管理，创新点包括：
- �� 引入学习的记忆控制器，动态筛选界面事件与视觉证据；
- �� 设计多操作模块（步骤处理、压缩、episodic写入与筛选），实现无缝集成；
- �� 利用自动化数据采集训练记忆策略，无需微调基础模型；
- �� 结合偏好学习优化检索与筛选效果，提升记忆的相关性与效率。
Methodology
- �� 采用预训练的GUI骨架（如UI-Venus、GUI-Owl）作为基础模型，冻结参数。
- �� 构建多模态记忆控制器（MementoCore），包括：
- 步骤处理：识别界面事件的重要性，生成事件摘要和ROI。
- 记忆压缩：对旧记忆进行总结，控制记忆体积。
- episodic写入：将完成的轨迹转存为可重用的记忆单元。
- episodic筛选：利用学习的相关性模型筛选最有用的过去经验。
- �� 训练过程：通过自动采集的交互轨迹，构建监督数据，采用结构化微调（SFT）和偏好偏差优化（DPO）进行训练。
- �� 在多个公开基准（GUI-Odyssey、MM-Mind2Web、MementoGUI-Bench）上评估，指标包括动作匹配、任务成功率及记忆一致性。
Experiments
- �� 使用GUI-Odyssey、MM-Mind2Web和自建的MementoGUI-Bench，涵盖移动端和Web环境。
- �� 评估指标包括：动作匹配（AMS）、轨迹成功率（Traj. SR）、步骤成功率（Step SR）、语义动作匹配（VAM）、任务进展（TPS）和记忆一致性（MCS）。
- �� 比较基线包括无历史、纯文本记忆和纯历史重放。
- �� 采用不同规模的记忆控制器（如8B、32B）验证模型容量影响。
- �� 进行消融实验，验证ROI级别的视觉信息和相关性筛选的重要性。
Results
- �� 在GUI-Odyssey中，UI-Venus-1.5-8B模型加入记忆后，动作匹配率从54.58提升至68.32，轨迹成功率从1.29提升至3.57，优于无记忆和纯文本记忆基线。
- �� 在MM-Mind2Web中，记忆增强模型在Step SR上提升显著，表现出长时序任务中的优势。
- �� 在MementoGUI-Bench上，VAM、TPS和MCS指标均显示加入记忆后，模型在语义一致性和任务推进方面表现更优，验证了记忆控制的有效性。
Applications
- �� 立即应用：可部署于智能助理、自动化测试、软件操作自动化等场景，提升复杂界面交互的自主性与效率。
- �� 长远愿景：推动自主系统在教育、医疗、工业自动化等领域实现更高层次的智能交互，减少人工干预，提升生产力。
Limitations & Outlook
- �� 依赖大量自动采集的训练数据，迁移到新环境或未覆盖场景时效果有限。
- �� 长轨迹中，信息压缩与筛选存在关键细节丢失风险。
- �� 在极端复杂界面或多任务场景中，记忆管理的效率和准确性仍需优化。
Plain Language Accessible to non-experts
想象你在一个复杂的厨房里做饭。每次你做菜时，都会遇到不同的食材、工具和步骤。有些事情你会记在脑子里，比如你刚切完菜，有些细节可能会被遗忘，比如调料放在哪了。为了做出好菜，你需要记住哪些步骤重要，哪些可以省略，还要记得之前用过的调料。MementoGUI就像一个聪明的厨房助手，它会帮你记住重要的步骤和细节，甚至会把一些过去的经验整理成小卡片，随时帮你回忆。这样，你就不用担心忘记关键步骤，也能更快完成复杂的菜谱。它通过学习，知道什么时候该记忆，什么时候该忘记，确保你每次都能顺利做出美味佳肴。
ELI14 Explained like you're 14
想象你在玩一个超级复杂的游戏，每次你要完成任务，比如找到宝藏或打败大Boss。游戏里有很多线索和提示，有时候你会忘记之前看到的线索，或者被一些无关的东西分心。MementoGUI就像你的智能助手，它会帮你记住重要的线索和关键的提示，把它们整理成小卡片，随时提醒你。它还能根据你之前的经验，告诉你哪些线索最有用，帮你更快找到宝藏。这样，你就不用担心忘记重要信息，也能更聪明地完成任务。它学习你的游戏习惯，知道什么时候该记忆，什么时候可以不用管，帮你变得更厉害！
Abstract
Recent GUI agents have made substantial progress in visual grounding and action prediction, yet they remain brittle in long-horizon tasks that require maintaining task state across many interface transitions. Existing agents typically rely on raw history replay or text-only memory, which either overwhelms the model with redundant screenshots or discards localized visual evidence needed for future decisions. To address these limitations, we introduce \textbf{MementoGUI}, a plug-in agentic memory framework that equips MLLM-based GUI agents with \textbf{MementoCore}, a learned controller for online memory selection, compression, and retrieval. Rather than treating interaction history as a fixed context, MementoGUI formulates long-horizon GUI control as an online memory-control problem: working memory selectively preserves task-relevant interface events with textual summaries and ROI-level visual evidence, while episodic memory retrieves reusable past trajectories through learned relevance selection. MementoCore modularizes memory control into specialized operators for step processing, memory compression, episodic writing, and episodic selection, enabling plug-in memory augmentation without finetuning the GUI agent backbone. We further develop a scalable data curation pipeline that converts computer-use trajectories into memory-controller training data, introduce \textbf{MementoGUI-Bench} for evaluating long-horizon decision-making in GUI agents, and design MLLM-based metrics for semantic action matching, task progress, and memory consistency. Experiments on GUI-Odyssey, MM-Mind2Web, and MementoGUI-Bench show that MementoGUI consistently improves GUI agents over no-history, history-replay, and text-only memory baselines, with larger MementoCore backbones further strengthening memory-augmented GUI control.