STAMP: Training Explicit Memory for Mobile GUI Agents in Controllable and Scalable Virtual Environments
STAMP framework trains explicit memory for mobile GUI agents using controllable virtual environments, enhancing Memory-World benchmark performance.
Key Findings
Methodology
STAMP framework trains explicit memory in mobile GUI agents through controllable virtual environments. Deterministic memory variables are programmatically injected to control what must be memorized, when it should be encoded, and when it must be retrieved, generating verifiable supervised data and enabling online reinforcement learning via environment-driven reward feedback.
Key Results
- Stamp-GUI excels on the Memory-World benchmark with a memory accuracy of 77.9% and task success rate of 75.3%.
- Compared to other GUI models, Stamp-GUI shows stronger task completion ability across multiple attempts.
- Ablation studies confirm the contribution of virtual environments and step balancing mechanisms to performance improvement.
Significance
This research addresses the memory deficit in mobile GUI agents for long-duration tasks, significantly enhancing task resilience and memory accuracy, providing a novel solution for intelligent agents in mobile applications.
Technical Contribution
STAMP framework generates memory-intensive tasks using controllable virtual environments, offering new theoretical guarantees and engineering possibilities, distinct from existing methods, especially in automatic verification of memory prediction and task execution.
Novelty
STAMP is the first framework to train explicit memory using controllable virtual environments, offering finer memory control and verification compared to existing action-centric datasets.
Limitations
- Virtual environments cannot fully capture the diversity and dynamics of real mobile applications, potentially affecting real-world transfer.
- Relies on multiple strong models, increasing computational and engineering costs.
- Does not cover broader forms of memory such as long-term user preferences or cross-task memory.
Future Work
Future research could explore virtual-to-real transfer effects, reduce pipeline costs, and expand the scope of memory modeling and evaluation.
AI Executive Summary
Mobile GUI agents often fail in long-duration tasks due to memory deficits. Existing methods cannot effectively solve this problem as they rely on static data and lack interactive verification. The STAMP framework trains explicit memory through controllable virtual environments, programmatically injecting memory variables to control content and timing. Experiments show Stamp-GUI excels on the Memory-World benchmark, significantly enhancing memory accuracy and task resilience. This research provides a novel solution for intelligent agents in mobile applications, but challenges remain in bridging virtual environments with real applications.
Deep Analysis
Background
Mobile GUI agents aim to assist users by directly perceiving and interacting with smartphone interfaces. However, these agents often fail in long-duration tasks due to memory deficits. Existing methods rely on static data and lack interactive verification, unable to effectively solve this problem.
Core Problem
Mobile GUI agents often fail in long-duration tasks due to memory deficits, unable to effectively store and retrieve task-specific information. Solving this problem is crucial for enhancing the task resilience of intelligent agents.
Innovation
STAMP framework trains explicit memory through controllable virtual environments, programmatically injecting memory variables to control content and timing. Compared to existing methods, it offers finer memory control and verification mechanisms.
Methodology
- �� Use controllable virtual environments to generate memory-intensive tasks
- �� Programmatically inject memory variables to control content and timing
- �� Enable online reinforcement learning via environment-driven reward feedback
- �� Generate verifiable supervised data to enhance memory accuracy
Experiments
Experiments evaluate Stamp-GUI's performance using the Memory-World benchmark. Ablation studies confirm the contribution of virtual environments and step balancing mechanisms to performance improvement. Results show Stamp-GUI excels in memory accuracy and task success rate.
Results
Stamp-GUI excels on the Memory-World benchmark with a memory accuracy of 77.9% and task success rate of 75.3%. Compared to other GUI models, Stamp-GUI shows stronger task completion ability across multiple attempts.
Applications
The framework can enhance the task resilience and memory accuracy of intelligent agents in mobile applications, particularly in tasks requiring long-duration memory.
Limitations & Outlook
Virtual environments cannot fully capture the diversity and dynamics of real mobile applications, potentially affecting real-world transfer. Relies on multiple strong models, increasing computational and engineering costs.
Plain Language Accessible to non-experts
Imagine you are cooking in a kitchen. You need to remember each step of the recipe, but the kitchen is small, and you can't keep all ingredients on the table. STAMP acts like a smart assistant, reminding you what to do next and helping you remember important steps. Even if you forget some details, it helps you complete the entire cooking process.
ELI14 Explained like you're 14
Imagine you're playing a game like Minecraft, where you need to remember lots of information, like where ores are or where monsters lurk. STAMP is like a super memory assistant, helping you remember this info and reminding you when needed. This way, you can find ores faster, defeat monsters, and become a game pro!
Glossary
Virtual Environment
A computer-generated environment that simulates the real world, used for testing and training algorithms.
Used to generate memory-intensive tasks for training mobile agents.
Explicit Memory
A form of memory that can be explicitly called and used, typically for storing task-related information.
STAMP framework enhances task resilience through explicit memory.
Reinforcement Learning
A machine learning method that trains models through reward and punishment mechanisms.
Used to optimize memory encoding actions online.
Ablation Study
An experimental method to evaluate the impact of model components by removing or altering them.
Used to verify the contribution of virtual environments and step balancing mechanisms.
Task Resilience
The ability of agents to maintain efficiency and accuracy in long-duration tasks.
Enhanced through explicit memory in STAMP framework.
Open Questions Unanswered questions from this research
- 1 How to implement memory-intensive tasks generated by virtual environments in real applications?
- 2 How to reduce computational and engineering costs of the STAMP framework?
- 3 How to expand memory modeling to cover long-term user preferences?
Applications
Immediate Applications
Mobile Application Intelligent Agents
Enhance task resilience and memory accuracy of intelligent agents in mobile applications, particularly in tasks requiring long-duration memory.
Long-term Vision
Cross-Application Memory System
Develop systems that share memory across multiple applications, enhancing user experience and task efficiency.
Abstract
Mobile GUI agents excel at immediate reactive control but frequently fail in realistic, long-horizon tasks that require memory. This failure stems from a fundamental conflict between limited context windows and token-heavy screenshots. To save the limited context, agents must progressively discard older visual history, permanently losing crucial transient information. Furthermore, existing action-centric datasets fail to teach agents what or when to explicitly memorize, and augmenting static real-world data is prohibitively expensive and lacks interactive verification. To resolve this, we present STAMP, a framework that trains explicit memory in mobile agents through controllable virtual environments, where deterministic memory variables are programmatically injected into synthesized tasks to control what must be memorized, when it should be encoded, and when it must later be retrieved, thereby producing verifiable supervised data at scale and enabling online reinforcement learning through environment-driven reward feedback. Evaluated on our newly introduced Memory-World benchmark, the resulting Stamp-GUI agent achieves state-of-the-art performance among GUI-specialized models and sets a new high watermark on our Memory-World benchmark, demonstrating exceptional memory accuracy and task resilience while maintaining strong general mobile navigation capabilities.