Memory as a Controlled Process: Learned Adaptive Memory Management for LLM Agents
MemCon models memory control as an MDP, using lightweight online UCB policy, achieving 15.2% success boost and 20% token savings without extra LLM calls.
Key Findings
Methodology
This work formulates memory operations as a Markov Decision Process (MDP), designing a lightweight contextual bandit policy with UCB exploration for online learning. MemCon wraps any memory backend, deciding actions such as retrieve, plan inject, re-retrieve, consolidate, and forget, based on task and memory state. It leverages prior knowledge for warm-starting Q-values, updates via binary task success feedback, and converges rapidly within tens of tasks, all without pretraining or additional LLM calls. The approach dynamically adjusts memory behavior according to task progress, balancing retrieval cost and information utility, thus enabling efficient, adaptive memory management in multi-task environments.
Key Results
- Across 6 benchmarks, 3 agent frameworks, and 3 LLM backbones, MemCon outperforms static baselines by up to 15.2% in task success rate, e.g., achieving 67.9% on ALF-World with gpt-4o. It reduces token consumption by 20%, demonstrating efficiency gains. Consistent improvements are observed across different memory backends, validating backend-agnostic design. Ablation studies show the learned control policy contributes most, with auxiliary augmented operations providing additional but smaller gains.
- The method generalizes well to diverse tasks, including interactive decision-making, QA, and web-tool use, across proprietary and open-source LLMs. It maintains robustness and efficiency, confirming its practical applicability in real-world autonomous agents.
- The approach’s simplicity, requiring no secondary LLM calls, and its rapid online convergence make it suitable for scalable deployment, offering a new paradigm for adaptive memory management that addresses long-standing challenges of static heuristics and memory inflation.
Significance
This research addresses core limitations of static memory access strategies in autonomous agents, introducing a dynamic, task-aware control layer that significantly enhances performance and efficiency. By formalizing memory management as an MDP and enabling online learning, it provides a scalable, generalizable framework that can be integrated into existing systems. The ability to adapt memory operations based on context reduces unnecessary retrieval, prevents memory overload, and improves task success, paving the way for more intelligent, autonomous systems capable of continuous learning and knowledge accumulation in complex environments.
Technical Contribution
The paper’s main technical innovation is the formalization of memory control as an MDP, solved via a lightweight contextual bandit with UCB exploration, enabling rapid online adaptation. The wrapper design decouples memory backend from control policy, making it compatible with diverse memory types. The introduction of augmented operations—plan injection and goal decomposition—addresses long-horizon challenges. The method’s theoretical guarantees include convergence within tens of tasks, with no need for pretraining or large-scale RL, making it computationally efficient and practically deployable. These advances open new avenues for scalable, adaptive memory management in AI systems.
Novelty
This is the first work to model memory access as an MDP and learn an online control policy without pretraining or secondary LLM calls. Unlike MemGPT, which promotes LLMs as controllers at high cost, MemCon employs a lightweight, task-aware bandit approach, achieving adaptability and efficiency simultaneously. Its backend-agnostic wrapper allows seamless integration with various memory systems, representing a fundamental shift from static heuristics to learned, context-sensitive strategies in memory management.
Limitations
- The current discretization of state space may limit performance in high-dimensional or continuous environments. Scalability to very long or highly dynamic task streams remains a challenge, requiring further compression or hierarchical strategies.
- While no extra LLM calls are needed during control, frequent memory operations could incur computational overhead in real-time settings, especially with complex backends.
- The approach relies on binary success feedback, which may be sparse or noisy in some real-world scenarios, potentially affecting learning stability and convergence.
Future Work
Future directions include extending the framework to continuous state spaces with deep RL, integrating hierarchical or meta-learning techniques for faster adaptation, and exploring multi-agent memory control. Additionally, combining this approach with lifelong learning paradigms could further enhance autonomous agents’ ability to accumulate and utilize knowledge over extended periods, making them more robust and versatile in complex, real-world environments.
AI Executive Summary
The rapid growth of large language models (LLMs) has fueled the development of autonomous agents capable of multi-task learning and complex reasoning. Central to their success is the external memory system, which stores past experiences and knowledge. However, existing memory access strategies are largely static, relying on fixed parameters and heuristics, which limits their adaptability across diverse task stages and environments. This rigidity results in either excessive retrieval, inflating context and cost, or insufficient access, missing reusable plans and insights.
To address these limitations, this paper introduces MemCon, a novel framework that models memory management as a Markov Decision Process (MDP). By formulating memory operations—such as retrieval, plan injection, re-retrieval, consolidation, and forgetting—as actions within this MDP, MemCon learns an online policy that dynamically adjusts memory behavior based on task progress and memory state. The core innovation lies in employing a lightweight contextual bandit with Upper Confidence Bound (UCB) exploration, which enables rapid convergence within tens of tasks without requiring pretraining or additional LLM calls. This approach is compatible with any memory backend, making it highly versatile.
Extensive experiments across six benchmarks, three agent frameworks, and three LLM architectures demonstrate that MemCon consistently outperforms static memory baselines by up to 15.2% in task success rate, while reducing token consumption by 20%. The results confirm that adaptive, task-aware memory control significantly enhances both efficiency and robustness, especially in long-horizon, multi-task scenarios. The framework’s simplicity, efficiency, and generality position it as a foundational step toward autonomous agents capable of continuous, scalable learning.
Looking ahead, future work will explore deeper integration with deep RL, hierarchical control, and lifelong learning paradigms. The goal is to create intelligent systems that not only learn from experience but also adapt their memory strategies in real-time, enabling sustained performance in increasingly complex environments. This research marks a pivotal advance in AI, bridging the gap between static heuristics and fully adaptive, context-sensitive memory management, and setting the stage for more autonomous, intelligent systems.
Deep Dive
Abstract
Large Language Model (LLM) agents increasingly rely on external memory systems to accumulate experience across tasks. Yet nearly all existing approaches, from graph-structured memories to reflective insight stores, access memory through fixed, hand-designed heuristics. We argue that this static view of memory is a core bottleneck for agentic learning because optimal memory behavior is fundamentally context-dependent. The early stages of the tasks, benefit from minimal retrieval because memory is sparse; recurring goal types benefit from plan reuse rather than generic nearest-neighbor lookup; stuck agents benefit from re-retrieval with alternative queries; and across long task streams, the memory store itself must be consolidated and pruned to remain useful. We present Memory as a Controlled Process (MemCon), a framework that models memory operations as a Markov Decision Process and learns an online policy that adaptively decides when, what, and how much to retrieve, when to inject a distilled plan, and when to consolidate or forget. MemCon is backend-agnostic: it wraps any existing memory implementation, learns from task-by-task binary feedback with no pretraining and no additional LLM calls, and uses a lightweight tabular contextual bandit with UCB exploration that converges within tens of tasks. Across 6 benchmarks, 3 agent frameworks, and 3 LLM backbones, MemCon consistently outperforms multiple memory baselines by up to 15.2 points in task success while reducing token consumption by 5--20%.