Harness the Memory: A Holistic Evaluation of Memory Substrates in Memory Agents
Holistic evaluation of 11 memory substrates across models and tasks reveals task-dependent performance and scalability trade-offs, advocating multi-substrate routing.
Key Findings
Methodology
The study employs a unified evaluation framework testing 11 diverse memory substrates—including flat vector indices, graph-based structures, hierarchical stores, and parametric tuning—across three backbone models (Qwen-8B, Qwen-32B-AWQ, GEMMA-4-26B-A4B-IT) and four benchmarks covering user and agent-centric tasks. Metrics include accuracy, success rate, latency, and storage costs. By controlling for variables, the analysis isolates the impact of each memory substrate on performance and efficiency, examining retrieval depth, scalability, and task-specific advantages. Ablation studies adjust retrieval breadth (k) and context length to understand their effects on attention and task outcomes.
Key Results
- No single memory substrate dominates across all tasks; for long-context QA, graph and hierarchical memories (e.g., M5) outperform flat indices (e.g., M2), achieving up to 648% accuracy improvement. Conversely, in sequential decision tasks, simpler memory (M2) offers better speed and cost efficiency. Increasing retrieval depth (k) boosts QA performance (e.g., M3 reaches 0.562 P4 on LoCoMo), but hampers decision-making by diverting attention (e.g., M5’s TSR drops to 4.5%). Scalability tests show parametric tuning (M9) performs well short-term but degrades with longer histories, while structured memories (M4, M5) scale better. These results highlight the importance of dynamic memory routing to adapt to task demands.
- The experiments demonstrate that task-specific memory needs vary: broad retrieval benefits factual QA, but excessive retrieval impairs decision tasks. Attention probing reveals that as retrieval depth increases, the model’s focus shifts away from current observations, reducing decision accuracy. Long-term scalability favors structured memories, which maintain performance over extended histories. The findings advocate for a multi-substrate memory system with adaptive routing, balancing retrieval breadth and write depth, to optimize performance across diverse scenarios.
- Overall, the study emphasizes that no single memory structure suffices; instead, a dynamic, multi-modal routing system is essential for robust, efficient long-term memory in LLMs. This approach addresses fundamental limitations of existing methods, paving the way for more adaptable, scalable AI agents capable of complex, sustained reasoning.
Significance
This research provides a comprehensive, empirical foundation for designing adaptive memory architectures in large language models. By quantifying the trade-offs between different memory substrates—such as retrieval speed, scalability, and task-specific accuracy—it informs the development of systems capable of dynamically selecting or combining memory types based on operational regimes. Such flexibility is crucial for deploying AI in real-world applications requiring long-term knowledge retention, multi-task handling, and resource efficiency. The insights gained address longstanding challenges in AI memory systems, bridging the gap between theoretical models and practical deployment, and fostering progress toward autonomous, continually learning agents.
Technical Contribution
The paper introduces a novel, controlled evaluation framework that systematically compares diverse memory substrates in a unified setting, across multiple models and tasks. It develops a multi-modal routing mechanism that dynamically allocates memory resources based on task demands, supported by extensive ablation and scalability analyses. The work advances theoretical understanding of retrieval-depth effects and scalability limits, providing concrete design principles for multi-substrate memory architectures. These contributions enable more flexible, efficient, and scalable long-term memory systems for large language models, facilitating their deployment in complex, real-world scenarios.
Novelty
This is the first comprehensive, multi-model, multi-task evaluation of diverse memory substrates under a unified framework, explicitly analyzing their performance, efficiency, and scalability trade-offs. The introduction of multi-modal memory routing as a dynamic scheduling mechanism distinguishes this work from prior static approaches. The empirical insights into how different memory structures perform across regimes, especially in long-horizon tasks, represent a significant advancement in understanding and designing adaptive memory systems for AI agents.
Limitations
- The evaluation focuses primarily on models like Qwen and GEMMA, limiting generalization to other architectures. Broader testing is needed to confirm universality.
- Hardware constraints (GPU clusters) influence scalability results; real-world deployment may face additional resource limitations.
- Some memory structures (e.g., parametric tuning) show poor scalability in extremely long sequences, requiring further optimization or hybrid approaches.
Future Work
Future research will explore integrating multi-modal data (visual, auditory) into the routing framework, enhancing multi-task adaptability. Developing more efficient compression and retrieval algorithms will address scalability challenges. Additionally, extending evaluations to more diverse models and real-world scenarios will validate the robustness of multi-substrate systems, ultimately enabling AI agents capable of sustained, autonomous learning in dynamic environments.
AI Executive Summary
The rapid evolution of large language models (LLMs) has underscored the importance of effective memory mechanisms for long-term knowledge retention and task adaptability. Traditional approaches often rely on single memory structures—such as dense vector indices, knowledge graphs, or parametric weights—each with inherent limitations in scalability, speed, or flexibility. This fragmentation hampers the development of truly adaptive AI agents capable of managing diverse tasks across varying operational regimes.
In response, this study introduces a comprehensive evaluation framework that systematically compares 11 different memory substrates—including flat indices, graph structures, hierarchical storages, and parameter tuning—across three state-of-the-art models and four benchmark tasks. The evaluation controls for variables such as model architecture and prompting, isolating the impact of each memory structure on performance and efficiency. Metrics cover accuracy, success rate, latency, storage costs, and retrieval overhead, providing a detailed landscape of strengths and weaknesses.
Results reveal that no single memory substrate excels universally. For long-context question answering, graph and hierarchical memories (e.g., M5) significantly outperform flat indices, achieving up to 648% accuracy improvements. Conversely, in sequential decision tasks, simpler memory structures (e.g., M2) offer better speed and lower costs. Increasing retrieval depth enhances QA performance but impairs decision-making by diverting attention from current observations. Scalability analyses show structured memories maintain performance over longer histories, while parametric tuning struggles with extreme sequences.
These insights highlight the necessity of a dynamic, multi-modal memory routing system that can adaptively select and combine different memory substrates based on task demands. Such a system balances retrieval breadth and write depth, optimizing both performance and resource utilization. The findings pave the way for more flexible, scalable, and reliable AI agents capable of sustained learning and complex reasoning in real-world environments, addressing key challenges in AI memory design.
Deep Analysis
Background
The development of large language models has transitioned from static, single-task systems to dynamic agents requiring persistent memory. Early methods like fine-tuning and external retrieval (e.g., RAG, knowledge graphs) addressed some issues but faced scalability and flexibility limitations. Recent advances explore diverse memory structures—vector indices, graph-based stores, hierarchical summaries, and parameter-based weights—yet lack systematic comparison. As tasks grow in complexity, the need for adaptive, multi-modal memory systems becomes evident. Prior work has shown that no single structure suffices across all scenarios, prompting research into dynamic routing and hybrid architectures. This paper situates itself within this context, aiming to empirically evaluate and guide the design of versatile memory systems for long-horizon AI agents.
Core Problem
The core challenge lies in balancing the trade-offs between memory accuracy, retrieval speed, scalability, and resource costs across diverse tasks. Existing systems often optimize for specific regimes, leading to suboptimal performance elsewhere. For example, dense vector indices excel in quick retrieval but falter with multi-hop queries; knowledge graphs provide relational reasoning but are costly to construct; parametric weights store knowledge internally but struggle with long-term updates. The difficulty is compounded by task-specific demands—question answering benefits from broad recall, while decision-making requires precise, action-critical information. Developing a unified, adaptive memory system that can dynamically select or combine these structures remains an open problem.
Innovation
The paper's key innovations include: 1) a unified, controlled evaluation framework testing 11 diverse memory substrates across multiple models and benchmarks, providing a comprehensive performance landscape; 2) the introduction of multi-modal memory routing, enabling dynamic scheduling of different structures based on task context; 3) extensive ablation studies analyzing retrieval depth, scalability, and task-specific performance, revealing nuanced trade-offs. These innovations facilitate understanding how different memory structures perform under varying regimes, guiding the design of adaptive, multi-substrate systems capable of balancing accuracy, efficiency, and scalability in complex, real-world scenarios.
Methodology
- �� Implement 11 memory methods spanning seven families: flat vector indices (M1, M2), text records (M3), structural graphs (M4, M5), hierarchical trees (M6), refinement-based memories (M7, M8), and internal weights (M9, M10, M11). Each method encodes and retrieves information differently.
- �� Use three backbone models (Qwen-8B, Qwen-32B-AWQ, GEMMA-4-26B-A4B-IT) and four benchmarks (LoCoMo, LongMemEval, ALFWorld, BigCode) for comprehensive testing.
- �� Control variables such as prompt templates, auxiliary LLMs, and interaction histories; vary retrieval depth (k) and context length to analyze effects.
- �� Measure 26 metrics, including accuracy, success rates, latency, storage costs, call counts, and attention shifts.
- �� Conduct ablation experiments to isolate effects of retrieval breadth, memory scale, and update semantics.
- �� Perform scalability tests simulating long sequences (up to 262K tokens) to evaluate performance degradation and resource costs.
- �� Analyze attention mechanisms via probing to understand how retrieval depth influences focus and task outcomes.
Experiments
The experimental setup involves four benchmarks: LoCoMo (multi-session dialogues), LongMemEval (long-context factual QA), ALFWorld (embodied planning), and BigCodeBench-Hard (code generation). Each benchmark tests different aspects of memory utility—recall, reasoning, action planning, and code reuse. Models are evaluated on metrics such as exact match, F1, success rate, latency, token usage, and call counts. Retrieval depth (k) is varied systematically to assess its impact on performance and efficiency. Long sequence tests extend input length to 262K tokens, examining scalability. Ablation studies compare different memory structures, revealing their strengths and weaknesses across tasks. Results are visualized in performance-latency trade-off plots, emphasizing the importance of adaptive routing for optimal deployment.
Results
The results demonstrate task-dependent memory preferences: graph and hierarchical memories (M5) excel in long-context QA, achieving up to 648% accuracy gains, while flat indices (M2) are more cost-effective in decision tasks. Increasing retrieval depth improves QA metrics (e.g., M3 reaches 0.562 P4) but reduces decision success (e.g., TSR drops to 4.5%). Scalability analysis shows structured memories maintain performance over longer histories, whereas parametric tuning (M9) degrades rapidly beyond 262K tokens. Attention probing indicates that excessive retrieval shifts focus away from current observations, impairing decision-making. The study advocates for multi-modal routing, dynamically selecting memory structures based on task regime, to optimize performance and resource use.
Applications
This research informs the design of adaptive, multi-task AI systems capable of long-term knowledge retention and complex reasoning. Practical applications include intelligent assistants, autonomous agents, and decision support systems that dynamically select memory structures to balance speed, accuracy, and scalability. The framework supports real-time multi-modal data integration, enabling AI to operate effectively in diverse environments, from customer service to scientific research. Long-term, it paves the way for autonomous, continually learning agents that can handle evolving knowledge bases and complex workflows with minimal human intervention.
Limitations & Outlook
The evaluation is primarily limited to models like Qwen and GEMMA, requiring validation across broader architectures. Hardware constraints impact scalability results, and some memory methods (e.g., parametric tuning) show poor long-term performance. The current framework does not incorporate multi-modal data fusion, which is essential for real-world deployment. Future work should address these gaps by testing more diverse models, optimizing algorithms for resource efficiency, and integrating multi-modal data to enhance adaptability and robustness.
Plain Language Accessible to non-experts
想象你在一个大型图书馆里管理各种不同的书架。有的书架很快就能找到你需要的书（像简单索引），但容量有限；有的书架用复杂的关系网连接不同的书（像图结构），可以快速找到相关书；还有的书架会把书按主题分层整理（像层级存储），方便长时间查阅。每次你要找资料时，会根据任务的不同，选择不同的书架。有时候，翻阅太多书反而会让你迷失方向（像过度检索影响注意力），因此，你需要聪明地调度这些书架，让你既能快速找到答案，又不浪费时间。这就像论文中提出的多模态记忆调度机制，旨在让人工智能像这个聪明的图书馆管理员一样，灵活调配不同的记忆底层，满足不同任务的需要。
ELI14 Explained like you're 14
想象你在学校图书馆找资料。有时候，你只需要一本书的内容（像问答任务），你会快速找到相关书架上的书；但有时候，你需要了解很多相关信息（像决策任务），就得翻阅很多书。这就像不同的记忆方式：一种是快速找到答案的索引（平坦索引），另一种是用关系网连接所有信息的图（图结构），还有一种是把书按主题分层整理（层级存储）。如果你不停地翻找很多书，反而会让你迷失方向，不能专注于任务（类似过度检索影响注意力）。所以，图书馆的管理员（模型）需要聪明地调配不同的书架（记忆底层），根据任务需要选择最合适的方式。这样，既能快速找到答案，又能保持效率，图书馆的管理就变得更智能、更高效了。
Abstract
Memory is becoming core infrastructure for long-horizon LLM agents, yet existing evaluations offer limited guidance on which memory substrate, namely the underlying medium in which memory is represented and stored, should be used under different operating regimes. We present a controlled harness evaluation of memory substrates for memory-augmented agents, covering dense and sparse indices, text records, structural stores, hierarchical stores, refinement-based memories, parametric updates, and activation-compatible context mechanisms. Across three backbone models and four benchmark suites spanning user-centric question answering and agent-centric decision-making, we instrument 26 performance and efficiency metrics under a unified harness. Our results show that no single substrate consistently dominates: broad retrieval benefits long-context factual QA, while excessive retrieval can harm sequential decision-making by shifting attention away from action-critical context. Scalability introduces a further routing axis, as substrates that perform well at moderate history lengths can become costly or brittle at longer horizons. These findings motivate substrate routing as a necessary component of adaptive agent memory systems and provide empirical guidance for designing efficient, reliable, and regime-aware long-term memory for LLM agents. Code will be made available upon acceptance.