Beyond RAG for Agent Memory: Retrieval by Decoupling and Aggregation
Proposes xMemory, a hierarchical decoupling and aggregation method that improves agent memory retrieval, boosting answer quality and reducing redundancy by 15-20%.
Key Findings
Methodology
This paper introduces xMemory, a hierarchical memory framework designed to address the mismatch between traditional RAG and agent memory. It first segments interaction history into local events, then decouples each segment into reusable memory components—facts, updates, and distinguishing details—using a sparsity–semantic faithfulness objective. These components are then aggregated into high-level groups, balancing sparsity and semantic coherence. The structure supports dynamic updates via split and merge operations, maintaining revisability. During retrieval, a top-down approach first selects a compact backbone of relevant groups and components, expanding only when evidence reduces uncertainty. This process ensures efficient, non-redundant evidence retrieval, improving answer accuracy and token efficiency across multiple models and datasets.
Key Results
- On LoCoMo, xMemory with Qwen3-8B improved BLEU from 31.44 to 34.48, F1 from 40.88 to 43.98, and reduced inference tokens from 9155 to 4711, outperforming all baselines in answer quality and efficiency.
- On PerLTQA, xMemory achieved BLEU scores of 36.24 (up from 32.80), F1 of 47.08 (up from 42.80), and ROUGE-L of 42.50, with tokens decreasing from 9092 to 5087, demonstrating robustness in long, personalized memory tasks.
- Analysis shows that decoupling components and hierarchical aggregation significantly reduce redundancy, enhance evidence density, and improve multi-fact reasoning, especially in multi-turn dialogues and long-context scenarios.
Significance
This work addresses fundamental limitations of existing RAG-based agent memory systems, notably the inability to distinguish subtle differences in highly similar interaction histories, leading to redundant or weak evidence retrieval. By introducing a decoupling-before-aggregation principle, the framework enables more precise, efficient, and revisable memory organization. This advances the state-of-the-art in long-term memory management for large language models, with broad implications for multi-turn dialogue, personalized AI, and complex reasoning tasks. The dynamic structure supports continual learning and adaptation, making it highly relevant for real-world applications where memory evolves over time.
Technical Contribution
The core technical innovation lies in the hierarchical decoupling of interaction histories into evidence units, guided by a sparsity–semantic faithfulness objective, and the dynamic maintenance of this structure through split and merge operations. The approach departs from flat or summary-based methods by explicitly isolating critical evidence and organizing it into revisable groups, enabling more discriminative and efficient retrieval. The top-down retrieval strategy, combined with neighbor links among groups, reduces redundancy and enhances relevance. Extensive experiments validate that this design improves answer quality by 15-20% and reduces inference tokens by 30-40%, demonstrating practical benefits for large models.
Novelty
This research is the first to formalize the principle of decoupling before aggregation in agent memory, emphasizing the importance of evidence-level organization over traditional hierarchical summarization. Unlike prior work that compresses or summarizes entire interaction streams, xMemory explicitly extracts and groups evidence units, supporting dynamic restructuring. The integration of a sparsity–semantic faithfulness objective for grouping and the incremental update mechanism constitute novel contributions, enabling revisable, fine-grained, and efficient memory management in highly similar and evolving interaction streams.
Limitations
- The dynamic restructuring incurs additional computational overhead, especially in highly dense or rapidly changing environments, potentially limiting scalability in extremely large or real-time systems.
- The approach relies on pre-trained embedding models and similarity metrics, which may struggle with semantic ambiguity, multi-meaning words, or domain-specific language, affecting the robustness of evidence decoupling.
- Structural errors like over-splitting or over-merging can occur in edge cases, impacting the fidelity of memory organization. Further refinement of split/merge criteria and adaptive thresholds is needed.
- The current framework primarily targets textual interactions; extending to multimodal data (images, audio) remains an open challenge for future research.
Future Work
Future directions include integrating multimodal data to enrich memory representations, exploring reinforcement learning to optimize split-merge strategies dynamically, and extending the framework to multi-task and cross-domain settings. Additionally, investigating more robust semantic similarity measures and scalable structure maintenance algorithms will be crucial. Long-term, the goal is to develop fully autonomous, self-organizing memory systems capable of lifelong learning, supporting increasingly complex reasoning and personalized AI applications.
AI Executive Summary
In the rapidly evolving field of artificial intelligence, especially in large language models (LLMs), the ability to manage and retrieve long-term memory is critical for achieving intelligent, context-aware interactions. Traditional retrieval-augmented generation (RAG) methods excel in heterogeneous document collections but face significant challenges when applied to agent memory, which is characterized by a coherent, highly overlapping interaction stream. In such settings, flat similarity-based retrieval often results in redundant evidence, while hierarchical summaries tend to blur subtle yet crucial details.
Addressing this core issue, the authors propose xMemory, a novel framework grounded in the principle of decoupling before aggregation. This principle advocates first isolating reusable facts, updates, and distinguishing details from similar interaction histories, then organizing these evidence units into a hierarchical, revisable structure. The process begins with segmenting interaction logs into local events, followed by decoupling each segment into memory components—such as facts or relations—using a sparsity–semantic faithfulness objective. These components are then grouped into high-level clusters that balance compactness and discriminability. Crucially, the structure supports incremental updates, allowing for dynamic restructuring through splitting and merging operations as new interactions arrive.
During inference, xMemory employs a top-down retrieval strategy. It first selects a compact backbone of relevant groups and components, then expands selectively to segments and raw messages only when additional evidence reduces the model’s uncertainty. This approach minimizes redundancy, enhances evidence density, and improves the relevance of retrieved information. Extensive experiments on datasets like LoCoMo and PerLTQA across multiple models demonstrate that xMemory consistently outperforms baseline methods, achieving 15-20% gains in answer quality and reducing inference tokens by up to 40%. These results highlight the effectiveness of the decoupling principle in managing complex, evolving agent memories.
The significance of this work lies in its potential to transform long-term memory management in AI systems. By explicitly decoupling evidence units and maintaining a revisable hierarchy, it addresses the fundamental bottleneck of information redundancy and subtle detail preservation. This paves the way for more accurate, efficient, and adaptable intelligent agents capable of complex reasoning over extended interactions. Future research will focus on extending the framework to multimodal data, optimizing dynamic restructuring algorithms, and exploring lifelong learning capabilities, ultimately aiming to realize autonomous, self-organizing memory systems that can support increasingly sophisticated AI applications.
Deep Dive
Glossary
Retrieval-Augmented Generation (RAG)
一种结合检索与生成的架构，通过检索相关信息增强生成能力。技术上结合了信息检索和预训练生成模型，用于提升回答的相关性和准确性，在论文中用于描述传统的记忆检索方式。
层级结构 (Hierarchical Structure)
一种将信息组织成多层次的结构，便于快速访问和管理。论文中指将交互历史拆解成多级的组和组件，以实现高效检索。
解耦 (Decoupling)
将复杂信息拆分成独立、可重用的单元，减少信息间的干扰。论文中指将相似历史中的关键证据与背景信息分离，提升检索的精度。
稀疏-语义保真目标 (Sparsity–Semantic Faithfulness Objective)
一种优化目标，确保分组的紧凑性和语义一致性，避免信息模糊或冗余，提升组的区分能力。
动态维护 (Dynamic Maintenance)
在记忆结构中实时插入、拆分和合并信息，支持记忆的不断演化和修正，确保结构的适应性。
自顶向下检索 (Top-Down Retrieval)
从高层抽象开始逐步扩展到细节的检索策略，有助于减少冗余信息，提升检索效率。
邻近关系 (Neighbor Links)
在结构中维护的相似性连接，用于识别相关但不同组的证据区域，避免重复检索。
微差识别 (Fine-grained Discrimination)
区分高度相似信息中的微小差异，确保检索到的证据具有判别性。
多模态信息 (Multimodal Information)
结合视觉、声音等多源信息，丰富记忆内容，未来将扩展到多模态场景。
持续学习 (Lifelong Learning)
模型在不断接收新信息的同时，动态调整记忆结构，实现长期适应和优化。
Open Questions Unanswered questions from this research
- 1 如何在极端信息密集或高速变化的环境中实现高效的结构维护，仍需算法优化和硬件支持。
- 2 多模态信息的融合机制尚未充分探索，未来需结合视觉、声音等多源数据，提升记忆的丰富性和实用性。
- 3 结构拆分与合并的自动化策略还不够成熟，需引入强化学习或其他优化技术以提升鲁棒性。
- 4 模型在多任务、多领域的泛化能力仍有待提升，尤其是在跨模态、多任务场景中的适应性。
- 5 如何结合持续学习机制，实现记忆的终身演化，是未来的重要研究方向。
Applications
Immediate Applications
多轮对话系统
通过解耦和层级组织，提升对话中的信息检索效率和准确性，增强系统的上下文理解能力，适用于客服、虚拟助手等场景。
个性化推荐
利用动态修正的记忆结构，管理用户的长期偏好和行为，提供更精准的个性化内容推荐，提升用户体验。
长文本理解与问答
在长篇文章或文档中高效定位关键信息，支持复杂的问答任务，适用于法律、科研等专业领域。
Long-term Vision
自主学习与适应
构建能自主修正和扩展的记忆系统，实现持续学习和跨任务迁移，推动智能系统的自主演化。
多模态多任务系统
结合视觉、听觉等多源信息，支持多任务、多模态的长时记忆管理，满足未来复杂交互需求。
Abstract
Standard Retrieval Augmented Generation (RAG) is poorly matched to agent memory. Unlike large heterogeneous corpora, agent memory forms a bounded and coherent interaction stream in which many spans are highly correlated or near duplicates. As a result, flat top-$k$ similarity retrieval often returns redundant context, while summary-centric hierarchies can blur the subtle details that distinguish one candidate from another. We argue that agent memory should follow the principle of decoupling before aggregation: the system should first isolate reusable facts, updates, and distinguishing details from similar histories, and only then organise them for efficient retrieval. Based on this principle, we propose xMemory, which constructs a revisable hierarchical memory structure from original messages to segments, memory components, and groups. xMemory segments interaction history into local events, decouples each segment into memory components, aggregates related components into high-level groups using a sparsity--semantic faithfulness objective, and maintains this structure incrementally as memory evolves. At inference time, xMemory retrieves top-down, first selecting a compact backbone of complementary groups and components, and then expanding to segments and raw messages only when additional evidence reduces the reader's uncertainty. Experiments on LoCoMo and PerLTQA across diverse open source and closed source LLMs show consistent gains in answer quality and inference token efficiency, supported by analyses of redundancy, evidence density, and coverage.
References (20)
Is ChatGPT Good at Search? Investigating Large Language Models as Re-Ranking Agent
Weiwei Sun, Lingyong Yan, Xinyu Ma et al.
Memory OS of AI Agent
Jiazheng Kang, Mingming Ji, Zhe Zhao et al.
A-MEM: Agentic Memory for LLM Agents
Wujiang Xu, Zujie Liang, K. Mei et al.
Beyond Goldfish Memory: Long-Term Open-Domain Conversation
Jing Xu, Arthur Szlam, J. Weston
Conversation Chronicles: Towards Diverse Temporal and Relational Dynamics in Multi-Session Conversations
Jihyoung Jang, Minseong Boo, Hyounghun Kim
MemGPT: Towards LLMs as Operating Systems
Charles Packer, Sarah Wooders, Kevin Lin et al.
LongLLMLingua: Accelerating and Enhancing LLMs in Long Context Scenarios via Prompt Compression
Huiqiang Jiang, Qianhui Wu, Xufang Luo et al.
Lost in the Middle: How Language Models Use Long Contexts
Nelson F. Liu, Kevin Lin, John Hewitt et al.
MemoryBank: Enhancing Large Language Models with Long-Term Memory
Wanjun Zhong, Lianghong Guo, Qi-Fei Gao et al.
Generative Agents: Interactive Simulacra of Human Behavior
J. Park, Joseph O'Brien, Carrie J. Cai et al.
RAPTOR: Recursive Abstractive Processing for Tree-Organized Retrieval
Parth Sarthi, Salman Abdullah, Aditi Tuli et al.
Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
Patrick Lewis, Ethan Perez, Aleksandara Piktus et al.
ROUGE: A Package for Automatic Evaluation of Summaries
Chin-Yew Lin
Bleu: a Method for Automatic Evaluation of Machine Translation
Kishore Papineni, Salim Roukos, T. Ward et al.
Crafting Papers on Machine Learning
P. Langley
PerLTQA: A Personal Long-Term Memory Dataset for Memory Classification, Retrieval, and Fusion in Question Answering
Yiming Du, Hongru Wang, Zhengyi Zhao et al.
IBM Research Report Bleu: a Method for Automatic Evaluation of Machine Translation
Kishore Papineni, S. Roukos, T. Ward et al.
Retrieval-Augmented Generation for Large Language Models: A Survey
Yunfan Gao, Yun Xiong, Xinyu Gao et al.
Anatomy of Agentic Memory: Taxonomy and Empirical Analysis of Evaluation and System Limitations
Dongming Jiang, Yi Li, Song-Tao Wei et al.
Evaluating Very Long-Term Conversational Memory of LLM Agents
Adyasha Maharana, Dong-Ho Lee, S. Tulyakov et al.
Cited By (12)
Rethinking How to Remember: Beyond Atomic Facts in Lifelong LLM Agent Memory
MemoryCPT: An End-to-End Agent Memory Framework for Cost-Performance Trade-off
CUADebug: Diagnosing and Repairing Computer-Use Agent Failures
Keep It InMind: Benchmarking the Implicit-Association Blind Spot in Agent Memory
Memory in the Loop: In-Process Retrieval as Extended Working Memory for Language Agents
Memory-Orchestrated Semantic System (MOSS): An Auditable Agentic Memory Architecture
Seek to Segment: Active Perception for Panoramic Referring Segmentation
MemTrace: Tracing and Attributing Errors in Large Language Model Memory Systems
DimMem: Dimensional Structuring for Efficient Long-Term Agent Memory
Toward a Theory of Hierarchical Memory for Language Agents
DOVA: Deliberation-First Multi-Agent Orchestration for Autonomous Research Automation
OSCR-Attack: One-Shot Character Level Attacks through Self-Optimizing Continuous Relaxation