MemoryCPT: An End-to-End Agent Memory Framework for Cost-Performance Trade-off
MemoryCPT employs an end-to-end trainable memory framework combining QAD and QAR, achieving a QPC of 0.138 with significant cost reduction and answer quality improvement.
Key Findings
Methodology
MemoryCPT integrates a two-stage training process: first, Query-agnostic Distillation (QAD) compresses complex memory construction into a low-rank adaptation (LoRA) model by learning explicit reasoning traces, enabling the model to internalize memory operations such as creation, merging, and abstraction. This offline stage ensures the model captures reusable memory construction skills. Second, Query-aware Retrieval and Summarization (QAR) employs Reciprocal Rank Fusion (RRF) to perform rough retrieval of candidate memories, followed by a reinforcement learning-based (GRPO) trained LoRA summarizer that generates query-specific compressed summaries. The reward function balances answer quality, measured by F1, against inference cost, normalized as Cost per Quality (QPC). During inference, the system retrieves relevant memories via RRF, generates a concise summary conditioned on the query, and feeds it into a frozen QA model to produce answers, optimizing the cost-performance trade-off.
Key Results
- On the LoCoMo dataset, MemoryCPT with Qwen-2.5-7B achieves an F1 score of 0.479, a Judge score of 0.755, and reduces inference cost to 4.31 USD per query, representing a 28% improvement over baseline F1 (0.373), with a sevenfold reduction in cost and a QPC of 0.138, outperforming traditional memory systems significantly.
- On LongMemEval, MemoryCPT maintains an F1 of 0.482, Judge of 0.644, and cost of 5.02 USD, surpassing LightMem, MemoryOS, and BudgetMem in both quality and efficiency. The ablation studies confirm that removing QAD or QAR degrades performance, highlighting their importance. The reward design with GRPO effectively balances quality and cost, demonstrating the robustness of the approach.
- The experiments show that end-to-end training of memory operations leads to superior cost-accuracy trade-offs compared to existing methods. MemoryCPT reduces inference costs by over 80% relative to some baselines while improving answer correctness, validating the effectiveness of combining explicit reasoning traces with reinforcement learning for cost-aware memory management.
Significance
This work addresses a critical bottleneck in deploying large-scale dialogue agents: balancing answer quality with inference cost. By introducing an end-to-end trainable memory pipeline that explicitly optimizes for cost-performance, MemoryCPT advances the state-of-the-art in long-term memory management for language models. It demonstrates that integrating structured reasoning, reinforcement learning, and parameter-efficient adaptation (LoRA) can produce systems that are both accurate and economical. The approach paves the way for scalable, real-world applications such as customer service, virtual assistants, and knowledge management systems, where maintaining high-quality responses at low cost is essential. Furthermore, the framework's modular design and explicit reward mechanism offer a blueprint for future research in cost-aware AI systems.
Technical Contribution
The core technical innovation lies in the combination of explicit reasoning trace distillation, structured memory operations, and reinforcement learning-based query-aware summarization within an end-to-end trainable pipeline. QAD leverages teacher-student training with structured JSON outputs to encode memory operations, enabling the model to learn reusable skills for constructing episodic and semantic memories efficiently. QAR employs RRF for rough retrieval and a GRPO-trained LoRA summarizer to generate query-specific compressed memories, guided by a reward balancing F1 and inference cost. The use of QPC as a unified metric guides the optimization process, ensuring that the system improves both answer quality and cost efficiency. This integrated approach offers a new paradigm for scalable, interpretable, and cost-effective long-term memory management in large language models.
Novelty
This paper pioneers the end-to-end training of a multi-stage memory pipeline that explicitly models memory construction and retrieval operations, guided by structured reasoning traces and reinforcement learning rewards. Unlike prior work relying on heuristics or manual prompts, MemoryCPT learns reusable memory skills offline and applies query-aware compression online, achieving a dynamic balance between information retention and cost. The integration of QAD’s explicit reasoning supervision, RRF-based retrieval, and GRPO-based summarization within a unified framework represents a significant departure from existing methods, establishing a new standard for cost-efficient long-term memory in dialogue agents.
Limitations
- The training process depends heavily on annotated reasoning traces and structured outputs, which require substantial manual effort and high-quality data, potentially limiting scalability in low-resource settings.
- While effective in multi-turn dialogues, the approach may face challenges in scenarios with extremely long histories or highly complex reasoning tasks, where information may still be lost or insufficiently captured.
- The end-to-end training and reinforcement learning components increase system complexity and computational costs, which could hinder real-time deployment or adaptation to new domains without further optimization.
Future Work
Future research will focus on reducing dependence on annotated reasoning traces, exploring unsupervised or semi-supervised methods for memory operation learning. Extending the framework to incorporate multi-modal data such as images or videos could broaden its applicability. Additionally, developing more efficient training algorithms and model compression techniques will facilitate deployment in resource-constrained environments. Further, enhancing interpretability and robustness, especially in adversarial or noisy settings, remains an important direction. The integration of continual learning mechanisms to adapt memory representations over time without catastrophic forgetting is also a promising avenue.
AI Executive Summary
In the rapidly evolving field of artificial intelligence, large language models (LLMs) have demonstrated remarkable capabilities in understanding and generating human-like text. However, their effectiveness in long-horizon, multi-turn dialogues remains constrained by the challenge of managing extensive interaction histories. Traditional approaches often rely on feeding entire conversation histories into the model, which leads to prohibitive inference costs and noisy inputs, ultimately limiting scalability and responsiveness.
Addressing this bottleneck, the MemoryCPT framework introduces an innovative, end-to-end trainable memory pipeline designed to optimize the trade-off between answer quality and inference cost. This pipeline is composed of two core components: offline Query-agnostic Distillation (QAD) and online Query-aware Retrieval and Summarization (QAR). QAD focuses on learning a compact, reusable memory construction skill set by distilling explicit reasoning traces generated by a teacher model. This process results in a low-rank adaptation (LoRA) model that efficiently encodes episodic and semantic memories, enabling rapid offline memory building.
The online component, QAR, employs a two-step retrieval and compression strategy. First, it uses Reciprocal Rank Fusion (RRF) to perform a rough retrieval of relevant memory candidates from a structured store, combining dense embedding similarity and sparse keyword matching. Then, a reinforcement learning-trained LoRA summarizer, guided by a reward function balancing F1 accuracy and inference cost, generates a query-specific compressed memory. This concise summary is fed into a frozen question-answering model, which produces the final response.
Experimental results on two benchmarks, LoCoMo and LongMemEval, demonstrate that MemoryCPT significantly outperforms existing memory systems. Specifically, on LoCoMo, it achieves an F1 score of 0.479, surpassing baseline methods by 28%, while reducing inference costs by over 80%. The QPC metric, which measures answer quality per unit cost, reaches 0.138, indicating superior efficiency. Ablation studies confirm that both the offline distillation and online query-aware compression are crucial for optimal performance.
This work marks a substantial advancement in long-term dialogue management, showing that explicit, structured reasoning combined with reinforcement learning can produce memory systems that are both accurate and cost-effective. Its modular design and explicit reward mechanisms open avenues for further improvements in scalable, interpretable AI systems. Looking ahead, integrating multi-modal data, reducing training dependencies, and enhancing real-time deployment are promising directions to extend this framework’s impact across various AI applications.
Deep Analysis
Background
The development of large-scale pre-trained language models (PLMs) such as GPT, LLaMA, and Qwen has revolutionized natural language understanding and generation. Early dialogue systems relied on limited context windows, which constrained their ability to handle long-term dependencies. To overcome this, researchers introduced various memory augmentation techniques, including summarization-based compression (Kim et al., 2026a), hierarchical memory organization (Milosevic, 2026), and multi-modal integration (Zhou et al., 2024). Despite these advances, existing systems often suffer from high inference costs, lack of end-to-end training, and limited interpretability. Recent efforts incorporating reinforcement learning (e.g., Memory-R1) have shown promise in adaptive memory management but still face challenges in balancing accuracy and efficiency. Overall, the field has recognized the importance of developing scalable, interpretable, and cost-efficient long-term memory solutions for dialogue agents.
Core Problem
The core challenge in long-horizon dialogue systems is managing extensive interaction histories without incurring excessive inference costs or losing critical information. Feeding entire histories into LLMs is computationally expensive and often introduces noise, reducing response quality. Existing methods like static summarization or heuristic retrieval lack adaptability and often result in either redundant information or critical data omission. The fundamental problem is designing a memory system that can dynamically select, compress, and utilize relevant information in a cost-effective manner while maintaining high answer accuracy. Achieving this requires a system capable of learning structured memory operations, optimizing retrieval strategies, and balancing answer quality against inference costs in an end-to-end manner.
Innovation
This paper introduces several key innovations:
- �� End-to-end trainable memory pipeline: Integrates offline memory construction with online query-specific retrieval and compression, enabling joint optimization.
- �� Query-agnostic Distillation (QAD): Uses teacher-generated reasoning traces to train a compact LoRA model that learns to perform memory operations such as creation, merging, and abstraction, ensuring reusable skills.
- �� Query-aware Retrieval and Summarization (QAR): Combines RRF-based rough retrieval with reinforcement learning (GRPO) trained summarizer to generate query-specific compressed memories, balancing answer quality and inference cost.
- �� Cost-aware reward design: Incorporates a novel metric, QPC, to guide the model toward optimal cost-performance trade-offs.
- �� Modular architecture: Separates offline memory building from online retrieval, improving scalability and interpretability.
These innovations collectively enable the system to perform efficient, accurate, and interpretable long-term memory management, addressing the limitations of prior approaches.
Methodology
- �� Offline memory construction (QAD):
- Input: Raw dialogue history H.
- Process: Teacher model (e.g., Qwen-3) generates explicit reasoning traces and structured JSON outputs indicating memory operations.
- Supervision: The student model (Qwen-2.5-7B-Instruct with LoRA-A) is trained to mimic these structured outputs via causal language modeling, optimizing the loss in Equation 9.
- Output: A compact, reusable memory construction model that encodes episodic and semantic memories.
- �� Online query processing (QAR):
- Input: User query q and memory store M.
- Retrieval: Use RRF to fuse dense embedding similarity and sparse BM25 keyword matching, retrieving top candidates.
- Compression: The GRPO-trained LoRA-B summarizer generates a query-specific concise summary mq conditioned on q and retrieved memories.
- Answer generation: The frozen QA model consumes (q, mq) to produce the final answer.
- Reward optimization: The summarizer's policy is trained via GRPO, with the reward combining F1 score and normalized inference cost (Equation 16).
- �� Training pipeline:
- Stage 1: Train LoRA-A on structured reasoning traces, then merge into base model.
- Stage 2: Use RRF to retrieve candidate memories, train LoRA-B with GRPO to generate summaries guided by the reward.
- During inference: retrieve, summarize, and answer without sampling or gold answer dependence.
- �� Technical details:
- Use of structured JSON outputs to represent memory operations.
- Explicit supervision of reasoning traces to improve interpretability.
- Cost modeling via token counts and reward balancing to optimize the cost-performance trade-off.
Experiments
- �� Datasets: Evaluation conducted on LoCoMo and LongMemEval, both designed for long-horizon multi-turn dialogues. LoCoMo includes 8 conversations with 314 questions, while LongMemEval contains 150 training, 98 validation, and 105 test questions, covering diverse categories.
- �� Baselines: Compared against No-Memory, LightMem, MemoryOS, BudgetMem, and Memory-R1, ensuring a comprehensive evaluation of different memory management strategies.
- �� Metrics: F1 score for answer correctness, Judge (LLM-based semantic correctness), inference cost in USD per question, and the derived QPC metric.
- �� Hyperparameters: Utilized Qwen-2.5-7B for memory encoding, trained LoRA-A with rank 16, learning rate 1e-4, dropout 0.05, and merged into baseA. QAR employed GRPO with α=0.8, top-20 retrievals, and 50 summaries.
- �� Ablation studies: Removing QAD or QAR significantly degraded performance, confirming their roles. Varying reward coefficients impacted the cost-quality balance, demonstrating the effectiveness of the reward design.
Results
- �� MemoryCPT achieves an F1 of 0.479 on LoCoMo, outperforming baseline methods by approximately 28%, with a cost of 4.31 USD/问，QPC提升至0.138，显著优于传统记忆系统。
- �� 在LongMemEval上，F1达0.482，成本为5.02美元/问，整体表现优越，优于LightMem、MemoryOS等方法。
- �� 消融分析显示，去除QAD导致F1下降20%，答题质量明显受损；去除QAR则成本大幅上升，答题准确率下降，验证了两阶段训练的协同作用。
- �� 通过奖励机制调节参数（如α），模型在不同场景下都能实现较优的成本-性能折中，验证了设计的有效性。
Applications
- �� 立即应用：该方法适用于企业客服、智能助手等场景，能够在多轮对话中保持高答题质量，同时显著降低推理成本。
- �� 长远愿景：未来将结合多模态信息（如图像、视频）扩展记忆系统的能力，支持多任务、多场景的知识管理，推动智能对话系统的商业化和普及。这一技术也可应用于知识库维护、智能问答和个性化推荐等领域，提升系统的响应速度和准确性。
Limitations & Outlook
- �� 训练依赖大量标注的推理轨迹和结构化输出，数据准备成本高，限制在低资源场景的推广。
- �� 在极端长文本或复杂推理任务中，信息可能遗失或推理不充分，仍需优化信息整合机制。
- �� 端到端训练和强化学习增加系统复杂度和计算成本，可能影响实时部署和模型的可解释性，未来需探索更高效的训练和压缩技术。
Plain Language Accessible to non-experts
想象你在管理一个超级大的图书馆，里面有成千上万的书和资料。当有人问你某个问题时，你不会每次都翻遍所有书，因为那样太慢也太累。相反，你提前把重要的内容整理成简短的笔记和标签，放在一个方便查找的地方。当有人问起时，你只需要快速查找相关的标签，然后用提前准备好的笔记回答他。MemoryCPT的工作原理就像这个图书管理员：它在后台提前学习如何把大量的对话内容整理成简洁的笔记（QAD阶段），然后在有人提问时，快速检索相关的笔记，并用它们生成答案（QAR阶段）。这样既节省时间，又保证回答的准确性。这个系统就像一个聪明的学生，提前准备好所有资料，遇到问题时就能快速找到答案，不用每次都从头开始查资料。它的聪明之处在于提前整理和压缩信息，让复杂的对话变得简单又高效，既省钱又能答得好。
ELI14 Explained like you're 14
想象你在学校的图书馆里，有很多书和资料。当你需要写作文或回答问题时，你不会每次都翻遍所有书，因为那太慢了。相反，你会提前做一些笔记，把重要的内容写在便签上。每次有人问你问题时，你只需要看一下这些便签，然后用它们回答。这就像MemoryCPT：它在后台提前学习如何整理和压缩大量的对话内容（QAD阶段），这样在真正需要回答时，就可以快速找到相关信息，并用它们生成答案（QAR阶段）。这样既节省时间，也能保证答案的准确。它就像一个聪明的学生，提前准备好资料，遇到问题就能快速答出来，不用每次都从头查资料。未来，这样的系统还能结合图片、视频等多种信息，让我们的智能助手变得更聪明、更贴心！
Abstract
Long-horizon LLM agents require memory systems that recover useful evidence from large interaction histories without passing excessive context to downstream models. Existing memory pipelines often rely on hand-crafted heuristics and repeated LLM calls, which can introduce redundant context and high inference cost. We propose MemoryCPT, an end-to-end trainable agent memory pipeline that spans offline memory construction and online query-conditioned context generation. MemoryCPT consists of two stages: Query-agnostic Distillation (QAD), which distills a modular memory-construction pipeline into a compact model using explicit reasoning traces; and Query-aware Retrieval and Summarization (QAR), which combines reciprocal rank fusion (RRF) with a LoRA-based summarizer trained via Group Relative Policy Optimization (GRPO) under a cost-aware reward. We further introduce Quality per Cost (QPC) to quantify answer quality per unit inference cost. Experiments on LoCoMo and LongMemEval show that MemoryCPT improves the cost-performance trade-off over the evaluated baselines, while ablation and sensitivity analyses characterize the contributions of its components and the effects of key design choices.
References (13)
DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models
Zhihong Shao, Peiyi Wang, Qihao Zhu et al.
LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive Memory
Di Wu, Hongwei Wang, Wenhao Yu et al.
Contextual Agentic Memory is a Memo, Not True Memory
Binyan Xu, Xilin Dai, Kehuan Zhang
Learning to summarize from human feedback
Nisan Stiennon, Long Ouyang, Jeff Wu et al.
Reflexion: language agents with verbal reinforcement learning
Noah Shinn, Federico Cassano, Beck Labash et al.
Compress-Distill: Reasoning Trace Compression for Efficient Knowledge Distillation
Maxime Griot, Paul S. Scotti, Tanishq Mathew Abraham
Lost in the Middle: How Language Models Use Long Contexts
Nelson F. Liu, Kevin Lin, John Hewitt et al.
A-MEM: Agentic Memory for LLM Agents
Wujiang Xu, Zujie Liang, K. Mei et al.
EvoEmbedding: Evolvable Representations for Long-Context Retrieval and Agentic Memory
Chang Nie, Chaoyou Fu, Junlan Feng et al.
Beyond RAG for Agent Memory: Retrieval by Decoupling and Aggregation
Zhanghao Hu, Qinglin Zhu, Hanqi Yan et al.
SimpleMem: Efficient Lifelong Memory for LLM Agents
Jiaqi Liu, Yaofeng Su, Peng Xia et al.
M$^3$Exam: Benchmarking Multimodal Memory for Realistic User-Agent Interactions
Zhengjun Huang, Wenxuan Liu, Zhoujin Tian et al.
What Deserves Memory: Adaptive Memory Distillation for LLM Agents
Wenquan Ma, Jiayan Nan, Wenlong Wu et al.