From Recall to Forgetting: Benchmarking Long-Term Memory for Personalized Agents
Memora benchmark and FAMA metric evaluate long-term memory in personalized agents, revealing models' struggles with memory consolidation and forgetting over months.
Key Findings
Methodology
This study introduces Memora, a benchmark simulating user interactions spanning weeks to months, covering tasks of remembering, reasoning, and recommending. It employs automated memory grounding checks and human validation to ensure data quality. The FAMA metric combines memory correctness with penalties for reliance on outdated or invalid memories. Evaluation involves four large language models (GPT-5.2, Claude, Gemini, Qwen) and six memory agents, testing their ability to maintain consistent, up-to-date memory states across extended periods.
Key Results
- Models frequently reuse invalid memories, with FAMA scores decreasing significantly at longer durations; for example, GPT-5.2’s memory accuracy drops from 25.32 weekly to 19.24 quarterly, indicating challenges in managing evolving information.
- Memory agents perform slightly better than pure models but still show substantial limitations; MemoBase’s average FAMA score drops from 43.6 weekly to 15.18 quarterly, highlighting persistent difficulties in long-term memory management.
- Introducing forgetting penalties reduces reliance on obsolete info but models still often depend on invalid memories, exposing gaps in dynamic memory updating capabilities.
Significance
This work advances beyond traditional retrieval-based benchmarks by emphasizing memory consolidation and mutation, crucial for realistic, persistent AI assistants. The FAMA metric offers a nuanced evaluation of how well models handle evolving information, addressing a key bottleneck in deploying truly long-term, personalized AI systems. It lays a foundation for future research aimed at enabling models to dynamically update, forget, and synthesize memories over extended periods, aligning AI behavior more closely with human cognition.
Technical Contribution
The paper proposes Memora, a benchmark that models long-term interactions over weeks and months, increasing the difficulty of memory consolidation and mutation. The FAMA metric explicitly penalizes reliance on invalid memories, providing a more comprehensive evaluation framework. The experimental results demonstrate that current models and memory agents struggle with maintaining consistent, up-to-date memories over extended durations, highlighting the need for improved dynamic memory mechanisms.
Novelty
This is the first comprehensive benchmark simulating multi-month conversations with explicit memory mutation and validation, moving beyond simple retrieval tasks. The FAMA metric uniquely combines correctness and forgetting penalties, providing a more realistic assessment of long-term memory management. These innovations shift the focus from static recall to dynamic, evolving memory systems, marking a significant step forward in AI research.
Limitations
- The benchmark relies on simulated interactions, which may not fully capture real user behaviors and complexities. Future work should incorporate real-world data for validation.
- Current models still struggle with frequent memory updates, especially in complex reasoning tasks, indicating the need for more sophisticated memory management algorithms.
- The evaluation primarily focuses on textual memory, with limited exploration of multi-modal or multi-source memory integration, which are critical for real-world applications.
Future Work
Future directions include integrating multi-modal data (images, audio), developing more robust memory update and conflict resolution strategies, and testing with real user data. Extending the benchmark to multi-user, multi-task scenarios will further enhance its relevance for practical deployment of persistent AI assistants.
AI Executive Summary
The rapid development of large language models has revolutionized AI capabilities, yet their long-term memory remains a significant challenge. Traditional benchmarks primarily evaluate short-term recall, failing to capture the complexities of memory consolidation, mutation, and forgetting over extended periods. This gap hampers the deployment of persistent, personalized AI agents capable of maintaining coherent, up-to-date user models.
In response, this work introduces Memora, a comprehensive benchmark that simulates multi-week to multi-month user interactions. By modeling three core tasks—remembering, reasoning, and recommending—Memora evaluates models’ abilities to integrate, update, and selectively forget information across extended timelines. The benchmark employs automated grounding checks and human validation to ensure data quality, and introduces FAMA, a novel metric that penalizes reliance on outdated or invalid memories.
Experimental results reveal that even state-of-the-art models like GPT-5.2 and advanced memory agents struggle to maintain consistent, accurate memories over longer durations. Scores decline markedly from weekly to quarterly settings, exposing fundamental limitations in current approaches to dynamic memory management. These findings underscore the importance of developing models with better memory mutation and forgetting mechanisms, essential for real-world applications requiring persistent, adaptive AI.
This research marks a significant step toward more human-like AI systems, capable of long-term learning and adaptation. It provides a rigorous framework for future innovations in memory architectures, evaluation metrics, and training strategies. Despite current limitations, the insights gained pave the way for creating AI that can truly remember, forget, and learn over extended periods, transforming personalized assistance, lifelong learning, and beyond.
Deep Analysis
Background
长时记忆在人工智能中的研究经历了从简单的记忆检索到复杂的跨会话信息整合。早期工作如PersonaChat、DSTC系列关注个性化对话中的记忆保持，逐步引入外部存储机制。近年来，随着大规模预训练模型的兴起，研究转向模型在长时间跨度内的记忆持续性和动态更新能力，代表性工作包括LoCoMo、LongMemEval等，强调跨会话信息的提取与总结。然而，这些方法多局限于短期或中期记忆，缺乏对记忆变异、遗忘机制的系统评估。现有基准如LoCoMo、MemDaily主要关注记忆的提取与重用，忽视了记忆的动态更新和遗忘处理，导致模型在实际应用中表现不稳定。长时记忆的研究尚未解决信息的持续整合、冲突解决和遗忘机制的有效实现，限制了个性化代理的实际应用。
Core Problem
当前大模型在长时间、多轮对话中表现出记忆不一致、信息过时等问题，严重影响用户体验。传统评估多关注记忆的存在与否，忽略了记忆的动态变化和遗忘机制。实际应用中，用户信息不断更新，模型需要在保持有用记忆的同时，及时遗忘无关或过时信息。缺乏系统化的评估体系，难以衡量模型在长时间、多任务、多场景中的记忆管理能力。这些问题限制了个性化代理的持续性和可靠性，亟需新的基准与指标来推动技术突破。
Innovation
本研究的核心创新包括：1）提出Memora基准，模拟真实长时间对话场景，增强记忆整合与变异的难度，突破现有短期评估的局限；2）引入FAMA指标，结合记忆有效性与遗忘惩罚，系统衡量模型对动态记忆的掌控能力；3）采用多阶段对话生成与自动验证机制，确保数据质量与评估的可靠性。这些创新使得模型在长时间、多任务、多场景中的表现得以全面评估，为未来研究提供了新的工具和思路。
Methodology
- �� 设计十个专业人物设定，涵盖偏好、活动、目标等多维信息，作为记忆模拟的基础。• 利用会话模拟器，生成跨周/月/季的用户交互，动态引入、更新或删除记忆实体，确保时间一致性。• 构建多轮对话，结合记忆操作指令，采用多智能体生成框架，确保对话内容符合记忆状态。• 自动验证环节，检测对话中是否正确表达目标记忆操作，排除偏离设定的内容。• 多模型投票与人类抽样验证，确保生成对话的质量与一致性。• 构建记忆问答集，设计记忆检索、推理与推荐任务，基于模拟记忆轨迹进行评估。• 引入FAMA指标，结合记忆存在与遗忘惩罚，量化模型在长时间跨度中的记忆管理能力。
Experiments
采用模拟多轮对话数据，覆盖不同时间尺度（周、月、季），测试四个大型语言模型及六个记忆代理的表现。指标包括FAMA得分、记忆保持率、遗忘惩罚等。对比不同模型在长时间跨度下的记忆一致性，分析其在记忆更新、冲突解决中的表现。通过消融实验验证记忆遗忘机制的影响，评估模型在复杂推理任务中的表现差异。实验还结合人类评审，确保评估的客观性与可靠性。整体设计旨在揭示模型在动态记忆环境中的实际能力。
Results
实验显示，模型在季度尺度下的记忆保持明显下降，FAMA平均得分从周度的高值降至季度的低值，表明长时间跨度内记忆管理困难。记忆代理虽略优于纯模型，但整体表现仍不足，尤其在推理任务中表现较差。引入遗忘惩罚后，模型对过时信息的依赖减少，但仍存在频繁重用无效记忆的问题。不同模型在记忆更新和冲突解决方面差异显著，验证了动态记忆管理的重要性。整体而言，研究揭示了当前模型在长时记忆中的瓶颈，为未来改进提供了明确方向。
Applications
该基准适用于开发更具人类认知能力的个性化助手、智能客服、长期陪伴机器人等场景。企业可利用Memora评估模型在实际应用中的记忆稳定性与适应性，优化模型设计。未来，结合多模态信息与真实用户数据，将推动个性化系统在教育、医疗、智能家居等领域的广泛应用，提升用户体验与系统可靠性。
Limitations & Outlook
目前评估主要基于模拟对话，可能与真实用户行为存在差异，需引入真实数据验证。模型在频繁更新的记忆中表现不足，尤其在复杂推理任务中存在较大差距。指标尚未涵盖多模态信息的动态管理，未来需扩展多源信息融合能力。此外，计算成本较高，长时间跨度的模拟与验证对硬件资源要求较大，限制了大规模应用推广。
Plain Language Accessible to non-experts
想象你在经营一家工厂，工厂每天都在生产不同的产品，员工们会记住每天的任务、客户的需求和生产的流程。随着时间推移，工厂的任务会发生变化，有些旧的流程需要忘掉，有新的需求需要记住。工厂管理者需要不断更新员工的记忆，确保他们知道最新的任务和客户信息。如果管理不善，员工可能会记错旧的任务，导致生产出错。这个工厂的管理就像人工智能中的长时记忆，模型需要学会记住重要信息、忘掉过时内容，并在需要时正确使用这些信息。研究者设计了一个模拟工厂的系统，让模型像管理者一样，处理长时间的任务变化，确保信息的准确性和及时更新。
ELI14 Explained like you're 14
你知道吗，就像你在学校里记笔记，有时候你会忘记一些旧的事情，但又会记住新发生的事情。比如，你昨天和朋友玩游戏，今天你可能会忘记一些细节，但记得你们玩了什么游戏。这个研究就像是让电脑学会像你一样记忆——它要记住重要的事情，也要忘掉不再重要的东西。科学家们设计了一个模拟场景，让电脑在几周甚至几个月的时间里不断学习和更新信息。比如，电脑要记住你的兴趣爱好、每天的活动，还要知道什么时候该忘掉一些旧的偏好。这样，电脑就能更像一个真正了解你、能陪你聊天的朋友。这项研究帮助电脑变得更聪明，能更好地记住和忘记信息，像人一样聪明地管理自己的记忆。
Glossary
Long-Term Memory (长时记忆)
指在长时间内存储和管理信息的能力，区别于短期记忆，涉及信息的存储、更新和遗忘。
论文中强调模型在长时间跨度内的记忆整合与遗忘机制。
Memory Consolidation (记忆巩固)
将短期记忆转化为长期存储的过程，确保信息在多次会话中持续可用。
评估模型在跨会话中信息的持续积累能力。
Forgetting-Aware Memory Accuracy (遗忘感知记忆准确率)
一种评估指标，衡量模型在使用有效记忆的同时，避免依赖已失效或被删除的记忆。
本文提出的核心评估指标。
Memory Mutation (记忆变异)
指在多次会话中对存储信息的更新、修改或删除过程。
衡量模型处理动态信息变化的能力。
FAMA (遗忘感知记忆准确率)
结合记忆有效性与遗忘惩罚的指标，用于评估模型在长时间跨度中的记忆管理表现。
论文中的主要评估工具。
Open Questions Unanswered questions from this research
- 1 如何在真实用户环境中验证模型的长时记忆能力，尤其是在多模态信息融合方面仍缺乏系统研究。
- 2 现有指标难以全面衡量模型在复杂动态场景中的记忆遗忘与更新能力，未来需开发更细粒度的评估体系。
Applications
Immediate Applications
个性化助手评估
企业可利用Memora检测模型在长时间交互中的记忆稳定性，优化个性化服务质量。
智能客服优化
通过评估模型的记忆管理能力，提升客服系统的连续性与用户满意度。
Long-term Vision
人类级认知系统
推动构建具备人类般记忆、推理与遗忘机制的智能体，实现更自然的交互与学习能力。
Abstract
Personalized agents that interact with users over long periods must maintain persistent memory across sessions and update it as circumstances change. However, existing benchmarks predominantly frame long-term memory evaluation as fact retrieval from past conversations, providing limited insight into agents' ability to consolidate memory over time or handle frequent knowledge updates. We introduce Memora, a long-term memory benchmark spanning weeks to months long user conversations. The benchmark evaluates three memory-grounded tasks: remembering, reasoning, and recommending. To ensure data quality, we employ automated memory-grounding checks and human evaluation. We further introduce Forgetting-Aware Memory Accuracy (FAMA), a metric that penalizes reliance on obsolete or invalidated memory when evaluating long-term memory. Evaluations of four LLMs and six memory agents reveal frequent reuse of invalid memories and failures to reconcile evolving memories. Memory agents offer marginal improvements, exposing shortcomings in long-term memory for personalized agents.