StreamMemBench: Streaming Evaluation of Agent Memory for Future-Oriented Assistance
StreamMemBench evaluates memory's ability to support future tasks from egocentric streams, focusing on evidence retention, initial use, feedback integration, and reuse, with detailed metrics.
Key Findings
Methodology
This work introduces StreamMemBench, a benchmark built on Egocentric Lifelog data, constructing evidence anchors linked to two-step task sequences. It evaluates memory systems across four metrics—Fidelity, IEU, FI, and FUR—by simulating interactions with eight models (e.g., MemoryOS, A-Mem) on two backbone architectures (DeepSeek-V4-Flash, Gemini-3-Flash). The benchmark tracks evidence from storage through initial response, feedback, and follow-up reuse, providing a comprehensive assessment of how well systems support future-oriented assistance in streaming scenarios. The evaluation involves simulating user feedback, analyzing the path from evidence retention to application, and diagnosing failures at each stage.
Key Results
- Despite some systems achieving 100% Fidelity (evidence retention), their performance on initial evidence use (IEU) and follow-up reuse (FUR) metrics remains low (~30%), indicating stored evidence often fails to influence actual task responses. For example, Mem0 and A-Mem show strong evidence retention but weak reuse, while MemOS stores evidence well but exhibits poor application performance.
- Feedback incorporation (FI) is relatively high across models, yet the transition from feedback to behavior adjustment (FUR) remains weak, revealing a gap between storing interaction corrections and applying them in future tasks.
- Temporal analysis shows evidence retention remains stable, but actual use declines over time, especially in late stream anchors, highlighting the challenge of maintaining effective memory across extended interactions.
Significance
This benchmark addresses a critical gap by systematically evaluating how personal agents retain and utilize streaming observations for future assistance. It emphasizes the importance of not just storing information but ensuring its effective application, which is vital for real-world AI deployment. The detailed, path-traced metrics enable precise diagnosis of failure points, guiding future improvements in memory architectures. The approach advances both academic understanding and industry practices, fostering development of more reliable, context-aware personal assistants capable of continuous learning and adaptation.
Technical Contribution
The paper proposes a novel evaluation framework combining egocentric lifelog data with multi-stage, path-traced metrics, capturing evidence retention, initial use, feedback integration, and reapplication. It introduces evidence anchors as a core concept, enabling fine-grained tracking of information flow. The methodology involves constructing task sequences grounded in real-world data, simulating user feedback, and analyzing system performance across diverse architectures and models. This comprehensive, multi-dimensional assessment surpasses traditional static benchmarks, providing a new standard for evaluating memory systems in streaming, real-world scenarios.
Novelty
This is the first benchmark explicitly designed to evaluate the entire trajectory from streaming egocentric observations to future task support, emphasizing evidence grounding and experience reuse. Unlike prior static dialogue or profile-based benchmarks, it incorporates real-world continuous data, simulating dynamic environments. The multi-stage, path-traced metrics provide unprecedented insight into how well systems retain, utilize, and adapt stored information over time, setting a new standard for real-time memory evaluation.
Limitations
- The benchmark relies on simulated user feedback, which may not fully capture the variability and complexity of real user interactions, potentially limiting ecological validity.
- Evaluation is primarily focused on short-term, single-session scenarios; long-term, multi-session performance remains to be validated.
- The computational cost of maintaining and evaluating multiple models across large streaming datasets poses practical challenges for deployment.
Future Work
Future research will incorporate multimodal data sources (visual, auditory) to enrich evidence anchors, aiming to improve robustness in real-world settings. Developing more efficient memory management algorithms to reduce storage and computation costs is also a priority. Extending the benchmark to multi-turn, multi-modal, multi-task environments will better reflect practical applications. Additionally, integrating user-specific personalization mechanisms could enhance the system’s ability to adapt over time, making personal agents more intuitive and reliable.
AI Executive Summary
The rapid advancement of large language models has brought personal assistants closer to human-like capabilities, yet their ability to remember and utilize streaming observations remains limited. Traditional benchmarks focus on static dialogue recall or profile management, neglecting the dynamic, continuous nature of real-world data streams. Recognizing this gap, researchers developed StreamMemBench, a novel benchmark designed to evaluate how well memory systems can support future-oriented assistance from egocentric lifelog data.
Built upon the EgoLife dataset, which captures daily life through wearable sensors, StreamMemBench constructs evidence anchors—key pieces of user-specific information—linked to two-step task sequences. Each sequence involves an initial assistance request grounded in the evidence, followed by a simulated user feedback, and a subsequent related task. This setup allows for a detailed, path-traced evaluation of four core metrics: evidence fidelity (storage accuracy), initial evidence use, feedback incorporation, and follow-up reuse.
Experimental results across eight memory systems and two backbone models reveal significant gaps. While many systems can store evidence with high fidelity, their ability to leverage this evidence in actual task responses, especially in follow-up scenarios, remains weak. Feedback integration improves within a single turn but often fails to translate into reliable future behavior. These findings underscore the complexity of maintaining and applying dynamic information in streaming environments.
This work provides a comprehensive framework for diagnosing and improving memory in AI assistants, emphasizing the importance of continuous, real-time evaluation. It highlights that effective memory management must go beyond storage, focusing on the entire trajectory from observation to application. Future directions include multimodal data integration, efficiency enhancements, and multi-turn, multi-modal testing, all aimed at creating smarter, more adaptive personal assistants capable of lifelong learning and seamless support.
Deep Analysis
Background
人工智能中的记忆机制经历了从简单存储到复杂推理的演变。早期系统如ELIZA依赖静态对话存储，后续如Memory Networks引入外部存储，增强了上下文理解能力。近年来，基于大模型的对话系统（如ChatGPT、Gemini）强调个性化和持续交互，但仍面临信息遗忘和迁移困难。代表性工作包括MemoryOS、A-Mem等，强调多层次、多粒度存储，试图解决信息碎片化问题。尽管如此，现有评估多集中于静态对话回忆，缺乏对连续流式观察中信息动态变化的关注，限制了系统在真实场景中的应用效果。
Core Problem
当前记忆系统在持续、多模态环境中表现有限，尤其在证据的动态保持与有效利用方面存在明显缺陷。系统虽能存储信息，但在实际应用中，存储的内容未必能被有效调用，导致任务支持不足。尤其在未来导向的场景中，系统需要从连续观察中提取关键证据，并在后续任务中复用，但现有方法缺乏系统性评估机制，难以衡量其实际能力。这一问题阻碍了个性化助理的深度发展，亟需建立更科学、全面的评估体系。
Innovation
本研究创新点在于提出基于egocentric lifelog的流式评估基准，结合证据锚点设计多阶段任务，系统性追踪信息从存储到应用的全过程。引入四维指标体系（证据保持、初次使用、反馈整合、复用），突破传统静态评估的局限，强调动态信息流的连续性。通过模拟用户反馈，验证系统在不同阶段的表现差异，为未来优化提供数据支撑。此方法创新性地将多模态、多任务场景融入评估框架，推动个性化助理在真实环境中的应用。
Methodology
- �� 构建证据锚点：利用Egocentric Lifelog数据，将每段连续观察转化为锚点，提取用户相关信息，关联对应任务。
- �� 设计两步任务：每个锚点支持初次任务（基于锚点证据）和后续任务（验证复用能力），模拟用户反馈。
- �� 评估指标：引入证据保持（Fidelity）、证据使用（IEU）、反馈整合（FI）和复用（FUR），全方位衡量系统性能。
- �� 实验设置：采用八个不同记忆系统（如MemoryOS、A-Mem等），在两种基础模型（DeepSeek-V4-Flash、Gemini-3-Flash）上测试，结合模拟用户反馈进行多轮评估。
- �� 追踪路径：每个锚点从存储、响应、反馈到复用，形成完整路径，分析系统在不同阶段的表现差异。
Experiments
数据来源于Egocentric Lifelog（Yang et al., 2025），涵盖7天连续观察，划分为3347个五分钟段，每段含12个观察点。提取7760个证据锚点，生成15520个任务查询，涉及多种日常任务。实验中，系统依次处理时间序列数据，响应任务，模拟用户反馈，并将交互存入记忆。采用多模型、多基础模型对比，评估指标包括Fidelity、IEU、FI、FUR，结合不同存储策略，分析在不同时间点的表现差异。通过多轮实验验证指标的有效性和系统的实际应用能力。
Results
实验显示，尽管部分系统在Fidelity指标（证据保持）达100%，但在实际任务中的证据利用（IEU）和复用（FUR）表现明显不足，平均仅为30%左右，说明存储信息未能有效转化为支持行为。Mem0和A-Mem在证据保持方面表现优异，但在后续任务中复用能力不足，MemOS虽存储稳定，但在实际应用中证据利用率低于预期。反馈整合（FI）指标在大部分系统中表现较好，但在从反馈到行为调整路径上仍存在断层。这些结果揭示了存储与应用之间的脱节，强调了全路径追踪的重要性。
Applications
本基准适用于个性化助理、智能家居、医疗辅助等场景，能帮助系统更好地理解用户环境，持续优化交互策略。未来可结合多模态数据（视觉、声音）丰富证据源，提升系统在复杂环境中的适应性。行业中，企业可利用此评估体系优化产品设计，增强用户体验，实现更智能的个性化服务。长远来看，推动持续学习和跨任务迁移，将使智能助理更贴近人类日常需求。
Limitations & Outlook
目前评估主要基于模拟用户反馈，实际用户行为的多样性和复杂性未能完全覆盖，可能影响指标的泛化性。系统在长时间、多任务、多模态环境中的表现尚未充分验证，未来需扩展多轮、多模态交互的评估框架。模型存储与计算成本较高，实际部署仍面临效率与规模的挑战。
Plain Language Accessible to non-experts
想象你在一家工厂工作，每天都在收集各种原料和信息，比如工人们的操作、机器的状态、生产的产品。工厂需要记住这些信息，才能在需要时快速找到关键原料或调整生产流程。现在，如果工厂只把信息存起来，但不去用它，遇到问题时也找不到解决办法。这个工厂就像一个没有智能的记忆系统。研究者设计了一套方法，让工厂不仅存信息，还能在需要时用上这些信息，甚至在遇到新问题时还能借鉴以前的经验。通过模拟工厂的操作，发现很多系统虽然存了信息，但在真正用起来时却表现不好，就像工厂记住了原料，但不知道什么时候用、怎么用。这个研究帮助我们理解，未来的智能系统要像一个聪明的工厂，不仅记住事情，还能灵活运用，解决实际问题。
ELI14 Explained like you're 14
想象你在学校里，有一个超级聪明的朋友，他不仅记得你说过的话，还能在你需要帮忙时马上用上这些信息。有时候，他记得你喜欢的游戏、喜欢吃的零食，还记得你上次说过的烦恼。可是，有时候他记得这些信息，却忘了怎么帮你用上，或者忘了你上次说过的事情。这就像一些智能助手，它们存储了很多信息，但在你需要帮助时，却不能很好地用上这些信息。这个研究就像在帮这些助手变得更聪明，让它们不仅记住事情，还能在对话中用得上，甚至在遇到类似问题时还能借鉴以前的经验。通过模拟各种场景，科学家发现很多助手虽然存了信息，但用得不够好。未来，我们希望这些助手变得更聪明，能像你的好朋友一样，记得你说过的话，还能帮你解决问题，变得更贴心、更聪明。
Abstract
A central role of personal-agent memory is to turn stored information and prior interactions into future-oriented assistance. In daily use, useful cues come from what the agent observes and how the user interacts with the agent, and the agent must carry them forward from the current request to similar future tasks. Existing memory benchmarks usually test dialogue recall or task improvement in isolation, leaving the trajectory from streaming observations to later assistance largely untested. We introduce StreamMemBench, a streaming benchmark that constructs a two-step task sequence around each evidence anchor from EgoLife egocentric streams. The initial task tests evidence use, while the follow-up task tests whether feedback and interaction experience are reused. Four metrics diagnose evidence recall, initial evidence use, feedback incorporation, and follow-up reuse. Experiments with eight memory systems across two backbones show that current systems often fail to use observed evidence or turn feedback into reliable follow-up behavior, even when evidence is stored or feedback is incorporated locally. StreamMemBench is publicly available at https://github.com/landian60/StreamMemBench.