DynamicMem: A Long-Horizon Memory Benchmark in Real-World Settings

Long-term memory in AI has evolved from simple persona-based models to sophisticated external memory modules. Early works like GPT-based personalization focused on static profiles, while recent efforts such as Memory-Augmented Neural Networks and retrieval-augmented models aimed to improve evidence access. However, these approaches often lack realistic modeling of behavior dynamics over months or years, especially across multiple applications. Existing benchmarks like LlamaIndex and Lifelong Learning datasets primarily evaluate short-term recall or static profiles, failing to capture the heterogeneity and external causality that drive real user behavior. As AI assistants become more integrated into daily life, the need for scalable, causally grounded long-term memory systems becomes urgent. This work builds on prior efforts but introduces a comprehensive synthetic benchmark that simulates realistic, multi-application user trajectories over 15 months, addressing the gap between short-term recall and real-world complexity.

Current evaluation methods for long-term memory in AI systems are limited by their inability to simulate realistic user behavior over extended periods. They often assume static profiles or unmotivated drift, ignoring the heterogeneity of user attributes, habits, and preferences. Moreover, existing benchmarks lack causal grounding, making it difficult to assess how well models adapt to external influences like seasons or life events. This results in overestimating system capabilities and underestimating the challenges of maintaining accurate, dynamic user models. The core problem is designing a benchmark that captures the complexity, heterogeneity, and external causality of real user behavior, enabling meaningful evaluation of memory systems’ scalability, stability, and update mechanisms over long horizons.

The paper introduces a multi-stage trajectory synthesis framework that models user profile evolution driven by external causal events, with explicit causal annotations. It separates user attributes, habits, and preferences into distinct categories, each with different temporal dynamics, and generates event chains conditioned on these factors. The logs are grounded in stateful app environments, ensuring behavioral consistency across applications. The evaluation schema employs five quarterly checkpoints, analyzing how memory accuracy degrades with increasing history length. This approach uniquely combines causal inference, multi-application coherence, and long-horizon synthetic data generation, setting a new standard for realistic, scalable long-term memory benchmarking in AI.

• �� Construct static base profiles with demographic and personality data. • Model external events (seasonal, life-changing) as causal drivers influencing profile components. • Generate typed deltas (add, acquire, shift, refine) for attributes, habits, and preferences, each with causal annotations. • Use large language models to produce event chains conditioned on current profile, external drivers, and previous states, ensuring intent coherence. • Ground each event into multi-app logs with explicit API calls, maintaining stateful consistency within and across apps. • Incorporate manual validation to correct implausible transitions and conflicts. • Evaluate at five quarterly checkpoints, measuring profile reconstruction and service accuracy, analyzing degradation over time.

The dataset comprises 10 synthetic personas from PersonaHub, covering 6 life domains, 16 applications, and 66 APIs, with trajectories spanning 15 months. The evaluation compares five memory systems, including retrieval-augmented and external memory models, on tasks like state completion and personalized service. Metrics include accuracy, recall, and error analysis, focusing on how performance varies with history length. Ablation studies test the impact of causal modeling and multi-scale profiles. Manual validation ensures trajectory realism. Experiments reveal that longer histories lead to declining accuracy, with retrieval errors dominating failure cases, emphasizing the importance of improving evidence access and causal reasoning.

Results show a clear degradation in profile reconstruction accuracy from 85% to 65% as history length increases, with over 93% of failures due to retrieval inaccuracies. No system effectively maintains stable facts while updating changing ones, especially for implicit preferences and entity references. Errors tend to cluster around specific categories, highlighting the need for better causal inference and memory management. The findings underscore that the main bottleneck lies in the memory retrieval process, not the model’s generative capacity, guiding future research toward more robust memory architectures.

This benchmark enables rigorous evaluation of AI assistants’ ability to remember and adapt over months, crucial for personalized healthcare, smart home management, and adaptive learning systems. It supports development of systems that can handle evolving user profiles, external influences, and multi-application coherence, fostering more natural, persistent interactions. The synthetic data approach preserves user privacy while providing rich, realistic scenarios for industry testing and academic research. Ultimately, it aims to accelerate the deployment of long-term, context-aware AI that can serve users reliably over extended periods.

The causal event modeling, while effective, simplifies real-world complexity, often missing unpredictable external influences. The trajectory generation process is computationally intensive, limiting scalability. The current framework emphasizes structured attributes, habits, and preferences, lacking richer multimodal data integration. Additionally, the focus on retrieval accuracy does not fully address downstream task performance, such as decision-making or proactive assistance. Future work should aim to incorporate more complex causal models, optimize efficiency, and extend multimodal capabilities to better mirror real user environments.