M$^3$Eval: Multi-Modal Memory Evaluation through Cognitively-Grounded Video Tasks
M³Eval, grounded in cognitive psychology, assesses multi-modal video memory across four dimensions with controlled tasks and quantitative metrics.
Key Findings
Methodology
This study adopts cognitive psychology paradigms, designing four video tasks—divided attention, interference robustness, interleaved events, and symbolic memory. Each task uses carefully crafted stimuli and multiple-choice questions to evaluate information retention, interference resistance, temporal organization, and symbolic abstraction. Experiments involve models like GPT-5, Gemini-3.1-Pro, and Qwen series, with metrics such as accuracy and intrusion rate, revealing weaknesses in multi-task, multi-modal memory handling. The framework isolates specific memory aspects, enabling precise analysis.
Key Results
- Models struggle with parallel streams, showing attention confusion in split-screen tasks, with accuracy around 50-70%, far below human performance (89.58%). Repetition improves accuracy by up to 15%, indicating potential strategies for enhancement.
- In interference tests, models exhibit high intrusion rates (>40%), and their resistance to proactive and retroactive interference is limited, contrasting with human asymmetry where retroactive interference dominates.
- Temporal organization tasks reveal low accuracy (30-45%) in interleaved event reconstruction, and symbolic memory (N-Back) accuracy remains below 70%, highlighting deficiencies in temporal and symbolic reasoning.
Significance
This work pioneers a systematic, theory-grounded evaluation of multi-modal video memory, addressing a critical gap in understanding how models encode, store, and retrieve information over long sequences. It provides a benchmark for diagnosing weaknesses and guiding future improvements, impacting long-form video analysis, continuous interaction, and symbolic reasoning in AI systems. The insights facilitate designing models with better disentanglement, robustness, and symbolic abstraction, essential for real-world applications like autonomous systems and intelligent assistants.
Technical Contribution
The framework integrates four cognitive paradigms into a unified evaluation platform, combining task isolation with quantitative metrics. It introduces novel experimental designs—split-screen, concatenation, interleaving, and N-Back—to disentangle specific memory facets. The approach enables detailed performance profiling of diverse models, revealing their limitations and guiding targeted enhancements. This systematic methodology advances the state-of-the-art in multi-modal memory evaluation, bridging cognitive theory and AI engineering.
Novelty
This is the first comprehensive benchmark explicitly grounded in cognitive psychology paradigms for multi-modal video memory. It innovatively combines multiple memory dimensions into a controlled, quantifiable framework, contrasting with prior works that focus solely on perception or reasoning. The task-oriented design allows for isolating and measuring distinct memory mechanisms, offering a new standard for future research.
Limitations
- The current evaluation mainly focuses on static or semi-dynamic scenes; dynamic, interactive environments require further extension.
- Symbolic abstraction remains limited, especially in complex reasoning scenarios; integrating neural-symbolic methods could improve this.
- Experiments are dataset-dependent; broader validation across diverse real-world scenarios is necessary for generalization.
Future Work
Future directions include expanding the benchmark to dynamic, interactive environments, integrating reinforcement learning for adaptive memory strategies, and developing hybrid neural-symbolic models. Additionally, exploring unsupervised or self-supervised training paradigms to enhance memory robustness and interpretability will be crucial. The ultimate goal is to create models capable of long-term, flexible, and symbolic memory akin to human cognition.
AI Executive Summary
The rapid advancement of multi-modal models has significantly improved visual perception and reasoning capabilities. However, their ability to retain, organize, and utilize information over long temporal horizons remains underexplored. Traditional benchmarks mainly evaluate perception accuracy, neglecting the core aspect of memory—how models encode, store, and retrieve information amidst interference and complexity.
To address this gap, we introduce M³Eval, a cognitively-grounded evaluation framework that systematically probes four key dimensions of memory in multi-modal video models: divided attention, interference robustness, temporal organization, and symbolic memory. Inspired by classic psychological paradigms, each task involves carefully designed video stimuli and targeted questions, quantifying model performance through metrics like accuracy, intrusion rate, and false memory detection.
Extensive experiments across state-of-the-art models reveal that current systems struggle with parallel streams, exhibit high interference susceptibility, and have limited temporal and symbolic reasoning. For instance, in split-screen tasks, models' accuracy drops to around 50%, far below human levels (~90%). Repetition strategies improve performance, indicating potential pathways for enhancement. In interference tests, intrusion rates exceed 40%, highlighting weak resistance to distractors. Temporal and symbolic tasks further expose deficiencies in organizing and abstracting long-term information.
These findings underscore the importance of developing more robust, disentangled, and symbolic memory mechanisms in multi-modal models. The benchmark provides a valuable tool for future research, guiding the design of systems capable of long-term, interference-resilient, and symbolic reasoning—crucial for applications like autonomous driving, video summarization, and intelligent assistants. Despite promising insights, challenges remain in extending the framework to dynamic, interactive scenarios and integrating symbolic reasoning more deeply. Continued efforts in this direction will be vital for realizing truly intelligent multi-modal AI systems.
Deep Dive
Abstract
As multi-modal models advance towards long-form video understanding, memory emerges as a critical capability. Despite substantial efforts in developing video datasets and benchmarks, existing works primarily focus on perception and reasoning, without systematically evaluating memory: what models retain, how faithfully information is preserved, and how robust memory remains under interference. To address this gap, we introduce M$^3$Eval, the first comprehensive evaluation framework and benchmark for probing different memory dimensions in multi-modal models. Grounded in cognitive psychology, our design features carefully constructed tasks that isolate key aspects of memory. Leveraging M$^3$Eval, we conduct extensive experiments across representative multi-modal models, revealing consistent weaknesses and distinctive behaviors. We find that models struggle to maintain disentangled representations when processing parallel video streams, exhibit interference patterns differing substantially from those observed in human memory, ground memory sources more reliably in the spatial domain than the temporal domain, and demonstrate limited symbolic memory. Collectively, our benchmark provides a valuable resource for future research, while our findings highlight memory as a fundamental yet underexplored capability and offer insights for designing more effective memory mechanisms in multi-modal models. Our code and dataset are available at https://pku-value-lab.github.io/m3eval-homepage.