Do You Remember? Toward Memory-Centric Multimodal AI
Proposes DoYouRemember architecture combining VQ-VAE, LoRA-tuned LLM, and diffusion decoder for image memory reconstruction.
Key Findings
Methodology
The proposed three-stage architecture includes: first, VQ-VAE compresses images into discrete visual tokens; second, a LoRA-tuned transformer model jointly attends to visual and text tokens for multimodal understanding; third, a diffusion model reconstructs images from LLM hidden states. Experiments with 1000 3D facial skin maps and 99,000 unlabeled facial images reveal that LLM hidden states contain negligible recoverable visual information, confirming understanding but not memory. Training a shared memory matrix M encounters systematic failure due to gradient cancellation, which is mitigated by local EMA updates—each image updates only its top 8 of 64 slots, maintaining diversity. The resulting 229K-parameter M approaches the VQ-VAE upper bound on unseen images; expanding to 1024 slots surpasses it, as continuous representations avoid quantization errors. This framework interprets memory as lossy compression, recall as decompression, and hallucination as an inherent property of lossy decompression.
Key Results
- On 1000 3D facial textures and 99,000 unlabeled images, LLM hidden states contain almost no recoverable visual info, while VQ-VAE tokens enable clear image reconstruction, confirming memory gaps. The 229K parameter memory matrix performs near the VQ-VAE limit; expanding to 1024 slots reduces LPIPS from 0.071 to 0.056, showing continuous representations outperform discrete quantization.
- Local EMA updates prevent gradient cancellation, allowing each image to update only a few slots, preserving diversity. Increasing slots improves reconstruction quality, demonstrating scalability and robustness.
- From an information-theoretic perspective, memory is lossy compression, recall is decompression, hallucination is an inherent feature, emphasizing the importance of compression-decompression understanding in multimodal memory models.
Significance
This work advances multimodal large models by addressing their limited long-term memory capabilities. By framing memory as lossy compression, it offers a theoretical foundation for models with persistent storage and reasoning. The scalable memory mechanism enables applications in long video understanding, virtual avatars, and interactive AI, overcoming current short-term memory bottlenecks. Moreover, it provides insights into human memory processes, highlighting the role of compression and reconstruction, with broad implications for AI and cognitive science.
Technical Contribution
The paper introduces a novel architecture integrating VQ-VAE, LoRA-tuned LLM, and diffusion decoders, with a scalable shared memory matrix M. The key innovation is the local EMA update strategy, which prevents gradient cancellation and enhances memory capacity. The approach achieves near or surpasses VQ-VAE performance with 16x parameter compression, demonstrating the effectiveness of continuous representations over discrete quantization. The framework also offers a new theoretical perspective on memory as lossy compression, with implications for future multimodal AI systems.
Novelty
This is the first systematic integration of a scalable, learnable memory matrix with VQ-VAE and diffusion models for multimodal image memory. Unlike traditional single-pass encoding-decoding, it emphasizes persistent storage and reconstruction, addressing the memory bottleneck in large models. The local EMA update strategy effectively resolves gradient cancellation, enabling large-scale memory expansion, representing a significant step forward in AI memory research.
Limitations
- The model's memory performance may degrade in highly dynamic or complex scenes, especially over extended periods, due to information loss. Its scalability to very high-resolution images or videos remains to be tested.
- Training relies heavily on the EMA update strategy; its effectiveness across diverse tasks and data distributions needs further validation. The computational overhead of large memory matrices could limit real-time applications.
- While the approach improves memory capacity, it does not yet fully emulate human-like memory processes such as forgetting or selective recall, which are important for practical deployment.
Future Work
Future research will explore adaptive memory update mechanisms, possibly integrating reinforcement learning to optimize retention and forgetting. Extending the framework to multi-modal data beyond images, such as audio and text, is also planned. Additionally, efforts will focus on reducing computational costs and improving robustness in real-world scenarios, aiming toward AI systems with more human-like, long-term memory capabilities.
AI Executive Summary
Current multimodal large language models excel at understanding visual and textual data but lack effective long-term memory. They process images through fixed encoders, generate one-shot outputs, and discard internal representations, limiting their ability to recall or reconstruct past inputs. This shortcoming constrains their performance in tasks requiring persistent knowledge, such as long video comprehension or interactive dialogue.
To address this, the authors propose "DoYouRemember," a three-stage architecture that introduces reconstructive memory into multimodal models. The first stage employs VQ-VAE to compress images into discrete tokens, significantly reducing storage requirements. The second stage involves a LoRA-finetuned transformer that jointly attends to visual and textual tokens, enabling multimodal understanding. The third stage uses a diffusion decoder to reconstruct images from the LLM's hidden states, testing whether the model truly 'remembers' visual content.
Experimental results demonstrate that the LLM's hidden states contain negligible recoverable visual information, while the VQ-VAE tokens enable high-quality image reconstruction. A shared memory matrix M, trained with a novel local EMA update strategy, effectively stores visual information across multiple images. This matrix, with only 229K parameters, approaches the performance upper bound set by VQ-VAE, and expanding to 1024 slots surpasses it, thanks to the continuous representation that avoids quantization errors. These findings are grounded in an information-theoretic framework, where memory is viewed as lossy compression, recall as decompression, and hallucination as an inherent property of lossy processes.
This work significantly advances the understanding of memory mechanisms in multimodal AI, providing a scalable, theoretically grounded approach to persistent visual memory. It opens pathways for future AI systems capable of long-term reasoning, learning, and interaction, with broad implications for both AI research and cognitive science. Despite current limitations, such as handling highly dynamic scenes and computational costs, the proposed framework sets a foundation for more human-like, long-term memory in artificial intelligence.
Deep Dive
Abstract
Human memory is reconstructive, not a faithful recording. Current multimodal LLMs (MLLMs) lack this capability: they process images through a frozen visual encoder, produce a one-shot text output, and discard internal representations. We present DoYouRemember, a three-stage architecture introducing reconstructive memory into MLLMs: (1) a VQ-VAE compresses images into discrete visual tokens, (2) a LoRA-fine-tuned LLM jointly attends to visual and text tokens, and (3) a Diffusion Decoder reconstructs images from the LLM's hidden states. On 1,000 3D facial skin texture maps and 99,000 unlabeled facial images, we find that LLM hidden states contain approximately zero recoverable visual information -- the same Decoder producing clear reconstructions from VQ-VAE tokens (pre-LLM) produces pure noise from LLM hidden states (post-LLM), demonstrating that the LLM understands images but does not remember them. Training a shared memory matrix M under backpropagation systematically fails due to gradient cancellation (O(1/sqrt(N)) attenuation). We identify three root causes and show that local EMA updating resolves all three: each image updates only its top-8 slots out of 64, preserving inter-slot diversity. The resulting M (229K parameters, 16x compressed) approaches the VQ upper bound on unseen test images. Scaling to 1,024 slots surpasses it (LPIPS 0.056 vs. 0.071), as M's continuous representation avoids VQ quantization error. We unify these findings under an information-theoretic framework: memory is lossy compression, recall is decompression, and hallucination is an inherent property of lossy decompression rather than a defect.