DMV-Bench: Diagnosing Long-Horizon Multimodal Agents' Visual Memory with Incidental Cue Injection
DMV-Bench tests multimodal agents' visual memory; DualMem excels in long-chain tasks.
Key Findings
Methodology
The study employs DMV-Bench, a benchmark simulating a realistic e-commerce environment with 1,000 product variants. Each product image embeds a unique visual cue, which agents must recall after multiple shopping sessions. Inspired by dual coding theory, the DualMem architecture combines visual and verbal encodings.
Key Results
- DualMem achieved an 81.2% success rate on DMV-Bench, significantly outperforming text-only baselines.
- Across different models and session chain lengths, DualMem consistently outperformed other multimodal memory systems.
- Even after adjusting for memory bank size and encoding position bias, DualMem's advantage persisted.
Significance
This study systematically analyzes the often-overlooked visual memory capabilities of multimodal agents, addressing a gap in long-term visual memory research. DMV-Bench enables a deeper understanding of agent memory performance in complex environments, advancing agent design.
Technical Contribution
Introduced the DualMem architecture, leveraging dual coding theory to integrate visual and verbal information, significantly enhancing memory capabilities. Compared to existing multimodal memory systems, DualMem excels in information retention and retrieval.
Novelty
DMV-Bench is the first interactive benchmark focused on visual memory, and DualMem is the first to apply dual coding theory to agent memory systems, offering a new research perspective.
Limitations
- DMV-Bench is currently limited to e-commerce environments, which may not generalize to other complex scenarios.
- DualMem's performance significantly declines with extremely long session chains.
Future Work
Future research could expand DMV-Bench's application scenarios and optimize DualMem's adaptability across different environments.
AI Executive Summary
In the realm of multimodal agents, visual memory has been an overlooked area. Existing benchmarks primarily focus on textual memory, while DMV-Bench, by embedding visual cues in an e-commerce environment, systematically analyzes visual memory performance for the first time.
The study introduces the DualMem architecture, which combines visual and verbal encoding through dual coding theory to enhance agent memory capabilities. In DMV-Bench tests, DualMem excelled in long-chain tasks, significantly outperforming baseline models.
This research not only fills a gap in visual memory studies but also provides new directions for future agent design. Although the current testing environment is limited, DMV-Bench offers a crucial tool for multimodal agent research.
Deep Analysis
Background
Agent memory research has largely focused on textual memory, overlooking the importance of visual memory. Existing benchmarks like VisualWebArena and MemoryArena primarily evaluate textual memory, leaving long-term visual retention capabilities underexplored.
Core Problem
Multimodal agents need to remember visual details in complex environments, but existing systems rely heavily on textual encoding, struggling with purely visual cues. This issue limits agents' performance in real-world applications.
Innovation
DMV-Bench systematically evaluates agents' visual memory capabilities by embedding visual cues in an e-commerce environment. The DualMem architecture combines visual and verbal encoding, significantly enhancing memory performance.
Methodology
- �� Embed visual cues in an e-commerce environment.
- �� Use DualMem architecture to combine visual and verbal encoding.
- �� Test agents' memory capabilities through multiple shopping sessions.
Experiments
Experiments were conducted on DMV-Bench, featuring 1,000 product variants. Memory performance was tested across different models and session chain lengths, comparing DualMem with other multimodal memory systems.
Results
DualMem achieved an 81.2% success rate in long-chain tasks, significantly outperforming baseline models. Its advantage persisted across different models and session chain lengths.
Applications
DMV-Bench can be used to evaluate agents' memory capabilities in complex environments, and the DualMem architecture can be applied to agent designs requiring long-term memory.
Limitations & Outlook
DMV-Bench is currently limited to e-commerce environments, and DualMem's performance declines significantly with extremely long session chains. Future research should expand application scenarios and optimize architecture performance.
Plain Language Accessible to non-experts
Imagine you're shopping in a large mall, and each product has a unique marker, like a small object of a specific color. You need to remember these markers after shopping to find the products again. DMV-Bench is like this mall, and DualMem is like your memory assistant, helping you remember these markers even if you didn't consciously try to memorize them.
ELI14 Explained like you're 14
Imagine you're playing a game where each level has hidden clues that help you find the key to the next level. DMV-Bench is like this game, and DualMem is like your super memory, helping you remember these clues even if you didn't try to remember them. Isn't that cool?
Glossary
DMV-Bench
An interactive benchmark for testing multimodal agents' visual memory, simulating a realistic e-commerce environment.
Used to evaluate agents' visual memory performance in long-term tasks.
DualMem
A memory architecture combining visual and verbal encoding, based on dual coding theory to enhance memory performance.
Used in DMV-Bench to enhance agents' visual memory capabilities.
Visual Cue
Unique visual markers embedded in product images, used to test agents' memory capabilities.
Core testing element in DMV-Bench.
Multimodal Agent
An agent capable of processing multiple forms of information, such as visual and textual.
Main subject of the study, evaluating its memory capabilities in complex environments.
Dual Coding Theory
A memory theory suggesting that information stored in both visual and verbal forms enhances memory effectiveness.
Theoretical basis for the DualMem architecture.
Open Questions Unanswered questions from this research
- 1 How to test visual memory in more complex environments? Current methods are mostly limited to e-commerce scenarios, needing expansion to other fields.
- 2 How to improve DualMem's performance in extremely long session chains? Current performance declines significantly, requiring further optimization.
Applications
Immediate Applications
E-commerce Assistant
Helps users remember product details while shopping, enhancing the shopping experience. Requires strong visual memory capabilities.
Long-term Vision
Intelligent Navigation Systems
Provides navigation support in complex environments, helping users remember important landmarks and paths. Requires optimized memory performance.
Abstract
Agent benchmarks for measuring memory largely study textual cases, in which information is deliberately extracted from the environment, written down, and then later retrieved. In other words, they assess what agents elected to record, not what they happened to see. We introduce DMV-Bench (code: https://github.com/yyyujintang/DMV-Bench), the first interactive benchmark for visual memory in multimodal agents, to study this often-neglected property. DMV-Bench is built on (1) a controlled home-furnishing e-commerce environment, supported by a catalog of 1,000 product variants, and (2) a text-leakage contract which ensures that the primary discriminative signal of each task is solely in the pixels. In DMV-Bench, agents undergo chains of autonomous shopping sessions in which every visited product image carries a unique, pre-rendered incidental cue that the agent is later asked to recall. We show that conventional solutions struggle with this task. Inspired by dual-coding theory, we propose a memory architecture that uses parallel visual and verbal codes, which we call DualMem. On DMV-Bench, DualMem outperforms a caption-only baseline and three recent multimodal agent-memory systems across multi-session chain lengths on multiple models. These gains persist even adjusting for memory-bank size and encoding-position bias. Further experiments also reveal an asymmetric division of labor between the two codes; a weighted coding scheme is often strongest. We view this as a step towards memory systems that preserve a richer record of agents' observations.