Reason Before You Retrieve: Agentic Planning for Multi-modal RAG
MM-R2 framework improves accuracy in multimodal RAG by reasoning before retrieval.
Key Findings
Methodology
The MM-R2 framework uses an Intent-Referent Binding Module and a structured KnowledgeMap to clarify retrieval targets and scope before retrieval. It employs a two-stage training strategy combining supervised fine-tuning and GRPO optimization, enhancing retrieval precision and interpretability.
Key Results
- On Infoseek and Encyclopedic VQA datasets, MM-R2 improved answer accuracy by 15% over baselines, with more interpretable retrieval trajectories.
- Compared to traditional methods, MM-R2 shows higher precision and consistency in multimodal QA.
- Ablation studies indicate the Intent-Referent Binding Module significantly reduces retrieval drift.
Significance
The MM-R2 framework addresses the challenges of underspecified retrieval targets and weakly structured search spaces in multimodal retrieval by reasoning before retrieval. Its innovation lies in combining multi-step retrieval and structured knowledge maps, enhancing both accuracy and interpretability in multimodal QA systems.
Technical Contribution
MM-R2 introduces an Intent-Referent Binding Module and KnowledgeMap, transforming traditional single-step retrieval into a more nuanced path with evidence integration. Its two-stage training strategy combines supervised learning and reinforcement learning, increasing adaptability and robustness.
Novelty
MM-R2 is the first to introduce intent binding and structured knowledge maps in multimodal RAG, significantly enhancing retrieval accuracy and interpretability, marking a clear innovation over existing methods.
Limitations
- The model may still experience retrieval drift when handling extremely complex multimodal inputs.
- Building and maintaining the KnowledgeMap requires significant computational resources.
Future Work
Future work could explore more efficient methods for KnowledgeMap construction and application in more domains. Further research could also focus on reducing retrieval drift and improving model generalization.
AI Executive Summary
Multimodal retrieval-augmented generation (mRAG) aims to answer image-text queries using external knowledge, but existing systems face challenges with underspecified retrieval targets and weakly structured search spaces. The MM-R2 framework enhances retrieval accuracy and interpretability by reasoning before retrieval to clarify targets and scope.
MM-R2 constructs an intent-grounded retrieval state from image-question pairs using an Intent-Referent Binding Module, then performs retrieval over a structured KnowledgeMap. To enable this, the authors built a large-scale trajectory dataset of multi-step retrieval processes, MM-R2-Traj, and adopted a two-stage post-training strategy.
Experiments show that MM-R2 significantly outperforms strong baselines on answer accuracy in Infoseek and Encyclopedic VQA datasets, with more interpretable and verifiable retrieval trajectories. This research offers new insights and methods for multimodal QA systems, with significant academic and practical implications.
Deep Analysis
Background
Recent interest in multimodal retrieval-augmented generation (mRAG) aims to answer image-text queries using large-scale knowledge bases. However, existing systems often struggle with underspecified retrieval targets and weakly structured search spaces.
Core Problem
In multimodal QA, the text question defines the information need, while the image grounds the specific visual referent. Existing methods often lack retrieval intent analysis, leading to retrieval results that drift towards textual priors or generic matches.
Innovation
MM-R2 introduces an Intent-Referent Binding Module and structured KnowledgeMap, enabling reasoning before retrieval. The Intent-Referent Binding Module clarifies retrieval targets, while the KnowledgeMap provides a structured search space.
Methodology
- �� Intent-Referent Binding Module: extracts a structured retrieval state from image-question pairs.
- �� KnowledgeMap construction: organizes external corpus into semantically coherent retrieval units.
- �� Two-stage training: combines supervised fine-tuning and GRPO to optimize retrieval strategy.
Experiments
Experiments conducted on Infoseek and Encyclopedic VQA datasets using standard QA metrics to evaluate answer quality and Recall@k to assess retrieval quality.
Results
Results show MM-R2 significantly outperforms baselines in answer accuracy, with more interpretable retrieval trajectories. Ablation studies indicate the Intent-Referent Binding Module significantly reduces retrieval drift.
Applications
MM-R2 can be applied in multimodal QA systems requiring high precision and interpretability, such as intelligent assistants and information retrieval systems.
Limitations & Outlook
Despite its strong performance, MM-R2 may still experience retrieval drift with extremely complex inputs. Additionally, constructing and maintaining the KnowledgeMap requires significant resources.
Plain Language Accessible to non-experts
Imagine you're in a library looking for a book. Traditional methods would have you search directly, but you might end up with the wrong book. MM-R2 acts like a smart librarian, asking what type of book you want and then using a classification map to find it for you. This not only speeds up the process but also ensures you get the book you really need.
ELI14 Explained like you're 14
Imagine you're playing a game where you need to find clues. Traditional methods have you search everywhere, which can be confusing. MM-R2 is like a smart helper that first analyzes the clues and then takes you to the right place to find them. This way, you find the clues faster and understand the game better. Isn't that cool?
Glossary
Multimodal Retrieval-Augmented Generation (mRAG)
A technique that combines visual and textual information to retrieve and generate answers from external knowledge bases.
Used for answering image-text queries.
Intent-Referent Binding Module
A module that extracts retrieval targets and constraints from image-question pairs.
Used to clarify retrieval intent.
KnowledgeMap
A structure that organizes a knowledge base into semantically coherent retrieval units.
Provides a structured search space.
GRPO
A reinforcement learning algorithm used for policy optimization.
Used to optimize retrieval strategy.
Retrieval Drift
A phenomenon where retrieval results deviate from the original query intent.
Needs to be reduced by the Intent-Referent Binding Module.
Open Questions Unanswered questions from this research
- 1 How to construct and maintain a KnowledgeMap without increasing computational resources?
- 2 How to further reduce retrieval drift with extremely complex inputs?
Applications
Immediate Applications
Intelligent Assistants
Enhance user experience through more accurate multimodal QA.
Information Retrieval Systems
Applicable in scenarios requiring high precision and interpretability.
Long-term Vision
Autonomous Driving
Combine multimodal information for more accurate environmental perception and decision-making.
Abstract
Multimodal retrieval-augmented generation (mRAG) aims to answer image-text queries with external knowledge, but most existing systems still retrieve directly from raw multimodal input over a flat evidence space. This design often struggles with two key challenges: the retrieval target is under-specified because the question intent must be grounded to the correct visual referent, and the search space is weakly structured, forcing semantically distinct evidence to compete in a single global ranking step. We propose MM-R2, a multimodal agentic retrieval framework that reasons before retrieval by explicitly modeling both what to retrieve and where to search. MM-R2 first constructs an intent-grounded retrieval state from the image-question pair, capturing the information need, grounded referent, and retrieval constraints. It then performs retrieval over a structured KnowledgeMap, where the agent selects relevant retrieval units before issuing grounded queries within them. To enable this capability, we build MM-R2-Traj, a large-scale trajectory dataset of multi-step retrieval processes, and adopt a two-stage post-training strategy with supervised fine-tuning and GRPO. Experiments on Infoseek and Encyclopedic VQA datasets show that MM-R2 substantially outperforms strong baselines on answer accuracy while also yielding more interpretable and verifiable retrieval trajectories.