Multimodal Graph RAG for Long-range Visually Rich Document Understanding
KG4VD leverages multimodal knowledge graphs and graph retrieval, achieving 44.52% accuracy on long-range document VQA benchmarks.
Key Findings
Methodology
This paper introduces KG4VD, a framework combining offline MMKG construction with online graph-based retrieval for long-distance multimodal document understanding. It employs adaptive extraction and layout region grounding to handle diverse page complexities. The approach uses a variant of PageRank (PPR) for entity expansion, integrating multimodal indexes to improve retrieval precision. The system demonstrates superior performance on multi-hop QA and VQA tasks, notably achieving 44.52% accuracy on the DLVQA benchmark, outperforming existing methods by over 3%.
Key Results
- On the MMLongBench-doc dataset, KG4VD surpasses ColQwen (+2.73%) and MegaRAG (+3.34%), reaching 44.52% overall accuracy, indicating strong long-range multimodal reasoning capabilities.
- In the DLVQA benchmark, KG4VD achieves the highest scores in faithfulness, completeness, and conciseness, outperforming baselines by approximately 10%.
- Experiments highlight that integrating layout-aware graph construction and multimodal indexing significantly improves evidence aggregation across complex documents.
Significance
This work addresses the critical challenge of long-distance understanding in visually rich documents, enabling AI systems to connect dispersed evidence across multiple pages. The proposed MMKG framework facilitates more accurate, faithful, and comprehensive document comprehension, with broad implications for automated legal review, financial analysis, scientific research, and digital archiving. It advances the state-of-the-art in multimodal reasoning, bridging the gap between local page-level understanding and holistic document comprehension, thus opening new avenues for intelligent document processing.
Technical Contribution
The core contribution lies in designing a zero-shot MMKG construction pipeline grounded in layout components, combined with adaptive extraction and cross-page entity fusion. The use of a PageRank-based expansion mechanism effectively filters noise and enhances relevant evidence retrieval. Additionally, the introduction of the DLVQA benchmark provides a standardized, comprehensive evaluation for long-range multimodal VQA, fostering further research in this domain.
Novelty
This is the first work to automate the construction of a comprehensive, layout-grounded MMKG for visually rich, multi-page documents, integrating multimodal cues with graph reasoning. The innovative graph expansion and entity fusion strategies enable effective long-distance reasoning, surpassing prior methods limited to page-level retrieval or text-only graphs, thus offering a fundamentally new approach to complex document understanding.
Limitations
- The approach relies on pre-trained layout parsers and entity extractors, which may struggle with highly complex or poorly scanned documents, affecting extraction accuracy.
- Graph expansion and reasoning are computationally intensive, especially for very large documents, requiring optimization for real-time applications.
- The current model's performance drops on dense, fine-grained pages, indicating room for improvement in visual grounding and entity resolution in challenging scenarios.
Future Work
Future directions include end-to-end training of the MMKG construction pipeline, integrating reinforcement learning for adaptive reasoning paths, and extending the framework to multilingual and domain-specific datasets. Enhancing efficiency and robustness in diverse real-world scenarios will be key to broader deployment.
AI Executive Summary
Understanding complex, visually rich documents over long distances remains a significant challenge in AI. Existing multimodal large language models (MLLMs) excel at local visual question answering but falter when faced with lengthy, dispersed content due to limited context windows. To overcome this, researchers have explored retrieval-augmented generation (RAG) techniques, which retrieve relevant pages or snippets to aid comprehension. However, these methods often lack the ability to connect evidence across multiple pages effectively, especially in documents with intricate layouts and visual elements.
This paper introduces KG4VD, a novel framework that constructs a reusable, layout-aware multimodal knowledge graph (MMKG) to facilitate long-range reasoning across complex documents. The offline construction phase employs adaptive extraction and layout region grounding to generate detailed page graphs, capturing entities, relations, and visual cues. These page graphs are then connected into a global MMKG through cross-page entity fusion, leveraging a variant of personalized PageRank (PPR) for controlled graph expansion. During inference, the system anchors retrieval on relevant page images, dynamically expanding the evidence graph based on question type—local, multi-hop, or document-level reasoning—thus balancing precision and recall.
Experimental results demonstrate that KG4VD outperforms existing baselines such as ColQwen and MegaRAG, achieving 44.52% accuracy on the challenging DLVQA benchmark. The approach effectively integrates visual layout, textual content, and cross-page relations, enabling more faithful and comprehensive answers to complex questions. This advancement significantly pushes the frontier of long-distance multimodal understanding, with promising applications in legal, financial, scientific, and digital archiving domains.
Looking ahead, the authors plan to refine the end-to-end training of the MMKG pipeline, incorporate reinforcement learning for adaptive reasoning, and extend the framework to multilingual and domain-specific contexts. As multimodal large models continue to evolve, KG4VD offers a scalable, interpretable, and effective solution for the next generation of intelligent document understanding systems.
Deep Dive
Abstract
Multimodal large language models (MLLMs) are widely applied to visual document understanding. However, comprehending long documents remains an issue by the limited context window. Though recent multimodal retrieval-augmented generation (MMRAG) can address this challenge by retrieving relevant pages. It still struggles with the visual question answering (VQA) requiring holistic comprehension of a document. To cope with this, knowledge graph (KG) that summarizes global knowledge of a document can provide an effective solution. However, most existing LLM-based KG construction methods handle only the language modality, leaving the automatic creation of multimodal KGs (MMKGs) for visually rich documents largely unexplored. In this paper, we introduce a multimodal graph-based RAG approach to tackle this problem. Existing LLM-based KG methods evaluate the QA performance relying on indirect evidence such as comprehensiveness, diversity, empowerment, and so on. The lack of annotated datasets for comprehensive document-level VQA poses a significant challenge to effective model evaluation. To overcome this limitation, we also introduce a new benchmark, DLVQA (document-level VQA), which provides reference summaries and corresponding supporting facts for global document-level questions. Experimental results show that our approach outperforms existing MMRAG or KG-based approaches on multi-hop QA/VQA benchmarks and DLVQA.