DocAttriBench: Benchmarking Answer Grounding in Document Visual Question Answering
DocAttriBench uses MAPPET for answer grounding in Document VQA, offering 237k documents and 296k QA pairs.
Key Findings
Methodology
We propose a Mask-based Perplexity-Derived Attribution method (MAPPET) that combines document layout and language modeling to identify the most informative element for each answer. MAPPET measures the increase in perplexity after masking candidate elements and attributes the answer to the element contributing most to model confidence.
Key Results
- MAPPET applied to multiple existing Document VQA datasets yields DAB, with 237k documents and 296k question-answer pairs with element-level grounding.
- Results show that while larger models generally achieve higher answer accuracy, even the strongest models often fail to localize the supporting elements.
- Fine-tuned models outperform their zero-shot counterparts in grounding accuracy, demonstrating DAB's effectiveness in developing grounding-capable multimodal models.
Significance
DocAttriBench provides a scalable benchmark for developing grounded, verifiable, and trustworthy Document VQA models. It addresses the issue of existing benchmarks lacking grounding annotations or providing limited-quality labels, reducing the cost of manually constructing grounded datasets through automation.
Technical Contribution
Through MAPPET, we provide a novel automatic visual answer grounding framework capable of achieving fine-grained element-level grounding in Document VQA. This method not only enhances model grounding capabilities but also opens new possibilities for future multimodal reasoning.
Novelty
DocAttriBench is the first large-scale Document VQA benchmark providing high-quality element-level grounding annotations. It significantly improves upon existing datasets in both scale and annotation quality.
Limitations
- MAPPET may perform poorly on complex layouts or multi-page documents, as these scenarios may lead to inaccurate perplexity calculations.
- Automated annotations may introduce errors or hallucinated grounding, especially when visual content does not fully support the answer.
Future Work
Future research can explore improving MAPPET's performance on complex documents and developing more robust multimodal models to enhance grounding and answer generation accuracy.
AI Executive Summary
Answer grounding in document visual question answering (VQA) remains an open challenge, with existing benchmarks lacking high-quality grounding annotations. DocAttriBench (DAB) introduces a scalable benchmark for Document VQA by employing a Mask-based Perplexity-Derived Attribution method (MAPPET) to identify the most informative elements for each answer. Applied to multiple existing Document VQA datasets, DAB comprises 237k documents and 296k question-answer pairs with element-level grounding.
Experimental results show that while larger models generally achieve higher answer accuracy, even the strongest models often fail to localize the supporting elements. Fine-tuned models outperform their zero-shot counterparts in grounding accuracy, demonstrating DAB's effectiveness in developing grounding-capable multimodal models.
DAB provides a scalable benchmark for developing grounded, verifiable, and trustworthy Document VQA models. It addresses the issue of existing benchmarks lacking grounding annotations or providing limited-quality labels, reducing the cost of manually constructing grounded datasets through automation. Future research can explore improving MAPPET's performance on complex documents and developing more robust multimodal models to enhance grounding and answer generation accuracy.
Deep Analysis
Background
Document visual question answering (VQA) aims to answer questions by analyzing visual and textual information within documents. Recently, multimodal large models (MLLMs) have made significant progress in document understanding. However, existing benchmark datasets lack sufficient grounding annotations, limiting further model development.
Core Problem
The core problem in Document VQA is accurately localizing the visual elements that support the answer. Existing datasets lack high-quality grounding annotations, making it difficult for models to achieve precise answer grounding in complex documents.
Innovation
DocAttriBench achieves answer grounding in Document VQA through the MAPPET method. MAPPET combines document layout and language modeling to identify the most informative elements for each answer. This approach not only enhances model grounding capabilities but also opens new possibilities for future multimodal reasoning.
Methodology
- �� Propose MAPPET method to identify answer-supporting elements through masked perplexity calculations.
- �� Apply MAPPET to multiple existing Document VQA datasets to generate DAB.
- �� DAB comprises 237k documents and 296k question-answer pairs with element-level grounding annotations.
Experiments
Experiments were conducted on multiple Document VQA datasets, evaluating model answer accuracy and grounding accuracy. Results show that fine-tuned models outperform their zero-shot counterparts in grounding accuracy.
Results
Results show that while larger models generally achieve higher answer accuracy, even the strongest models often fail to localize the supporting elements. Fine-tuned models outperform their zero-shot counterparts in grounding accuracy.
Applications
DocAttriBench provides a scalable benchmark for developing grounded, verifiable, and trustworthy Document VQA models.
Limitations & Outlook
MAPPET may perform poorly on complex layouts or multi-page documents, as these scenarios may lead to inaccurate perplexity calculations. Automated annotations may introduce errors or hallucinated grounding, especially when visual content does not fully support the answer.
Plain Language Accessible to non-experts
Imagine you're looking at a picture-filled encyclopedia with lots of questions and answers. DocAttriBench acts like a smart assistant that helps you find the exact picture or text location for each answer. Using a method called MAPPET, it identifies which parts of the picture or text are most important for the answer. It's like having a guide that tells you exactly which page and section to look at when you're searching for an answer. This method not only makes finding answers more efficient but also ensures accuracy.
ELI14 Explained like you're 14
Imagine you're playing a game about an encyclopedia, and every time you answer a question, the game asks you to point out where the answer came from. DocAttriBench is like a super helper that quickly finds the exact spot for each answer. Using a method called MAPPET, it figures out which parts of the picture or text are most important. It's like having a guide that tells you exactly which page and section to look at when you're searching for an answer. This method not only makes finding answers more efficient but also ensures accuracy.
Glossary
MAPPET (Mask-based Perplexity-Derived Attribution)
A method combining document layout and language modeling to identify the most informative elements for each answer.
Used in DocAttriBench for answer grounding in Document VQA.
DocAttriBench
A large-scale Document VQA benchmark providing element-level grounding annotations.
Used to evaluate multimodal large models in Document VQA.
Multimodal Large Models (MLLMs)
Machine learning models capable of handling multiple data types, such as text and images.
Applied in document understanding and answer grounding.
Perplexity
Measures the probability of a language model generating a sentence; lower perplexity indicates higher model confidence.
Used in MAPPET to assess element importance.
Visual Answer Grounding
The task of identifying the specific visual evidence that supports an answer.
Used in Document VQA to enhance answer accuracy and verifiability.
Open Questions Unanswered questions from this research
- 1 How to improve MAPPET's performance on complex documents?
- 2 How to reduce errors and hallucinated grounding in automated annotations?
- 3 How to further enhance multimodal model grounding capabilities?
Applications
Immediate Applications
Document Analysis
Researchers can use DocAttriBench to evaluate and improve the performance of Document VQA models.
Long-term Vision
Intelligent Document Processing
Future systems could automatically identify and locate key information in documents, transforming document handling.
Abstract
Answer grounding in document visual question answering remains an open challenge: most benchmarks lack grounding annotations or provide limited-quality labels, while constructing grounded datasets still requires costly manual effort. We introduce DocAttriBench (DAB), a large-scale benchmark for fine-grained, element-level source attribution in Document VQA, grounding answers to specific layout elements such as text blocks, tables, and images. To build DAB, we propose a Mask-based Perplexity-Derived Attribution method (MAPPET) that combines document layout and language modeling to identify the most informative element for each answer. MAPPET measures the increase in perplexity after masking candidate elements and attributes the answer to the element contributing most to model confidence. Applying MAPPET to multiple existing Document VQA datasets yields DAB, with 237k documents and 296k question-answer pairs with element-level grounding. We benchmark grounding-capable multimodal LLMs on DAB, evaluating answer accuracy, attribution accuracy, and overall answer quality. Results show that while larger models generally achieve higher answer accuracy, even the strongest models often fail to localize the supporting elements. DAB provides a scalable benchmark for developing grounded, verifiable, and trustworthy Document VQA models. Dataset and code are available at https://aimagelab.github.io/DocAttriBench/.