DocScope: Benchmarking Verifiable Reasoning for Trustworthy Long-Document Understanding
DocScope benchmarks trustworthy reasoning in long-document understanding, finding only 29% complete evidence chains.
Key Findings
Methodology
DocScope uses a four-stage evaluation protocol: Page Localization, Region Grounding, Fact Extraction, and Answer Verification. Each stage independently assesses the model's reasoning trajectory, ensuring outputs are not only accurate but verifiable. Tested with 1,124 questions and 273 documents, all evidence annotations are human-completed.
Key Results
- Result 1: Even among correct answers, the highest observed rate of complete evidence chains is only 29%.
- Result 2: Region grounding is the weakest stage across all models.
- Result 3: The primary challenge is aggregating evidence dispersed across long distances and multiple document clusters.
Significance
DocScope provides a comprehensive evaluation framework for long-document QA systems, addressing gaps in existing benchmarks regarding evidence verification. It emphasizes not just answer accuracy but the verifiability of the reasoning process, offering crucial insights for future research.
Technical Contribution
By introducing a four-stage evaluation protocol, DocScope offers a fine-grained assessment framework that independently diagnoses each reasoning stage's performance. This approach allows researchers to better understand model performance and limitations in long-document understanding.
Novelty
DocScope is the first to formalize long-document QA as a structured reasoning trajectory prediction problem, emphasizing evidence chain completeness and verifiability.
Limitations
- Limitation 1: Current models perform poorly in handling evidence dispersed across long distances and multiple document clusters.
- Limitation 2: Region grounding remains a weak stage across models.
Future Work
Future research could focus on improving region grounding performance and more effectively integrating dispersed evidence.
AI Executive Summary
Understanding long documents is increasingly important in the application of multimodal large language models. However, existing evaluation methods often focus only on answer accuracy, neglecting the verifiability of the reasoning process. DocScope introduces a four-stage evaluation protocol, providing a comprehensive framework that allows researchers to better understand model performance and limitations in long-document understanding.
DocScope's experimental results show that even among correct answers, the highest observed rate of complete evidence chains is only 29%. This indicates that answer accuracy cannot substitute for trajectory-level evaluation. Region grounding is the weakest stage across all models, and the primary challenge is aggregating evidence dispersed across long distances and multiple document clusters.
DocScope provides a comprehensive evaluation framework for long-document QA systems, addressing gaps in existing benchmarks regarding evidence verification. It emphasizes not just answer accuracy but the verifiability of the reasoning process, offering crucial insights for future research. Future research could focus on improving region grounding performance and more effectively integrating dispersed evidence.
Deep Analysis
Background
In recent years, multimodal large language models have made significant advances in document understanding tasks. However, as these models are increasingly deployed in realistic application scenarios, merely generating answers is no longer sufficient to meet user requirements. Users need not only accurate answers but also responses that can be verified and audited based on the source document. Thus, evaluating whether multimodal large language models can produce trustworthy, verifiable reasoning over long, visually rich documents becomes a critical question.
Core Problem
Existing long-document understanding benchmarks often address only a subset of the evidence-verification problem, leaving several critical dimensions underexplored. Especially when handling evidence dispersed across long distances and multiple document clusters, current evaluation methods often fail to fully reflect the model's capabilities. This results in a pronounced decoupling between answer correctness and reasoning trustworthiness.
Innovation
DocScope is the first to formalize long-document QA as a structured reasoning trajectory prediction problem, emphasizing evidence chain completeness and verifiability. By introducing a four-stage evaluation protocol, DocScope can independently diagnose each reasoning stage's performance, allowing researchers to better understand model performance and limitations in long-document understanding.
Methodology
- �� Page Localization: Identifying the pages containing evidence.
- �� Region Grounding: Locating specific evidence regions on the pages.
- �� Fact Extraction: Extracting relevant factual statements from the evidence regions.
- �� Answer Verification: Verifying whether the final answer is derived from these facts.
Experiments
DocScope comprises 1,124 questions derived from 273 documents. The experiments benchmarked 6 proprietary models, 12 open-weight models, and several domain-specific systems. Evaluation metrics include precision, recall, and F1 scores for page localization, strict and lenient F1 for region grounding, and consistency rate for fact extraction.
Results
The experimental results show that even among correct answers, the highest observed rate of complete evidence chains is only 29%. Region grounding is the weakest stage across all models, and the primary challenge is aggregating evidence dispersed across long distances and multiple document clusters. Different models exhibit distinct capability profiles at different stages.
Applications
DocScope provides a comprehensive evaluation framework for long-document QA systems, addressing gaps in existing benchmarks regarding evidence verification. It emphasizes not just answer accuracy but the verifiability of the reasoning process, offering crucial insights for future research.
Limitations & Outlook
Current models perform poorly in handling evidence dispersed across long distances and multiple document clusters. Region grounding remains a weak stage across models, and future research could focus on improving this stage's performance and more effectively integrating dispersed evidence.
Plain Language Accessible to non-experts
Imagine you're in a library trying to find a specific chapter in a book to answer a question. First, you need to find the book (page localization), then locate the specific chapter within the book (region grounding). Next, you need to extract relevant information from the chapter (fact extraction), and finally, use this information to answer the question (answer verification). If you make a mistake at any step, the final answer might be inaccurate or unverifiable. DocScope acts like a librarian, helping you make the right choices at each step.
ELI14 Explained like you're 14
Imagine you're at school in the library, and your teacher gives you a question you need to answer using a big book. First, you have to find the right page in the book (page localization), then find the specific paragraph on that page (region grounding). Next, you need to pick out the important information from the paragraph (fact extraction), and finally, use that information to answer the teacher's question (answer verification). If you mess up any step, the answer might be wrong! DocScope is like a super-smart helper, guiding you to make the right choices at each step.
Glossary
Page Localization
Identifying the pages in a document that contain evidence.
Used in DocScope to evaluate the model's ability to identify evidence pages.
Region Grounding
Locating specific evidence regions on the identified pages.
Used to evaluate the model's ability to precisely locate evidence on a page.
Fact Extraction
Extracting relevant factual statements from the evidence regions.
Used to evaluate the model's ability to extract accurate facts from evidence.
Answer Verification
Verifying whether the final answer is derived from the extracted facts.
Used to evaluate the trustworthiness and accuracy of the model's answers.
Evidence Chain
The complete reasoning path from question to answer.
Used in DocScope to evaluate the completeness of the model's reasoning process.
Open Questions Unanswered questions from this research
- 1 How to improve region grounding performance?
- 2 How to more effectively integrate dispersed evidence?
Applications
Immediate Applications
Academic Research
Researchers can use DocScope to evaluate long-document QA systems' performance and identify their shortcomings in evidence verification.
Long-term Vision
Intelligent Assistants
Future intelligent assistants could leverage DocScope's evaluation framework to improve the accuracy and trustworthiness of long-document understanding.
Abstract
Evaluating whether Multimodal Large Language Models can produce trustworthy, verifiable reasoning over long, visually rich documents requires evaluation beyond end-to-end answer accuracy. We introduce DocScope, a benchmark that formulates long-document QA as a structured reasoning trajectory prediction problem: given a complete PDF document and a question, the model outputs evidence pages, supporting evidence regions, relevant factual statements, and a final answer. We design a four-stage evaluation protocol -- Page Localization, Region Grounding, Fact Extraction, and Answer Verification -- that audits each level of the trajectory independently through inter-stage decoupling, with all judges selected and calibrated via human alignment studies. DocScope comprises 1,124 questions derived from 273 documents, with all hierarchical evidence annotations completed by human annotators. We benchmark 6 proprietary models, 12 open-weight models, and several domain-specific systems. Our experiments reveal that answer accuracy cannot substitute for trajectory-level evaluation: even among correct answers, the highest observed rate of complete evidence chains is only 29\%. Across all models, region grounding remains the weakest trajectory stage. Furthermore, the primary difficulty stems from aggregating evidence dispersed across long distances and multiple document clusters, while an oracle study identifies faithful perception and fact extraction as the dominant capability bottleneck. Cross-architecture comparisons further suggest that activated parameter count matters more than total scale. The benchmark and code will be publicly released at https://github.com/MiliLab/DocScope.