Doc-V*:Coarse-to-Fine Interactive Visual Reasoning for Multi-Page Document VQA
Doc-V* enhances multi-page document VQA by sequential evidence aggregation, outperforming RAG baseline by 47.9%.
Key Findings
Methodology
Doc-V* employs an OCR-free agentic framework, starting with a thumbnail overview, followed by semantic retrieval and targeted page fetching to aggregate evidence. It is trained using imitation learning and Group Relative Policy Optimization to balance answer accuracy and evidence-seeking efficiency.
Key Results
- Across five benchmarks, Doc-V* improves out-of-domain performance by 47.9% over the RAG baseline, demonstrating effective evidence aggregation with selective attention.
- On the DUDE dataset, Doc-V* achieves an ANLS of 64.5, surpassing all open-source baselines.
- On SlideVQA, Doc-V* scores 77.2 F1, significantly outperforming other models.
Significance
Doc-V* is significant for both academia and industry, addressing the trade-off between scalability and precision in existing methods for long documents. It provides a more efficient evidence aggregation approach for complex document understanding tasks.
Technical Contribution
Doc-V* introduces the concept of active perception, contrasting with existing passive methods. By sequentially aggregating evidence, Doc-V* improves document understanding without increasing input pages.
Novelty
Doc-V* is the first to frame multi-page document VQA as a sequential evidence aggregation process, employing active perception strategies to significantly enhance cross-domain performance.
Limitations
- Doc-V* may face efficiency limitations when handling extremely long documents, especially with over 100 pages.
- Sensitivity to retrieval tools and hyperparameters remains, potentially affecting model robustness.
Future Work
Future research could explore the application of Doc-V* to more document types and further optimize its performance on extremely long documents.
AI Executive Summary
Multi-page Document Visual Question Answering (DocVQA) is a complex task requiring reasoning over semantics, layouts, and visual elements in long, dense documents. Existing OCR-free methods face a trade-off between scalability and precision when handling long documents. Doc-V* employs an agentic framework to sequentially aggregate evidence, significantly improving multi-page document VQA performance.
Doc-V* starts with a thumbnail overview to grasp the document's overall structure, followed by semantic retrieval and targeted page fetching to aggregate evidence. This method is trained using imitation learning and Group Relative Policy Optimization to balance answer accuracy and evidence-seeking efficiency. Across five benchmarks, Doc-V* outperforms open-source baselines and approaches proprietary models.
The innovation of Doc-V* lies in its active perception strategy, enabling the model to effectively aggregate evidence without increasing input pages. This approach not only enhances cross-domain performance but also demonstrates significant robustness in handling complex document understanding tasks. Future research could further explore its application to more document types and optimize its performance on extremely long documents.
Deep Analysis
Background
Multi-page Document Visual Question Answering requires reasoning over semantics, layouts, and visual elements in long, complex documents. Traditional OCR methods linearize document images into text, losing fine-grained layout information. Recently, OCR-free methods model documents directly as images, using multimodal large language models for visual-semantic reasoning.
Core Problem
Existing methods face a trade-off between scalability and precision when handling long documents. End-to-end models process entire documents as long image sequences, but scale poorly due to attention costs and context length limits. Visual retrieval-augmented generation systems improve efficiency but are sensitive to retrieval errors and hyperparameters.
Innovation
Doc-V* employs an agentic framework with active perception, starting with a thumbnail overview, followed by semantic retrieval and targeted page fetching to sequentially aggregate evidence. Unlike passive methods, Doc-V* dynamically adjusts its strategy to integrate discontinuous visual evidence.
Methodology
- �� Thumbnail Overview: Obtain overall document structure.
- �� Semantic Retrieval: Retrieve relevant pages based on semantic information.
- �� Targeted Page Fetching: Fetch specific pages to aggregate evidence.
- �� Evidence Aggregation: Integrate evidence in working memory for reasoning.
Experiments
Experiments were conducted on five benchmarks, including DUDE and MP-DocVQA. Metrics such as ANLS, F1, and accuracy were used to evaluate model performance. Results show that Doc-V* significantly outperforms existing baselines in cross-domain performance.
Results
Doc-V* achieves an ANLS of 64.5 on the DUDE dataset, surpassing all open-source baselines. On SlideVQA, it scores 77.2 F1, significantly outperforming other models. Results demonstrate the effectiveness of Doc-V* in evidence aggregation and selective attention.
Applications
Doc-V* is applicable to scenarios requiring complex document understanding tasks, such as financial reports, academic papers, and industrial manuals. Its active perception strategy improves the efficiency and accuracy of document understanding.
Limitations & Outlook
Doc-V* may face efficiency limitations when handling extremely long documents, especially with over 100 pages. Sensitivity to retrieval tools and hyperparameters remains, potentially affecting model robustness.
Plain Language Accessible to non-experts
Imagine you're in a huge library looking for a specific book. Traditional methods would involve browsing each book from start to finish, which is time-consuming and prone to missing details. Doc-V* acts like a smart librarian, quickly scanning the shelves to identify potential books, then checking each one step-by-step until the answer is found. This approach saves time and ensures the most relevant information is found.
ELI14 Explained like you're 14
Imagine you're at your school library looking for a book about dinosaurs. You wouldn't flip through every book from start to finish. Instead, you'd glance at the titles on the shelves, open a few to check the table of contents, and find the most likely book. Doc-V* is like this smart search process, quickly finding the information you need!
Glossary
OCR-free
A method that doesn't rely on optical character recognition, processing images directly.
Used to process document images directly without text linearization.
Active Perception
A process of dynamically adjusting perception strategies to gather information.
Doc-V* uses active perception strategies to sequentially aggregate evidence.
Group Relative Policy Optimization
An optimization strategy that balances answer accuracy and evidence-seeking efficiency through reward signals.
Used to train the Doc-V* model to enhance performance.
Sequential Evidence Aggregation
A method of integrating evidence step-by-step for reasoning.
Doc-V* improves multi-page document VQA performance through sequential evidence aggregation.
ANLS
A metric for evaluating answer accuracy in document QA tasks.
Used to evaluate Doc-V*'s performance on the DUDE dataset.
Open Questions Unanswered questions from this research
- 1 How to further improve Doc-V*'s efficiency on extremely long documents? Current methods may face limitations with over 100 pages.
- 2 How to reduce Doc-V*'s sensitivity to retrieval tools and hyperparameters to enhance model robustness?
Applications
Immediate Applications
Financial Report Analysis
Doc-V* can be used to quickly analyze complex financial reports, improving the efficiency and accuracy of information extraction.
Long-term Vision
Cross-domain Document Understanding
Doc-V*'s active perception strategy can be applied to more domains, enhancing understanding capabilities for different document types.
Abstract
Multi-page Document Visual Question Answering requires reasoning over semantics, layouts, and visual elements in long, visually dense documents. Existing OCR-free methods face a trade-off between capacity and precision: end-to-end models scale poorly with document length, while visual retrieval-based pipelines are brittle and passive. We propose Doc-$V^*$, an \textbf{OCR-free agentic} framework that casts multi-page DocVQA as sequential evidence aggregation. Doc-$V^*$ begins with a thumbnail overview, then actively navigates via semantic retrieval and targeted page fetching, and aggregates evidence in a structured working memory for grounded reasoning. Trained by imitation learning from expert trajectories and further optimized with Group Relative Policy Optimization, Doc-$V^*$ balances answer accuracy with evidence-seeking efficiency. Across five benchmarks, Doc-$V^*$ outperforms open-source baselines and approaches proprietary models, improving out-of-domain performance by up to \textbf{47.9\%} over RAG baseline. Other results reveal effective evidence aggregation with selective attention, not increased input pages.