Hierarchical multimodal transformers for Multi-Page DocVQA
Hierarchical multimodal transformer Hi-VT5 processes multi-page documents, achieving 50.01% accuracy and 0.6572 ANLS on MP-DocVQA.
Key Findings
Methodology
The paper introduces Hi-VT5, a hierarchical encoder-decoder model built on T5, designed to handle long multi-page documents. The encoder processes each page separately, extracting relevant information through a transformer that incorporates OCR text, layout, and visual features, with special [PAGE] tokens summarizing each page. These summaries are concatenated and fed into the decoder, which generates answers and predicts the answer page. Pretraining involves a layout-aware denoising task to align spatial and semantic features. The model supports input sequences up to 20480 tokens, addressing the quadratic complexity challenge of standard transformers for long inputs.
Key Results
- On MP-DocVQA, Hi-VT5 achieves 50.01% accuracy and 0.6572 ANLS in the oracle setup, outperforming baselines like BERT (34.78%), Longformer (45.87%), and LayoutLMv3 (42.70%). The page prediction accuracy reaches 79.23%, demonstrating effective localization of answer context.
- Training on only two pages per document yields performance close to full-document training, validating the hierarchical design and training efficiency.
- The model's ability to identify the answer page enhances interpretability and reasoning, making it suitable for real-world applications involving complex, multi-page documents.
Significance
This work addresses the critical challenge of understanding multi-page long documents, a common scenario in industry and administration. By overcoming sequence length limitations and integrating multimodal features, Hi-VT5 significantly advances document AI, enabling accurate, explainable question answering over complex documents. It paves the way for intelligent automation in sectors like finance, legal, and government, where multi-page document comprehension is essential.
Technical Contribution
The paper proposes a novel hierarchical transformer architecture that combines page-level summaries with a global document representation, supported by a layout-aware pretraining strategy. The integration of OCR, visual, and layout features in a scalable transformer framework represents a key technical innovation, allowing processing of sequences up to 20480 tokens without increasing model complexity. The approach also introduces a page prediction head for explainability, setting new standards for multi-page document understanding.
Novelty
This is the first work to implement a hierarchical, multimodal transformer tailored for multi-page document question answering, effectively addressing the sequence length barrier. The combination of page-wise encoding, layout-aware pretraining, and answer page prediction distinguishes it from prior single-page models, offering a scalable solution for real-world long document scenarios.
Limitations
- The model's performance degrades on documents exceeding 20 pages, and training costs are high due to the need for extensive GPU resources. Handling ultra-long documents remains a challenge.
- Robustness against OCR errors and complex layouts, especially handwritten or heavily formatted pages, needs improvement.
- Current training relies on partial document segments; full-length document training is computationally intensive and requires further optimization.
Future Work
Future directions include enhancing robustness to OCR inaccuracies, extending the model to handle longer documents beyond 20 pages, and optimizing training efficiency. Incorporating external knowledge bases and reasoning modules could further improve answer accuracy and interpretability. Additionally, exploring unsupervised pretraining strategies tailored for multi-page documents may broaden applicability.
AI Executive Summary
In the era of digital information overload, managing and extracting knowledge from complex, multi-page documents remains a significant challenge. Traditional question answering models excel on short texts but falter when faced with lengthy, multi-page files common in legal, financial, and administrative domains. Existing datasets and models largely focus on single-page scenarios, leaving a gap between research and real-world needs.
This paper introduces Hi-VT5, a hierarchical multimodal transformer architecture designed explicitly for multi-page document question answering. The core innovation lies in processing each page independently to generate concise summaries, which are then combined to form a comprehensive understanding of the entire document. The model leverages multimodal features—text, layout, and visual cues—and employs a novel pretraining task that aligns spatial and semantic representations, enabling it to handle sequences up to 20,480 tokens.
Extensive experiments on the newly created MP-DocVQA dataset demonstrate the effectiveness of Hi-VT5. The model achieves 50.01% accuracy and a 0.6572 ANLS in the oracle setup, outperforming baseline models like BERT, Longformer, and LayoutLMv3 by significant margins. Notably, the model can also predict the page where the answer is located with 79.23% accuracy, providing interpretability and reasoning transparency.
This work marks a substantial step forward in document AI, offering a scalable, explainable solution for understanding complex, multi-page documents. It opens new avenues for automating workflows in sectors such as finance, legal, and government, where long, structured documents are ubiquitous. Future research will focus on improving robustness, extending sequence length, and integrating external knowledge sources, aiming to realize fully autonomous, intelligent document processing systems.
Deep Analysis
Background
随着数字化信息的快速增长，企业和机构对高效、智能的文档理解需求日益增加。早期方法多依赖规则和模板，难以适应多样化布局和复杂内容。近年来，Transformer架构的引入极大推动了文本理解的发展，代表性模型如BERT、LayoutLM系列通过融合布局信息显著提升了文档理解能力。单页问答数据集如SingleDocVQA推动了模型性能，但实际场景中多页长文档的理解仍面临序列长度限制、布局复杂和多模态融合等难题。现有模型在处理超长文本时表现不佳，亟需创新架构突破。
Core Problem
多页长文档的理解涉及信息碎片化、序列长度限制和复杂布局等核心难题。传统Transformer模型受限于最大输入长度（如512或1024 tokens），难以直接处理数百页内容。OCR识别错误、布局多样性和多模态信息融合增加了理解难度。此外，缺乏有效机制识别答案所在页面，影响模型的推理和解释能力。这些问题严重制约多页文档在实际应用中的效果，亟需新型模型架构和训练策略。
Innovation
提出Hi-VT5架构，创新性地结合层次化多模态Transformer，突破序列长度限制，支持多页长文本问答。主要创新包括：
- �� 页级编码器：对每页内容进行独立编码，提取关键信息，支持长序列输入。
- �� 特殊[PAGE]标记：融合每页摘要，增强多页信息整合能力。
- �� 布局感知预训练：引入去噪任务，提升模型对复杂布局的理解。
- �� 多模态融合：结合OCR文本、视觉特征和布局信息，提升理解和推理能力。这些创新使模型能在单阶段完成多页问答，显著优于传统单页模型。
Methodology
- �� 输入：每页OCR文本、视觉特征和布局信息，结合问题。
- �� 编码：每页通过Transformer编码器提取关键信息，生成页级摘要[PAGE]标记。
- �� 融合：所有页的[PAGE]标记拼接，形成全局表示。
- �� 解码：解码器基于全局表示生成答案，并预测答案页。
- �� 预训练：布局感知去噪任务，掩盖部分tokens，模型利用布局信息预测。
- �� 微调：在部分长文档上训练后，使用全长文档进行微调，优化性能。
Experiments
在MP-DocVQA数据集上，模型采用部分样本训练，评估指标包括准确率和ANLS。对比基线如BERT、Longformer、LayoutLMv3，验证模型在多页长文本中的优势。模型参数约316M，支持20480 tokens输入，训练采用两页片段策略以降低成本。通过消融实验验证层次结构、布局感知任务和多模态融合的重要性。模型在答案准确率和页面预测准确率上均优于基线，证明其有效性。
Results
模型在oracle设置中达成50.01%的准确率，ANLS为0.6572，明显优于BERT（34.78%）和LayoutLMv3（42.70%）。页面预测准确率达79.23%，显示模型良好的推理和解释能力。微调实验表明，训练仅用两页即可获得接近全页性能，验证层次结构设计的有效性。模型在长文档场景中表现优异，显著提升多页文档问答的实用性。
Applications
该模型适用于自动处理财务报告、法律文件、行政档案等多页复杂文档。结合OCR和多模态特征，支持自动问答、信息抽取和文档索引，极大提升工作效率。未来可结合知识图谱实现推理增强，推动智能文档管理系统的发展。
Limitations & Outlook
模型在超长（超过20页）文档上表现尚有限，训练成本高，需大量GPU资源。对手写文本和复杂布局鲁棒性不足，OCR错误影响性能。未来需优化模型结构，降低计算成本，增强多模态融合能力。
Plain Language Accessible to non-experts
想象你在整理一本很厚的百科全书，每一章都很多页，里面有图片、表格和文字。现在你要回答一个问题，比如“哪一章讲的是动物？”传统方法就像只看一页，可能漏掉重要信息。而这个新方法就像有一个聪明的助手，先帮你总结每一章的重点内容，然后再告诉你答案在哪一章，甚至还能告诉你答案的具体位置。它还会告诉你答案来自哪一页，就像告诉你“答案在第5章第3页”。这样，你就可以快速找到答案，也能理解它是怎么找到的。这就像有个聪明的导游，帮你在复杂的书中找到想要的内容，既快又准。
ELI14 Explained like you're 14
假设你有一本很厚的书，要找出答案，比如“哪一页讲的是恐龙？”以前的方法就像只翻一页，可能错过重要信息。现在，这个新方法像有个超级助手，先帮你总结每一页的内容，然后告诉你答案在哪一页，还能告诉你答案是在哪一页找到的。它用一种聪明的方式，把所有页面的重点都记下来，然后再组合起来，帮你快速找到答案。这就像你有个聪明的朋友，帮你在一堆书里找到你要的内容，而且还能告诉你答案在哪一页，方便你自己去看。这让找信息变得又快又准，也更容易理解答案是怎么来的。
Abstract
Document Visual Question Answering (DocVQA) refers to the task of answering questions from document images. Existing work on DocVQA only considers single-page documents. However, in real scenarios documents are mostly composed of multiple pages that should be processed altogether. In this work we extend DocVQA to the multi-page scenario. For that, we first create a new dataset, MP-DocVQA, where questions are posed over multi-page documents instead of single pages. Second, we propose a new hierarchical method, Hi-VT5, based on the T5 architecture, that overcomes the limitations of current methods to process long multi-page documents. The proposed method is based on a hierarchical transformer architecture where the encoder summarizes the most relevant information of every page and then, the decoder takes this summarized information to generate the final answer. Through extensive experimentation, we demonstrate that our method is able, in a single stage, to answer the questions and provide the page that contains the relevant information to find the answer, which can be used as a kind of explainability measure.