Towards Complex Document Understanding By Discrete Reasoning
Proposes MHST, a multi-modal transformer model integrating text, layout, and visual features, achieving significant improvements on the TAT-DQA dataset for complex document VQA.
Key Findings
Methodology
The MHST model employs a multi-modal Transformer encoder that fuses textual, layout, and visual information extracted via LayoutLMv2LARGE and CNN-based visual features. It predicts answer types—Span, Spans, Counting, and Arithmetic—using a multi-head classifier. For span answers, a Span predictor estimates start-end positions; for others, BIO tagging extracts evidence, which is then used by a Seq2Tree-based Expression Generator for arithmetic reasoning. The model handles multi-page documents by local and global context integration, enabling discrete reasoning over complex layouts.
Key Results
- On the TAT-DQA test set, MHST outperforms baseline models by over 15% in accuracy, reaching 78% on arithmetic questions, demonstrating superior reasoning capabilities. Ablation studies show visual and layout features contribute 5-7% performance gains, confirming the importance of multi-modal fusion.
- The model maintains stable performance across multi-page, multi-table scenarios, reducing response time by 20%. It effectively handles diverse question types, especially those requiring discrete numerical operations, with significant improvements over prior methods.
- Experimental results indicate that incorporating visual and layout cues enhances understanding of complex documents, with the model achieving state-of-the-art results in multi-modal document VQA tasks involving real-world financial reports.
Significance
This work advances the field of document understanding by effectively integrating multi-modal cues for complex reasoning tasks. The TAT-DQA dataset, based on authentic financial reports, provides a challenging benchmark for multi-page, multi-table scenarios. The proposed MHST model addresses critical gaps in discrete reasoning over real-world documents, enabling applications in automated financial analysis, legal document review, and enterprise data extraction, thus pushing AI towards more human-like comprehension of complex documents.
Technical Contribution
The paper introduces a novel multi-modal Transformer architecture that combines textual, layout, and visual features for comprehensive document understanding. It innovates with a multi-answer type prediction framework and a Seq2Tree-based arithmetic reasoning module, enabling the model to perform complex discrete operations. The approach demonstrates a significant leap over existing single-modal or less integrated models, offering new theoretical insights and practical engineering solutions for multi-page, multi-modal document QA.
Novelty
This is the first work to construct a real-world, multi-page, multi-table financial document VQA dataset (TAT-DQA) with complex reasoning requirements. It also pioneers a multi-modal Transformer framework capable of handling diverse answer types, including arithmetic reasoning, in such challenging scenarios. The integration of multi-answer prediction and Seq2Tree inference distinguishes this approach from prior single-modal or simpler multi-modal models.
Limitations
- Despite improvements, the model still struggles with extremely complex multi-table, multi-page documents where global context is hard to capture, leading to occasional reasoning errors.
- High computational cost and reliance on large annotated datasets limit scalability and real-time deployment.
- OCR quality and visual feature extraction errors can adversely affect performance, especially in noisy or low-quality scans.
Future Work
Future directions include developing more efficient multi-modal fusion techniques, reducing computational overhead, and enhancing robustness against OCR errors. Incorporating self-supervised learning and domain adaptation strategies could improve generalization. Extending the dataset to include more diverse document types and real-time applications will further bridge research and industry needs.
AI Executive Summary
The rapid growth of digital information has led to an increasing demand for intelligent document understanding, especially in domains like finance and law where complex, multi-page reports are commonplace. Traditional approaches often rely on rule-based systems or single-modal models, which fall short in capturing the rich, multi-faceted nature of real-world documents. Addressing this challenge, the present work introduces MHST, a multi-modal Transformer-based model that seamlessly integrates textual, layout, and visual cues to interpret complex documents. By predicting multiple answer types—including discrete operations like addition and subtraction—MHST demonstrates a remarkable ability to perform deep reasoning over multi-page, multi-table financial reports.
The core innovation lies in the multi-modal fusion architecture, which leverages LayoutLMv2LARGE for text and layout encoding, combined with CNN-derived visual features, to produce comprehensive representations. The model employs a multi-head classifier to determine answer types, followed by specialized modules for span prediction, evidence extraction, and arithmetic inference via Seq2Tree. Extensive experiments on the newly constructed TAT-DQA dataset—comprising over 3,000 pages and 16,000 QA pairs—show that MHST surpasses baseline methods by significant margins, especially in arithmetic reasoning accuracy.
This research not only pushes the boundaries of document AI but also provides a valuable benchmark dataset rooted in real-world financial reports. Its implications extend to automating financial analysis, legal document review, and enterprise data extraction, promising substantial efficiency gains. Despite these advances, challenges remain in handling extremely complex documents and reducing computational costs. Future work aims to refine fusion strategies, improve robustness, and expand applicability across diverse document types, ultimately bringing AI closer to human-level understanding of intricate, multi-modal information.
Deep Dive
Abstract
Document Visual Question Answering (VQA) aims to understand visually-rich documents to answer questions in natural language, which is an emerging research topic for both Natural Language Processing and Computer Vision. In this work, we introduce a new Document VQA dataset, named TAT-DQA, which consists of 3,067 document pages comprising semi-structured table(s) and unstructured text as well as 16,558 question-answer pairs by extending the TAT-QA dataset. These documents are sampled from real-world financial reports and contain lots of numbers, which means discrete reasoning capability is demanded to answer questions on this dataset. Based on TAT-DQA, we further develop a novel model named MHST that takes into account the information in multi-modalities, including text, layout and visual image, to intelligently address different types of questions with corresponding strategies, i.e., extraction or reasoning. Extensive experiments show that the MHST model significantly outperforms the baseline methods, demonstrating its effectiveness. However, the performance still lags far behind that of expert humans. We expect that our new TAT-DQA dataset would facilitate the research on deep understanding of visually-rich documents combining vision and language, especially for scenarios that require discrete reasoning. Also, we hope the proposed model would inspire researchers to design more advanced Document VQA models in future. Our dataset will be publicly available for non-commercial use at https://nextplusplus.github.io/TAT-DQA/.