VisualMRC: Machine Reading Comprehension on Document Images

The evolution of machine reading comprehension (MRC) has transitioned from simple fact-based question answering datasets like SQuAD to more complex scenarios involving multi-turn dialogues and long documents. Prior works such as BERT-based models and span-extraction techniques significantly improved short-text understanding. Simultaneously, the rise of visual question answering (VQA) datasets like VQA v2, VizWiz, and DocVQA introduced multimodal challenges, emphasizing OCR and image content reasoning. However, these datasets primarily focus on short answers or span extraction, lacking emphasis on long, abstractive responses and layout-aware understanding. Recent efforts like DocVQA incorporate document images but are limited to specific domains and do not emphasize natural language generation. This gap motivates the creation of VisualMRC, designed to evaluate models' ability to understand and generate long, contextually rich answers grounded in complex visual layouts and multimodal information.

Despite progress, current models struggle with understanding complex document layouts, integrating visual and textual cues, and generating coherent long answers. Existing datasets either lack diversity, focus on short span answers, or do not incorporate detailed visual layout information. Real-world documents—such as webpages, reports, and forms—contain multiple regions with different semantic roles, requiring models to detect, interpret, and reason over these regions simultaneously. The core challenge lies in designing models that can effectively fuse visual features, spatial relationships, and textual content to produce accurate, natural language responses, especially in multi-domain, multi-format contexts. Overcoming these bottlenecks is crucial for deploying AI in practical document understanding applications.

The paper introduces several innovations: 1) a large-scale, multi-domain dataset (VisualMRC) with annotated regions, long answers, and diverse content; 2) a multimodal Transformer-based architecture that extends T5 and BART by integrating ROI detection, visual features, and semantic labels; 3) a saliency mechanism that dynamically identifies relevant tokens, improving focus and reasoning; 4) a multi-task training framework combining visual region recognition and language generation, enabling end-to-end learning of layout-aware comprehension. These innovations collectively address the limitations of prior short-answer, layout-agnostic models, enabling deep understanding of complex documents and natural language answer generation.

• �� Data collection: Gathered 10,197 webpage screenshots across 35 domains, annotated regions with semantic labels, and extracted OCR text using Tesseract.
• �� ROI detection: Trained Faster R-CNN to identify key regions like titles, paragraphs, images, and data tables, providing bounding boxes and class labels.
• �� OCR processing: Extracted textual content from each ROI, forming a rich textual representation.
• �� Input construction: Concatenated question tokens, ROI labels, positional encodings, OCR tokens, and visual features into a unified sequence.
• �� Model architecture: Built upon Transformer encoder-decoder (T5/BART), extended with region embeddings, position embeddings, and visual features.
• �� Saliency mechanism: Calculated token relevance scores via a learned sigmoid function, guiding the model to focus on critical tokens.
• �� Training: Minimized combined negative log-likelihood and saliency loss, employing multi-task learning.
• �� Evaluation: Used BLEU, METEOR, ROUGE-L, CIDEr, and BERTScore on the test set, with ablation studies to assess component contributions.

Experiments utilized the VisualMRC dataset, with models trained on ground-truth ROIs and OCR features. Baselines included T5 and BART without visual inputs, and the SOTA VQA model Hu et al. 2020. Evaluation metrics encompassed BLEU, CIDEr, METEOR, ROUGE-L, and BERTScore, providing comprehensive assessment of answer quality. Ablation studies examined the impact of ROI labels, visual features, and saliency mechanisms. Results showed that the full model achieved BLEU-4 of 43.0 and CIDEr of 346.0, outperforming baselines by significant margins. The experiments confirmed the effectiveness of multimodal fusion and layout-aware design, with detailed analysis of error cases and component contributions.

The proposed model achieved BLEU-4 of 43.0, CIDEr of 346.0, surpassing baseline models by over 50%. Incorporating visual features and semantic ROI labels improved performance by around 10%, demonstrating the benefit of multimodal integration. The model significantly outperformed Hu et al. 2020's VQA in long answer generation, especially in reasoning tasks involving complex layouts. Ablation results confirmed that ROI labels, visual features, and saliency mechanisms each contributed substantially to overall performance, validating the design choices. Despite these advances, the model still lags behind human performance, indicating room for further improvement in robustness and reasoning capabilities.

This model can be deployed in enterprise document management, legal and financial document analysis, and automated report summarization. It requires OCR and ROI detection tools, along with fine-tuning on domain-specific data. Its ability to generate detailed, context-aware answers makes it suitable for intelligent document retrieval, content extraction, and digital assistant systems. Long-term, integrating such models into comprehensive knowledge management platforms could revolutionize how organizations process and understand large-scale unstructured documents, enabling fully automated, intelligent workflows.

The model's robustness decreases with highly complex or noisy documents, especially those with intricate layouts or poor image quality. Training relies on extensive annotated data, which is costly and limits scalability. Fusion strategies may not fully capture deep relationships between regions, and computational costs remain high. Future work should focus on improving robustness, reducing training costs, and enhancing reasoning over multi-layered information for broader real-world applicability.