InfographicVQA
Introduces InfographicVQA dataset and evaluates Transformer-based models, revealing significant performance gaps in understanding complex infographics.
Key Findings
Methodology
The study assembled a dataset of 5,485 infographics with 30,035 question-answer pairs, emphasizing elementary reasoning and arithmetic. It employs Transformer-based models M4C and LayoutLM, which fuse question, OCR text, and visual features through multi-layer attention mechanisms. The models incorporate spatial position encoding and early visual-text fusion, trained via masked language modeling and span prediction objectives. Performance is evaluated using ANLS and accuracy metrics, revealing substantial gaps compared to human performance, highlighting the complexity of infographics and the need for advanced multimodal reasoning.
Key Results
- M4C achieves an ANLS of 0.134 and about 6.58% accuracy on the test set, far below human scores of 0.98 ANLS and 95.7% accuracy, indicating difficulty in handling complex layouts and multi-source data. LayoutLM's performance on image-span questions is similarly limited, with ANLS around 0.136. Incorporating OCR and vocabulary upper bounds boosts performance to 77.4%, demonstrating that most answers are obtainable via OCR or common answer sets. The results underscore the challenge in multi-source fusion and reasoning operations, guiding future model improvements.
Significance
This work pioneers a dedicated dataset and task for automatic understanding of infographics, a prevalent form of visual communication. It bridges the gap between natural scene VQA and document understanding, addressing the unique challenges posed by complex layouts, data visualizations, and textual content. The dataset offers a benchmark for developing models capable of multi-source reasoning, with broad implications for automated data extraction, intelligent reporting, and educational tools. It advances the field by highlighting current limitations and setting clear goals for future research in multimodal document comprehension.
Technical Contribution
The primary contribution is the creation of a large-scale, diverse infographic dataset tailored for VQA, combined with the adaptation of Transformer architectures—M4C and LayoutLM—for this task. The models integrate spatial layout encoding, early visual-text fusion, and multi-operation reasoning, pushing the boundaries of multimodal understanding. The training involves pretraining with masked tasks and fine-tuning for span prediction, establishing a new baseline for complex document image understanding. This work also provides insights into feature extraction challenges and multimodal fusion strategies necessary for future improvements.
Novelty
This is the first comprehensive effort to formulate VQA specifically for infographics, integrating layout, textual, and visual data in a unified framework. Unlike prior datasets focused on natural images or isolated document types, InfographicVQA captures the diversity and complexity of real-world infographics, emphasizing reasoning over layout and data visualizations. The innovative model design combines spatial encoding, early visual-text fusion, and multi-source reasoning, setting a new standard for multimodal document understanding. This approach opens new research directions for AI systems to interpret complex visual data.
Limitations
- Models still struggle with complex reasoning involving multiple steps and arithmetic operations, indicating the need for more sophisticated reasoning modules and feature representations.
- The dataset is primarily English-centric, limiting cross-lingual generalization. Expanding to other languages and domains remains a challenge.
- High computational costs for training and inference pose practical barriers for deployment in real-world applications, necessitating efficiency improvements.
Future Work
Future directions include enhancing visual feature extraction with advanced CNNs or vision transformers, improving layout-aware reasoning modules, and integrating multi-step reasoning capabilities. Expanding the dataset to include multi-lingual infographics and diverse domains will improve model robustness. Additionally, exploring end-to-end training with reinforcement learning or meta-learning could help models better handle complex reasoning tasks. Ultimately, the goal is to develop AI systems that understand and interpret infographics as seamlessly as humans, enabling smarter data-driven decision-making and automated content analysis.
AI Executive Summary
Infographics serve as a powerful means of conveying complex information quickly, combining text, visuals, and data visualizations in a compact form. However, automating their understanding remains a significant challenge due to their intricate layouts and multi-source data. Existing visual question answering (VQA) models excel in natural images but falter when faced with the layered, structured nature of infographics.
To address this gap, we introduce the InfographicVQA dataset, comprising 5,485 diverse infographics annotated with over 30,000 question-answer pairs. The dataset emphasizes questions requiring elementary reasoning, counting, and arithmetic operations, reflecting real-world information extraction needs. We adapt state-of-the-art models—M4C, a multimodal Transformer, and LayoutLM, a layout-aware BERT variant—to this task, integrating spatial position encoding and early visual-text fusion.
Experimental results reveal a stark performance gap: M4C achieves an ANLS of 0.134 and about 6.6% accuracy, far below human scores of 0.98 and 95.7%. LayoutLM's performance on image-span questions is similarly limited. Incorporating OCR and vocabulary-based upper bounds boosts accuracy to over 77%, indicating that most answers are accessible via OCR or common answer sets. These findings underscore the complexity of infographics and the need for more sophisticated models capable of multi-source reasoning.
This work establishes a new benchmark for multimodal document understanding, highlighting critical challenges and guiding future research. Advancing visual feature extraction, layout modeling, and multi-step reasoning will be essential to bridge the gap between current AI capabilities and human-level understanding of complex visual data. Ultimately, this research paves the way for smarter automated data extraction, educational tools, and decision support systems, transforming how machines interpret visual information in our data-rich world.
Deep Dive
Abstract
Infographics are documents designed to effectively communicate information using a combination of textual, graphical and visual elements. In this work, we explore the automatic understanding of infographic images by using Visual Question Answering technique.To this end, we present InfographicVQA, a new dataset that comprises a diverse collection of infographics along with natural language questions and answers annotations. The collected questions require methods to jointly reason over the document layout, textual content, graphical elements, and data visualizations. We curate the dataset with emphasis on questions that require elementary reasoning and basic arithmetic skills. Finally, we evaluate two strong baselines based on state of the art multi-modal VQA models, and establish baseline performance for the new task. The dataset, code and leaderboard will be made available at http://docvqa.org