VeriSciQA: An Auto-Verified Dataset for Scientific Visual Question Answering
Proposes Cross-Modal Verification to build VeriSciQA, a dataset with 20,272 high-quality scientific visual QA pairs.
Key Findings
Methodology
This paper introduces a cross-modal verification framework that generates questions and answers from figure-citing paragraphs using large language models (e.g., GPT-4), then verifies them against figures with vision-language models (e.g., o4-mini). The process involves extracting atomic claims, generating multiple-choice questions with distractors, and applying cascaded filters—source consistency, visual dependence, and figure support—to eliminate erroneous QA pairs. This leverages the inherent text-figure alignment in scientific papers, reducing hallucinations and information asymmetry issues. The resulting dataset, VeriSciQA, contains 20,272 QA pairs across 20 scientific domains and 12 figure types, demonstrating high quality and diversity.
Key Results
- Models fine-tuned on VeriSciQA outperform those trained on existing datasets, with the best open-source model reaching 65% accuracy and proprietary models achieving 80.5%. Performance scales positively with data size, with an average improvement of 2.05% as data increases from 500 to 20,272 examples. Human evaluation confirms dataset quality, and models show robust generalization across tasks and figure types.
Significance
This work addresses the critical challenge of generating reliable, large-scale scientific visual QA datasets, which are essential for advancing AI understanding of complex scientific figures. By integrating textual and visual verification, the framework significantly reduces hallucination and misinformation, providing a scalable solution for the scientific community. The high-quality dataset supports training more accurate and interpretable models, fostering progress in scientific AI applications such as automated literature analysis, reproducibility, and research assistance. It also opens avenues for further research in multimodal scientific understanding, bridging gaps between textual descriptions and visual data.
Technical Contribution
The core innovation lies in combining a text-based question generation module with a vision-based verification module within a cascaded filtering pipeline. This approach exploits the natural alignment between figure-citing paragraphs and figures, ensuring questions are grounded in authors’ intent and visual evidence. The framework's modular design allows scalable, automated dataset construction, overcoming limitations of previous template-based or hallucination-prone LVLM synthesis methods. It introduces specific algorithms for atomic claim extraction, distractor generation conditioned on figures, and multi-stage verification, which collectively improve data quality and diversity.
Novelty
This study is the first to systematically integrate cross-modal verification with scientific paragraph-based question generation, leveraging the inherent text-figure alignment in peer-reviewed papers. Unlike prior works that rely solely on LVLMs for data synthesis, this framework explicitly filters out hallucinated or unsupported QA pairs, ensuring high fidelity. Its scalable, automated pipeline addresses the key bottleneck of data scarcity and quality, setting a new standard for scientific visual QA datasets and enabling broader open-source research in this domain.
Limitations
- Despite significant improvements, the framework may struggle with highly complex or poorly annotated figures, where visual understanding is limited. The reliance on citation paragraphs assumes accurate and detailed descriptions, which may not always be available, leading to potential gaps in data quality. Additionally, the computational cost of multi-stage verification is high, posing challenges for scaling to even larger datasets or real-time applications.
Future Work
Future research will focus on enhancing the understanding of complex figures through advanced multimodal models, integrating domain-specific knowledge, and developing more efficient verification algorithms. Expanding the dataset to include more diverse figure types and languages, as well as incorporating active learning strategies, will further improve scalability and robustness. Additionally, exploring zero-shot and few-shot learning paradigms using this high-quality data could accelerate scientific AI development.
AI Executive Summary
The rapid development of large vision-language models (LVLMs) has opened new frontiers in AI's ability to interpret complex visual data. However, applying these models to scientific figures—such as charts, graphs, and diagrams—remains challenging due to the dense, specialized information they encode. Existing datasets like ArXivQA and SciFiBench have made strides in scaling data generation but suffer from quality issues, including hallucinations and misalignments, which hinder model training and deployment.
To address these limitations, this paper introduces a novel cross-modal verification framework that leverages the inherent text-figure alignment in scientific papers. The approach begins by extracting figure-citing paragraphs, which contain authors’ interpretations and verifiable claims about figures. Using large language models like GPT-4, the system generates questions and answers from these claims, then creates distractors conditioned on the figure with vision-language models like o4-mini. The core innovation lies in applying cascaded filters—text-based source consistency, visual dependence, and figure support—to ensure only accurate, well-grounded QA pairs are retained.
This pipeline results in VeriSciQA, a dataset comprising over 20,000 high-quality QA pairs spanning 20 scientific domains and 12 figure types. Extensive experiments demonstrate that models fine-tuned on VeriSciQA outperform counterparts trained on existing datasets, with accuracy improvements from 50% to 65% for open-source models and up to 80.5% for proprietary systems. Performance scales positively with data size, and human evaluations confirm the dataset’s high fidelity.
The significance of this work lies in its scalable, automated approach to constructing reliable scientific visual QA datasets, which are crucial for advancing AI's interpretability and utility in scientific research. By reducing hallucination and information bias, the framework paves the way for more accurate, interpretable models capable of supporting scientific discovery, reproducibility, and automation. Future directions include enhancing understanding of complex figures, expanding dataset diversity, and optimizing computational efficiency, ultimately accelerating the integration of AI into scientific workflows.
Deep Dive
Limitations & Outlook
What gaps remain?
Abstract
Large Vision-Language Models (LVLMs) show promise for scientific applications, yet open-source models still struggle with Scientific Visual Question Answering (SVQA), namely answering questions about figures from scientific papers. A key bottleneck is the lack of public, large-scale, high-quality SVQA datasets. Although recent work uses LVLMs to synthesize data at scale, we identify systematic errors in their resulting QA pairs, stemming from LVLMs' inherent limitations and information asymmetry between figures and text. To address these challenges, we propose a Cross-Modal verification framework that generates questions and answers purely from figure-citing paragraphs, then verifies them against the figures themselves, leveraging the inherent text-figure alignment in scientific papers to filter out erroneous QA pairs. We instantiate this framework to curate VeriSciQA, a dataset of 20,272 QA pairs spanning 20 scientific domains and 12 figure types. Difficulty assessment reveals a notable accuracy gap between the best open-source model (65%) and the best proprietary model (80.5%), demonstrating room for improvement. Moreover, models fine-tuned on VeriSciQA achieve consistent improvements on SVQA benchmarks, with performance gains that scale with data size, surpassing models trained on existing datasets. Human evaluation further validates the improved quality of VeriSciQA. These results demonstrate that continued data expansion via our scalable framework can further advance SVQA capability in the open-source community. Our dataset is publicly available at https://huggingface.co/datasets/datajuicer/VeriSciQA.