MMSci: A Dataset for Graduate-Level Multi-Discipline Multimodal Scientific Understanding

The evolution of AI in scientific research has seen rapid progress with multimodal models like CLIP, Flamingo, and GPT-4V, enabling better understanding of visual data. Early datasets such as FigureQA and SciCap focused on synthetic or limited chart types, insufficient for real-world scientific visuals. Recent efforts include arXiv-based collections, but these lack peer review and comprehensive coverage across disciplines. Scientific figures are inherently complex, ranging from schematic diagrams to microscopy images, requiring domain expertise. High-quality, diverse datasets are essential for training models capable of deep understanding and reasoning about such visuals, which remains a significant challenge in AI research.

Current models underperform on complex scientific figures, especially in multi-disciplinary contexts requiring detailed understanding and reasoning. Existing datasets lack diversity and realism, limiting model generalization. The core issue is the scarcity of high-quality, annotated, multi-disciplinary data that captures the complexity of real scientific visuals. This hampers the development of AI systems capable of autonomous interpretation, which is crucial for automating literature analysis, hypothesis generation, and experimental planning in scientific workflows.

This work introduces several innovations: 1) a large-scale, multi-disciplinary dataset with 72 fields and diverse figure types, 2) a comprehensive benchmark with figure captioning and matching tasks, 3) novel evaluation metrics tailored for scientific visuals, including large-model-based assessments, 4) a training pipeline combining fine-tuning and continual pretraining on interleaved article-image data. These innovations enable models to learn nuanced understanding of scientific visuals, surpassing prior datasets limited to simple charts or synthetic data, and establish new standards for scientific multimodal AI.

• �� Data collection: Extracted 131,393 peer-reviewed articles from Nature Communications, including full texts, figures, and captions.
• �� Sub-figure extraction: Used regex to identify sub-figure labels, resulting in 514,054 sub-captions.
• �� Figure classification: Employed GPT-4o to categorize images into schematic, microscopic, experimental, etc.
• �� Benchmark design: Developed figure captioning and multiple-choice matching tasks, with detailed evaluation metrics.
• �� Model training: Fine-tuned Qwen2-VL-7B on task-specific datasets, incorporating multi-turn conversations generated via GPT-4.
• �� Pretraining: Conducted continual pretraining on combined article and figure data to enhance knowledge representation.
• �� Evaluation: Assessed models using BLEU, ROUGE, METEOR, BERTScore, CIDEr, FACTSCORE, and G-EVAL, comparing against human experts.

The experimental setup involved training models on the collected dataset with multiple tasks, including figure captioning and matching. Baseline models included open-source options like LLaVA, Kosmos-2, and proprietary models like GPT-4V, Claude-3.5. Fine-tuning strategies included task-specific supervised learning and continual pretraining. Evaluation metrics covered accuracy, BLEU, ROUGE, CIDEr, and large-model-based scores. Ablation studies examined the impact of data diversity, multi-task learning, and pretraining. The dataset’s broad disciplinary coverage allowed testing across various scientific domains, ensuring robustness and generalization of the models.

Fine-tuned Qwen2-VL-7B achieved 87.48% accuracy on multiple-choice tasks, outperforming GPT-4o and human experts. The model demonstrated superior performance in detailed figure captioning, with CIDEr scores significantly higher than baseline models. Pretraining on article-figure interleaved data improved downstream tasks in materials science, with notable gains in complex schematic and microscopic images. Ablation studies confirmed that diversity and multi-task training contributed substantially to performance improvements. These results validate the dataset’s effectiveness in training models for real-world scientific figure understanding.

The dataset and trained models can be applied in automated literature review, scientific knowledge extraction, and intelligent research assistants. They enable automatic interpretation of complex figures, aiding researchers in hypothesis formulation and data analysis. Industry applications include scientific publishing, data visualization, and educational tools, where automated figure understanding can streamline workflows and enhance comprehension. The models require domain-specific fine-tuning, but their potential to accelerate scientific discovery is substantial.

Despite progress, models still face challenges with highly specialized or extremely complex figures requiring deep domain knowledge. Computational costs for training and inference are high, limiting scalability. The dataset’s focus on peer-reviewed articles may introduce disciplinary bias, affecting generalization. Future work should incorporate external knowledge bases, improve reasoning capabilities, and expand dataset coverage to address these issues, aiming for more robust and efficient scientific AI systems.