From Terminology to Diagrams: Visual-Instruction Generation for Scientific Diagram Understanding
SciGram framework generates 1.4M visual instructions, advancing scientific diagram understanding and surpassing SOTA.
Key Findings
Methodology
The SciGram framework uses a four-stage process: terminology extraction, atomic fact generation, diagram retrieval, and instruction generation. Qwen2-VL generates multimodal instructions, including captions and MCQs.
Key Results
- SciGram models improved TQA diagram question accuracy to 80.12%, a 3.04% increase over LLaVA OV.
- Achieved 97.62% accuracy on SQA visual support questions, surpassing prior SOTA by 2.92%.
- Excelled on AI2D dataset with 92.42% accuracy on transparent labels.
Significance
This study addresses long-standing challenges in scientific diagram understanding, providing new tools for education and research. SciGram fills gaps in existing datasets and significantly enhances model performance on diagram-centric tasks.
Technical Contribution
Introduced terminology-driven instruction generation, ensuring broad scientific concept coverage. Compared to natural image-based models, SciGram excels in scientific diagram understanding using fewer training instances.
Novelty
First to propose a terminology-driven framework for scientific diagram instruction generation, combining curriculum-derived terms and web diagrams to create a large-scale multimodal dataset.
Limitations
- Some diagrams from web retrieval contain noise; ~24% are non-scientific diagrams.
- 16% of generated MCQs have annotation errors.
- Certain questions rely on prior knowledge, reducing diagram dependence.
Future Work
Future efforts could optimize diagram filtering, improve annotation verification, and explore cross-disciplinary applications.
AI Executive Summary
Scientific diagrams convey functional or relational meaning, unlike natural images. Existing vision-language models struggle with understanding these diagrams.
This study introduces the SciGram framework, which constructs a dataset of 194K diagrams and 1.4M instructions through terminology extraction, atomic fact generation, diagram retrieval, and instruction generation. Models fine-tuned on SciGram outperform or match SOTA on benchmarks like TQA, SQA, and AI2D.
SciGram-enhanced models significantly improve scientific diagram understanding, offering new tools for educational technology and research while demonstrating the potential of terminology-driven instruction generation for multimodal learning.
Deep Analysis
Background
Scientific diagrams are essential tools in education and research. While vision-language models excel at natural image tasks, they struggle with diagram understanding due to limited training data. Benchmarks like AI2D, TQA, and SQA provide diagram-centric tasks but lack sufficient data for effective model training.
Core Problem
Scientific diagrams are abstract and symbolic, conveying complex relationships and processes. Understanding them requires grounding in scientific context, but existing datasets lack coverage, limiting model performance.
Innovation
Introduced a terminology-driven instruction generation framework combining curriculum-derived terms and web diagrams to create the SciGram dataset. Ensures broad scientific concept coverage and semantic consistency.
Methodology
- �� Terminology extraction: Extract scientific terms from curricula and filter using RoBERTa embeddings.
- �� Atomic fact generation: Generate concise scientific statements using LLaMA-Instruct based on term combinations.
- �� Diagram retrieval: Query web for diagrams, apply hashing and filtering.
- �� Instruction generation: Use Qwen2-VL to create captions and MCQs for multimodal training.
Experiments
Fine-tuned LLaVA models on SciGram and evaluated performance on TQA, SQA, and AI2D benchmarks. Experiments included baseline comparisons and ablation studies, focusing on visual support tasks.
Results
Fine-tuned models achieved 80.12% accuracy on TQA diagram questions, 97.62% on SQA visual support, and 92.42% on AI2D transparent labels.
Applications
SciGram can be used for educational tools like classroom aids and scientific research involving complex visual data analysis.
Limitations & Outlook
Web-retrieved diagrams contain noise, MCQ annotation errors are significant, and some questions rely on prior knowledge. Future work should refine filtering and validation processes.
Plain Language Accessible to non-experts
Imagine you're in a kitchen following a recipe. Scientific diagrams are like the recipe, showing how ingredients combine into a dish. The SciGram framework acts like a smart assistant, extracting key steps, generating detailed instructions, and providing visual examples for each dish. This helps models better understand the content and relationships in scientific diagrams.
ELI14 Explained like you're 14
Hey, imagine you're playing a puzzle game! Scientific diagrams are like the game's map, showing where treasures and traps are. SciGram is like a super helper—it not only finds the map but also explains what each mark means and quizzes you with questions to test your understanding. Cool, right?
Glossary
Terminology-driven
A method for generating data based on domain-specific terms, ensuring broad scientific concept coverage.
Core framework for constructing the SciGram dataset.
Scientific Diagram
Graphical representations conveying scientific concepts, relationships, or processes.
The primary target of this study.
Vision-Language Model
A machine learning model combining visual and linguistic information for tasks.
Used for scientific diagram understanding.
MCQ Generation
Method for creating multiple-choice questions based on diagram content.
Used to evaluate visual reasoning capabilities.
Ablation Study
Analysis by removing model components to assess their impact on performance.
Used to analyze SciGram dataset contributions.
Open Questions Unanswered questions from this research
- 1 How to reduce noise in web-retrieved diagrams?
- 2 How to enhance diagram dependence in MCQs?
- 3 How to extend to cross-disciplinary domains?
Applications
Immediate Applications
Educational Aid
Helps students understand scientific diagrams, improving classroom learning outcomes.
Scientific Research
Supports scientists in analyzing complex diagram data, enhancing research efficiency.
Long-term Vision
Cross-disciplinary Expansion
Apply the framework to social sciences and medicine for comprehensive multimodal learning.
Abstract
Vision-language models (VLMs) have demonstrated strong performance in visual question answering with natural images. However, they continue to struggle with scientific diagrams, which are designed to convey functional or relational meaning rather than literal scenes. We therefore introduce a framework for generating large-scale diagram-grounded instruction data by leveraging terminology derived from scientific curricula. Our approach systematically extracts domain concepts, synthesizes atomic facts, retrieves relevant diagrams from the web, and generates multimodal supervision in the form of diagram captions and multiple-choice questions. Using this pipeline, we construct SciGram, a dataset of over 194K diagrams and 1.4M visual instructions across life, earth, and physical sciences. Despite relying on noisy web data and synthetic annotations, models fine-tuned on SciGram achieve substantial improvements on diagram-centric benchmarks, including TQA, ScienceQA, and AI2D, outperforming or matching state-of-the-art VLMs while using fewer training instances. Furthermore, augmenting existing models such as LLaVA OneVision with SciGram establishes new state-of-the-art performance on diagram question answering. Our results highlight the effectiveness of terminology-grounded instruction generation as a general strategy for improving vision-language reasoning in scientific domains. To support future research in scientific diagram understanding, we release both the SciGram dataset and models.