BioMol-MQA: A Multi-Modal Question Answering Dataset For LLM Reasoning Over Bio-Molecular Interactions
BioMol-MQA constructs a multi-modal drug interaction QA dataset integrating knowledge graphs, text, and SMILES, enhancing LLM reasoning in biomedical tasks.
Key Findings
Methodology
This work introduces BioMol-MQA, a dataset combining drug-protein knowledge graphs, textual summaries, and molecular structures (SMILES). The creation process involves data collection from biomedical databases, text augmentation via Wikipedia, molecular relationship extraction using GPT-4, and question generation through automated prompts. The dataset emphasizes complex, relational multi-modal reasoning, with questions designed to require retrieval and synthesis across multiple data sources. Experiments demonstrate that current LLMs perform poorly without retrieval augmentation, highlighting the importance of RAG frameworks for accurate biomedical reasoning.
Key Results
- Models without background data achieve less than 30% accuracy; with background, accuracy exceeds 70%, confirming the effectiveness of retrieval-augmented approaches.
- Combining knowledge graph, text, and molecular data improves question-answering accuracy by approximately 25% over single-modality models.
- Question complexity metrics, such as an average length of 66.6 tokens and entropy of 5.67, reflect the dataset’s challenging nature.
Significance
This dataset addresses a critical gap in biomedical AI, enabling models to reason over complex multi-modal data for drug safety and interaction prediction. It advances the development of robust, explainable AI systems capable of supporting clinical decision-making and drug discovery, ultimately contributing to safer pharmacological practices and personalized medicine.
Technical Contribution
The paper proposes an end-to-end pipeline for multi-modal knowledge graph construction, molecular relationship extraction with GPT-4, and automated question generation. It introduces multi-hop reasoning questions and integrates molecular interactions into knowledge graphs, pushing the frontier of multi-modal reasoning and retrieval in biomedical contexts.
Novelty
This is the first comprehensive dataset that combines knowledge graphs, textual summaries, and molecular structures for complex question answering in biomedicine. The use of GPT-4 for molecular relationship extraction and question synthesis represents a novel application of large language models in biomedical multi-modal data integration.
Limitations
- Limited dataset size focusing mainly on drug-protein interactions; broader biomedical relations need inclusion for generalization.
- Current models struggle with multi-modal fusion and reasoning, requiring further architectural innovations.
- High-quality annotations depend on manual verification, limiting scalability and automation.
Future Work
Future efforts will expand data sources, improve molecular relationship extraction accuracy, and develop end-to-end multi-modal reasoning architectures. Incorporating dynamic knowledge updates and exploring transfer learning across biomedical domains are also promising directions.
AI Executive Summary
The rapid advancement of large language models (LLMs) has revolutionized natural language processing, yet their application in high-stakes biomedical fields remains limited by data complexity and reasoning challenges. Traditional datasets often focus on single modalities, such as text or structured data, which restricts models’ ability to understand intricate biological relationships like drug interactions. Recognizing this gap, the authors present BioMol-MQA, a novel multi-modal question-answering dataset designed specifically for biomedical reasoning involving drugs, proteins, and molecular structures.
BioMol-MQA integrates three core data modalities: a comprehensive knowledge graph capturing drug-protein interactions, textual summaries from Wikipedia providing contextual background, and SMILES strings encoding molecular structures. The dataset construction involves meticulous data collection, text refinement via GPT-4 to enhance complexity, molecular relationship extraction leveraging GPT-4’s chemistry reasoning, and automated question generation that synthesizes multi-modal information into challenging queries. These questions require models to retrieve relevant information from multiple sources and perform complex reasoning, including multi-hop inference.
Experimental results reveal that existing LLMs perform poorly on these tasks without retrieval augmentation, with accuracy below 30%. When provided with relevant background data, performance improves significantly, surpassing 70%. Combining all three modalities yields approximately 25% better accuracy than single-modality baselines, demonstrating the value of multi-modal integration. The dataset’s complexity metrics, such as an average question length of 66.6 tokens and high entropy scores, underscore its challenging nature.
This work offers a vital resource for advancing biomedical AI, enabling models to reason over complex, multi-faceted data crucial for drug safety, personalized medicine, and drug discovery. It also highlights the importance of robust retrieval frameworks like RAG, which ground language models in factual, domain-specific knowledge. Future directions include expanding the dataset, refining molecular relationship extraction, and developing end-to-end multi-modal reasoning systems capable of dynamic knowledge updating and real-world deployment.
Deep Analysis
Background
Biomedical AI的发展经历了从单一文本处理到多模态融合的演变。早期研究如BioBERT和PubMedBERT在文本理解方面取得突破，但在药物交互和关系推理中仍受限。知识图谱如Hetionet和DrugBank提供结构化关系，但缺少丰富的语义信息。近年来，结合多模态数据的研究逐渐兴起，旨在提升模型的推理能力。现有数据集如MoleculeQA和PubChemQA存在模板化、单一模态限制，难以模拟真实药物交互场景。本文旨在构建一个集成知识图谱、文本和分子结构的多模态数据集，推动药物安全、个性化医疗等应用。
Core Problem
当前模型在药物交互推理中表现不足，主要原因是缺乏丰富的多模态数据和复杂关系建模能力。药物-蛋白关系、分子结构和临床文本信息未能有效融合，导致推理难度大、准确率低。高风险场景如药物安全性评估亟需更强的推理能力，但现有数据资源和模型支持有限，成为制约其应用的瓶颈。如何设计一个能充分利用多模态信息、实现复杂关系推理的系统，成为亟待解决的问题。
Innovation
本研究的创新点包括：1）构建融合药物、蛋白质、文本和SMILES的多模态知识图谱，丰富关系表达；2）利用GPT-4分析SMILES字符串，提取潜在分子关系，增强知识图谱的细粒度信息；3）自动生成复杂多模态问答，模拟真实药物交互场景，推动多模态推理研究。相比传统单模态或模板化数据集，极大提升了数据复杂性和推理难度，为未来多模态AI模型提供了丰富资源。
Methodology
- �� 数据采集：整合药物-蛋白知识图谱，标注交互类型，获得节点和边信息。
- �� 文本增强：用Wikipedia和PubMed摘要丰富实体背景信息，提升语义密度。
- �� 分子关系提取：利用GPT-4分析SMILES字符串，识别分子间潜在相互作用，如氢键、π-π堆积等。
- �� 问答生成：基于知识图谱三元组，自动生成多模态问答，确保关系推理的复杂性。
- �� 质量验证：结合人工和自动评估，筛选高质量问答，确保难度与多样性。
Experiments
采用药物-蛋白知识图谱作为基础，划分训练、验证、测试集。模型在多模态检索与推理任务上评估，指标包括准确率、F1值和复杂度指标。对比单模态模型，验证多模态融合优势。进行关系提取消融实验，分析不同关系类型对性能的影响。模型在不同难度级别的问答上均表现出显著提升，验证数据集的挑战性。
Results
未提供背景信息时，模型准确率不足30%；引入背景后，提升至70%以上。多模态模型比单一模态提升约25%的准确率。问答复杂度指标显示，数据集具有较高的语义和结构复杂性，验证其挑战性。关系提取的准确率超过95%，支持多层次推理。
Applications
该数据集可用于药物安全性评估、药物-蛋白关系推理、个性化医疗方案设计等。模型需结合多模态信息进行推理，适应临床和药物研发中的复杂场景。未来还可扩展到其他生物医学关系网络，推动行业智能化升级。
Limitations & Outlook
目前规模有限，主要集中在药物-蛋白关系，泛化能力不足。多模态信息融合和关系推理仍需优化，模型复杂度高，计算成本大。高质量标注依赖人工验证，自动化程度不足，未来需加强自动标注和知识动态更新。
Plain Language Accessible to non-experts
想象你在厨房里准备一道复杂的菜肴。你需要各种原料（药物、蛋白质、分子结构）和不同的工具（文本描述、知识图谱）来理解每个原料的特性和它们之间的关系。单靠一种信息（比如只看食谱或只看原料）很难做出美味的菜肴。你必须结合多种信息，理解每个原料的作用、相互影响，才能做出正确的判断。这就像研究药物交互一样，只有同时考虑药物的结构、作用和关系，才能确保安全有效。这个过程就像厨师调配食材，既要理解每个原料的本质，又要知道它们怎么配合，才能做出令人满意的菜肴。
ELI14 Explained like you're 14
想象你在玩一个超级复杂的拼图游戏。每个拼图块代表一种药物或蛋白质，它们之间有很多隐藏的关系。你需要把这些拼图块拼在一起，才能看到完整的图片。可是，有时候只看一块拼图是不够的，你还得看它旁边的拼图，甚至要知道每个拼图的详细信息，比如它的形状和颜色。这个研究就像教你用多种线索拼出药物之间的关系。科学家们用电脑和智能程序帮忙，把这些信息结合起来，像拼图一样拼出药物的关系图。这样一来，医生就能更好地理解药物的作用，确保用药安全，避免不良反应。就像拼图游戏一样，越拼越清楚，最后能拼出完整的画面。
Abstract
Retrieval augmented generation (RAG) has shown great power in improving Large Language Models (LLMs). However, most existing RAG-based LLMs are dedicated to retrieving single modality information, mainly text; while for many real-world problems, such as healthcare, information relevant to queries can manifest in various modalities such as knowledge graph, text (clinical notes), and complex molecular structure. Thus, being able to retrieve relevant multi-modality domain-specific information, and reason and synthesize diverse knowledge to generate an accurate response is important. To address the gap, we present BioMol-MQA, a new question-answering (QA) dataset on polypharmacy, which is composed of two parts (i) a multimodal knowledge graph (KG) with text and molecular structure for information retrieval; and (ii) challenging questions that designed to test LLM capabilities in retrieving and reasoning over multimodal KG to answer questions. Our benchmarks indicate that existing LLMs struggle to answer these questions and do well only when given the necessary background data, signaling the necessity for strong RAG frameworks.