MedHopQA: A Disease-Centered Multi-Hop Reasoning Benchmark and Evaluation Framework for LLM-Based Biomedical Question Answering

Biomedical text mining has traditionally been organized around task-specific information extraction. Named entity recognition, relation extraction, event detection, and document classification have been developed and evaluated as discrete, pipelined subtasks, each with dedicated models, training data, and evaluation benchmarks. This task-specific paradigm has supported large-scale biocuration efforts by enabling the systematic extraction of structured knowledge from the literature through pre-defined schemas.

In contrast, natural language question answering provides end users with a more flexible interaction, in which information needs are posed directly as ad-hoc questions and system responses returned in fluent natural language. The rise of large language models (LLMs) has made this setting increasingly practical, enabling systems to retrieve, combine, and interpret information to produce coherent answers. However, natural language question answering exposes different aspects of system behavior, particularly the ability to reliably integrate information across sources and perform multi-step reasoning. As a result, this shift introduces new demands on dataset construction and evaluation, requiring benchmarks that capture not only factual correctness but – crucially – can reliably evoke and evaluate reasoning over biomedical knowledge.

Existing biomedical QA benchmarks are limited in evaluating LLM reasoning capabilities. Multiple-choice formats allow models to succeed by answer elimination rather than inference, and widely circulated exam-style datasets are subject to performance saturation and training data contamination. Multi-hop reasoning, the ability to integrate information across multiple sources to derive an answer, is central to clinically meaningful tasks such as diagnostic support, literature-based discovery, and hypothesis generation, yet remains underrepresented in current biomedical QA benchmarks.

MedHopQA addresses the limitations of existing benchmarks through the following innovations:

• �� Multi-hop structure: Each question requires synthesizing information from two distinct Wikipedia articles, ensuring reasoning challenge.

• �� Open-ended answer format: Answers are provided in free-text format, eliminating answer cueing effects and requiring models to generate rather than select the correct inferential output.

• �� Community-scale evaluation: By embedding and withholding answers on a CodaBench leaderboard, MedHopQA reduces leaderboard gaming and contamination risk.

• �� Ontology enhancement: Gold annotations are augmented with ontology-grounded synonym sets (MONDO, NCBI Gene, NCBI Taxonomy) to support both lexical and concept-level evaluation.

The MedHopQA dataset was constructed through a multi-stage human-AI collaborative process, with the following steps:

• �� Source Material and Seed Page Selection: The seed dataset was constructed from Wikipedia's curated list of disease pages, each paired with pages reachable via its outgoing hyperlinks.

• �� Human Annotation: Sixteen researchers selected page pairs and formulated questions requiring synthesis of information from both articles.

• �� AI Data Augmentation: An AI generation module was used to produce additional QA pairs, which entered the same triage pool as human-authored ones.

• �� Triage and Verification: All QA pairs entered a shared triage pool and were distributed to reviewers for validation, ensuring question quality.

In a zero-shot setting, MedHopQA evaluated four frontier LLMs (GPT-5.1, Gemini 2.5 Pro, Claude Sonnet 4.5, and GPT-4o). The experimental design included:

• �� Dataset: 1,000 expert-curated question-answer pairs embedded within a publicly downloadable set of 10,000 questions.

• �� Baselines: Comparison with existing biomedical QA benchmarks.

• �� Evaluation Metrics: Accuracy, semantic discrimination ability, and performance variation across answer types.

• �� Hyperparameters: Default model settings, no fine-tuning performed.

• �� Ablation Studies: Analysis of performance variation across different answer types.

Experimental results show:

• �� Overall accuracy ranges from 66.3% to 83.4%, with strongest performance on chemical and anatomical questions.

• �� Performance variation is most pronounced on disease and gene/protein categories, where fine-grained semantic discrimination is required.

• �� In a zero-shot setting, LLMs still face challenges in multi-hop reasoning tasks, indicating the need for further model improvements.

Application scenarios for MedHopQA include:

• �� Diagnostic Support: Assisting physicians in integrating information from different sources to make more accurate diagnostic decisions through multi-hop reasoning.

• �� Literature Discovery: Supporting researchers in discovering new associations and hypotheses in the literature by integrating information from different documents.

• �� Hypothesis Generation: Providing scientists with new hypotheses and research directions by discovering new research paths through multi-hop reasoning.

Limitations of MedHopQA include:

• �� Reliance on Wikipedia as a knowledge source, which may lead to insufficient coverage of certain specific domains or the latest research.

• �� The complexity of question design requires significant human and time investment in dataset construction and validation.

• �� Despite measures to reduce contamination risk, potential similar instances in training data cannot be completely ruled out. Future work directions include expanding MedHopQA to cover more biomedical domains and knowledge sources, further increasing the diversity and challenge of the questions.