SciFact-Open: Towards open-domain scientific claim verification
Introduced SciFact-Open, leveraging model fusion and retrieval pooling on 500K abstracts, with F1 dropping over 15 points compared to closed-domain benchmarks.
Key Findings
Methodology
This work constructs the SciFact-Open dataset using a model fusion pooling strategy, combining predictions from four state-of-the-art scientific claim verification models. Abstracts retrieved via BM25 and neural re-rankers are scored for confidence, and top candidates are pooled for human annotation. This approach reduces annotation burden while maximizing evidence coverage. The dataset includes 279 claims verified against 500,000 abstracts, with annotations capturing support, refutation, and conflicting evidence. Evaluation metrics include F1, precision, and recall, revealing significant performance drops of at least 15 points compared to in-domain results, highlighting generalization challenges in open environments.
Key Results
- Models trained on smaller corpora exhibit at least a 15-point F1 performance decline when tested on SciFact-Open, indicating poor generalization to large, diverse datasets.
- Evidence analysis shows frequent phenomena such as evidence supporting only specific sub-cases of claims, and the presence of conflicting evidence, which complicates verification.
- Pooling multiple models' top predictions increases evidence recall but still leaves gaps, especially in handling evidence specificity mismatches and contradictions, emphasizing the need for models to better understand evidence hierarchies.
Significance
This study advances scientific claim verification by providing a large-scale, realistic benchmark that mimics real-world scenarios. It exposes the limitations of current models in open-domain settings, urging the community to develop more robust, generalizable methods. The dataset facilitates research into evidence diversity, conflicts, and specificity, essential for trustworthy scientific information extraction, which impacts scientific communication, knowledge management, and AI-assisted research workflows.
Technical Contribution
Key innovations include the integration of multi-model fusion pooling for evidence collection, large-scale dataset construction from 500K abstracts, and detailed analysis of evidence phenomena such as specificity mismatches and conflicts. The approach combines traditional IR techniques with deep learning classifiers, enabling scalable evidence retrieval and verification. The study also systematically evaluates model performance across different evidence complexities, providing insights into model robustness and failure modes in open-domain contexts.
Novelty
This is the first large-scale open-domain scientific claim verification dataset based on 500K abstracts, employing a multi-model fusion pooling strategy to simulate real-world evidence retrieval. It systematically analyzes evidence phenomena like specificity mismatches and conflicting evidence, which have been underexplored in prior work. The combination of large-scale data, evidence diversity analysis, and model evaluation under realistic conditions marks a significant step forward.
Limitations
- Despite pooling strategies, evidence diversity and conflicts remain challenging, with models struggling to correctly interpret evidence specificity and resolve contradictions, especially in complex claims.
- The annotation process relies on expert judgment, which introduces subjectivity and limits scalability. Expanding the dataset further requires substantial manual effort.
- Model performance still degrades significantly in highly ambiguous or nuanced cases, indicating the need for integrating reasoning and external knowledge sources to improve robustness.
Future Work
Future directions include integrating knowledge graphs and reasoning modules to improve evidence understanding, developing semi-automatic annotation techniques to scale data collection, and designing models that better handle evidence conflicts and hierarchical evidence structures. Additionally, exploring domain adaptation and continual learning could enhance models’ robustness across diverse scientific fields.
AI Executive Summary
Scientific claim verification is a vital task for ensuring the reliability of scientific information, yet current models often falter when faced with the vast and diverse landscape of real-world research literature. Traditional datasets and methods, while effective in controlled environments, lack the scale and complexity needed to simulate practical scenarios. To address this gap, this work introduces SciFact-Open, a large-scale dataset comprising 500,000 scientific abstracts, designed to evaluate the robustness and generalization of claim verification systems in open-domain settings.
The core challenge lies in retrieving relevant evidence from an enormous corpus and accurately classifying claims as supported, refuted, or unsupported. To this end, the authors adopt a pooling strategy inspired by information retrieval competitions, combining predictions from four state-of-the-art models—VERT5ERINI, PARAGRAPHJOINT, MULTIVERS, and MULTIVERS10. They select the top d predictions from each model based on confidence scores, then have experts annotate these candidates for evidence. This approach balances coverage and annotation efficiency, capturing diverse evidence phenomena.
Experimental results reveal a stark performance gap: models trained on smaller, closed datasets experience at least a 15-point F1 drop when evaluated on SciFact-Open, exposing significant generalization issues. Further analysis uncovers phenomena such as evidence supporting only specific subtypes of claims, and the frequent occurrence of conflicting evidence—both complicating verification. The dataset’s detailed annotations enable systematic study of these phenomena, guiding future model improvements.
This research offers a crucial step toward deploying scientific claim verification in real-world applications, such as automated literature review, scientific knowledge bases, and fact-checking tools. It emphasizes the importance of developing models capable of understanding evidence hierarchies, handling conflicts, and generalizing across diverse scientific domains. Despite its advances, challenges remain in scaling annotations, resolving evidence conflicts, and integrating external knowledge. Continued efforts in these directions will be essential for trustworthy AI-driven scientific reasoning.
Deep Analysis
Background
科学验证作为自然语言处理中的重要任务，经历了从封闭域到开域的逐步演变。早期工作如Wadden等（2020）提出了SCIFACT数据集，主要在有限的科研文献范围内验证声明。随着大规模科研文献资源如S2ORC、PubMed Central的出现，研究逐步转向多样化和复杂化，但多依赖人工标注和小规模实验。近年来，深度学习模型如BERT、RoBERTa在文本理解中表现出色，但在科学验证中的泛化能力仍有限。现有方法多采用检索-分类架构，结合信息检索技术（如BM25）和Transformer模型实现声明与证据匹配。然而，面对大规模开域语料，模型面临证据稀疏、冲突和特异性偏差等新挑战，亟需新的数据集和评估策略推动技术突破。
Core Problem
在大规模科研文献中自动验证声明的难点在于证据的多样性、复杂性和冲突性。传统封闭域模型在有限语料中表现良好，但在开域环境中性能大幅下降，原因在于模型难以泛化到未见过的文献分布。此外，证据的特异性和层级关系模糊，导致模型难以正确匹配声明与证据，冲突证据的存在也增加了验证难度。这些问题限制了科学知识自动化验证的实际应用，影响科研信息的快速传播与准确性保障。
Innovation
本研究的创新主要体现在：1）提出基于多模型融合的池化策略，有效扩展证据采集范围，提升召回率；2）构建50万规模的开域科研摘要数据集，突破封闭域限制，模拟真实科研场景；3）引入证据特异性和冲突分析，揭示模型在大规模环境中的表现差异，为未来模型设计提供理论依据；4）系统评估模型在复杂证据环境下的鲁棒性，提出改进方向。此策略结合信息检索和深度学习，显著提升了开域验证的可行性和效果。
Methodology
- �� 采用BM25和神经重排序器筛选候选摘要，确保检索的相关性和多样性；• 利用四个最先进模型（如VERT5ERINI、PARAGRAPHJOINT、MULTIVERS、MULTIVERS10）对每个声明进行预测，获得支持、反驳或无信息标签；• 计算模型置信度，排序候选摘要，选取前d个作为标注池；• 人工专家对池中的CAP进行证据确认和标签标注，确保标注质量；• 结合多模型预测结果，减少遗漏，提高证据覆盖率；• 评估指标包括F1、精确率、召回率，分析模型在不同场景下的表现差异。
Experiments
实验基于构建的50万摘要数据集，采用四个主流模型进行证据预测，设置不同负采样比率，评估模型在开域验证任务中的性能。通过比较不同模型的F1、平均精度，分析模型泛化能力。还进行多模型融合效果的对比，验证池化策略的有效性。采用人工标注的验证集，确保评估的可靠性。实验还包括证据的特异性和冲突分析，揭示模型在复杂证据环境中的表现差异。
Results
模型在50万摘要中验证声明的F1平均下降超过15个百分点，显著低于封闭域表现。多模型融合提升了证据召回率，但仍存在证据不匹配和冲突问题。高频、研究热点领域的声明表现优于边缘领域，证据的特异性和冲突现象频繁出现。实验还发现，负采样比率对模型泛化影响显著，合理调节可改善模型性能。整体结果表明，开域验证任务复杂度高，模型需进一步增强推理和知识融合能力。
Applications
本研究提供了面向科研自动验证的基础数据和方法，适用于科研信息管理、学术搜索引擎、科学知识图谱等场景。未来可结合知识图谱和推理模型，实现更精准的声明验证，提升科研信息的可信度和传播效率。该技术还可应用于科研审查、政策制定等领域，推动科学信息的自动化处理和智能化管理。
Limitations & Outlook
当前模型在证据特异性匹配和冲突识别方面仍不足，部分复杂声明难以准确验证。标注过程依赖专家，成本较高，难以快速扩展。大规模语料带来计算负担，模型在极端模糊或多义场景表现仍不理想。未来需引入更强的推理和知识融合机制，提升模型鲁棒性和解释能力。
Plain Language Accessible to non-experts
想象你在一个大型图书馆里查找关于某个科学问题的证据。每本书都可能提供一些信息，但找到真正支持或反驳你问题的内容很难。科学验证就像是用多个不同的搜索工具（模型）帮你筛选出最可能的书，然后由专家逐一确认。这些工具各有偏好，有的擅长找支持证据，有的更善于发现反对证据。通过结合多个工具的结果，你可以更全面地了解问题的真相。这个过程就像是在大海捞针，但借助智能工具，效率大大提升。研究者们希望让电脑也能像专家一样，快速判断科学声明的真假，帮助科研变得更可靠、更高效。
ELI14 Explained like you're 14
想象你在学校的图书馆里找资料，想知道一个科学说法是不是对的。你会用不同的搜索器，比如一个找支持的书，一个找反对的书。每个搜索器都给出一些建议，但有时候它们会给出相反的答案。为了弄清楚真相，你会把这些建议结合起来，看哪些更靠谱。研究人员也是这样，他们用很多智能程序（模型）帮忙找证据，然后专家确认哪些是真的，哪些是错的。这个过程就像是用多个朋友帮你判断，最后得出一个比较靠谱的结论。这样，电脑也能帮忙判断科学说法的真假，让科学变得更快、更准！
Abstract
While research on scientific claim verification has led to the development of powerful systems that appear to approach human performance, these approaches have yet to be tested in a realistic setting against large corpora of scientific literature. Moving to this open-domain evaluation setting, however, poses unique challenges; in particular, it is infeasible to exhaustively annotate all evidence documents. In this work, we present SciFact-Open, a new test collection designed to evaluate the performance of scientific claim verification systems on a corpus of 500K research abstracts. Drawing upon pooling techniques from information retrieval, we collect evidence for scientific claims by pooling and annotating the top predictions of four state-of-the-art scientific claim verification models. We find that systems developed on smaller corpora struggle to generalize to SciFact-Open, exhibiting performance drops of at least 15 F1. In addition, analysis of the evidence in SciFact-Open reveals interesting phenomena likely to appear when claim verification systems are deployed in practice, e.g., cases where the evidence supports only a special case of the claim. Our dataset is available at https://github.com/dwadden/scifact-open.