Neural Rankers for Effective Screening Prioritisation in Medical Systematic Review Literature Search
Pre-trained BERT-based neural rankers outperform traditional methods in medical systematic review screening prioritization, with significant improvements in key metrics.
Key Findings
Methodology
This study employs monoBERT cross-encoder architecture, combining zero-shot and fine-tuning strategies, leveraging title and abstract representations. Fine-tuning involves contrastive loss with relevant and non-relevant document triples, using models like BioBERT and PubMedBERT. Experiments on CLEF TAR datasets compare neural models with BM25 and QLM baselines, evaluating metrics such as AP, Recall@p%, and WSS. Results demonstrate that fine-tuned neural rankers significantly outperform traditional methods across all metrics, especially in high-recall scenarios, indicating strong potential for practical deployment.
Key Results
- Fine-tuned BioBERT achieves highest performance, with Recall@1% reaching 0.1801, AP at 0.2922, and WSS100 surpassing baseline by ~20%. Zero-shot models underperform, highlighting the importance of fine-tuning. Multi-modal representations (title+abstract) outperform title-only. Convergence analysis shows models stabilize after 100 epochs, suggesting early stopping can optimize training. These findings confirm neural models' effectiveness in prioritizing relevant literature for systematic reviews.
Significance
This work advances the application of deep pre-trained language models in medical literature screening, addressing the bottleneck of manual, labor-intensive review processes. By significantly improving ranking accuracy and reducing screening workload, it accelerates evidence synthesis, crucial for timely clinical decision-making. The approach offers scalable, adaptable solutions for large-scale literature management, with potential integration into automated review pipelines, transforming how medical evidence is curated and utilized in research and practice.
Technical Contribution
The paper introduces a novel application of multi-modal text representations (title+abstract) combined with contrastive fine-tuning of pre-trained models like BioBERT for ranking tasks. It demonstrates that fine-tuning on limited labeled data yields substantial performance gains over zero-shot approaches. The study also provides comprehensive empirical comparisons, establishing a new state-of-the-art in systematic review screening prioritization, and suggests model fusion strategies for further enhancement. These innovations push the boundaries of neural IR in specialized domains.
Novelty
This is the first systematic study applying pre-trained language models specifically to the task of ranking documents in medical systematic reviews. Unlike prior work limited to general IR tasks, this research tailors models like BioBERT and PubMedBERT with domain-specific fine-tuning, addressing the unique challenges of long, technical medical texts. The combination of multi-modal representations and contrastive learning constitutes a novel methodological contribution, setting a new benchmark for future research.
Limitations
- Model performance depends on the quality and quantity of labeled training data; limited samples may restrict generalization. Long medical texts risk truncation, affecting relevance scoring. High computational costs for training and inference pose deployment challenges. Further work is needed to optimize model efficiency and robustness across diverse medical domains.
Future Work
Future directions include integrating multi-modal data (images, tables), employing active learning to iteratively improve ranking, and exploring domain adaptation for broader medical fields. Developing lightweight models for real-time deployment and extending to multilingual datasets will enhance practical utility. Combining neural rankers with relevance feedback mechanisms could further boost performance, enabling fully automated, scalable literature screening pipelines.
AI Executive Summary
Medical systematic reviews are vital for synthesizing evidence but face the challenge of screening vast amounts of literature efficiently. Traditional keyword-based methods are limited in understanding complex medical texts, leading to high costs and delays. Recent advances in deep learning, particularly pre-trained language models like BERT, offer promising solutions. This study introduces neural rankers based on BioBERT and PubMedBERT, leveraging multi-modal text representations and contrastive fine-tuning to prioritize relevant documents effectively.
Experiments on the CLEF TAR datasets demonstrate that fine-tuned neural models significantly outperform traditional baselines such as BM25 and QLM across multiple metrics, including AP, Recall@p%, and WSS. The results show that models like BioBERT can achieve Recall@1% of 0.1801 and AP of 0.2922, reducing the screening workload by approximately 20%. These improvements are particularly notable in high-recall scenarios, crucial for comprehensive systematic reviews.
The implications of this work are substantial. Automating and optimizing literature screening accelerates evidence synthesis, enabling faster clinical guideline development and research updates. The approach also reduces human labor, making large-scale reviews more feasible within limited budgets. The study highlights the importance of domain-specific fine-tuning and multi-modal representations, setting a new standard for neural IR in medical applications.
Despite promising results, challenges remain. Model performance depends on labeled data quality, and computational costs are high. Long medical texts may be truncated, affecting accuracy. Future work will focus on integrating additional data modalities, active learning, and model compression to facilitate real-world deployment. Overall, this research paves the way for smarter, faster, and more reliable systematic review processes, with broad impacts on healthcare and biomedical research.
Deep Analysis
Background
随着医学文献的爆炸式增长，系统综述成为整合临床证据的核心工具。传统方法依赖关键词匹配和统计模型（如BM25、QLM），在理解长文本和捕捉语义关系方面存在局限。近年来，深度学习和预训练模型（如BERT、BioBERT）在信息检索中表现出色，推动了自动筛查技术的发展。尽管如此，将这些模型应用于医学系统综述的筛选排序仍处于探索阶段，缺乏系统性研究，亟需结合医学文本特性进行优化。
Core Problem
核心问题在于如何高效、准确地对海量医学文献进行排序，优先筛查最相关的文献。传统方法在语义理解和长文本处理上表现不足，难以满足高召回和效率的双重需求。现有深度模型多用于一般信息检索，缺乏针对医学文本的优化策略，限制了其实际应用效果。如何在保证高准确率的同时，降低计算成本，是亟待解决的难题。
Innovation
本研究的创新点包括：1) 结合多模态文本（标题+摘要）提升模型理解能力，解决长文本信息丢失问题；2) 利用对比学习机制进行微调，增强模型对医学文献的判别能力；3) 在多个指标上优于传统检索算法，验证神经模型在医学筛查中的优越性。创新在于将预训练模型系统性引入医学筛查场景，突破了以往仅在通用场景中的应用限制，为医学信息检索提供了新思路。
Methodology
- �� 构建monoBERT跨编码器架构，将查询和候选文献拼接输入模型，输出相关性分数。• 采用零-shot策略，直接利用预训练模型进行排序；同时，利用有限标注数据进行微调，优化模型参数。• 设计对比损失，利用相关与非相关文档对进行训练，增强模型判别能力。• 采用标题和摘要两种表示方式，比较其对排序效果的影响。• 在CLEF TAR数据集上进行多轮实验，评估指标包括AP、Recall@p%、WSS，验证模型性能。• 结合不同预训练模型（BioBERT、PubMedBERT）进行性能对比，分析微调效果。
Experiments
实验采用CLEF TAR 2017-2019数据集，比较BM25、QLM、零-shot和微调模型的性能。指标包括Last_Rel、AP、Recall@p%、WSS，反映排序准确性和工作量节省。通过交叉验证调优超参数（学习率、批次大小），确保结果稳健。模型收敛分析显示，训练100轮后性能稳定，早停策略可优化训练效率。多模态表示（标题+摘要）效果优于标题单一表示，验证模型在实际筛查中的应用潜力。
Results
微调BioBERT在所有指标上显著优于传统方法，Recall@1%达0.1801，AP达0.2922，WSS100超越基线约20%。零-shot模型表现有限，未能超越BM25。多模态表示（标题+摘要）效果更佳。模型在训练100轮后趋于收敛，早停策略可提升效率。整体验证神经模型在医学筛查中的应用潜力巨大，特别是在高效筛查和节省人力方面表现优异。
Applications
该模型可应用于医学系统综述的文献筛查环节，帮助研究人员快速识别高相关性文献。结合自动化筛查平台，实现大规模文献的智能排序，降低人工成本。未来可集成到临床决策支持系统中，加快证据整合，提升临床实践效率。
Limitations & Outlook
模型对长文本（如完整摘要）存在信息截断风险，影响排序效果。微调依赖有限标注样本，样本不足时性能下降。训练和推理成本较高，实际部署需考虑硬件资源限制。未来需优化模型压缩和多模态融合策略，以增强实用性。
Plain Language Accessible to non-experts
想象你在一个大厨房里准备一道复杂的菜肴。每次你需要从成千上万的食材中挑选出最合适的几样，才能做出美味佳肴。传统方法就像用肉眼挑选，效率低且容易错过重要食材。现在，科学家们开发了一种智能助手，它像个超级厨师，能快速理解每个食材的特点，并帮你排序出最重要的几样。这个助手经过大量厨房经验的训练（预训练模型），还能根据你提供的菜谱（筛选条件），自动筛选出最符合要求的食材。通过不断学习和调整，它变得越来越聪明，帮你节省时间，也让菜肴更美味。这就像用AI帮忙做饭，让复杂的厨房工作变得简单高效。
ELI14 Explained like you're 14
想象你在学校的图书馆里找书。很多书都和你的作业有关，但你没有时间看全部。于是，你的朋友帮你用一种特别聪明的方法，把书按重要性排好。这个方法就像一个超级聪明的机器人，它学会了如何判断哪本书最值得先看。刚开始，它只是用以前学到的知识快速判断（零-shot），但效果不够好。后来，它还专门学习了你的偏好（微调），变得更聪明了。它会看书的标题和摘要，然后告诉你哪些书最重要。结果，你可以只看前几本书，就能完成作业，节省了很多时间。这就像用AI帮你筛选资料，让学习变得更轻松、更快！
Abstract
Medical systematic reviews typically require assessing all the documents retrieved by a search. The reason is two-fold: the task aims for ``total recall''; and documents retrieved using Boolean search are an unordered set, and thus it is unclear how an assessor could examine only a subset. Screening prioritisation is the process of ranking the (unordered) set of retrieved documents, allowing assessors to begin the downstream processes of the systematic review creation earlier, leading to earlier completion of the review, or even avoiding screening documents ranked least relevant. Screening prioritisation requires highly effective ranking methods. Pre-trained language models are state-of-the-art on many IR tasks but have yet to be applied to systematic review screening prioritisation. In this paper, we apply several pre-trained language models to the systematic review document ranking task, both directly and fine-tuned. An empirical analysis compares how effective neural methods compare to traditional methods for this task. We also investigate different types of document representations for neural methods and their impact on ranking performance. Our results show that BERT-based rankers outperform the current state-of-the-art screening prioritisation methods. However, BERT rankers and existing methods can actually be complementary, and thus, further improvements may be achieved if used in conjunction.