Benchmarking LLM Agents on Meta-Analysis Articles from Nature Portfolio

The rapid growth of scientific literature across disciplines has created a pressing need for automated tools capable of synthesizing evidence efficiently. Traditional manual meta-analyses, although considered the gold standard, are labor-intensive and time-consuming, often taking months to complete. Early efforts in automating systematic reviews focused on text mining and rule-based systems, such as the development of tools like RobotReviewer and Abstrackr, which aimed to assist in screening and data extraction. With the advent of deep learning, models like BERT and SciBERT have been employed to improve relevance ranking and information extraction. However, these approaches typically target isolated tasks and lack the capacity to handle the full pipeline with verifiable ground truth. Recent advances in retrieval-augmented generation (RAG) models have shown promise in generating summaries and answering scientific questions, but their application in the context of strict eligibility criteria remains limited. The need for a comprehensive benchmark that captures the entire meta-analysis workflow, including the critical screening step governed by PI/ECO criteria, has been recognized as a major gap in the field. MetaSyn addresses this gap by providing a structured, verifiable dataset that enables systematic evaluation of AI models across all stages, fostering progress toward fully automated evidence synthesis.

Despite significant progress in individual components of scientific literature processing, the core challenge remains in the screening stage: accurately distinguishing studies that meet complex eligibility criteria from those that do not. Existing models excel at retrieval but falter when it comes to applying nuanced, multi-dimensional PI/ECO standards, especially in the presence of topically similar but ineligible studies. This bottleneck impairs the reliability and trustworthiness of automated meta-analyses, limiting their adoption in critical domains like healthcare and environmental policy. Additionally, current evaluation frameworks often rely on relevance labels that do not fully capture the protocol adherence, making it difficult to diagnose specific failure modes. The lack of a unified, verifiable benchmark hampers systematic progress, as models cannot be reliably compared or optimized for the entire pipeline. Addressing this problem requires a dataset with explicit, expert-verified ground truth at each stage and metrics that disentangle retrieval from eligibility reasoning.

MetaSyn introduces several key innovations. First, it constructs a large, expert-curated dataset with 442 meta-analyses, each with detailed search strategies, PI/ECO questions, and verified study lists, providing a verifiable ground truth for the entire pipeline. Second, it develops a multi-stage evaluation framework that separates retrieval, screening, and synthesis, enabling precise diagnostics of model performance at each step. Third, it incorporates stage-specific metrics validated against expert judgments, ensuring that improvements are meaningful and aligned with scientific standards. Fourth, the evaluation includes multiple RAG variants and a protocol-driven agent, highlighting the performance gaps and potential pathways for enhancement. Finally, the dataset's diversity across clinical and non-clinical domains ensures broad applicability, encouraging the development of models capable of handling complex eligibility criteria in real-world scenarios.

• �� Data collection: Extracted 442 meta-analyses from Nature journals, ensuring each included a complete list of analyzed studies, search strategies, and eligibility criteria. Paired each with a PubMed corpus of 140,585 articles, including positive and hard negative samples.
• �� Ground truth extraction: Human experts reviewed supplementary materials, forest plots, and methods sections to confirm study inclusion and extract detailed metadata.
• �� Research question structuring: Used GLM-4.6 to parse abstracts and generate initial PI/ECO components, followed by manual correction.
• �� Retrieval model training: Fine-tuned dense retrievers like DPR and ColBERT on the dataset, optimizing for Recall@K.
• �� Screening model design: Developed multi-stage classifiers based on BERT and T5, trained to discriminate eligible studies according to PI/ECO standards.
• �� Evaluation: Employed stage-specific metrics—Recall, Precision, F1, and expert validation—to assess each pipeline component.
• �� Ablation studies: Tested the impact of different retriever architectures, prompting strategies, and multi-modal inputs.
• �� Error analysis: Analyzed misclassifications with expert input to identify reasoning gaps and guide model improvements.

The experimental setup involved training and testing models on the MetaSyn dataset, with a focus on nine RAG variants and a protocol-driven agent. The models were evaluated across retrieval, screening, and synthesis tasks, with metrics including Recall@K, precision, recall, F1, and expert-validated synthesis accuracy. Hyperparameters such as learning rate, embedding dimensions, and retrieval top-K thresholds were tuned via grid search. Cross-validation ensured robustness, and ablation studies examined the contribution of different model components. Expert annotations validated the model outputs, especially in the screening phase, where eligibility judgments are most challenging. The experiments aimed to quantify the gap between retrieval recall and screening coverage, diagnose failure modes, and establish baseline performance levels for future improvements.

The models achieved high retrieval recall, with MA-Retriever reaching 90.9% at K=200, but the overall inclusion coverage in end-to-end pipelines was capped at 52.7%. Stage-wise analysis revealed that the primary bottleneck was in the screening phase, where models struggled to accurately discriminate PI/ECO-ineligible studies, especially in hard negative cases. The discrepancy between retrieval and screening performance underscores the need for more sophisticated eligibility reasoning modules. Expert validation confirmed that current models lack the nuanced understanding required for complex eligibility criteria, leading to significant false negatives and positives. These results highlight the importance of multi-stage evaluation and targeted model enhancements to bridge the performance gap.

The findings can be immediately applied to develop AI-assisted tools for systematic reviews in medicine, environmental science, and social sciences, enabling faster, more reliable literature screening. Such tools can assist researchers and clinicians by automating the initial search and eligibility assessment, reducing manual workload and increasing reproducibility. In the long term, integrating structured reasoning, multi-modal data, and domain knowledge into models could lead to fully automated meta-analysis pipelines, transforming evidence-based decision-making. These advancements would benefit regulatory agencies, healthcare providers, and policymakers by providing timely, high-quality syntheses of scientific evidence, ultimately accelerating scientific discovery and improving public health outcomes.

Current models are limited by their shallow understanding of complex eligibility criteria, often misclassifying studies that require multi-faceted reasoning. The dataset's reliance on PubMed restricts coverage of other important biomedical databases, affecting generalizability. Computational costs remain high, especially for long texts and multi-modal inputs, hindering scalability. Additionally, the manual annotation process, while rigorous, introduces potential biases and inconsistencies. Future work should focus on enhancing reasoning depth, expanding data sources, improving model efficiency, and developing explainability features to foster broader adoption.