FEVER: a large-scale dataset for Fact Extraction and VERification

The rapid growth of online information has heightened the need for automated fact verification, crucial for combating misinformation. Early datasets like Vlachos and Riedel (2014) provided small-scale resources, but lacked diversity and multi-page support. The Fake News Challenge (Pomerleau and Rao, 2017) expanded scope but still faced limitations in evidence complexity. Advances in natural language inference (Dagan et al., 2009) and question answering (Chen et al., 2017) laid groundwork, yet struggled with multi-hop, multi-source scenarios. The emergence of deep pre-trained models (e.g., BERT) offers new opportunities, but data scarcity remains a bottleneck. This context motivated the creation of FEVER, aiming to bridge the gap between research and real-world needs.

The core challenge lies in developing systems capable of retrieving, integrating, and reasoning over evidence from multiple documents and sentences to verify claims. Existing methods rely heavily on shallow matching, which fails in complex, multi-sentence, multi-page contexts. The bottleneck is evidence selection accuracy, especially when claims involve paraphrasing, negation, or multi-hop inference. Additionally, the scarcity of large, annotated datasets hampers deep learning approaches, limiting their ability to generalize to diverse real-world scenarios. Addressing these issues requires a comprehensive dataset and robust pipeline architecture capable of handling complex evidence structures.

This work introduces several innovations: 1) a large-scale, multi-category dataset with 185,445 claims and multi-page evidence annotations, surpassing prior resources; 2) a multi-component pipeline integrating TF-IDF-based document retrieval, similarity-driven sentence selection, and deep neural models for textual entailment; 3) oracle-based validation experiments that reveal evidence retrieval as the primary bottleneck, guiding future improvements. The dataset’s diversity, including mutated claims and multi-source evidence, enables training models for complex reasoning tasks. The pipeline’s modular design facilitates targeted enhancements, fostering progress in multi-hop, multi-document verification.

• �� Data collection: Extract sentences from Wikipedia June 2017 dump, generate claims via human annotation and mutation, focusing on entities and facts. • Evidence annotation: Annotators select sentences supporting or refuting claims, often across multiple pages, ensuring high inter-annotator agreement. • System architecture: • Document retrieval using TF-IDF and DrQA to fetch top-k documents. • Sentence selection: Rank sentences by similarity to claim, tune cutoff thresholds. • Textual entailment: Use models like Decomposable Attention and MLP to classify claim-evidence pairs. • Oracle experiments: Replace system components with gold annotations to measure maximum achievable accuracy. • Evaluation: Measure overall accuracy, evidence F1, and analyze bottlenecks across different scenarios.

The dataset was split into training, validation, and test sets, ensuring each Wikipedia page appears in only one set. Baseline models used TF-IDF for document retrieval, sentence ranking, and Decomposable Attention for entailment. Hyperparameters like number of documents (k=5) and sentences (l=5) were optimized via grid search. Oracle experiments demonstrated potential upper bounds, with document retrieval reaching 86.59% accuracy. The full pipeline achieved 31.87% accuracy with evidence, and 50.91% ignoring evidence. Ablation studies showed that removing sentence selection significantly degraded performance, emphasizing its importance. The models struggled with multi-hop and multi-page evidence, indicating areas for future improvement.

The best system achieved 31.87% accuracy when evidence was required, highlighting the task's difficulty. Oracle experiments indicated that evidence retrieval could reach 86.59%, but the overall accuracy was limited by evidence matching and reasoning. Incorporating deep models like BERT is expected to improve performance. The experiments confirmed that evidence selection and multi-hop reasoning are primary bottlenecks, especially in complex scenarios involving multiple documents and sentence mutations. The results suggest that future work should focus on improving evidence retrieval robustness and reasoning depth to handle real-world complexities.

The FEVER system can be applied to automated fact-checking in journalism, scientific research validation, and legal evidence analysis. It enables scalable, real-time verification of claims against large knowledge bases, reducing misinformation spread. Future integration with knowledge graphs and multi-modal data can further enhance its utility, making AI systems more trustworthy and transparent in critical domains.

Current models are limited by shallow retrieval techniques, insufficient for deep semantic understanding required in multi-hop scenarios. Manual annotation quality, while high, is costly and not scalable. The system struggles with ambiguous, vague, or paraphrased claims, especially across multiple pages. Computational costs of large models and multi-source fusion remain high. Future work must address these challenges by developing more sophisticated retrieval, reasoning, and annotation methods.