SemEval-2021 Task 9: Fact Verification and Evidence Finding for Tabular Data in Scientific Documents (SEM-TAB-FACTS)
SEM-TAB-FACTS introduces a new dataset and tasks for fact verification in scientific tables, featuring 180K+ statements and 16M+ evidence annotations.
Key Findings
Methodology
SEM-TAB-FACTS includes two subtasks: Subtask A verifies if a statement is supported, refuted, or unknown by a table; Subtask B identifies specific table cells providing evidence. The dataset contains 981 human-generated tables and 1,980 auto-generated tables, totaling 180K+ statements and 16M+ evidence annotations.
Key Results
- Result 1: In Subtask A, the King001 team achieved the highest F1 score of 84.48% for three-way classification, outperforming the baseline of 48.24%.
- Result 2: In Subtask B, the best F1 score was 65.17% by BreakingBERT@IITK, highlighting the challenge of cell-level evidence selection.
- Result 3: Experiments showed that models like TAPAS and Table-BERT excel in table understanding tasks, especially with ensemble strategies.
Significance
This research addresses a critical gap in table fact verification by introducing the first dataset focused on scientific tables. It provides a benchmark for future research, advancing table understanding and natural language inference.
Technical Contribution
The study introduces the first dataset for scientific table fact verification with cell-level evidence annotations. Compared to TabFact and INFOTABS, SEM-TAB-FACTS adds complexity with hierarchical headers and scientific terminology.
Novelty
SEM-TAB-FACTS is the first dataset to focus on scientific table fact verification, introducing cell-level evidence annotations and addressing the unique challenges of complex scientific tables.
Limitations
- Limitation 1: Auto-generated statements may introduce biases, affecting model generalization.
- Limitation 2: Complex scientific tables with hierarchical headers pose challenges for existing models.
- Limitation 3: Dataset domain coverage is limited, potentially restricting cross-domain applicability.
Future Work
Future work could expand domain coverage, improve model understanding of complex table structures, and explore more efficient evidence selection algorithms.
AI Executive Summary
Tables are critical in scientific documents for summarizing data, but their complex structures make automated understanding and fact verification challenging. Existing NLP tasks like SQuAD and SemEval-2019 focus on text, neglecting the unique challenges of tables.
SEM-TAB-FACTS addresses this gap by introducing a new dataset and tasks for fact verification in scientific tables. The dataset includes 981 human-generated and 1,980 auto-generated tables, with over 180K statements and 16M evidence annotations. Two subtasks are defined: verifying whether a statement is supported, refuted, or unknown by a table, and identifying specific table cells that provide evidence.
Experiments reveal that ensemble models based on TAPAS and Table-BERT perform well, with the best F1 score of 84.48% in Subtask A. However, challenges remain in handling complex table structures and reducing biases in auto-generated data. Future research could focus on expanding domain coverage and improving model robustness for table understanding tasks.
Deep Analysis
Background
Tables are a common format for presenting data in scientific documents. However, their structured nature and hierarchical headers make automated understanding challenging. Existing NLP tasks like SQuAD and SemEval-2019 focus on text, leaving table-specific challenges unaddressed. Recent models like TAPAS and Table-BERT have shown promise but are limited to simpler table structures.
Core Problem
The core problem is automating fact verification for scientific tables, including determining if a statement is supported, refuted, or unknown, and identifying specific cells as evidence. This is crucial for combating misinformation and enhancing scientific communication but is challenging due to the complexity of scientific tables and their specialized vocabulary.
Innovation
Key innovations include:
- �� The first dataset for scientific table fact verification, with cell-level evidence annotations.
- �� Two subtasks: statement verification and evidence selection.
- �� Coverage of scientific tables with hierarchical headers and complex data structures, unlike prior datasets like TabFact.
Methodology
The methodology includes:
- �� Dataset creation: Extracting tables from scientific articles, combining human and auto-generated statements.
- �� Subtask A: Using models like Table-BERT and TAPAS to classify statements as supported, refuted, or unknown.
- �� Subtask B: Employing ensemble methods to identify table cells providing evidence for statements.
Experiments
Experiments used 981 human-generated and 1,980 auto-generated tables. Baseline models included Table-BERT and TAPAS. Metrics focused on F1 scores, with Subtask A achieving a best score of 84.48% and Subtask B reaching 65.17%.
Results
Key results include:
- �� Ensemble strategies significantly improve accuracy in statement verification and evidence selection.
- �� King001 achieved the highest F1 score in Subtask A (84.48%).
- �� Subtask B results highlight the difficulty of cell-level evidence selection.
Applications
Applications include:
- �� Automated fact-checking for scientific documents, aiding researchers in verifying data.
- �� Government report auditing to ensure data accuracy and reduce misinformation.
Limitations & Outlook
Limitations include:
- �� Biases in auto-generated statements may hinder model generalization.
- �� Complex scientific tables challenge current models.
- �� Limited domain coverage may restrict cross-domain applicability.
Plain Language Accessible to non-experts
Imagine you're in a kitchen with a recipe (the statement) and a pantry full of ingredients (the table). Your job is to check if the recipe can be made with the ingredients available. SEM-TAB-FACTS is like a smart assistant that not only tells you if the recipe is possible but also points out the exact ingredients you need. This makes verifying scientific data much faster and more reliable.
ELI14 Explained like you're 14
Think of this like a game where you have a big table full of numbers and facts, and someone gives you a sentence like, 'Is there more chocolate than vanilla ice cream?' You have to look at the table and decide if it's true, false, or if you can't tell. This research teaches computers to play that game really well!
Glossary
Natural Language Inference (NLI)
Determines if one statement logically follows from another. Often used for fact verification.
Used to verify if a statement is supported or refuted by a table.
TAPAS
A pre-trained model combining table and text data for reasoning tasks.
Used as a baseline for both subtasks.
Table-BERT
A model that converts tabular data into natural language for classification tasks.
Key model for statement verification.
Cell-level Evidence
Specific table cells that support or refute a statement.
Core focus of Subtask B.
Model Ensembling
Combining predictions from multiple models to improve accuracy.
Widely used by top-performing teams in Subtask A.
Open Questions Unanswered questions from this research
- 1 How can models better handle complex table structures with hierarchical headers?
- 2 How can biases in auto-generated statements be minimized to improve dataset quality?
Applications
Immediate Applications
Scientific Document Verification
Helps researchers quickly verify data in scientific literature, reducing errors.
Government Report Auditing
Automates fact-checking in government reports to ensure data accuracy.
Long-term Vision
Cross-domain Table Understanding
Expands to fields like healthcare and finance, enabling universal table understanding.
Abstract
Understanding tables is an important and relevant task that involves understanding table structure as well as being able to compare and contrast information within cells. In this paper, we address this challenge by presenting a new dataset and tasks that addresses this goal in a shared task in SemEval 2020 Task 9: Fact Verification and Evidence Finding for Tabular Data in Scientific Documents (SEM-TAB-FACTS). Our dataset contains 981 manually-generated tables and an auto-generated dataset of 1980 tables providing over 180K statement and over 16M evidence annotations. SEM-TAB-FACTS featured two sub-tasks. In sub-task A, the goal was to determine if a statement is supported, refuted or unknown in relation to a table. In sub-task B, the focus was on identifying the specific cells of a table that provide evidence for the statement. 69 teams signed up to participate in the task with 19 successful submissions to subtask A and 12 successful submissions to subtask B. We present our results and main findings from the competition.