Are Large Language Models Table-based Fact-Checkers?
Study shows LLMs can perform table-based fact verification with prompt engineering and instruction tuning.
Key Findings
Methodology
The study evaluates LLMs like ChatGPT and LLaMA for table-based fact verification (TFV) using zero-shot and few-shot learning with prompt engineering and instruction tuning. TabFact dataset is used for experiments.
Key Results
- Result 1: ChatGPT achieves 73.6% accuracy in zero-shot TFV, significantly outperforming random guessing (50%).
- Result 2: In few-shot learning, ChatGPT achieves 75.7% accuracy (2-shot), surpassing Codex (72.6%).
- Result 3: Instruction-tuned LLaMA-2 achieves 82.3% accuracy, close to BART (82.5%).
Significance
This is the first systematic evaluation of LLMs as standalone table-based fact-checkers. Results highlight their potential and limitations, paving the way for future cross-domain table reasoning models.
Technical Contribution
Introduced novel prompt engineering strategies and instruction tuning methods to enhance LLaMA's table reasoning capabilities, providing a new perspective for applying LLMs to structured data tasks.
Novelty
First study to directly use LLMs for TFV, contrasting prior works that used LLMs as auxiliary components.
Limitations
- Limitation 1: LLaMA models perform poorly in zero-/few-shot settings due to limited reasoning capabilities.
- Limitation 2: Instruction-tuned models still lag behind specialized small-scale models.
- Limitation 3: Models struggle with complex reasoning tasks like counting, leading to hallucinations.
Future Work
Future research could explore table decomposition strategies, chain-of-thought reasoning, and fine-tuning LLMs specifically for table-based tasks.
AI Executive Summary
Table-based fact verification (TFV) determines whether a statement aligns with structured table data. Traditional methods rely on small-scale models, limited by data scarcity and weak generalization. Large language models (LLMs) like ChatGPT and LLaMA exhibit strong zero- and few-shot learning capabilities, but their potential for TFV remains unexplored.
This study systematically evaluates LLMs for TFV using prompt engineering and instruction tuning. Experiments on the TabFact dataset show that ChatGPT achieves strong performance in zero- and few-shot settings, while instruction-tuned LLaMA models significantly improve their accuracy, nearing specialized models like BART.
However, challenges remain, particularly in handling complex reasoning tasks. Future directions include developing better table decomposition strategies, leveraging chain-of-thought reasoning, and creating cross-domain table reasoning datasets to enhance LLMs' capabilities in TFV tasks.
Deep Analysis
Background
TFV has gained attention due to applications like fake news detection. Traditional methods include program-based and pretrain-based approaches, but both require large labeled datasets and struggle with complex reasoning.
Core Problem
Existing methods fail to generalize in data-scarce or cross-domain scenarios. The challenge lies in leveraging LLMs' zero- and few-shot learning capabilities to address these limitations.
Innovation
This study pioneers the direct application of LLMs for TFV. It introduces novel prompt designs and instruction tuning strategies, significantly improving LLaMA's performance.
Methodology
- �� Designed various zero-shot prompts, including Sentence and Dialogue formats.
- �� Evaluated models on TabFact dataset under zero- and few-shot settings.
- �� Fine-tuned LLaMA using LoRA for efficient instruction tuning.
Experiments
Experiments used TabFact dataset with ChatGPT and LLaMA models. Zero-/few-shot setups were tested, and instruction tuning was applied to LLaMA. Accuracy was the primary metric.
Results
ChatGPT achieved 73.6% accuracy in zero-shot TFV. Instruction-tuned LLaMA-2 reached 82.3%, comparable to BART.
Applications
The findings can be applied to fake news detection and data validation, especially in data-scarce scenarios.
Limitations & Outlook
LLaMA models show limited reasoning in zero-/few-shot setups and struggle with complex tasks. Instruction tuning improves performance but doesn't surpass specialized models.
Plain Language Accessible to non-experts
Imagine you have a table of student grades, and someone claims, 'Only John scored full marks in math.' You need to check the table to verify this claim. LLMs act like a super-smart assistant that can analyze the table and give you an answer. By giving it clear instructions or examples, it gets better at the task. This study explores how to make these assistants smarter and more reliable for such tasks.
ELI14 Explained like you're 14
Imagine you have a table of your friends' test scores. Someone says, 'Only Alex got an A in math.' You need to check if that's true. LLMs are like a super-smart friend who can look at the table and tell you the answer. If you show them a few examples first, they get even better! But sometimes, they mess up, especially with tricky questions like counting. Scientists are working to fix that!
Glossary
TabFact
A dataset with 16,000 tables and 118,000 statements for evaluating table-based fact verification.
Used to test models' table reasoning abilities.
Prompt Engineering
Designing input prompts to guide LLMs for specific tasks.
Used in zero-/few-shot experiments to optimize performance.
Instruction Tuning
Fine-tuning LLMs with task-specific instructions to improve performance.
Applied to enhance LLaMA's TFV capabilities.
Zero-shot Learning
The ability to perform tasks without any task-specific training data.
Evaluated to test LLMs' inherent reasoning capabilities.
LoRA
A parameter-efficient fine-tuning method that freezes most model weights.
Used for instruction tuning LLaMA models.
Open Questions Unanswered questions from this research
- 1 How can LLMs handle complex table reasoning tasks like counting?
- 2 What are the best strategies for decomposing large tables for reasoning?
- 3 How can cross-domain datasets improve LLMs' generalization in TFV?
Applications
Immediate Applications
Fake News Detection
Quickly verify data in news articles to reduce misinformation.
Financial Report Analysis
Assist in analyzing financial data and verifying key claims in reports.
Long-term Vision
General Table Reasoning Assistant
Develop intelligent assistants for cross-domain table reasoning in research and business.
Abstract
Table-based Fact Verification (TFV) aims to extract the entailment relation between statements and structured tables. Existing TFV methods based on small-scaled models suffer from insufficient labeled data and weak zero-shot ability. Recently, the appearance of Large Language Models (LLMs) has gained lots of attraction in research fields. They have shown powerful zero-shot and in-context learning abilities on several NLP tasks, but their potential on TFV is still unknown. In this work, we implement a preliminary study about whether LLMs are table-based fact-checkers. In detail, we design diverse prompts to explore how the in-context learning can help LLMs in TFV, i.e., zero-shot and few-shot TFV capability. Besides, we carefully design and construct TFV instructions to study the performance gain brought by the instruction tuning of LLMs. Experimental results demonstrate that LLMs can achieve acceptable results on zero-shot and few-shot TFV with prompt engineering, while instruction-tuning can stimulate the TFV capability significantly. We also make some valuable findings about the format of zero-shot prompts and the number of in-context examples. Finally, we analyze some possible directions to promote the accuracy of TFV via LLMs, which is beneficial to further research of table reasoning.