Fact-Check Your Information (FYI): A Design Probe to Understand How People Actually Fact-Check Data-Driven Articles
FYI integrates AI and manual tools to enhance fact-checking efficiency for data-driven articles.
Key Findings
Methodology
FYI is a browser extension embedding a fact-checking workflow directly into the reading environment. It offers four tools: Auto Check, AI Chat, Table Explorer, and Chart Builder. An exploratory study with 22 participants analyzed how users navigate between AI and manual tools to verify data claims.
Key Results
- Result 1: Participants adopted three workflows: AI-first with manual confirmation, manual-first with AI support, and parallel co-review. Visualization was the primary mechanism for auditing AI conclusions.
- Result 2: Trust in AI was dynamic, increasing when tools converged and decreasing when AI outputs were inconsistent.
- Result 3: FYI's multi-modal tools significantly improved verification efficiency while preserving user decision-making authority.
Significance
This study addresses a critical gap in empirical research on data claim fact-checking, revealing user strategies and trust dynamics in mixed-initiative systems. It highlights the importance of treating AI as a complement rather than a definitive authority and emphasizes visualization and flexible workflows to enhance user trust and engagement.
Technical Contribution
FYI integrates automated verification (Auto Check), conversational AI (AI Chat), tabular inspection (Table Explorer), and visual analytics (Chart Builder) into a unified environment. By logging user interactions, it provides new insights into trust calibration and workflow patterns in human-AI collaboration.
Novelty
FYI is the first system to integrate multi-modal tools into a single environment for data claim verification. It uniquely enables empirical observation of user behavior and trust dynamics, distinguishing it from prior fragmented systems.
Limitations
- Limitation 1: The study involved only 22 participants, which may limit the generalizability of the findings.
- Limitation 2: FYI relies on user-uploaded structured datasets, making it unsuitable for unstructured data claims.
- Limitation 3: The AI modules depend on GPT-4.1, which may inherit its limitations.
Future Work
Future research could expand to larger sample sizes to validate findings, improve AI transparency and reliability, and explore support for more complex data claims.
AI Executive Summary
Data-driven journalism and policy reports often include statistical claims grounded in structured datasets. Verifying these claims requires connecting them to the underlying data, a process that existing systems fail to integrate effectively, leaving readers struggling to balance AI assistance with manual inspection.
FYI is a browser extension that embeds a complete fact-checking workflow directly into the reading environment. It provides four tools: Auto Check, AI Chat, Table Explorer, and Chart Builder. In a study with 22 participants, users adopted three distinct workflows and relied heavily on visualization to audit AI conclusions. Trust in AI was dynamic, increasing with consistent outputs and decreasing with inconsistencies.
The findings suggest that future systems should treat AI as a starting point rather than a definitive authority, elevate visualization as a core capability, and support flexible, user-driven workflows. FYI is released as open-source software, providing a platform for further research in data fact-checking.
Deep Analysis
Background
Data-driven articles often rely on statistical claims, but verifying these claims requires connecting them to structured datasets. Existing systems are either fully automated, which can be unreliable, or highly manual, which imposes a cognitive burden on users. A unified approach is needed.
Core Problem
The core challenge is the lack of a system that allows users to seamlessly transition between AI-generated outputs and manual data inspection. Fully automated systems struggle with precise numerical claims, while manual systems demand high user effort.
Innovation
Key innovations include: 1) Integration of multi-modal tools (Auto Check, AI Chat, Table Explorer, Chart Builder); 2) Support for flexible workflows combining AI and manual tools; 3) Logging user interactions to study trust calibration and workflow patterns.
Methodology
- �� Automated detection: Uses GPT-4.1 to identify data claims in text.
- �� Manual annotation: Users can highlight text to mark claims.
- �� Multi-modal verification: Provides Auto Check, AI Chat, Table Explorer, and Chart Builder for claim investigation.
- �� User determination: Users submit final judgments with optional corrections.
Experiments
The study involved 22 participants who used FYI to fact-check claims in a data-driven article. User interactions, tool usage sequences, and trust dynamics were logged to analyze workflows and trust calibration.
Results
Three workflows emerged: AI-first with manual confirmation, manual-first with AI support, and parallel co-review. Visualization was the primary tool for auditing AI conclusions. Trust in AI increased with consistent outputs and decreased with inconsistencies.
Applications
FYI is suitable for fact-checking data-driven journalism and policy reports, particularly those requiring verification of structured data claims. It also has potential as an educational tool for teaching data literacy and critical thinking.
Limitations & Outlook
FYI currently supports only structured datasets and may not handle unstructured claims. The small sample size limits generalizability, and the AI modules depend on GPT-4.1, which has inherent limitations.
Plain Language Accessible to non-experts
Imagine you're reading an article claiming, 'Higher budgets lead to higher box office revenue.' You want to verify this. FYI acts like a smart assistant embedded in your browser. It highlights sentences that might need checking and offers four tools: Auto Check for quick AI validation, AI Chat for asking questions, Table Explorer for viewing raw data, and Chart Builder for creating graphs. For example, you could use Table Explorer to find budget and revenue data, then use Chart Builder to plot trends. Finally, you decide whether the claim is true and submit your judgment.
ELI14 Explained like you're 14
Imagine seeing an article saying, 'Playing video games makes your grades worse.' You think, 'Really? Is that true?' FYI is like a super-smart helper right in your browser. It highlights sentences that might need checking. Then, you can use tools like a chart maker to see trends or ask the AI directly, 'Is this true?' You get to decide if the claim is right or wrong and submit your answer. Cool, right?
Glossary
Data Claim
A quantitative statement based on structured data, such as statistical summaries or trends.
Sentences in articles that require verification, e.g., 'Higher budgets lead to higher revenue.'
Auto Check
A tool in FYI that automatically generates verification code and outputs conclusions.
Used to quickly verify the accuracy of data claims.
Chart Builder
A tool for users to create visualizations to explore data patterns.
Helps users understand data distributions and trends visually.
Trust Calibration
The process of adjusting user trust in AI based on the quality of its outputs.
Trust increases when AI outputs are consistent and reliable.
Mixed-Initiative System
A system combining AI automation and user-driven actions.
FYI allows users to switch between AI and manual tools for fact-checking.
Open Questions Unanswered questions from this research
- 1 How can AI modules be improved for complex data claims?
- 2 How can FYI support unstructured data verification?
- 3 How can the user interface be optimized to reduce learning curves?
Applications
Immediate Applications
Journalism Fact-Checking
Journalists can use FYI to quickly verify statistical claims in data-driven articles, improving reporting credibility.
Educational Tool
FYI can be used in classrooms to teach students data analysis and critical thinking.
Long-term Vision
Promoting Data Literacy
By popularizing FYI, public understanding of data and critical thinking can improve, reducing misinformation spread.
Abstract
Data-driven journalism and policy reports frequently rely on statements grounded in statistical evidence, referred to as data claims. Verifying such a claim requires connecting it to the underlying structured dataset. However, existing systems typically isolate automated fact-checking from manual data exploration, leaving it unclear how readers coordinate AI assistance with manual inspection of the evidence in practice. We present FYI, a browser extension that embeds fact-checking in the reading environment, and use it as a design probe to study how people detect, verify, and determine the validity of data claims against the underlying dataset. FYI provides four complementary tools spanning the spectrum from full automation to manual data exploration. In an exploratory study (N=22), participants used FYI to fact-check claims in a data-driven article. We find that participants adopted three distinct workflow archetypes---AI-first with manual confirmation, manual-first with AI supplement, and parallel co-review---with visualization serving as the primary mechanism for auditing AI conclusions. Trust in AI shifted dynamically, growing when multiple tools converged and eroding when AI outputs were inconsistent. These findings suggest that fact-checking systems should treat AI as a starting point that human verification complements rather than a definitive authority, elevate visualization as a core verification capability, and support flexible, user-driven workflows. We release FYI as open-source software for further research at https://github.com/DataVisards/FYI.