The FACTS Leaderboard: A Comprehensive Benchmark for Large Language Model Factuality
The FACTS Leaderboard evaluates large language models' factuality using four sub-leaderboards, with an average score of 68.8.
Key Findings
Methodology
The FACTS Leaderboard evaluates model factuality through four sub-leaderboards: Multimodal, Parametric, Search, and Grounding. Each uses automated scoring models, and the final score is the average of the four.
Key Results
- Gemini 3 Pro scored highest on the FACTS Leaderboard with 68.8, excelling in search scenarios with a score of 83.8.
- GPT 5 showed weaker performance in multimodal and parametric tasks, scoring 44.1 and 55.8 respectively.
- Claude 4.5 performed well in grounding tasks but had an overall score of only 51.3.
Significance
This study provides a comprehensive benchmark for evaluating the factuality of large language models, addressing gaps in current evaluation tools, especially in multimodal and information retrieval scenarios.
Technical Contribution
Introduces a new evaluation framework combining multiple evaluation scenarios, offering a more comprehensive factuality assessment compared to existing methods.
Novelty
First to integrate multimodal, parametric, search, and grounding into a single evaluation framework, providing a comprehensive factuality assessment tool.
Limitations
- Model performance in multimodal tasks is unstable, potentially affected by image quality and question complexity.
- In parametric tasks, models have limited memory for rare facts.
Future Work
Future work could expand to more evaluation scenarios such as video understanding and real-time information retrieval to enhance comprehensiveness and accuracy.
AI Executive Summary
Large language models still face challenges in generating factually accurate text. The FACTS Leaderboard evaluates model factuality through four sub-leaderboards: Multimodal, Parametric, Search, and Grounding. Each uses automated scoring models, and the final score is the average of the four. Experimental results show that Gemini 3 Pro excels in search scenarios, while GPT 5 shows weaker performance in multimodal and parametric tasks. This study provides a comprehensive benchmark for evaluating the factuality of large language models, addressing gaps in current evaluation tools. Future work could expand to more evaluation scenarios such as video understanding and real-time information retrieval to enhance comprehensiveness and accuracy.
Deep Analysis
Background
In recent years, large language models have made significant advances in natural language processing. However, the generated text often contains factual errors, posing challenges to their reliability in practical applications. Existing evaluation methods typically focus on specific aspects, lacking a comprehensive factuality assessment tool.
Core Problem
The text generated by large language models often lacks factuality, especially in multimodal and information retrieval scenarios. This not only affects the practical application of models but also impacts user trust.
Innovation
The FACTS Leaderboard provides a comprehensive factuality evaluation framework by integrating four sub-leaderboards: Multimodal, Parametric, Search, and Grounding. Each sub-leaderboard targets different factuality dimensions, using automated scoring models for evaluation.
Methodology
- �� Multimodal: Evaluates the accuracy of model responses to image-based questions.
- �� Parametric: Assesses the model's internal knowledge in closed-book scenarios.
- �� Search: Evaluates model performance in information retrieval scenarios.
- �� Grounding: Assesses long-form responses' document dependency.
Experiments
Experiments used multiple large language models, including Gemini 3 Pro and GPT 5. The evaluation dataset includes public and private parts to prevent overfitting. Each sub-leaderboard's score is calculated using automated scoring models, and the final score is the average of the four.
Results
Gemini 3 Pro scored highest on the FACTS Leaderboard, particularly excelling in search scenarios. GPT 5 showed weaker performance in multimodal and parametric tasks. Claude 4.5 performed well in grounding tasks but had a lower overall score.
Applications
This evaluation framework can be used to assess the factuality performance of large language models in various scenarios, helping developers identify strengths and weaknesses and guide model improvements.
Limitations & Outlook
Model performance in multimodal tasks is unstable, potentially affected by image quality and question complexity. In parametric tasks, models have limited memory for rare facts.
Plain Language Accessible to non-experts
Imagine you're in a library with many books and pictures. You need to answer questions about these books and pictures. The FACTS Leaderboard is like a librarian helping you check if your answers are correct. It checks if you correctly use information from the books and don't misunderstand the pictures. This way, you can better understand the knowledge in the library.
ELI14 Explained like you're 14
Imagine you're playing a game where you need to answer fun questions about the world. The FACTS Leaderboard is like a game referee checking if your answers are right. If you get it right, you score points; if not, it tells you where you went wrong. This way, you learn more about the world and score high in the game!
Glossary
FACTS Leaderboard
An online leaderboard for evaluating the factuality of large language models.
Used to comprehensively assess model factuality across various scenarios.
Multimodal
Combining multiple information sources (like text and images) for analysis and judgment.
Evaluates model accuracy in answering image-based questions.
Parametric
Using internal model knowledge to answer questions without external information.
Assesses model internal knowledge in closed-book scenarios.
Information Retrieval
Obtaining and using information through search tools.
Evaluates model performance in information retrieval scenarios.
Grounding
Generating long-form responses based on provided documents.
Assesses document dependency of long-form responses.
Open Questions Unanswered questions from this research
- 1 How to improve model stability in multimodal tasks, especially when handling complex images.
- 2 How to enhance model memory for rare facts, especially in parametric tasks.
Applications
Immediate Applications
Model Evaluation
Developers can use the FACTS Leaderboard to evaluate model factuality performance, identify strengths and weaknesses, and guide improvements.
Long-term Vision
Intelligent Assistants
In the future, intelligent assistants can use this evaluation framework to improve answer accuracy, providing more reliable information services to users.
Abstract
We introduce The FACTS Leaderboard, an online leaderboard suite and associated set of benchmarks that comprehensively evaluates the ability of language models to generate factually accurate text across diverse scenarios. The suite provides a holistic measure of factuality by aggregating the performance of models on four distinct sub-leaderboards: (1) FACTS Multimodal, which measures the factuality of responses to image-based questions; (2) FACTS Parametric, which assesses models' world knowledge by answering closed-book factoid questions from internal parameters; (3) FACTS Search, which evaluates factuality in information-seeking scenarios, where the model must use a search API; and (4) FACTS Grounding (v2), which evaluates whether long-form responses are grounded in provided documents, featuring significantly improved judge models. Each sub-leaderboard employs automated judge models to score model responses, and the final suite score is an average of the four components, designed to provide a robust and balanced assessment of a model's overall factuality. The FACTS Leaderboard Suite will be actively maintained, containing both public and private splits to allow for external participation while guarding its integrity. It can be found at https://www.kaggle.com/benchmarks/google/facts .