Asking and Answering Questions to Evaluate the Factual Consistency of Summaries
QAGS uses question answering to evaluate factual consistency of summaries, outperforming ROUGE with a correlation of 54.52 on CNN/DM.
Key Findings
Methodology
This paper introduces QAGS, an automatic evaluation framework combining question generation (QG) and question answering (QA) models. The process involves generating questions from summaries, answering these questions using both source texts and summaries, and comparing answers via similarity metrics. The core algorithm models the probability distributions of questions and answers conditioned on summaries and source texts, leveraging deep neural models like BART for QG and BERT for QA. This approach addresses the limitations of n-gram overlap metrics by focusing on semantic content, providing interpretability through question-answer pairs that highlight factual discrepancies. The system is trained on datasets like CNN/DailyMail and XSUM, with extensive experiments validating its correlation with human judgments and robustness across model quality, domain shifts, and question quantity.
Key Results
- On CNN/DM, QAGS achieves a Pearson correlation of 54.52 with human judgments, significantly higher than ROUGE-2 at 17.72, demonstrating superior alignment with factual accuracy assessments.
- Increasing the number of questions from 10 to 50 improves correlation, with diminishing returns beyond 20 questions, yet even 5 questions outperform traditional metrics.
- QAGS maintains robustness across different QA model qualities and domain shifts, with only slight performance drops when using less accurate models or out-of-domain data.
Significance
This work addresses the critical challenge of factual inconsistency in neural summarization, providing a reliable, interpretable, and scalable metric that aligns closely with human evaluations. It advances the field by moving beyond surface-level n-gram metrics, enabling better model development and quality assurance in real-world applications like news summarization, legal document generation, and medical report synthesis.
Technical Contribution
The paper pioneers integrating deep question generation and answering models into an evaluation framework, introducing a novel answer similarity measure that captures semantic correctness. It demonstrates that such model-based metrics can outperform traditional overlap-based metrics in correlating with human judgments, offering a new paradigm for automatic factuality assessment. The approach also provides interpretability, as the generated questions and answers pinpoint specific factual errors.
Novelty
This is the first work to embed question answering into automatic summarization evaluation, directly testing factual consistency through model-generated questions and answers. Unlike previous reliance on lexical overlap or entailment models, QAGS leverages the semantic understanding of deep neural networks, setting a new standard for factuality metrics.
Limitations
- The accuracy of QAGS heavily depends on the quality of the underlying QG and QA models, which may introduce biases or errors, especially in low-resource or domain-specific settings.
- It requires significant computational resources due to multiple question-answering passes, limiting real-time deployment.
- The method may struggle with highly abstractive summaries or those containing hallucinated facts not present in source texts, leading to potential false positives or negatives.
Future Work
Future directions include improving question diversity and quality, integrating multi-modal data for broader applicability, and developing lightweight models for real-time evaluation. Additionally, combining QAGS with human-in-the-loop systems or reinforcement learning could further enhance factuality detection and model training.
AI Executive Summary
Automatic summarization has seen rapid progress with neural models like Transformer architectures (Vaswani et al., 2017), producing increasingly fluent and topical summaries. However, a persistent challenge remains: ensuring these summaries are factually accurate. Traditional evaluation metrics such as ROUGE and BLEU primarily measure lexical overlap, which often fails to detect semantic errors or factual inconsistencies. This gap hampers the development of reliable summarization systems, especially in high-stakes domains like news, legal, and medical fields.
To address this, the authors propose QAGS, a novel evaluation framework that leverages question answering (QA) and question generation (QG) models. The core idea is straightforward yet powerful: if a summary is factually consistent with the source, questions generated from the summary should be answerable in the source text with similar answers. Conversely, discrepancies in answers indicate factual errors. The process involves generating questions from summaries using neural QG models, answering these questions in both source and summary contexts with deep QA models, and comparing the answers using token-level F1 similarity. The final score is an average over multiple questions, providing a robust measure of factual consistency.
Extensive experiments on CNN/DailyMail and XSUM datasets demonstrate that QAGS correlates with human judgments at a coefficient of 54.52 and 17.49 respectively, far surpassing traditional metrics like ROUGE-2. The method is shown to be robust to variations in model quality, domain shifts, and the number of questions asked. Importantly, QAGS offers interpretability: the generated questions and answers highlight specific factual discrepancies, aiding model debugging and improvement.
This work significantly advances automatic evaluation by providing a scalable, interpretable, and highly correlated metric for factuality. Its implications extend to improving the training and deployment of summarization models, ensuring content reliability in real-world applications. Despite some limitations related to computational costs and dependency on QA model performance, QAGS opens new avenues for research in factual consistency detection, multi-modal evaluation, and human-AI collaboration for content verification.
Deep Analysis
Background
Automatic summarization技术经历了从基于规则和统计方法到深度学习模型的演变。Transformer（Vaswani et al., 2017）极大提升了生成质量，ROUGE（Lin, 2004）成为主流评价指标。然而，传统指标主要衡量词汇重叠，难以捕捉语义偏差，导致生成内容的事实正确性难以保证。近年来，研究引入自然语言推理（Bowman et al., 2015）和知识图谱（Hoffart et al., 2014）以增强事实检测能力，但仍存在理解不足和解释困难的问题。随着模型能力的提升，确保内容的事实一致性成为核心难题，亟需更有效的自动评估方法。
Core Problem
当前自动摘要中存在大量事实偏差，传统评估指标难以检测这些偏差，导致模型输出的可信度不足。人工评估虽准确但成本高、效率低，难以大规模应用。现有自动指标对语义错误敏感度不足，无法有效识别偏差，特别是在抽象性强或偏离源内容的摘要中。如何设计一种既自动又准确、能提供错误定位的评估指标，成为研究难点。本文提出利用问答机制，通过模型生成的问题和答案对比源文本与摘要内容，从而实现更精确的事实检测。
Innovation
主要创新包括：1）引入基于问答的评估框架，利用深度学习中的QG和QA模型，生成问题并回答，检测摘要与源文本的事实一致性；2）结合答案相似度指标，增强语义偏差的敏感性；3）避免对参考文本的依赖，直接用源文本验证摘要内容；4）通过大规模实验验证其优越性，显著优于传统的n-gram指标和NLI方法。这些创新共同推动了自动事实检测的技术发展，为自动评估提供了新思路。
Methodology
- �� 使用预训练的Transformer模型（如BART、BERT）进行问答生成和回答。
- �� 利用命名实体识别（spaCy）提取答案候选。
- �� 生成多轮问题，筛选高质量问题（过滤低质量、重复、长度不足的问题）。
- �� 用提取式问答模型回答源文本和摘要，得到两个答案集。
- �� 计算答案的F1相似度，作为单个问题的匹配度。
- �� 通过平均所有问题的相似度，得到最终的QAGS分数。
- �� 在CNN/DailyMail和XSUM数据集上，通过人工标注的事实一致性判断验证指标效果。
Experiments
在两个公开数据集上，采用不同的问答生成和回答模型，评估QAGS与人类判断的相关性。对比ROUGE、BLEU、METEOR、BERTScore等指标，分析问答数量、模型质量、领域偏移的影响。还进行了消融实验验证指标的鲁棒性和稳定性，确保统计显著性。实验结果显示，QAGS在不同条件下均优于传统指标，验证其广泛适用性和稳定性。
Results
QAGS在CNN/DM和XSUM上与人类判断的Pearson相关系数分别达到54.52和17.49，显著优于ROUGE-2的17.72和8.95。增加问答数量（如20个）能提升相关性，边际递减明显。问答模型的不同质量对结果影响有限，显示指标鲁棒性。采用答案F1比精确匹配更能捕捉语义偏差，整体性能优越。这些结果表明，QAGS在实际应用中具有较强的可靠性和实用性。
Applications
该方法适用于新闻、法律、医疗等行业的自动内容生成，用于检测和提升内容的真实性。尤其在需要高可信度的场景，如新闻报道自动化、法律文件审核、医疗报告生成等，QAGS能有效识别事实偏差，保障内容质量。未来结合多模态信息和用户反馈，将进一步提升评估的准确性和适应性，推动自动内容验证技术的发展。
Limitations & Outlook
依赖于问答模型的性能，模型偏差可能影响评估效果，尤其在低资源或特定领域表现不足。计算成本较高，尤其在问答数量较多时，实际部署存在挑战。对于极度抽象或偏离源内容的摘要，检测能力有限，可能出现误判。未来需优化模型结构和算法，提高效率和适应性。
Plain Language Accessible to non-experts
想象你在厨房里做饭，食材代表信息，菜谱代表摘要。传统评价就像只看食材的颜色和形状，难以判断菜味是否正宗。而QAGS就像请厨师问你：‘这道菜用了什么调料？’你回答：‘盐和胡椒’。如果你和厨师的答案一样，说明菜做得还不错；如果不一样，说明可能有问题。通过问答，能更准确地判断菜的味道是否符合菜谱。这种方法就像QAGS用问答帮你检查摘要的事实是否正确，既聪明又实用。
ELI14 Explained like you're 14
想象你和朋友在玩拼图游戏，你的任务是拼出一幅图片。传统的方法就像只看拼图的边缘，觉得拼得差不多就算好，但其实里面可能错得很离谱。现在，你的哥哥告诉你：‘我们可以问拼图上的内容，比如：‘这张图片里有一只狗吗？’你回答：‘有，是一只黑色的狗’。如果你的答案和哥哥的答案一样，说明拼图拼得还算对；如果不一样，说明拼图可能拼错了。这样一问一答，就能更准确地判断拼图是不是拼对了。这就像QAGS用问答帮你检查摘要是不是有事实错误一样，既聪明又有趣！
Abstract
Practical applications of abstractive summarization models are limited by frequent factual inconsistencies with respect to their input. Existing automatic evaluation metrics for summarization are largely insensitive to such errors. We propose an automatic evaluation protocol called QAGS (pronounced "kags") that is designed to identify factual inconsistencies in a generated summary. QAGS is based on the intuition that if we ask questions about a summary and its source, we will receive similar answers if the summary is factually consistent with the source. To evaluate QAGS, we collect human judgments of factual consistency on model-generated summaries for the CNN/DailyMail (Hermann et al., 2015) and XSUM (Narayan et al., 2018) summarization datasets. QAGS has substantially higher correlations with these judgments than other automatic evaluation metrics. Also, QAGS offers a natural form of interpretability: The answers and questions generated while computing QAGS indicate which tokens of a summary are inconsistent and why. We believe QAGS is a promising tool in automatically generating usable and factually consistent text.