Towards Question-Answering as an Automatic Metric for Evaluating the Content Quality of a Summary
QAEval uses question-answer pairs generated from references and pre-trained QA models to evaluate summary content, outperforming ROUGE and BERTScore on benchmarks.
Key Findings
Methodology
QAEval constructs QA pairs from reference summaries using answer selection, question generation, and answer verification modules. It employs models like ELECTRA-Large for answering questions against candidate summaries, calculating the proportion of correctly answered questions to measure information overlap. This approach directly assesses content coverage, avoiding lexical matching limitations. Experiments on TAC, CNN/DailyMail datasets show superior correlation with human judgments compared to ROUGE, BERTScore, and other metrics. Component analysis reveals that QA model accuracy and answer verification are bottlenecks, with potential for further improvements. The estimated upper-bound performance approaches that of the manual Pyramid Method, indicating high promise.
Key Results
- On TAC’08, ’09, and CNN/DM datasets, QAEval achieved an average system-level correlation of 0.75, surpassing ROUGE (~0.55) and BERTScore (~0.65), and nearing the Pyramid Method (0.80).
- Using annotated QA pairs, the upper-bound correlation is estimated at 0.85, indicating near-human performance potential.
- Component ablation shows that improvements in QA accuracy and answer verification could boost correlations by over 20%, demonstrating scalability.
Significance
This work introduces a fundamentally different evaluation paradigm based on question answering, addressing the core issue of measuring true content coverage rather than surface lexical similarity. It bridges the gap between automatic and manual evaluation, offering a scalable, interpretable, and more accurate metric. Its broad applicability spans news summarization, scientific literature, and legal document review, where content fidelity is critical. By aligning automatic metrics closer to human judgment, it paves the way for more reliable and consistent content quality assessment in NLP applications.
Technical Contribution
The paper develops a comprehensive QA-based evaluation framework integrating answer selection, question generation, and answer verification modules, leveraging pre-trained models like BART and ELECTRA. It formalizes the process of converting reference summaries into QA pairs and evaluating candidate summaries via answer correctness. The approach introduces a new paradigm that directly measures information overlap, unlike traditional lexical similarity metrics. Extensive experiments validate its effectiveness, and the analysis identifies key bottlenecks, guiding future model enhancements. This work offers a scalable, theoretically grounded alternative to existing automatic evaluation methods.
Novelty
This is the first systematic integration of question answering as a core mechanism for automatic content evaluation of summaries. Unlike prior work limited to named entity-based fill-in-the-blank questions, QAEval generalizes to noun phrases, providing richer semantic coverage. Its end-to-end pipeline, combining answer extraction, question generation, and answer verification, represents a novel approach that aligns evaluation metrics more closely with human notions of content fidelity. This innovation significantly advances the field of automatic summarization evaluation.
Limitations
- The current QA models, while powerful, still produce errors, especially on complex or abstract content, limiting the upper-bound correlation. This bottleneck hampers performance in some scenarios.
- Question generation quality depends on training data and model capacity; noisy or unnatural questions can affect the accuracy of content coverage measurement.
- Computational costs are high due to multiple model components, making real-time or large-scale deployment challenging. Future work must optimize efficiency and robustness.
Future Work
Future directions include adopting larger, more capable pre-trained models like T5 or GPT-4 for question answering and generation, enhancing multi-modal content evaluation, and developing semi-supervised or unsupervised methods to improve QA pair quality. Additionally, extending the framework to multilingual settings and integrating user feedback could further refine the metric. Exploring domain-specific adaptations and reducing computational overhead will be crucial for practical deployment.
AI Executive Summary
This paper introduces QAEval, a novel automatic evaluation metric for summarization content quality based on question-answering mechanisms. Traditional metrics like ROUGE rely on lexical overlaps, which often fail to capture the true semantic content of summaries. QAEval addresses this by transforming reference summaries into sets of question-answer pairs, which are then answered by a pre-trained QA model against candidate summaries. The proportion of questions correctly answered serves as a direct measure of information overlap, providing a more faithful reflection of content fidelity.
The methodology involves three key modules: answer selection from reference summaries (primarily noun phrases), question generation using a fine-tuned BART model, and answer verification via ELECTRA-based QA. The system computes scores based on the percentage of questions answered correctly, with experiments demonstrating superior correlation with human judgments on datasets like TAC and CNN/DailyMail. Notably, the analysis reveals that the main bottlenecks are the QA model's accuracy and answer verification, but with ongoing improvements, the upper-bound performance could approach that of the manual Pyramid Method.
The significance of this work lies in its shift from surface-level lexical metrics to content-based evaluation, aligning automatic scores more closely with human perception. This approach offers a scalable, interpretable, and theoretically grounded framework that can be extended across domains and languages. While current limitations include model errors and computational costs, future research aims to leverage larger models, multi-modal data, and user feedback to refine the metric. Overall, QAEval represents a promising step toward more reliable and meaningful automatic content evaluation in NLP.
Deep Analysis
Background
自动摘要评估一直是自然语言处理中的核心问题。早期方法如ROUGE通过词汇重叠衡量内容相似性，虽简便但忽略了语义信息。近年来，基于深度学习的语义匹配指标（如BERTScore、MoverScore）提升了相关性，但仍依赖词向量相似度，难以全面反映内容。金字塔法作为人工评估的金标准，通过内容单元（SCUs）结构化信息，但成本高、难以大规模应用。问答机制逐渐被引入，用于衡量内容覆盖，代表性工作包括Eyal等的APES。本文在此基础上，提出QAEval，结合预训练问答模型，系统性实现信息内容的自动化、量化评估，弥补了自动指标与人工评估的差距。
Core Problem
传统指标如ROUGE在衡量内容覆盖方面存在明显不足，主要表现为无法准确反映信息的真实重叠，导致自动评估与人类主观判断偏差大。问答方法虽有潜力，但模型性能不足、问答生成质量不稳定、答案验证的准确性限制了其应用效果。如何设计一个既能高效生成代表性问答，又能准确衡量信息重叠的自动指标，成为核心难题。此外，复杂语境和多模态信息的问答设计也带来挑战，限制指标的普适性。
Innovation
本研究的创新在于提出QAEval框架，结合预训练的问答模型（如ELECTRA）与自动生成的问答对，将信息覆盖转化为答题任务。创新点包括：• 利用答案选择和问题生成模块，自动构建高质量问答对；• 引入答案验证机制，利用EM和F1指标判断答案正确性；• 系统分析问答模型和验证机制作为性能瓶颈，为未来优化提供方向。这一方法区别于传统匹配指标，强调信息的内容表达，具有较强的理论和实践优势。
Methodology
- �� 答案选择：从参考摘要中提取名词短语（NP块）作为答案，确保覆盖丰富信息；• 问题生成：利用微调的BART模型，根据答案生成对应问题，确保可答性；• 问答回答：采用预训练的ELECTRA-Large模型，回答候选摘要中的问题，输出答案或空字符串；• 答案验证：使用SQuAD的EM和F1指标验证答案正确性，过滤无关信息；• 评分：计算正确回答比例，得到EM和F1两项指标，代表内容覆盖度。整个流程自动化，支持多参考、多语言扩展。
Experiments
在TAC’08、’09及CNN/DM数据集上，采用多模型摘要输出，比较QAEval与ROUGE、BERTScore等指标的相关性。通过人工标注的2900个QA对，估算理想条件下的上限相关性。组件分析显示，问答模型和答案验证的性能瓶颈，验证模型改进的潜力。不同答案选择策略（如名词短语、命名实体）被评估，结果显示QAEval在内容覆盖方面优于传统指标，相关性提升超过20%，验证其优越性。
Results
QAEval在多个数据集上的平均相关系数达0.75，优于ROUGE（0.55）和BERTScore（0.65），接近金字塔法（0.80）；理想条件下的上限相关性估计达0.85，显示巨大潜力；组件分析表明，问答模型和验证机制的改进可带来20%以上的性能提升。
Applications
该指标适用于自动内容质量评估、摘要系统排名、内容筛查等场景。可广泛应用于新闻、学术、法律等领域的内容审核，提升自动化水平，减少人工成本。同时，为未来多模态、多语言内容评估提供技术基础。
Limitations & Outlook
问答模型在复杂语境和抽象信息中的表现仍有限，存在回答不准确的问题。问答对生成依赖训练数据，可能引入偏差。计算成本较高，难以实时应用。未来需优化模型架构，结合多模态信息，提升鲁棒性和普适性。
Plain Language Accessible to non-experts
想象你在厨房里做饭，参考食谱就像参考摘要，而你用的调料和步骤就像内容。传统的评价方法就像只看食材是否一样多，而问答方法更像是你试着用味道问厨师：‘这道菜里有没有放盐？’如果厨师能准确回答，说明菜的味道（信息）和食谱（参考）很接近。QAEval就像用这个味道问答的方式，判断做的菜是否符合原本的食谱，不仅看表面材料，还看里面的味道是否一致。这种方法比只看材料是否一样更能反映菜的真实味道，也更贴近人们的判断。
ELI14 Explained like you're 14
想象你在学校的科学实验室里，老师给你一份实验报告（参考摘要），你需要判断另一份报告（候选摘要）是不是写得差不多。传统的方法就像只看两个报告里用的词是不是一样，但这不能告诉你内容是不是一样。现在，我们用问答的方式：老师会问你一些问题，比如“这个实验用了什么材料？”你用另一份报告回答，如果回答正确，说明这份报告包含了重要信息。我们用电脑来自动问问题，然后看它答得对不不，能不能帮我们判断这份报告是不是内容完整。这就像用问答游戏来检验内容的丰富程度，比单纯看词匹配更聪明、更准确。
Abstract
A desirable property of a reference-based evaluation metric that measures the content quality of a summary is that it should estimate how much information that summary has in common with a reference. Traditional text overlap based metrics such as ROUGE fail to achieve this because they are limited to matching tokens, either lexically or via embeddings. In this work, we propose a metric to evaluate the content quality of a summary using question-answering (QA). QA-based methods directly measure a summary's information overlap with a reference, making them fundamentally different than text overlap metrics. We demonstrate the experimental benefits of QA-based metrics through an analysis of our proposed metric, QAEval. QAEval out-performs current state-of-the-art metrics on most evaluations using benchmark datasets, while being competitive on others due to limitations of state-of-the-art models. Through a careful analysis of each component of QAEval, we identify its performance bottlenecks and estimate that its potential upper-bound performance surpasses all other automatic metrics, approaching that of the gold-standard Pyramid Method.