A Critical Evaluation of Evaluations for Long-form Question Answering
This study evaluates LFQA with human experts and automatic metrics, revealing no single metric reliably predicts human preferences.
Key Findings
Methodology
The authors recruited domain experts across seven fields to perform pairwise answer preference judgments, providing detailed justifications. They curated 12 automatic metrics, analyzing their correlation with human preferences on overall quality and specific attributes like coherence and factuality. The expert evaluations emphasized answer comprehensiveness and factual correctness, revealing the complexity and subjectivity involved. The dataset includes 260 expert ratings and explanations, supporting a multi-attribute evaluation framework. The analysis highlights the limitations of existing automatic metrics and advocates for multi-dimensional assessment to better capture answer quality.
Key Results
- Experts preferred model answers in 62.4% of cases overall, with strong domain variation—favoring models in economics and law (90%) but favoring humans in history (75.6%). Automatic metrics like QAFactEval and RankGen showed some correlation with fine-grained attributes but poorly predicted overall preferences. Experts prioritized answer completeness and factuality, whereas crowdworkers focused on surface features like conciseness. Disagreement among experts underscores the subjective nature of quality judgments. The study advocates multi-attribute evaluation to address these challenges, providing a comprehensive dataset and tools for future research.
- Analysis of automatic metrics revealed that models like T5-11B could understand justifications better, with higher token-level exact match scores. However, overall correlation with human preferences remained weak. The evaluation across domains demonstrated the difficulty of capturing answer quality with a single score, emphasizing the need for multi-faceted metrics that evaluate factuality, completeness, coherence, and understandability. The results suggest that combining expert judgment with advanced automatic metrics can improve LFQA evaluation, guiding model development towards more reliable and comprehensive answers.
- The findings underscore the importance of multi-dimensional evaluation frameworks. While models like GPT-3 can generate answers comparable to or better than humans in some domains, their performance drops in complex, nuanced questions. The study's multi-attribute approach provides a nuanced understanding of answer quality, encouraging future work to develop better automatic metrics and scalable expert annotation methods. Overall, this research advances the field by highlighting the gaps in current evaluation practices and proposing practical solutions for more accurate, holistic assessment of long-form answers.
Significance
This research marks a pivotal step in LFQA evaluation, revealing the inadequacies of existing automatic metrics and emphasizing the importance of multi-attribute assessment. By involving domain experts, it addresses the limitations of crowd-based judgments and provides a nuanced understanding of answer quality. The findings have broad implications for both academia and industry, especially in high-stakes fields like healthcare and law, where answer correctness and completeness are critical. The publicly released dataset and tools facilitate future innovations in automatic evaluation, fostering the development of more reliable, scalable metrics. Ultimately, this work paves the way for more trustworthy and effective long-form question answering systems, aligning evaluation practices with real-world needs.
Technical Contribution
The paper introduces a comprehensive multi-attribute evaluation framework for LFQA, combining expert judgments with a systematic analysis of 12 automatic metrics. It develops a dataset of 260 expert ratings with detailed justifications, enabling nuanced analysis of answer properties. The study evaluates the correlation between automatic metrics and human preferences, revealing their limitations and potential in fine-grained attributes. It also proposes training a new learned metric based on large pretrained models (e.g., T5) fine-tuned on preference data, demonstrating improved understanding of answer quality aspects. This approach advances automatic evaluation by integrating multi-dimensional signals and expert knowledge, setting a new standard for LFQA assessment.
Novelty
This is the first comprehensive study to analyze the multi-faceted nature of LFQA evaluation, involving domain experts and a broad suite of automatic metrics. Unlike prior works focusing solely on overall scores or simple string similarity, it emphasizes answer attributes like factuality and completeness. The integration of expert justifications and training of learned metrics based on preference data represents a significant innovation, bridging the gap between human judgment and automatic evaluation. The research also provides a publicly available dataset, fostering community engagement and future development of more sophisticated, multi-dimensional evaluation methods.
Limitations
- Expert annotation, while precise, is costly and not scalable for large datasets, limiting widespread adoption. The subjective nature of preferences across domains introduces variability, complicating standardization. Current automatic metrics, despite showing some promise in fine-grained attributes, cannot reliably predict overall answer quality, especially in complex or ambiguous cases. The dataset is primarily based on the ELI5 corpus, which may not generalize well to other domains or question types. Additionally, the computational costs of training large models for learned metrics pose practical challenges for widespread use.
Future Work
Future research should focus on developing scalable, domain-adaptive automatic metrics that better predict human preferences across diverse answer properties. Incorporating user feedback and real-world application scenarios can refine evaluation frameworks. Exploring semi-supervised or active learning approaches to reduce annotation costs, as well as expanding datasets to include more varied domains, will be vital. Additionally, integrating explainability into automatic metrics can enhance interpretability and trustworthiness. Ultimately, establishing standardized, multi-attribute evaluation benchmarks will accelerate progress toward more reliable and comprehensive LFQA systems.
AI Executive Summary
Long-form question answering (LFQA) has become a critical area in natural language processing, aiming to generate detailed, accurate, and comprehensive answers to complex questions. Despite rapid advances in model capabilities, evaluating the quality of these answers remains a significant challenge. Traditional metrics like ROUGE and BLEU, originally designed for translation and summarization, are inadequate for capturing the nuanced qualities of long, open-ended responses. Moreover, crowd-sourced evaluations, while more aligned with human judgment, suffer from subjectivity and lack domain expertise, especially when assessing factual correctness and completeness.
This study pioneers a systematic evaluation of LFQA by involving domain experts across seven fields—biology, physics, chemistry, economics, law, technology, and history—to provide preference judgments and detailed justifications. The findings reveal that experts tend to prefer answers based on properties such as factuality and comprehensiveness, with preferences varying significantly across domains. Interestingly, in some fields like economics and law, models like GPT-3 outperform humans in answer preference, whereas in history, human answers are favored. These results highlight the complexity and domain-dependence of answer quality.
Complementing human evaluation, the authors analyze 12 automatic metrics, including ROUGE, BERTScore, QAFactEval, and RankGen. The analysis shows that no single metric reliably predicts overall human preferences. However, some metrics correlate with specific answer attributes like coherence and factuality, suggesting potential pathways for developing multi-faceted automatic evaluation systems. The authors further train a learned metric based on large pretrained models, demonstrating improved understanding of answer quality attributes.
The research advocates shifting from single-score evaluations toward multi-attribute frameworks that consider factuality, completeness, coherence, and understandability. Such an approach aligns better with human judgment and is crucial for advancing LFQA systems for real-world applications, especially in high-stakes domains. The study also releases a dataset of expert annotations and evaluation tools, fostering future research. Overall, this work significantly advances the methodology for evaluating long-form answers, emphasizing the importance of nuanced, multi-dimensional assessment for building trustworthy AI systems.
Deep Analysis
Background
随着自然语言处理技术的快速发展，长文本问答（LFQA）逐渐成为研究焦点。早期方法主要依赖简单的字符串匹配指标（如ROUGE）评估生成答案的相似性，但难以反映答案的真实性、完整性和可理解性。近年来，模型如GPT-3的出现极大提升了生成能力，但评估体系仍滞后，主要采用非专业众包评价，存在偏差和局限。学术界逐步认识到多维评价的重要性，尝试引入专家评审和细粒度指标，但缺乏系统性分析。本研究结合专家偏好和自动指标，首次全面评估LFQA的多属性特征，为未来建立科学、全面的评估体系奠定基础。
Core Problem
当前LFQA的核心难题在于如何科学、客观地评估答案质量。传统指标如ROUGE、BLEU等在长文本中表现不佳，不能充分反映答案的真实性、完整性和易懂性。众包评价成本低，但缺乏专业性，偏差大。自动指标难以准确预测人类偏好，尤其在专业或复杂问题中表现不足。如何设计既能反映多属性，又具自动化、可扩展的评价体系，成为亟待解决的关键。这关系到模型的优化和实际应用的可靠性。
Innovation
本研究的创新点包括：1）引入领域专家进行偏好判断，确保评价的专业性和准确性；2）分析答案的多属性（如事实性、完整性、连贯性），推动多维评价体系建立；3）系统评估12种自动指标，揭示其在细粒度属性上的潜力与不足；4）提出结合专家偏好和自动指标的多属性评价框架，突破传统单一指标限制。这些创新为LFQA的科学评价提供了新思路，有助于模型的持续优化。
Methodology
- �� 采集7个专业领域的专家评价，提供偏好判断和详细理由。• 设计A/B偏好测试，比较模型答案与高/低投票人类答案。• 使用GPT-3（175B）生成长文本答案，采集不同领域的样本。• 评估12种自动指标（如ROUGE、QAFactEval、RankGen），分析其与人类偏好的相关性。• 收集260个专家评分和理由，建立多维评价数据集。• 通过人工分析识别影响偏好的关键属性（如事实性、完整性、易懂性）。• 训练自动指标模型，结合偏好数据优化预测能力。
Experiments
采用ELI5数据集，结合最新问题，采集专家偏好标注。模型使用GPT-3生成答案，自动指标评估答案质量。设计多领域（如生物、经济、历史）样本，比较模型与人类答案偏好。评估指标包括传统（ROUGE、BLEU）和新兴（QAFactEval、RankGen），分析其相关性。通过统计分析和回归模型验证指标在不同属性上的预测能力。实验结果显示，自动指标在整体偏好预测上表现不足，但在细粒度属性（如连贯性、事实性）方面有潜力。
Results
专家偏好模型答案比例达62.4%，在经济和法律领域偏向模型答案，历史偏向人类答案。自动指标如QAFactEval在事实性检测上表现优越，但整体预测能力有限。不同领域专家对答案评价标准差异明显，反映偏好具有高度主观性。自动指标在预测细粒度属性（如连贯性）方面表现较好，但难以全面替代人类判断。研究提出多维评价体系，强调答案的完整性、事实性和易懂性，为模型优化提供指导。这些发现推动了LFQA评估方法的创新。
Applications
该研究推动LFQA在专业问答、知识库构建、智能助手等场景中的应用。多属性评价体系有助于提升答案的可靠性和用户体验，特别在医疗、法律等领域，确保生成内容的真实性和完整性。未来，结合自动指标与专家评审，可实现高效、准确的自动评估，支持大规模模型训练和部署。长远来看，完善的多维评价体系将推动自然语言生成技术迈向更高的智能水平，满足复杂场景的需求。
Limitations & Outlook
当前评价体系依赖专家标注，成本较高，难以大规模推广。自动指标在整体偏好预测上仍不足，特别是在复杂或专业性强的问题中表现欠佳。数据集局限于ELI5，泛化到其他任务或领域仍需验证。模型在处理极端或模糊问题时可能出现偏差，未来需提升模型理解与评估的鲁棒性。评估体系的多维性也带来复杂性，如何平衡不同属性的权重仍是挑战。
Plain Language Accessible to non-experts
想象你在厨房里做饭。每道菜都要看味道、颜色、营养和摆盘。只用味道好不好来评价，可能会忽略菜的营养或外观。科学评价长答案也是一样，不能只看一句话是否通顺，还要考虑内容是否完整、是否真实、是否容易理解。专家就像厨师，他们用专业知识判断每个方面。自动指标像是厨师的评分表，只能看部分内容，不能完全替代专业厨师的判断。为了做出最好的菜（答案），我们需要多角度、多方面的评价体系，就像厨房里要兼顾味道、健康和美观一样。
ELI14 Explained like you're 14
想象你在学校的食堂吃饭。每个人都喜欢不同的菜，有人喜欢辣的，有人喜欢甜的。老师们想知道哪道菜最好吃，但不能只看一眼就决定。于是，他们会问学生：你觉得这道菜怎么样？是不是营养丰富？颜色漂亮吗？是不是容易吃？每个学生的答案都不一样，有的说味道好，有的说健康重要。科学评价长文本答案也是这样，要看内容是不是全面、真实、容易懂。专家就像厨师，他们用专业知识判断答案的好坏。自动评价指标就像评分表，帮忙快速打分，但不能完全代替专家的判断。为了让答案更好，我们需要用多角度、多方面的评价方法，就像做菜一样，要考虑味道、健康和外观。
Abstract
Long-form question answering (LFQA) enables answering a wide range of questions, but its flexibility poses enormous challenges for evaluation. We perform the first targeted study of the evaluation of long-form answers, covering both human and automatic evaluation practices. We hire domain experts in seven areas to provide preference judgments over pairs of answers, along with free-form justifications for their choices. We present a careful analysis of experts' evaluation, which focuses on new aspects such as the comprehensiveness of the answer. Next, we examine automatic text generation metrics, finding that no existing metrics are predictive of human preference judgments. However, some metrics correlate with fine-grained aspects of answers (e.g., coherence). We encourage future work to move away from a single "overall score" of the answer and adopt a multi-faceted evaluation, targeting aspects such as factuality and completeness. We publicly release all of our annotations and code to spur future work into LFQA evaluation.