LFQA-HP-1M: A Large-Scale Human Preference Dataset for Long-Form Question Answering
Introduces LFQA-HP-1M dataset with nine scoring metrics, achieving model performance comparable to human preferences in long-form QA evaluation.
Key Findings
Methodology
This work constructs LFQA-HP-1M, a dataset with 1.3 million human pairwise preference annotations across diverse sources like SHP-2, LFQA Eval, and Chatbot Arena. It employs a rigorous filtering pipeline based on a refined long-form question definition. Nine fine-grained rubrics (e.g., completeness, coherence, factuality) are developed, operationalized via tools like G-EVAL and Veriscore. Logistic regression models and SOTA LLMs (e.g., GPT-4) are benchmarked for preference prediction. The study systematically analyzes transitivity, bias, and adversarial robustness, demonstrating that simple interpretable models perform competitively, and models are vulnerable to biases and adversarial perturbations.
Key Results
- LFQA-HP-1M contains approximately 300K unique questions, with preference prediction accuracy of 68%, close to GPT-4’s 69%. The nine rubrics reveal that coherence and completeness are the strongest predictors of human preference, while relevance and grammar show negative weights, indicating possible constant or non-discriminative features. The linear models achieve performance comparable to GPT-4, validating the interpretability and effectiveness of the rubric-based approach.
- Preference consistency tests show that models maintain a reasonable level of transitivity, though some intransitive cycles exist, mainly influenced by answer length and position biases. Adversarial experiments indicate models are somewhat robust to semantic-preserving perturbations but still susceptible to certain adversarial attacks, highlighting areas for improvement.
- This dataset and evaluation framework enable transparent, interpretable, and scalable assessment of long-form answers, addressing limitations of existing metrics and black-box models. It provides a foundation for future research on preference alignment, bias mitigation, and robustness enhancement in large language models.
Significance
This research addresses critical gaps in LFQA evaluation by providing the largest publicly available preference dataset and a multi-faceted rubric framework. It enhances transparency and interpretability in automatic evaluation, facilitating better model alignment with human preferences. The insights into biases and robustness inform future development of more reliable and fair AI systems, impacting both academia and industry by enabling more user-aligned and trustworthy long-form generation models.
Technical Contribution
The paper’s main contributions include the creation of LFQA-HP-1M, a large-scale annotated preference dataset; the design of nine comprehensive scoring rubrics; and the demonstration that simple linear models based on these rubrics can rival complex LLM evaluators. It introduces a systematic analysis of biases and adversarial robustness, proposing evaluation protocols that improve transparency and reliability. These innovations advance the state-of-the-art in reference-free, preference-based evaluation of long-form QA systems.
Novelty
This is the first work to release such a large-scale, detailed preference dataset specifically for LFQA, integrating nine interpretability-driven rubrics. Unlike prior metrics that lack transparency, this approach combines explainable scoring with competitive predictive performance. The systematic bias and robustness analysis further distinguish this work, offering new insights into the limitations of current LLM evaluators and proposing practical solutions for improvement.
Limitations
- Models show limited robustness to complex semantic adversarial attacks, and biases related to answer length and position persist. The dataset is predominantly English and domain-specific, limiting cross-lingual and multi-domain applicability.
- Current evaluation relies on static rubrics and tools, which may not capture all nuanced preferences. Computational costs for large-scale annotation and model inference remain high.
- Further research is needed to extend the framework to multilingual settings and to incorporate dynamic, user-specific preferences for personalized evaluation.
Future Work
Future directions include expanding the dataset to multiple languages and domains, developing adaptive and personalized preference models, and integrating user feedback for continuous improvement. Enhancing model robustness against adversarial attacks and biases remains a priority. Additionally, exploring more sophisticated scoring mechanisms and explainability techniques will further improve transparency and trustworthiness of automated LFQA evaluation.
AI Executive Summary
Long-form question answering (LFQA) has become a vital component of advanced AI systems, aiming to generate detailed, multi-sentence responses to complex queries. Despite significant progress, evaluating the quality of these answers remains challenging. Traditional metrics like ROUGE and BLEU are limited in capturing human preferences, which are inherently multi-dimensional and nuanced. To address this, the authors introduce LFQA-HP-1M, the largest publicly available dataset with 1.3 million human pairwise preference annotations, curated from diverse sources such as SHP-2, LFQA Eval, and Chatbot Arena. This dataset is constructed through a rigorous filtering pipeline based on a refined definition of long-form questions, ensuring high precision and relevance.
Building upon this dataset, the authors propose a rubric-driven evaluation framework comprising nine detailed scoring metrics, including completeness, coherence, factuality, and relevance. These rubrics are operationalized using tools like G-EVAL and Veriscore, enabling automated, interpretable scoring aligned with human judgments. The core methodology involves training simple logistic regression models on these rubric scores, which achieve preference prediction accuracy of 68%, comparable to GPT-4’s 69%. This demonstrates that transparent, rule-based models can rival complex large language models in preference prediction.
The study further investigates biases such as answer length and positional effects, revealing that models exhibit some biases but maintain a degree of transitive consistency. Adversarial robustness tests show that models are somewhat resilient to semantic-preserving perturbations but still vulnerable to certain adversarial attacks, indicating room for improvement.
Overall, this work advances LFQA evaluation by providing a transparent, scalable, and interpretable framework grounded in human preferences. It offers valuable resources for researchers and industry practitioners aiming to develop more aligned, trustworthy, and robust long-form generation systems. Future efforts will focus on expanding the dataset to multilingual and multi-domain settings, enhancing model robustness, and refining evaluation metrics to better reflect diverse user preferences.
Deep Analysis
Background
长文问答（LFQA）近年来成为自然语言处理中的重要研究方向，旨在生成详细、结构合理的多句回答，满足复杂问题的解答需求。早期工作如SQuAD、NarrativeQA、ELI5等提供了基础数据集，但多依赖参考答案，难以反映多样化偏好。随着大规模预训练模型的发展，LFQA的生成质量显著提升，但评估仍面临挑战，尤其是偏好偏差和指标缺乏透明性。现有指标如ROUGE、BERTScore等在反映人类偏好方面表现有限，模型偏好判定多依赖黑箱模型，缺乏可解释性。部分研究如Rosset等提出偏好一致性原则，但缺乏规模化数据支持。本文在此背景下，构建了规模最大、细粒度的偏好数据集，结合多维评分指标，推动了评估方法的革新。
Core Problem
当前长文问答评估主要依赖参考文本或单一指标，难以全面反映人类偏好。模型偏好判定存在偏见（如长度、位置偏差）且缺乏透明度，限制了模型优化和偏好对齐的有效性。缺少大规模、细粒度、公开的偏好数据集，导致评估结果不够可靠，难以指导模型改进。此外，模型鲁棒性不足，容易受到微调或对抗扰动影响，影响实际应用中的稳定性和可信度。
Innovation
本文的核心创新在于：1）构建规模最大、标注细粒度的LFQA偏好数据集LFQA-HP-1M，涵盖多源、多领域；2）提出九项评分指标（如完整性、连贯性、事实性），实现多维度、透明化评估；3）验证线性模型在偏好预测中的竞争优势，增强模型可解释性；4）系统分析模型偏见和鲁棒性，提出对抗扰动评估方法，提升模型稳定性。这些创新突破了传统单一指标和黑箱模型的局限，为偏好导向的自动评估提供了新思路。
Methodology
- �� 数据采集：整合SHP-2、LFQA Eval、Chatbot Arena等多源偏好数据，筛选长文问答。• LFQA定义：制定严格的长文问答定义，排除非长文问题。• 过滤策略：采用问答提示和模型筛选，确保数据质量。• 指标设计：提出九项评分指标，结合LLM（如G-EVAL、Veriscore）自动评估。• 模型训练：用逻辑回归模型结合指标预测偏好，验证指标有效性。• 鲁棒性分析：测试模型传递性、偏见和对抗扰动，评估稳定性。
Experiments
- �� 数据集：LFQA-HP-1M，包含300K问题对。• 评估指标：偏好准确率68%，与GPT-4（69%）相近。• 模型：线性逻辑回归、GPT-4、Gemini-2.5等。• 设计：随机抽样偏好样本，验证指标预测能力。• 其他：偏好一致性检验、偏见检测、对抗扰动测试。
Results
- �� 线性模型在偏好预测中达68%准确率，表现接近GPT-4。• 九项指标中，连贯性和完整性为偏好最强指标，相关性和语法为负向指标。• 模型偏好具有一定传递性，但存在长度和位置偏差。• 对抗扰动实验显示模型对微调保持一定鲁棒性，但仍存在提升空间。
Applications
- �� 立即应用：自动化长文问答评价，提升模型偏好对齐效率。• 长远愿景：实现可解释、透明的偏好评估体系，推动个性化问答系统和人机交互的智能化发展。
Limitations & Outlook
- �� 当前偏好模型对极端偏好和复杂语义扰动敏感，鲁棒性不足。• 数据集偏向英文和特定领域，跨语言适应性有限。• 评估指标仍需丰富，未来需结合用户反馈持续优化。
Plain Language Accessible to non-experts
想象你在一家厨房做饭，厨师需要判断哪份菜更好吃。传统方法可能只看颜色或味道的描述，但这不能完全反映大家的喜好。现在，厨师用了一套评分标准，比如菜的完整度、味道是否浓郁、摆盘是否漂亮、是否符合健康要求等。每一道菜都根据这些标准打分，然后用一个简单的公式，结合这些分数，判断哪份菜更受大家喜欢。这就像我们用九个不同的尺子量一份长文问答，最后用一个透明的公式告诉我们哪份回答更符合人类偏好。这种方法比单纯看答案是否正确更公平、更直观，也能帮助厨师不断改进菜肴，满足不同人的口味。
ELI14 Explained like you're 14
你知道吗？当你在网上问问题，别人给你答案，有些答案比其他的更受欢迎。科学家们也在研究，怎么让电脑判断哪个答案更好。以前的方法就像只看答案是不是拼写正确，或者是不是用了一些关键词，但这些都不能完全反映人们的喜好。现在，研究人员设计了一套“评分尺子”，比如答案是不是完整、是不是有逻辑、是不是很有趣、是不是很准确。他们用这些尺子给每个答案打分，然后用一个简单的公式，告诉电脑哪个答案更符合人类的偏好。这样，电脑就能更聪明地判断答案的好坏，也能帮人们找到更满意的回答。这就像你用多个标准来评价一份作业，最后得出一个公平的结论。
Abstract
Long-form question answering (LFQA) demands nuanced evaluation of multi-sentence explanatory responses, yet existing metrics often fail to reflect human judgment. We present LFQA-HP-1M, a large-scale dataset comprising 1.3M human pairwise preference annotations for LFQA. We propose nine rubrics for answer quality evaluation, and show that simple linear models based on these features perform comparably to state-of-the-art LLM evaluators. We further examine transitivity consistency, positional bias, and verbosity biases in LLM evaluators and demonstrate their vulnerability to adversarial perturbations. Overall, this work provides one of the largest public LFQA preference datasets and a rubric-driven framework for transparent and reliable evaluation.