QuALITY: Question Answering with Long Input Texts, Yes!
Introduces QuALITY, a long-text (≈5000 words) multiple-choice QA dataset, with models achieving only 55.4% accuracy versus 93.5% by humans.
Key Findings
Methodology
The study presents QuALITY, a challenging long-document QA dataset built via a multi-stage crowdsourcing pipeline. Writers, trained to read entire passages, craft questions requiring integration across multiple sections. Validation involves rapid (45s) answering to identify difficult questions, alongside traditional untimed validation for answer correctness. The dataset comprises 6,737 questions over texts averaging 5,159 tokens. Models like Longformer, RoBERTa, DeBERTaV3, and T5 are tested, combined with retrieval techniques (ROUGE, DPR, fastText). The best model achieves 55.4% accuracy, significantly below human performance (93.5%), highlighting the task's difficulty.
Key Results
- The top-performing DeBERTaV3-large with DPR retrieval reaches 55.4% accuracy, with the hard subset at 46.7%, far below human accuracy of 93.5%. Incorporating retrieval improves performance slightly, but understanding long texts remains a challenge.
- Model performance varies across question types and difficulty levels, with notable struggles on questions requiring deep reasoning and multi-paragraph comprehension.
- Human annotators consistently outperform models, emphasizing the gap in current AI capabilities for holistic long-text understanding.
Significance
This work advances long-text comprehension by providing a high-quality, challenging dataset that reflects real-world complexity. It addresses the limitations of existing short-context datasets, pushing models toward deeper understanding, reasoning, and information integration—crucial for applications like legal analysis, academic research, and news summarization. The dataset also establishes a benchmark for future innovations in long-document AI systems.
Technical Contribution
The paper introduces a novel combination of long-document encoders (LED, Longformer) with retrieval-based augmentation (DPR, ROUGE, fastText). It innovates with a multi-stage data collection process, including rapid validation to filter out easy questions, ensuring high difficulty. The integration of retrieval and encoding strategies provides a new pathway for handling lengthy, complex texts in QA tasks, setting a foundation for more sophisticated models.
Novelty
This is the first large-scale dataset with an average length exceeding 5,000 tokens specifically designed for multiple-choice QA, emphasizing real-world complexity. The multi-stage validation process, especially the speed-based difficulty filtering, ensures high challenge level. The combined use of retrieval and long-range encoders marks a significant methodological advancement over prior short-text datasets.
Limitations
- Despite progress, model accuracy remains far below human levels, especially on questions demanding deep reasoning and multi-step inference, indicating room for architectural improvements.
- The dataset is primarily in English and based on specific sources, limiting cross-lingual and domain generalization.
- High computational costs and resource requirements restrict large-scale deployment and real-time applications.
Future Work
Future directions include developing more efficient long-range models, integrating multi-modal data (images, videos), and expanding to multilingual datasets. Improving reasoning capabilities and reducing computational costs are priorities. Additionally, exploring transfer learning from related tasks and unsupervised pretraining on diverse long texts could further enhance performance.
AI Executive Summary
Understanding long documents remains a significant challenge in natural language processing. Existing models excel at short passages but struggle with the complexity and scale of real-world texts like legal documents, scientific articles, or lengthy narratives. To address this gap, this study introduces QuALITY, a large-scale, high-quality dataset specifically designed for multi-choice question answering over long texts averaging over 5,000 words. The dataset was carefully constructed through an innovative crowdsourcing pipeline that emphasizes difficulty and ambiguity, ensuring that questions require comprehensive understanding and reasoning across multiple parts of the passage.
The data collection process involved expert writers who read entire articles and crafted questions that could not be answered through simple skimming or keyword search. Validation included rapid answering within 45 seconds to identify questions that challenge even quick search strategies, as well as traditional untimed validation for correctness. This rigorous approach resulted in a dataset with 6,737 questions, nearly half of which are classified as hard, based on human performance.
Model experiments employed state-of-the-art long-range encoders such as Longformer and LED, combined with retrieval mechanisms like DPR, ROUGE, and fastText to enhance input relevance. Despite these efforts, the best models achieved only 55.4% accuracy, significantly below human performance at 93.5%. The results highlight the persistent difficulty of long-text comprehension, especially for questions requiring deep reasoning, multi-paragraph integration, and inference.
This research marks a substantial step forward by providing a challenging benchmark that closely mirrors real-world demands. It underscores the necessity for further innovations in model architecture, training strategies, and multi-modal integration to bridge the gap between AI and human understanding. The dataset and insights from this work will serve as a foundation for future advances in AI systems capable of truly understanding and reasoning over lengthy, complex texts.
Deep Analysis
Background
长文本理解是自然语言处理中的核心难题之一。早期工作如SQuAD（2016）主要关注短文本问答，难以应对法律、科研、新闻等实际场景中的长篇内容。NarrativeQA（Koˇciský et al., 2018）引入长篇文本问答，但多偏重文学作品，且题目多为简洁回答。近年来，长文本问答成为研究热点，CosmosQA（Huang et al., 2019）和RACE（Lai et al., 2017）提供更长文本，但仍缺乏真实复杂场景的挑战性数据。长文本理解的难点在于信息跨段整合、深层推理和计算资源限制，亟需高质量、难度适中的数据集推动技术突破。
Core Problem
现有模型在长文本问答中的表现远低于人类，主要受限于输入长度限制、信息整合能力不足和推理复杂性。短文本数据集无法反映实际场景中的理解难度，长文本任务需要跨越多个段落、理解上下文关系、进行深度推理。如何设计具有挑战性且高质量的长文本数据，成为制约研究的重要瓶颈。此外，模型在长文本中的表现受硬件和算法效率限制，亟需创新架构和训练策略。
Innovation
本研究的创新点包括：1）构建平均长度达5159词的长文本多选问答数据集QuALITY，代表真实复杂场景；2）采用多阶段众包流程，结合快速验证（45秒内回答）筛选难题，确保题目难度；3）引入多模型融合策略，包括长文本编码（LED、Longformer）和检索增强（DPR、ROUGE、fastText），提升理解能力；4）设计二步检索+模型策略，有效缓解输入长度限制，提升性能。这些创新突破了以往短文本为主的评估体系，为长文本理解提供新思路。
Methodology
- �� 数据采集：利用CC-BY许可的长文本（如Gutenberg、Slate）构建题库，最大长度6k词，内容丰富。• 问题设计：由专业写手阅读全文，撰写具有挑战性的问题，结合多段信息。• 验证流程：采用速度验证（45秒内回答）筛选出难题，确保模型不能简单搜索即答。无时间验证确保答案唯一。• 模型训练：使用Longformer、LED支持长文本编码，结合DPR、ROUGE、fastText进行检索，筛选相关句子作为输入。• 多模型融合：将检索结果与预训练模型（RoBERTa、DeBERTaV3、T5）结合，进行多选问答。• 评估指标：采用准确率，最大达55.4%，硬例（Hard subset）为46.7%，显示长文本理解仍有巨大提升空间。
Experiments
实验采用长文本数据集QuALITY，划分训练、验证和测试集，确保不同写手的样本分布。模型包括Longformer、LED、RoBERTa、DeBERTaV3和T5，结合多种检索机制（ROUGE、DPR、fastText）筛选信息。模型训练采用交叉验证，超参数调优，重点在于检索策略的效果和模型融合性能。对比单模型、二步检索+模型和oracle检索等多种策略，分析不同方法在硬例和易例上的表现差异。通过ablation研究验证各组件贡献，确保模型在长文本理解中的有效性。模型性能指标主要为准确率，最大达55.4%，硬例表现为46.7%，显示长文本理解仍有巨大提升空间。
Results
模型在长文本问答中的表现显著低于人类，最高准确率55.4%，硬例为46.7%，远低于人类的93.5%。引入检索机制后，模型性能略有提升，验证了信息筛选的重要性。不同模型间的性能差异明显，DeBERTaV3-large结合DPR表现最佳。模型在复杂推理、多段信息整合方面仍存在瓶颈，提示未来需在模型结构和训练策略上创新。数据分析显示，长文本理解的难点在于信息的跨段整合和深层推理，模型在硬例上的表现尤为不足。
Applications
该数据集和模型策略可应用于法律文档分析、学术论文理解、新闻摘要等场景，尤其适合需要理解长篇内容的自动问答系统。对企业和科研机构而言，提升长文本理解能力有助于自动化信息提取和决策支持。未来，结合多模态信息（如图像、视频）将进一步拓展应用范围，实现更智能的内容理解与交互。
Limitations & Outlook
模型性能仍远低于人类，特别在复杂推理和跨段信息整合方面存在不足。数据集主要基于英文长文本，跨语言适应性有限。训练成本高，硬件要求大，限制大规模部署。未来需优化模型架构，降低计算成本，并扩展多语言、多模态能力。
Plain Language Accessible to non-experts
想象你在读一本很厚的书，要找到答案需要记住很多不同章节的内容。普通的AI模型就像只看书的前几页，难以理解整本书的意思。我们设计了一个特别的“长书理解”任务，让AI像读完整本书一样，回答关于内容的问题。为了让AI学会这个，我们用很多长篇文章，问它一些难题，比如“为什么发生了这件事？”或“这个角色的感受如何？”这些问题需要AI理解很多不同部分的内容，像拼拼图一样把信息拼在一起。我们还用一种特殊的方法，让AI在很短时间内只能快速浏览，测试它是否真的理解了长篇内容。结果显示，AI的表现远远不及人类，说明长文本理解还需要很多努力，就像让一个人看完一本厚书后还能讲出所有细节一样困难。这个研究帮助我们更好地训练AI，让它未来能像人一样理解复杂的长篇内容。
ELI14 Explained like you're 14
想象你在看一本超级厚的小说，要找到答案就像拼拼图一样，要记住很多不同的细节。普通的AI就像只看前几页，根本搞不懂整本书的意思。我们做了个特别的挑战，让AI像读完整本书一样，回答一些很难的问题，比如“为什么这个角色会这么做？”或者“他心里在想什么？”这些问题需要AI把很多不同部分的内容都记住、理解，才能答得好。为了测试AI是不是理解了，我们还让它在很短时间内快速浏览内容，看它能不能找到答案。结果发现，AI答题的准确率只有55%，远远比不上人类的93%。这说明，理解长篇内容还很难，就像让一个人看完一本厚书还能讲出所有细节一样困难。这个研究让我们知道，要让AI理解长篇内容，还需要很多创新和努力。未来，我们希望让AI变得更聪明，能像人一样理解复杂的长篇故事和信息。
Abstract
To enable building and testing models on long-document comprehension, we introduce QuALITY, a multiple-choice QA dataset with context passages in English that have an average length of about 5,000 tokens, much longer than typical current models can process. Unlike in prior work with passages, our questions are written and validated by contributors who have read the entire passage, rather than relying on summaries or excerpts. In addition, only half of the questions are answerable by annotators working under tight time constraints, indicating that skimming and simple search are not enough to consistently perform well. Our baseline models perform poorly on this task (55.4%) and significantly lag behind human performance (93.5%).