SealQA: Raising the Bar for Reasoning in Search-Augmented Language Models
SealQA benchmark evaluates search-augmented LLMs' reasoning under noisy/conflicting info; models perform poorly, accuracy below 50%.
Key Findings
Methodology
SealQA comprises three challenging tasks: SEAL-0, SEAL-Hard, and LongSeal, designed to simulate real-world noisy search environments. Each question is crafted through multi-round expert review, ensuring high difficulty and diversity across domains like science, politics, and history. The benchmark incorporates simulated noise—conflicting, outdated, or misleading documents—and evaluates models using state-of-the-art algorithms such as GPT-4.1, DeepSeek-R1-671B, and LLAMA-3.2-70B. Experiments involve models with and without search tools, analyzing accuracy, error types, and robustness. The evaluation reveals that even top models struggle, with accuracy often below 50%, highlighting their vulnerability to noise and long-text reasoning challenges.
Key Results
- On SEAL-0, the best models like GPT-4.1 achieve only 43.2% accuracy, far below human performance (>70%). Search tools provide limited gains, and performance plateaus or declines with increased reasoning effort, indicating noise amplification.
- LongSeal results show models' inability to reliably identify relevant evidence among thousands of distractors, with accuracy below 20%. Increasing context length or search effort does not significantly improve results, exposing fundamental limitations.
- Models are especially weak in cross-lingual reasoning, false-premise detection, and handling rapidly changing information, exposing gaps in current approaches for real-world applications.
Significance
This work exposes critical weaknesses of current search-augmented language models in noisy, conflicting environments, emphasizing the need for more robust reasoning and filtering mechanisms. It provides a rigorous benchmark to evaluate future models' resilience, guiding research toward models capable of reliable information filtering, even amid misinformation and complex long-text scenarios. The release of SealQA aims to accelerate development of AI systems that can operate reliably in real-world, high-noise settings, impacting industries like journalism, scientific research, and decision-making.
Technical Contribution
The paper introduces a comprehensive evaluation framework combining multi-domain, multi-source, and long-text reasoning challenges. It innovates by integrating expert-crafted adversarial questions, noise simulation, and multi-model assessment, revealing model vulnerabilities. The benchmark emphasizes the importance of filtering, reasoning depth, and robustness, pushing the boundaries of current evaluation practices. It also highlights the limitations of increasing computational effort, advocating for more fundamental improvements in model architecture and training strategies.
Novelty
SealQA is the first benchmark systematically simulating real-world noisy search environments with multi-source, conflicting, and long documents. Unlike existing benchmarks focused on static knowledge or short queries, it emphasizes the challenges of filtering misinformation, reasoning over lengthy texts, and handling dynamic, ambiguous data. Its multi-layered question design and expert vetting set a new standard for evaluating model robustness in practical scenarios.
Limitations
- The dataset size remains limited, focusing on high-difficulty questions, which may not fully represent all real-world scenarios. Future work should expand scope and diversity.
- Evaluation mainly covers text-based models; integration with multimodal data (images, videos) remains unexplored.
- The analysis of internal model mechanisms is limited; further interpretability studies are needed to understand failure modes.
Future Work
Future directions include scaling up SealQA with more questions across additional domains and languages, incorporating multimodal data, and developing models with enhanced filtering and reasoning capabilities. Exploring explainability techniques to understand failure modes, and designing adaptive noise simulation methods to better mimic real-world environments, will be key. Additionally, integrating reinforcement learning and multi-task training could improve robustness, aiming for models that reliably operate in complex, noisy information landscapes.
AI Executive Summary
The rapid evolution of large language models (LLMs) has revolutionized natural language processing, enabling impressive feats in understanding and generation. However, their deployment in real-world scenarios—such as search engines, scientific research, and decision support—faces significant hurdles. Traditional benchmarks like MMLU or GPQA primarily evaluate static knowledge and simple reasoning, failing to capture the messiness, ambiguity, and noise inherent in actual information retrieval. Recognizing this gap, the authors introduce SealQA, a rigorous benchmark designed to evaluate the reasoning capabilities of search-augmented models under realistic noisy conditions.
SealQA comprises three challenging tasks: SEAL-0, SEAL-Hard, and LongSeal. Each task involves questions crafted through expert review to ensure high difficulty, often causing multiple models to fail repeatedly. The benchmark simulates real-world search environments by injecting conflicting, outdated, or misleading documents, testing models' abilities to filter, reconcile, and reason over complex, multi-source information. Experiments with state-of-the-art models—including GPT-4.1, DeepSeek-R1-671B, and LLAMA-3.2-70B—reveal a stark reality: even the most advanced models perform poorly, with accuracy rarely exceeding 50%. Search tools and increased reasoning effort provide limited improvements, and performance often plateaus or declines, indicating that current models are highly sensitive to noise.
These findings underscore the urgent need for more robust reasoning architectures and filtering mechanisms. The benchmark exposes fundamental weaknesses, especially in handling conflicting information, long documents, and rapidly changing data. By releasing SealQA publicly, the authors aim to catalyze research into models that can operate reliably in noisy, real-world environments, ultimately advancing AI's practical deployment in critical sectors like journalism, scientific discovery, and policy-making. Despite its limitations, SealQA sets a new standard for evaluating model robustness, guiding future innovations toward truly resilient AI systems.
Deep Analysis
Background
近年来，随着GPT系列、LLaMA等大规模预训练模型的崛起，NLP在理解和生成方面取得了巨大突破。然而，模型在实际应用中面临信息噪声、虚假内容和长文本推理的挑战。传统评估指标如MMLU、GPQA主要关注知识掌握和短时推理，难以反映模型在复杂、多源、多变环境中的表现。随着搜索增强技术的兴起，模型开始结合外部搜索工具，但在噪声和虚假信息环境下的鲁棒性仍不足。现有评估体系未能充分模拟真实搜索场景中的信息干扰，限制了模型的实际应用能力。
Core Problem
当前大模型在面对真实搜索环境中的噪声、虚假信息和长文本推理时表现不佳，尤其在筛选有用信息、识别虚假内容方面存在明显短板。现有评估多偏向静态知识或简化场景，无法反映复杂环境中的实际需求。这限制了模型在新闻、科研、决策支持等领域的应用效果。解决这一问题需要设计更具代表性和挑战性的基准，系统性测试模型的推理、筛选和抗干扰能力，推动技术突破。
Innovation
本研究的创新点包括：• 提出SealQA基准，结合多场景、多难度、多源信息的真实搜索环境模拟；• 设计专家多轮审核的问题，确保高难度和真实性；• 引入噪声模拟机制，包括虚假信息、冲突答案和长文本干扰，模拟真实搜索场景；• 采用多模型、多算法评测，揭示模型在复杂环境中的脆弱性。与传统基准不同，SealQA强调模型在真实噪声环境中的推理和筛选能力，推动模型更好适应实际应用。
Methodology
- �� 构建多样化问题集，涵盖科学、技术、历史、政治等领域，确保问题复杂且真实；
- �� 每个问题由多轮专家评审，筛选出在多模型、多次尝试中都失败的高难度题；
- �� 设计噪声模拟机制，包括虚假信息、冲突答案和长文本干扰，模拟真实搜索场景；
- �� 采用多模型评测，结合搜索工具（如FRESHPROMPT、API搜索）进行推理，分析模型在不同噪声水平下的表现；
- �� 统计准确率、错误类型和鲁棒性指标，进行多轮对比分析。
Experiments
实验采用多种前沿模型（GPT-4.1、DeepSeek-R1-671B、LLAMA-3.2-70B等），结合不同搜索策略（内置搜索、外部搜索、无搜索）进行评测。指标包括准确率、错误类型、对噪声的敏感性。设置不同推理努力级别，分析推理深度对性能的影响。还引入人类评测，比较模型与专家的差距。实验重点验证模型在噪声、虚假信息和长文本环境中的表现差异，揭示模型的脆弱环节。
Results
模型在SEAL-0中的最高准确率仅为43.2%，远低于人类水平（超过70%）。引入搜索工具后，表现提升有限，反而在噪声环境中表现更差。LongSeal测试中，模型在大量干扰信息中难以识别正确证据，准确率普遍低于20%。增加推理计算量未能显著提升，反而可能导致性能下降。这些结果表明，当前模型在复杂、多源信息环境中的推理和筛选能力仍不足，亟需新技术突破。
Applications
该基准适用于科研机构和企业评估模型在复杂搜索场景中的鲁棒性，推动开发更可靠的AI问答系统。未来在新闻筛查、科学研究、智能决策等领域具有重要应用价值。模型需结合多模态信息、解释机制，提升在真实环境中的表现，满足行业对信息可信度和抗干扰能力的需求。
Limitations & Outlook
目前基准规模有限，未覆盖所有复杂场景，未来需扩大数据集和问题多样性。模型评估多集中在特定架构，尚未验证新兴多模态、多任务模型的表现。对模型机制的深入分析不足，未来应结合解释性方法，理解模型推理失败的根源。评估成本较高，实际应用中仍需优化效率和适应性。
Plain Language Accessible to non-experts
想象你在一个大型图书馆找信息，但书架上有很多虚假或误导的书。你需要通过仔细筛选、判断哪些内容是真的，哪些是误导，才能找到正确答案。这个过程就像模型在搜索信息时遇到的噪声和冲突。SealQA就像设计了一个特别难的“谜题”，让模型在面对大量虚假信息和长篇资料时，仍能找到正确的答案。它测试模型的筛选能力、推理能力和抗干扰能力，就像你在图书馆里学会了如何识别真假书籍一样。这个基准帮助我们了解，未来的AI能不能像人一样聪明，能在复杂的环境中找到真正有用的信息。
ELI14 Explained like you're 14
想象你在学校的图书馆里找资料，但书架上有很多假消息和误导内容。你得花时间筛选，判断哪些是真的，哪些是假的，才能找到正确答案。这就像模型在搜索信息时遇到的困难。SealQA就设计了一些特别难的问题，让模型在面对很多虚假和混乱的信息时，仍然能找到正确的答案。它测试模型的筛选能力和推理能力，就像你学会了如何分辨真假书一样。这个基准可以帮助我们知道，未来的AI是否能像人一样聪明，能在复杂的环境中找到真正有用的信息。
Abstract
We introduce SealQA, a new challenge benchmark for evaluating SEarch-Augmented Language models on fact-seeking questions where web search yields conflicting, noisy, or unhelpful results. SealQA comes in three flavors: (1) Seal-0 (main) and (2) Seal-Hard, which assess factual accuracy and reasoning capabilities, with Seal-0 focusing on the most challenging questions where chat models (e.g., GPT-4.1) typically achieve near-zero accuracy; and (3) LongSeal, which extends SealQA to test long-context, multi-document reasoning in "needle-in-a-haystack" settings. Our evaluation reveals critical limitations in current models: Even frontier LLMs perform poorly across all SealQA flavors. On Seal-0, frontier agentic models equipped with tools like o3 and o4-mini achieve only 17.1% and 6.3% accuracy, respectively, at their best reasoning efforts. We find that advanced reasoning models such as DeepSeek-R1-671B and o3-mini are highly vulnerable to noisy search results. Notably, increasing test-time compute does not yield reliable gains across o3-mini, o4-mini, and o3, with performance often plateauing or even declining early. Additionally, while recent models are less affected by the "lost-in-the-middle" issue, they still fail to reliably identify relevant documents in LongSeal when faced with numerous distractors. To facilitate future work, we release SealQA at huggingface.co/datasets/vtllms/sealqa.