Ko-WideSearch: A Korean Breadth-Search Benchmark for Exhaustive Set Enumeration by Web Agents
Introduces Ko-WideSearch, a Korean breadth-search benchmark with 228 tables, evaluating web agents' ability for exhaustive set enumeration and attribute filling.
Key Findings
Methodology
This paper constructs Ko-WideSearch using an automated synthesize-and-verify pipeline, covering 228 tasks across three difficulty tiers. Structural parameters—table width and 2D composite key—are independently tuned to vary complexity. A single normalization-aware comparator ensures stable scoring across formats. The pipeline generates gold standard tables by exhaustive web search, with multiple verification gates certifying completeness, non-memorization, and cross-source attribute correctness. Evaluation of 20 web agents reveals that while set membership recovery (Item-F1) reaches 92.8%, attribute-level accuracy (Row-F1) lags at 53.7%, especially in open-ended text cells. Harder tasks see accuracy decline, with the best models achieving only 19.3% success rate in full table correctness.
Key Results
- Models recover most members (Item-F1 92.8%) but struggle with correct attribute filling (Row-F1 53.7%), indicating they identify entities but fail to fill all attributes correctly. Increasing task difficulty reduces overall accuracy, with the strongest model achieving only 19.3% full table success. Standard answer types like dates and names are handled well, but free-text answers often fail, highlighting challenges in multi-source, multi-page structured data retrieval.
Significance
This work fills a critical gap by providing the first systematic benchmark for breadth search in Korean web data, addressing the challenge of comprehensive structured information extraction. It advances the evaluation of web agents' capabilities in multi-source, multi-page, structured data scenarios, with implications for knowledge graph construction, multi-modal information integration, and real-world information retrieval systems. The benchmark's design promotes development of more robust, scalable, and accurate web-based information extraction models, pushing forward the frontier of automated knowledge acquisition.
Technical Contribution
The paper introduces an automated pipeline for large-scale, verified gold table generation, leveraging multi-source web data. It innovates with adjustable structural parameters—table width and 2D composite key—to generate tasks of varying difficulty. A unified, type-aware normalization comparator ensures fair scoring across diverse formats. Multi-source attribute verification enhances correctness, reducing source fragility. These contributions significantly reduce manual effort, improve scalability, and enable systematic evaluation of breadth search capabilities in non-English contexts.
Novelty
This is the first comprehensive Korean breadth-search benchmark integrating automated synthesis, multi-source verification, and adjustable structural difficulty. It extends the WideSearch paradigm to Korean sources, surpassing previous hand-crafted datasets in scale and complexity. The combination of structural parameters and verification mechanisms offers a novel framework for evaluating web agents' structured data retrieval in low-resource language settings.
Limitations
- Models still struggle with open-ended free-text answers, especially when information spans multiple sources, indicating room for improvement in multi-source reasoning. The benchmark relies on static web snapshots, limiting evaluation of dynamic web environments. High computational costs for generating and verifying large datasets pose scalability challenges, requiring future optimization.
Future Work
Future directions include integrating multi-modal data, handling dynamic web content, and developing more efficient verification algorithms. Expanding the benchmark to cover more diverse domains and languages will enhance generalizability. Additionally, improving model architectures to better handle multi-source, multi-page structured data remains a key research goal.
AI Executive Summary
Breadth search represents a fundamental challenge in web information retrieval, aiming to exhaustively enumerate structured data across multiple sources and pages. Unlike traditional deep search benchmarks that focus on multi-step reasoning to find a single answer, breadth search requires comprehensive collection of all relevant entities and their attributes, often spread over diverse web pages. Despite its importance, systematic evaluation frameworks for breadth search, especially in non-English languages like Korean, have been lacking.
This paper introduces Ko-WideSearch, a novel benchmark designed to evaluate web agents' ability to perform exhaustive set enumeration and attribute filling in Korean web data. Built through an automated synthesize-and-verify pipeline, the benchmark encompasses 228 tasks spanning various categories, with adjustable difficulty levels achieved by tuning structural parameters such as table width and 2D composite keys. The pipeline ensures high-quality gold standards by certifying completeness, non-memorization, and cross-source correctness, thus enabling reliable evaluation.
Experimental results across 20 state-of-the-art models reveal that while most models can identify the correct set of entities with high precision (Item-F1 ~92.8%), their ability to correctly fill all attributes (Row-F1) remains limited (~53.7%). Harder tasks, involving wider tables and complex cross-page relationships, further degrade performance, with the best models achieving only 19.3% success in full table correctness. These findings highlight the persistent gap between entity recognition and attribute completion, especially in multi-source, multi-page scenarios.
The benchmark's design and findings have broad implications. They provide a critical evaluation tool for advancing web-based structured data extraction, particularly in low-resource languages. The methodology promotes scalable, automated dataset creation, reducing manual effort and enabling continuous progress. Future work aims to incorporate dynamic web content, multi-modal data, and more efficient verification techniques, paving the way for more robust and comprehensive web agents capable of truly exhaustive information retrieval.
Deep Analysis
Background
随着互联网信息的爆炸式增长，结构化信息的自动化检索成为关键技术之一。早期工作如BrowseComp和WideSearch主要关注深度推理，强调多步答案推导，已在英语环境中取得一定成果。然而，宽度搜索作为一种全面枚举实体及其属性的能力，尚缺乏系统性评估工具，特别是在韩语等低资源语言环境中。韩语网页具有特殊的结构和术语，增加了信息检索的难度。现有韩语评测多偏重理解能力，缺少结构化信息的全集枚举能力评估。本文基于此背景，提出了面向韩语环境的宽度搜索基准，填补了这一空白。
Core Problem
当前Web代理在深度推理方面表现优异，但在实现对结构化信息的全面枚举方面存在明显不足，尤其是在多源、多页、多结构化场景中。韩语网页的特殊结构、术语和搜索习惯使得信息检索更具挑战性。缺乏系统化的宽度搜索能力评估基准，限制了模型的优化空间。如何设计一个既能模拟真实复杂场景，又能保证评估可信度的基准，是亟待解决的核心问题。
Innovation
本文的创新点包括：1）引入自动合成与验证流程，确保大规模金标准的可信性；2）利用结构参数（表宽和二维复合键）调节任务难度，生成多层次场景；3）开发类型感知的归一化比较器，解决格式差异带来的评分偏差；4）实现多源属性验证机制，确保答案的真实性。此套方法显著降低了手工标注成本，提升了评估的可扩展性和可靠性，为韩语环境下宽度搜索的研究提供了新范式。
Methodology
- �� 任务定义：每个任务命名一个实体（如电视季、朝代、选举），要求列出所有成员及其属性。
- �� 自动合成：利用模型生成候选答案，结合网页数据，自动构建金标准表格。
- �� 多源验证：包括非记忆性验证（模型不能从记忆中复现答案）、完整性验证（重新枚举成员确认集是否完整）和多源验证（属性值跨页面验证）。
- �� 结构调节：通过调节表宽和二维复合键，生成不同难度。
- �� 比较器设计：采用类型感知的归一化匹配，确保格式差异不影响评分。
- �� 评估：模型输出表格后，自动解析并评分，指标包括Item-F1、Row-F1和完整表格成功率。
Experiments
采用真实网页数据，涵盖190个韩语实体，16个类别，分三难度层级。评估20个模型，比较其在不同结构参数下的表现。指标包括集体成员识别（Item-F1）、逐行填充（Row-F1）和整体成功率。还进行模型类型（专用、开源、前沿）对比，验证基准的区分能力和实用性。实验结果显示，模型在成员识别方面表现优异（Item-F1达92.8%），但在属性填充方面差距明显（Row-F1 53.7%），特别在复杂任务中表现更差。
Results
所有模型在识别成员方面表现良好（Item-F1 92.8%），但在逐行属性填充方面表现较差（Row-F1 53.7%），说明模型能找到正确成员但难以完整填写所有属性。随着任务难度增加，准确率逐步下降，最强模型的完整表格成功率仅19.3%。自由文本回答失败率高，而标准答案（如日期、姓名）表现较好。这反映了多源、多页、多结构化场景中的挑战。
Applications
该基准可用于评估未来Web代理在多源信息整合、结构化数据全集枚举等方面的能力，推动知识图谱自动构建、智能问答系统的发展。也可作为模型训练目标，提升其在复杂环境中的表现。未来结合多模态信息，将增强模型对动态网页的适应性。
Limitations & Outlook
模型在开放文本单元的回答准确性不足，尤其在多源信息融合时存在信息丢失或误差。评估依赖静态网页快照，难以反映网页动态变化。高复杂度任务对计算资源要求较大，未来需优化流程以提升效率。
Plain Language Accessible to non-experts
想象你在一家大型工厂工作，工厂里有许多不同的机器，每台机器负责生产不同的零件。你需要找到所有生产某种零件的机器，并知道每台机器的工作时间、产量和负责人。这就像在网页上找信息，你要找到所有相关网页（机器），并从不同页面收集完整的资料（零件信息）。有些信息在同一页面，有些在不同页面，你得逐个访问，确保每个零件的所有细节都正确。这就像工厂的检查员，要确保每台机器都正常工作，所有零件都符合标准。这个过程需要不断查找、核对和确认，才能确保信息完整、准确，就像宽度搜索一样，全面、细致地把所有信息都找出来。
ELI14 Explained like you're 14
想象你在学校的图书馆里找资料，你想列出所有关于某个话题的书，比如关于“韩国的运动队”。你不仅要知道有哪些书，还要知道每本书的作者、出版年份、出版社和内容简介。有时候，这些信息散落在不同的书架上，你得走遍整个图书馆，把所有相关的书都找到，然后逐一核对信息是否正确。这个任务就像在网页上搜索信息，你要找到所有相关网页，收集每个网页上的资料，然后确保它们都正确无误。这个过程很难，因为信息分散、格式不同，还要保证没有遗漏。就像你要做一份完整的资料清单，确保每个细节都准确无误，才能算完成任务。
Glossary
Breadth Search (宽度搜索)
一种全面枚举结构化信息的方法，旨在从网页中提取所有相关数据，而非深度推理。技术上涉及多源、多页、多结构化数据的自动整合。
本文的核心评估任务，检验Web代理的全集枚举能力。
Automated Verification (自动验证)
利用模型和规则自动检查生成数据的完整性和正确性，确保大规模金标准的可信性。
保证评估数据的质量和可信度。
2D Composite Key (二维复合键)
由两个属性组成的联合主键，用于表示复杂的关系网格结构，提升任务难度。
调节任务结构复杂度的重要参数。
Normalization-aware Comparator (归一化比较器)
一种考虑格式差异的比较工具，用于确保不同格式的答案能公平评分。
解决答案格式多样性带来的评分偏差。
Cross-source Verification (多源验证)
从不同网页源头核实属性信息的一致性，确保答案的可靠性。
提升答案的真实性和完整性。
Open Questions Unanswered questions from this research
- 1 如何进一步提升模型在开放文本单元的回答准确率，特别是在多源信息融合场景中？
- 2 动态网页环境下信息检索的有效策略尚未充分研究。
Abstract
Web-agent benchmarks overwhelmingly measure depth -- pinning one obscure answer behind a chain of constraints -- while breadth, exhaustively enumerating a closed set and filling each item's attributes, is barely evaluated, especially outside English. Breadth is also hard to build: certifying that a gold set is complete and every cell correct is far costlier than checking a single answer. I introduce \textsc{Ko-WideSearch}, a Korean breadth-search benchmark built by an automated synthesize-and-verify pipeline. Each task names a set-parent entity -- a TV season, a dynasty, a league, an administrative region, an election -- and asks for its full membership plus a per-item attribute table, graded by Item-, Column-, and Row-F1. It spans 228 tables over 190 entities and sixteen categories across three difficulty tiers, set by two structural knobs I dial independently -- table width and a 2-D composite key -- so cross-product membership climbs from 0\% to 100\% across the tiers. A single normalization-aware comparator is shared between gold construction and grading, so stable date and count columns are not over-dropped on formatting alone. Across twenty web agents, the failure is consistent: agents recover the set but not the rows (e.g.\ Item-F1 92.8 against Row-F1 53.7), accuracy falls steadily as the knobs harden, and neither more search nor more spend closes the gap. Broken down by cell, the hard part is finding the right value, not formatting it: open-ended free-text cells fail most, while cells with a standard answer such as a date or a name usually come out right.