CorporateBench: Large-Scale Q&A Benchmarking with Temporal Knowledge Bases
CorporateBench is a large-scale, temporally evolving knowledge base benchmark for multi-task enterprise QA, with over 230,000 documents.
Key Findings
Methodology
This study constructs a procedural, temporally evolving knowledge base simulating corporate environments with organizational hierarchies, tasks, and communication. Over 230,000 documents are generated, ensuring logical consistency across the dataset. Five models, including GPT-5.1 and Haiku 4.5, are evaluated on information extraction and knowledge base querying tasks using metrics like F1 and exact match. The evaluation reveals performance degradation at larger scales, especially in relation extraction and complex reasoning, highlighting current model limitations in enterprise scenarios.
Key Results
- Entity extraction remains relatively stable (F1: 0.7-0.8), but relation extraction drops significantly (F1: 0.2-0.4) at XL scale, indicating difficulty in maintaining relational consistency across large datasets.
- Knowledge base question answering (KB QA) outperforms retrieval-based methods (RAG), with top F1 scores around 0.75, but performance declines sharply as data size increases.
- Multi-task evaluation shows models struggle with long documents, revealing a performance gap in real-world enterprise-scale reasoning, emphasizing the need for more robust models.
Significance
This benchmark addresses a critical gap in enterprise AI evaluation by providing a realistic, large-scale, and logically consistent dataset. It enables rigorous assessment of models' abilities to handle complex, dynamic, multi-document scenarios, facilitating progress towards practical enterprise AI solutions. The framework bridges the gap between synthetic and real-world data, supporting industry adoption and academic research in scalable knowledge reasoning.
Technical Contribution
The paper introduces a novel framework combining procedural generation and temporal knowledge graphs to produce large, coherent enterprise datasets. It designs multi-task evaluation tasks, including information extraction, KB question answering, and topic classification, with precise ground truth annotations. This scalable, logic-consistent dataset enables benchmarking of models' long-text understanding and multi-document reasoning at unprecedented scale, revealing current limitations and guiding future improvements.
Novelty
This is the first work to integrate time-evolving knowledge graphs with procedural data generation for large-scale enterprise simulation. Unlike prior synthetic benchmarks, CB ensures logical consistency and realism at multi-million document scale, providing a controllable, verifiable environment for comprehensive model evaluation in enterprise contexts.
Limitations
- Models exhibit significant difficulty in relation extraction and maintaining relational consistency at scale, especially with complex, multi-relational graphs.
- While the synthetic data mimics enterprise environments, it may not fully capture industry-specific nuances and heterogeneity.
- The evaluation focuses on extraction and reasoning tasks, with limited assessment of robustness, efficiency, and deployment readiness in real-world settings.
Future Work
Future directions include integrating multimodal data (images, speech), developing more scalable and robust long-context models, and collecting real enterprise data under privacy constraints. Extending the benchmark to include more diverse industries and real-world scenarios will further enhance its applicability and impact.
AI Executive Summary
In the era of enterprise digital transformation, vast amounts of internal communication and documentation form the backbone of organizational knowledge. However, evaluating AI models capable of understanding and reasoning over such large, complex, and dynamic datasets remains a significant challenge. Existing benchmarks are limited by scale, realism, or scope, often relying on synthetic or small datasets that do not reflect real-world enterprise conditions.
To address this gap, the authors introduce CorporateBench (CB), a large-scale, temporally evolving knowledge base benchmark designed specifically for enterprise multi-task question answering. CB is built upon a procedural, knowledge graph-driven framework that simulates diverse companies across industries and scales, from small startups to multinational corporations. The dataset contains over 230,000 documents, including emails, reports, and meeting records, all generated to maintain logical consistency and temporal coherence. This setup allows models to be evaluated on their ability to perform information extraction, knowledge base querying, and topic classification, reflecting real-world enterprise challenges.
The evaluation of five state-of-the-art models reveals critical insights: while entity extraction remains relatively robust, relation extraction and complex reasoning degrade sharply at larger scales. For instance, relation F1 scores drop from around 0.8 in small datasets to below 0.3 in XL-scale datasets, exposing the limitations of current models in maintaining relational integrity across vast, interconnected data. KB question answering outperforms retrieval-based methods but still faces performance drops as data size increases, highlighting the need for more scalable and robust architectures.
This benchmark offers a vital tool for academia and industry, enabling rigorous testing and development of AI systems tailored for enterprise environments. It bridges the gap between synthetic and real data, providing a controlled yet realistic platform for long-text understanding and multi-document reasoning. Future work will focus on multimodal integration, improving model scalability, and expanding to real-world datasets, aiming to realize truly intelligent enterprise AI systems capable of managing complex, dynamic organizational knowledge at scale.
Deep Analysis
Background
The evolution of enterprise AI has seen significant advances in natural language processing, with models like BERT, GPT series, and specialized knowledge graph techniques. Prior works include knowledge graph construction (Google Knowledge Graph), enterprise QA benchmarks (e.g., LegalBench, FinancialQA), and synthetic data generation methods. Despite progress, these efforts often focus on small-scale or static datasets, lacking the realism and scale needed for practical deployment. Recent trends emphasize dynamic, time-sensitive knowledge modeling, yet scalable benchmarks remain scarce due to data privacy and generation challenges. This work builds upon these foundations, leveraging procedural generation and temporal knowledge graphs to create large, realistic datasets that simulate enterprise environments over time.
Core Problem
Current AI models struggle with long, multi-document reasoning tasks typical in enterprise settings. Existing benchmarks are limited in scale or realism, often failing to capture the complexity of real organizational data. The core challenge lies in generating large, logically consistent datasets that reflect the dynamic, multi-relational nature of corporate environments. Without such benchmarks, progress in scalable, reliable enterprise AI remains hindered. Moreover, models exhibit performance drops at scale, especially in relation extraction and reasoning over evolving data, underscoring the need for more representative evaluation platforms.
Innovation
This work introduces several innovations: 1) a procedural, time-evolving knowledge graph framework for large-scale enterprise data simulation; 2) multi-task evaluation covering information extraction, KB querying, and topic classification; 3) generation of over 230,000 documents with guaranteed logical consistency and temporal coherence. These innovations enable realistic, scalable benchmarking, addressing the limitations of prior synthetic datasets and small-scale real data. The approach allows for controlled, verifiable evaluation of models' long-text understanding and reasoning capabilities in enterprise contexts, a significant step forward in the field.
Methodology
- �� Construct a formal ontology representing enterprise entities, relationships, and roles, using RDF triples.
- �� Sample organizational hierarchies, employee roles, projects, and tasks from the knowledge graph, ensuring temporal continuity.
- �� Simulate company operations over a quarter: assign tasks, schedule meetings, hire/fire employees, and update hierarchies.
- �� Generate textual documents (emails, reports, meeting notes) by templating from the knowledge graph, inserting evidence strings to substantiate relationships.
- �� Sample documents to include relationship evidence, ensuring logical consistency and verifiability.
- �� Annotate entities with properties like names, titles, and aliases, using recursive prompting.
- �� Generate ground truth triples for entity and relation extraction, and formulate questions for QA tasks.
- �� Evaluate models on these tasks, analyzing performance degradation with increasing data size.
Experiments
Four synthetic companies across different sizes (from 12 to 10,000 employees) were generated, each with corresponding document corpora. Models like GPT-5.1, Haiku 4.5, and Gemini 2.5 were tested on extraction and QA tasks, with metrics including F1 and accuracy. Validation involved human annotation and statistical analysis of network properties, topic distributions, and document coherence. The experiments demonstrated that entity extraction remains stable, but relation extraction and complex reasoning significantly decline at larger scales, exposing current model limitations in enterprise scenarios.
Results
Entity extraction maintained high F1 scores (~0.75-0.82), but relation extraction performance dropped sharply in XL datasets (F1 ~0.2-0.4). KB QA outperformed retrieval-based approaches, with scores around 0.75, yet performance deteriorated with larger data. Topic classification F1 scores reached 0.95 in smaller datasets but declined at larger scales. These results highlight the gap between current model capabilities and the demands of real-world enterprise data, emphasizing the need for more scalable, relationally aware models.
Applications
CB can be used for developing enterprise knowledge management systems, automating internal question answering, and training employees on organizational structures. It provides a benchmark for testing model robustness, scalability, and reasoning in realistic settings. Long-term, it supports the creation of intelligent enterprise AI capable of managing complex, dynamic organizational knowledge, enabling smarter decision-making and operational efficiency.
Limitations & Outlook
Models show significant performance drops in relation extraction and relational consistency at large scales, indicating current architectures lack robustness for complex, multi-relational data. The synthetic data, while realistic, cannot fully replicate industry-specific nuances. Additionally, the evaluation focuses on extraction and reasoning, with limited assessment of deployment efficiency, robustness under noisy data, and adaptability to real enterprise environments. Future work should address these gaps to enhance practical applicability.
Plain Language Accessible to non-experts
想象一个大公司就像一个超级复杂的学校，有许多老师、学生、课程和活动。每天都在发生很多事情：有人发邮件、开会、完成任务。每个人都在不同的班级、部门工作，关系错综复杂。科学家们用一种叫“知识库”的超级智能系统，把所有这些信息都存起来，就像学校的档案馆一样。为了让电脑学会理解这些信息，他们用程序模拟出一个虚拟的学校，里面有虚拟的老师和学生，模拟他们的关系和活动。然后，电脑可以在这个虚拟学校里练习，学习怎么找到老师的联系方式、谁在开会、谁在负责什么。这样，当真正的公司需要问问题时，电脑就能像老师一样帮忙，快速找到答案。这种模拟让电脑变得更聪明，也让我们知道它在处理复杂信息时的能力。
ELI14 Explained like you're 14
想象你在一个超级大的学校里，里面有很多班级、老师和学生，每天都在发生各种事情。有的老师发邮件通知学生，有的学生在讨论作业，有的老师在开会。现在，科学家们想让电脑帮忙理解这些复杂的学校生活，帮忙回答问题，比如“谁在上个星期负责了数学课？”或者“哪个学生参加了学校的科学比赛？”但问题是，这些信息太多，电脑很难一下子全部看懂，就像你在一堆书里找一篇重要的文章一样困难。于是，科学家们用一种特殊的方法，模拟出一个虚拟的学校，里面的老师、学生和活动都按照一定的规则生成，既真实又大规模。这样，电脑可以在这个虚拟学校里练习，学习怎么找到需要的信息。最后，电脑变得更聪明，能帮忙解答实际学校或企业中的复杂问题，就像一个超级聪明的助手一样！
Abstract
LLMs are increasingly able to answer complex questions about enterprise-scale document collections. But evaluation is hard: companies don't want to share internal communications, and synthetic datasets have been overly simple. We present CorporateBench (CB), a human-validated multi-task Q&A benchmark whose scale approaches the conditions LLMs encounter in corporate communication networks, with evaluation corpora surpassing 230,000 documents. CB evaluates LLMs across two dimensions (information extraction and knowledge base querying) through four synthetically generated firms ranging from 12 to 10,000 employees. Each corpus is sampled from a temporally evolving knowledge base describing a consistent world, guaranteeing cross-document logical consistency even across hundreds of thousands of documents. We evaluate five LLMs on CB, revealing increasingly poor performance as input size approaches realistic scales. CB provides LLM developers a metric for corporate communication reasoning, filling a crucial gap in the benchmarking ecosystem.