T2R-bench: A Benchmark for Generating Article-Level Reports from Real World Industrial Tables

Industrial data analysis has long relied on manual processes, which are inefficient and prone to errors. Recent advances in deep learning, especially large language models like GPT-4, LLaMA, and specialized table reasoning models, have shown promise in automating NLP tasks. However, these models are primarily evaluated on academic datasets such as WikiSQL, TabFact, and ToTTo, which lack the complexity and scale of real industrial tables. Industrial tables often feature multi-table relationships, hierarchical headers, and massive data volumes, posing significant challenges for current models. Despite some industry tools like Power BI and SAP integrating automation, there is no standardized benchmark to evaluate model performance comprehensively in real-world scenarios. This gap limits progress and hinders the deployment of AI solutions that can handle the intricacies of industrial data, emphasizing the need for realistic, large-scale benchmarks that reflect actual operational environments.

Existing models struggle to accurately interpret and generate reports from complex industrial tables due to structural diversity, large data sizes, and multi-table dependencies. The lack of a standardized, comprehensive benchmark hampers meaningful evaluation and comparison. Consequently, models often produce reports with numerical inaccuracies, incomplete information, or incoherent narratives, which are unsuitable for industrial decision-making. Addressing these issues requires developing datasets that mirror real-world complexities and establishing evaluation metrics that capture multiple aspects of report quality, including factual correctness, completeness, and readability. Without such benchmarks, progress in industrial table reasoning remains limited, impeding the deployment of AI-driven automation in enterprise settings.

This research introduces several key innovations: 1) A large-scale, real-world industrial dataset covering diverse sectors and table types, including extremely large and multi-table structures; 2) A multi-criteria evaluation system combining numerical accuracy, information coverage, and report coherence, validated against human judgments; 3) A semi-automatic annotation pipeline that ensures high-quality question and keypoint labeling, reflecting real industrial tasks; 4) Emphasis on multi-modal and multi-structure data, pushing models beyond academic benchmarks. These innovations enable a more realistic assessment of model capabilities, fostering development of practical solutions for industrial automation.

• �� Data collection: sourced from public industrial platforms, statistical bureaus, and open datasets, focusing on diversity in size, structure, and domain.
• �� Question annotation: generated via seed questions, prompt templates, and self-instruct methods, refined through expert filtering to ensure relevance and answerability.
• �� Report keypoint annotation: multiple models generate reports, from which core information is extracted and verified by human annotators.
• �� Evaluation design: developed metrics for numerical consistency (NAC), semantic coverage (ICC), and overall report quality (GEC), validated through human comparison.
• �� Experiments: tested 25 models across different table types, analyzing performance gaps and guiding future improvements.

The dataset includes over 8.3% extremely large tables (>50K cells), 28.9% complex structured tables, and 23.6% multi-table scenarios. Models evaluated include open-source (Qwen, LLaMA, DeepSeek) and proprietary (GPT-4, Claude). Metrics encompass numerical accuracy, semantic coverage, and holistic quality scores. Experiments involve ablation studies on table complexity, multi-table reasoning, and data scale, with detailed analysis of model strengths and weaknesses. Results highlight the persistent gap between current models and industrial requirements, especially in complex, large-scale, multi-table contexts, emphasizing the need for specialized architectures.

Deepseek-R1 achieves a top overall score of 62.71%, with notable weaknesses in numerical fidelity and information completeness. Models perform better on simpler tables but falter on multi-table and large-scale data, with scores dropping below 50% in some scenarios. Ablation studies reveal that incorporating structural reasoning modules and multi-modal inputs can improve performance by approximately 10%. The results demonstrate that current models lack the robustness needed for industrial deployment, underscoring the importance of dataset realism and multi-criteria evaluation for guiding future research.

The benchmark supports development of industrial AI tools for automatic report generation in finance, manufacturing, healthcare, and logistics. It enables enterprises to automate data analysis, reduce manual labor, and improve decision accuracy. Additionally, it can serve as a testing ground for new model architectures tailored to complex, large-scale data, accelerating AI adoption in enterprise workflows. Long-term, such systems could evolve into fully autonomous industrial data analysts, integrating multi-modal inputs and real-time reasoning to support continuous operational improvements.

Models still struggle with extremely large and structurally complex tables, especially in multi-table reasoning and high-dimensional inference. The evaluation metrics, while comprehensive, may not fully capture subjective report quality aspects like style and coherence. Data sources are primarily public, which may introduce biases; proprietary or more diverse datasets are needed for better generalization. Computational costs for training and inference remain high, posing challenges for real-time deployment. Future work should focus on scalable architectures, improved evaluation methods, and broader data collection to address these limitations.