Benchmarking and Enhancing LLMs for Rule-Intensive Review of National Standard Documents
Introduces GB/T-Bench and GB/T-Reviewer, leveraging hierarchical error taxonomy and multi-agent collaboration to significantly improve rule-based national standard document review.
Key Findings
Methodology
This work constructs a hierarchical GB/T Review Taxonomy encompassing five dimensions—structure, scope, normative modality, terminology, and references—divided into 25 diagnosable error types. A combined deterministic rule and constrained LLM rewriting mechanism generates 7,306 traceable error instances from 488 documents, forming GB/T-Bench. An evaluation protocol based on exact matching of error location, review dimension, and error type ensures precise model assessment. To enhance review performance, a multi-agent GB/T-Reviewer framework decomposes review tasks into global inspection, targeted diagnosis, rule scanning, and verification modules, coordinating their outputs for robust error detection. Experiments on 14 mainstream LLMs reveal a significant performance gap with human experts, with the best model achieving only 0.3280 CMCS, while the multi-agent system reaches 0.5094, demonstrating the effectiveness of structured skill coordination.
Key Results
- Results show that current large models underperform significantly compared to human experts in rule-intensive document review, with the top CMCS being 0.3280 (GPT-5.6), versus 0.6640 for experts. Incorporating the multi-agent framework raises CMCS to 0.5094, substantially narrowing the gap. The models excel at detecting explicit structural and scope errors but lag in normative language, terminology consistency, and cross-section validation. Ablation studies confirm that the multi-agent collaboration notably improves detection accuracy, especially for high-risk error types, validating the design's robustness.
- Error localization and fine-grained classification remain challenging, with models often missing subtle normative errors or misclassifying similar error types. The results indicate that decomposing review knowledge into specialized modules and coordinating their outputs enhances diagnostic precision. The experiments also highlight the importance of error generation quality, as the controlled dataset ensures fair evaluation and meaningful insights into model capabilities.
- Overall, the findings underscore that while large models show promise, significant advancements are needed for trustworthy, automated rule-based document review. The multi-agent approach offers a scalable pathway to bridge the gap, with potential for broader application in high-stakes industries such as legal, medical, and financial sectors.
Significance
This research pioneers a comprehensive, hierarchical evaluation framework tailored for rule-intensive national standard documents, addressing a critical gap in existing benchmarks that focus mainly on knowledge understanding. By integrating structured error taxonomy, controllable data generation, and multi-agent collaboration, the study advances the state-of-the-art in trustworthy AI for professional document analysis. Its implications extend beyond standardization, offering a blueprint for automating complex regulatory and compliance tasks across industries. The demonstrated performance improvements validate the potential of structured skill decomposition and multi-agent systems to transform high-stakes document review workflows, reducing costs, increasing consistency, and enhancing transparency. As AI continues to mature, this framework provides a scalable, interpretable, and reliable foundation for deploying automated review tools in real-world settings.
Technical Contribution
The paper's key technical contributions include the development of a hierarchical GB/T Review Taxonomy with 25 error types, enabling fine-grained error classification aligned with GB/T standards. It introduces a controllable error generation pipeline combining deterministic rules and constrained LLM rewriting, ensuring high-quality, traceable error samples for evaluation. The core innovation lies in the multi-agent GB/T-Reviewer framework, which decomposes review tasks into specialized modules—global inspection, dimension-specific agents, error-type agents, and rule scanners—coordinated via information fusion. This modular design overcomes the limitations of monolithic models in rule comprehension and detailed localization, providing a scalable, interpretable architecture with improved diagnostic accuracy. The evaluation on 14 models demonstrates that structured skill decomposition significantly boosts performance, setting new benchmarks for rule-based document review.
Novelty
This work is the first to establish a hierarchical, rule-based error taxonomy specifically tailored for national standard documents, coupled with a controllable error generation mechanism for creating high-quality evaluation datasets. The integration of multi-agent collaboration to decompose complex review tasks into specialized modules is a novel architectural approach, addressing the core challenge of fine-grained error localization and classification in rule-intensive scenarios. Unlike prior benchmarks focused on reasoning or knowledge understanding, this framework emphasizes diagnostic precision, interpretability, and scalability in professional document review, marking a significant leap forward in trustworthy AI for high-stakes applications.
Limitations
- Despite the improvements, the multi-agent system still faces challenges in handling highly ambiguous or context-dependent errors, especially in documents with complex cross-references or fuzzy language. The reliance on rule-based error injection may limit the diversity of training samples, affecting robustness in real-world scenarios.
- The dataset construction, while carefully controlled, may not fully capture the complexity and variability of real-world errors, necessitating future efforts to incorporate more authentic error cases and expand the dataset's coverage.
- Computational costs remain high when processing very long documents or deploying multiple specialized agents simultaneously, which could hinder scalability in large-scale industrial applications. Future work should focus on optimizing system efficiency and exploring adaptive agent collaboration strategies.
AI Executive Summary
In the era of rapid digital transformation, the quality assurance of national standard documents is more critical than ever. These documents serve as the backbone of industry regulation, product quality, and public safety, yet their review process remains predominantly manual, labor-intensive, and costly. Traditional methods rely heavily on human experts who meticulously examine lengthy, highly structured texts for compliance, consistency, and correctness. This approach, while accurate, is slow and difficult to scale, especially given the exponential growth in the number of standards and updates.
Recent advances in large language models (LLMs) like GPT-5, Claude, and Qwen have demonstrated remarkable capabilities in natural language understanding, reasoning, and knowledge application. However, their performance in specialized, rule-intensive tasks such as standard document review has not been thoroughly evaluated. Existing benchmarks mainly focus on knowledge comprehension, question answering, or regulatory compliance detection, leaving a significant gap in assessing models' ability to perform detailed, rule-based inspections.
To address this gap, the authors introduce GB/T-Bench, a comprehensive benchmark tailored for the structured review of China’s GB/T standards. This benchmark is built upon a hierarchical review taxonomy that categorizes errors into five core dimensions—structure, scope, normative modality, terminology, and references—and further subdivides into 25 specific error types. Using a combination of deterministic rules and constrained LLM rewriting, the authors generated a large, traceable dataset of over 7,300 error instances across 488 documents, ensuring high-quality, controllable evaluation samples.
Complementing the dataset, the authors propose GB/T-Reviewer, a multi-agent framework that decomposes the complex review task into specialized modules—global inspection, dimension-specific agents, error-type agents, and rule scanners—coordinated through information fusion. This architecture mimics expert review workflows, enabling more precise error localization, classification, and cross-section verification. Extensive experiments on 14 mainstream LLMs reveal that, despite significant progress, models still lag behind human experts, with the best single model achieving only 0.3280 CMCS, whereas experts reach 0.6640. Notably, the multi-agent system improves the CMCS to 0.5094, demonstrating the value of structured skill decomposition.
This work marks a substantial step toward trustworthy AI in high-stakes document review, providing a systematic evaluation framework and a scalable, interpretable architecture. It opens new avenues for automating regulatory compliance, standardization, and quality assurance processes across industries, ultimately reducing costs, increasing consistency, and enhancing transparency. Future research will focus on expanding dataset diversity, integrating multimodal data, and optimizing system efficiency to realize fully autonomous, reliable standard document review in real-world applications.
Deep Analysis
Background
Standard documents, especially national standards like China’s GB/T series, are essential for ensuring product quality, safety, and interoperability across industries. Traditionally, their review process relies on domain experts meticulously checking lengthy texts for structural correctness, terminological consistency, scope adherence, and reference validity. This manual process is time-consuming, costly, and difficult to scale with the increasing volume of standards. Recent developments in large language models (LLMs) have shown promise in automating various NLP tasks, including legal and regulatory document understanding. However, their application to rule-based, structured review remains underexplored. Existing benchmarks such as LegalBench, LexGLUE, and CUAD focus on reasoning or question answering but lack the granularity needed for detailed error localization and classification in standards. The complexity of standards—涉及多层次结构、交叉引用、规范性语言—poses significant challenges for models. Therefore, establishing a specialized, hierarchical evaluation framework aligned with GB/T drafting and审查实践 is crucial to advance trustworthy AI in this domain.
Core Problem
The core challenge addressed by this research is how to systematically evaluate and improve large language models’ ability to perform rule-intensive, structured review of national standard documents. Existing models excel at general language understanding but struggle with precise error localization, classification, and cross-section verification required in standards review. The lack of a tailored benchmark hampers meaningful assessment and targeted improvement. Moreover, the complexity of standards—lengthy, highly structured, governed by explicit rules—demands models to understand nuanced normative language, semantic consistency, and detailed references. Manual review remains costly and slow, creating a bottleneck for standardization processes. The key problem is designing a comprehensive, controllable evaluation dataset and a robust, scalable framework that can decompose review tasks into manageable, specialized skills, mimicking expert workflows and enabling targeted model enhancements.
Innovation
This work introduces several key innovations. First, it develops a hierarchical GB/T Review Taxonomy that categorizes errors into five core dimensions—structure, scope, normative modality, terminology, and references—and further refines into 25 error types, providing a detailed diagnostic framework. Second, it proposes a controllable error generation pipeline combining deterministic rule-based modifications and constrained LLM rewriting, ensuring high-quality, traceable error samples for evaluation. Third, it constructs GB/T-Bench, a large, expert-validated dataset of 7,306 errors across 488 documents, supporting fine-grained assessment. Fourth, it designs GB/T-Reviewer, a multi-agent architecture that decomposes the review process into specialized modules—global inspection, dimension-specific agents, error-type agents, and rule scanners—coordinated via information fusion, to mimic expert review workflows. This modular, collaborative approach overcomes the limitations of monolithic models in rule understanding and detailed localization, significantly improving diagnostic accuracy and interpretability. Overall, the framework advances the state-of-the-art in automated, trustworthy document review.
Methodology
- �� 构建GB/T审查分类体系：基于GB/T标准草案和专家经验，定义五个核心维度（结构、范围、规范性、术语、引用）及其25个细粒度错误类型。
- �� 设计错误生成机制：结合规则注入和受控大模型重写，生成7306个可追溯的错误实例，确保数据多样性和可控性。
- �� 构建数据集：涵盖488份标准文档，包含64,991个章节，经过专家质量控制，确保标注准确。
- �� 设计评估协议：采用误差位置、维度和类型的精确匹配，结合文档级覆盖指标，全面衡量模型性能。
- �� 构建多智能体架构：将审查任务拆分为全局检测、维度专家、错误类型专家和规则扫描模块，通过信息融合实现高效诊断。
- �� 实验验证：在14个主流大模型上进行评估，分析其在不同错误类别和复杂场景中的表现差异。
Experiments
实验采用作者构建的GB/T-Bench数据集，包含7306个错误实例，覆盖五个维度和25类错误类型。评估对象包括14个主流大模型（如GPT-5.6、Claude、Qwen等），采用精确诊断匹配指标（CMCS）和诊断召回（DMTR）衡量性能。模型在不同任务设置（单模型、多智能体）下进行对比，分析其在结构、范围、规范性、术语和引用错误检测中的表现差异。实验还包括消融分析，验证多智能体架构对性能提升的贡献。模型输入标准文档，输出错误位置、类型和描述，自动比对标注，统计准确率和召回率。结果显示，多智能体架构显著优于单模型，最高CMCS达0.5094，验证了设计的有效性。
Results
实验表明，当前模型在规则密集型标准文档审查中，表现仍远低于人类专家。最高CMCS为0.3280（GPT-5.6），而专家达0.6640。引入多智能体后，CMCS提升至0.5094，缩小差距。模型在结构和范围错误检测较好，但在规范性用语、术语一致性和交叉验证方面仍有不足。消融分析显示，多智能体架构显著提升高风险错误类别的检测能力，验证了其有效性。这些结果为未来高规范场景的自动化审查提供了技术基础。
Applications
该框架可应用于国家标准、行业规范、法规合规等场景，帮助行业降低人力成本、提升效率。具体包括：自动检测标准中的结构偏差、术语不一致、引用错误等，适用于标准制定、质量控制和合规审查。未来，结合行业特定知识和多模态信息，有望实现全流程自动化审查，提升行业整体规范化水平。该技术还可推广到医疗、金融等高风险行业，推动可信AI的行业落地。
Limitations & Outlook
尽管多智能体架构提升了模型性能，但在极端复杂或模糊表达的标准场景中仍存在误检漏检问题，特别是在交叉章节和模糊表达中表现不足。数据生成依赖规则注入和受控重写，可能存在偏差，难以完全模拟真实错误场景，未来应引入更多真实错误样本以增强鲁棒性。模型在处理超长文档和多层次交叉验证时，计算成本较高，存在信息遗失和效率瓶颈。未来需优化模型结构和推理策略，以满足实际应用需求。
Plain Language Accessible to non-experts
想象你在一家工厂工作，工厂的任务是生产一份完美的产品——就像标准文件一样。每个工人（模型）都要按照严格的规则检查每个环节，确保没有瑕疵。传统上，这个检查由经验丰富的工人（专家）完成，既费时又费力。现在，工厂引入了一套智能机器人（大模型），它可以帮忙检查，但机器人还不够聪明，偶尔会漏掉一些瑕疵。为了让机器人变得更聪明，工厂设计了一套“多机器人合作”系统，每个机器人专门负责某一方面，比如结构、术语或引用。它们相互配合，确保每个环节都符合标准。经过不断训练和优化，这个系统变得越来越可靠，能大大提高检查速度和准确性。这个比喻说明了本文提出的多智能体协作框架如何让自动化标准审查变得更高效、更可信，就像工厂里的机器人合作一样。
ELI14 Explained like you're 14
想象你在学校里要检查一份非常复杂的作业，里面有很多不同的部分，比如语法、内容、格式和引用。以前，老师一个人要花很多时间逐个检查，既累又容易出错。现在，你的朋友们（就像不同的机器人）帮忙检查：一个专门看语法，一个负责内容，一个关注格式。他们一起合作，把作业检查得更快、更准。这就像论文里的多智能体系统，每个“机器人”都擅长某一方面，然后合作完成整个任务。虽然他们还不是完美的，但比以前一个人盯着看要好多了。这个故事告诉我们，利用多个“助手”合作，可以让复杂的任务变得简单又高效，就像在学校里分工合作一样。
Glossary
GB/T Standard (GB/T标准)
中国国家标准的规范文件，规定了行业技术要求和审查规则，作为本研究的基础。
用于定义审查分类体系和错误类型。
CMCS (Complete Model Coverage Score)
衡量模型在错误诊断中覆盖全部正确错误的比例，反映模型的全面诊断能力。
作为评估模型性能的核心指标之一。
GB/T Review Taxonomy (GB/T审查分类体系)
基于GB/T标准制定的五维层级化错误分类体系，用于细粒度错误诊断。
指导错误生成和模型评估。
Multi-Agent Framework (多智能体架构)
由多个具有专业技能的模块协同工作，提升复杂任务中的整体性能。
用于标准文档的规则审查。
Controlled Rewriting (受控重写)
在生成错误样本时，模型在规则限制下对文本进行重写，确保错误的可控性。
用于数据增强和错误生成。
Error Type (错误类型)
标准化文档中可能出现的具体错误类别，如结构错误、术语不一致等。
细化模型诊断目标。
Error Location (错误位置)
错误在文档中的具体章节或段落位置，用于精确定位。
模型输出的诊断结果之一。
Diagnosis Protocol (诊断协议)
基于误差位置、维度和类型的匹配规则，用于模型性能评估。
确保评估的公平性和一致性。
Rule Scanning (规则扫描)
利用预定义规则检测文档中的潜在错误。
作为多智能体中的关键模块。
Error Instance (错误实例)
由错误生成机制产生的带有明确标签的样本，用于模型训练和评估。
保证数据的可追溯性和多样性。
Long-Document Understanding (长文本理解)
模型对超长文档中信息的理解和推理能力。
在标准文档审查中尤为重要。
Cross-Section Verification (交叉验证)
验证不同章节或部分之间的一致性。
提升诊断的细粒度和准确性。
Error Localization (错误定位)
在文档中准确找到错误的具体位置。
模型输出的关键指标。
Fine-Grained Diagnosis (细粒度诊断)
对错误进行详细分类和描述。
提升模型的解释能力。
Expert Review (专家评审)
由行业专家进行的人工标准审查，作为模型性能的基准。
用于验证模型的诊断效果。
Abstract
Large language models (LLMs) increasingly support complex professional tasks, yet their capabilities in rule-intensive document review remain insufficiently evaluated. National standard documents, such as China GB/T standards, offer a representative testbed: they are lengthy, highly structured, and governed by explicit rules for scope, terminology, normative wording, and cross-section consistency. Existing benchmarks focus on domain knowledge and question answering, largely overlooking intrinsic quality review for professional documents. Such reviews rely heavily on human experts, making them costly and difficult to scale. To bridge this gap, we introduce GB/T-Bench, the first benchmark for the structured review of national standard documents. Its GB/T Review Taxonomy is a hierarchical schema covering document structure, scope alignment, normative modality, terminology consistency, and normative references, with 25 diagnosable error types. A controllable counterexample generation mechanism combines deterministic rules and constrained LLM rewriting to process 488 documents into 7,306 traceable review error instances for evaluation. We also develop a diagnosis-oriented evaluation protocol requiring exact matches on error location, review dimension, and error type, plus document-level coverage metrics. We further propose GB/T-Reviewer, a multi-agent framework that converts review knowledge into specialized skills and coordinates global inspection, targeted diagnosis, rule scanning, and result verification. Experiments with 14 mainstream LLMs reveal a substantial human-LLM gap: the strongest model achieves only 0.3280 CMCS versus 0.6640 for experts. GB/T-Reviewer raises the best CMCS to 0.5094, showing the value of structured skill coordination for rule-intensive document review. This work paves the way for trustworthy AI in standardization and other high-stakes document domains.
References (20)
TaskWeaver: A Code-First Agent Framework
Bo Qiao, Liqun Li, Xu Zhang et al.
Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond
Jinze Bai, Shuai Bai, Shusheng Yang et al.
HSKBenchmark: Modeling and Benchmarking Chinese Second Language Acquisition in Large Language Models through Curriculum Tuning
Qihao Yang, Xuelin Wang, Jiale Chen et al.
CODE-ACCORD: A Corpus of building regulatory data for rule generation towards automatic compliance checking
Hansi Hettiarachchi, Amna Dridi, Mohamed Medhat Gaber et al.
CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquing
Zhibin Gou, Zhihong Shao, Yeyun Gong et al.
LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding
Yushi Bai, Xin Lv, Jiajie Zhang et al.
The MiniMax-M2 Series: Mini Activations Unleashing Max Real-World Intelligence
MiniMax Aili Chen, Aonian Li, Baichuan Zhou et al.
Evaluation and Benchmarking of LLM Agents: A Survey
Mahmoud Mohammadi, Yipeng Li, Jean-Pierre Lo et al.
MAUD: An Expert-Annotated Legal NLP Dataset for Merger Agreement Understanding
Steven H. Wang, Antoine Scardigli, Leonard Tang et al.
Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution
Peng Wang, Shuai Bai, Sinan Tan et al.
AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversation
Qingyun Wu, Gagan Bansal, Jieyu Zhang et al.
ReAct: Synergizing Reasoning and Acting in Language Models
Shunyu Yao, Jeffrey Zhao, Dian Yu et al.
FinMME: Benchmark Dataset for Financial Multi-Modal Reasoning Evaluation
Junyu Luo, Zhizhuo Kou, Liming Yang et al.
SPaR.txt, a Cheap Shallow Parsing Approach for Regulatory Texts
Ruben Kruiper, Ioannis Konstas, A. Gray et al.
CUAD: An Expert-Annotated NLP Dataset for Legal Contract Review
Dan Hendrycks, Collin Burns, Anya Chen et al.
Qwen-Image Technical Report
Chenfei Wu, Jiahao Li, Jingren Zhou et al.
Toolformer: Language Models Can Teach Themselves to Use Tools
Timo Schick, Jane Dwivedi-Yu, Roberto Dessì et al.
PolicyQA: A Reading Comprehension Dataset for Privacy Policies
Wasi Uddin Ahmad, Jianfeng Chi, Yuan Tian et al.
MMLongBench-Doc: Benchmarking Long-context Document Understanding with Visualizations
Yubo Ma, Yuhang Zang, Liangyu Chen et al.
LLM-based Collaborative Agents with Pedagogy-guided Interaction Modeling for Timely Instructive Feedback Generation in Task-oriented Group Discussions
Qihao Yang, Yu Yang, Sixu An et al.