Encyclo-K: Evaluating LLMs with Dynamically Composed Knowledge Statements
Proposes Encyclo-K, a knowledge statement-based dynamic benchmark, achieving contamination resistance and multi-knowledge assessment, with top model accuracy at 62.07%.
Key Findings
Methodology
Encyclo-K extracts standalone knowledge statements from authoritative textbooks, then employs random sampling at test time to assemble multi-statement questions. Each question contains 8-10 statements, with options combining 2-4 statements. Incorrect statements are generated via models like DeepSeek, ensuring high deception. Random seeds produce multiple question sets, maintaining stable model rankings. The approach effectively avoids data contamination and assesses comprehensive understanding across disciplines.
Key Results
- Even the strongest GPT-5.1 achieves only 62.07% accuracy, indicating high evaluation difficulty. Performance varies significantly across models, with reasoning models from 16.04% to 62.07%, and chat models from 9.71% to 50.40%, demonstrating the challenge of multi-statement comprehension. The stability of rankings across different question sets confirms the robustness of the sampling method.
- Models perform over 20 percentage points worse on multi-statement questions compared to single-concept ones, highlighting the difficulty of cross-knowledge understanding. The broad performance distribution enables fine-grained capability differentiation.
- The stable ranking and high difficulty validate the effectiveness of dynamic, multi-statement evaluation in measuring deep knowledge integration.
Significance
This work introduces a novel, contamination-resistant framework for large language model evaluation, emphasizing multi-knowledge comprehension. It addresses critical limitations of static benchmarks, enabling scalable, reliable, and comprehensive assessment of models’ knowledge integration and reasoning abilities, thus advancing both academic research and practical deployment.
Technical Contribution
The key innovation is treating knowledge statements as atomic units, combined with random sampling to generate diverse, dynamic questions. This approach ensures evaluation robustness against data contamination and enhances multi-knowledge understanding. The multi-statement fusion mechanism and multi-seed sampling provide a new paradigm for large-scale, scalable model assessment, with low annotation costs due to automated format verification.
Novelty
This is the first framework to leverage knowledge statements as the core unit for dynamic, combinatorial question generation, effectively addressing data contamination and multi-knowledge comprehension challenges. Unlike traditional static benchmarks, it introduces a scalable, adaptive evaluation paradigm that better reflects real-world knowledge integration demands.
Limitations
- While effective, the current approach relies on textbook-based knowledge sources, which may not cover emerging or highly specialized fields. Additionally, the model performance gap indicates room for improvement in complex reasoning and multi-modal understanding. Computational costs for large-scale sampling and question generation are non-trivial, necessitating further optimization.
Future Work
Future directions include expanding knowledge sources to include scientific papers and multimedia content, developing more sophisticated question templates for complex reasoning, and integrating multi-modal data. Additionally, exploring continual learning and adaptation in models based on dynamic evaluation feedback will be crucial for pushing the boundaries of knowledge understanding.
AI Executive Summary
As large language models (LLMs) continue to evolve rapidly, evaluating their true understanding and reasoning capabilities remains a fundamental challenge. Traditional benchmarks like SuperGLUE and MMLU, though influential, suffer from critical limitations such as data contamination, single-knowledge-point focus, and high annotation costs. These issues hinder fair, comprehensive assessment, especially as models grow larger and more capable.
To address these shortcomings, this paper introduces Encyclo-K, a novel evaluation framework based on knowledge statements extracted from authoritative textbooks. Unlike question-centric benchmarks, Encyclo-K treats knowledge statements as atomic units, which are randomly sampled and combined at test time to generate diverse, multi-statement questions. This dynamic approach ensures that models cannot memorize specific questions, effectively resisting data contamination. Each question, comprising 8-10 statements, challenges models to perform joint comprehension, pushing beyond simple recall to deeper integration.
Extensive experiments across more than 50 models reveal that even state-of-the-art models like GPT-5.1 achieve only 62.07% accuracy, underscoring the difficulty of the task. Performance varies widely across disciplines and model types, with reasoning models outperforming chat models significantly. The stability of model rankings across multiple random question sets confirms the robustness of the evaluation method. Importantly, models show a consistent performance drop of over 20 points on multi-statement questions, highlighting the importance of multi-knowledge understanding.
This work's significance lies in establishing a scalable, contamination-resistant, and comprehensive evaluation paradigm that better reflects real-world knowledge integration challenges. By reducing annotation costs and enabling continuous dataset refresh, Encyclo-K paves the way for more reliable and nuanced assessments of large language models’ capabilities. Future work will focus on expanding knowledge sources, refining question complexity, and integrating multi-modal data, aiming to further bridge the gap between model performance and human-level understanding.
Deep Analysis
Background
The rapid development of large language models (LLMs) such as GPT, BERT, and their后续版本推动了自然语言处理的革命，但其评估体系仍主要依赖静态问答，存在数据污染、单一知识点限制和高昂标注成本等问题。传统基准如SuperGLUE、MMLU在推动模型能力提升的同时，难以全面衡量模型的跨领域知识融合和推理能力。随着模型规模不断扩大，评估方法亟需创新以应对模型泛化和知识整合的挑战。近年来，动态评估和多知识点测试逐渐成为研究热点，但缺乏系统性解决方案。本文提出的Encyclo-K正是在此背景下，旨在构建一种抗污染、全面、多知识点的动态评估体系，填补现有方法的空白。
Core Problem
现有评估体系面临三大难题：一是问答题易被训练数据污染，影响评估的公平性；二是单一知识点难以反映模型的跨学科理解能力；三是高成本的专家标注限制了大规模应用。尤其是在模型不断变大、能力不断提升的背景下，静态问答难以满足多维度、多知识点的评估需求，亟需一种低成本、鲁棒性强的创新框架。
Innovation
核心创新包括：1）将知识陈述作为评估单元，避免问答题的局限；2）利用随机采样动态组合知识陈述，生成多样化题目，确保评估的动态性和抗污染能力；3）引入多陈述融合机制，提升模型跨知识点理解能力；4）采用低成本标注策略，只需验证格式，无需专业知识。此创新突破了传统静态问答的限制，显著增强了评估的多样性和鲁棒性。
Methodology
- �� 从权威教材中提取独立知识陈述，确保内容完整、无歧义。• 利用DeepSeek等模型自动生成错误陈述，丰富题目选项。• 设计题目模板，将8-10个陈述随机组合成多选题，每题提供4-8个选项，每个选项由2-4个陈述组成。• 在测试阶段，采用多轮随机采样，生成不同题集，确保模型排名稳定。• 采用正则表达式匹配模型回答，保证答案提取的准确性。• 评估指标包括准确率、模型差异和抗污染能力，验证设计效果。
Experiments
在50余个模型上进行评估，包括OpenAI、Qwen、DeepSeek等，比较不同规模和架构模型的表现。采用多轮随机采样生成题目，确保评估的鲁棒性。指标包括准确率、跨学科差异、模型排名稳定性。还进行消融实验，验证多陈述融合和随机采样的效果。通过对比静态问答，突出动态生成的优势。实验还分析模型在不同学科、不同题型上的表现差异，验证方法的广泛适用性。
Results
模型在Encyclo-K上的平均准确率为62.07%，远低于传统基准，显示出极高的挑战性。模型在不同学科表现差异显著，推理模型从16.04%到62.07%，聊天模型从9.71%到50.40%。多陈述题目平均性能低于单一知识点题目20%以上，验证了跨知识点理解难度。随机采样保持模型排名稳定，证明设计的鲁棒性。结果显示，模型在多知识点融合和推理任务中仍有巨大提升空间。
Limitations & Outlook
当前方法依赖教材内容，可能存在知识覆盖不足的问题。模型在复杂推理、多模态融合方面仍表现有限。动态生成机制在极端复杂题型上效果待提升，且对计算资源要求较高。未来需优化题目设计，增强多领域适应性，提升模型在实际场景中的表现。
Plain Language Accessible to non-experts
想象你在一家大型厨房里，厨师们用各种食材（知识陈述）准备菜肴。每次做菜前，厨师会随机挑选一些食材（随机组合陈述），然后按照食谱（题目模板）把它们拼在一起，做出一道新菜。这些菜肴代表不同的知识点组合，厨师（模型）需要理解每个食材的特点，才能判断菜是否合格。这样做可以测试厨师对多种食材的搭配能力，而不是只看他会做单一菜肴。每次换食材和菜谱，厨师都要重新应对，既公平又全面。
ELI14 Explained like you're 14
想象你在学校的厨房里，老师让你用不同的食材做一道菜。老师会随机挑选一些食材，比如鸡肉、蔬菜、米饭，然后让你把它们拼在一起，看看你能不能理解每个食材的味道和搭配。每次老师换不同的食材组合，你都要用之前学到的知识去判断这道菜是不是好吃。这样一来，老师就能知道你是不是真正懂得怎么搭配食材，而不是只会做一道简单的菜。这就像在测试你对很多不同知识点的理解能力，而不是只考你记住了什么。
Glossary
Knowledge Statement (知识陈述)
描述某一知识点的完整句子或段落，能单独理解，无需依赖其他内容。
作为动态评估题目的基本单元。
Dynamic Sampling (动态采样)
在测试时随机抽取知识陈述进行组合，生成不同题目，避免模型记忆。
确保评估的鲁棒性和多样性。
Multi-Statement Comprehension (多陈述理解)
模型同时理解多个知识陈述，进行联合推理的能力。
衡量模型跨知识点整合能力。
Contamination Resistance (污染抵抗)
通过随机组合避免训练数据中的题目直接出现，确保评估公平性。
提升评估的客观性。
Knowledge Composition (知识组合)
将多个知识陈述随机拼接成题目，测试模型的综合理解能力。
核心创新之一。
Open Questions Unanswered questions from this research
- 1 如何扩展知识源以覆盖更多专业领域？
- 2 多模态知识融合在动态评估中的应用潜力？
- 3 模型在极端复杂多陈述推理中的表现机制仍未充分理解。
Applications
Immediate Applications
模型能力评估工具
为研究者提供抗污染、多知识点的模型性能衡量方法，支持模型开发。
知识管理优化
利用动态组合提升企业知识库的知识整合和检索效率。
Long-term Vision
智能教育评估平台
构建面向教育行业的个性化评测体系，提升学生跨学科理解能力。
Abstract
Benchmarks play a crucial role in tracking the rapid advancement of large language models (LLMs) and identifying their capability boundaries. However, existing benchmarks predominantly curate questions at the question level, suffering from three fundamental limitations: vulnerability to data contamination, restriction to single-knowledge-point assessment, and reliance on costly domain expert annotation. We propose Encyclo-K, a statement-based benchmark that rethinks benchmark construction from the ground up. Our key insight is that knowledge statements, not questions, can serve as the unit of curation, and questions can then be constructed from them. We extract standalone knowledge statements from authoritative textbooks and dynamically compose them into evaluation questions through random sampling at test time. This design directly addresses all three limitations: the combinatorial space is too vast to memorize, and model rankings remain stable across dynamically generated question sets, enabling reliable periodic dataset refresh; each question aggregates 8-10 statements for comprehensive multi-knowledge assessment; annotators only verify formatting compliance without requiring domain expertise, substantially reducing annotation costs. Experiments on over 50 LLMs demonstrate that Encyclo-K poses substantial challenges with strong discriminative power. Even the top-performing OpenAI-GPT-5.1 achieves only 62.07% accuracy, and model performance displays a clear gradient distribution--reasoning models span from 16.04% to 62.07%, while chat models range from 9.71% to 50.40%. These results validate the challenges introduced by dynamic evaluation and multi-statement comprehensive understanding. These findings establish Encyclo-K as a scalable framework for dynamic evaluation of LLMs' comprehensive understanding over multiple fine-grained disciplinary knowledge statements.