Characterising LLM-Generated Competency Questions: a Cross-Domain Empirical Study using Open and Closed Models

Ontology Engineering (OE) is a crucial field in information science, aimed at organizing and sharing knowledge through ontology construction. Requirement elicitation is a foundational stage in the OE lifecycle, determining the functional scope and semantic adequacy of the resulting model. Competency questions (CQs) are recognized as the standard mechanism for this task, serving as a natural language interface between domain experts and ontology engineers. By framing requirements as answerable questions, CQs direct the modeling of concepts and relations, underpin validation and testing, and inform assessments of ontology reuse. However, the manual formulation of CQs remains a major bottleneck, as it requires substantial domain and modeling expertise, leading to their under-utilization in practice. To mitigate this, the OE community has increasingly turned towards automation, from pattern-based approaches to the use of Large Language Models (LLMs).

Despite the rapid adoption of LLMs in OE, understanding their output properties across multiple dimensions and characteristics remains a significant challenge. For example, the influence of model architecture, parameter size, and input domain on features such as linguistic structure, complexity, and semantic diversity of generated CQs is poorly investigated. Treating LLMs as a monolithic solution ignores the substantial variability in their output profiles that ontology engineers must understand to effectively select LLM-based OE tools.

The innovation of this paper lies in the introduction of a multi-dimensional framework called CompCQ for systematically comparing competency questions generated by different LLMs. This framework analyzes the complexity and readability of generated questions by quantifying linguistic, syntactic, and semantic features. The study uses open models like KimiK2-1T, LLama3.1-8B, LLama3.2-3B, and closed models like Gemini 2.5 Pro and GPT 4.1, conducting experiments across five domains including cultural heritage and healthcare. Through this approach, we identify salient properties of generated questions, including readability, relevance with respect to the input text, and structural complexity.

• �� Introduce the CompCQ framework: a multi-dimensional framework for quantifying and comparing LLM-generated competency questions.
• �� Use multiple LLMs for experiments: including open models (KimiK2-1T, LLama3.1-8B, LLama3.2-3B) and closed models (Gemini 2.5 Pro, GPT 4.1).
• �� Cross-domain analysis: conduct experiments across five domains including cultural heritage and healthcare to identify salient properties of generated questions.
• �� Quantify features: analyze the readability, structural complexity, and semantic diversity of generated questions.

The experimental design covers multiple LLMs, including open and closed models, across five domains including cultural heritage and healthcare. The models used include KimiK2-1T, LLama3.1-8B, LLama3.2-3B, Gemini 2.5 Pro, and GPT 4.1. Metrics used in the experiments include readability, structural complexity, and semantic diversity. To ensure reproducibility, all models were prompted via their respective APIs with parameters set to temperature=0, Top-P=1, and seed=46.

Experimental results show that LLM performance reflects distinct generation profiles shaped by use cases. Closed models excel in stability and readability, while open models offer higher diversity but sometimes sacrifice clarity. In the personalized depression treatment ontology domain, KimiK2 generated questions scored highest in complexity and readability metrics, with an FKGL of 21, indicating a high educational level required for comprehension. Conversely, the Gemini model consistently produced the most concise and readable questions across most domains, achieving the lowest FKGL scores, indicating a generation style favoring simple, direct questions.

The applications of this study include requirement elicitation and validation processes in ontology engineering. By automatically generating high-quality competency questions, it significantly reduces human intervention and improves efficiency. Additionally, this approach can be applied to other fields requiring natural language interfaces, such as knowledge graph construction and semantic search.

Despite providing a systematic method for comparing LLM-generated competency questions, the CompCQ framework faces challenges in handling complex domains where open models exhibit significant increases in question complexity, potentially leading to comprehension difficulties. Additionally, some models generate fewer questions in specific domains, possibly failing to cover the full requirements. Closed models, while stable, lack diversity. Future research could explore combining multiple LLMs to enhance question coverage and diversity.