Question Difficulty Estimation for Large Language Models via Answer Plausibility Scoring

In information retrieval (IR) and natural language processing (NLP) systems, questions are a fundamental means by which users express their information needs. With the development of large language models (LLMs), accurately estimating question difficulty has become an important research topic. Traditional methods often rely on readability formulas, retrieval signals, or popularity statistics, which may not fully capture the reasoning challenges faced by modern LLMs. In recent years, as LLMs are increasingly applied to question-answering (QA) tasks, researchers have begun to explore more complex and refined difficulty estimation methods to better evaluate and improve the performance of these models.

The core problem is how to accurately estimate question difficulty to better evaluate and improve the performance of large language models. Traditional difficulty estimation methods often rely on readability formulas, retrieval signals, or popularity statistics, which may not fully capture the reasoning challenges faced by modern LLMs. Especially when dealing with complex reasoning questions, existing methods often lack sufficient interpretability and practicality.

The core innovation of the Q-DAPS method lies in its estimation of question difficulty by computing the entropy of plausibility scores over candidate answers. • Generating candidate answers with plausibility scores: Prompting LLMs to generate multiple candidate answers, each assigned a plausibility score. • Popularity debiasing: Using Wikipedia page view data to adjust the plausibility scores of candidate answers to reduce the impact of popularity bias. • Entropy computation: Calculating the entropy of the debiased plausibility scores as a metric for question difficulty.

The detailed steps of the Q-DAPS method are as follows:

• �� Candidate Answer Generation: Using the LLaMA 3.3 model to generate candidate answers and assign plausibility scores to each.
• �� Popularity Debiasing: Extracting Wikipedia page view data for candidate answers and adjusting plausibility scores to reduce popularity bias.
• �� Entropy Calculation: Computing the entropy of the debiased plausibility scores and normalizing it to a difficulty score in the range of [0,1].
• �� Result Validation: Conducting experiments on multiple QA datasets to validate the accuracy and robustness of the Q-DAPS method.

The experimental design includes systematic evaluations on four QA datasets: TriviaQA, NQ, MuSiQue, and QASC. Baseline methods used include readability formulas, retrieval signals, and popularity statistics. The main metrics used in the experiments include Spearman's ρ and Cohen's d. Additionally, ablation studies were conducted to verify the robustness of the Q-DAPS method across different model sizes and question types.

Experimental results show that the Q-DAPS method significantly outperforms baseline methods across multiple datasets. • On the TriviaQA dataset, the entropy-plausibility score of the Q-DAPS method improved by over 20% compared to average plausibility scores. • On the MuSiQue dataset, Q-DAPS demonstrated strong robustness across different model sizes and question types, achieving a Spearman's ρ of -0.89. • Ablation studies showed that even without popularity debiasing, Q-DAPS maintained high performance across multiple datasets.

Application scenarios for the Q-DAPS method include: • Model Selection: Choosing a stronger LLM when most domain questions are difficult. • Question Routing: Sending high-difficulty questions to human reviewers in a company knowledge base. • Safeguard Triggering: Requiring citations or user confirmation for exam questions.

The limitations of the Q-DAPS method include: • Reliance on Wikipedia page view data for popularity debiasing, which may not be accurate in certain domains. • Performance declines when the correct answer is not provided. • In terms of computational complexity, Q-DAPS requires extensive candidate answer generation and plausibility computation, which may demand significant computational resources. Future research directions include exploring how to effectively apply Q-DAPS in domains without popularity data and further optimizing its computational efficiency.