Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMs
Proposes a black-box confidence estimation framework combining prompting, sampling, and aggregation, improving calibration and failure prediction metrics.
Key Findings
Methodology
This study develops a systematic framework integrating prompting strategies (e.g., self-probing, multi-step reasoning, Top-K), diverse sampling methods (random, paraphrasing), and multiple aggregation techniques (consistency, average confidence, ranking). These components are combined to estimate model confidence in a black-box setting. Experiments across five models (GPT-4, LLaMA 2) and five datasets (commonsense, math, symbolic, professional, ethical) demonstrate that tailored prompts and response diversity significantly improve calibration (reducing ECE from 52.0 to below 0.1) and failure prediction (AUROC up to 68%). The framework effectively addresses overconfidence issues prevalent in large models.
Key Results
- Models tend to be highly overconfident in verbalized confidence, with most values between 80%-100%, and an average ECE of 52.0 for GPT-3, which reduces to 18.0 for GPT-4 after calibration techniques. AUROC for failure prediction improves from near random (50%) to over 62% with prompt optimization.
- Prompt strategies inspired by human dialogue (self-probing, multi-step reasoning) consistently enhance calibration and failure detection, with no single method dominating across all datasets. Response diversity via sampling and ranking further boosts performance, achieving AUROC up to 68%.
- Scaling models improves both calibration and failure prediction, yet challenges remain in high-stakes tasks requiring specialized knowledge, indicating ongoing need for refinement.
Significance
This work advances the practical deployment of large language models by enabling reliable confidence estimates without internal access. It addresses critical issues of overconfidence and unreliable failure detection, essential for applications in healthcare, finance, and safety-critical systems. The proposed framework broadens the scope of uncertainty quantification in black-box models, fostering safer and more trustworthy AI systems.
Technical Contribution
The paper introduces a unified framework combining prompting, sampling, and aggregation for black-box confidence estimation. It innovates by leveraging multiple responses and ranking-based methods, providing a versatile toolkit adaptable to various models and tasks. Compared to white-box approaches, it offers a practical solution compatible with commercial APIs, with theoretical insights into response variance and confidence calibration.
Novelty
This is the first comprehensive systematization of black-box confidence elicitation, integrating diverse prompt designs, multi-response sampling, and advanced aggregation metrics. Unlike prior works limited to token likelihoods or fine-tuning, it emphasizes response consistency and semantic confidence, filling a significant gap in practical uncertainty estimation for proprietary models.
Limitations
- Performance drops significantly on tasks requiring domain-specific knowledge, indicating that the current strategies are insufficient for complex, high-stakes scenarios.
- The increased computational cost from multiple sampling and response aggregation may hinder real-time deployment in resource-constrained environments.
- Overconfidence persists in some cases, especially in professional or ethical reasoning tasks, suggesting further research is needed to address intrinsic model biases.
Future Work
Future directions include integrating external knowledge bases, exploring multi-modal signals, and developing adaptive prompting strategies tailored to specific tasks. Enhancing efficiency through smarter sampling and aggregation methods will also be prioritized. Additionally, extending the framework to multilingual and domain-specific models can broaden its applicability.
AI Executive Summary
Large language models (LLMs) have revolutionized natural language processing, yet their confidence estimates often remain unreliable, especially in black-box settings where internal model details are inaccessible. Traditional confidence calibration methods rely on token likelihoods or fine-tuning, which are impractical for commercial APIs like GPT-4 or LLaMA 2. Recognizing this gap, the current study introduces a novel confidence elicitation framework that leverages prompting, response sampling, and multi-metric aggregation to estimate model confidence without internal access.
The core idea involves designing human-inspired prompts—such as self-probing, multi-step reasoning, and Top-K guesses—that coax models into verbalizing their confidence levels. These responses are then diversified through sampling techniques, including random temperature adjustments and paraphrasing, to generate multiple outputs. The responses are subsequently analyzed using various aggregation strategies—like measuring agreement, averaging confidence scores, and ranking responses—to produce a robust estimate of the model’s certainty.
Experimental results across five prominent models (GPT-4, LLaMA 2, Vicuna) and five datasets (covering commonsense, mathematical, symbolic, professional, and ethical reasoning) demonstrate that this approach significantly improves calibration (reducing ECE from over 50% to below 20%) and enhances failure prediction (AUROC up to 68%). Notably, larger models tend to be more overconfident but also show better calibration and failure detection capabilities, highlighting the importance of model scaling.
This framework offers a practical solution for deploying trustworthy LLMs in real-world applications, such as medical diagnosis, financial decision-making, and safety-critical systems. It addresses key challenges like overconfidence and unreliable failure detection, providing a pathway toward safer AI. Despite these advances, limitations remain in handling complex, domain-specific tasks, and computational costs are non-trivial. Future research will focus on integrating external knowledge, improving efficiency, and extending the approach to diverse languages and specialized domains, aiming to make black-box confidence estimation a standard component of trustworthy AI systems.
Deep Dive
Abstract
Empowering large language models to accurately express confidence in their answers is essential for trustworthy decision-making. Previous confidence elicitation methods, which primarily rely on white-box access to internal model information or model fine-tuning, have become less suitable for LLMs, especially closed-source commercial APIs. This leads to a growing need to explore the untapped area of black-box approaches for LLM uncertainty estimation. To better break down the problem, we define a systematic framework with three components: prompting strategies for eliciting verbalized confidence, sampling methods for generating multiple responses, and aggregation techniques for computing consistency. We then benchmark these methods on two key tasks-confidence calibration and failure prediction-across five types of datasets (e.g., commonsense and arithmetic reasoning) and five widely-used LLMs including GPT-4 and LLaMA 2 Chat. Our analysis uncovers several key insights: 1) LLMs, when verbalizing their confidence, tend to be overconfident, potentially imitating human patterns of expressing confidence. 2) As model capability scales up, both calibration and failure prediction performance improve. 3) Employing our proposed strategies, such as human-inspired prompts, consistency among multiple responses, and better aggregation strategies can help mitigate this overconfidence from various perspectives. 4) Comparisons with white-box methods indicate that while white-box methods perform better, the gap is narrow, e.g., 0.522 to 0.605 in AUROC. Despite these advancements, none of these techniques consistently outperform others, and all investigated methods struggle in challenging tasks, such as those requiring professional knowledge, indicating significant scope for improvement. We believe this study can serve as a strong baseline and provide insights for eliciting confidence in black-box LLMs.