Scaling with Confidence: Calibrating Confidence of LLMs for Adaptive Test Time Scaling
C3RL calibrates LLM confidence, while CAS matches majority-vote accuracy with up to 12.33× less sampling.
Key Findings
Methodology
C3RL combines correctness, calibration, and dataset-informed reference-accuracy rewards: R=R_correctness+R_calibration+R_reference. The model emits an answer and a 1–10 verbalized confidence score, with t=5 separating certain from uncertain outputs. CAS aggregates confidence by unique answer, identifies the largest and second-largest sums c1 and c2, and uses a Beta-distribution stopping probability to decide whether more samples are needed.
Key Results
- For Qwen2.5VL-7B-Instruct, C3RL obtains 53.3% average multimodal accuracy, 0.644 AUROC, and 0.109 ECE; ECE is 0.096 on MMMU and 0.074 on MathVista, compared with 0.182 average ECE for RLCR.
- CAS preserves majority-voting accuracy while reducing sampling: it uses 5.19 samples on average for OOD text, saving 12.33× budget; 10.91 samples in-domain, saving 5.87×; and 7.13 on multimodal OOD tasks, saving 8.98×.
- Removing the reference-accuracy reward increases Qwen’s average ECE from 0.107 to 0.142. This indicates that the reward suppresses the shortcut of producing incorrect answers with deliberately low confidence and protects previously reliable behavior.
Significance
The paper unifies answer correctness with knowing when an answer may be wrong. It addresses a persistent weakness of RLVR, which rewards correctness but does not directly discourage overconfident hallucination, and shows that calibration is not merely an evaluation property: it can control computation. For high-stakes QA, tutoring, and multimodal agents, the framework enables extra reasoning on uncertain cases and early termination on reliable ones.
Technical Contribution
C3RL uses three complementary rewards to avoid the degeneration associated with calibration-only objectives: correctness preserves task performance, threshold-based calibration teaches a certainty boundary, and reference accuracy exploits “all correct,” “partially correct,” and “all incorrect” tags from ten sampled responses. CAS converts verbalized confidence into a Beta-based stopping probability, rather than relying only on answer frequency or entropy, creating a tunable adaptive test-time scaling mechanism.
Novelty
Relative to SaySelf, RLCR, and Adaptive-Consistency, the work presents a unified pipeline connecting fine-grained confidence RL with adaptive test-time scaling. Its fundamental innovation is not merely asking for confidence, but using dataset-level reference accuracy to prevent calibration from decoupling from correctness, then using the calibrated signal itself to determine when inference should stop.
Limitations
- Training relies on NuminaMath-TIR, WebInstruct-verified, LogicNLI, and LogiQA, with reference tags produced from ten samples of Qwen or Llama; calibration may therefore inherit base-model biases.
- Confidence is a discrete 1–10 verbalized score, while t=5, α, β, and Pstop require tuning. Robustness in open-ended generation, long dialogue, and severe distribution shift remains under-tested.
- The model and benchmark coverage is limited; larger models, real parallel-serving costs, and safety-critical error costs are not systematically evaluated.
Future Work
Future work should examine continuous or interval confidence, automatically adapt thresholds across tasks, and obtain reference labels without relying on base-model sampling. Evaluation should expand to code, long-context reasoning, and agentic tasks, while jointly optimizing latency, cost, and accuracy under parallel CAS. Human handoff, abstention, and safety-audit outcomes also deserve direct study.
AI Executive Summary
Large language models increasingly solve mathematical, scientific, and visual reasoning problems, yet they often sound certain when they are wrong. Conventional RLVR rewards correctness but not reliable self-knowledge; calibration-oriented methods such as SaySelf can instead encourage low-confidence errors. The result is a difficult trade-off between capability and trustworthiness.
Yang and colleagues propose Correctness and Confidence Calibration Reinforcement Learning, or C3RL. The model emits a 1–10 confidence score. Correct answers receive correctness reward; confidence aligned with correctness receives calibration reward; and a reference-accuracy reward uses ten-sample tags—“all correct,” “partially correct,” or “all incorrect”—to prevent calibrated but wrong behavior. Their inference method, Confidence-based Adaptive Test Time Scaling (CAS), aggregates confidence over candidate answers and applies a Beta-distribution stopping probability.
Across text and multimodal benchmarks, C3RL reduces calibration error without materially sacrificing accuracy: on MMMU and MathVista, ECE reaches 0.096 and 0.074. CAS matches majority voting while using only 5.19 samples on OOD text, a 12.33× budget saving. The study reframes confidence as an operational control signal, although its discrete scores, model-generated reference labels, and limited model coverage require broader validation.
Deep Analysis
Background
LLMs perform strongly on reasoning but remain vulnerable to overconfident hallucination. Prompt-based confidence depends on wording, while sampling consistency and majority voting are expensive. SaySelf and RLCR improve calibration with Brier-like or correctness-aware rewards, but may trade accuracy for cautious behavior. This paper links calibration directly to adaptive computation.
Core Problem
The goal is not merely higher accuracy: confidence should reflect the probability of being correct without allowing a model to minimize calibration loss by giving wrong answers with low confidence. Fixed-budget majority voting also wastes computation on easy items and under-invests in difficult ones.
Innovation
- �� C3RL jointly optimizes correctness, calibration, and reference accuracy.
- �� Its 1–10 signal and t=5 threshold provide finer supervision than binary sure/unsure labels.
- �� CAS uses confidence-weighted competition among answers and a Beta stopping probability.
- �� The training-to-inference pipeline turns calibrated confidence into a computation policy.
Methodology
- �� Data: NuminaMath-TIR (69k), WebInstruct-verified (4.5k), LogicNLI (16k), and LogiQA (8,678), yielding 106k training and 1,073 test samples.
- �� Tagging: the base model samples ten answers at temperature 0.7; Accq=ΣI(ai=a*)/10 determines all-correct, partially-correct, or all-incorrect tags.
- �� C3RL: R_correctness=I(a=a*); certain-correct and uncertain-incorrect receive +β, while the opposite cases receive −β; correcting an all-incorrect item earns +α, and hallucinating on an all-correct item earns −α.
- �� CAS: confidence is summed per unique answer, producing c1 and c2; the normalized Beta integral gives Prob(c1,c2), and sampling stops when it exceeds Pstop or reaches N.
Experiments
Evaluation covers AGIEval, MMLU, MMMU, MathVista, and LogicVista, while CAS uses MMLU validation, FOLIO, GSM8K, and multimodal sets. Baselines are Base, Self-Consistency, SFT+Ref, RLVR, SaySelf, and RLCR. Metrics are Accuracy, AUROC, and 20-bin ECE. CAS is compared with majority voting at N=64 and Adaptive-Consistency; removing the reference reward provides the main ablation.
Results
For Qwen, C3RL reaches 64.5% accuracy, 0.696 AUROC, and 0.107 ECE in-domain, and 53.3%, 0.644, and 0.109 on multimodal benchmarks. Against SaySelf, OOD accuracy rises by 7.9 percentage points on text and 19.2 on multimodal data. CAS needs only 5.19 samples on OOD text versus Adaptive-Consistency’s 12.87 while matching majority-vote accuracy; removing reference reward worsens ECE.
Applications
Potential uses include human escalation for low-confidence medical or legal answers, extra reasoning for difficult educational problems, and adaptive computation in visual QA and agents. Deployment requires domain-specific calibration checks, maximum-sample limits, abstention rules, and safety monitoring.
Limitations & Outlook
Reference labels depend on ten base-model samples and may reproduce systematic bias; the discrete 1–10 scale and t=5 are not guaranteed to transfer. CAS still updates candidate statistics during generation, so latency and parallel-serving costs require engineering. Larger models, continuous confidence, broader tasks, and risk-weighted evaluation are important next steps.
Plain Language Accessible to non-experts
Imagine a student taking an exam. Ordinary training gives only right-or-wrong marks, so the student may confidently bluff when unsure. C3RL acts like a teacher who also grades honesty: a correct answer is rewarded; a correct answer paired with strong confidence is good; an incorrect answer paired with uncertainty is at least appropriately cautious. The teacher also checks how the student performed on ten earlier attempts, discouraging sudden mistakes on questions the student usually solves.
CAS is an intelligent invigilator. It lets the student try several times and compares both repeated answers and confidence. If one answer is repeatedly given with strong confidence, it ends the process early. If answers disagree or confidence is weak, it allows more attempts. Easy questions therefore consume little time, while difficult ones receive more attention. In the paper, CAS uses only 5.19 attempts on average for OOD text while matching a method fixed at 64 attempts, saving up to 12.33 times the sampling budget.
ELI14 Explained like you're 14
Picture a quiz game where you answer a question and also rate your confidence from 1 to 10. A 1 means “I’m guessing,” while a 10 means “absolutely sure.” If you miss a question but shout 10 every time, the game cannot trust you. If you get everything right but always choose 1, the game also cannot tell when your answer is dependable. C3RL trains the player to be both accurate and honest.
It also checks your history. If you solved this kind of problem correctly many times, you should not suddenly make a careless guess. If you often miss it, you should lower your confidence. CAS then acts like a teammate deciding whether to keep trying: when one answer is popular and very confident, stop; when several answers are tied, keep sampling.
The researchers tested math, logic, science, and image questions. C3RL achieved an ECE of 0.074 on MathVista, meaning confidence was closely aligned with results. On some OOD text tasks, CAS needed only 5.19 attempts instead of a fixed 64. Simple levels finish quickly, while tricky levels get extra thinking—like spending more game lives only when the boss fight is hard!
Glossary
Confidence calibration
Calibration means that stated confidence matches the long-run chance of being correct. For example, answers marked 80% confident should be correct about 80% of the time.
The paper evaluates calibration with AUROC and ECE.
C3RL
Correctness and Confidence Calibration Reinforcement Learning is an RL method combining correctness, calibration, and reference-accuracy rewards.
It is the training component of the proposed framework.
CAS
Confidence-based Adaptive Test Time Scaling dynamically decides whether to generate more samples or stop.
It uses C3RL’s verbalized confidence to control inference budget.
ECE
Expected Calibration Error measures the gap between average confidence and accuracy across confidence bins; lower is better.
The paper uses 20-bin ECE.
AUROC
Area Under the Receiver Operating Characteristic measures how well confidence separates correct from incorrect answers; higher is better.
It evaluates discrimination across confidence thresholds.
Open Questions Unanswered questions from this research
- 1 Can discrete 1–10 confidence transfer reliably to open-ended writing, code, and long dialogue? Continuous probabilities, intervals, or calibrated token-level uncertainty may be needed.
- 2 Reference tags come from base-model samples and may contain systematic bias. Independent human labels, external verifiers, or uncertainty models are needed to test causal benefits.
- 3 Whether fewer samples produce lower real serving cost depends on batching, parallel generation, latency, and caching; the paper does not fully resolve this systems question.
Applications
Immediate Applications
Educational question-answering
An assistant can return easy, high-confidence answers quickly while allocating more generations to proofs, logic problems, or repeatedly missed exercises. Low-confidence outputs can trigger explanations, warnings, or teacher review.
Multimodal QA and human escalation
For charts, medical images, or product photos, CAS can stop early for reliable interpretations and continue sampling or escalate uncertain ones. Deployment should first measure ECE on target data and define safety thresholds.
Long-term Vision
Risk-aware autonomous agents
Future agents could use calibrated confidence to choose reasoning depth, tool calls, human approval, or abstention. This would align computation with error cost in finance, healthcare, and security operations, provided robust domain calibration is achieved.
Abstract
Training large language models (LLMs) with reinforcement learning (RL) has significantly advanced their performance on reasoning and question-answering tasks. However, prevailing RL reward designs typically prioritize response correctness, neglecting to incentivize models to express their confidence accurately. This leads to a critical problem: performance gains are often accompanied by poor calibration between confidence and accuracy, misleading models to overconfidently hallucinate when uncertain. To address this limitation, we propose $\textbf{C}$orrectness and $\textbf{C}$onfidence $\textbf{C}$alibration $\textbf{R}$einforcement $\textbf{L}$earning ($\textbf{C3RL}$), a novel RL algorithm integrating correctness, calibration and dataset-informed reference accuracy rewards together. Comprehensive evaluation across 8 text and multimodal datasets demonstrates that C3RL enhances calibration without sacrificing accuracy, outperforming the current state-of-the-art method in both performance and calibration metrics. Utilizing the well-calibrated verbalized confidence from C3RL, we further introduce $\textbf{C}$onfidence-based $\textbf{A}$daptive Test Time $\textbf{S}$caling ($\textbf{CAS}$), an adjustable inference-time strategy that allocates computational resources based on response confidence. Experiments show that CAS surpasses majority voting on both in-domain and out-of-domain datasets while reducing the inference budget by up to 12.33 times. We believe the synergy of C3RL and CAS paves the way for deploying more reliable and resource-efficient LLMs. The code, data and models will be released.