The Illusion of Certainty: Decoupling Capability and Calibration in On-Policy Distillation
CaOPD framework decouples capability and calibration, reducing overconfidence by using empirical success rates, achieving better calibration without sacrificing accuracy.
Key Findings
Methodology
This paper analyzes the intrinsic conflict between capability transfer and confidence calibration in on-policy distillation (OPD). It introduces CaOPD, which samples multiple model rollouts to estimate empirical success rates, replacing teacher-provided confidence tokens with these real success signals. The approach leverages self-distillation with target replacement, effectively decoupling the model’s ability to answer from its confidence estimation. Theoretical analysis demonstrates how information asymmetry causes entropy collapse and optimism bias in standard OPD. Extensive experiments across NLP tasks and model scales validate that CaOPD improves calibration metrics (ECE, Brier Score) while maintaining or enhancing task accuracy, with negligible additional computational cost.
Key Results
- Across tasks like Science Q&A and Tool Use, CaOPD reduces ECE by over 30%, improves Brier Score, and significantly enhances confidence discrimination (SPR). For example, in Qwen3-8B, calibration error drops from 0.157 to 0.141, and SPR increases from 0.085 to 0.555, indicating more reliable confidence estimates.
- Compared to RL-based calibration methods, CaOPD avoids the capability trade-off, maintaining high accuracy (e.g., 80.6% in SDPO) while drastically reducing overconfidence (OCG from +32% to -0.7%). It effectively aligns confidence with true success probability.
- Ablation studies show sampling with K≥8 achieves optimal calibration with minimal overhead, and the decoupling mechanism prevents overconfidence saturation, enabling models to distinguish successes from failures reliably.
Significance
This work addresses a fundamental limitation in current knowledge distillation methods—namely, the inability to calibrate confidence without impairing capability transfer. By formalizing the information asymmetry problem and proposing a practical solution, CaOPD advances the development of trustworthy AI systems. Its ability to produce models with both high accuracy and reliable confidence signals is crucial for deploying AI in safety-critical applications like autonomous systems, scientific research, and decision support, thereby fostering greater trust and safety in AI systems.
Technical Contribution
The paper introduces a novel theoretical framework revealing how privileged teacher contexts induce entropy collapse and optimism bias in OPD. It proposes a practical algorithm—target replacement with empirical success rate estimation—that decouples confidence supervision from capability transfer. This approach integrates seamlessly into existing distillation pipelines, requiring no reward shaping or complex optimization, and offers provable guarantees of improved calibration while preserving task performance.
Novelty
This is the first comprehensive analysis of the intrinsic conflict between capability and confidence calibration in on-policy distillation. The core innovation lies in leveraging model rollouts to estimate unbiased success probabilities and replacing teacher confidence tokens accordingly. This decoupling strategy fundamentally shifts how confidence is learned, setting a new paradigm for trustworthy model training that surpasses prior RL-based or post-hoc calibration methods.
Limitations
- The approach relies on multiple rollouts and verifier evaluations, which can increase computational costs, especially for very large models or real-time applications.
- In open-ended or unsupervised tasks lacking explicit success metrics, estimating empirical success rates may be challenging, affecting calibration quality.
- The method's effectiveness in extreme out-of-distribution or highly adversarial scenarios remains to be fully validated, necessitating further research.
Future Work
Future directions include optimizing rollout sampling strategies to reduce computational overhead, extending calibration techniques to multi-modal and multi-task settings, and integrating adaptive methods for unsupervised domains. Additionally, exploring theoretical bounds and real-world deployment scenarios will further solidify the practical impact of CaOPD.
AI Executive Summary
In recent years, large language models (LLMs) have achieved remarkable success in complex reasoning and problem-solving tasks. However, their confidence estimates often suffer from severe overconfidence, limiting their trustworthiness in critical applications such as autonomous decision-making, scientific research, and safety-critical systems. Traditional knowledge distillation methods focus on transferring capabilities but overlook the calibration of confidence, leading to models that are highly capable yet unreliable in their self-assessment.
This paper investigates the root cause of this discrepancy, revealing that on-policy distillation (OPD) inherently induces an information asymmetry. The teacher, conditioned on privileged context, produces low-entropy, overly confident trajectories, which the student then mimics. This results in entropy collapse and an optimism bias, where models systematically overestimate their success probabilities. Recognizing this fundamental limitation, the authors propose a novel framework—Calibration-Aware On-Policy Distillation (CaOPD)—that explicitly decouples the learning of capability from confidence calibration.
CaOPD leverages model rollouts to empirically estimate the true success probability of the student model at deployment time, replacing teacher-provided confidence tokens with these unbiased estimates. This target replacement mechanism ensures that the model learns to produce confidence scores aligned with actual performance, effectively mitigating overconfidence without sacrificing task accuracy. Extensive experiments across NLP tasks demonstrate that CaOPD significantly reduces calibration errors (over 30%) while maintaining or improving accuracy, outperforming RL-based calibration methods that often compromise capability.
The approach is computationally efficient, requiring only modest additional sampling, and is robust across different model sizes and domains. Its theoretical foundation and practical effectiveness mark a significant step toward trustworthy AI, enabling models that are both highly capable and reliably calibrated. Future work aims to extend this framework to multi-modal, multi-task, and real-world deployment scenarios, fostering broader adoption of trustworthy AI systems.
Deep Dive
Abstract
On-policy distillation (OPD) is an increasingly important paradigm for post-training language models. However, we identify a pervasive Scaling Law of Miscalibration: while OPD effectively improves task accuracy, it systematically traps models in severe overconfidence. We trace this failure to an information mismatch: teacher supervision is formed under privileged context available during training, whereas the deployed model must report confidence using only deployment-time information. We formalize this perspective theoretically, showing that teacher-conditioned success is generally not a valid target for deployment-time confidence and that helpful privileged context induces entropy collapse and a systematic optimism bias. To address this, we propose a calibration-aware OPD framework, CaOPD, that estimates empirical confidence from model rollouts, replaces self-reported confidence with this student-grounded target, and distills the revised response through the same self-distillation pipeline. Experiments across various models and domains show that CaOPD achieves Pareto-optimal calibration while maintaining competitive capability, generalizing robustly under out-of-distribution and continual learning. Our findings highlight that capability distillation does not imply calibrated confidence, and that confidence should be treated as an essential objective in post-training. Code: https://github.com/SalesforceAIResearch/CaOPD