Privileged Likelihood Is Not Automatically Value: Three Checks for Token Credit in On-Policy Self-Distillation
This paper analyzes the validity of privileged likelihood as token credit, proposing three checks to validate its usefulness in on-policy self-distillation.
Key Findings
Methodology
The paper introduces a formal three-question framework to evaluate whether token likelihood scores reflect better actions, whether feedback construction alters the comparison, and what behaviors the training loss reinforces. Through mathematical derivations, it distinguishes between self-dependence and cross-fitting scenarios. Experiments on a 20B parameter GPT model using AIME 2025 data compare additive scores and outcome-only controls, employing metrics like AUC to assess correlation with task success. The analysis emphasizes the importance of validating score meaning, feedback design, and training behavior separately to ensure meaningful token credit.
Key Results
- On AIME 2025, the additive privileged likelihood score achieved an AUC of 0.505, close to chance, and slightly favored incorrect traces after length correction. Paired comparisons showed outcome-only control reaching 64.2% accuracy, whereas five token-score variants only scored between 24.2% and 33.9%, highlighting the limited reliability of likelihood scores as outcome indicators.
- Experiments reveal that scoring using self-written feedback introduces self-dependence, heavily influenced by content, which can distort the score’s reflection of true behavior quality. Cross-fitting reduces this dependence but does not guarantee usefulness, underscoring the need for separate validation of feedback and scoring mechanisms.
- Overall, the results demonstrate that token likelihood signals can be active and well-formed yet fail to provide meaningful outcome credit, cautioning against uncritical use of privileged likelihood in training and evaluation.
Significance
This work critically examines the assumptions behind privileged likelihood as a proxy for token contribution, revealing its limitations and potential pitfalls. It provides a rigorous framework to validate whether such scores truly reflect task success, informing better design of reward signals in large language models. The findings have implications for improving model interpretability, robustness, and fairness, especially in self-supervised and reinforcement learning contexts, by emphasizing the necessity of separate validation steps for feedback quality, score interpretability, and training objectives.
Technical Contribution
The paper offers a formal, mathematically grounded framework to distinguish between different notions of token credit, incorporating information-theoretic measures and probabilistic analysis. It clarifies conditions under which likelihood ratios can represent outcome action values, and demonstrates how content dependence and feedback construction influence score validity. Empirical validation on large models highlights the practical limitations of privileged likelihood, guiding future reward design and interpretability efforts in self-distillation and RLHF.
Novelty
This is the first systematic formalization of the three critical questions—behavior tracking, feedback comparison, and training reinforcement—in evaluating privileged likelihood as token credit. It combines theoretical proofs with large-scale experiments, revealing that likelihood scores, despite being structurally sound, often do not encode meaningful outcome information. This work advances the understanding of probabilistic scoring in model training, setting a new standard for validation and interpretability.
Limitations
- The experiments are limited to the AIME 2025 dataset and a 20B parameter GPT model, requiring further validation across different tasks and larger models to confirm generality.
- Feedback construction relies on external language models, which may introduce biases or inconsistencies, and the impact of diverse feedback sources remains unexplored.
- The study does not deeply analyze how different training objectives or loss functions affect the behavior of likelihood scores, suggesting future research directions.
Future Work
Future research will focus on developing more robust feedback generation methods, incorporating multi-modal signals, and multi-turn interactions to improve the task relevance of privileged likelihood. Additionally, exploring different model architectures and training regimes will help understand how to better align scores with true outcome contributions, ultimately leading to more reliable reward signals and interpretability in large-scale language models.
AI Executive Summary
This study critically examines the role of privileged likelihood as a token-level credit signal in on-policy self-distillation. While intuitively appealing, the authors demonstrate through formal analysis and empirical validation that likelihood changes do not automatically equate to meaningful outcome contributions. Using a 20B parameter GPT model trained on AIME 2025 data, the experiments reveal that additive privileged likelihood scores perform near chance (AUC=0.505), and after length correction, tend to favor incorrect traces. Paired comparisons further show that outcome-only controls significantly outperform token-score variants, with accuracy reaching 64.2%. These findings underscore the importance of validating what scores measure, how feedback is constructed, and what training objectives reinforce, before relying on likelihood signals as outcome credits. The paper emphasizes that content dependence and self-referential feedback can distort scores, leading to misleading training updates. Its formal framework clarifies the conditions under which likelihood ratios can represent true outcome action values, highlighting the pitfalls of arbitrary feedback and content-dependent scoring. The insights provided are crucial for designing more reliable reward signals in large language models, especially in reinforcement learning and self-distillation contexts. Overall, this work advances the understanding of probabilistic scoring mechanisms, advocating for rigorous validation to ensure that training signals genuinely reflect task success rather than superficial correlations or content artifacts.
Deep Dive
Abstract
Outcome verifiers score completed reasoning traces but do not assign credit to intermediate tokens. Privileged self-distillation attempts to fill this gap by rescoring a model's own rollout with training-only information. A token likelihood change, however, is not automatically outcome credit. We separate three questions: whether the score tracks better actions, whether feedback construction changes what is compared, and what behavior the training loss reinforces. We establish these distinctions formally. When a rollout is scored using hindsight feedback written about that same rollout, its content determines both the tokens and the scoring context, creating direct self-dependence. Using feedback from another rollout of the same problem removes this dependence but does not guarantee a useful score. In matched experiments with a 20B model on AIME 2025, the implemented additive score is near chance (AUC=0.505) and slightly favors incorrect traces after length adjustment. In the paired comparison, the outcome-only control records 64.2\%, versus 24.2\%--33.9\% for five token-score variants. The results motivate validating score meaning, feedback construction, and training behavior separately before calling a likelihood signal credit.