Diagnosing Correctness Probes under Self-Judgement Confounding
This study distinguishes OC and SJ signals, showing SJ directions transfer better across domains, highlighting semantic ambiguity in correctness probes.
Key Findings
Methodology
The paper constructs four response types based on OC and SJ labels, applying linear factor analysis on hidden states extracted at the final answer token. It evaluates transferability across tasks like mathematical reasoning and factual recall using models up to 14B parameters. The analysis involves layer-wise AUC metrics, null controls, and cross-domain transfer tests, decomposing activation directions into SJ- and OC-associated components. This approach isolates semantic signals and assesses their stability and transferability, revealing differences in how models internalize correctness and self-assessment signals.
Key Results
- Across all models, the SJ-related direction (Wmeta) consistently exhibits above-chance transfer (AUC > 0.5) in cross-domain evaluations, indicating robust preservation of self-judgement polarity. Conversely, the OC-related direction (Wtruth) performs below chance, often reversing polarity, suggesting limited transferability. Layer-wise analysis shows SJ signals strengthen in middle-to-late layers, maintaining stability across tasks such as MMLU and TruthfulQA without target-specific fitting. Control experiments confirm these effects are robust against variations in answer length, confidence, and null conditions.
- In within-domain tests, Wmeta reliably predicts SJ in held-out questions with AUCs ranging from 0.649 to 0.915, while Wtruth's OC predictions are inconsistent, with some estimates below 0.5. Cross-domain transfer experiments demonstrate that directions fitted in one domain (Math or Movies) effectively preserve SJ polarity in the other, with 91% of transfer layers showing positive ∆CB values. These findings highlight the dominance of SJ signals in transfer scenarios, whereas OC signals lack robustness, often reversing or vanishing in new contexts.
- Further, the transferability of SJ directions extends to unseen datasets like MMLU and binary TruthfulQA, with AUCs consistently above 0.5, while OC directions remain near chance or below. Control analyses, including null and random directions, confirm the observed effects are not artifacts. Overall, the results emphasize that internal self-assessment signals are more stable and generalizable than objective correctness signals, raising questions about semantic interpretations of transfer metrics.
Significance
This research advances understanding of internal model representations, revealing that self-judgement signals (SJ) are more robust and transferable than objective correctness (OC). It challenges assumptions that transferability indicates semantic correctness, highlighting the importance of internal signals in interpretability and safety. The findings suggest that relying solely on transfer metrics may misrepresent a model’s true knowledge state, urging the development of more nuanced diagnostic tools. These insights are crucial for building trustworthy AI systems, especially in high-stakes applications like medical diagnosis or legal decision-making, where understanding internal biases and self-assessment mechanisms is vital.
Technical Contribution
The paper introduces a novel linear factor analysis framework to decompose internal activation directions into OC- and SJ-associated components, enabling precise evaluation of their transferability. It demonstrates that SJ signals exhibit greater stability across layers, tasks, and domains, validated through extensive cross-domain transfer experiments and null controls. The methodology combines layer-wise AUC analysis, fixed-window evaluation, and bootstrap inference, establishing a rigorous approach for probing internal semantic signals. This work bridges the gap between internal representation analysis and practical transfer diagnostics, offering a new paradigm for interpretability research.
Novelty
This is the first comprehensive study to explicitly differentiate OC and SJ signals via linear directions and demonstrate their contrasting transfer behaviors. Unlike prior work focusing on output correctness, it emphasizes internal self-assessment signals, revealing their superior transfer stability. The approach of combining factorial analysis with cross-domain validation is novel, providing deeper insights into the semantic nature of internal model representations and their implications for interpretability and robustness.
Limitations
- The linear assumption may oversimplify complex internal representations, potentially missing nonlinear relationships between signals.
- The experiments focus on instruction-tuned models, limiting generalization to other training paradigms or untrained models.
- Dataset biases and task selection could influence transfer results; broader validation across diverse tasks and models is needed.
Future Work
Future research should explore nonlinear and multi-dimensional representations of correctness signals, integrating multimodal data for richer interpretability. Extending analysis to untrained or differently trained models will clarify the universality of findings. Developing dynamic self-evaluation mechanisms that adapt across tasks and domains could enhance model reliability. Additionally, combining internal signals with external verification methods may lead to more trustworthy AI systems, especially in safety-critical applications.
AI Executive Summary
Deep Dive
Plain Language Accessible to non-experts
想象你在一个工厂里工作，工厂里有很多不同的机器，每台机器都能做不同的事情。有些机器能判断自己做得对不对，有些只是盲目地做事。科学家们想知道，这些机器内部是怎么判断自己做得对的。有一种方法是看机器的“内部信号”，就像是机器的灯光或声音，告诉你它觉得自己做得好不好。研究发现，有一种信号（叫SJ）在不同工厂（任务）之间都能保持一致，说明它反映了机器的“自我感觉”。而另一种信号（叫OC）虽然代表真正的正确答案，但在不同工厂之间却不那么稳定，有时候还会反转。这个发现帮助我们理解，机器的“自我感觉”比“真正的正确”更容易在不同环境中保持一致，但这并不意味着它一定是真正的正确。
ELI14 Explained like you're 14
想象你在学校参加考试，答完题后老师会问你：‘你觉得你答得对吗？’你可能会觉得自己答得很好，也可能觉得不太确定。其实，老师还会给你一个客观的评分，告诉你答案到底对不对。科学家们也在研究类似的事情，他们用超级聪明的电脑（叫大模型）来回答问题。但是，这些电脑有时候会自己判断答案对不对（自我判断），也有时候会被外部标准（客观正确性）评判。研究发现，这些电脑的“自我判断”信号在不同任务和环境中都很稳定，就像是它们的“直觉”；而“客观正确性”的信号则不那么稳定，有时候还会反转。就像你觉得自己答得好，但其实错了。这个发现告诉我们，要理解这些电脑是不是真的“懂”了答案，还需要看它们的“直觉”信号，而不能只看表面上的正确答案。
Abstract
Hidden-state readouts can predict whether language-model outputs are correct, but objective correctness (OC) usually agrees with the model's own self-judgement (SJ), leaving the decoded signal semantically ambiguous. We construct conflict cases in which OC and SJ predict opposite readout orderings. On high-confidence disagreements, conventional correctness-labelled contrasts often rank incorrect/self-endorsed responses above correct/self-rejected responses, following SJ rather than OC. We estimate factorial SJ- and OC-associated directions and evaluate their polarity across mathematical reasoning and factual recall. Across four instruction-tuned models up to 14B parameters, the SJ-associated direction transfers above chance in both cross-domain directions for every model, whereas the OC-associated direction has a below-chance point estimate for the expected OC ordering in every corresponding condition. This transfer asymmetry develops across middle-to-late layers, persists under answer-likelihood, sequence-length, and null-direction controls, and extends to MMLU and binary TruthfulQA without target-domain direction fitting. Across the studied models and diagnostic subsets, the most reliably transferable component preserves SJ-associated polarity. Transferability alone therefore does not establish objective-correctness semantics.