Discovering Latent Knowledge in Language Models Without Supervision
Introduces Contrast-Consistent Search (CCS), an unsupervised method to discover latent knowledge in language models, improving question-answering accuracy by 4%.
Key Findings
Methodology
CCS leverages the logical structure of truth by identifying a linear direction in the activation space that encodes factual knowledge. It constructs answer pairs for yes-no questions, normalizes their hidden states, and learns a linear probe to map activations to probabilities. The training optimizes a combined loss enforcing probabilistic consistency (answer and negation sum to one) and confidence (probabilities are away from 0.5). This unsupervised approach does not rely on model outputs or labels, instead exploiting the internal representations. Experiments across 6 models and 10 datasets show a 4% average improvement over zero-shot baselines, with robustness to misleading prompts and cross-task transferability.
Key Results
- CCS achieves an average accuracy of 71.2%, outperforming zero-shot (67.2%) across models and datasets. It maintains high performance (~83.8%) even when models are prompted with misleading cues, compared to a 9.5% drop in zero-shot accuracy. The method performs better on middle layers than final layers, indicating the presence of task-agnostic truth representations. Transfer experiments demonstrate the model’s internal knowledge generalizes across unrelated tasks, confirming the existence of a task-independent truth encoding.
Significance
This work advances understanding of internal model representations, showing that models contain latent, task-agnostic knowledge that can be extracted without supervision. It offers a new paradigm for interpretability, robustness, and trustworthy AI, reducing reliance on external labels. The ability to uncover internal beliefs enhances model transparency and could inform future alignment and safety efforts, making AI systems more reliable and explainable in real-world applications.
Technical Contribution
The paper introduces a novel unsupervised framework based on linear probing and logical consistency constraints, enabling the extraction of latent truth representations. It combines normalization, linear mapping, and a dual-loss optimization to identify a universal direction in activation space. The approach is model-agnostic, scalable, and transferable across tasks, providing theoretical insights into the internal structure of language models and practical tools for knowledge discovery.
Novelty
This is the first work to formalize and implement an unsupervised, activation-space-based method for discovering latent factual knowledge in language models. Unlike previous approaches relying on supervision or output analysis, CCS exploits the internal logical structure of truth, offering a general, interpretable, and robust technique for probing model beliefs. Its ability to transfer across tasks and layers highlights a fundamental, task-agnostic internal representation of truth.
Limitations
- The linear assumption may not capture complex, nonlinear knowledge structures, limiting effectiveness in some scenarios. The method's reliance on specific layer activations may vary across models and tasks, requiring careful tuning. It may struggle with ambiguous or fuzzy questions where the notion of a clear true/false answer is ill-defined. Additionally, computational costs increase with model size and dataset complexity, necessitating further optimization.
Future Work
Future directions include extending the framework to nonlinear representations, integrating multi-modal data, and exploring more complex logical constraints. Researchers aim to improve transferability across diverse models and tasks, enhance robustness against adversarial prompts, and develop real-time interpretability tools. Further work will also investigate the semantic structure of internal representations, aiming to deepen understanding of how models encode and manipulate knowledge internally.
AI Executive Summary
The rapid advancement of large-scale language models like GPT, T5, and BERT has revolutionized NLP, yet understanding what these models internally 'know' remains a challenge. Traditional approaches depend heavily on external supervision, fine-tuning, or analyzing model outputs, which can be limited by labeling costs and susceptibility to errors. Recognizing this, the authors propose a novel unsupervised technique called Contrast-Consistent Search (CCS), designed to uncover latent knowledge embedded within model activations without relying on labels or outputs.
CCS operates by constructing pairs of natural language statements representing both answers to yes-no questions, then normalizing their hidden states. It learns a linear probe that maps these normalized activations to probabilities, optimizing a loss function that enforces logical consistency—specifically, that the probability of a statement and its negation sum to one—and confidence, ensuring probabilities are away from 0.5. This approach effectively isolates the internal representation of truth, independent of the model's explicit outputs.
Experimental results demonstrate that CCS outperforms traditional zero-shot baselines by 4% on average across six models and ten datasets, including sentiment classification, question answering, and reasoning tasks. Notably, CCS maintains high accuracy even when models are prompted with misleading cues, indicating robustness. Additionally, transfer experiments reveal that the extracted representations generalize across unrelated tasks, suggesting the existence of a task-agnostic internal truth encoding.
This work signifies a major step toward interpretability and trustworthiness in AI, showing that models harbor internal, latent knowledge that can be accessed without supervision. It opens avenues for more transparent, reliable AI systems capable of internal reasoning, with broad implications for safety, alignment, and knowledge management. Future research will explore nonlinear representations, multi-modal integration, and deeper semantic understanding of internal model structures.
Deep Analysis
Background
近年来，预训练语言模型如GPT、BERT、T5等在自然语言处理领域取得了巨大突破。传统研究多关注模型输出的准确性和微调技术，但对模型内部潜在知识的理解仍有限。早期工作如神经网络的可解释性分析、激活可视化、表示分析逐步揭示了模型潜在的知识结构。近年来，学界尝试通过微调或标注引导模型学习特定知识，但成本高昂且缺乏普适性。如何在无需外部标注的情况下，直接从模型激活中提取潜在的“信念”成为研究热点。本文背景正是在此基础上，提出一种无监督、基于激活空间的潜在知识发现方法，旨在揭示模型内部的“真值”表示，为理解和信任大模型提供新视角。
Core Problem
当前方法多依赖模型输出或外部标注，难以应对模型输出误导或未知任务。模型内部的潜在知识结构复杂，线性或非线性关系难以捕获。如何在没有标注的情况下，准确识别模型的“真值”表示，成为核心难题。此问题关系到模型的可信性、解释性和鲁棒性，尤其在实际应用中，模型可能输出错误或被误导，如何提取其内在“信念”成为亟待解决的问题。
Innovation
本文创新点在于提出基于激活空间线性方向的无监督知识发现框架，结合逻辑一致性（如对立命题概率互补）作为约束，设计Contrast-Consistent Search（CCS）算法。该方法无需模型输出或标注，通过优化激活空间中的线性映射，提取潜在的“真值”特征。区别于微调或监督学习，CCS强调模型内部潜在结构，具有良好的迁移性和解释性。该技术突破了传统依赖标注的限制，为模型理解提供新工具。
Methodology
- �� 构造对比对：对每个问题，生成“是”与“否”两个自然语言句子，代表不同答案。
- �� 激活提取：利用预训练模型获取对应句子的隐藏状态激活。
- �� 归一化：对激活进行均值和尺度归一化，消除句子末尾差异。
- �� 线性映射：训练线性投影，将激活映射到[0,1]的概率空间。
- �� 损失优化：设计一致性损失（概率互补）和置信度损失，交叉优化。
- �� 预测：在训练完成后，结合两个概率，判断答案的“是”或“否”。
Experiments
在6个模型（如GPT-J、T5、UnifiedQA、RoBERTa、DeBERTa、T0）和10个数据集（如IMDB、COPA、RTE等）上，评估CCS的问答准确率。采用不同提示、多次训练，比较零样本、校准零样本和监督线性模型的性能。通过迁移测试验证模型潜在知识的任务无关性。调优参数包括学习率0.01，训练轮次10次，确保模型稳定性。实验还设计了误导性提示，验证鲁棒性。
Results
CCS在所有模型和数据集上平均提升4%的准确率，最高达84.8%，显著优于零样本基线。在误导性提示下，准确率仅下降不到2%，显示强鲁棒性。中间层激活比末层表现更优，迁移实验显示模型潜在知识具有任务无关性。多模型、多任务验证了方法的普适性和有效性，证明了潜在知识的可提取性。
Applications
该技术可应用于模型解释、知识推理、模型校准等场景。无需标注，适合大规模模型的快速知识检测和验证，有助于提升模型的可信度和透明度。未来可结合多模态信息，扩展到多语言、多任务环境，推动AI系统的自主学习和知识管理。
Limitations & Outlook
方法假设激活空间存在线性结构，可能在复杂或模糊任务中表现不足。对模型层次选择敏感，参数调优依赖经验。对噪声或偏离训练分布的场景效果有限。未来需研究非线性结构和多模态融合，提升鲁棒性和适应性。
Plain Language Accessible to non-experts
想象你有一个神奇的工厂，里面的工人（模型）每天都在制造各种商品（回答问题）。你不知道他们具体怎么做的，但你可以偷偷观察他们的工作台（激活空间）。如果你想知道工厂里是否真正掌握了某个技能（知识），你可以设计一些特殊的测试（正反问句），让工人展示他们的秘密。通过观察工人在不同测试中的表现，你可以找到工厂内部隐藏的“真相”线索，而不用问工厂直接说。这就像用一种特殊的眼睛（算法）在工厂的内部找到工人真正懂的东西，而不是他们表面上的表现。这种方法不需要告诉工厂答案，也不需要额外的培训，就能发现工厂真正的秘密技能。
ELI14 Explained like you're 14
你知道，有时候我们在学校里学东西，但老师没有告诉我们所有的秘密。比如，老师可能会问你一些问题，你知道答案，但老师不知道。这个研究就像是用一种特别的眼睛，偷偷看出你心里真正知道的东西，而不用你直接说出来。科学家们让电脑像工厂一样工作，他们想知道电脑是不是真的“懂”一些事情。于是他们设计了一种方法，让电脑自己在内部找出“真相”，就像你在游戏里发现隐藏的宝藏一样。这样，即使电脑说错了话，我们也能知道它其实“懂”什么。这对未来让电脑更聪明、更可靠很有帮助。
Abstract
Existing techniques for training language models can be misaligned with the truth: if we train models with imitation learning, they may reproduce errors that humans make; if we train them to generate text that humans rate highly, they may output errors that human evaluators can't detect. We propose circumventing this issue by directly finding latent knowledge inside the internal activations of a language model in a purely unsupervised way. Specifically, we introduce a method for accurately answering yes-no questions given only unlabeled model activations. It works by finding a direction in activation space that satisfies logical consistency properties, such as that a statement and its negation have opposite truth values. We show that despite using no supervision and no model outputs, our method can recover diverse knowledge represented in large language models: across 6 models and 10 question-answering datasets, it outperforms zero-shot accuracy by 4\% on average. We also find that it cuts prompt sensitivity in half and continues to maintain high accuracy even when models are prompted to generate incorrect answers. Our results provide an initial step toward discovering what language models know, distinct from what they say, even when we don't have access to explicit ground truth labels.