Prompt Embedding Probes (PEP): Hallucination Detection in LLMs from Hidden States
PEP introduces trainable prompt embeddings into frozen LLMs for improved answer-level hallucination detection, outperforming standard linear probes in in-distribution settings.
Key Findings
Methodology
PEP extends linear probes by inserting M trainable prompt embeddings into the input embeddings of a frozen LLM. During inference, prompts are optimized jointly with a linear classifier to detect hallucinations at specific layers and positions. This approach leverages prompt learning to enhance internal signal extraction without updating model weights. Experiments on TriviaQA, GSM8K, and MedQA with Qwen3 models (0.6B, 8B, 32B) demonstrate that PEP surpasses standard probes in in-distribution ROC AUC, maintains effectiveness in pre-generation and cross-model transfer, but faces challenges in out-of-distribution generalization.
Key Results
- Across datasets and model scales, PEP improves ROC AUC by 3-11 points over linear probes, with the largest gains in GSM8K and TriviaQA. In the TBG setting, PEP outperforms linear probes by 2-10 points, confirming robustness before answer completion. Cross-model transfer experiments show that detectors trained on one model's outputs perform best on their own, but transfer to other models drops, especially across datasets, indicating limited generalization. Hyperparameter analysis reveals that inserting 1-10 prompts near the middle layers and at sequence start yields optimal results.
Significance
This work advances white-box hallucination detection by integrating prompt learning into internal probing, enabling parameter-efficient, high-performance answer verification without modifying the backbone. It offers a practical solution for real-time, low-cost reliability monitoring in high-stakes applications, addressing a critical challenge in deploying trustworthy LLMs. The method bridges the gap between interpretability and efficiency, providing insights into internal representations and guiding future research in model safety and explainability.
Technical Contribution
PEP innovatively combines prompt tuning with linear probing, introducing a small set of trainable prompt embeddings that adapt internal signals for hallucination detection. Unlike traditional fine-tuning, PEP keeps the model frozen, reducing computational costs. It demonstrates that prompt embeddings can significantly improve internal signal extraction, leading to better detection accuracy across multiple datasets and models. This approach opens new avenues for lightweight, adaptable interpretability tools in large models.
Novelty
This is the first systematic integration of prompt learning into white-box internal probing for hallucination detection. Unlike prior work limited to neuron selection or simple linear probes, PEP employs trainable prompt embeddings to enhance internal signal utilization. This novel combination yields superior detection performance while maintaining minimal parameter updates, setting a new standard for efficient model interpretability and safety.
Limitations
- Cross-dataset transfer remains weak due to dataset-specific answer formats and task differences, limiting generalization outside training distributions.
- Training involves multiple forward passes and gradient updates, increasing computational cost compared to simple linear probes.
- Current focus on answer-level detection overlooks finer-grained claim or token-level hallucination detection, which is necessary for detailed interpretability.
Future Work
Future directions include developing multi-task training strategies to improve cross-dataset generalization, designing more robust prompt embedding schemes, and extending detection granularity to claim-level or token-level. Additionally, exploring more efficient training algorithms and integrating with external verification systems could further enhance practical deployment in safety-critical domains.
AI Executive Summary
Large language models (LLMs) have revolutionized natural language processing, enabling fluent and contextually rich text generation. However, hallucinations—confident but false outputs—remain a significant obstacle, especially in high-stakes fields like medicine and law. Existing detection methods often rely on external verification or response consistency checks, which are computationally expensive and not suitable for real-time deployment. To address this, the present study introduces Prompt Embedding Probes (PEP), a white-box approach that leverages internal model representations for hallucination detection.
PEP innovatively inserts a small set of trainable prompt embeddings into the input of a frozen LLM. These prompts are optimized jointly with a linear classifier to enhance the internal signals used for hallucination detection at specific layers and positions. This method combines the efficiency of prompt tuning with the interpretability of internal probing, requiring only a tiny fraction of trainable parameters compared to full model fine-tuning.
Extensive experiments on datasets like TriviaQA, GSM8K, and MedQA across multiple model scales demonstrate that PEP consistently outperforms standard linear probes, achieving improvements of up to 11 ROC AUC points in in-distribution settings. The method also maintains effectiveness in pre-generation scenarios and across different models, though cross-dataset transfer remains challenging. Hyperparameter analysis indicates that inserting prompts near the middle layers and at the sequence start yields optimal results.
These findings suggest that prompt-based internal probing offers a promising, scalable solution for real-time hallucination detection, with significant implications for deploying trustworthy LLMs in sensitive applications. While limitations in out-of-distribution generalization persist, this work paves the way for more robust, interpretable, and efficient model safety mechanisms in the future.
Deep Analysis
Background
近年来，LLMs在自然语言处理领域取得了突破性进展，代表如GPT、BERT、Qwen系列不断扩大规模，显著提升了文本生成的连贯性和复杂性。然而，幻觉问题依然严重，模型可能输出虚假或不支持的内容。早期方法多依赖外部验证或后处理机制，成本高且难以实现实时监控。白盒检测技术逐渐兴起，包括线性探测、神经元选择等，旨在利用模型内部表示实现更高效的幻觉识别。提示学习的出现为模型内部信号的利用提供了新思路，结合白盒检测成为研究热点，推动模型安全和解释性的发展。
Core Problem
核心问题在于如何在保持模型参数冻结的情况下，有效利用隐藏状态信息检测幻觉。传统线性探测在复杂任务中效果有限，难以捕获深层次信号。模型内部表示的多样性和任务差异导致检测泛化困难，尤其在跨数据集和跨模型迁移中表现不佳。此外，现有方法多依赖单一特征或复杂微调，成本高且难以部署。设计参数少、效果优的白盒检测方案成为亟待解决的难题。
Innovation
本研究提出PEP，将提示学习引入白盒检测框架，创新点包括：
1）引入少量可训练提示嵌入，增强隐藏状态的判别能力；
2）在保持模型参数冻结的基础上，优化提示嵌入和线性分类器；
3）在多任务、多模型、多尺度场景中验证其有效性。该方法突破传统线性探测的局限，通过提示学习实现更灵活的信号利用，显著提升检测性能，为模型解释和安全监控提供新工具。
Methodology
- �� 输入：将查询x和答案a编码为输入序列z。• 提示嵌入：初始化M个提示p(j)，每个大小为d。• 插入：将提示嵌入插入输入序列前端，形成增强序列˜E。• 传递：增强序列经过冻结模型，提取第ℓ层隐藏状态˜H(ℓ)，对应插入位置π(q)。• 分类：用线性分类器w对隐藏状态进行二分类，判断是否幻觉。• 训练：只优化提示嵌入和线性层参数，保持模型参数不变。• 评估：在TriviaQA、GSM8K、MedQA上验证不同提示数量、插入位置、层数的效果。
Experiments
采用TriviaQA、GSM8K、MedQA三大数据集，代表事实问答、数学推理和医学问答。模型包括Qwen3（0.6B、8B、32B）和Gemma3（12B）。指标为ROC AUC。训练中调整提示嵌入数量（1-10）、插入位置（开头、结尾、两端）、探测层（中间层、末层）。在多任务、多模型、多尺度场景中验证PEP的检测能力和鲁棒性，进行跨模型和跨数据集迁移实验。
Results
PEP在所有任务中均优于线性探测，平均提升ROC AUC达5-11个百分点，尤其在中大型模型中效果更佳。在预生成场景中，提升幅度为2-10个百分点。跨模型迁移中，模型特异性提示嵌入提升检测准确，但跨数据集迁移效果有限，提示泛化仍需改进。参数分析显示，插入位置偏好在模型中层附近，少量提示在序列开头效果最佳。
Plain Language Accessible to non-experts
想象你在厨房做饭，食材代表模型的知识，厨师代表模型本身。幻觉就像厨师误把假材料当成真食材，做出不真实的菜肴。为了避免这个问题，你可以在食材上贴标签（提示嵌入），帮助厨师辨别真假。PEP就像给厨师提供一小块标签，告诉它哪些食材可信，哪些可能是虚假的。这样，即使厨师不改变自己，只用这块标签，也能更好地判断菜是否安全。这个方法简单又高效，不需要重新训练厨师，只需在厨房里放点小标签，就能大大提高菜的质量和安全性。
ELI14 Explained like you're 14
想象你在学校写作文，老师告诉你要写得真实可信，但有时候你会写错或胡说八道。为了帮你写得更靠谱，老师给你一些提示，比如“写真话”。PEP就像老师给你一些特别的小贴纸，贴在作文里，提醒你哪些内容要特别小心。你不用重新学写作，只是在原有基础上加点提示，就能写得更靠谱。这就像给你的作文加了个“真话标签”，让老师更容易判断内容是否可信。这个方法简单又实用，能帮助你写出更真实的作文，避免胡说八道。
Abstract
Large language models (LLMs) can generate fluent and useful responses but remain prone to hallucinations. We introduce Prompt Embedding Probes (PEP), a white-box method for answer-level hallucination detection from the hidden states of a frozen LLM. PEP extends standard linear probes by augmenting the input with a small number of learnable prompt embeddings. We evaluate PEP on TriviaQA, GSM8K, and MedQA using Qwen3 models at multiple scales. PEP improves hidden-state-based detection over standard linear probes in the main in-distribution setting. We further evaluate PEP for pre-generation prediction, cross-model transfer, and out-of-distribution generalization. PEP remains effective in the pre-generation and cross-model settings, whereas robust cross-dataset transfer remains difficult. These results show that prompt-based adaptation can strengthen hidden-state probing while keeping the backbone frozen and adding only a small number of trainable parameters.