CrossHallu: Do Hallucination Signals Generalize Across Languages and Domains in Large Language Model's Internals?
CrossHallu evaluates internal signals' cross-lingual and cross-domain generalization in LLMs, showing most models transfer hallucination signals effectively.
Key Findings
Methodology
This study extends HalluShift, extracting hidden states, attention, and token probabilities from six LLMs during generative QA. Features include layer-wise Wasserstein distances, cosine similarity, max/min probabilities, and entropy. A lightweight MLP classifier trained on GPT-4o-labeled data evaluates hallucination signals across monolingual, cross-lingual, cross-domain, and combined scenarios. Datasets include TruthfulQA, its Arabic translation, and HalluScore. The framework systematically compares transfer performance, emphasizing feature stability and layer-wise dynamics.
Key Results
- Most models, especially multilingual ones like Aya, demonstrate strong cross-lingual signal transfer, with AUROC reaching 82.86%. Cross-domain transfer within Arabic datasets shows variable performance, with models trained on cleaner datasets like TruthfulQA AR performing better. In combined cross-lingual and cross-domain settings, Aya achieves AUROC of 75.57%, indicating internal signals' robustness across both language and domain shifts.
- Multilingual models outperform Arabic-specific models in cross-lingual transfer, supported by visualization of embedding overlaps. Deep transformer layers exhibit more stable signals, with feature analysis revealing that deeper layers encode more language-agnostic and domain-invariant information, crucial for generalization.
- Feature analysis highlights that layer-wise Wasserstein distances and cosine similarities effectively distinguish hallucinated from factual responses, with certain features like entropy and probability spread being key indicators. These insights guide future model design for improved internal signal transferability.
Significance
This work advances understanding of internal model signals, demonstrating their potential to generalize across languages and domains. It addresses critical challenges in deploying LLMs in multilingual, real-world settings, especially for high-stakes applications requiring factual accuracy. By validating the language-agnostic nature of internal hallucination signals, it paves the way for more reliable, scalable hallucination detection methods, reducing reliance on surface-level heuristics and enhancing trustworthiness in AI systems. The findings also inform future model training strategies to foster more universal internal representations, benefiting both academia and industry.
Technical Contribution
The paper introduces a comprehensive evaluation framework combining multi-layer feature extraction, similarity measures, and GPT-4o-based labeling, enabling systematic transferability assessment. It innovates by integrating multi-modal features, analyzing layer-wise dynamics, and validating signals across six diverse models. The approach offers theoretical insights into the internal mechanisms of hallucination, providing a foundation for future research on internal signal robustness and universality. It also demonstrates practical utility in improving hallucination detection in multilingual and multi-domain contexts.
Novelty
This is the first systematic study to evaluate the cross-lingual and cross-domain transfer of internal hallucination signals in large models. It combines multi-modal feature analysis with GPT-4o factual assessment, moving beyond surface heuristics. The research reveals that deep transformer layers encode language-agnostic signals, a novel insight that challenges prior assumptions about language-specific internal representations and opens avenues for universal hallucination detection.
Limitations
- The reliance on GPT-4o for factual labeling introduces potential biases and limits the scalability of labeling in diverse languages. The simplicity of the classifier may restrict capturing complex signal patterns, and the current focus on Arabic and English leaves other low-resource languages untested. Additionally, the study's scope is limited to generative QA, requiring validation in broader NLP tasks. Future work should explore more sophisticated models, larger multilingual datasets, and multi-modal signals to enhance robustness.
Future Work
Future directions include integrating multi-modal data (visual, audio) to enrich internal signals, employing self-supervised learning for more robust feature representations, and extending evaluations to additional languages and tasks. Developing adaptive, multi-task models that learn universal signals across domains and languages could further improve generalization. Additionally, exploring explainability techniques to interpret internal signals will deepen understanding of model hallucinations, ultimately leading to more trustworthy AI systems.
AI Executive Summary
Large Language Models (LLMs) have revolutionized natural language processing, yet hallucination remains a persistent challenge, especially when deploying models across multiple languages and domains. Existing detection methods often rely on surface features like confidence scores, which lack robustness and fail to capture the internal dynamics of models. To address this, CrossHallu introduces a systematic framework for evaluating whether internal signals—such as hidden states, attention patterns, and token probabilities—can reliably indicate hallucinations across languages and tasks.
The core methodology involves extracting multi-layer features from six diverse LLMs during generative question-answering, training a lightweight classifier on GPT-4o-labeled data, and testing across monolingual, cross-lingual, and cross-domain scenarios. The datasets include the English TruthfulQA, its Arabic translation, and HalluScore, covering both high-resource and low-resource settings. Results demonstrate that most models, particularly multilingual ones like Aya and Phi4-mini, transfer hallucination signals effectively, with AUROC scores exceeding 75% in challenging transfer scenarios.
Significantly, deep transformer layers exhibit more stable, language-agnostic signals, supporting the hypothesis that internal representations can generalize beyond training languages. Visualization of embedding spaces confirms that multilingual models develop shared semantic spaces, facilitating transfer. These insights have profound implications for deploying reliable, multilingual AI systems in real-world applications such as content moderation, medical diagnosis, and legal analysis.
Despite promising results, limitations include dependence on GPT-4o for labeling and the need for more diverse datasets. Future work will explore multi-modal signals, larger multilingual corpora, and explainability techniques to further improve robustness and interpretability. Overall, this research marks a crucial step toward universal hallucination detection, enhancing the trustworthiness and safety of large language models in complex, multilingual environments.
Deep Analysis
Background
The evolution of large language models (LLMs) like GPT-4, LLaMA, and PaLM has significantly advanced NLP capabilities, enabling high-quality text generation, summarization, and question-answering. Early efforts focused on surface-level confidence heuristics, but these proved insufficient for high-stakes applications due to hallucinations—confidently generated but factually incorrect outputs. Recent research shifts toward internal representations, leveraging attention patterns, hidden states, and token probabilities to detect hallucinations more reliably. Notable works include Liu et al. (2025) on internal feature-based detection and Dasgupta et al. (2025) on the HalluShift framework. However, most studies are limited to English, with scarce validation across languages and domains, especially in low-resource settings. As multilingual models like mT5 and Aya emerge, understanding whether internal signals are language-agnostic becomes critical for scalable, reliable AI deployment.
Core Problem
The key challenge is whether hallucination signals derived from internal model states can transfer across languages and domains. Existing methods lack systematic validation in multilingual, multi-domain contexts, risking unreliable detection in real-world scenarios. This is particularly problematic for low-resource languages where data scarcity hampers training. The core bottleneck lies in the assumption that deep internal representations encode universal signals, yet this remains unverified. Addressing this gap requires rigorous cross-lingual and cross-domain evaluation, understanding how features evolve across layers, and whether models develop shared semantic spaces that support transfer. Solving this problem is vital for deploying trustworthy, multilingual AI systems in diverse applications.
Innovation
The study introduces a comprehensive evaluation framework for internal hallucination signals, combining multi-layer feature extraction, similarity metrics, and GPT-4o-based factual labeling. It innovates by systematically testing six models across monolingual, cross-lingual, cross-domain, and combined transfer scenarios, revealing the layer-wise stability of signals. The integration of multi-modal features and visualization of embedding spaces offers new insights into the internal dynamics of multilingual models. Unlike prior work limited to surface heuristics, this approach emphasizes the universality and robustness of internal signals, providing a foundation for scalable hallucination detection in multilingual, multi-domain settings.
Methodology
- �� Extract hidden states, attention tensors, and token probabilities from six LLMs during QA generation.
- �� Select specific transformer layers (even layers for hidden states, odd layers for attention) to capture diverse internal dynamics.
- �� Compute layer-wise Wasserstein distances and cosine similarities to measure representation shifts.
- �� Derive statistical features such as max/min token probabilities, entropy, low-probability counts, and probability gradients.
- �� Normalize features using StandardScaler or QuantileTransformer to address language-specific distribution differences.
- �� Train a fixed three-layer MLP classifier on GPT-4o-labeled data, evaluating in monolingual, cross-lingual, cross-domain, and combined transfer scenarios.
- �� Visualize feature spaces with t-SNE to analyze internal representation alignment and separability.
- �� Perform ablation studies to identify key features and layer contributions, ensuring robustness across models and datasets.
Experiments
The experimental setup involves training classifiers on monolingual datasets (TruthfulQA English and Arabic, HalluScore), then testing in cross-lingual (English↔Arabic), cross-domain (TruthfulQA↔HalluScore), and combined scenarios. The datasets are split into 75% training and 25% testing, with careful feature normalization to mitigate language distribution differences. Multiple models, including Allam, Silma, Phi4-mini, Ministral, Qwen2.5, and Aya, are evaluated using AUROC, F1-score, and PR-AUC. The experiments include ablation analyses on feature importance, layer-wise stability, and visualization of embedding overlaps, providing comprehensive insights into transferability and internal signal robustness.
Results
Results show that deep transformer layers encode hallucination signals that transfer effectively across languages and domains, especially in multilingual models like Aya, with AUROC exceeding 75% in challenging transfer scenarios. Cross-lingual transfer performance degrades compared to monolingual baselines but remains significant, indicating partial language-agnostic internal representations. Cross-domain transfer within Arabic datasets is more consistent, with models trained on cleaner data (TruthfulQA AR) performing better on HalluScore. Visualization confirms that multilingual models develop shared semantic spaces, facilitating transfer, while Arabic-specific models show more language bias. Feature analysis highlights the importance of layer-wise Wasserstein distances and entropy measures in distinguishing hallucinations.
Applications
This framework can be applied to improve factual accuracy in multilingual chatbots, content moderation systems, and AI assistants operating in diverse linguistic environments. It enables real-time hallucination detection without additional inference costs, suitable for deployment in high-stakes domains like healthcare, legal, and finance. The insights into internal signal transfer also guide model training and architecture design, promoting more robust, generalizable AI systems capable of functioning reliably across languages and tasks.
Limitations & Outlook
The reliance on GPT-4o for labeling introduces potential biases and limits scalability, especially for low-resource languages. The current feature set and classifier are relatively simple, possibly missing complex internal dynamics. Experiments are confined mainly to Arabic and English, requiring validation in additional languages. The study focuses on QA tasks, and extending to other NLP tasks like summarization or translation remains future work. Computational costs for feature extraction and evaluation may also pose challenges for large-scale deployment.
Plain Language Accessible to non-experts
想象你在学校里，有很多不同的班级，每个班级都有自己的老师和学生。有时候，老师会讲错题或者误解学生的意思，但他们会用一些特别的线索，比如老师的语气、回答的速度、或者他们说话时的表情，来判断老师是否理解错了。科学家们也在做类似的事情，他们让计算机像老师一样，观察模型内部的“信号”——比如它们在回答问题时的“注意力”或“信心”——来判断它们是否“胡说八道”。
这些“信号”就像老师的线索，告诉我们模型是不是在出错。研究发现，不同语言的模型内部信号其实很相似，就像不同班级的老师用的线索一样。这意味着，我们可以用一种方法检测英语、阿拉伯语甚至其他语言模型的“胡说八道”。这就像是给老师们装上了“智能眼镜”，让他们自己知道什么时候说错话。这样，我们就能让这些模型变得更可靠，能在学校、医院、法律等重要地方帮忙，减少错误，变得更聪明、更值得信赖。
ELI14 Explained like you're 14
你知道，现在的智能机器人可以帮你回答问题、写作文，但有时候它们会胡说八道，给出不正确的答案。科学家们想知道，机器人内部是不是有一些“秘密信号”，能告诉我们它们是不是在胡说八道。就像你玩游戏时，游戏会有一些提示，告诉你是不是在作弊或者出错。研究人员用一种叫CrossHallu的方法，从机器人“脑袋”里的秘密通道里提取信息，比如它们的“注意力”或“自信心”，然后用一个简单的“判断器”来判断这些信息是否显示机器人在“胡说八道”。
他们发现，不同语言的机器人，内部的信号其实很像，就像不同国家的老师用的线索一样。这意味着，我们可以用一种方法检测英语、阿拉伯语甚至其他语言的机器人是否在说谎。这就像是给机器人装上了“侦探眼镜”，让它们自己知道什么时候出错。这样一来，机器人就能变得更聪明、更可靠，可以在学校、医院、法律等重要场合帮忙，大家都能更信任它们！
Abstract
Recent hallucination detection techniques in large language models (LLMs) focus on directly extracting features from a model's internal representations and training a classifier on these features to detect hallucinations, demonstrating promising results. Notwithstanding this advancement, most internal-state hallucination detection techniques have been explored predominantly in English, raising the question of whether such internal signals generalize across different languages and domains. To address this gap, we present CrossHallu, the first study to evaluate the cross-lingual and cross-domain generalization of hallucination detection using internal representations from six LLMs on the generative question-answering task. We conduct a systematic Arabic <-> English evaluation using TruthfulQA, an Arabic translated version of TruthfulQA, and HalluScore. This evaluation encompasses monolingual training and testing, cross-lingual transfer, cross-domain transfer, and combined cross-lingual and cross-domain transfer. The results reveal that internal-state hallucination signals in LLMs transfer across languages and domains for most models, with cross-lingual performance highly dependent on both class separability and language alignment in the feature space, whereas cross-domain transfer within Arabic varies depending on the training and testing datasets used for the hallucination detector. The code is publicly available at https://github.com/aishaalansari57/CrossHal.