Detecting and Mitigating Hallucinations in Machine Translation: Model Internal Workings Alone Do Well, Sentence Similarity Even Better

The evolution of neural machine translation (NMT) has been driven by the advent of Transformer architectures (Vaswani et al., 2017), which significantly improved translation fluency and accuracy. Early methods relied on statistical models, but deep learning shifted the paradigm toward end-to-end neural systems. Despite these advances, hallucinations—outputs that deviate from the source—remain problematic, especially in high-stakes applications. Prior work, such as Lee et al. (2019) and Raunak et al. (2021), attempted to mitigate hallucinations via data augmentation and perturbation techniques, but these approaches often lacked robustness. Guerreiro et al. (2022) introduced a “detect-then-rewrite” framework, combining multiple heuristics and external quality estimation models like COMET, to identify and correct hallucinations. Meanwhile, internal interpretability methods, including Layer-wise Relevance Propagation (LRP) and attribution techniques like ALTI, have been explored to understand model decision processes. Cross-lingual sentence embeddings (LASER, LaBSE) have emerged as promising tools for semantic similarity assessment, enabling unsupervised quality checks. However, integrating these methods into a scalable, accurate hallucination detection system remains an open challenge, especially in diverse linguistic contexts and complex scenarios.

Despite progress, hallucination detection in NMT remains difficult due to their rarity and the subtlety of their manifestation. Existing methods rely heavily on external models trained on annotated data, which are costly and often fail to generalize across different error types. Moreover, these models struggle with partial or oscillatory hallucinations, which are more nuanced and harder to detect. The core issue is how to leverage the internal model signals—such as source token influence—to reliably identify severe hallucinations without external dependencies. This is crucial for deploying trustworthy translation systems in real-world applications, where false negatives can lead to misinformation, and false positives reduce system usability. The challenge is compounded by the diversity of hallucination types and the need for methods that are both accurate and computationally feasible.

This paper introduces a pioneering approach that utilizes layer-wise attribution (ALTI+) within Transformer models to quantify source contribution as an internal hallucination indicator. Unlike traditional reliance on sequence probabilities, this method offers a more direct measure of how much source information influences the output. Additionally, the integration of cross-lingual sentence embeddings (LASER, LaBSE) provides an unsupervised semantic similarity metric that detects content deviations without requiring annotated data. The “detect-then-rewrite” framework employs Monte Carlo dropout to generate multiple hypotheses, which are then reranked based on semantic similarity scores, effectively reducing hallucination rates. These innovations collectively enable high-precision, low-cost hallucination detection and correction, advancing the field toward more interpretable and reliable NMT systems.

• �� Compute layer-wise token attributions (ALTI+) within the Transformer to assess source contribution for each target token, averaging across tokens to obtain a global source influence score.
• �� Use pretrained cross-lingual sentence embedding models (LASER, LaBSE) to embed source and translation sentences into a shared semantic space.
• �� Calculate semantic similarity scores (cosine similarity, entailment probabilities) between source and translation embeddings as an unsupervised hallucination indicator.
• �� Generate multiple translation hypotheses via Monte Carlo dropout during inference, introducing stochasticity to capture model uncertainty.
• �� Rerank hypotheses based on LaBSE similarity scores, selecting the most semantically consistent translation.
• �� Evaluate detection performance using annotated datasets (Guerreiro et al., 2022), employing ROC AUC and precision-recall metrics to compare internal attribution-based and external embedding-based methods.
• �� Conduct ablation studies to analyze the contribution of each component, optimize hyperparameters, and validate robustness across hallucination types.

The experimental setup utilizes Guerreiro et al. (2022)’s German-English hallucination dataset, comprising 3415 manually annotated examples across various error types. Baseline methods include sequence log-probability, chrF, and external models like COMET and XNLI. The proposed internal attribution method (ALTI+) is compared against these baselines for detecting severe and partial hallucinations. Cross-lingual embeddings (LASER, LaBSE) are evaluated for their semantic similarity scores in unsupervised detection. Hypotheses are generated using Monte Carlo dropout with varying numbers of samples (up to 20), and reranked with LaBSE similarity. Metrics such as ROC AUC, PR@R90, and average COMET scores are used to quantify detection and mitigation performance. Ablation experiments assess the impact of each component, and cross-scenario tests evaluate generalization to different hallucination types.

The internal ALTI+ source contribution metric achieved ROC AUC of 84.9%, outperforming Seq-Logprob by over 2x in severe hallucination detection. LaBSE and XNLI embeddings improved recall at 90% precision by approximately 80%, with LaBSE providing a clear severity ranking among hallucination types. Combining hypothesis generation via Monte Carlo dropout with LaBSE reranking led to a 20% increase in translation quality (measured by COMET). The detection methods effectively identified fully detached and strongly detached hallucinations, with LaBSE outperforming LASER due to its training for translation ranking. The framework demonstrated robustness across error types, with significant improvements over previous external-only approaches.

The developed detection and mitigation strategies can be integrated into real-time translation pipelines, especially in high-stakes domains like legal, medical, and diplomatic translation, where accuracy is critical. The low reliance on external models reduces deployment costs and complexity. Additionally, the approach can serve as a post-processing step to filter or correct hallucinated outputs, enhancing user trust. Long-term, combining these techniques with adaptive learning and multimodal data could enable autonomous, trustworthy multilingual AI systems capable of self-monitoring and correction, transforming global communication and information dissemination.

The current approach may struggle with highly complex or context-dependent hallucinations, especially in multi-turn dialogues or long documents. High computational costs associated with large cross-lingual embedding models (e.g., LaBSE) limit scalability in resource-constrained environments. Detection accuracy for oscillatory or partial hallucinations, which are more subtle, remains imperfect, necessitating further refinement. Additionally, the reliance on annotated datasets for evaluation constrains the generalization to unseen error types or languages, highlighting the need for more diverse training data and unsupervised learning enhancements.