Diversity-Oriented Fine-Tuning for Uncertainty-Based Hallucination Detection

The evolution of large language models (LLMs) such as GPT, LLaMA, and PaLM has significantly advanced NLP capabilities, enabling tasks like question answering, summarization, and dialogue generation. Despite these achievements, a persistent issue is the generation of hallucinations—plausible but factually incorrect outputs. Existing detection methods include external verification against knowledge bases, uncertainty estimation via token likelihoods, and representation analysis of hidden states. However, these approaches often struggle with models producing repetitive, low-entropy answers, which mask hallucinations and reduce detection sensitivity. Recent research emphasizes the importance of output diversity, but most techniques focus on inference-time sampling strategies like temperature scaling or nucleus sampling, which are limited by their reactive nature. This background underscores the need for training-stage solutions that embed diversity into the model's generative behavior, paving the way for more robust hallucination detection.

The core challenge lies in the tendency of large models to produce high-confidence, repetitive answers across multiple samples, resulting in low semantic entropy and false negatives in hallucination detection. This issue is exacerbated in open-domain settings where multiple valid answers exist, causing semantic entropy metrics to fragment and misclassify correct answers as uncertain. The fundamental bottleneck is the model's training bias towards high-confidence, low-variability outputs, which diminishes the effectiveness of entropy-based detection methods. Overcoming this requires mechanisms that actively promote output diversity during training, enabling the model to generate a broader range of semantically equivalent responses, thereby exposing inconsistencies indicative of hallucinations.

This work introduces two key innovations: 1) Multi-answer supervised fine-tuning (SFT), which constructs datasets with multiple semantically equivalent but lexically diverse responses, encouraging the model to produce varied answers; 2) Direct preference optimization (DPO), which trains the model to prefer diverse, semantically consistent answers over repetitive ones through pairwise comparison. These strategies are integrated into a unified training pipeline, systematically increasing output variability. Unlike prior methods that rely solely on inference-time diversity, this approach embeds diversity into the training process, providing theoretical guarantees for improved detection sensitivity. The combination of semantic clustering via bi-directional entailment and entropy measurement forms the core technical mechanism, enabling models to better reveal hallucinations.

• �� Construct diverse training datasets by iterative generation and semantic verification, ensuring multiple answers per question.
• �� In SFT, maximize the likelihood of all diverse answers conditioned on the prompt, promoting lexical and structural variation.
• �� In DPO, utilize pairwise preference data where the model learns to favor diverse, semantically aligned responses over repetitive, low-entropy answers.
• �� Combine SFT and DPO: first fine-tune with multi-answer data, then optimize with preference pairs to reinforce diversity.
• �� During training, adjust generation temperature and verify semantic consistency using entailment models.
• �� Evaluate by clustering generated answers with bi-directional entailment, computing semantic entropy, and measuring AUROC for hallucination detection.

Experiments involve three datasets—TriviaQA, SQuAD, NQ-Open—and models like LLaMA-3-8B and Qwen-2.5-7B. The training pipeline includes generating multiple diverse answers per question at high temperature (T=0.8) and a single reference answer at low temperature (T=0.05). Fine-tuning employs LoRA for parameter efficiency, with 4-5 epochs, learning rates around 1e-4 for SFT and 2e-6 for DPO. Baselines include unmodified models and models with standard SFT. Evaluation metrics focus on AUROC, comparing detection of hallucinations via semantic entropy scores. Ablation studies analyze the impact of diversity levels, training strategies, and dataset quality, validating the robustness and generalizability of the approach.

Results show that models fine-tuned with diversity strategies outperform baselines across all datasets, with AUROC improvements averaging 4.5%. The combined SFT+DPO approach achieves the highest scores, e.g., 0.839 AUROC on Qwen. Analysis reveals increased semantic divergence among multiple responses, especially in hallucination cases, leading to higher entropy scores and better discrimination. The models exhibit enhanced sensitivity to semantic inconsistencies, reducing false negatives. Comparisons with inference-time diversity methods confirm that training-stage diversity embedding yields superior detection performance, demonstrating the importance of proactive diversity promotion during training.

This method is directly applicable to AI safety, fact verification, and content moderation systems, where detecting false or hallucinated outputs is critical. It enables deploying more trustworthy LLMs in real-world scenarios like automated fact-checking, dialogue systems, and knowledge-based assistants. The training approach can be integrated into existing pipelines with minimal overhead, provided diverse datasets are available. Long-term, this strategy can facilitate the development of inherently safer models, reducing reliance on external verification and post-hoc filtering, thus broadening AI’s safe deployment in sensitive domains.

The approach may face challenges in open-domain settings with inherently high answer variability, risking false positives due to semantic fragmentation. Data collection for diverse answer sets is resource-intensive, limiting scalability. Additionally, in rare or complex factual scenarios, models may still produce low-entropy hallucinations, indicating the need for further refinement. Future work should focus on adaptive diversity control, external knowledge integration, and reducing annotation costs to enhance robustness and applicability.