Stick to What You Know: A Study of Knowledge-Aligned Supervised Fine-Tuning
Knowledge-aligned SFT reduces hallucinations, enhancing factuality.
Key Findings
Methodology
The study introduces knowledge-aligned supervised fine-tuning (SFT) by constraining training targets to the base model's parametric knowledge, reducing hallucinations. It compares generation and estimation alignment methods, introducing Evidence Rewrite and Recall Rewrite.
Key Results
- Recall Rewrite significantly reduces hallucinations on WildHalu and Biography datasets, achieving support rates of 84.2% and 56.2%, respectively.
- Evidence Rewrite maintains high support rates while reducing hallucinations, especially on the Biography dataset.
- Compared to standard SFT, knowledge-aligned methods improve refusal behavior on UnknownBench.
Significance
This research significantly reduces factual hallucinations in language model generation through knowledge-aligned SFT. It holds importance for both academia and industry, particularly in applications requiring high-confidence responses.
Technical Contribution
Introduces two new knowledge alignment methods: Evidence Rewrite and Recall Rewrite, offering new pathways to reduce hallucinations. These methods significantly enhance factuality while maintaining model capabilities compared to existing methods.
Novelty
First to introduce knowledge alignment in SFT, reducing hallucinations by constraining training targets within the base model's known range. This approach fundamentally differs from traditional SFT methods.
Limitations
- The method relies on the base model's parametric knowledge, potentially unable to handle entirely unknown information.
- In some cases, it may lead to excessive refusal behavior, impacting user experience.
Future Work
Future research could explore integrating external knowledge bases to further enhance factuality and evaluate application effects across different domains.
AI Executive Summary
Current large language models often exhibit factual hallucinations, where generated content does not align with real-world facts. This issue is particularly problematic in applications requiring high-confidence responses. Existing supervised fine-tuning methods, while improving model instruction-following capabilities, may also lead to hallucinations.
This paper proposes a knowledge-aligned supervised fine-tuning method by constraining training targets within the base model's parametric knowledge to reduce hallucinations. The study compares generation and estimation alignment methods, introducing two new variants: Evidence Rewrite and Recall Rewrite. Experimental results demonstrate that these methods significantly reduce hallucinations on WildHalu and Biography datasets while maintaining overall model capabilities.
Despite these advancements, the method has limitations, such as limited handling of entirely unknown information. Future research could explore integrating external knowledge bases to further enhance factuality.
Deep Analysis
Background
Large language models often generate hallucinations, where content does not align with real-world facts. This issue is particularly severe in applications requiring high-confidence responses. Existing supervised fine-tuning methods, while improving model instruction-following capabilities, may also lead to hallucinations.
Core Problem
The core problem is reducing hallucinations in language model generation. Existing SFT methods may lead models to generate content beyond their knowledge scope, resulting in hallucinations.
Innovation
This paper proposes knowledge-aligned SFT by constraining training targets within the base model's parametric knowledge to reduce hallucinations. It introduces two new variants: Evidence Rewrite and Recall Rewrite, offering new pathways to reduce hallucinations.
Methodology
- �� Evidence Rewrite: Verifies base model generations using external evidence.
- �� Recall Rewrite: Retains claims only when they can be consistently recalled by the model.
- �� Compares generation and estimation alignment methods.
Experiments
Experiments were conducted on Qwen 3 4B and OLMo 3 7B models using WildHalu and Biography datasets. The study evaluated the performance of different methods in reducing hallucinations and maintaining model capabilities.
Results
Recall Rewrite significantly reduces hallucinations on WildHalu and Biography datasets, achieving support rates of 84.2% and 56.2%. Evidence Rewrite maintains high support rates while reducing hallucinations.
Applications
This method can be applied in fields requiring high-confidence responses, such as medical diagnosis and legal consultation.
Limitations & Outlook
The method relies on the base model's parametric knowledge, potentially unable to handle entirely unknown information. In some cases, it may lead to excessive refusal behavior, impacting user experience.
Plain Language Accessible to non-experts
Imagine you're cooking in a kitchen. You have a cookbook (the model's knowledge), but some recipes are unfamiliar (unknown information). To avoid making a bad dish (hallucination), you decide to only cook dishes you're familiar with (knowledge alignment). If you're unsure about a recipe, you consult the cookbook or ask others (evidence rewrite). This approach ensures every dish you make is within your knowledge range.
ELI14 Explained like you're 14
Imagine you're playing a game, and your character has specific skills (the model's knowledge). When you encounter a task, you can only use your character's skills to complete it (knowledge alignment). If you're unsure how to complete a task, you can check the game guide or ask other players (evidence rewrite). This way, you won't make mistakes in the game!
Glossary
Supervised Fine-Tuning
A training method that adjusts a model's behavior by providing specific input-output pairs.
Used to transform a pre-trained base model into an instruction-following model.
Hallucination
The phenomenon where a model generates content that does not align with real-world facts.
A common issue in large language model generation.
Knowledge Alignment
A method that constrains training targets within the model's known range.
Used to reduce hallucinations.
Evidence Rewrite
A method that verifies base model generations using external evidence.
Ensures the factuality of generated content.
Recall Rewrite
A method that retains claims only when they can be consistently recalled by the model.
Used to reduce hallucinations.
Open Questions Unanswered questions from this research
- 1 How can hallucinations be further reduced without increasing refusal rates?
- 2 How can external knowledge bases be integrated to enhance factuality?
Applications
Immediate Applications
Medical Diagnosis
Doctors can use this method to ensure the accuracy of diagnostic advice, reducing the risk of misdiagnosis.
Long-term Vision
Legal Consultation
Can be used in legal consultation to ensure advice aligns with current laws, reducing misleading information.
Abstract
Supervised fine-tuning (SFT) trains a base language model to imitate target responses, and these targets may require knowledge the base model has not robustly internalized. We study this as a source of hallucinations and frame a group of mitigation methods as \emph{knowledge-aligned SFT}: constraining SFT training targets to the base model's parametric knowledge. Under a unified setup, we compare existing generation-based and estimation-based knowledge-alignment methods and introduce two new variants: Evidence Rewrite, which verifies base-model generations using external evidence, and Recall Rewrite, which retains claims only when they can be consistently recalled by the base model. Experiments with Qwen 3 4B and OLMo 3 7B show that knowledge-aligned SFT can reduce factual hallucinations on WildHalu and Biography while largely preserving general capabilities. Recall Rewrite yields the strongest factuality gains and improves refusal behavior on UnknownBench. It thereby confirms that SFT targets beyond the base model's knowledge drive hallucination behavior.