Inject, Align, Recover: Staged Post-Training for Retrieval-Free Document Knowledge Internalization
Proposes IAR three-stage framework for retrieval-free document knowledge internalization, boosting domain QA and general skills.
Key Findings
Methodology
IAR comprises Inject, Align, and Recover stages. Inject employs continuation, rewriting, and instruction-conditioned reconstruction to densely embed document knowledge into the model. Align fine-tunes the model with answer-only QA supervision to map knowledge to the QA interface. Recover uses model merging strategies (SLERP, TIES, DARE) to balance domain accuracy and general capability, mitigating catastrophic forgetting. Experiments across models (Llama, Qwen) and datasets (CC, CCI) demonstrate significant improvements in retrieval-free QA performance and robustness, with average gains of 3.6% in domain accuracy and 12.1% in general benchmarks.
Key Results
- Across 8 configurations and 7 models, IAR outperforms Vanilla SFT, achieving an average of 3.6 percentage points higher in domain QA accuracy and 12.1 points in general performance, surpassing CPT+SFT and parameter-efficient methods.
- In CC and CCI datasets, IAR consistently improves metrics, notably in Qwen-3-4B, with up to 9% gains over baseline, and effectively mitigates knowledge loss via model merging strategies.
- Model merging methods like TIES and DARE effectively balance knowledge retention and general ability, ensuring models maintain instruction-following while internalizing domain knowledge.
Significance
This work advances the field by systematically addressing the challenge of knowledge internalization without retrieval, crucial for privacy-sensitive, latency-critical, or retrieval-limited applications. The three-stage framework offers a flexible, scalable approach to embed and access knowledge internally, reducing reliance on external retrieval systems. Its demonstrated effectiveness across multiple models and datasets highlights its potential to reshape how large language models acquire and utilize knowledge, fostering more autonomous and efficient AI systems with broad industrial and academic impact.
Technical Contribution
The core innovation lies in decoupling knowledge injection, accessibility, and capability recovery into three distinct phases, each with tailored objectives and strategies. Inject enhances knowledge expression through dense supervised signals; Align ensures knowledge is accessible via QA fine-tuning; Recover employs advanced model merging techniques to preserve instruction-following and general skills. This structured approach contrasts with traditional monolithic fine-tuning or continual pretraining, providing a clear theoretical and practical framework for knowledge internalization, with detailed algorithms and hyperparameter tuning for optimal performance.
Novelty
This is the first comprehensive framework integrating structured document knowledge injection, QA alignment, and model merging for retrieval-free knowledge internalization. Unlike prior work focusing solely on continued pretraining or parameter-efficient fine-tuning, IAR emphasizes the importance of separating knowledge expression, accessibility, and capability recovery, offering a new paradigm for embedding large-scale document collections into language models without retrieval dependencies.
Limitations
- The multi-stage process increases training complexity and computational costs, potentially limiting scalability in resource-constrained environments.
- Effectiveness depends on careful hyperparameter tuning and merge strategies; unstable merging can lead to performance drops.
- In scenarios with rapidly changing knowledge or highly dynamic domains, static internalization may become outdated, requiring mechanisms for continual knowledge updates.
Future Work
Future research will explore dynamic knowledge updating, integrating continual learning and knowledge editing techniques. Extending the framework to multi-modal data (images, videos) could further enhance model understanding. Additionally, developing more efficient merging algorithms and reducing computational overhead will be key to broader deployment. Investigating adaptive strategies for knowledge refresh and domain adaptation remains an open avenue.
AI Executive Summary
Large language models (LLMs) have demonstrated remarkable capabilities in storing and retrieving factual knowledge, but their reliance on external retrieval systems poses limitations in latency, privacy, and robustness. Traditional fine-tuning methods like Vanilla SFT often result in sparse knowledge coverage and potential degradation of instruction-following abilities. Continuous pretraining (CPT) enhances understanding but does not directly improve question-answering (QA) performance on fixed corpora, especially when retrieval is unavailable.
This paper introduces a novel three-stage framework—Inject, Align, and Recover (IAR)—designed to internalize document knowledge into LLMs without relying on retrieval. The Inject stage employs dense supervision through continuation, rewriting, and instruction-conditioned reconstruction, enriching the model’s internal knowledge representation. The Align stage fine-tunes the model with answer-only QA supervision, ensuring that the injected knowledge becomes accessible during inference. The Recover stage utilizes advanced model merging techniques such as SLERP, TIES, and DARE to restore the model’s general capabilities while maintaining domain-specific knowledge.
Extensive experiments across multiple models (Llama, Qwen, Phi, SmolLM) and datasets (CC, CCI) demonstrate that IAR consistently outperforms vanilla supervised fine-tuning and comparable methods, achieving an average increase of 3.6% in domain QA accuracy and 12.1% in general benchmarks. Notably, IAR effectively mitigates catastrophic forgetting, balancing domain specialization with instruction-following skills.
The significance of this work lies in its systematic approach to knowledge internalization, offering a scalable, flexible solution that reduces dependence on retrieval systems. It paves the way for more autonomous AI systems capable of robust, fast, and privacy-preserving question answering. Future directions include dynamic knowledge updating, multi-modal integration, and more efficient merging strategies, promising further advancements in the field of knowledge-embedded language models.
Deep Analysis
Background
近年来，大规模预训练模型在知识存储和利用方面取得巨大突破（如GPT、BERT、Llama系列），但其知识主要通过参数存储，面临知识更新缓慢和表达有限的问题。检索增强生成（RAG）等技术引入外部存储，改善知识访问，但依赖检索器的效率和准确性。持续预训练（Gururangan et al., 2020）增强模型理解，但难以保证问答能力的保持。参数微调（LoRA、FAPM）等技术在保持能力的同时适应新知识，但存在知识表达稀疏的问题。本文试图突破单一微调的局限，通过系统化策略实现无检索知识内化，提升模型自主知识掌握能力。
Core Problem
核心问题在于如何在不依赖检索的情况下，将固定文档集合的知识有效内化到模型参数中。传统微调（SFT）覆盖面有限，难以覆盖全部文档信息；持续预训练虽能增强理解，但可能损失原有指令遵循能力。知识表达稀疏且难以访问，导致模型在实际应用中表现不足。特别是在知识更新频繁或知识范围超出训练数据时，模型的知识表达和能力恢复仍面临挑战。如何在保证通用能力的同时实现深度领域知识内化，成为亟待解决的问题。
Innovation
本研究提出分阶段策略：• Inject阶段通过续写、重写和指令条件重建，将文档知识密集注入模型，增强知识表达能力；• Align阶段利用只答题的监督，将知识映射到问答界面，确保知识的可访问性；• Recover阶段采用多策略模型合并技术，有效缓解灾难性遗忘，平衡领域性能与通用能力。这一体系化设计区别于传统单一微调或持续预训练，强调知识的结构化注入与能力的恢复，提供了理论上的操作框架和实践中的优化路径。
Methodology
- �� Inject：利用续写（预测后缀）、重写（知识重建）和指令条件重建（指令格式化）目标，将文档内容转化为监督信号，增强模型对文档的理解和表达能力。• Align：在注入知识后，使用只答题的监督（answer-only QA）微调模型，确保知识在问答任务中的可访问性。• Recover：从原始指令模型出发，结合模型合并策略（如SLERP、TIES、DARE），在验证集上选择平衡领域准确率和通用指标的模型，缓解知识内化带来的能力退化。整个流程通过多轮调优和模型合并，优化模型在无检索条件下的问答表现。
Experiments
在两个主要数据集（CC、CCI）上，采用多模型（Llama、Qwen、Phi、SmolLM）进行验证。对比基线包括Vanilla SFT、持续预训练（CPT+SFT）、参数微调（LoRA、FAPM）等。评估指标涵盖领域问答准确率、通用能力（IFEval、MMLU、MSBench）。实验中调整不同的Inject目标比例，验证Align的问答映射效果，以及Recover的模型合并策略。采用不同的超参数设置，确保结果的稳健性。通过多轮模型合并和验证，筛选出最优的操作点。
Results
IAR在7个模型、8个数据配置中，平均提升领域问答准确率3.6%，通用指标12.1%。在CC数据集，IAR优于Vanilla SFT，Qwen-3-4B模型提升最大，达9个百分点。模型合并策略（如TIES）有效缓解能力退化，保持指令遵循能力。不同模型和数据集的实验显示，结构化知识注入和模型合并策略是提升无检索问答性能的关键因素。整体来看，IAR在保持通用能力的同时，显著增强了领域知识的表达和访问能力。
Applications
该方法适用于需要在无检索条件下进行领域问答的场景，如企业内部知识库、隐私敏感环境、快速应答系统等。模型经过知识内化后，可直接部署，无需实时检索，提升响应速度和隐私保护。未来还可结合知识更新机制，实现模型的持续学习和动态知识维护，推动智能问答系统的自主知识掌握。
Limitations & Outlook
目前方法对模型计算资源要求较高，模型合并和多阶段训练增加了复杂度。极端知识更新场景下，模型的知识表达仍有限，存在知识过时或遗漏的风险。不同模型和数据集的适应性存在差异，部分合并策略效果不稳定，未来需优化算法和调参策略。
Plain Language Accessible to non-experts
想象你有一间厨房，平时用食谱做菜。以前，每次做新菜都得查书或网上找食谱，费时又不一定学到所有技巧。现在，你的厨艺老师告诉你一套特别的方法：第一步，把所有食谱都整理成一本手册（Inject），让你反复练习和理解；第二步，专门练习用这些食谱做菜（Align），确保你能用它们做出美味；第三步，把你的厨艺和老师的经验结合（Recover），让你既能用新食谱，又不失去原有的厨艺水平。这种方法让你在不用查资料的情况下，依然能做出好菜，还能保持厨艺的广泛性。
ELI14 Explained like you're 14
想象你在学校学新技能，比如弹吉他。以前，你得每天看教程，反复练习，但有时候会忘记一些技巧。现在，假如有个神奇的老师，他会先帮你整理所有的技巧（Inject），让你多练习；然后，他会帮你用这些技巧答题（Align），确保你知道怎么用；最后，他会把你的练习和老师的经验结合起来（Recover），让你既能弹得快，又不会忘记基础。这种方法让你不用一直看教程，也能变成弹吉他的高手，既快又稳。
Abstract
Large language models often fail to answer questions about a bounded document collection when the source documents are not retrieved at inference time. We study this setting as document knowledge internalization: converting a fixed corpus into usable parametric knowledge for retrieval-free question answering. We propose IAR (Inject, Align, and Recover), a three-stage post-training framework that separates structured document knowledge injection, QA behavior alignment, and general ability recovery. Unlike conventional continued pretraining, Inject converts source documents into continuation, rewrite, and instruction-conditioned reconstruction objectives. Align then adapts the injected model with answer-only QA supervision, while Recover merges the domain-adapted model with the base instruction model to recover general capabilities. Across Common Corpus (CC) and CCI, and across Llama, Phi, Qwen, and SmolLM model families, IAR improves the domain-primary domain-general frontier for retrieval-free document internalization. In the main comparison, IAR improves over Vanilla SFT on all four reported metrics in 7 of 8 dataset-model settings, with average gains of 3.6 percentage points in domain QA accuracy and 12.1 percentage points in mean general performance across IFEval, MMLU, and MSBench. Extended CC baselines show that LoRA and FAPM can win individual general metrics, but among methods that also reach leading or near-leading domain internalization, IAR retains one of the strongest general profiles.