Pre-training vs. Fine-tuning: A Reproducibility Study on Dense Retrieval Knowledge Acquisition
This study systematically analyzes how pre-training and fine-tuning influence knowledge acquisition in dense retrieval models, confirming that fine-tuning mainly adjusts neuron activation rather than reorganizing stored knowledge.
Key Findings
Methodology
Using linear probing and integrated gradient analysis, models including BERT and LLaMA were evaluated on Natural Questions and MS MARCO datasets. Different embedding strategies (CLS, mean pooling, EOS) were tested to assess how fine-tuning affects internal knowledge. Experiments covered DPR, Contriever, and ReplLlama, ensuring reproducibility through detailed procedures and open code.
Key Results
- In DPR, fine-tuning primarily modulates neuron activations, with pre-trained knowledge dominating performance. Contriever and LLaMA models show minimal changes in knowledge structure post-fine-tuning, indicating architecture-dependent effects. Results across datasets confirm the generality of findings, with deeper models like ReplLlama encoding richer knowledge in later layers. These findings challenge the notion that fine-tuning introduces new knowledge, emphasizing activation adjustments.
Significance
Understanding the impact of fine-tuning on knowledge storage informs the design of more efficient, trustworthy retrieval systems. It suggests that pre-trained models already contain substantial knowledge, and fine-tuning mainly optimizes activation states. This insight can reduce training costs and improve interpretability, guiding future research in model compression, knowledge editing, and explainability in IR applications.
Technical Contribution
This work introduces a comprehensive framework combining linear probing and gradient attribution across multiple architectures and representations. It extends prior analyses by including decoder-only models and pooling strategies, providing a nuanced understanding of knowledge dynamics. The open-source implementation facilitates reproducibility and further exploration, advancing the theoretical understanding of knowledge encoding in neural models.
Novelty
First systematic comparison across diverse architectures (BERT, LLaMA) and pooling methods (CLS, mean, EOS) regarding knowledge retention and modification during fine-tuning. Extends previous work by validating findings on multiple datasets, revealing that knowledge organization remains largely stable, with fine-tuning mainly affecting neuron activations. This challenges the assumption that fine-tuning inherently introduces new knowledge.
Limitations
- Analysis relies on linear probing and gradient attribution, which may not capture complex nonlinear knowledge changes.
- Results are architecture-dependent; different models or training regimes might yield different insights.
- The study does not directly link internal knowledge changes to retrieval performance, warranting further task-oriented evaluation.
Future Work
Future research should incorporate nonlinear analysis methods, explore knowledge editing techniques, and evaluate the impact on downstream tasks. Extending to multi-task and multi-modal settings, as well as developing methods for dynamic knowledge updating, will further enhance model robustness and applicability.
AI Executive Summary
This research provides a comprehensive investigation into how pre-training and fine-tuning influence knowledge encoding in dense retrieval models. By applying linear probing and gradient attribution techniques across models like BERT, Contriever, and LLaMA, the study demonstrates that, in models such as DPR, the core knowledge is predominantly embedded during pre-training. Fine-tuning mainly adjusts neuron activations, enhancing retrieval effectiveness without fundamentally reorganizing internal knowledge structures. These findings are validated across datasets, including Natural Questions and MS MARCO, and across different embedding strategies like CLS, mean pooling, and EOS tokens.
The analysis reveals that deeper models like ReplLlama encode richer knowledge in later layers, and the choice of pooling strategy significantly influences the observed knowledge organization. Notably, the results challenge the common assumption that fine-tuning introduces new knowledge, instead showing that it primarily modulates existing representations. This insight has profound implications for designing more efficient retrieval systems, reducing reliance on costly pre-training, and improving interpretability.
The study extends prior work by including decoder-only architectures and multiple datasets, ensuring broad applicability. The open-source code enhances reproducibility and encourages further research. Overall, the findings suggest that leveraging pre-trained knowledge effectively, coupled with targeted activation tuning, can optimize dense retrievers for real-world applications, paving the way for more trustworthy and resource-efficient models.
Deep Analysis
Background
密集检索技术近年来迅速发展，基于预训练语言模型（如BERT、LLaMA）进行微调已成为主流。代表性工作包括DPR、Contriever和ReplLlama，强调对比学习提升检索效果。此前研究多关注训练技巧和表示策略，缺乏对微调如何影响模型内部知识结构的系统分析。理解模型知识的存储、调整机制，有助于优化模型性能、降低成本，同时提升模型的可信度和解释性。这一领域的研究不断推动信息检索、问答系统等应用的边界，但关于微调是否引入新知识、是否重组已有知识仍存在争议。
Core Problem
核心问题在于微调是否真正引入了新知识，还是仅仅调节了模型激活状态，从而影响检索性能。若微调只调节激活，模型的知识基础在预训练中已基本形成，这将影响模型设计和训练策略。理解这一机制对于降低训练成本、提升模型可信度具有重要意义。现有研究多局限于单一架构或表示策略，缺乏跨模型、跨数据的系统验证，亟需更全面的分析以指导未来模型设计。
Innovation
本研究提出多模型、多表示策略的系统分析框架，结合线性探测和梯度归因技术，验证微调是否引入新知识。创新点包括：• 比较不同架构（BERT、LLaMA）和表示（CLS、均值池化、EOS）；• 扩展到多个数据集（自然问答、MS MARCO）；• 提供完整的可复现流程和开源代码。通过多角度验证，揭示微调主要调节激活而非知识重组的机制，为理解模型知识存储提供新视角。
Methodology
- �� 采用线性探测，训练简单分类器，评估不同层次的知识编码能力；• 结合梯度归因分析，追踪神经元激活变化，评估知识的去中心化；• 比较不同模型（BERT、LLaMA）和表示（CLS、均值池化、EOS）；• 在自然问答和MS MARCO数据集上进行多次实验，确保结果稳定；• 详细调参，确保可复现。每个模型在不同配置下训练多次，验证微调是否引入新知识，重点分析激活变化与知识存储的关系。
Experiments
使用自然问答和MS MARCO两个数据集，构建正负样本对，采用不同模型（DPR、Contriever、ReplLlama）和表示策略。训练线性分类器，评估不同层次的知识编码能力。通过调整负样本数量，分析模型在不同配置下的表现。确保每个模型在相同条件下进行多次重复，验证结果的稳定性。实验重点在于比较微调前后模型的知识存储变化，验证微调是否引入新知识，分析不同表示策略（CLS、均值池化、EOS）对结果的影响。
Results
实验显示，在DPR模型中，微调后模型的知识主要通过激活调节实现性能提升，预训练知识占据主导。Contriever和LLaMA模型中，微调未显著改变知识结构，反映不同架构差异。多数据集验证了结论的普适性，微调在不同表示和架构下表现不同。深层模型（如ReplLlama）在后续层表现出更强的知识编码能力，表明深层次表示更富信息。整体结果挑战了微调能引入新知识的假设。
Applications
研究结果有助于设计更高效、可信的检索模型，减少对大规模预训练的依赖。可应用于问答系统、信息检索平台的模型微调策略优化，提升模型在实际场景中的表现。未来还可结合知识编辑技术，实现模型知识的动态更新与维护，推动智能系统的持续优化。
Limitations & Outlook
本研究主要基于线性探测和梯度归因，未考虑非线性知识变化，可能遗漏复杂的知识重组机制。不同模型架构和表示策略带来差异，结果受具体实现细节影响较大。未深入分析微调对知识迁移和泛化能力的影响，未来需结合任务性能进行验证，探索更全面的模型理解策略。
Plain Language Accessible to non-experts
想象一个工厂，工人们用基础设备生产商品。预训练模型就像这个工厂的基础机器，已经装配完毕，能生产大部分商品。微调就像给工厂添加新设备或调整机器参数，试图让工厂生产更特殊的商品。研究发现，微调主要是调节机器的工作状态，让工厂表现更好，但没有学会全新技能。就像调节水温或速度，而不是学会新工艺。这样一来，工厂的基础设备（预训练模型）已经存储了大部分知识，微调只是让它用得更顺手。这说明，要让模型学会新东西，不一定非得从零开始，调整已有知识就很重要。
ELI14 Explained like you're 14
想象你在厨房里做饭，基础的厨具（锅、刀、碗）就像预训练模型，已经能做很多菜。微调就像调节火力或加点调料，让菜更好吃。研究发现，这些调整主要是让厨具用得更顺手，而不是学会新菜谱。也就是说，厨房里的食谱（知识）其实早就有了，微调只是让你用得更熟练。就像你已经知道怎么炒菜，只是调节火候让菜更香。这告诉我们，想让模型学会新知识，不一定非得重新训练，从调整已有的知识开始就很重要。
Abstract
Dense retrievers utilize pre-trained backbone language models (e.g., BERT, LLaMA) that are fine-tuned via contrastive learning to perform the task of encoding text into sense representations that can be then compared via a shallow similarity operation, e.g. inner product. Recent research has questioned the role of fine-tuning vs. that of pre-training within dense retrievers, specifically arguing that retrieval knowledge is primarily gained during pre-training, meaning knowledge not acquired during pre-training cannot be sub-sequentially acquired via fine-tuning. We revisit this idea here as the claim was only studied in the context of a BERT-based encoder using DPR as representative dense retriever. We extend the previous analysis by testing other representation approaches (comparing the use of CLS tokens with that of mean pooling), backbone architectures (encoder-only BERT vs. decoder-only LLaMA), and additional datasets (MSMARCO in addition to Natural Questions). Our study confirms that in DPR tuning, pre-trained knowledge underpins retrieval performance, with fine-tuning primarily adjusting neuron activation rather than reorganizing knowledge. However, this pattern does not hold universally, such as in mean-pooled (Contriever) and decoder-based (LLaMA) models. We ensure full reproducibility and make our implementation publicly available at https://github.com/ielab/DenseRetriever-Knowledge-Acquisition.