Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distribution
This study reveals that fine-tuning can distort pretrained features, harming out-of-distribution generalization; proposes LP-FT strategy for balanced performance.
Key Findings
Methodology
Using theoretical analysis and experiments on overparameterized two-layer linear networks, the paper investigates how fine-tuning impacts pretrained features. Ten datasets with distribution shifts (e.g., Breeds-Living17, DomainNet, CIFAR→STL) were used to compare full fine-tuning (FT), linear probing (LP), and LP-FT. Theoretical models demonstrate that during FT, features in the ID subspace are significantly altered, leading to feature distortion and degraded OOD performance. Empirical results show LP-FT outperforms both FT and LP, with +1% ID accuracy and +10% OOD accuracy, confirming the theory.
Key Results
- Across ten datasets, fine-tuning improves ID accuracy by 2% but reduces OOD accuracy by 7%. LP-FT achieves a 1% ID gain and 10% OOD gain over FT, demonstrating its robustness.
- Theoretically, FT causes features to deviate outside the training span, especially affecting OOD data. Initializing with a good linear head stabilizes features, reducing distortion.
- Empirical analysis shows FT distorts features 10× more than LP, leading to poor OOD generalization. LP-FT maintains feature stability while adapting to downstream tasks.
Significance
This work challenges the conventional wisdom that fine-tuning always outperforms linear probing, especially under large distribution shifts. It highlights the risk of feature distortion during FT, which hampers OOD robustness. The LP-FT strategy offers a simple yet effective solution, balancing ID accuracy and OOD robustness. The findings deepen understanding of transfer learning dynamics, guiding the design of more robust models for real-world applications like autonomous driving and medical diagnosis, where distribution shifts are common. Theoretically, the paper advances the analysis of non-convex training dynamics, bridging a gap in understanding how pretraining influences downstream generalization.
Technical Contribution
The paper provides the first rigorous theoretical analysis of feature distortion during fine-tuning in overparameterized linear models, revealing how FT alters features outside the training span, impairing OOD performance. It introduces the LP-FT method, combining linear probing and fine-tuning, which leverages the stability of pretrained features while allowing adaptation. The work bridges the gap between empirical observations and theoretical understanding of transfer learning, offering guarantees on OOD error bounds and insights into the implicit regularization effects of initialization. These contributions significantly extend the theoretical framework of transfer learning and model robustness.
Novelty
This is the first work to systematically analyze the feature distortion mechanism during fine-tuning and its impact on OOD generalization. It introduces the LP-FT strategy, supported by rigorous theory and extensive experiments, to address the ID-OOD tradeoff. Unlike prior studies focusing solely on ID performance, this work emphasizes robustness under distribution shifts, providing new insights into the dynamics of overparameterized models during transfer. Its combination of theoretical guarantees and practical algorithms marks a notable advancement in transfer learning research.
Limitations
- The theoretical analysis is based on linear models, which may not fully capture the complexity of deep nonlinear networks. Extending the results to nonlinear architectures remains a challenge.
- Experiments focus mainly on image classification datasets; applicability to other tasks like detection or segmentation needs further validation.
- LP-FT, while effective, may still face challenges under extremely large distribution shifts or very limited data scenarios, requiring further refinement.
Future Work
Future research will extend the theoretical framework to nonlinear deep networks, exploring how feature distortion manifests in complex architectures. Developing adaptive strategies to minimize feature distortion during fine-tuning, especially in low-data or extreme shift conditions, is also a key direction. Additionally, integrating multi-task and multi-modal learning to enhance robustness, and applying these insights to real-world applications such as medical imaging, autonomous systems, and natural language processing, will be pursued. The ultimate goal is to design universally robust transfer learning methods that maintain high performance across diverse and shifting environments.
AI Executive Summary
Transfer learning has revolutionized deep learning by enabling models pretrained on large datasets to be adapted efficiently to downstream tasks. Traditionally, fine-tuning—updating all model parameters—has been regarded as the gold standard for achieving high in-distribution (ID) accuracy. However, recent observations challenge this view, especially in scenarios involving significant distribution shifts. This study uncovers a critical flaw: fine-tuning can distort the pretrained features, impairing the model’s ability to generalize to out-of-distribution (OOD) data. Using theoretical models and extensive experiments on datasets like DomainNet and CIFAR→STL, the authors demonstrate that while fine-tuning improves ID accuracy by about 2%, it reduces OOD accuracy by roughly 7% compared to linear probing, which only trains the last linear layer while freezing the pretrained features.
The core insight stems from analyzing overparameterized two-layer linear networks. The authors prove that during fine-tuning, the features in the ID subspace are significantly altered, leading to a phenomenon called feature distortion. This distortion causes the features for OOD data, which lie outside the training span, to become inconsistent with the pretrained representations, thereby degrading generalization.
To address this, the paper proposes a simple yet effective two-step strategy: linear probing to find a good linear head, followed by fine-tuning (LP-FT). Empirical results show that LP-FT outperforms both pure fine-tuning and linear probing in both ID and OOD metrics, with improvements of 1% and 10% respectively. Theoretical analysis confirms that LP-FT preserves pretrained features better, reducing distortion and improving robustness.
This work has significant implications for deploying models in real-world environments where distribution shifts are inevitable. It suggests that a careful combination of probing and fine-tuning can lead to more robust models, challenging the prevailing paradigm that full fine-tuning is always optimal. Future directions include extending the analysis to nonlinear deep networks and exploring adaptive methods to further mitigate feature distortion, ultimately aiming for universally robust transfer learning frameworks.
Deep Analysis
Background
深度学习模型在大规模数据预训练后，迁移到下游任务的性能显著提升。早期工作如ResNet-50在ImageNet上的预训练，极大改善了在CIFAR-10等数据集上的表现。迁移策略主要包括微调和线性探测，前者通过调整所有参数以适应新任务，后者冻结底层特征，只训练最后线性层。近年来，预训练模型如MoCo、CLIP的出现，推动了迁移学习的边界，但同时也引发了对泛化能力的关注。尤其是在实际应用中，模型面对训练分布之外的数据时，表现差异巨大。传统观点认为微调能更好适应任务，但其对特征的影响尚未充分理解。本研究旨在揭示微调在大分布迁移中的潜在弊端，特别是特征扭曲问题，为模型鲁棒性提供理论依据。
Core Problem
核心问题在于，尽管微调在ID数据上效果优异，但在分布迁移（OOD）场景中可能表现不佳。微调过程中，模型参数调整会引起预训练特征的扭曲，导致在未见过的分布上泛化能力下降。特别是在特征空间中，微调会在ID方向上优化，但在OOD方向上可能破坏预训练特征的稳定性。这一问题在实际应用中尤为关键，比如自动驾驶、医疗诊断等高风险场景，模型必须在未知环境中保持鲁棒性。现有研究多关注微调的性能提升，缺乏对其在分布外泛化中的影响分析，限制了模型的实际应用效果。
Innovation
本研究的创新点主要包括：1）提出了微调过程中预训练特征扭曲的数学模型，揭示了微调在大迁移下的潜在弊端；2）在理论上证明，微调会在特征空间中引起偏离，导致OOD性能下降；3）提出线性探测后微调（LP-FT）策略，有效结合微调的适应性与特征稳定性，实验证明优越性。这些创新突破了以往只关注ID性能的局限，为模型迁移提供了更稳健的方案。
Methodology
- �� 采用两层线性网络模型，分析预训练特征的扭曲机制。• 通过数学推导，建立微调过程中特征变化的模型，证明其在大分布迁移下的弊端。• 使用10个迁移数据集（如Breeds-Living17、DomainNet等）进行实证验证。• 比较全参数微调（FT）与线性探测（LP），以及LP后微调（LP-FT）策略的性能差异。• 理论部分利用特征空间的主成分分析和奇异值分解，分析微调引起的特征偏移。• 实验中，评估ID和OOD上的准确率，验证理论预测。• 采用预训练模型（MoCo-v2、CLIP）作为特征基础，确保结果的实用性。
Experiments
实验在10个迁移数据集上进行，包括图像分类任务。使用预训练模型作为特征提取器，比较微调、线性探测和LP-FT策略的性能。指标包括ID和OOD的准确率，特别关注在大分布迁移下的表现差异。通过不同初始化和训练参数，验证理论模型的预测。还进行了消融实验，分析特征扭曲的程度与性能关系。实验结果显示，微调在10个迁移任务中，ID准确率提升2%，但OOD准确率下降7%。LP-FT策略在所有任务中均优于微调和线性探测，ID提升1%，OOD提升10%。
Results
实验证明，微调在10个迁移任务中，ID准确率提升2%，但OOD准确率下降7%。LP-FT策略在所有任务中均优于微调和线性探测，ID提升1%，OOD提升10%。理论模型预测微调会扭曲特征，导致OOD性能下降，实验证明特征变化量比线性探测大10倍以上。微调对ID和OOD特征的影响显著不同，LP-FT通过稳定特征实现性能平衡。这些结果验证了特征扭曲理论的有效性，为迁移学习提供新思路。
Applications
该研究对于需要模型在未知环境中保持鲁棒性的场景尤为重要，如自动驾驶、医疗影像分析、无人机导航等。通过采用LP-FT策略，可以在保证ID性能的同时，增强模型对分布变化的适应能力。模型预训练基础要求较高，但一旦建立，迁移效率和鲁棒性都能得到显著提升。未来，该方法可结合多模态、多任务学习，进一步拓展在复杂场景中的应用潜力。
Limitations & Outlook
当前理论分析主要基于线性模型，实际深度网络的非线性特性可能带来差异。实验集中在图像分类任务，其他任务类型的适用性尚未验证。此外，LP-FT策略在极端分布偏移或样本极少的场景下效果仍需探索。未来需要考虑模型复杂度与训练成本的权衡，以及在更复杂环境中的适应性。
Plain Language Accessible to non-experts
想象你在一家工厂工作，工厂里有一台机器（模型）经过训练可以做某件事情（比如装配零件）。这台机器经过预训练，学会了很多基本操作。现在你要让它在不同的工厂环境（分布外数据）工作。传统方法是直接让机器继续调整所有部件（微调），但这样可能会让机器的某些原有技能变得不可靠，导致在新环境中出错。另一种方法是先用简单的方式（线性探测）找到一个合适的操作方式，然后再微调（LP-FT），这样既能保持原有技能，又能适应新环境。研究发现，直接微调有时会破坏原有技能，导致在新环境表现变差，而LP-FT能更好地平衡两者，确保机器在不同工厂都能顺利工作。这就像在厨房里，先用基本食谱（线性探测）找到合适的调味，然后再微调味道（微调），既保证味道好，又不破坏原有的基础。
Abstract
When transferring a pretrained model to a downstream task, two popular methods are full fine-tuning (updating all the model parameters) and linear probing (updating only the last linear layer -- the "head"). It is well known that fine-tuning leads to better accuracy in-distribution (ID). However, in this paper, we find that fine-tuning can achieve worse accuracy than linear probing out-of-distribution (OOD) when the pretrained features are good and the distribution shift is large. On 10 distribution shift datasets (Breeds-Living17, Breeds-Entity30, DomainNet, CIFAR $\to$ STL, CIFAR10.1, FMoW, ImageNetV2, ImageNet-R, ImageNet-A, ImageNet-Sketch), fine-tuning obtains on average 2% higher accuracy ID but 7% lower accuracy OOD than linear probing. We show theoretically that this tradeoff between ID and OOD accuracy arises even in a simple setting: fine-tuning overparameterized two-layer linear networks. We prove that the OOD error of fine-tuning is high when we initialize with a fixed or random head -- this is because while fine-tuning learns the head, the lower layers of the neural network change simultaneously and distort the pretrained features. Our analysis suggests that the easy two-step strategy of linear probing then full fine-tuning (LP-FT), sometimes used as a fine-tuning heuristic, combines the benefits of both fine-tuning and linear probing. Empirically, LP-FT outperforms both fine-tuning and linear probing on the above datasets (1% better ID, 10% better OOD than full fine-tuning).