SR-init: An interpretable layer pruning method
SR-init assesses layer redundancy by measuring accuracy drop after stochastic re-initialization, enabling interpretable pruning with significant parameter reduction.
Key Findings
Methodology
The approach involves randomly re-initializing each layer of a pre-trained model using Kaiming initialization, then fixing other layers. The accuracy drop on validation data quantifies layer sensitivity. Layers with minimal accuracy decline are deemed redundant. This process includes: • Randomly resetting layer parameters with Kaiming method; • Fixing other layers; • Evaluating model accuracy; • Calculating accuracy difference as a redundancy indicator; • Pruning layers below a threshold. Feature visualization confirms the interpretability of this criterion, linking low sensitivity to layer redundancy.
Key Results
- On ResNet56, CIFAR-10 and CIFAR-100, SR-init reduces parameters by 63.98% and 37.71%, with negligible accuracy loss (-0.56% and 0.8%). On ImageNet, ResNet50 achieves 39.29% parameter reduction, 15.59% FLOPs reduction, with only 0.6% accuracy drop. These results demonstrate effective compression with minimal performance impact.
- Compared to state-of-the-art methods like Chen et al.[10] and Xu et al.[12], SR-init achieves better parameter reduction and comparable accuracy, highlighting its practical advantage.
- Feature visualization shows that layers with low accuracy drop maintain discriminative regions, validating the correlation between accuracy decline and layer redundancy.
Significance
This work introduces a novel, interpretable layer pruning criterion rooted in internal model redundancy. By analyzing the impact of stochastic re-initialization, it reveals the intrinsic structure of neural networks, facilitating efficient model compression while enhancing explainability. Such insights are crucial for deploying deep models on resource-constrained devices, bridging the gap between performance and interpretability, and guiding future research in model optimization.
Technical Contribution
The paper's main contribution is establishing a direct link between accuracy drop after stochastic re-initialization and layer importance, providing a simple yet effective criterion for pruning. The method avoids complex importance metrics, instead leveraging performance sensitivity, and is validated through feature visualization. It offers a new theoretical perspective and practical tool for model compression, with potential extensions to various architectures and tasks.
Novelty
This is the first work to utilize stochastic re-initialization-induced accuracy changes as a measure of layer redundancy, providing a transparent and interpretable criterion. Unlike prior methods based solely on gradients or importance scores, SR-init emphasizes the intrinsic sensitivity of layers, offering a more intuitive understanding of model structure.
Limitations
- The method assumes stable pre-trained models; in cases of unstable training or highly sensitive architectures, accuracy drops may be unreliable. Threshold tuning remains empirical, requiring dataset-specific calibration. Random re-initialization can cause performance fluctuations, especially in very deep networks, limiting generalizability.
- Further research is needed to adaptively select thresholds and extend to other architectures like Transformers, as well as to multi-task scenarios.
Future Work
Future directions include developing adaptive threshold strategies, integrating with automated neural architecture search, and applying to diverse tasks such as object detection or NLP. Exploring the theoretical bounds of accuracy sensitivity and extending the approach to unsupervised or semi-supervised settings are promising avenues.
AI Executive Summary
Deep neural networks have revolutionized fields like computer vision and natural language processing, yet their enormous size and computational demands hinder deployment on resource-limited devices. Traditional pruning techniques often rely on importance scores or gradient-based metrics, which lack clear interpretability. This paper introduces SR-init, a novel layer pruning strategy grounded in the analysis of performance degradation caused by stochastic re-initialization of individual layers.
The core idea is simple yet powerful: randomly re-initialize each layer's parameters using Kaiming initialization, then measure the resulting accuracy drop on validation data. Layers with minimal impact are considered redundant, as their perturbation does not significantly affect overall performance. This insight provides an interpretable criterion for pruning, emphasizing the internal structure and redundancy of neural networks.
Extensive experiments on ResNet56 and ResNet50 across CIFAR-10, CIFAR-100, and ImageNet datasets demonstrate the effectiveness of SR-init. The method achieves up to 63.98% parameter reduction with less than 1% accuracy loss on CIFAR datasets, and 39.29% parameter reduction with only 0.6% accuracy drop on ImageNet. Feature visualization techniques like Grad-CAM and Guided-Backpropagation confirm that layers with low sensitivity retain critical discriminative features, validating the interpretability of the approach.
Compared with existing state-of-the-art methods, SR-init offers a superior balance between compression and accuracy, especially in parameter reduction. Its simplicity, interpretability, and effectiveness make it a promising tool for deploying deep models on edge devices, as well as for understanding the internal redundancy of neural networks. Future work aims to refine threshold selection, extend to other architectures, and explore multi-task applications, further advancing the field of model compression and interpretability.
Deep Analysis
Background
Deep neural networks（DNNs）在图像识别、自然语言处理等领域取得了巨大成功，但模型庞大带来的计算和存储负担成为瓶颈。为应对这一挑战，模型压缩技术不断发展，包括权重剪枝、滤波器剪枝和层剪枝。权重剪枝通过稀疏化参数实现压缩，适合硬件加速但难以带来实际速度提升；滤波器剪枝通过删除冗余通道，提升推理速度，但受模型结构限制；层剪枝则直接删除整体冗余层，最适合资源受限设备。近年来，基于重要性指标的剪枝方法如Chen等[10]提出利用线性探针评估层重要性，但缺乏对剪枝依据的解释性。本文关注模型内部冗余的可解释性，提出利用随机重初始化引起的性能变化分析层冗余，填补了现有方法在可解释性方面的空白。
Core Problem
现有剪枝方法多依赖梯度或重要性评分，缺乏对剪枝依据的直观解释，难以理解模型内部冗余的本质。尤其在深层网络中，如何量化每一层的冗余度成为关键难题。传统方法在保持性能的同时，效果有限。本文试图通过分析随机重初始化对模型性能的影响，揭示层级冗余的本质，为剪枝提供更具解释性的依据。
Innovation
核心创新包括：1）引入随机重初始化机制，测量每层对模型性能的影响，建立性能变化与冗余的关系；2）提出基于准确率变化的层敏感性指标，作为剪枝依据；3）结合特征可视化验证指标的合理性，增强模型可解释性。该方法简洁高效，避免复杂梯度计算，提供直观的层重要性评估方式，突破传统剪枝的局限。
Methodology
- �� 利用Kaiming初始化对每一层参数进行随机重初始化，确保参数符合正态分布；
- �� 固定其他层，仅重初始化目标层，构建重初始化模型；
- �� 在验证集上评估模型准确率，计算准确率下降作为层敏感性指标；
- �� 设定阈值，剪除低敏感层，达到模型压缩目的；
- �� 结合特征可视化验证剪枝合理性，确保模型内部冗余的解释性。
Experiments
在CIFAR-10、CIFAR-100和ImageNet数据集上，采用ResNet系列模型验证。训练采用SGD，学习率0.01，批量256，训练150轮。剪枝后，采用余弦退火微调模型，评估参数量、FLOPs和准确率。对比多种剪枝方法，验证参数压缩率、性能保持和可解释性。特征可视化采用Grad-CAM和Guided-Backpropagation，分析不同层的冗余特性。
Results
在ResNet56上，参数压缩比例达63.98%，准确率仅下降0.56%；在CIFAR-100上，参数压缩37.71%，准确率下降0.8%；在ImageNet上，ResNet50参数减少39.29%，FLOPs降低15.59%，准确率仅下降0.6%。这些数据表明，SR-init在模型压缩和性能保持方面具有优越表现，验证了其有效性和可解释性。特征可视化分析显示，低敏感层的目标区域与模型冗余高度相关，进一步支持了指标的合理性。
Applications
该方法适用于需要模型轻量化的边缘设备、移动端应用，以及需要模型可解释性的场景。通过简洁的剪枝流程，开发者可以快速获得高效、可解释的模型，提升部署效率。未来可结合自动阈值调节，拓展到多任务、多结构模型中，推动深度学习模型的普适性和可解释性发展。
Limitations & Outlook
该方法依赖预训练模型的性能稳定性，在极端或特殊结构模型中可能表现不佳。阈值设定具有一定经验性，需调优。随机重初始化可能引起性能波动，影响剪枝效果。未来需研究自适应阈值和多任务场景的适应性，提升鲁棒性。
Plain Language Accessible to non-experts
想象你在整理一个复杂的工厂，每个工厂环节都在生产某个产品。有些环节可能重复多次，做的事情几乎一样。你可以试着把那些重复的环节暂时关闭，看看整体生产是否还顺利。若关闭后生产没有明显变差，说明这个环节其实多余，可以省掉。深度学习模型也是一样的，有些层就像那些重复的环节，它们对最终结果影响不大。研究人员用一种特别的方法，随机“重启”每一层，观察模型的表现会掉多少。掉得少的层，说明它们可以被省掉，模型变得更简单、更快。这就像你整理衣柜，把重复的衣服扔掉，既省空间，又不影响穿着。这个思路帮助我们理解模型内部的冗余，找到真正重要的部分，提升效率。
ELI14 Explained like you're 14
想象你在玩一款超级复杂的游戏，里面有很多不同的关卡。有些关卡其实重复了很多内容，玩多次也没啥新意。你可以试着跳过那些重复的关卡，看看游戏还能不能顺利通关。如果没有太大影响，说明那些关卡其实没啥用，可以省掉。深度学习模型也是一样的，有些层就像那些重复的关卡，它们对最终结果影响不大。研究人员用一种特别的方法，随机“重启”每一层，看看模型的表现会掉多少。掉得少的层，说明它们可以被省掉，模型变得更轻、更快。这样一来，模型既保持了效果，又变得更轻便，就像把衣柜里的重复衣服扔掉一样，既省空间又好看。这个方法让我们更好理解模型的内部结构，知道哪些部分是真正重要的，哪些可以省略。
Abstract
Despite the popularization of deep neural networks (DNNs) in many fields, it is still challenging to deploy state-of-the-art models to resource-constrained devices due to high computational overhead. Model pruning provides a feasible solution to the aforementioned challenges. However, the interpretation of existing pruning criteria is always overlooked. To counter this issue, we propose a novel layer pruning method by exploring the Stochastic Re-initialization. Our SR-init method is inspired by the discovery that the accuracy drop due to stochastic re-initialization of layer parameters differs in various layers. On the basis of this observation, we come up with a layer pruning criterion, i.e., those layers that are not sensitive to stochastic re-initialization (low accuracy drop) produce less contribution to the model and could be pruned with acceptable loss. Afterward, we experimentally verify the interpretability of SR-init via feature visualization. The visual explanation demonstrates that SR-init is theoretically feasible, thus we compare it with state-of-the-art methods to further evaluate its practicability. As for ResNet56 on CIFAR-10 and CIFAR-100, SR-init achieves a great reduction in parameters (63.98% and 37.71%) with an ignorable drop in top-1 accuracy (-0.56% and 0.8%). With ResNet50 on ImageNet, we achieve a 15.59% FLOPs reduction by removing 39.29% of the parameters, with only a drop of 0.6% in top-1 accuracy. Our code is available at https://github.com/huitang-zjut/SR-init.