Advancing Weight and Channel Sparsification with Enhanced Saliency
Proposes IEE, an iterative exploitation-exploration framework, boosting importance scores for sparse pruning, achieving +1.3% Top-1 accuracy on ImageNet ResNet50.
Key Findings
Methodology
The IEE framework divides the model into an active sparse structure and an exploration space. It iteratively performs exploitation (training the active structure) and exploration (briefly reactivating and training the exploration space). During exploitation, parameters are optimized; during exploration, less crucial parameters are pruned and reactivated temporarily, then re-evaluated using a unified importance score (e.g., magnitude or Taylor score). The most important parameters from exploration are grown into the active structure, forming a closed feedback loop. This approach enhances importance estimation, reduces performance degradation caused by score inaccuracies, and is compatible with both unstructured and structured sparsity. It leverages hardware-aware pruning and maintains resource constraints, leading to efficient, high-performance sparse models.
Key Results
- On ImageNet, ResNet50 with 90% ERK sparsity, IEE surpasses prior methods, achieving +1.3% Top-1 accuracy (74.6% vs. 73.3%), with over 70% training cost reduction compared to HALP. It also outperforms RigL by 1.3% accuracy at the same sparsity.
- In MobileNetV1, IEE achieves better latency-accuracy tradeoffs, with significant reductions in training FLOPs and improved inference speed. For unstructured sparsity, it improves CIFAR-10 and ImageNet performance over baseline methods.
- Ablation studies confirm that reactivating exploration space parameters and using a unified importance score are key to performance gains, demonstrating the method’s robustness across datasets and architectures.
Significance
This work addresses the core challenge of unreliable importance scores in pruning by introducing a cyclic exploitation-exploration process, significantly improving sparse model accuracy and efficiency. It bridges the gap between static pruning and dynamic training, enabling models to adaptively discover better sparse structures. The approach is versatile, applicable to both unstructured and structured sparsity, and compatible with hardware-aware constraints, making it highly relevant for deploying efficient deep learning models in real-world scenarios. It advances the theoretical understanding of importance estimation and offers practical tools for model compression, accelerating the deployment of deep networks on resource-limited devices.
Technical Contribution
The main innovation is the iterative exploitation-exploration cycle that leverages a unified importance metric, combined with a short-term reactivation of exploration parameters, providing a performance preview. This mechanism effectively mitigates the score bias problem, enhances the reliability of importance evaluation, and enables simultaneous optimization of sparse structure and model accuracy. The method also integrates hardware-aware pruning using knapsack solvers, ensuring latency constraints are met. These contributions collectively push the boundaries of dynamic sparse training, offering a theoretically sound and practically effective framework.
Novelty
This is the first work to systematically incorporate a short-term reactivation and training of exploration space parameters within an iterative exploitation-exploration loop, unifying pruning and growth importance scores. Unlike RigL, which relies on immediate gradients for growth, IEE maintains a consistent importance criterion, reducing score bias and improving long-term structure optimization. Its compatibility with both structured and unstructured sparsity, along with hardware-aware constraints, marks a significant departure from prior methods, setting a new standard in sparse model training.
Limitations
- The approach depends on the choice of importance metric; while simple metrics like magnitude and Taylor scores work well, more complex tasks may require better metrics, which could increase computational overhead.
- Short-term reactivation introduces additional training steps, potentially raising computational costs, especially for very large models or datasets.
- Current validation is primarily on image classification; extending to other domains like NLP or object detection needs further validation and adaptation.
Future Work
Future directions include developing multi-metric importance evaluation, integrating reinforcement learning for adaptive exploration, and extending the framework to multi-task and multi-modal scenarios. Additionally, exploring more efficient reactivation strategies and applying the method to other domains such as NLP and speech recognition will broaden its impact.
AI Executive Summary
The rapid growth of deep neural networks has led to increasingly large models with high computational and storage demands, hindering deployment in resource-constrained environments. Pruning techniques, which remove redundant parameters to create sparse models, have become a vital research area. Traditional pruning relies heavily on importance scores—such as magnitude or Hessian-based metrics—to identify which weights or channels to prune. However, these scores are often imperfect, leading to performance drops after pruning. Dynamic sparse training methods like RigL attempt to address this by periodically adjusting the sparse structure during training, but they suffer from issues such as criterion inconsistency and short-sighted growth strategies.
In this context, the paper introduces the Iterative Exploitation and Exploration (IEE) framework, a novel approach that enhances importance scores through a cyclic process. The core idea is to divide the model into an active sparse structure and an exploration space. During exploitation, the active structure is trained to convergence; during exploration, parameters in the exploration space are briefly reactivated and trained, providing a performance preview. Based on a unified importance metric, the most promising parameters are then grown into the active structure, forming a feedback loop that iteratively refines the sparse architecture.
Experimental results demonstrate that IEE significantly outperforms existing methods. On ImageNet with ResNet50, it achieves a +1.3% increase in Top-1 accuracy at 90% ERK sparsity, while reducing training costs by over 70% compared to HALP. Similar improvements are observed with MobileNetV1, and in unstructured sparsity scenarios on CIFAR-10 and ImageNet, the method surpasses RigL and other baselines. Ablation studies confirm the importance of the reactivation step and the use of a unified importance score.
Overall, IEE offers a powerful, versatile framework for high-performance sparse model training, addressing key limitations of prior methods. Its ability to integrate hardware-aware constraints and improve importance estimation paves the way for more efficient deployment of deep learning models in real-world applications. Future work will explore multi-metric importance evaluation, broader domain adaptation, and multi-task scenarios, further extending its impact.
Deep Analysis
Background
Deep neural networks的不断扩大带来了模型规模和计算成本的持续增长，限制了其在边缘设备和实时场景中的应用。剪枝技术通过移除冗余参数实现模型压缩和加速，早期方法如SNIP、GraSP、RigL等，依赖参数幅值、Hessian信息或梯度信息进行重要性评估，取得一定成功。然而，这些方法普遍存在评分不准、结构限制和训练成本高的问题。结构化剪枝（如通道剪枝）能带来硬件加速，但难以适应复杂模型。动态稀疏训练（如RigL）试图在训练过程中动态调整稀疏结构，但面临评分不一致、短视策略等瓶颈。近年来，研究者开始关注如何提升重要性评分的可靠性，结合多阶段优化，推动稀疏模型的实用化。
Core Problem
现有剪枝方法多依赖单一指标，难以准确识别重要参数，导致剪枝后性能下降。动态稀疏训练中的评分不一致和短视策略限制了模型潜力。如何在保证性能的同时降低训练成本，提升结构探索效率，成为亟待解决的问题。特别是在结构化剪枝中，梯度信息难以反映参数重要性，限制了方法的实用性。此外，缺乏有效的探索机制，难以充分挖掘潜在的稀疏结构。
Innovation
本文提出的IEE框架创新在于引入利用-探索的循环机制，结合短期参数重训练，为重要性评分提供性能预览。通过“重激活”探索空间参数，短暂训练后依据统一指标（如Taylor评分）重新评估参数重要性，实现剪枝与生长的闭环优化。这一机制缓解了评分偏差，兼容结构化与无结构稀疏，显著提升模型性能，降低训练成本。结合硬件感知策略，确保稀疏结构满足延迟约束，具有理论创新和工程实用性。
Methodology
- �� 将模型参数划分为活跃结构ΘK和探索空间ΘP。• 利用预设重要性指标（幅值或Taylor评分）评估参数。• 在每个周期中，先训练ΘK H步，评估其重要性。• 根据重要性指标，剪除ΘK中最不重要的参数，加入ΘP。• 短期冻结ΘK，训练ΘP Q步，评估潜在贡献。• 依据统一指标，选择ΘP中最重要的参数，加入ΘK，完成利用-探索循环。• 反复迭代优化稀疏结构，直至训练结束。• 结合硬件感知策略，确保满足延迟约束。
Experiments
在ImageNet上，验证ResNet50和MobileNetV1的性能，比较不同稀疏率下的Top-1准确率、FLOPs和训练成本。采用Taylor评分和幅值作为重要性指标，结合硬件感知的HALP方案，进行结构化剪枝。无结构稀疏在CIFAR-10和ImageNet上验证，超越RigL等动态训练方法。设置不同的更新周期和探索次数，进行消融实验，分析各组件对性能的贡献。评估指标包括准确率提升、训练成本降低和推理速度提升，验证方法的普适性和优越性。
Results
在ImageNet上，ResNet50模型通过IEE实现90% ERK稀疏率，Top-1准确率提升1.3%，训练成本降低70%，优于RigL和HALP。MobileNetV1在结构化剪枝中表现出更优的延迟-准确率折衷，训练成本显著减少。无结构稀疏方面，利用幅值和Taylor评分，模型在CIFAR-10和ImageNet上的性能均优于对比方法。消融实验显示，重激活探索空间参数和统一指标评估是性能提升的关键。整体结果表明，IEE框架在模型压缩和性能提升方面具有广泛应用潜力。
Applications
该方法适用于需要模型压缩和加速的场景，如边缘设备、移动端和实时推理系统。只需预训练模型或从零开始训练，结合硬件感知策略，可实现高效稀疏模型部署。未来，结合多指标和强化学习，能进一步提升模型的自适应能力，支持多任务、多模态应用，推动深度学习模型的实际落地。
Limitations & Outlook
目前方法依赖于单一重要性指标，可能在复杂任务中表现有限。探索空间的短期训练增加了计算负担，尤其在大规模模型中。主要验证在图像分类，迁移到其他任务（如目标检测、NLP）仍需验证。未来需优化指标融合策略，降低计算复杂度，并扩展到多任务场景。
Plain Language Accessible to non-experts
想象你在整理一个超大的仓库，里面堆满了各种商品。每次整理时，你会优先把最重要的商品放在显眼位置，不那么重要的放到角落。可是如果只看商品的大小（越大越重要），可能会漏掉一些其实很有用的小商品。于是，你决定用一种聪明的方法：先把所有商品都稍微整理一下，看看哪些商品可能还会用到，然后再根据实际情况调整，把重要的商品放在前面，次要的放到后面。这个过程不断重复，仓库变得越来越整齐，存取也更快。这就像论文中的方法，把模型的参数当作商品，通过不断评估和调整，找到最重要的参数，让模型变得更快、更小，但性能依然很强。
ELI14 Explained like you're 14
你知道学校里有很多书，有些书很重要，有些用得少。老师想把重要的书放在前面，方便大家学习，但如果只看书的厚度（越厚越重要），可能会漏掉一些其实很关键的薄书。于是，老师决定用一种聪明的方法：先把所有书都拿出来，稍微翻阅一下，看看哪些书可能还会用到，然后再根据实际情况，把重要的书放到前面，不重要的放到后面。这个过程会反复几次，书架变得越来越合理，找书也更快。这就像论文里的方法，把模型里的参数当作书，通过不断评估和调整，找到最重要的参数，让模型变得更快、更小，但性能还很棒。
Abstract
Pruning aims to accelerate and compress models by removing redundant parameters, identified by specifically designed importance scores which are usually imperfect. This removal is irreversible, often leading to subpar performance in pruned models. Dynamic sparse training, while attempting to adjust sparse structures during training for continual reassessment and refinement, has several limitations including criterion inconsistency between pruning and growth, unsuitability for structured sparsity, and short-sighted growth strategies. Our paper introduces an efficient, innovative paradigm to enhance a given importance criterion for either unstructured or structured sparsity. Our method separates the model into an active structure for exploitation and an exploration space for potential updates. During exploitation, we optimize the active structure, whereas in exploration, we reevaluate and reintegrate parameters from the exploration space through a pruning and growing step consistently guided by the same given importance criterion. To prepare for exploration, we briefly "reactivate" all parameters in the exploration space and train them for a few iterations while keeping the active part frozen, offering a preview of the potential performance gains from reintegrating these parameters. We show on various datasets and configurations that existing importance criterion even simple as magnitude can be enhanced with ours to achieve state-of-the-art performance and training cost reductions. Notably, on ImageNet with ResNet50, ours achieves an +1.3 increase in Top-1 accuracy over prior art at 90% ERK sparsity. Compared with the SOTA latency pruning method HALP, we reduced its training cost by over 70% while attaining a faster and more accurate pruned model.