Discrimination-aware Channel Pruning for Deep Neural Networks
Discrimination-aware channel pruning (DCP) integrates auxiliary losses to select channels with true discriminative power, achieving 30% channel reduction in ResNet-50 with a 0.39% accuracy boost on ImageNet.
Key Findings
Methodology
This paper introduces discrimination-aware channel pruning (DCP), which incorporates multiple intermediate discriminative losses into the network to enhance the discriminative capacity of feature maps. The approach formulates channel importance as an `2,0`-norm constrained optimization problem, solved via a greedy algorithm that iteratively selects channels based on gradient magnitudes. The process involves fine-tuning the network with combined losses, then pruning channels by balancing reconstruction error and discriminative power. This method effectively filters out redundant channels that do not contribute to classification, leading to more compact models without performance loss.
Key Results
- On ImageNet ILSVRC-12, pruning 30% of channels in ResNet-50 with DCP results in a 0.39% top-1 accuracy increase over the baseline, while reducing parameters and FLOPs by approximately 30% and 33%, respectively. When pruning 50%, DCP outperforms ThiNet by 0.81% in top-1 accuracy, demonstrating superior pruning effectiveness.
- On CIFAR-10, DCP achieves significant parameter and FLOP reductions for VGGNet and ResNet-56, with negligible accuracy degradation. For MobileNet v1 and v2, pruning 30% channels yields accuracy improvements of 0.41% and 0.22%, respectively, confirming its broad applicability.
- In face recognition tasks on LFW, DCP compresses SphereNet-4 by 50-65%, maintaining high accuracy (above 98%) while drastically reducing FLOPs and parameters, validating its effectiveness in real-world applications.
Significance
This work shifts the paradigm from reconstruction-error-centric pruning to discriminative power preservation, enabling models to retain essential features for classification. It addresses the challenge of redundant channel retention in deep networks, providing a theoretically grounded and practically efficient solution. The approach enhances deployment feasibility of deep models on resource-constrained devices, facilitating broader AI adoption in edge computing, mobile applications, and embedded systems. By integrating auxiliary losses, DCP offers a new perspective on model compression, bridging the gap between accuracy and efficiency.
Technical Contribution
The main technical innovation lies in the integration of multiple intermediate discriminative losses with a `2,0`-norm constrained optimization framework, solved via a greedy channel selection algorithm. This contrasts with prior methods that focus solely on reconstruction errors or sparse regularization. The method guarantees the preservation of channels with true discriminative ability, supported by theoretical convexity analysis. Its iterative fine-tuning and channel selection process ensures high accuracy retention, even in very deep networks, with reduced computational overhead compared to exhaustive search.
Novelty
This is the first work to incorporate multi-layer discriminative losses into the channel pruning process, explicitly targeting channels that contribute to the classification task. Unlike existing approaches like ThiNet or Slimming, which rely mainly on weight magnitudes or activation statistics, DCP emphasizes the functional importance of channels through auxiliary losses. This novel integration results in more effective pruning, especially for deep architectures like ResNet-50, achieving better accuracy and compression simultaneously.
Limitations
- The effectiveness depends on the quality of the pre-trained model; poor initial training may limit the discriminative loss's guidance. The added auxiliary losses increase training complexity and computational cost, especially for very deep networks. The greedy algorithm, while efficient, may not always find the globally optimal set of channels, potentially leaving room for further improvement with more sophisticated search strategies.
Future Work
Future directions include automating the weighting of auxiliary discriminative losses, integrating reinforcement learning for more optimal channel selection, and extending the framework to multi-task learning scenarios. Additionally, exploring adaptive stopping criteria and combining DCP with quantization or low-rank methods could further enhance compression efficiency and model robustness.
AI Executive Summary
Deep neural networks have revolutionized computer vision, yet their large size and computational demands hinder deployment on resource-limited devices. Traditional model compression techniques like quantization and sparse regularization have made progress but often compromise accuracy or require complex training. Channel pruning offers a promising alternative by removing redundant filters directly, but existing methods primarily optimize reconstruction error, risking retention of non-discriminative channels. This paper introduces discrimination-aware channel pruning (DCP), a novel approach that integrates auxiliary discriminative losses into the training process to identify and preserve channels that truly contribute to classification tasks.
The core idea is to embed multiple intermediate discriminative losses at different network stages, which guide the pruning process to focus on channels with genuine importance. The authors formulate the importance of channels as an `2,0`-norm constrained optimization problem, solved via a greedy algorithm that iteratively selects the most impactful channels based on gradient magnitudes. This process involves fine-tuning the network with combined losses, then pruning channels by balancing the reconstruction error and discriminative capacity. The approach ensures that the pruned model maintains or even improves accuracy while significantly reducing parameters and FLOPs.
Extensive experiments on datasets such as ImageNet (ILSVRC-12), CIFAR-10, and LFW demonstrate the method's effectiveness. For instance, pruning 30% of channels in ResNet-50 results in a 0.39% accuracy gain, outperforming the original model, with a 33% reduction in FLOPs. On CIFAR-10, the method achieves superior compression with negligible accuracy loss, and on face recognition tasks, it maintains high accuracy with substantial model size reduction. These results highlight the potential of DCP to facilitate efficient deployment of deep models in real-world scenarios.
Overall, this work advances model compression by emphasizing the preservation of discriminative features, offering a theoretically sound and practically effective solution. It opens avenues for further research into automated loss weighting, multi-task pruning, and integration with other compression techniques, promising a future where deep models are both powerful and resource-efficient.
Deep Dive
Abstract
Channel pruning is one of the predominant approaches for deep model compression. Existing pruning methods either train from scratch with sparsity constraints on channels, or minimize the reconstruction error between the pre-trained feature maps and the compressed ones. Both strategies suffer from some limitations: the former kind is computationally expensive and difficult to converge, whilst the latter kind optimizes the reconstruction error but ignores the discriminative power of channels. To overcome these drawbacks, we investigate a simple-yet-effective method, called discrimination-aware channel pruning, to choose those channels that really contribute to discriminative power. To this end, we introduce additional losses into the network to increase the discriminative power of intermediate layers and then select the most discriminative channels for each layer by considering the additional loss and the reconstruction error. Last, we propose a greedy algorithm to conduct channel selection and parameter optimization in an iterative way. Extensive experiments demonstrate the effectiveness of our method. For example, on ILSVRC-12, our pruned ResNet-50 with 30% reduction of channels even outperforms the original model by 0.39% in top-1 accuracy.