Filter Pruning via Geometric Median for Deep Convolutional Neural Networks Acceleration
Proposes Geometric Median-based filter pruning (FPGM), reducing over 52% FLOPs on ResNet-110 and 42% on ResNet-101 without accuracy loss.
Key Findings
Methodology
This work analyzes the limitations of norm-based filter pruning, which assumes filters with smaller norms are less important. To address this, it introduces a method leveraging the geometric median (GM) to identify redundant filters within each layer. The core process involves computing the GM of filters in a layer, then selecting filters closest to this median as candidates for removal, indicating high redundancy. This approach does not depend on the distribution of filter norms and thus remains effective even when the norm distribution is non-ideal. The method integrates pruning into training, avoiding additional fine-tuning. Experiments on CIFAR-10 and ImageNet demonstrate its effectiveness, achieving significant FLOPs reduction with minimal or no accuracy loss.
Key Results
- On CIFAR-10, FPGM reduces over 52% FLOPs on ResNet-110, with a 2.69% relative accuracy increase. Similar results are observed on ResNet-20 and ResNet-56, maintaining or improving accuracy.
- On ImageNet, FPGM achieves over 42% FLOPs reduction on ResNet-101 without top-5 accuracy drop, outperforming existing pruning methods in robustness and compression ratio.
- Compared to norm-based pruning, FPGM exhibits superior robustness, especially when the filter norm distribution is non-ideal, maintaining high performance under various conditions.
Significance
This research overcomes the dependency on filter norm distributions in pruning strategies, enabling more robust and effective model compression. Its independence from distribution assumptions makes it adaptable to various architectures and datasets, facilitating deployment on resource-constrained devices. The approach enhances the practicality of deep learning models in real-world applications, addressing the longstanding challenge of balancing model size, speed, and accuracy. It also opens new avenues for theoretical understanding of filter redundancy and robustness in neural networks.
Technical Contribution
The core technical contribution is the introduction of a geometric median-based criterion for filter redundancy detection, which is robust to the distribution of filter norms. The algorithm computes the GM within each layer, then identifies filters closest to this median for pruning. This process is integrated into training, allowing for iterative compression without performance degradation. The method's theoretical foundation guarantees that the selected filters are highly replaceable, leading to efficient compression and acceleration. Extensive experiments validate its superiority over traditional norm-based methods, establishing a new standard for robust filter pruning.
Novelty
This work is the first to utilize the geometric median as a measure of filter redundancy in CNN pruning. Unlike prior approaches relying solely on filter norms, GM-based pruning does not assume specific norm distributions, making it more robust and versatile. The approach introduces a novel perspective on filter importance, emphasizing replaceability and redundancy over magnitude. This innovation significantly improves the robustness and effectiveness of pruning, especially in scenarios where norm-based criteria fail due to non-ideal distributions, marking a substantial advancement in model compression techniques.
Limitations
- Computing the geometric median, especially in high-dimensional spaces with many filters, can be computationally intensive, potentially limiting scalability to very large models without optimization.
- The method primarily targets convolutional layers; its applicability to fully connected layers or other architectures like transformers remains to be validated.
- Extreme compression ratios may still cause performance drops, indicating the need for adaptive or hybrid pruning strategies in future work.
Future Work
Future research could focus on optimizing the geometric median computation for large-scale models, possibly through approximation algorithms. Extending the approach to other network components, such as fully connected layers and transformer modules, is also promising. Combining GM-based pruning with quantization and sparsity regularization could further enhance compression ratios. Additionally, developing adaptive pruning schedules and automated hyperparameter tuning would improve robustness and ease of deployment, paving the way for broader adoption in real-world applications.
AI Executive Summary
Deep convolutional neural networks (CNNs) have revolutionized computer vision tasks, achieving remarkable accuracy across various benchmarks. However, their large size and high computational demands hinder deployment on resource-constrained devices like smartphones and embedded systems. Traditional model compression techniques, such as filter pruning, often rely on heuristics like filter norm magnitude to identify redundant filters. While effective in some cases, these methods depend heavily on the distribution of filter norms, which may not always be ideal. When the norm distribution is narrow or the minimum norm is not close to zero, such approaches can fail, leading to performance degradation.
To address these limitations, this work introduces a novel filter pruning strategy based on the geometric median (GM). Instead of relying solely on filter norms, the method calculates the GM of filters within each layer, which serves as a robust central point representing the layer's filter set. Filters closest to this median are considered highly redundant, as they can be effectively replaced by others without significant loss of information. By pruning these filters, the method achieves substantial reduction in FLOPs while maintaining, or even improving, accuracy.
Extensive experiments on CIFAR-10 and ImageNet datasets validate the effectiveness of the approach. In ResNet-110, the method reduces over 52% FLOPs with a 2.69% accuracy gain; on ResNet-101, over 42% FLOPs are cut without accuracy loss. Compared to traditional norm-based pruning, the geometric median approach demonstrates superior robustness, especially in scenarios where filter norm distributions are non-ideal. The technique integrates seamlessly into training, eliminating the need for additional fine-tuning, and offers a practical pathway toward deploying efficient, high-performance CNNs on edge devices.
Overall, this research provides a significant step forward in model compression, emphasizing the importance of filter redundancy and robustness over simple magnitude heuristics. Its implications extend to real-world applications requiring fast, lightweight models, and open avenues for further innovations combining geometric insights with advanced pruning and quantization techniques.
Deep Dive
Plain Language Accessible to non-experts
想象你在整理一个厨房里的食材。每次做饭前，你会检查所有食材的状态，发现有些食材其实可以用得很少甚至不用了，因为它们和其他食材重复或多余。传统的方法就像只看食材的重量，轻的可能代表不重要，但有时候轻的食材其实很重要。现在，这个新办法像是用一种聪明的测量工具，能找到那些其实可以被替代或省略的食材，而不用依赖重量大小。它会找出那些和其他食材差不多、可以互相替代的食材，然后把它们去掉。这样一来，厨房变得更整洁，做饭也更快。这个方法让厨房里的食材变得更合理，既省时间又不影响味道。
ELI14 Explained like you're 14
想象你在玩一款游戏，里面有很多角色和装备。有些装备看起来很酷，但其实用处不大，或者和其他装备重复。以前的人常用的方法是只看装备的重量，轻的就觉得没用，但其实轻的装备有时候很重要。现在，这个新方法像是用一种聪明的算法，能找到那些其实可以不用的装备，而不用只看重量。它会找出那些和其他装备差不多、可以互相替代的装备，然后把它们去掉。这样你的角色就能变得更快、更强，装备也更轻便。就像整理书包一样，把不重要的东西扔掉，剩下的都能帮你更好地完成任务。这种方法让游戏变得更有趣，也让你更快赢得胜利！
Abstract
Previous works utilized ''smaller-norm-less-important'' criterion to prune filters with smaller norm values in a convolutional neural network. In this paper, we analyze this norm-based criterion and point out that its effectiveness depends on two requirements that are not always met: (1) the norm deviation of the filters should be large; (2) the minimum norm of the filters should be small. To solve this problem, we propose a novel filter pruning method, namely Filter Pruning via Geometric Median (FPGM), to compress the model regardless of those two requirements. Unlike previous methods, FPGM compresses CNN models by pruning filters with redundancy, rather than those with ''relatively less'' importance. When applied to two image classification benchmarks, our method validates its usefulness and strengths. Notably, on CIFAR-10, FPGM reduces more than 52% FLOPs on ResNet-110 with even 2.69% relative accuracy improvement. Moreover, on ILSVRC-2012, FPGM reduces more than 42% FLOPs on ResNet-101 without top-5 accuracy drop, which has advanced the state-of-the-art. Code is publicly available on GitHub: https://github.com/he-y/filter-pruning-geometric-median