DepthShrinker: A New Compression Paradigm Towards Boosting Real-Hardware Efficiency of Compact Neural Networks
DepthShrinker merges linear layers by removing unimportant activation functions, boosting hardware utilization and achieving 3.06% accuracy gain and 1.53× throughput over SOTA pruning.
Key Findings
Methodology
DepthShrinker employs a differentiable search to learn activation importance, removing redundant functions post-training. Consecutive linear layers are then merged into dense convolutions, leveraging the observation that certain activations are dispensable for inference. The framework involves three steps: identifying unimportant activations via learnable masks, fine-tuning the pruned model, and merging layers into a single dense operation. Experiments on models like MobileNetV2 and EfficientNet-Lite0 demonstrate significant hardware efficiency improvements while maintaining accuracy.
Key Results
- On Tesla V100, the compressed models achieved a 3.06% higher accuracy and 1.53× throughput compared to MetaPruning, with minimal accuracy loss. MobileNetV2 and ResNet models showed up to 4× throughput gains on GPUs, with accuracy preserved.
- Ablation studies confirmed that removing and merging layers based on learned importance yields substantial hardware utilization improvements, validating the effectiveness of the approach.
- Real-device profiling indicated that dense replacements of depthwise convolutions significantly boost throughput, especially on parallel architectures like V100 and RTX 2080Ti.
Significance
This work addresses the core bottleneck of hardware underutilization in efficient DNNs by transforming irregular, sparse operations into dense, hardware-friendly ones. It bridges the gap between theoretical efficiency and real-world deployment, enabling compact models to fully leverage modern accelerators. The approach offers a new paradigm—soft layer pruning—that complements existing compression techniques, making models both smaller and faster without sacrificing accuracy. Its practical impact spans edge devices, mobile applications, and high-performance servers, promising broader adoption of efficient AI.
Technical Contribution
The key technical innovation lies in the joint importance learning of activation functions via differentiable masks, enabling selective removal without retraining from scratch. The layer fusion algorithm mathematically merges multiple linear layers into a single dense convolution, based on kernel convolution formulas (e.g., Eq.6). This approach maintains model expressiveness while significantly improving hardware utilization. The framework is compatible with various architectures and can be extended with platform-aware latency optimization, opening new avenues for hardware-aware neural architecture design.
Novelty
This is the first systematic framework that combines activation importance learning with layer fusion to achieve hardware-efficient dense networks from existing efficient models. Unlike traditional pruning, which removes entire channels or layers, DepthShrinker soft-prunes by merging layers, preserving model capacity. Its novelty also lies in exploiting the dispensability of activation functions post-training, a concept rarely utilized in model compression, to facilitate layer merging and hardware acceleration.
Limitations
- The method relies on pre-trained models, and its effectiveness on training-from-scratch scenarios remains untested.
- Extreme compression ratios may cause slight accuracy degradation, especially if importance learning misclassifies some activations.
- Platform-specific latency measurements are needed for optimal fusion, adding complexity for deployment on diverse hardware.
Future Work
Future research will focus on adaptive importance learning that dynamically adjusts during training, integrating platform-aware latency models for optimal layer merging. Extending the framework to support quantization-aware fusion and exploring automated neural architecture search for even more efficient dense networks are promising directions. Additionally, applying this approach to other tasks like object detection and NLP could broaden its impact.
AI Executive Summary
Deep neural networks have revolutionized AI applications, yet their deployment on real hardware faces a persistent challenge: inefficient utilization of computational resources. Many high-performance models rely on operations like depthwise convolutions, which, despite their theoretical efficiency, suffer from irregular computation patterns that hinder hardware parallelism. This discrepancy limits the practical speed and energy efficiency gains promised by these compact architectures. Existing compression techniques such as pruning and quantization reduce model size but often fail to fully leverage modern accelerators’ parallel capabilities, leaving significant room for improvement.
In response, this paper introduces DepthShrinker, a novel framework that fundamentally rethinks how compact neural networks can be optimized for hardware. The core insight is that certain activation functions, while beneficial during training, can be safely removed after training without degrading accuracy. By learning the importance of each activation via a differentiable mask, the method identifies redundant functions. Removing these functions simplifies the network structure, enabling the fusion of consecutive linear layers into a single dense convolution. This process transforms irregular, sparse operations into dense, hardware-friendly computations that better utilize parallel processing units.
Experimental results on ImageNet demonstrate the effectiveness of DepthShrinker. On Tesla V100, the compressed models outperform state-of-the-art pruning methods like MetaPruning, achieving a 3.06% accuracy increase and 1.53× throughput boost. Similar gains are observed on edge devices, with significant reductions in latency and improvements in hardware utilization. The approach maintains the original model’s accuracy, validating the hypothesis that many activation functions are dispensable for inference.
This work offers a new paradigm—soft layer pruning—complementing traditional compression techniques. It paves the way for more efficient deployment of compact models across diverse hardware platforms, from edge devices to data centers. While promising, the method relies on pre-trained models and platform-specific latency measurements, which could be further refined. Future directions include adaptive importance learning, platform-aware optimization, and extension to other neural tasks, promising a broader impact on AI hardware efficiency.
Deep Dive
Glossary
Activation Function (激活函数)
非线性变换，用于增加模型表达能力，常见有ReLU、ReLU6等。In this paper, some activation functions are shown to be removable after training without影响模型性能。
用于判断哪些激活函数可以在推理时移除。
Layer Fusion (层融合)
将连续的线性操作（如卷积、全连接）合并成一个操作，减少计算步骤，提高硬件利用率。
核心技术之一，用于提升模型硬件效率。
Differentiable Mask (可微掩码)
一种可学习的二值或连续掩码，用于选择性地移除或保留网络中的激活函数。
实现激活重要性学习的关键机制。
Dense Convolution (密集卷积)
具有完整数据重用和规则计算模式的卷积操作，适合GPU等硬件高效执行。
通过融合层实现硬件友好计算。
Model Fine-tuning (模型微调)
在模型结构调整后，通过继续训练优化性能，确保精度不受影响。
在激活函数移除后进行微调。
Open Questions Unanswered questions from this research
- 1 如何在训练从零开始的模型中应用激活函数移除策略？
- 2 不同硬件平台的融合策略如何自动调优？
- 3 极端压缩比下模型性能的极限是什么？
Applications
Immediate Applications
边缘设备加速
在智能手机、嵌入式设备上部署紧凑模型，提升实时推理速度，降低能耗。
云端推理优化
在数据中心GPU上优化模型，提升吞吐率，减少延迟，支持大规模应用。
Long-term Vision
自动化模型设计
结合神经架构搜索，自动生成硬件友好的密集网络架构，实现端到端优化。
Abstract
Efficient deep neural network (DNN) models equipped with compact operators (e.g., depthwise convolutions) have shown great potential in reducing DNNs' theoretical complexity (e.g., the total number of weights/operations) while maintaining a decent model accuracy. However, existing efficient DNNs are still limited in fulfilling their promise in boosting real-hardware efficiency, due to their commonly adopted compact operators' low hardware utilization. In this work, we open up a new compression paradigm for developing real-hardware efficient DNNs, leading to boosted hardware efficiency while maintaining model accuracy. Interestingly, we observe that while some DNN layers' activation functions help DNNs' training optimization and achievable accuracy, they can be properly removed after training without compromising the model accuracy. Inspired by this observation, we propose a framework dubbed DepthShrinker, which develops hardware-friendly compact networks via shrinking the basic building blocks of existing efficient DNNs that feature irregular computation patterns into dense ones with much improved hardware utilization and thus real-hardware efficiency. Excitingly, our DepthShrinker framework delivers hardware-friendly compact networks that outperform both state-of-the-art efficient DNNs and compression techniques, e.g., a 3.06% higher accuracy and 1.53$\times$ throughput on Tesla V100 over SOTA channel-wise pruning method MetaPruning. Our codes are available at: https://github.com/facebookresearch/DepthShrinker.