LayerMerge: Neural Network Depth Compression through Layer Pruning and Merging
LayerMerge jointly prunes and merges convolutional and activation layers, achieving 1.36× speed-up with minimal accuracy loss.
Key Findings
Methodology
LayerMerge formulates a joint layer pruning problem, combining convolution and activation layer removal, as a surrogate optimization solved via dynamic programming. The approach constructs importance and latency lookup tables, enabling efficient search for the optimal layer subset under latency constraints. The method involves defining layer sets A (activation) and C (convolution), with constraints to prevent kernel size explosion, and optimizing for maximum performance. Fine-tuning follows pruning and merging, ensuring minimal performance degradation. The framework is validated on ResNet-34, MobileNetV2, and DDPM, outperforming existing depth compression and layer pruning techniques.
Key Results
- On ImageNet, ResNet-34 compressed with LayerMerge achieved 1.36× acceleration, accuracy 74.26%, surpassing Depth-78% and LayerOnly-73%. MobileNetV2 reached 2.49× speed-up with 70.99% accuracy. In DDPM, generated image quality remained high after compression, confirming broad applicability. The method maintained stable performance even at high compression ratios, with improvements over state-of-the-art baselines.
- Across multiple architectures and tasks, joint optimization yielded about 10% better compression ratios and robustness compared to separate pruning or deep compression. The importance and latency tables enabled polynomial-time solutions, with the entire process completing within seconds on standard hardware.
- Ablation studies confirmed the necessity of joint optimization, showing that separate pruning or compression underperformed. The algorithm complexity is O(L² P K₀), with practical runtime suitable for real-world deployment, demonstrating scalability and efficiency.
Significance
This work addresses the core challenge of kernel size growth during layer merging, a bottleneck in neural network compression. By jointly pruning convolutional and activation layers, it enables more aggressive compression without hardware latency penalties. The approach advances both theoretical understanding and practical deployment, facilitating faster, more efficient models for edge devices, real-time inference, and resource-constrained environments. Its versatility across tasks and architectures underscores its potential to reshape model deployment strategies, reducing computational costs while maintaining high accuracy.
Technical Contribution
The paper introduces a novel joint pruning framework that combines convolution and activation layer removal, formulated as a surrogate optimization problem. The core innovation lies in constructing importance and latency lookup tables, leveraging the layer structure, and solving via dynamic programming for global optimality. Theoretical analysis guarantees solution optimality with polynomial complexity. This approach extends traditional layer pruning by controlling kernel size growth, enabling high compression ratios with minimal performance loss. It also provides a scalable, hardware-aware optimization paradigm for neural network acceleration.
Novelty
This is the first work to jointly prune convolutional and activation layers with explicit kernel size control, addressing the kernel growth issue in layer merging. The surrogate optimization with dynamic programming guarantees globally optimal solutions efficiently. Unlike prior methods that prune layers or channels independently, this integrated approach achieves superior compression and speed-up, validated across multiple architectures and tasks, marking a significant step forward in neural network compression.
Limitations
- The method relies on discretized latency models, which may not perfectly reflect real hardware behavior, especially in heterogeneous environments. Extreme compression ratios could lead to performance drops if the importance estimation is inaccurate.
- Computational overhead increases with model size and layer count, potentially limiting scalability for very large models. The approach currently targets convolutional architectures and may need adaptation for other types.
- Validation primarily on vision tasks; applicability to NLP or multimodal models remains to be demonstrated. Future work should explore broader domain adaptation and hardware-aware extensions.
Future Work
Future directions include integrating hardware-specific latency models for more precise optimization, extending the framework to NLP and multimodal models, and developing automated, end-to-end compression pipelines. Additionally, exploring multi-objective optimization to balance latency, energy, and accuracy, and adapting the approach for large-scale pretraining models, will broaden its impact and practical utility.
AI Executive Summary
LayerMerge introduces a novel approach to neural network depth compression by jointly pruning convolutional and activation layers, effectively addressing the kernel size growth problem that hampers traditional layer merging techniques. The core innovation lies in formulating a surrogate optimization problem that captures the trade-off between model performance and inference latency, which is efficiently solved using dynamic programming. This method constructs importance and latency lookup tables based on the network's layer structure, enabling the selection of the most impactful layers to prune while respecting latency constraints.
Experimental results across multiple architectures—including ResNet-34, MobileNetV2, and DDPM—demonstrate that LayerMerge consistently outperforms existing methods such as depth compression and layer pruning. For instance, on ImageNet, it achieves a 1.36× speed-up with a slight accuracy increase to 74.26%, surpassing the baseline methods. In MobileNetV2, the acceleration reaches 2.49× with minimal performance loss, highlighting its effectiveness in resource-constrained scenarios. Moreover, the approach maintains high-quality outputs in generative tasks, validating its versatility.
The significance of this work extends beyond mere acceleration. By controlling kernel size growth and enabling joint layer pruning, it offers a scalable, hardware-aware framework that can be adapted to various models and tasks. Its theoretical guarantees and practical efficiency make it a promising tool for deploying large models on edge devices, reducing computational costs while preserving accuracy. Despite some limitations in hardware modeling and scalability for extremely large models, the method opens new avenues for future research in multi-objective optimization, automated compression pipelines, and broader domain applications, paving the way for more efficient AI systems.
Deep Dive
Abstract
Recent works show that reducing the number of layers in a convolutional neural network can enhance efficiency while maintaining the performance of the network. Existing depth compression methods remove redundant non-linear activation functions and merge the consecutive convolution layers into a single layer. However, these methods suffer from a critical drawback; the kernel size of the merged layers becomes larger, significantly undermining the latency reduction gained from reducing the depth of the network. We show that this problem can be addressed by jointly pruning convolution layers and activation functions. To this end, we propose LayerMerge, a novel depth compression method that selects which activation layers and convolution layers to remove, to achieve a desired inference speed-up while minimizing performance loss. Since the corresponding selection problem involves an exponential search space, we formulate a novel surrogate optimization problem and efficiently solve it via dynamic programming. Empirical results demonstrate that our method consistently outperforms existing depth compression and layer pruning methods on various network architectures, both on image classification and generation tasks. We release the code at https://github.com/snu-mllab/LayerMerge.