To prune, or not to prune: exploring the efficacy of pruning for model compression

Over the past decade, deep neural networks have achieved remarkable success across vision, NLP, and translation tasks, driven by increasing data and computational power. Early methods like LeCun’s Optimal Brain Damage and Optimal Brain Surgeon introduced second-order approximations for pruning, but their computational cost limited practical use. More recent techniques, such as magnitude-based pruning (Han et al., 2015), have become popular due to efficiency. With hardware advancements supporting sparse matrix operations, model sparsity has gained prominence. Despite progress, achieving high sparsity without performance loss remains challenging, especially across diverse architectures. This paper builds on these foundations, proposing a simple, effective pruning schedule that adapts during training, validated across multiple models and tasks, aiming to bridge the gap between theoretical compression and practical deployment.

The core challenge is balancing model size reduction with performance preservation. Large models contain redundant parameters, but naive pruning often leads to accuracy degradation. Existing methods depend on fixed thresholds or structural assumptions, limiting flexibility and generality. Hardware constraints, such as storage overhead for sparse matrices and lack of universal sparse computation support, further complicate deployment. The key question is how to design a universal, easy-to-implement pruning strategy that can be applied during training, achieve high sparsity, and maintain accuracy across architectures and tasks. Addressing this problem is critical for enabling deep learning in resource-limited environments like mobile devices, IoT, and embedded systems.

The primary innovation is the development of a schedule-based gradual pruning algorithm that dynamically adjusts sparsity during training without requiring layer-specific thresholds or structural assumptions. Key features include:

1) a sparsity schedule (equation 1) that prunes rapidly initially, then more gradually,

2) integration into standard training routines with minimal hyperparameter tuning,

3) applicability across diverse architectures (CNNs, RNNs, seq2seq), and

4) empirical validation showing superior performance compared to traditional fixed-threshold pruning.

This approach simplifies the pruning process, reduces tuning complexity, and enhances generality, making high sparsity models feasible in practical scenarios.

• �� Select target models (InceptionV3, MobileNet, LSTM, seq2seq).• Insert binary masks into each layer’s weights, controlling participation in forward pass.• Sort weights by magnitude, prune smallest according to schedule (equation 1).• Increase sparsity from si to sf over n steps, updating masks every ∆t steps.• Use a schedule that prunes rapidly at first, then more slowly, to allow recovery.• Incorporate pruning into training with learning rate schedules, avoiding low learning rates during pruning.• Stop mask updates once target sparsity sf is reached, continue training to recover performance.• Evaluate models at various sparsity levels, compare accuracy, parameter count, and storage overhead.• Analyze hardware implications, storage formats, and performance trade-offs.

Experiments cover image classification (ImageNet with InceptionV3, MobileNet), language modeling (Penn Treebank with stacked LSTMs), and machine translation (Google NMT). For each, models are trained with standard hyperparameters, then pruned to target sparsities (50%-87.5%). Performance metrics include accuracy, perplexity, BLEU scores, and parameter counts. Ablation studies vary pruning schedules, sparsity levels, and compare sparse versus dense models of similar size. Hardware considerations, such as storage overhead for sparse matrices, are also examined. Results demonstrate that high sparsity models retain near-original performance, outperforming similarly sized dense models, validating the approach’s robustness and broad applicability.

In InceptionV3, 87.5% sparsity reduces parameters from 27.1M to 3.3M with only 1.5% accuracy loss. MobileNet with 75% sparsity achieves 67.7% top-1 accuracy, surpassing denser models of similar parameter count. In language modeling, 90% sparse LSTM reduces parameters from 66M to 6.6M, with perplexity increasing marginally. In NMT, pruning to 85% sparsity yields BLEU scores within 0.25 of the baseline, with 10x fewer parameters. These results confirm that sparse models can be both compact and high-performing, across vision, NLP, and translation tasks, demonstrating the method’s versatility.

The approach enables deploying high-accuracy, low-memory models on mobile devices, embedded systems, and IoT applications. It reduces storage, computational load, and energy consumption, facilitating real-time inference in resource-limited environments. The method is suitable for tasks requiring efficient models, such as on-device image recognition, speech processing, and translation. Combining pruning with hardware-aware sparse storage formats can further optimize inference speed and energy efficiency. Long-term, this technique could transform AI deployment, making sophisticated models accessible in everyday devices, reducing hardware costs, and enabling scalable AI solutions across industries.

While effective, the method’s performance at extremely high sparsity (>95%) can degrade, especially on complex tasks. Sparse matrix storage overhead, such as index vectors, may offset compression gains, particularly on hardware lacking optimized sparse support. Hyperparameter tuning, including pruning schedule and frequency, remains necessary, which can be time-consuming. The approach's effectiveness varies across architectures; further work is needed to adapt it to transformer models and multi-modal tasks. Hardware implementation challenges, such as supporting efficient sparse matrix operations, must be addressed for practical deployment.