Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding

Deep neural networks have achieved state-of-the-art performance in vision tasks, but their large parameter count leads to high storage and computational costs. Early work on pruning and sparsity, such as LeCun’s early neural network pruning, addressed overfitting and model complexity. Recent advances like AlexNet and VGG-16 pushed accuracy but at the expense of massive models (hundreds of MBs). Existing compression methods, such as weight pruning, quantization, and low-rank approximation, often trade off accuracy or lack generality. Hardware constraints further limit deployment, especially on mobile devices. The need for efficient, high-ratio compression methods that preserve accuracy remains urgent, driving research into multi-stage, hardware-friendly solutions.

The core challenge is reducing the enormous size and energy demands of deep neural networks for resource-constrained devices without degrading their accuracy. Traditional compression techniques often cause accuracy loss or require complex retraining. Moreover, the large number of parameters makes inference slow and energy-consuming, especially on mobile hardware. Achieving a balance between high compression ratio, low latency, and accuracy preservation is difficult. Existing methods lack a systematic approach that combines multiple techniques effectively, limiting their practical utility. The problem is compounded by the need for hardware-compatible formats that facilitate acceleration.

This work introduces a comprehensive deep compression framework integrating three key innovations: • Pruning: removes redundant connections based on weight thresholds, reducing parameters by up to 13×; • Trained quantization: employs k-means clustering to share weights, decreasing bits per weight from 32 to 5, with retraining to fine-tune shared weights; • Huffman coding: exploits the biased distribution of weights and indices for lossless compression, achieving further size reduction. These steps are combined sequentially, with retraining after each to maintain accuracy. The approach is hardware-agnostic, enabling deployment across CPUs, GPUs, and mobile devices, and achieves unprecedented compression ratios without accuracy loss.

• �� Train the original network to learn weights and connectivity; • Apply a threshold to prune low-magnitude weights, creating a sparse network; • Store sparse structure using compressed sparse row/column formats, encoding relative indices; • Cluster remaining weights with k-means, choosing initial centroids via linear, density, or random methods; • Fine-tune shared centroids through backpropagation, updating based on grouped gradients; • Quantize weights into k clusters, replacing original weights with cluster indices; • Use Huffman coding on the weight distribution and index differences to compress further; • Store the final compressed model in a format suitable for hardware acceleration.

The authors tested on MNIST with LeNet models and on ImageNet with AlexNet and VGG-16. Compression ratios ranged from 35× to 49×, with no accuracy loss. The models were evaluated on multiple hardware platforms: CPU, GPU, and mobile GPU, demonstrating 3-4× speedup and 3-7× energy efficiency improvements. Ablation studies examined the effects of pruning thresholds, quantization bits, and initialization strategies, confirming the robustness of the approach. The experiments validated that high compression could be achieved without sacrificing model performance, making the method practical for real-world deployment.

AlexNet's size shrank from 240MB to 6.9MB, maintaining top-1 accuracy at 57.2%. VGG-16 was reduced from 552MB to 11.3MB, with no accuracy degradation. On hardware, compressed models achieved 3-4× layerwise speedup and 3-7× energy savings. The combined pruning and quantization method outperformed individual techniques, confirming their synergy. The use of bias-aware Huffman coding further compressed the models by 20-30%. These results demonstrate that large, accurate models can be made lightweight and energy-efficient, suitable for mobile and embedded applications.

This compression technique is ideal for deploying deep neural networks on mobile devices, embedded systems, and real-time vision systems such as autonomous vehicles and surveillance. It enables models to fit into limited on-chip memory, reduces bandwidth requirements, and lowers power consumption. The hardware-friendly format facilitates integration with accelerators, making real-time inference feasible in resource-constrained environments. It also benefits cloud-edge scenarios where bandwidth and latency are critical, broadening AI's practical reach.

The current approach primarily targets convolutional and fully connected layers; its effectiveness on recurrent or attention-based models remains untested. Extreme quantization may cause accuracy drops, requiring careful parameter tuning. The retraining process is computationally intensive, limiting rapid deployment. Additionally, hardware implementation complexity increases with compression, necessitating further optimization for real-world accelerators. Future work should address these limitations to enhance generality and efficiency.