A Survey of Quantization Methods for Efficient Neural Network Inference

Deep neural networks have revolutionized AI, but their large size limits deployment on resource-constrained devices. Early efforts focused on pruning, sparse representations, and hardware optimization, yet model size and inference latency persisted. Recent breakthroughs in low-bit quantization—such as XnorNet, DoReFa, and QAT—have enabled significant compression and acceleration, especially with INT8 and lower bit-widths. These methods address the need for efficient inference without substantial accuracy loss, facilitating applications in mobile, IoT, and autonomous systems. As hardware platforms diversify, research increasingly emphasizes hardware-aware quantization algorithms that adapt to specific architectures, balancing precision and efficiency.

Despite progress, deploying neural networks with ultra-low precision (INT2, INT4) remains challenging due to accuracy degradation, especially in complex models like Transformers. Precise calibration of quantization ranges, error accumulation during training, and hardware compatibility issues further complicate deployment. The core problem is how to maintain high accuracy while drastically reducing bit-width, ensuring models are both resource-efficient and reliable. Additionally, dynamic calibration introduces computational overhead, limiting real-time applicability. Addressing these issues requires innovative algorithms that adaptively optimize quantization parameters, integrate with training, and align with hardware constraints.

This work introduces a multi-faceted approach combining: • Layerwise, channelwise, and block-level quantization to adapt ranges to distribution variances; • Learning-based algorithms like LSQ and PACT that dynamically optimize quantization step sizes and activation clipping; • Hybrid strategies integrating static and dynamic calibration for improved accuracy and efficiency. These innovations enable models to operate at ultra-low bit-widths with minimal performance loss, bridging the gap between theoretical compression and practical deployment, especially on edge hardware.

• �� Define quantization functions (e.g., Q(r) = Int(r/S) + Z) with learnable parameters; • Use static calibration via calibration datasets or dynamic calibration during inference to set clipping ranges; • Apply multi-granularity schemes—layerwise, channelwise, blockwise—to tailor quantization to distribution characteristics; • Implement algorithms like LSQ and PACT to learn optimal step sizes and clipping thresholds during training; • Combine quantization with pruning and distillation for further compression; • Evaluate on datasets such as ImageNet and COCO, measuring accuracy, model size, and latency, with ablation studies to isolate effects of each component.

Models tested include ResNet-50, MobileNetV2, and Transformer architectures, evaluated on ImageNet and COCO datasets. Metrics include top-1 accuracy, model size, inference latency, and energy consumption. Comparisons involve INT8, INT4, and INT2 models, with ablation studies on calibration methods, granularity, and algorithms. Hyperparameters such as learning rate, quantization step size, and training epochs are optimized for each setting. Results show that INT4 models incur less than 1% accuracy loss, while INT2 models retain over 90% accuracy, with significant reductions in model size and inference time. Experiments also demonstrate robustness across tasks and hardware platforms.

INT4 quantized ResNet-50 achieves <1% accuracy drop, with 4× compression and 30% faster inference. Transformer models with INT2-4 quantization retain >90% accuracy, reducing model size to 20%. Layer, channel, and block-level quantization with dynamic calibration improve edge inference speed and energy efficiency, demonstrating practical deployment potential. Ablation studies confirm the effectiveness of learned step sizes and calibration strategies, with hybrid approaches outperforming single-method baselines.

This technology enables efficient deployment of deep models in mobile phones, autonomous vehicles, and IoT devices, where computational resources and energy are limited. It supports real-time inference with minimal latency and power consumption, crucial for applications like facial recognition, speech processing, and medical diagnostics. The algorithms are compatible with existing hardware accelerators, facilitating widespread adoption. Long-term, integrating quantization with neural architecture search (NAS) and automated calibration could lead to fully autonomous, resource-aware AI systems.

Extreme low-bit quantization (INT2) still faces accuracy challenges, especially in complex tasks. Dynamic calibration, while accurate, incurs high computational costs, limiting real-time use. Training stability and convergence issues remain, requiring further algorithmic refinement. Hardware-specific constraints may restrict general applicability, and robustness against adversarial attacks under quantization needs investigation. Addressing these limitations will be key to broader adoption and practical deployment.