The case for 4-bit precision: k-bit Inference Scaling Laws
This study establishes that 4-bit quantization offers near-universal optimality for zero-shot performance and model size trade-offs in large language models, validated by 35,000 experiments.
Key Findings
Methodology
This work conducts over 35,000 zero-shot experiments across models like BLOOM, OPT, NeoX/Pythia, and GPT-2, varying input precision (16-bit) and parameter quantization (k=3~8 bits). It systematically evaluates the impact of different quantization data types (Quantile, Float, Int), block sizes (64~128), and outlier-dependent quantization (Proxy Quantization). Performance curves are fitted using linear interpolation, revealing that 4-bit parameters consistently achieve the best zero-shot accuracy for a fixed total bit budget across all model scales. The experiments incorporate diverse model sizes (19M to 176B), ensuring broad applicability.
Key Results
- Across all models and scales, 4-bit quantization nearly universally outperforms 3-bit and higher-bit models in zero-shot accuracy, with performance improvements comparable to increasing bit precision from 4 to 5 bits. Small block sizes (e.g., 64) significantly enhance the scaling behavior of 4-bit models, with improvements equivalent to a 0.24-bit increase per parameter. Data types like Quantile and Float outperform Integer types in 4-bit models. While Proxy Quantization stabilizes 3-bit models, it does not alter the fundamental optimality of 4-bit parameters.
- In the 6-8 bit range, existing quantization methods (centering, blocking, data types) do not improve the scaling laws, as parameters are already sufficiently expressive. The main gains in performance come from reducing block size and selecting appropriate data types. The research confirms that 4-bit quantization, combined with small blocks and float or quantile data types, offers the best trade-off between model size and zero-shot accuracy.
- The study also finds that outlier-dependent quantization improves the stability of 3-bit models but does not surpass 4-bit performance. The stability issues in 3-bit models are linked to emergent outlier features, which can be partially mitigated by proxy quantization, yet the fundamental scaling law favors 4-bit parameters for optimal performance.
Significance
This research provides a comprehensive quantitative framework for understanding the trade-offs between model size, quantization precision, and zero-shot performance in large language models. Demonstrating that 4-bit quantization nearly universally achieves optimal balance, it guides practical deployment strategies, reducing hardware costs and inference latency. The findings have profound implications for deploying massive models on resource-constrained devices, enabling broader accessibility and faster inference without significant performance loss. Moreover, the established scaling laws serve as a foundation for future research in low-bit quantization, pushing the boundaries of model compression and efficiency in AI systems.
Technical Contribution
The paper introduces a novel approach to modeling the scaling behavior of quantized large language models through linear interpolation of performance curves across different bit precisions. It systematically compares multiple data types and block sizes, establishing that 4-bit quantization with float or quantile data types and small blocks (64-128) yields near-optimal zero-shot accuracy. The integration of outlier-dependent proxy quantization enhances stability in 3-bit models, although it does not alter the fundamental optimality of 4-bit parameters. These contributions provide a rigorous theoretical and empirical basis for designing low-bit quantized models with predictable scaling behavior.
Novelty
This work is the first comprehensive study to establish universal scaling laws for zero-shot performance across multiple large language model families at various bit precisions, especially highlighting the near-universality of 4-bit quantization. It innovatively combines performance curve fitting with quantization strategies, including data type optimization and outlier-aware methods, to systematically analyze their impact on scaling laws. Unlike prior work focusing on individual models or specific quantization techniques, this research offers a broad, comparative framework that guides practical low-bit model deployment, representing a significant advancement in the field.
Limitations
- The analysis primarily focuses on zero-shot performance metrics, with limited exploration of fine-tuned or task-specific scenarios, which may exhibit different scaling behaviors.
- The study does not extensively investigate ultra-low bit quantization (e.g., 2-bit), where performance degradation and stability issues are more severe, leaving open questions about the lower bounds of quantization.
- Experiments are concentrated on certain model families; generalization to other architectures or emerging models requires further validation.
Future Work
Future research should explore adaptive and learned data types, as well as one-shot quantization techniques, to push beyond 4-bit limits while maintaining performance. Developing more robust outlier detection and handling algorithms could further stabilize ultra-low-bit models. Extending the analysis to multi-task and multimodal models, and integrating hardware-aware optimization, will facilitate practical deployment in resource-constrained environments. The ultimate goal is to establish a unified theoretical framework for low-bit quantization that can guide the design of next-generation efficient AI systems.
AI Executive Summary
Large language models (LLMs) have revolutionized natural language processing but face significant challenges in deployment due to their enormous size, requiring vast storage and computational resources. Quantization offers a promising solution by reducing parameter bit-widths, thus shrinking model size and inference latency. However, the optimal balance between quantization precision and performance remains unclear, especially at ultra-low bit levels.
This study systematically investigates the scaling laws governing zero-shot performance of quantized LLMs across multiple model families, including BLOOM, OPT, NeoX/Pythia, and GPT-2. By conducting over 35,000 experiments spanning parameter sizes from 19 million to 176 billion, the authors analyze how different quantization strategies—such as data types (Quantile, Float, Int), block sizes (64, 128), and outlier-dependent methods—affect performance. The key finding is that 4-bit quantization consistently offers the best trade-off, nearly universally outperforming 3-bit models and matching the performance of higher-bit models with significantly fewer bits.
The research reveals that small block sizes (around 64) and float or quantile data types are crucial for optimizing 4-bit models. While 3-bit models can be stabilized using outlier-aware proxy quantization, their scaling performance remains inferior to 4-bit counterparts. Notably, increasing bit precision beyond 4 bits yields diminishing returns, and existing methods do not improve scaling laws in the 6-8 bit range.
These insights have profound implications for deploying large models in resource-constrained environments, enabling faster inference and reduced storage without sacrificing accuracy. The findings serve as a practical guideline for model compression, emphasizing the importance of quantization data types and block sizes. Future directions include developing adaptive quantization schemes, extending analysis to ultra-low bits, and integrating hardware-aware optimizations, ultimately aiming to democratize access to powerful AI models with minimal resource requirements.
Deep Dive
Limitations & Outlook
What gaps remain?
Abstract
Quantization methods reduce the number of bits required to represent each parameter in a model, trading accuracy for smaller memory footprints and inference latencies. However, the final model size depends on both the number of parameters of the original model and the rate of compression. For example, a 30B 8-bit model and a 60B 4-bit model have the same number of bits but may have very different zero-shot accuracies. In this work, we study this trade-off by developing inference scaling laws of zero-shot performance in Large Language Models (LLMs) to determine the bit-precision and model size that maximizes zero-shot performance. We run more than 35,000 experiments with 16-bit inputs and k-bit parameters to examine which zero-shot quantization methods improve scaling for 3 to 8-bit precision at scales of 19M to 176B parameters across the LLM families BLOOM, OPT, NeoX/Pythia, and GPT-2. We find that it is challenging to improve the bit-level scaling trade-off, with the only improvements being the use of a small block size -- splitting the parameters into small independently quantized blocks -- and the quantization data type being used (e.g., Int vs Float). Overall, our findings show that {4-bit} precision is almost universally optimal for total model bits and zero-shot accuracy.