KronQ: LLM Quantization via Kronecker-Factored Hessian
KronQ employs Kronecker-factored Hessian approximation with gradient covariance, achieving state-of-the-art 2-bit quantization, e.g., perplexity 7.93 on LLaMA-3-70B.
Key Findings
Methodology
KronQ builds on Kronecker-factored Hessian approximation, integrating gradient covariance HG into the quantization loss. It extends GPTQ’s column-wise OBS by incorporating output-side second-order info, using a single backward pass to estimate HG. The framework employs bidirectional incoherence processing—applying orthogonal transforms to both input and output Hessians—to reduce quantization error. Additionally, it introduces a sensitivity metric based on the traces of HX and HG for inter-layer mixed-precision allocation, enabling more efficient bit-width distribution. The approach maintains computational efficiency while significantly improving low-bit quantization performance, validated on LLaMA-2/3 models across various tasks.
Key Results
- On LLaMA-3-70B, 2-bit weight-only quantization with GPTQ diverged (>2000 perplexity), whereas KronQ achieved 7.93 perplexity, showing a substantial performance boost. Across multiple models (7B, 13B, 70B), KronQ consistently outperformed baselines in perplexity and zero-shot accuracy on datasets like WikiText-2, PiQA, and ArcC. The method's ability to leverage output gradient covariance and Hessian traces led to better layer sensitivity ranking and more effective mixed-precision strategies.
- Experimental results demonstrate that KronQ maintains superior performance in both weight-only and weight-activation quantization, especially at W2/W3/W4 levels. Its novel sensitivity metric effectively distinguishes layer importance, resulting in optimized bit allocation. The method also exhibits robustness against outliers and model-specific anomalies, outperforming prior methods such as GuidedQuant and YAQA.
- The integration of gradient covariance into the Kronecker Hessian approximation and the bidirectional incoherence process collectively contribute to significant reductions in perplexity and improvements in model accuracy, validating the theoretical advantages of the proposed framework.
Significance
This work advances the state-of-the-art in post-training quantization by addressing the critical limitation of neglecting output-side second-order information. By incorporating gradient covariance, KronQ unlocks the potential for ultra-low-bit quantization without sacrificing model performance, enabling efficient deployment of massive language models on edge hardware. Its innovative combination of Kronecker approximation, bidirectional incoherence, and sensitivity-based layer scheduling offers a comprehensive solution that bridges the gap between theoretical Hessian-based optimization and practical model compression. The approach paves the way for more scalable, hardware-friendly AI systems, reducing memory footprint and inference latency, and broadening AI accessibility.
Technical Contribution
KronQ introduces a Kronecker-factored Hessian approximation that explicitly models output gradient covariance HG, which was previously ignored in PTQ. It extends the GPTQ framework by incorporating HG into the quantization objective, enabling more accurate error minimization. The method employs bidirectional incoherence processing—applying orthogonal transforms to both input HX and output HG matrices—to reduce spectral coherence, thus lowering quantization error. It derives a novel sensitivity metric based on the traces of HX and HG, facilitating automated mixed-precision layer allocation. The algorithm maintains computational efficiency, requiring only a single backward pass for HG estimation, and achieves significant performance gains at ultra-low bit-widths, especially in 2-bit quantization scenarios.
Novelty
This is the first work to incorporate gradient covariance HG into Kronecker-factored Hessian approximation for post-training quantization of large models. Unlike prior methods that only utilize input activation statistics, KronQ leverages output gradient information to better characterize layer sensitivity. Its bidirectional incoherence strategy—applying orthogonal transforms to both input and output Hessians—reduces spectral coherence, significantly improving quantization accuracy. The sensitivity metric based on the product of Hessian traces enables principled layer-wise mixed-precision allocation, a novel approach in this context. These innovations collectively push the boundary of ultra-low-bit quantization performance.
Limitations
- The method relies on a single backward pass to estimate HG, which may be insufficient for highly complex or non-convex models, potentially affecting accuracy in certain scenarios.
- Kronecker approximation assumes independence between input and output Hessian factors, which may not hold in models with strong layer interactions or non-linearities, limiting effectiveness.
- Computational overhead, while minimized, still exists during the orthogonal transforms and sensitivity calculations, especially for very large models or real-time applications.
Future Work
Future directions include refining gradient covariance estimation through multiple backward passes, extending the framework to non-linear layers, and exploring hardware-aware optimizations. Integrating adaptive strategies for dynamic layer sensitivity assessment and expanding to other model architectures could further enhance robustness and applicability. Additionally, combining KronQ with hardware-specific quantization schemes may facilitate deployment on edge devices, broadening AI's reach.
AI Executive Summary
Massive language models (LLMs) have revolutionized natural language processing, yet their deployment remains constrained by enormous memory and computational demands. Quantization offers a practical solution by reducing model precision, but existing methods like GPTQ often fall short at ultra-low bit-widths due to neglecting output-side second-order information. This gap hampers the ability to compress models without performance loss, especially at 2-bit precision. To address this, the paper introduces KronQ, a novel quantization framework leveraging Kronecker-factored Hessian approximation that incorporates gradient covariance HG. This approach captures output-layer sensitivities more accurately, enabling more effective error minimization.
KronQ extends GPTQ by integrating HG into the quantization objective, achieved through a single backward pass to estimate the gradient covariance matrix. It employs bidirectional incoherence processing—applying orthogonal transforms to both input HX and output HG matrices—to reduce spectral coherence, which is a key source of quantization error. Furthermore, the method derives a sensitivity metric based on the product of Hessian traces, allowing automated layer-wise mixed-precision allocation. These innovations collectively enable the model to maintain high accuracy even at 2-bit quantization.
Experimental results on LLaMA-2 and LLaMA-3 models demonstrate that KronQ surpasses prior state-of-the-art methods, achieving perplexity as low as 7.93 on LLaMA-3-70B at 2-bit weight quantization, where GPTQ diverges. The method consistently improves zero-shot performance across multiple benchmarks, validating its robustness and practical utility. Its ability to exploit output gradient information marks a significant step forward in model compression, making ultra-low-bit quantization feasible for large-scale models.
This work not only advances theoretical understanding by integrating output Hessian information into quantization but also offers a scalable, efficient algorithm suitable for real-world deployment. It opens new avenues for deploying massive models on edge hardware, reducing costs, and democratizing AI technology. Future research will focus on refining gradient estimation, extending to non-linear layers, and hardware-specific optimizations, promising broader impact across AI applications.
Deep Dive
Abstract
Post-training quantization (PTQ) is a widely adopted technique for compressing large language models (LLMs) without retraining. Most existing second-order PTQ methods, including GPTQ, construct quantization objectives from input activation statistics, effectively assuming that all output channels contribute equally to the layer-wise reconstruction objective. We propose KronQ, a PTQ framework that challenges this assumption by introducing the gradient covariance into the quantization pipeline. Under the Kronecker-factored Hessian approximation, the quantization loss depends jointly on both the activation and gradient covariances, and KronQ exploits this at two complementary levels. (1) KronQ introduces bidirectional incoherence processing, extending the existing input-side random rotation to the output dimension using the gradient covariance, reducing weight magnitude variance across both input and output dimensions. (2) KronQ derives a new sensitivity metric for inter-layer mixed-precision allocation, driven by the gradient and activation Hessian traces. Notably, in the case of 2-bit weight-only quantization on LLaMA-3-70B, while GPTQ and GPTAQ diverge or produce degenerate quantizations (>2000 perplexity on WikiText-2), \KronQ{} achieves 7.93 perplexity.