Qift: Shift-Friendly No-Zero W2 Post-Training Quantization for Rotated W2A4/KV4 LLM Inference
Qift proposes fixed no-zero W2 level sets based on Gaussian-like weight distribution, significantly improving LLaMA W2A4 inference performance.
Key Findings
Methodology
This paper investigates the scalar level-set geometry of two-bit weights after Hadamard rotation, revealing their approximate Gaussian distribution. Based on this, Qift introduces fixed no-zero level sets {±1, ±3} and {±1, ±4}, designed to match the rotated weights' statistical properties. The approach involves analyzing the inner/outer centroid ratio, identifying the optimal range of 0.25–0.33 for effective quantization. Empirical validation on LLaMA-2-7B and LLaMA-3.1-8B shows consistent improvements over standard symmetric level sets in perplexity, accuracy, and GPTQ residuals, without requiring training or learned codebooks. The method maintains per-channel scales and is hardware-friendly, suitable for deployment in resource-constrained environments.
Key Results
- In LLaMA-2-7B, pure W2A4 perplexity drops from 53.8 to 33.5 with Qift, with significant gains in downstream accuracy. The mixed W2/W4 configuration at 16 layers narrows the gap to W3A4, maintaining half the layers at two-bit precision. Across multiple downstream tasks, Qift improves perplexity and residuals, demonstrating robustness and versatility.
- In LLaMA-3.1-8B, similar performance gains are observed, with perplexity reductions exceeding 40%. The analysis of centroid ratios confirms that the effective optimal ratio lies between 0.25 and 0.33, aligning with the Gaussian-like source model. Ablation studies show that removing zero levels and optimizing centroid ratios are key to performance improvements.
- Overall, Qift effectively addresses the challenges of extreme low-bit quantization, providing a simple, source-aware, deployment-friendly alternative to learned codebooks, with consistent gains across models and tasks.
Significance
This work advances the state-of-the-art in low-bit quantization for large language models, especially in the W2A4 regime. By leveraging the statistical properties of rotated weights, it offers a theoretically grounded and practically effective design for fixed scalar level sets. The approach simplifies hardware implementation, reduces model size, and maintains high inference accuracy, making it highly relevant for deploying LLMs on edge devices and resource-limited hardware. It also opens new avenues for research into source-aware quantization level design, bridging the gap between theoretical models and practical engineering.
Technical Contribution
The key innovation lies in treating the scalar level set as a design variable informed by the Gaussian-like distribution of rotated weights. The fixed no-zero level sets {±1, ±3} and {±1, ±4} are derived from Lloyd-Max principles, optimized via centroid ratio analysis. This approach avoids the complexity of learned codebooks or asymmetric zero-points, maintaining simplicity and hardware compatibility. The method integrates seamlessly with existing PTQ pipelines like GPTQ and GPTAQ, providing a plug-and-play solution that significantly improves quantization performance in the W2A4 regime.
Novelty
This is the first work to explicitly incorporate the statistical shape of rotated weights into the scalar level-set design for two-bit quantization. Unlike prior methods relying on learned or asymmetric codebooks, Qift employs a fixed, source-aware, no-zero level set based on Gaussian approximation, offering a novel, simple, and effective solution. Its focus on the inner/outer centroid ratio as a key design parameter is a unique contribution that enhances understanding and performance of extreme low-bit quantization.
Limitations
- The approach assumes the rotated weights are approximately Gaussian; deviations from this assumption, such as highly skewed distributions, may reduce effectiveness. In such cases, performance could degrade, requiring adaptive or learned adjustments.
- While fixed level sets simplify deployment, they may lack flexibility in highly heterogeneous models or tasks with different weight distributions, necessitating further adaptation.
- The method primarily targets W2A4 inference; extending to even lower bits or more complex architectures might require additional innovations or hybrid strategies.
Future Work
Future research could explore adaptive centroid ratio tuning based on real-time weight distribution diagnostics, extending the fixed level set concept. Integrating this with learned or hybrid quantization schemes may further improve robustness. Additionally, applying the approach to other model architectures, such as vision transformers or multimodal models, and optimizing hardware implementations for even lower latency and power consumption, are promising directions.
AI Executive Summary
The rapid growth of large language models (LLMs) has brought about significant challenges in storage and inference efficiency. While high-precision parameters enable impressive performance, they demand vast memory and computational resources. Quantization, especially at extremely low bit-widths like two bits, offers a promising solution but faces hurdles due to quantization errors and distribution mismatches.
This paper introduces Qift, a novel fixed no-zero scalar level set design tailored for rotated W2A4/KV4 inference. By analyzing the statistical properties of weights after Hadamard rotation, the authors demonstrate that these weights approximate a Gaussian distribution centered near zero. Leveraging this insight, they propose level sets {±1, ±3} and {±1, ±4}, which avoid placing a quantization level at zero, thus better capturing the dense central weight distribution. This approach is grounded in Lloyd-Max scalar quantization principles and validated through extensive experiments on LLaMA models.
Empirical results show that Qift significantly reduces perplexity, improves downstream task accuracy, and enhances GPTQ residual behavior compared to traditional symmetric level sets. Notably, in a 16-layer mixed W2/W4 setting, the performance approaches that of W3A4, while maintaining only two-bit weights in half of the transformer layers. The method's simplicity, hardware friendliness, and source-aware design make it a practical alternative to complex learned codebooks, facilitating efficient deployment of large models on resource-constrained hardware.
Overall, this work bridges theoretical insights with engineering practicality, advancing the frontier of extreme low-bit quantization. It highlights the importance of source distribution modeling in quantizer design and opens avenues for further adaptive and hybrid quantization strategies, promising broader impacts in AI hardware and model compression fields.
Deep Analysis
Background
近年来，随着大规模语言模型（LLMs）在自然语言处理中的广泛应用，模型的存储和推理效率成为研究热点。传统高精度参数存储带来庞大硬件成本，极端低比特量化（如二比特）虽能大幅压缩模型，但在重建误差和性能保持方面面临巨大挑战。早期工作如量子化（Quantization）和剪枝（Pruning）解决了部分存储问题，但在极低比特下，重建误差显著增加，模型性能明显下降。旋转变换技术（如QuaRot）通过正交变换缓解通道内离群值，提升量化效果。GPTQ、GPTAQ等方法引入校准和误差补偿，改善量化精度，但对重建级别集的设计仍未充分考虑旋转后权重的统计特性。本文基于权重旋转后高斯样本，重新设计二比特重建级别，填补了该研究空白。
Core Problem
极端二比特量化的核心难题在于有限重建级别导致的高误差，尤其在旋转变换后，权重趋于高斯分布，传统对称级别集（如{-2, -1, 0, 1}）在零点位置造成信息浪费，影响模型性能。现有方案多依赖训练或学习码本，复杂度高且硬件实现困难。如何设计一个符合旋转后高斯样源统计特性的固定级别集，成为提升极端低比特量化性能的关键。
Innovation
本研究提出Qift方案，创新点在于：• 基于旋转后权重的高斯样特性，设计无零点的固定四级级别集（如{±1, ±3}和{±1, ±4}），避免在重建级别集中放置零点，提升表达能力。• 通过分析内外质心比，确定0.25–0.33范围内的最优比值，优化级别布局。• 方案无需训练、无需学习码本，保持每通道尺度，硬件友好，兼容多种PTQ技术。• 理论验证和实证结果显示，该设计在极端压缩场景中优于传统对称级别集，显著提升模型性能。
Methodology
- �� 研究旋转后权重的统计特性，假设其为近似零中心高斯分布。• 设计无零点的四级级别集（MNZ和POT-MNZ），避免在重建级别集中放置零点，提升表达能力。• 进行内外质心比分析，确定0.25-0.33范围内的最优比值，确保模型性能。• 在LLaMA-2-7B和LLaMA-3.1-8B模型中，应用固定级别集替代传统对称集，结合现有PTQ技术（如GPTQ、GPTAQ）进行校准和重建。• 评估困惑度、准确率和GPTQ残差，验证方案有效性。• 实现硬件友好的整数级别结构，确保部署便利。
Experiments
采用LLaMA-2-7B和LLaMA-3.1-8B模型，进行纯W2A4和混合W2/W4设置。使用WikiText-2作为校准集，评估困惑度和下游任务表现。对比标准W2级别集和多种设计变体，验证不同内外质心比的影响。调整尺度参数，确保最优重建误差。实验结果显示，Qift在极端压缩场景中表现优异，显著优于传统对称方案，验证了其有效性。
Results
Qift在纯W2A4设置中，困惑度由53.8降至33.5，模型精度提升显著。在16层混合W2/W4配置中，困惑度接近W3A4水平，性能差距缩小一半以上。多任务评估显示，Qift提升了下游任务的准确率，GPTQ残差也得到改善。不同级别比分析表明，内外质心比在0.25-0.33范围内最优，验证了高斯样源模型的合理性。整体而言，Qift在极端低比特场景中实现了性能突破。
Applications
该方法适用于边缘设备和低功耗场景的LLM部署，特别在存储受限或对推理速度要求高的应用中。通过固定级别集，简化硬件实现，降低部署成本。未来可结合硬件优化，推动极端低比特模型在移动端、物联网等场景的应用，提升模型普及率和实用性。
Plain Language Accessible to non-experts
想象你有一个装满各种颜色球的盒子，代表模型中的参数。传统方法就像用很多不同大小的球来表示参数，但这样很占空间。现在，你希望用只有几种大小的球来代表所有参数，这样可以节省空间。问题是，如果用的球太少，不能准确表达参数的细节，就会导致模型表现变差。这个研究就像设计一种特殊的球，虽然只有几种大小，但它们的分布和参数的实际情况很像。通过巧妙安排这些球的大小和位置，就能用很少的球，准确地表达参数的特性，让模型既节省空间，又保持性能。这就像用几种不同的尺子拼凑出复杂的图形，既简单又有效。
ELI14 Explained like you're 14
想象你在玩一个游戏，你的角色有很多装备，比如武器、盔甲和道具。每件装备都可以用数字表示，但如果数字太大，存放和传输就很麻烦。于是，你决定用只有几个数字的小箱子来代表所有装备，比如只用1、2、3、4这几个数字。问题是，这些数字能不能准确表达装备的强度和特点呢？如果数字太少，可能会丢失一些细节，导致游戏表现变差。这个研究就像设计一种特别的数字箱子，把装备的特点用几种数字巧妙地表示出来，让游戏既流畅又不失趣味。它告诉我们，用少量的数字也能很好地表达复杂的东西，只要设计得巧妙，就像用有限的颜色画出丰富的画面一样。
Abstract
Two-bit weight quantization is attractive for memory-efficient LLM inference, but the standard W2 level set {-2,-1,0,+1} often collapses under aggressive W2A4/KV4 settings. We study the scalar level-set geometry of two-bit weights in a Hadamard-rotated quantization pipeline. Conventional asymmetric W2 substantially improves over the standard level set, indicating that W2A4 failure is not only a bit-width problem but also a reconstruction-level problem. Across all 224 linear modules in each of LLaMA-2-7B and LLaMA-3.1-8B, pretrained weights are already nearly zero-centered, while Hadamard rotation primarily Gaussianizes their standardized shape: excess kurtosis and Q-Q error drop by orders of magnitude. Based on this approximate zero-centered Gaussian-like source model, we propose Qift, a fixed no-zero W2 level set for rotated W2A4/KV4 inference. The main level set is {+/-0.5, +/-1.5}, equivalently {+/-1, +/-3} under a half-scale reparameterization; a power-of-two variant uses {+/-1, +/-4} for sign-and-shift decoded weight application. Qift redesigns the fixed two-bit code-to-level mapping and is training-free, learned-codebook-free, group-grid-free, and zero-point-free, retaining the standard per-channel scale. A scale-invariant ratio analysis identifies an effective inner/outer centroid ratio range of 0.25 to 0.33, explaining why mirror no-zero (MNZ), Lloyd, NF2, and PoT-MNZ perform well while {+/-1, +/-2} does not. On both models, the no-zero level sets consistently improve pure W2A4 perplexity, L-layer mixed W2/W4 perplexity, downstream accuracy, and GPTQ residual behavior over the standard W2 level set. At L=16 mixed precision, they substantially narrow the gap to W3A4 while keeping half of the transformer layers at two-bit precision, giving a simple, source-aware, and deployment-friendly alternative to more complex learned W2 codebooks.