GSQ: Highly-Accurate Low-Precision Scalar Quantization for LLMs via Gumbel-Softmax Sampling

In the inference process of large language models (LLMs), memory and bandwidth costs have become a major challenge. To address this issue, weight quantization has become a standard method for efficient deployment. Existing quantization methods are mainly divided into two categories: simple scalar quantization techniques, such as GPTQ or AWQ, which are widely used but plateau in accuracy at 3-4 bits per parameter; and 'second-generation' vector or trellis quantized methods, such as QTIP, GPTVQ, and AQLM, which push the accuracy frontier at low bit-widths but are notoriously hard to implement and scale. In recent years, with the widespread application of large language models, how to reduce computational and storage costs while ensuring model performance has become an important research direction.

Existing scalar quantization methods perform poorly at low bit-widths, while vector quantization methods, although advantageous in accuracy, are difficult to implement and scale. How to improve low-bit quantization accuracy while maintaining compatibility with existing scalar inference kernels has become an urgent problem to solve. Especially in the application of large-scale models, achieving efficient quantization and deployment without affecting model performance is a challenging problem.

The core innovation of GSQ lies in its combination of Gumbel-Softmax relaxation, significantly enhancing scalar quantization accuracy at low bit-widths. • GSQ uses Gumbel-Softmax relaxation to jointly learn per-coordinate grid assignments and per-group scales. • GSQ closes most of the gap between scalar quantization and the QTIP frontier at 2 and 3 bits, while maintaining full compatibility with existing scalar inference kernels. • GSQ demonstrates scalability on trillion-scale Mixture-of-Experts models such as Kimi-K2.5, where vector-quantized methods are difficult to apply.

The detailed steps of the GSQ method are as follows: • Use Gumbel-Softmax relaxation to make the discrete grid selection process differentiable. • Jointly learn per-coordinate grid assignments and per-group scales. • Match the cardinality of the relaxation to the small number of levels available in the target bit-width regime, making the relaxation tight and the optimization tractable. • Conduct experiments on the Llama-3.1-8B/70B-Instruct models to verify GSQ's accuracy improvement at 2 and 3 bits. • Conduct experiments on trillion-scale Mixture-of-Experts models such as Kimi-K2.5 to verify GSQ's scalability.

The experimental design includes testing on the Llama-3.1-8B/70B-Instruct models and the Kimi-K2.5 model. • On the Llama models, quantize all non-embedding and non-head linear layers to verify GSQ's accuracy performance at 2 and 3 bits. • On the Kimi-K2.5 model, only quantize non-shared expert weights to verify GSQ's scalability on large-scale models. • Use 2-bit and 3-bit weight-only quantization configurations with a group size of 128. • Use a symmetric scalar quantizer where each group shares a single scale value.

Experimental results show that at 2 bits, GSQ improves average zero-shot accuracy by 4.76 and 4.14 points over the best scalar baseline (EfficientQAT), respectively. At 3 bits, GSQ matches or surpasses all scalar baselines and is essentially on par with QTIP on the 70B model. Notably, these results are obtained without zero-point parameters, indicating that the gains come from better optimization of the discrete assignments. GSQ scales to trillion-scale Mixture-of-Experts models such as Kimi-K2.5, achieving low-bit, near-lossless quantization where vector-quantized methods are difficult to apply.

Application scenarios of GSQ include: • Achieving efficient low-bit quantization in the inference process of large language models, reducing memory and bandwidth costs. • Demonstrating scalability in large-scale Mixture-of-Experts models, providing new possibilities for model compression and deployment. • Providing an efficient quantization solution in applications that require maintaining compatibility with existing scalar inference kernels.

The limitations of GSQ include: • It may encounter instability in certain model layers, such as the down_proj of the second layer in the Llama-3.1-8B model, where compression is less effective. • While GSQ performs well in most cases, its performance at extremely low bit-widths (e.g., 1 bit) may not match some specially designed vector quantization methods. • The training process of GSQ requires additional computational resources, especially when handling large-scale models, which may increase training time and cost. Future research directions include further optimizing the GSQ training process to reduce computational resource requirements and exploring its performance at even lower bit-widths.