CAT-Q: Cost-efficient and Accurate Ternary Quantization for LLMs

Recent years have seen exponential growth in large language models (LLMs) like GPT-3, BERT, and their variants, achieving remarkable performance across NLP tasks. However, their enormous parameter counts—ranging from billions to hundreds of billions—bring prohibitive storage, computational, and energy costs, hindering widespread deployment. Early compression techniques such as weight pruning and knowledge distillation provided some relief but were limited in compression ratio and efficiency. Quantization emerged as a promising solution, reducing precision from FP32 to INT8, INT4, or even binary/ternary formats. Quantization-aware training (QAT) methods, like BitNet and TriLM, achieved high accuracy but require extensive retraining on massive datasets, often involving hundreds of billions of tokens. Post-training quantization (PTQ), which uses minimal calibration data, offers a practical alternative but struggles with ultra-low-bit accuracy, especially in models exceeding tens of billions of parameters. Existing PTQ methods are limited in scalability and performance, motivating the need for novel approaches that combine efficiency with accuracy.

Deploying super-large models efficiently remains a major challenge. The core issues include the difficulty of maintaining model fidelity when reducing weights to only three states ({-1, 0, 1}), which risks severe information loss. Additionally, the optimization process for ternary weights is inherently non-differentiable and prone to convergence issues, especially under PTQ constraints with limited calibration data. Existing solutions like QAT are resource-intensive and not scalable for models beyond a few billion parameters. Therefore, developing a method that can perform accurate, stable, and fast PTQ with minimal data and computational overhead is critical for practical large-model deployment.

The paper introduces CAT-Q, which innovates through: 1) Learnable modulation (LM), adjusting weight distributions via learnable parameters to reduce distribution mismatch; 2) Softened ternarization (ST), employing a novel transition function that smoothly approximates hard ternarization, ensuring stable convergence; 3) Sliding-layer optimization, jointly refining multiple layers to leverage inter-layer dependencies and reduce quantization errors. This integrated framework enables high-precision, low-cost PTQ for models up to 235B parameters, a feat previously unattainable with minimal calibration data. The combination of theoretical guarantees and practical efficiency marks a significant advance in model compression.

• �� Start with pre-trained high-precision weights W. • Apply LM by introducing learnable parameters (δμ, δα, δ∆) to adjust the mean, scale, and threshold, producing transformed weights ˆW. • Use a differentiable transition function f(W; s, Δ) that gradually shifts from identity to ternarization, controlled by a schedule parameter t. • During calibration, perform a two-stage process: first, asymptotic ternarization with continuous mapping; second, hard ternarization for final weights. • Optimize multiple layers jointly via sliding-layer output matching, minimizing the L2 loss between high-precision and quantized outputs. • Use gradient-based updates to refine learnable parameters, ensuring convergence and performance stability.

The evaluation spans models from 1.7B to 235B parameters, including dense and MoE architectures (Qwen3, Llama2, Ring-flash). Using only 512 samples (~1 million tokens), the method quantizes models into ternary weights. Performance is tested on five zero-shot reasoning benchmarks, comparing against QAT and other PTQ methods. Ablation studies verify the contributions of LM and ST components. Results show less than 2% accuracy loss on 30B models, and only 3% on 235B models, with quantization times under 60 hours on 8 GPUs. The experiments demonstrate scalability, efficiency, and superior accuracy retention.

CAT-Q consistently outperforms prior PTQ and QAT methods, achieving near-original performance with minimal calibration data. For example, on Qwen3-30B, accuracy on PIQA is 68.4%, just 2% below the full-precision baseline, with only 512 samples. On 235B models, performance drops by less than 3%, while training time is reduced from hundreds of hours to hours. The method maintains robustness across architectures, confirming its scalability and practicality for real-world deployment.

Ideal for deploying large models in resource-constrained environments like edge devices, mobile applications, and cloud services. Its minimal data requirement and fast quantization process enable rapid adaptation, reducing hardware costs and energy consumption. The approach can facilitate widespread AI adoption, especially in scenarios demanding quick model updates or low-latency inference. Future integration with hardware accelerators could further enhance efficiency.

Primarily optimized for linear layers; effectiveness on nonlinear modules like activation functions remains to be validated. Performance may degrade in tasks requiring ultra-low-bit representations or in models with high sparsity. The current framework may need adaptation for specialized architectures or multi-modal models. Further research is needed to improve robustness and extend applicability to diverse model types.