Quant.npu: Enabling Efficient Mobile NPU Inference for on-device LLMs via Fully Static Quantization
Quant.npu improves mobile NPU inference efficiency via fully static quantization, reducing latency by 15.1%.
Key Findings
Methodology
Quant.npu is an integer-only fully static quantization framework that incorporates learnable quantization parameters and rotation matrices, enabling low-bit activation-weight quantization without runtime parameter re-computation. It uses rotation-and-bit-width-aware initialization and distribution-aware selective optimization to stabilize the optimization process, and introduces a sensitivity-guided adaptive mixed-precision scheme.
Key Results
- Quant.npu reduces inference latency by 15.1% on real mobile NPUs, achieving accuracy comparable to state-of-the-art methods.
- Tested on Qualcomm SM8650 with 4 mobile-sized LLMs, average accuracy drop of 2.58%, PPL increase of 1.23.
- Quant.npu consistently achieves higher accuracy on downstream tasks at identical inference latency compared to existing approaches.
Significance
This research significantly enhances inference efficiency for large language models on mobile devices, addressing incompatibility issues between existing dynamic quantization methods and NPU hardware constraints, providing new solutions for privacy protection and low-latency operations on mobile devices.
Technical Contribution
Quant.npu bridges the gap between dynamic quantization and static compilation constraints, offering new theoretical guarantees and engineering possibilities, particularly in optimization stability and inference efficiency.
Novelty
Quant.npu is the first to achieve fully static quantization combined with rotation matrices, resolving the contradiction between dynamic quantization and NPU static compilation constraints, providing a more efficient inference solution.
Limitations
- Accuracy may degrade in extreme activation outlier scenarios.
- Further optimization of rotation matrix selection is needed.
Future Work
Future research could explore more complex rotation matrix optimization methods and applications on more hardware platforms.
AI Executive Summary
As large language models are increasingly deployed on mobile devices, existing dynamic quantization methods fail to meet the static compilation requirements of NPU hardware. Quant.npu proposes an integer-only fully static quantization framework that combines learnable quantization parameters and rotation matrices, achieving low-bit activation-weight quantization without runtime parameter re-computation. Experimental results show that this method achieves accuracy comparable to state-of-the-art methods while reducing inference latency by 15.1% on real mobile NPUs. This innovation not only resolves the contradiction between dynamic quantization and static compilation constraints but also provides new solutions for privacy protection and low-latency operations on mobile devices. Future research could further optimize rotation matrix selection and explore applications on more hardware platforms.
Deep Analysis
Background
As large language models become more prevalent on mobile devices, NPUs have become crucial hardware for accelerating inference. However, existing dynamic quantization methods fail to meet the static compilation requirements of NPUs, leading to inefficient inference. Researchers have attempted to solve this issue through optimization of rotation matrices and quantization parameters.
Core Problem
Existing dynamic quantization methods are incompatible with the static compilation requirements of NPU hardware, leading to inefficient inference. Improving inference efficiency while maintaining model accuracy is a pressing challenge.
Innovation
Quant.npu achieves fully static quantization by combining learnable quantization parameters and rotation matrices, resolving the contradiction between dynamic quantization and static compilation constraints. It stabilizes the optimization process through rotation-and-bit-width-aware initialization and distribution-aware selective optimization.
Methodology
- �� Rotation-and-bit-width-aware Initialization: Choose appropriate initialization methods based on activation distribution. • Distribution-aware Selective Optimization: Divide the quantization process into two stages to reduce optimization complexity. • Adaptive Mixed Precision: Select appropriate bit-widths based on sensitivity metrics.
Experiments
Tested on Qualcomm SM8650 with 4 mobile-sized LLMs, evaluated using 6 accuracy datasets. The introduction of rotation matrices improved model accuracy and inference efficiency.
Results
Quant.npu achieves accuracy comparable to state-of-the-art methods while reducing inference latency by 15.1% on real mobile NPUs. It consistently achieves higher accuracy on downstream tasks at identical inference latency compared to existing approaches.
Applications
Quant.npu can be used for large language model inference on mobile devices, providing more efficient privacy protection and low-latency operations.
Limitations & Outlook
Accuracy may degrade in extreme activation outlier scenarios. Further optimization of rotation matrix selection is needed.
Plain Language Accessible to non-experts
Imagine a kitchen where the chef needs to quickly prepare ingredients. Existing methods are like the chef recalculating the amount of ingredients each time, while Quant.npu is like having all ingredient amounts prepared in advance, allowing the chef to cook according to the prepared amounts. This method not only saves time but also ensures the quality of the dishes.
ELI14 Explained like you're 14
Hey, imagine you're playing a game and your character needs to level up quickly. Existing methods are like recalculating experience points each time you level up, while Quant.npu is like having all experience points prepared in advance, allowing you to level up according to the prepared points. This not only saves time but also ensures your character's strength.
Glossary
Fully Static Quantization
A quantization method where all parameters are determined before runtime, eliminating dynamic computation.
Used to improve NPU inference efficiency.
Rotation Matrix
Used to adjust activation distribution, reducing the impact of outliers on quantization.
Combined with quantization parameter optimization.
Sensitivity Metric
Used to evaluate activation distribution sensitivity to quantization, guiding mixed precision selection.
Used in adaptive mixed precision schemes.
Mixed Precision
Selects different bit-widths as needed to balance accuracy and efficiency.
Used to optimize inference efficiency.
Activation Outliers
Extreme values in activation distribution that may cause quantization errors.
Affect quantization accuracy.
Open Questions Unanswered questions from this research
- 1 How to further optimize rotation matrices to improve quantization efficiency?
- 2 What are the challenges of applying Quant.npu on more hardware platforms?
Applications
Immediate Applications
Mobile Device Inference
Quant.npu can be used to improve inference efficiency for large language models on mobile devices, reducing latency.
Long-term Vision
Privacy Protection
By improving inference efficiency on mobile devices, Quant.npu can achieve better privacy protection.
Abstract
Large language models (LLMs) are increasingly deployed on mobile devices, where Neural Processing Units (NPUs) necessitate fully static quantization for optimal inference efficiency. However, existing post-training quantization (PTQ) methods predominantly rely on dynamic activation quantization, rendering them incompatible with NPU hardware constraints. To bridge the gap between high-fidelity PTQ and NPU-constrained inference, we propose Quant.npu, a integer-only fully static quantization framework. It incorporates learnable quantization parameters and rotation matrices, enabling low-bit activation-weight quantization without runtime quantization parameters re-computation. Crucially, we identify that initialization and selective optimization of quantization parameters is pivotal for optimization stability, as improper initialization and naive joint optimization induce gradient instability that disrupts the optimization of rotation matrices. To address this, we propose a rotation-and-bit-width-aware initialization tailored to diverse activation profiles and a distribution-aware selective optimization (two-stage quantization pipeline) tailored to rotated and unrotated tensors. Furthermore, we introduce a sensitivity-guided adaptive mixed-precision scheme to balance accuracy with inference efficiency. Extensive experiments on real-world mobile NPUs demonstrate that Quant.npu achieves comparable accuracy to state-of-the-art methods, while reducing inference latency by up to 15.1%.