Up or Down? Adaptive Rounding for Post-Training Quantization
AdaRound uses Taylor expansion-based adaptive rounding, outperforming nearest neighbor, achieving <1% accuracy loss on 4-bit ResNet without fine-tuning.
Key Findings
Methodology
This work develops a theoretical framework that considers data distribution and task loss, transforming the weight rounding problem into a per-layer Quadratic Unconstrained Binary Optimization (QUBO). By approximating the task loss with a second-order Taylor series, a local layer-wise MSE loss is formulated. The approach involves analyzing the Hessian of the task loss, simplifying it via diagonal approximation, and solving the resulting binary optimization through continuous relaxation and regularization. The AdaRound algorithm employs layer-wise optimization, combining local MSE and asymmetric reconstruction, requiring only minimal unlabeled data and no fine-tuning, yet achieving significant performance gains.
Key Results
- On ImageNet, AdaRound quantizes ResNet18 and ResNet50 to 4 bits with less than 1% accuracy loss, outperforming nearest neighbor by over 10%. Similar improvements are observed on models like MobileNetV2 and InceptionV3. Ablation studies confirm the importance of Taylor expansion and local MSE objectives. The method maintains high accuracy while drastically reducing storage and computation, facilitating edge deployment.
- The layer-wise Taylor-based approximation effectively captures the impact of quantization, enabling optimization without full model fine-tuning. The experimental results demonstrate robustness across architectures and datasets, with consistent accuracy improvements and efficiency gains.
- In practical terms, AdaRound enables near-lossless 4-bit quantization of large models, significantly reducing model size and inference latency, thus broadening the applicability of deep learning in resource-constrained environments.
Significance
This research addresses the fundamental limitation of traditional rounding strategies in neural network quantization, introducing a theoretically grounded, data-adaptive approach that significantly enhances post-training quantization performance. Its ability to achieve high accuracy with minimal data and no fine-tuning simplifies deployment pipelines, reduces costs, and accelerates the adoption of low-bit models in industry. The method's theoretical foundation and empirical validation fill a critical gap in the literature, providing a new paradigm for model compression and efficient inference, especially in edge and embedded systems.
Technical Contribution
The main technical innovations include: 1) deriving a task loss approximation via second-order Taylor expansion, capturing the influence of weight perturbations; 2) formulating the weight rounding as a layer-wise QUBO problem, enabling systematic optimization; 3) proposing AdaRound, which combines continuous relaxation with regularization and asymmetric reconstruction to efficiently solve the binary optimization. The approach surpasses existing PTQ methods by leveraging model-data interactions and theoretical insights, achieving state-of-the-art results at low bit-widths without retraining.
Novelty
This work is the first to incorporate second-order task loss analysis into post-training weight rounding, transforming the problem into a layer-wise QUBO. Unlike previous heuristic or stochastic methods, AdaRound optimizes rounding directions based on model-specific Hessian information, leading to substantial accuracy improvements at 4-bit quantization. Its integration of Taylor expansion, binary optimization, and asymmetric reconstruction represents a novel, comprehensive framework that advances the state-of-the-art in PTQ.
Limitations
- The method relies on the diagonal Hessian approximation, which may introduce bias in some models with strong parameter interactions, potentially limiting performance in certain architectures.
- Solving QUBO problems, although approximated efficiently, remains computationally intensive for very large models or layers with high dimensionality, requiring further optimization.
- The current approach focuses on weight quantization; extending it to activation quantization and hardware-aware schemes remains future work. Additionally, the method's effectiveness at ultra-low bit-widths (e.g., 2-bit) needs further validation.
Future Work
Future directions include integrating AdaRound with quantization-aware training for further accuracy gains, developing hardware-friendly quantization grids, and exploring joint optimization of weights and activations. Improving QUBO solvers for large-scale problems and extending the framework to mixed-precision quantization will further enhance its practical impact. Additionally, investigating the Hessian approximation's accuracy and robustness could lead to more precise and scalable solutions.
AI Executive Summary
Deep neural networks have revolutionized fields such as computer vision and natural language processing, but their deployment on resource-constrained devices remains challenging due to large model sizes and computational demands. Quantization, which reduces the precision of weights and activations, offers a promising solution. However, traditional post-training quantization methods, primarily relying on nearest neighbor rounding, often suffer from significant accuracy degradation at low bit-widths, such as 4 bits. This problem stems from ignoring the complex interactions between model parameters and data distribution.
In this context, the authors introduce AdaRound, an innovative quantization scheme that leverages a second-order Taylor series approximation of the task loss to guide weight rounding. By formulating the problem as a layer-wise QUBO, AdaRound optimizes each layer's rounding decisions through a continuous relaxation, regularization, and an asymmetric reconstruction strategy. This approach effectively captures the joint effects of weight perturbations, enabling the model to maintain high accuracy without retraining.
Extensive experiments on ImageNet with ResNet architectures demonstrate that AdaRound achieves near-lossless 4-bit quantization, with less than 1% accuracy loss, outperforming traditional methods by over 10%. The method's efficiency and minimal data requirements make it highly suitable for practical deployment, especially in edge devices. Its theoretical foundation and empirical success mark a significant step forward in neural network compression.
Despite its strengths, the approach faces challenges such as reliance on Hessian diagonal approximation and computational costs for very large models. Future research aims to integrate AdaRound with quantization-aware training, develop hardware-friendly schemes, and extend the framework to activation quantization and mixed-precision settings. Overall, AdaRound paves the way for more efficient, accurate, and accessible deep learning models in real-world applications.
Deep Analysis
Background
Neural network模型的快速发展带来了模型规模的不断扩大，存储和计算成本激增，限制了其在边缘设备上的应用。早期研究如Hamming码和硬件设计专注于8-16位硬件实现（Hammerstrom, 1990；Holi & Hwang, 1993），但随着模型复杂度提升，低比特量化成为节省资源的关键。近年来，量化感知训练（QAT）和后训练量化（PTQ）成为主流，Jacob et al. (2018)、Gupta et al. (2015)提出了训练中模拟量化的方法，取得了显著效果。许多研究开始关注微调范围、通道拆分和不同层的比特宽度调节（Kim et al., 2019；Zhao et al., 2019），以降低性能损失。尽管如此，微调繁琐，难以快速部署，促使研究转向无需微调的PTQ方法。本文聚焦于此，试图通过优化舍入策略，提升量化效果。
Core Problem
传统的后训练量化方法多采用最邻近舍入策略，忽视了模型参数与数据的复杂关系，导致在极低比特宽度（如4-bit）下性能下降明显。如何在不微调的情况下，优化舍入方向，最大限度减少量化引入的性能损失，成为亟待解决的问题。现有方法缺乏理论支撑，难以充分利用模型的二阶信息，导致优化空间受限。此外，舍入的非凸性和高维性使得问题难以求解，尤其是在大规模网络中，计算复杂度成为瓶颈。
Innovation
本文的核心创新在于：1）引入泰勒级数展开，系统分析舍入对任务损失的影响，建立理论基础；2）将舍入问题转化为每层的QUBO优化，利用连续松弛和正则化策略高效求解；3）设计了AdaRound，结合局部均方误差（MSE）和异步重建机制，显著优于传统的最邻近舍入。该方法不依赖微调，且只需少量无标签数据，极大提升了后训练量化的性能和实用性。
Methodology
- �� 建立任务损失的二阶泰勒展开，近似舍入对模型性能的影响。
- �� 将舍入问题转化为每层的QUBO优化，定义二值变量对应舍入方向。
- �� 利用连续松弛（h(V)）和正则化（freg）策略，解决NP-hard问题。
- �� 设计异步重建机制，考虑激活函数和前向误差，逐层优化。
- �� 采用Adam优化器，结合正则化和退火策略，训练舍入向量。
- �� 在多层网络中逐层应用，避免误差累积，提升整体性能。
Experiments
采用ImageNet数据集，评估ResNet18、ResNet50、MobileNetV2等模型的4-bit量化性能。对比最邻近舍入、随机舍入和AdaRound，使用Top-1准确率作为指标。设置尺度参数s，利用少量无标签图像进行优化，超参数包括学习率、正则化系数等。通过消融实验验证不同近似和正则化策略的影响，分析算法鲁棒性和泛化能力。
Results
AdaRound在ResNet18和ResNet50上实现了4-bit量化后，准确率误差控制在1%以内，优于传统方法10%以上。在多个网络（如InceptionV3、DeepLabV3）上也获得了类似提升。消融实验显示，泰勒展开和局部MSE目标是性能提升的关键因素。算法在保持模型性能的同时，大幅降低存储和计算成本，为边缘设备部署提供了可行方案。
Applications
该方法适用于模型压缩、边缘推理和硬件加速场景。只需少量无标签数据和简单的后处理步骤，即可在现有模型基础上实现低比特宽度的高性能量化。适合在资源受限的设备上部署深度学习模型，降低能耗和延迟，推动智能应用的普及。
Limitations & Outlook
算法在极端低比特（如2-bit）时效果可能受限，且对模型结构的依赖较强。Hessian矩阵的对角近似在某些模型中可能引入偏差，影响性能。大规模模型中，QUBO求解的时间成本仍较高，未来需优化求解策略和近似模型。此外，未考虑激活量化和硬件友好格点设计，仍有提升空间。
Abstract
When quantizing neural networks, assigning each floating-point weight to its nearest fixed-point value is the predominant approach. We find that, perhaps surprisingly, this is not the best we can do. In this paper, we propose AdaRound, a better weight-rounding mechanism for post-training quantization that adapts to the data and the task loss. AdaRound is fast, does not require fine-tuning of the network, and only uses a small amount of unlabelled data. We start by theoretically analyzing the rounding problem for a pre-trained neural network. By approximating the task loss with a Taylor series expansion, the rounding task is posed as a quadratic unconstrained binary optimization problem. We simplify this to a layer-wise local loss and propose to optimize this loss with a soft relaxation. AdaRound not only outperforms rounding-to-nearest by a significant margin but also establishes a new state-of-the-art for post-training quantization on several networks and tasks. Without fine-tuning, we can quantize the weights of Resnet18 and Resnet50 to 4 bits while staying within an accuracy loss of 1%.