The Structure of Quantization Damage in LLMs: Why the Next Bit Should Be Spent Globally
Using causal mixed-precision intervention, the study finds damage is widespread; global fine-grained quantization outperforms local layer repair by 21-52 points.
Key Findings
Methodology
This paper employs progressive layer-wise elevation to 8-bit, measuring accuracy recovery as a causal ground truth. It tests three hypotheses: damage localization in task circuits, computation sites, or weight statistics, using localizers like circuit drift, activation patching, and weight variance. Comparing local repair versus global fine-grained quantization under matched budgets, results show damage is diffuse, with recovery benefits spread across many layers. The approach combines systematic experiments across 9 open models, validating the effectiveness of global strategies over targeted fixes.
Key Results
- Across 8 models, restoring 75% of the accuracy gap requires roughly half the layers, with Qwen3-8B showing concentrated damage in few layers. Global fine-grained quantization consistently outperforms local repair, improving accuracy by 21-52 points. Local indicators fail to predict which layers are most beneficial, damage is broadly distributed, and architecture influences recovery sites within families. Residual damage is near lossless at 8-bit, emphasizing the diffuse nature of quantization damage.
- The experiments demonstrate that a global approach to quantization granularity yields superior performance, with the most concentrated model Qwen3-8B favoring a few layers but still benefiting from global strategies. The damage distribution is not aligned with task circuits, computation sites, or weight statistics, challenging previous assumptions. The findings suggest that resource allocation for quantization should prioritize global fine-grained strategies rather than targeted layer protection.
Significance
This work fundamentally shifts the understanding of quantization damage, revealing its diffuse distribution across layers and invalidating the reliance on local signals for repair. It provides a robust theoretical and empirical foundation for adopting global fine-grained quantization strategies, which significantly improve model performance under limited precision budgets. The insights have broad implications for deploying large language models efficiently, reducing costs, and maintaining accuracy, thus advancing both academic research and industrial applications in model compression and efficient inference.
Technical Contribution
The paper introduces causal mixed-precision intervention as a validation tool for damage localization, demonstrating that damage is broadly dispersed rather than concentrated. It systematically evaluates localizers like circuit drift, activation patching, and weight statistics, showing their predictive limitations. The study establishes the superiority of global fine-grained quantization in resource-limited scenarios, providing a new theoretical perspective on damage distribution and resource allocation. It also uncovers architecture-dependent patterns within model families, enriching the understanding of damage structure and repair strategies.
Novelty
This is the first comprehensive application of causal intervention to validate the spatial distribution of quantization damage in large models. It challenges the conventional focus on local signals, demonstrating that damage is diffuse and that global strategies outperform targeted fixes. The integration of architecture-aware analysis and causal validation offers a novel framework for understanding and optimizing quantization, filling a significant gap in current literature.
Limitations
- The study is limited to models up to 8B parameters and specific quantization algorithms (RTN, GPTQ, AWQ); larger models and other algorithms need validation.
- Local repair strategies tested are greedy and layer-based; more sophisticated, non-greedy, or weight-level methods remain unexplored.
- Causal intervention incurs high computational costs, limiting practical deployment; approximate methods are needed for real-world application.
Future Work
Future research will extend to larger models, explore multi-layer joint repair, and incorporate structural pruning. Developing automated, scalable resource allocation tools based on architecture-aware damage analysis will be prioritized. Additionally, investigating other quantization algorithms and their damage patterns will refine strategies, aiming for more efficient and robust model deployment.
AI Executive Summary
Large language models (LLMs) have revolutionized natural language processing, but their deployment remains costly due to massive parameter sizes. Quantization offers a practical solution by reducing precision, yet introduces uneven accuracy degradation. Traditional approaches often rely on heuristics or local signals like weight variance or circuit drift to guide repair, but these lack systematic validation.
This paper pioneers the use of causal mixed-precision intervention to empirically analyze where quantization damage occurs within models. By progressively elevating individual layers to 8-bit precision and measuring accuracy recovery, the authors test three hypotheses: damage localization in task circuits, computation sites, or weight statistics. Across nine models spanning four architectures, the results reveal damage is broadly dispersed, with no single layer or small set reliably predictive of repair benefit. Instead, approximately half the layers need to be restored to recover most of the performance gap.
Furthermore, the study compares global fine-grained quantization against local layer repair under matched budgets. Findings show that applying finer granularity globally consistently outperforms targeted fixes, improving accuracy by 21-52 points across models. This challenges the conventional wisdom that critical layers should be prioritized, emphasizing instead the importance of resource-wide strategies. The damage’s architecture-dependent distribution within families suggests that tailored approaches may further enhance efficiency.
Overall, the research shifts the paradigm from local, heuristic-based repair to a holistic, data-driven resource allocation. It underscores that damage is diffuse, and effective quantization requires global strategies, especially when resources are limited. These insights pave the way for more robust, scalable, and cost-effective deployment of large models, with significant implications for future model compression and inference optimization.
Deep Dive
Abstract
Post-training quantization (PTQ) is widely used to reduce the cost of serving large language models (LLMs), but its accuracy cost is uneven and is often tuned per model. We study where quantization damage occurs and how to allocate a small additional precision budget. Using causal mixed-precision intervention as ground truth (raise each layer to 8-bit in turn and measure the accuracy it recovers) across 9 open-weight models in 4 architecture families, we test 3 intuitive hypotheses: that quantization damage lives in task circuits, where the model computes, or in weight statistics. None of them predicts which layers benefit from restored precision. Recovery is instead diffuse: for 8 of 9 models, recovering 75% of the gap takes roughly half the layers; the lone exception, Qwen3-8B, is sharply concentrated. At a matched precision budget, spending it globally on finer quantization granularity beats locally repairing the most recoverable layers for all 8 group-128-compatible models (all but OpenLLaMA, whose width rules out group-128), by 21-52 points, including the concentrated Qwen3-8B. We report 2 secondary findings: the residual is budget-limited (8-bit is near-lossless in our evaluation across RTN, GPTQ, and AWQ), and the location of peak recovery correlates with architecture within a family, though not across families. Within this budget setting, global granularity is a better default than selectively protecting critical layers. More broadly, cheap signals that correlate with quantization damage do not necessarily identify where restoring precision improves accuracy; this must be tested with causal intervention.