LAQuant: A Simple Overhead-free Large Reasoning Model Quantization by Layer-wise Lookahead Loss
LAQuant uses layer-wise lookahead loss for overhead-free quantization, improving AIME25 Pass@1 by 15.11 percentage points.
Key Findings
Methodology
LAQuant is a layer-wise weight-only quantization-aware training method that addresses KV-cache fidelity and Hessian-subspace alignment issues. It combines reasoning-domain calibration with a one-layer lookahead loss to maintain the next-layer residual stream through implicit cross-layer co-adaptation.
Key Results
- LAQuant improved AIME25 Pass@1 by 15.11 percentage points over ParoQuant under W3G128 quantization for Qwen3-4B, achieving a 3.42x decoding speedup on RTX A6000.
- Compared to ParoQuant++, LAQuant improved AIME25 Pass@1 by 1.93 percentage points under matched calibration.
- LAQuant demonstrated superior performance across multiple reasoning tasks, especially in long-decoding scenarios.
Significance
LAQuant provides an overhead-free solution for quantizing large reasoning models, significantly enhancing efficiency and accuracy in reasoning tasks. It addresses the accuracy degradation of traditional quantization methods in long-decoding tasks, offering new optimization insights for academia and industry.
Technical Contribution
Technically, LAQuant offers a new quantization-aware training framework by combining layer-wise lookahead loss with Hessian-aligned calibration data, avoiding online transformation overhead while maintaining KV-cache fidelity.
Novelty
LAQuant is the first to introduce layer-wise lookahead loss in large reasoning model quantization, differing from previous methods by achieving cross-layer co-adaptation without online transformations.
Limitations
- LAQuant may still experience accuracy degradation in extremely long-sequence tasks, especially with insufficient calibration data.
- The method is sensitive to calibration data choice, requiring alignment with the reasoning distribution.
Future Work
Future research directions include optimizing calibration data selection strategies and testing applications on larger-scale models and more task scenarios.
AI Executive Summary
Large reasoning models excel in math and coding tasks, but their long-decoding process incurs significant computational costs, hindering deployment. Traditional quantization methods suffer accuracy loss in long-decoding tasks, which LAQuant addresses through layer-wise lookahead loss. By combining reasoning-domain calibration with a one-layer lookahead loss, LAQuant significantly enhances efficiency and accuracy in reasoning tasks, especially in long-decoding scenarios. Experimental results show LAQuant's superior performance across multiple reasoning tasks, particularly improving AIME25 by 15.11 percentage points and achieving a 3.42x decoding speedup. However, LAQuant may still face accuracy challenges in extremely long-sequence tasks, with future research focusing on optimizing calibration data selection strategies.
Deep Analysis
Background
In recent years, large reasoning models have excelled in STEM and coding tasks, but their long-decoding process incurs significant computational costs, hindering deployment. Traditional quantization methods suffer accuracy loss in long-decoding tasks, despite performing well in short-decoding tasks.
Core Problem
The core challenges in quantizing large reasoning models for long-decoding tasks are maintaining KV-cache fidelity and aligning calibration data with the deployment distribution's Hessian subspace. These issues lead to accuracy degradation in traditional quantization methods.
Innovation
LAQuant addresses KV-cache fidelity and Hessian subspace alignment issues by combining layer-wise lookahead loss with Hessian-aligned calibration data. Its innovation lies in achieving cross-layer co-adaptation without online transformations.
Methodology
- �� Use reasoning-domain calibration data to maintain optimization target alignment
- �� Employ a one-layer lookahead loss for implicit cross-layer co-adaptation
- �� Avoid online transformation overhead by implementing standard weight quantization
Experiments
Experiments were conducted on models like Qwen3-4B using W3G128 quantization, with evaluation metrics including AIME25 Pass@1 and decoding speed. Results show LAQuant's superior performance across multiple reasoning tasks.
Results
LAQuant improved AIME25 Pass@1 by 15.11 percentage points and achieved a 3.42x decoding speedup on RTX A6000. Compared to ParoQuant++, LAQuant improved AIME25 Pass@1 by 1.93 percentage points under matched calibration.
Applications
LAQuant is applicable to large models requiring efficient reasoning, particularly in long-decoding tasks. Its overhead-free quantization method can be widely applied in various industrial reasoning tasks.
Limitations & Outlook
LAQuant may still experience accuracy degradation in extremely long-sequence tasks, especially with insufficient calibration data. Future research will focus on optimizing calibration data selection strategies.
Plain Language Accessible to non-experts
Imagine a large factory where each worker needs to complete tasks efficiently. Traditional methods are like giving workers large hammers for delicate tasks, leading to inefficiency. LAQuant equips each worker with the right tools, allowing them to work more efficiently without additional steps. By considering the tools needed for the next step in advance, workers can collaborate better, avoiding wasted time and resources.
ELI14 Explained like you're 14
Hey there! Imagine you're playing a super complex game where each move takes forever to complete. Traditional methods are like using an old, slow game console, making you frustrated. LAQuant is like upgrading to a super-fast console that not only speeds up the game but also helps you complete tasks more accurately! Isn't that cool?
Glossary
Quantization
The process of converting model weights from high precision to low precision to improve computational efficiency.
Used to accelerate the decoding process of large reasoning models.
KV Cache
A cache storing the key and value projections in a model, used to accelerate reasoning.
KV-cache fidelity is crucial for model performance in long-decoding tasks.
Hessian Subspace
Represents the alignment between calibration data and deployment distribution.
Used to analyze the impact of calibration data choice on model performance.
Lookahead Loss
A loss function that optimizes the current layer by considering the next layer's output.
Used to achieve cross-layer co-adaptation and maintain model performance.
Calibration Data
Specific datasets used to adjust model parameters for improved reasoning accuracy.
The choice of calibration data significantly impacts model performance.
Open Questions Unanswered questions from this research
- 1 How can model accuracy be further improved in extremely long-sequence tasks? Current methods perform poorly with insufficient calibration data.
- 2 How to select optimal calibration data to ensure alignment with the deployment distribution?
- 3 How does LAQuant perform on larger-scale models?
Applications
Immediate Applications
Efficient Reasoning
LAQuant can be used for large models requiring fast reasoning, suitable for real-time industrial applications.
Long-term Vision
Large-Scale Model Optimization
By further optimizing quantization methods, LAQuant has the potential to achieve more efficient reasoning on larger-scale models.
Abstract
Large reasoning models (LRMs) reach competition-level math and coding accuracy via long autoregressive decoding, making per-token decoding cost a primary deployment concern. Weight quantization is the standard tool for acceleration, but representative recipes -- including state-of-the-art end-to-end (E2E) QAT -- lose accuracy on long-decoding reasoning benchmarks despite preserving perplexity and short-decode accuracy. Through a systematic gradient-direction analysis, we identify two factors driving this gap: (i) KV-cache fidelity preservation under the QAT loss, which E2E supervision attenuates via the softmax Fisher metric; and (ii) Hessian-subspace alignment between calibration data and the deployment distribution. We propose LookAhead Quantization (LAQuant), a layer-wise weight-only QAT method that addresses both factors without online-transform overhead by combining reasoning-domain calibration with a one-layer lookahead loss whose implicit cross-layer co-adaptation preserves the next-layer residual stream. For Qwen3-4B under W3G128 quantization, LAQuant improves AIME25 Pass@1 over ParoQuant by 15.11pp (1.93pp over ParoQuant++ at matched calibration) while achieving a 3.42x decoding speedup over FP16 on RTX A6000, compared with ParoQuant's 3.01x.