Quantize the Target, Quantize the Drafter: Efficient Inference with Qwen3.5-4B
Combining quantized target model and speculative decoding, Qwen3.5-4B achieves 6.978x speedup on NVIDIA A10G.
Key Findings
Methodology
The study employs quantization-aware distillation (QAD) and speculative decoding to optimize the inference speed of Qwen3.5-4B on NVIDIA A10G GPU. The target model's accuracy is recovered using QAD while maintaining the original quantization grid. Speculative decoding is accelerated using a specially trained block-diffusion drafter, combined with sliding-window attention to reduce long-context decoding latency.
Key Results
- For short inputs (64/128), inference time reduced from 2582ms to 232ms, achieving a 6.978x speedup.
- For medium inputs (2048/256), time reduced from 5441ms to 782ms, maintaining required quality thresholds.
- For long inputs (8192/256), time reduced from 6576ms to 2313ms, ranking third.
Significance
This research achieves efficient inference of large language models on resource-constrained GPUs, significantly reducing latency while maintaining model accuracy. The results provide new insights for practical deployment of large-scale models, especially in resource-limited scenarios.
Technical Contribution
Technical contributions include recovering quantized model accuracy through QAD, accelerating speculative decoding with a block-diffusion drafter, and introducing sliding-window attention to reduce long-context decoding latency. These methods significantly improve inference speed without sacrificing quality.
Novelty
This is the first to combine QAD and speculative decoding for large model inference optimization, especially in resource-constrained GPU environments. It significantly improves inference speed compared to existing methods while maintaining model accuracy.
Limitations
- In extreme long-context inputs, sliding-window attention may lead to loss of context information, affecting model performance.
- The quantization process may cause slight accuracy loss, requiring further optimization.
Future Work
Future research could explore more efficient quantization strategies and attention mechanisms to further enhance inference speed and accuracy. Additionally, extending the study to other hardware platforms and larger models is promising.
AI Executive Summary
In large language model inference, latency and computational resources are major challenges. Existing solutions often struggle to balance accuracy and speed, especially on resource-constrained hardware.
This study proposes a novel approach combining quantization-aware distillation and speculative decoding, achieving efficient inference of Qwen3.5-4B on NVIDIA A10G GPU. The method not only recovers the accuracy of the quantized model but also reduces long-context decoding latency through sliding-window attention.
Experimental results demonstrate a 6.978x average speedup without sacrificing model quality. This achievement offers new insights for deploying large-scale language models in practical scenarios, particularly where computational resources are limited. However, future exploration of more efficient quantization strategies and attention mechanisms is needed to further optimize performance.
Deep Analysis
Background
Large language models have made significant advances in natural language processing, but their high computational cost and latency limit practical applications. Recently, quantization and speculative decoding have become popular research directions to address this issue. Quantization reduces computation by lowering model bit-width, while speculative decoding accelerates inference by quickly generating drafts.
Core Problem
Achieving efficient inference of large language models on resource-constrained hardware is a critical and challenging problem. Existing methods struggle to balance accuracy and speed, particularly with long-context inputs where latency is a major issue.
Innovation
The core innovation of this study is combining quantization-aware distillation and speculative decoding to enhance inference speed. Quantization-aware distillation recovers accuracy by guiding a quantized student model with a high-precision teacher. Speculative decoding accelerates generation with a block-diffusion drafter, and sliding-window attention further reduces long-context decoding latency.
Methodology
- �� Use quantization-aware distillation (QAD) to recover quantized model accuracy.
- �� Train a block-diffusion drafter to accelerate speculative decoding.
- �� Introduce sliding-window attention to reduce long-context decoding latency.
- �� Conduct experiments on NVIDIA A10G GPU for validation.
Experiments
Experiments use the Nemotron-Post-Training-Dataset-v2 dataset to evaluate inference speed and model accuracy across different input lengths. MMLU-Pro, IFEval, and GPQA-Diamond serve as quality benchmarks to ensure model accuracy while accelerating.
Results
Results show that the model achieves 6.978x, 6.963x, and 2.844x speedup in short, medium, and long input scenarios, respectively. The model meets or exceeds required quality thresholds across all benchmarks.
Applications
This method can be directly applied to large-scale language model inference scenarios requiring rapid response, such as real-time translation and intelligent customer service. Its efficiency on resource-constrained hardware makes it highly applicable to edge computing devices.
Limitations & Outlook
While the method significantly improves inference speed, sliding-window attention may lead to context information loss in extreme long-context inputs. Additionally, the quantization process may cause slight accuracy loss, requiring further optimization.
Plain Language Accessible to non-experts
Imagine a large library where a librarian needs to quickly find specific books. Traditional methods involve checking each book one by one, but our approach is like equipping the librarian with a fast indexing system. Quantization-aware distillation is like providing a simplified catalog that, while less detailed, is accurate enough. Speculative decoding is like the librarian predicting which books might be needed in advance, reducing back-and-forth searching time. Sliding-window attention helps the librarian focus on recent books rather than the entire library, allowing faster retrieval of relevant books. This method greatly improves search efficiency without sacrificing accuracy.
ELI14 Explained like you're 14
Imagine you're playing a game where you need to quickly find hidden treasures. The traditional way is to search step by step, but that's too slow! Our research is like giving you a map, not very detailed but enough to find the treasure. Quantization-aware distillation is like this map, guiding you in the right direction. Speculative decoding is like guessing where the treasure might be, so you don't have to search everywhere. Sliding-window attention is like focusing on recent clues rather than the whole map, so you can find the treasure faster. This method lets you find treasures quickly without wasting time!
Glossary
Quantization-Aware Distillation (QAD)
A method that uses a high-precision teacher model to guide a quantized student model to recover accuracy.
Used to recover accuracy of the quantized model while maintaining the original quantization grid.
Speculative Decoding
A technique that accelerates inference by quickly generating drafts.
Used to accelerate the inference process of the Qwen3.5-4B model.
Block-Diffusion Drafter
A specially trained model for speculative decoding that can predict multiple future tokens.
Used to accelerate the speculative decoding process.
Sliding-Window Attention
A method that limits the attention computation range, reducing long-context decoding latency.
Used to reduce long-context decoding latency.
NVIDIA A10G GPU
A graphics processing unit used for efficient inference, featuring 24 GB VRAM.
Used to evaluate the inference speed of the Qwen3.5-4B model.
Open Questions Unanswered questions from this research
- 1 How to further improve the inference speed of quantized models without affecting accuracy?
- 2 How to reduce the information loss caused by sliding-window attention in extreme long-context inputs?
Applications
Immediate Applications
Real-Time Translation
By improving inference speed, achieve faster real-time translation applications suitable for multi-language communication scenarios.
Long-term Vision
Edge Computing
Deploy efficient large language models on edge devices, enabling real-time response and processing on smart devices.
Abstract
This report describes our approach to the Efficient Qwen Competition, where the goal is to enable low-latency serving of Qwen3.5-4B on a resource-constrained NVIDIA A10G GPU. Our system combines a quantized target model with speculative decoding. To recover accuracy, we apply quantization-aware distillation to the target model while retaining the original quantization grid. To speed up decoding, a block-diffusion drafter specialized for the quantized target model is trained using a two-stage procedure: first learning from the high-precision target and then adapting to the low-precision target. Because the drafter is invoked at every speculative decoding step, we further reduce its overhead with quantization and sliding-window attention, preserving draft-token acceptance while improving long-context decoding latency. As a result, our submission achieves a 6.978$\times$ average speedup over the baseline while satisfying the required quality thresholds, ranking 3rd overall. We hope these results provide useful insights for practical LLM inference. The code and resources are available at https://github.com/nota-github/adaptfm-quant-dflash