Margins, Not Windows: Training-Free Per-Step Lossy Speculative Decoding
AdaptiveSpec boosts inference speed by 56% and recovers 93% accuracy by dynamically adjusting draft tree shapes and lossy verification.
Key Findings
Methodology
AdaptiveSpec is a training-free per-step adaptive speculative decoding method that dynamically adjusts draft tree depth, width, and node count using internal signals, and employs a margin-based lossy verification rule to enhance inference efficiency.
Key Results
- On GSM8K, MATH-500, and HumanEval, AdaptiveSpec achieved up to 56% throughput improvement on the DeepSeek-R1-8B model while maintaining 93% task accuracy.
- On the Llama-3.1-8B model, AdaptiveSpec's dynamic adjustment achieved a 49% single-cell gain.
- On the Qwen3-8B model, AdaptiveSpec's combination of dynamic adjustment and lossy verification averaged a 38% speedup.
Significance
AdaptiveSpec significantly enhances inference efficiency for large language models by dynamically adjusting draft tree shapes and employing a lossy verification rule, addressing the efficiency bottleneck of traditional autoregressive decoding, and providing a more efficient inference solution for academia and industry.
Technical Contribution
AdaptiveSpec achieves dynamic draft tree adjustment and lossy verification without external training, overcoming the limitations of existing methods and offering new engineering possibilities.
Novelty
AdaptiveSpec is the first to achieve dynamic draft tree and verification rule adjustment without training, significantly improving inference efficiency.
Limitations
- In low acceptance rate scenarios, lossy verification may lead to error accumulation.
- Adaptability to different models needs further validation.
Future Work
Future work will explore the applicability of AdaptiveSpec to more models and tasks and optimize its performance on different hardware.
AI Executive Summary
Current large language models are limited by autoregressive design, especially in long-chain reasoning tasks. AdaptiveSpec significantly enhances inference efficiency by dynamically adjusting draft tree shapes and employing a lossy verification rule. Its core technologies include draft signal-based tree shape adjustment and probability margin-based verification rules, allowing throughput improvement without compromising accuracy. Experimental results show significant speed improvements across multiple datasets while maintaining high accuracy. This method provides new insights for efficient inference in large language models, though there is a risk of error accumulation in some cases. Future research will further optimize its performance on different models and hardware.
Deep Analysis
Background
Large language models are increasingly used in natural language processing, but their autoregressive design leads to low inference efficiency. Existing speculative decoding methods like EAGLE-3 have made improvements but still suffer from fixed draft tree shapes and strict verification rules.
Core Problem
The core problem of autoregressive decoding is that each token generation depends on the previous token, leading to significant inference speed decline as model size increases, especially in long-chain reasoning tasks.
Innovation
AdaptiveSpec's core innovations include dynamic draft tree shape adjustment and lossy verification rules without training. By dynamically adjusting draft tree depth, width, and node count using internal signals and employing a probability margin-based verification rule, it enhances inference efficiency.
Methodology
- �� Dynamic adjustment of draft tree shapes using draft signals.
- �� Margin-based lossy verification rule.
- �� Implemented on the SGLang engine to ensure production-grade performance.
Experiments
Experiments were conducted on GSM8K, MATH-500, and HumanEval datasets, comparing AdaptiveSpec with EAGLE-3, TALON, and FLy methods, tested on an A100 GPU.
Results
AdaptiveSpec achieved significant speed improvements across multiple datasets, particularly on the DeepSeek-R1-8B model, with throughput improvements up to 56% while maintaining high task accuracy.
Applications
AdaptiveSpec can be used in large-scale natural language processing tasks requiring efficient inference, such as real-time translation and dialogue systems, significantly improving system response speed.
Limitations & Outlook
In low acceptance rate scenarios, lossy verification may lead to error accumulation. Additionally, its adaptability to different models and tasks needs further validation.
Plain Language Accessible to non-experts
Imagine a factory where workers need to complete their tasks based on the results of the previous step, leading to low efficiency. AdaptiveSpec is like an intelligent system for the factory, dynamically adjusting the number of workers and their working methods based on the complexity of each step, thus improving overall efficiency.
ELI14 Explained like you're 14
Imagine you're playing a game that requires unlocking levels one by one, and you have to wait for each level to finish before moving on. AdaptiveSpec is like a super helper that lets you unlock multiple levels at once and adjusts strategies based on your performance, helping you finish faster!
Glossary
Speculative Decoding
A method to accelerate inference by drafting and verifying tokens.
Used to improve inference efficiency in large language models.
Tree-attention Drafter
A model that generates multiple candidate tokens.
Used in the draft phase to generate candidate tokens.
AdaptiveSpec
A method for dynamically adjusting draft tree shapes and verification rules.
The core method proposed in this paper.
Margin-based Rule
A lossy verification rule based on probability margins.
Used in the verification phase for token acceptance.
SGLang
A production-grade inference engine.
Used to implement AdaptiveSpec.
Open Questions Unanswered questions from this research
- 1 How to optimize AdaptiveSpec's performance on different hardware?
- 2 What is AdaptiveSpec's applicability to other models?
Applications
Immediate Applications
Real-time Translation
By improving inference speed, AdaptiveSpec can be used in real-time translation systems to enhance response speed.
Long-term Vision
Intelligent Dialogue Systems
Applying AdaptiveSpec in dialogue systems can enhance user experience and reduce latency.
Abstract
Speculative decoding accelerates LLM inference by drafting candidate tokens and verifying them in parallel. Tree-attention drafters such as EAGLE-3 are widely adopted, yet typically hold two decisions fixed: (1) a strict token-match verification rule and (2) a static draft-tree shape. Prior work relaxes each in isolation under limiting assumptions: long draft chains for training-free lossy verification, and adaptive tree shaping under a fixed token budget. We introduce AdaptiveSpec, a training-free per-step speculative decoding method that adapts both decisions from internal signals already produced during decoding. A per-step margin rule promotes a mismatched draft-proposed token when the ratio of the target's probability on the drafted token to its top-1 probability exceeds a threshold with no dependence on draft length or underlying drafter architecture. A per-step tree policy adjusts the draft tree's depth, width, and node count directly from a fused signal of draft top-1 confidence and a rolling acceptance history capturing recent draft-target agreement, allowing the total draft count to vary rather than only be redistributed. The two adaptations operate on orthogonal axes and compound in effect. Implemented on the SGLang production-grade serving engine, AdaptiveSpec improves throughput over the state-of-the-art autoregressive speculative decoding method EAGLE-3 by up to 56%, recovering 93% to fully lossless task accuracy across GSM8K, MATH-500, and HumanEval on three target models (DeepSeek-R1-Distill-Llama-8B, Llama-3.1-8B-Instruct, Qwen3-8B).