DeLS-Spec: Decoupled Long-Short Contexts for Parallel Speculative Drafting
DeLS-Spec improves inference speed and acceptance length by combining long and short contexts.
Key Findings
Methodology
DeLS-Spec combines a fixed DFlash model as a long-context expert with a lightweight local head as a short-context expert. The local head can be trained independently using a standard next-token prediction objective, without joint training with the target model or DFlash backbone.
Key Results
- Experiments on Qwen3 models show DeLS-Spec improves speedup and average acceptance length over DFlash across math, code, and dialogue benchmarks.
- On math tasks, DeLS-Spec improves DFlash speed from 4.74× to 5.02×, with acceptance length increasing by 0.44.
- On code generation tasks, DeLS-Spec improves HumanEval speed from 4.61× to 4.85×, with acceptance length increasing from 5.84 to 6.21.
Significance
DeLS-Spec offers a practical method to enhance existing speculative decoding systems by decoupling long and short contexts. It reduces training costs and increases modularity and flexibility, with broad applications in academia and industry.
Technical Contribution
DeLS-Spec improves DFlash-style block-parallel drafting without retraining the DFlash backbone. It introduces a lightweight local head providing intra-block causal information, which can be trained independently.
Novelty
DeLS-Spec is the first to decouple long-short contexts for speculative decoding, avoiding the high costs of training models from scratch as in existing methods.
Limitations
- DeLS-Spec may not capture all causal dependencies in complex contexts, particularly when the residual term is ignored.
- Ignoring the residual term may affect accuracy in certain scenarios.
Future Work
Future work could explore better modeling of interactions between long and short contexts and applying DeLS-Spec to larger-scale models.
AI Executive Summary
DeLS-Spec is a novel speculative decoding method designed to address the lack of causal conditioning in existing methods. By decoupling long and short contexts, DeLS-Spec offers a modular and flexible solution that significantly reduces training costs.
In experiments, DeLS-Spec performs exceptionally well on Qwen3 models, particularly in math and code generation tasks. It not only improves inference speed but also increases acceptance length, demonstrating its superiority across different benchmarks.
While DeLS-Spec excels in many areas, there is still room for improvement. Future research could further optimize the interaction between long and short contexts and explore larger-scale application scenarios.
Deep Analysis
Background
In recent years, speculative decoding has become an important method for accelerating large language model inference. Traditional autoregressive decoding requires generating tokens one by one, leading to latency. Block-parallel drafters like DFlash improve efficiency by predicting an entire block at once but lack explicit intra-block causal conditioning.
Core Problem
Existing block-parallel drafting methods lack causal conditioning within blocks, limiting acceptance length. Methods like Domino and DSpark attempt to address this but require training models from scratch, leading to flexibility and cost issues.
Innovation
DeLS-Spec addresses the lack of causal conditioning by fixing DFlash as a long-context expert and introducing a lightweight local head as a short-context expert. This method avoids retraining the DFlash backbone, reducing training costs.
Methodology
- �� Fix DFlash as a long-context expert
- �� Introduce a lightweight local head as a short-context expert
- �� Train the local head independently using a standard next-token prediction objective
- �� Combine long and short context logits during inference
Experiments
Experiments were conducted on Qwen3-4B and Qwen3-8B models, using math, code, and dialogue benchmarks for evaluation. Results show DeLS-Spec outperforms DFlash across all benchmarks.
Results
DeLS-Spec improves DFlash speed from 4.74× to 5.02× on math tasks, with acceptance length increasing by 0.44. On code generation tasks, HumanEval speed increases from 4.61× to 4.85×, with acceptance length increasing from 5.84 to 6.21.
Applications
DeLS-Spec can be used in large-scale language models requiring fast inference, such as real-time dialogue systems and code generation tools. Its modular design makes it easy to integrate into existing systems.
Limitations & Outlook
DeLS-Spec may not capture all causal dependencies in complex contexts, and ignoring the residual term may affect accuracy in certain scenarios. Future research could further optimize the interaction between long and short contexts.
Plain Language Accessible to non-experts
Imagine you are cooking in a kitchen. DFlash is like an experienced chef who knows how to make delicious dishes based on recipes but sometimes overlooks details like subtle changes in seasoning. DeLS-Spec is like an assistant focusing on these details, ensuring every dish is perfect. By combining the strengths of both, DeLS-Spec can complete tasks faster while ensuring quality.
ELI14 Explained like you're 14
Imagine you're playing a fast-paced game. DFlash is like a pro player who knows how to quickly pass levels but sometimes misses hidden treasures. DeLS-Spec is like an assistant helping him find these treasures, making him perform better in the game. Together, they not only clear levels faster but also earn more rewards!
Glossary
Speculative Decoding
A method to accelerate language model inference by parallelly verifying multiple future tokens to reduce latency.
Used to improve inference speed of large language models.
DFlash
A block-parallel drafter that improves efficiency by predicting an entire block at once.
Used as a long-context expert in DeLS-Spec.
Long-Short Contexts
Combining long and short contexts in speculative decoding to improve accuracy and efficiency.
Core innovation of DeLS-Spec.
Local Head
A lightweight model component focusing on intra-block causal conditioning.
Used in DeLS-Spec to capture short-context information.
Logits
Unnormalized probabilities output by a model used to predict the next token.
Combined in DeLS-Spec from long and short contexts.
Open Questions Unanswered questions from this research
- 1 How can interactions between long and short contexts be better modeled in more complex contexts?
- 2 What challenges exist in applying DeLS-Spec to larger-scale models?
Applications
Immediate Applications
Real-time Dialogue Systems
By improving inference speed and accuracy, DeLS-Spec can significantly enhance user experience in real-time dialogue systems.
Long-term Vision
Large-scale Language Models
DeLS-Spec's modular design makes it applicable to future larger-scale language models, advancing natural language processing.
Abstract
Speculative decoding accelerates LLM inference by drafting multiple tokens and verifying them in parallel. Block-parallel drafters such as DFlash further improve drafting efficiency by predicting an entire block in one pass, but their position-wise predictions lack explicit intra-block causal conditioning. Recent methods such as Domino and DSpark attempt to introduce such causality into block-parallel drafting, but they require training the draft model from scratch, which limits their flexibility and increases training cost. We propose DeLS-Spec, a decoupled long-short context speculative decoding method. DeLS-Spec treats the fixed DFlash model as a long-context expert and introduces a lightweight local head as a short-context expert. The local head can be trained independently with a standard next-token prediction objective, without joint training with the target model or the DFlash backbone, leading to extremely low training cost. At inference time, DeLS-Spec combines long-context and short-context logits, and the local head is not tied to a specific DFlash checkpoint, making the method more modular and flexible. Experiments on Qwen3 models show that DeLS-Spec consistently improves speedup and average acceptance length over DFlash across math, code, and dialogue benchmarks.