DFlare: Scaling Up Draft Capacity for Block Diffusion Speculative Decoding
DFlare enhances draft model capacity with layer-wise fusion, achieving over 5x LLM inference speedup.
Key Findings
Methodology
DFlare introduces a lightweight layer-wise fusion mechanism, allowing each draft layer to access multiple target model hidden layers, enhancing per-layer expressiveness. This mechanism uses learnable weight combinations, adding negligible computational overhead. By increasing training data, DFlare fully exploits the enhanced model capacity.
Key Results
- On Qwen3-4B, DFlare achieves an average speedup of 5.52x, improving over DFlash by 11%.
- On Qwen3-8B, DFlare achieves a speedup of 5.46x, an 8% improvement.
- On GPT-OSS-20B, DFlare achieves a speedup of 3.91x, a 5% improvement.
Significance
DFlare significantly improves LLM inference efficiency, especially in tasks like mathematical reasoning, code generation, and conversation. By optimizing draft model structure and training data, DFlare offers a more efficient solution for large-scale model applications, addressing existing method bottlenecks.
Technical Contribution
DFlare overcomes DFlash's limitations by enhancing per-layer expressiveness through layer-wise fusion and heterogeneous KV projections. This design allows the draft model to absorb more target model knowledge without increasing computational complexity.
Novelty
DFlare is the first to introduce layer-wise fusion, allowing each draft layer to independently access multiple target model layers, overcoming the limitations of shared information in previous methods and significantly enhancing model scalability.
Limitations
- DFlare's performance on extremely large models needs further validation, especially in more complex tasks.
- The method heavily relies on training data, and its effectiveness may be limited when data is insufficient.
Future Work
Future work could explore DFlare's application in more tasks, especially multimodal tasks. Further optimization of training data selection and model structure design will also be important research directions.
AI Executive Summary
DFlare significantly enhances LLM inference efficiency through an innovative layer-wise fusion mechanism. Traditional methods like DFlash face bottlenecks in draft model layer information sharing, limiting model scalability. DFlare overcomes this by allowing each draft layer to independently access multiple target model hidden layers.
In experiments, DFlare achieved speedups of 5.52x, 5.46x, and 3.91x on Qwen3-4B, Qwen3-8B, and GPT-OSS-20B, respectively, showing significant improvements over DFlash. These results indicate that DFlare not only excels in inference speed but also performs well across different tasks.
DFlare opens new possibilities for practical applications of large-scale language models, especially in scenarios requiring efficient inference. However, its performance on extremely large models still needs further validation, and future research could explore its application in multimodal tasks.
Deep Analysis
Background
In recent years, large language models have excelled in NLP tasks, but their inference speed has become a bottleneck. Existing methods like DFlash accelerate inference with draft models but face limitations in layer information sharing, affecting model scalability.
Core Problem
The slow inference speed of large language models is a significant issue, especially in real-time applications. Existing methods face bottlenecks in draft model layer information sharing, limiting model scalability.
Innovation
DFlare introduces a layer-wise fusion mechanism, allowing each draft layer to independently access multiple target model hidden layers, enhancing model expressiveness and inference speed. Unlike traditional methods, DFlare absorbs more target model knowledge without increasing computational complexity.
Methodology
- �� Introduce layer-wise fusion, allowing each draft layer to access multiple target model hidden layers through learnable weight combinations.
- �� Use heterogeneous KV projections to separate the information spaces of draft and target models.
- �� Increase training data to fully exploit the enhanced model capacity.
Experiments
Experiments were conducted on Qwen3-4B, Qwen3-8B, and GPT-OSS-20B, evaluating mathematical reasoning, code generation, and conversation tasks. Results show that DFlare outperforms DFlash in all tasks, especially in inference speed.
Results
DFlare achieved speedups of 5.52x on Qwen3-4B, 5.46x on Qwen3-8B, and 3.91x on GPT-OSS-20B. Compared to DFlash, DFlare shows significant improvements across all tasks.
Applications
DFlare is suitable for large-scale language model applications requiring efficient inference, such as real-time dialogue systems and code generation tools. Its efficient inference capability can significantly enhance the response speed of these applications.
Limitations & Outlook
DFlare's performance on extremely large models needs further validation, especially in more complex tasks. Additionally, the method heavily relies on training data, and its effectiveness may be limited when data is insufficient.
Plain Language Accessible to non-experts
Imagine a factory where traditional methods are like assembly lines, with each worker doing the same job, leading to inefficiency. DFlare is like a smart factory where each worker has a specialty and can adjust their tasks as needed, improving overall efficiency. This allows DFlare to complete tasks faster.
ELI14 Explained like you're 14
Imagine playing a game where your character needs to react quickly. Traditional methods are like your character acting in sequence, which is slow. DFlare is like giving your character superpowers, allowing them to handle multiple tasks at once, making the game faster and more exciting!
Glossary
Large Language Model
A large-scale neural network capable of processing and generating natural language, typically containing billions of parameters.
Used as the target model for accelerated inference in this paper.
Draft Model
A lightweight model used to quickly predict multiple future tokens, which are then verified by the target model.
A key component for accelerating inference in DFlare.
Layer-wise Fusion
A mechanism allowing each draft layer to independently access multiple target model hidden layers.
One of DFlare's core innovations.
Inference Acceleration
The process of improving model processing speed by optimizing model structure and algorithms.
The main goal of DFlare.
Heterogeneous KV Projections
A projection mechanism that separates the information spaces of draft and target models.
Used to enhance the expressiveness of the draft model.
Open Questions Unanswered questions from this research
- 1 DFlare's performance on extremely large models needs validation, especially in more complex tasks.
- 2 How to maintain DFlare's efficiency with insufficient data remains a challenge.
Applications
Immediate Applications
Real-time Dialogue Systems
DFlare can significantly enhance the response speed of dialogue systems, suitable for scenarios requiring rapid interaction.
Long-term Vision
Multimodal Tasks
Future exploration of DFlare's application in multimodal tasks could further enhance model versatility and efficiency.
Abstract
Block diffusion speculative decoding accelerates LLM inference by predicting all tokens within a block simultaneously for the target model to verify in parallel. Predicting an entire block at once requires a sufficiently capable draft model and effective utilization of the target model's internal knowledge. However, the state-of-the-art method DFlash constrains all draft layers to share a single fused representation derived from only a few target layers, limiting per-layer expressiveness and hindering further scaling of draft capacity. In this paper, we present \modelname, which flares out the narrow conditioning bottleneck of DFlash through a lightweight layer-wise fusion mechanism: each draft layer attends to its own learnable combination of a broad set of target layers at negligible overhead, simultaneously injecting richer target knowledge and providing every draft layer with a distinct input. This enhanced per-layer expressiveness enables scaling the draft model to deeper architectures with consistent gains. We further scale training data from 800K to 2.4M samples to fully exploit the enlarged capacity. On six benchmarks spanning mathematical reasoning, code generation, and conversation, \modelname attains average wall-clock speedups of 5.52x on Qwen3-4B, 5.46x on Qwen3-8B, and 3.91x on GPT-OSS-20B, improving over DFlash by roughly 11\%, 8\%, and 5\% respectively. Our code is available at https://github.com/Tencent/AngelSlim.