D-cut: Adaptive Verification Depth Pruning for Batched Speculative Decoding
D-cut improves high-concurrency inference speed by adaptive pruning, achieving up to 3× acceleration.
Key Findings
Methodology
D-cut employs cross-request pruning and runtime-adaptive depth selection to optimize verification budgets. It ranks draft tokens by confidence and models runtime costs for dynamic pruning.
Key Results
- Under high concurrency, D-cut improves average speedup from 1.26× to 1.65×, restoring acceleration for dense models.
- On MoE models, D-cut achieves up to 3× speedup over autoregressive decoding.
- On Qwen3-8B with Math500, D-cut outperforms DFlash significantly at batch size 64.
Significance
D-cut addresses the high verification cost in batched speculative decoding, enabling efficient deployment of LLMs in high-concurrency scenarios. It has significant implications for real-world applications requiring scalable inference.
Technical Contribution
D-cut introduces a combination of cross-request pruning and runtime-adaptive depth selection, reducing verification costs without modifying the target model while preserving lossless output.
Novelty
D-cut uniquely formulates verification budget allocation as a dynamic optimization problem, incorporating runtime cost modeling for hardware adaptability, excelling in high-concurrency scenarios.
Limitations
- Limited benefits in low-concurrency settings where pruning opportunities are fewer.
- Runtime cost modeling requires recalibration for different hardware environments.
- Reliance on confidence scores may falter if predictions are inaccurate.
Future Work
Future research could explore improved confidence estimation, extend D-cut to diverse architectures, and enhance performance in low-concurrency scenarios.
AI Executive Summary
Autoregressive decoding in large language models (LLMs) is inherently slow due to its step-by-step token generation. Speculative decoding accelerates this by drafting multiple tokens and verifying them in parallel. However, under high concurrency, long drafts lead to excessive verification costs, negating the speedup.
D-cut introduces cross-request pruning and runtime-adaptive depth selection to dynamically allocate verification budgets, prioritizing high-confidence draft tokens. Key innovations include confidence-based global ranking and runtime cost modeling, ensuring the pruning strategy aligns with hardware constraints.
Experiments show that D-cut significantly improves inference speed for both dense and mixture-of-experts (MoE) models. It restores acceleration for dense configurations where speculative decoding struggles and achieves up to 3× speedup over autoregressive decoding in MoE settings. This method offers a promising direction for optimizing large-scale concurrent inference while addressing practical deployment challenges.
Deep Analysis
Background
LLMs excel in NLP tasks but suffer from slow autoregressive decoding. Speculative decoding accelerates inference by drafting and verifying tokens in parallel, but high concurrency introduces inefficiencies due to excessive verification costs.
Core Problem
High-concurrency inference faces a bottleneck where long drafts waste resources on rejected tokens, making speculative decoding slower than autoregressive methods.
Innovation
D-cut introduces cross-request pruning and runtime-adaptive depth selection. By globally ranking draft tokens based on confidence and modeling runtime costs, it dynamically adjusts verification budgets to optimize speed.
Methodology
- �� Confidence-based ranking: Draft tokens are globally ranked by confidence, prioritizing high-confidence positions.
- �� Runtime cost modeling: Hardware performance is profiled at startup to build cost curves.
- �� Dynamic budget allocation: Verification depth is dynamically adjusted based on cost curves to maximize speedup.
Experiments
Evaluations on Qwen3-8B and other models using Math500 and similar datasets compare D-cut against baselines like DFlash. Tests include varying batch sizes and hardware environments.
Results
D-cut achieves significant speedups under high concurrency, with average acceleration increasing from 1.26× to 1.65×. On MoE models, it delivers up to 3× speedup over autoregressive decoding.
Applications
D-cut is ideal for scenarios requiring efficient handling of large-scale concurrent requests, such as chatbots, search engines, and code generation tools.
Limitations & Outlook
D-cut's benefits are limited in low-concurrency settings, and runtime cost modeling requires recalibration for different hardware. Future work could address these challenges.
Plain Language Accessible to non-experts
Imagine a factory where workers assemble products. Traditional methods have workers complete one product at a time. Speculative decoding drafts multiple products simultaneously, but verifying them all wastes resources. D-cut is like a smart manager who inspects only the most promising products first, saving time and effort.
ELI14 Explained like you're 14
Think of picking players for your soccer team. Normally, you'd test one player at a time, but that's slow. Speculative decoding invites many players, then picks the best. D-cut goes further—it checks their stats first and only tests the most promising ones. Cool, right?
Glossary
Speculative Decoding
An inference acceleration method that drafts and verifies multiple tokens in parallel.
Used to speed up LLM inference.
Cross-request Pruning
Dynamically reallocates verification budgets across concurrent requests based on confidence.
Key mechanism in D-cut.
Runtime Cost Modeling
Analyzing hardware performance to optimize algorithm parameters.
Used for dynamic budget allocation in D-cut.
Mixture-of-Experts (MoE)
A neural network architecture combining multiple expert models.
Evaluated for D-cut's performance.
Verification Budget
The computational resources allocated for verifying draft tokens.
Optimized dynamically by D-cut.
Open Questions Unanswered questions from this research
- 1 How to improve confidence prediction accuracy for better pruning.
- 2 Strategies to enhance D-cut's performance in low-concurrency scenarios.
- 3 Adapting runtime cost modeling to dynamic hardware environments.
Applications
Immediate Applications
Real-time Chatbots
Optimize inference speed to handle high user concurrency efficiently.
Code Generation Tools
Accelerate inference for real-time code suggestions in development environments.
Long-term Vision
Scalable LLM Deployment
Enable efficient inference for large-scale cloud services, reducing computational costs.
Abstract
Speculative decoding accelerates large language model (LLM) inference without compromising output quality. Recent parallel drafting methods further improve single-request performance by decoupling draft length from drafting latency, enabling longer drafts and higher mean accepted tokens (MAT). However, under high request concurrency, long drafts waste substantial computation on rejected tokens, increasing verification cost and potentially making speculative decoding slower than autoregressive decoding. We present D-Cut, an adaptive pruning method that selects draft tokens jointly across the batch and concentrates the verification budget on tokens most likely to be accepted. D-Cut is motivated by two observations. First, acceptance lengths vary considerably across concurrent requests; D-Cut therefore performs cross-request pruning, allocating the verification budget adaptively according to draft confidence. Second, verification cost depends strongly on the deployment environment, including GPU architecture and parallelism strategy; D-Cut incorporates a runtime cost model to adapt its pruning depth to the target environment. Experiments on dense and mixture-of-experts (MoE) models show that, under high concurrency, D-Cut improves the average speedup from \(1.26\times\) to \(1.65\times\), restores acceleration in dense-model configurations where long-draft baselines are slower than autoregressive decoding, and achieves up to \(3.0\times\) speedup over autoregressive decoding on MoE models.