Less Experts, Faster Decoding: Cost-Aware Speculative Decoding for Mixture-of-Experts
EcoSpec optimizes MoE model inference by considering expert activation costs, achieving a 1.62x decoding speedup.
Key Findings
Methodology
EcoSpec is a cost-aware speculative decoding framework that uses a lightweight expert predictor and dynamic expert buffer to select draft paths with high acceptance likelihood while reusing existing experts, without altering the target model's verification rules.
Key Results
- On the DeepSeek-V3.1 model, EcoSpec achieved a 1.62x decoding speedup while reducing the number of active experts.
- For the Qwen3-235B-A22B model, EcoSpec reduced expert activation from 23.7 to 20.5, improving overall decoding efficiency.
- In the GPT-OSS-120B model, EcoSpec reduced memory overhead from expert activation, significantly enhancing inference speed.
Significance
EcoSpec significantly enhances decoding efficiency in large-scale MoE models by optimizing expert activation costs, addressing the expert scattering issue in traditional speculative decoding, and providing a more efficient inference solution for large-scale language models.
Technical Contribution
EcoSpec introduces the consideration of expert activation costs in speculative decoding, offering a method to optimize expert reuse without altering verification rules, significantly reducing memory traffic and inference latency.
Novelty
EcoSpec is the first to incorporate expert activation costs into speculative decoding selection strategies, differing from previous strategies that focused solely on acceptance probability, providing a more cost-effective decoding solution.
Limitations
- EcoSpec relies on the accuracy of expert predictions, which may not perform well on certain tasks.
- Managing the expert buffer may become a bottleneck in extremely large-scale models.
Future Work
Future work could explore more complex expert prediction models to improve prediction accuracy and investigate applicability across different model architectures.
AI Executive Summary
Sparse Mixture-of-Experts (MoE) models excel in scaling large language models but their inference efficiency is affected by expert activation patterns. Existing speculative decoding strategies primarily optimize acceptance probability, which can lead to expert scattering and increased memory traffic in large-scale MoE models. EcoSpec addresses this by incorporating expert activation costs into draft selection strategies. Experiments show that EcoSpec achieves significant decoding speed improvements across multiple large-scale MoE models, up to 1.62x. This research provides new insights for efficient inference in large-scale language models, though improvements in expert prediction accuracy and buffer management are still needed.
Deep Analysis
Background
Large language models have demonstrated remarkable capabilities in logic, coding, and creative tasks, but their inference latency and serving cost have become bottlenecks. Sparse Mixture-of-Experts (MoE) models increase model capacity by reducing per-token computation, but their inference efficiency is affected by expert activation patterns.
Core Problem
The inference efficiency of MoE models is limited by expert activation patterns, and traditional speculative decoding strategies can lead to expert scattering, increasing memory traffic and inference latency.
Innovation
EcoSpec introduces the consideration of expert activation costs into speculative decoding draft selection strategies, reducing expert scattering and improving memory utilization efficiency.
Methodology
- �� Use a lightweight expert predictor to estimate expert activation for each draft token. • A dynamic expert buffer records already covered experts to reduce redundant activation. • Select draft paths with high acceptance probability and reuse existing experts.
Experiments
Experiments were conducted on DeepSeek-V3.1, Qwen3-235B-A22B, and GPT-OSS-120B models using reasoning, coding, question-answering, and dialogue benchmarks to validate EcoSpec's performance improvements.
Results
EcoSpec achieved a 1.62x decoding speedup across multiple models, significantly reducing the number of active experts and memory traffic, validating its effectiveness in large-scale MoE models.
Applications
EcoSpec is applicable to large-scale language models requiring efficient inference, especially in scenarios where memory traffic and inference latency are bottlenecks.
Limitations & Outlook
EcoSpec relies on the accuracy of expert predictions, which may not perform well on certain tasks; managing the expert buffer may become a bottleneck in extremely large-scale models.
Plain Language Accessible to non-experts
Imagine a factory where workers are experts and machines are models. Traditional methods have each worker working independently, leading to inefficiencies. EcoSpec acts like a smart manager, efficiently assigning workers to tasks, reducing redundancy, and improving efficiency.
ELI14 Explained like you're 14
Imagine you're playing a game with many characters, each with different skills. EcoSpec is like a smart player who knows how to use each character's skills wisely, not wasting resources, and quickly winning the game!
Glossary
Mixture-of-Experts
A model architecture that increases capacity by using multiple expert modules, with only a subset activated per input.
Used for inference acceleration in large-scale language models.
Speculative Decoding
An inference acceleration method that proposes multiple candidate tokens for parallel verification to speed up generation.
Used to reduce inference latency in large models.
EcoSpec
A cost-aware speculative decoding framework that optimizes expert activation costs to improve decoding efficiency.
Applied in large-scale MoE models to reduce memory traffic.
Expert Scattering
High-probability draft tokens may activate disjoint experts, increasing memory traffic.
A problem in traditional speculative decoding strategies.
Expert Predictor
A lightweight model used to predict the set of experts likely to be activated by draft tokens.
Used in EcoSpec's draft selection strategy.
Open Questions Unanswered questions from this research
- 1 How to improve expert prediction accuracy without increasing complexity?
- 2 How to effectively manage the expert buffer in extremely large-scale models?
Applications
Immediate Applications
Efficient Inference
EcoSpec can be used in large-scale language model applications requiring fast response, such as real-time translation and dialogue systems.
Long-term Vision
Intelligent Model Optimization
By optimizing expert activation, EcoSpec could drive smarter model designs in the future, reducing computational resource waste.
Abstract
Sparse Mixture-of-Experts (MoE) models have become an important approach for scaling Large Language Models (LLMs), but their inference efficiency depends strongly on expert activation patterns. Speculative decoding (SD) accelerates autoregressive generation by verifying multiple draft tokens in parallel, yet existing draft selection strategies primarily optimize acceptance likelihood. In large-scale MoE models, however, selecting draft tokens also determines the union of experts activated during verification. We observe that confidence-driven SD can introduce \textit{expert scattering}: high-probability draft tokens may route to disjoint experts, increasing expert-weight memory traffic and reducing the speedup from speculation. Motivated by this observation, we revisit draft-tree selection under the non-uniform memory-cost structure of MoE inference. We propose \textsc{EcoSpec}, a cost-aware speculative decoding framework that incorporates predicted marginal expert activation cost into draft selection. With a lightweight expert predictor and a dynamic expert buffer, \textsc{EcoSpec} favors draft paths that preserve high acceptance likelihood while reusing experts already covered by the current verification set, without modifying the target-model verification rule. We evaluate \textsc{EcoSpec} on three large-scale MoE models, including DeepSeek-V3.1 (671B), Qwen3-235B-A22B, and GPT-OSS-120B, across reasoning, coding, question-answering, and dialogue benchmarks. \textsc{EcoSpec} consistently reduces active expert footprints and improves end-to-end decoding speed, achieving up to $1.62\times$ speedup. These results show that accounting for expert activation cost is important for efficient speculative decoding in large-scale MoE models.