S2-MoE: Enabling Efficient Self-Speculative Decoding for Mixture-of-Experts on Edge Devices
S2-MoE employs routing-aware adaptive speculative expansion and expert reuse gating to accelerate MoE inference on edge devices, achieving up to 5.3× speedup.
Key Findings
Methodology
The proposed S2-MoE framework integrates routing-aware adaptive speculative expansion, reuse-aware expert gating, and shared context-based self-speculative decoding. It dynamically adjusts the expansion of draft tokens based on predicted verification utility, reduces redundant verification by promoting expert parameter reuse, and aligns draft and target decoding through shared KV caches. Implementation in llama.cpp demonstrates end-to-end efficiency. Experiments across multiple MoE models and datasets show significant speedups, with the highest reaching 5.3× and an average of 2.0×, outperforming standard autoregressive decoding.
Key Results
- On Jetson Orin, S2-MoE achieves up to 5.3× speedup, averaging around 2.0×, with verification costs reduced by over 30% and expert reuse increased by 15%. On RTX 4090, speedups range from 1.2× to 2.9×, with notable efficiency gains.
- Across models like DeepSeek, Qwen3, GPT-OSS and datasets such as Natural Questions, results show consistent improvements in inference speed, verification time, and expert activation efficiency. Ablation studies confirm the importance of routing-aware expansion and shared context.
- The experiments validate the accuracy of the verification cost prediction model, which closely matches measured latency (R² > 0.94), ensuring reliable adaptive strategies. The approach demonstrates robustness and scalability across hardware and model scales.
Significance
This work addresses the critical bottleneck of high verification costs in deploying large sparse models on resource-constrained edge devices. By innovatively combining routing-aware speculation and expert reuse, it significantly enhances inference speed and efficiency, making large-scale LLMs more practical for real-world edge applications. The methodology paves the way for deploying sophisticated NLP models in privacy-sensitive and latency-critical scenarios, such as mobile devices, IoT, and autonomous systems, thus broadening the impact of advanced AI.
Technical Contribution
The paper introduces a novel routing-aware adaptive speculative expansion mechanism that predicts verification costs via expert routing signals, enabling dynamic candidate expansion. It also proposes a reuse-aware expert gating strategy that promotes activation of similar experts across tokens, boosting parameter reuse. Additionally, the shared context design ensures the draft and target models share the same KV cache, reducing error accumulation. These innovations collectively improve the efficiency and robustness of MoE inference on edge hardware, with theoretical validation and practical system implementation.
Novelty
This study is the first to systematically incorporate routing-aware utility into speculative decoding for MoE models, addressing the challenge of high verification costs due to diverse expert activations. Unlike prior confidence-based or static strategies, it dynamically balances acceptance likelihood and verification overhead based on expert routing predictions. The shared context approach further reduces error propagation, setting a new standard for efficient MoE inference in resource-limited environments.
Limitations
- The accuracy of expert routing prediction directly impacts the efficiency; significant routing errors may reduce speedups. The method assumes reliable latency estimation for experts, which may vary across hardware. In extremely constrained environments, the overhead of cost prediction and gating mechanisms might offset gains. Future work should focus on improving routing prediction robustness, reducing computational overhead, and extending to multi-modal models.
Future Work
Future directions include refining expert routing prediction models, integrating hardware-aware scheduling, and extending the framework to multi-task and multi-modal scenarios. Additionally, exploring hardware acceleration for gating and verification processes could further reduce latency. Developing adaptive mechanisms for ultra-low-resource devices and broadening applicability to diverse NLP tasks will also be key areas of research.
AI Executive Summary
Large language models (LLMs) have revolutionized natural language processing, but deploying them efficiently on edge devices remains a major challenge due to limited memory and bandwidth. Traditional autoregressive decoding, while effective, suffers from high verification costs, especially in sparse models like Mixture-of-Experts (MoE), where diverse expert activations lead to redundant computations. To address this, the paper introduces S2-MoE, a novel decoding framework that combines routing-aware adaptive speculative expansion, expert reuse-guided gating, and shared context-based decoding. These innovations enable the model to dynamically select promising draft tokens, minimize redundant verification, and maintain alignment between draft and target models, significantly boosting inference speed.
Experiments on platforms like Jetson Orin and RTX 4090 demonstrate that S2-MoE achieves up to 5.3× acceleration, with an average of 2.0×, across various MoE architectures and datasets. The approach reduces verification overhead by over 30%, increases expert parameter reuse, and maintains high acceptance rates, thus making large sparse models more practical for real-world edge applications. The system's design leverages expert routing predictions to estimate verification costs accurately, allowing adaptive expansion that balances potential gains against computational expenses.
This work marks a significant step forward in edge AI, enabling faster, more resource-efficient deployment of large language models. It offers a scalable solution that addresses core bottlenecks of verification cost and expert diversity, opening new avenues for privacy-preserving, low-latency AI services in mobile and IoT devices. Despite its success, challenges remain in routing prediction accuracy and hardware resource constraints, guiding future research toward more robust, hardware-aware, multi-modal solutions.
Deep Analysis
Background
近年来，深度学习特别是大规模语言模型（如GPT、BERT、Qwen）在自然语言处理领域取得巨大突破。随着模型参数规模不断扩大，模型性能持续提升，但推理效率和资源消耗成为瓶颈。稀疏模型如MoE通过激活部分专家实现参数稀疏调度，有效降低计算和存储成本，适合边缘设备部署。然而，传统解码策略在保持高质量输出的同时，验证成本高昂，限制了其在实际场景中的应用。投机解码（SD）被提出以提升推理速度，但在MoE模型中，由于专家激活多样性和验证成本高，效果受限。现有研究多关注系统优化或启发式策略，缺乏系统性解决验证冗余和专家重用问题的方法。因此，如何在有限资源下实现高效、低成本的MoE推理，成为当前研究的热点。
Core Problem
边缘设备上的MoE模型推理面临多重挑战。首先，验证成本高，因不同候选词激活不同专家，导致验证过程中的参数访问频繁且冗余。其次，专家激活的多样性降低了参数重用率，增加了验证复杂度。再次，传统投机解码在MoE中难以实现长接受长度和高速度的平衡，因验证成本随候选数增加而剧增。轻量化策略虽降低了模型复杂度，但也带来了预测准确率下降和验证效率不足的问题。解决这些问题，需设计新机制以降低验证成本、提升专家重用率，并优化投机策略的适应性，从而实现边缘设备上的高效推理。
Innovation
本研究提出三大创新：1）路由感知的自适应投机扩展，利用专家路由预测验证成本和接受潜力，动态调节候选扩展范围，减少冗余验证。2）专家重用感知的门控机制，通过软偏置促进相似专家的跨候选激活，提高参数利用率。3）共享上下文机制，使草稿模型和目标模型共享KV缓存，抑制误差积累。这些创新结合，有效缓解了MoE模型中验证成本高、专家激活多样带来的性能瓶颈，显著提升推理效率和鲁棒性。
Methodology
- �� 设计路由感知的投机扩展策略，利用专家路由预测验证成本，动态调节候选扩展范围。• 构建专家重用门控机制，通过软偏置增强相似专家的激活频率，提升参数重用率。• 实现共享上下文机制，使草稿和目标模型共享KV缓存，减少误差积累。• 在llama.cpp中集成系统，利用专家路由预测验证成本，优化投机策略。• 采用多模型、多数据集进行验证，评估速度提升、验证成本和专家激活效率。• 通过消融实验验证各个模块的贡献，确保整体性能提升。• 设计动态调节机制，适应不同硬件环境和模型规模，增强系统鲁棒性。
Experiments
实验在Jetson Orin和RTX 4090平台上进行，测试模型包括DeepSeek、Qwen3和GPT-OSS，数据集为Natural Questions。指标涵盖推理速度、验证时间、专家激活次数和接受率。采用不同的投机策略和参数配置，进行消融分析。结果显示，最高速度提升达5.3倍，平均约2.0倍，验证成本降低30%以上，专家重用率提升15%，验证效率显著改善。
Results
在Jetson Orin平台，S2-MoE实现最高5.3倍速度提升，平均约2.0倍，验证成本降低30%以上，专家重用率提升15%。在RTX 4090上，速度提升范围为1.2到2.9倍，验证时间明显缩短。多模型、多数据集验证显示其普适性和鲁棒性。消融实验验证了路由感知扩展和共享上下文的关键作用，显著降低验证冗余和误差累积。整体结果表明，该方法在边缘场景中具有极强的实用价值和推广潜力。
Applications
该技术适用于边缘智能终端、移动设备和物联网平台，支持大规模稀疏模型的快速推理。只需在硬件有限的环境中部署经过优化的MoE模型，便可实现高效、低延迟的自然语言处理任务。未来还可结合硬件加速和多模态信息，推动智能设备的自主学习和交互能力提升，为智能制造、智能家居等行业带来变革。
Limitations & Outlook
目前方法依赖于专家路由预测的准确性，若路由偏差较大，将影响验证效率和模型性能。硬件资源极度有限时，实时调节和验证成本估算仍存在挑战。未来需优化专家门控机制，增强模型适应性和鲁棒性，降低复杂度，提升在极端环境下的表现。
Plain Language Accessible to non-experts
想象你在厨房做饭，厨师需要准备各种食材（模型参数），每次做菜都要拿出不同的食材（专家）来搭配。传统做法是每次都全部拿出所有食材，既麻烦又浪费时间。现在，有一种聪明的厨师（S2-MoE）可以根据菜谱（路由预测）提前判断哪些食材会用到，只拿需要的部分，减少准备时间。更厉害的是，他还能根据菜的味道（验证成本）灵活调整用料，确保每次都做得快又好吃。这就像在厨房里用智能工具，既省事又高效，能在有限的厨房空间（边缘设备）里做出美味佳肴（高效推理）。
ELI14 Explained like you're 14
想象你在学校的食堂里吃饭，厨师要准备很多不同的菜（模型的专家），每次做饭都要用不同的食材。以前，厨师会把所有食材都准备好，然后根据学生的点单（模型输入）逐个做菜，但这样很浪费时间，也不够快。现在，有个聪明的厨师（S2-MoE）可以提前猜到学生会点什么菜（用路由预测），只准备可能用到的食材，节省了很多时间。而且，他还能根据菜的味道（验证成本）灵活调整食材用量，确保每份饭都快又好吃。这就像用智能工具帮你节省时间，做饭变得又快又好，特别适合厨房空间有限的学校食堂。
Abstract
Deploying large language models (LLMs) for inference on edge devices is challenging due to severe memory and bandwidth constraints. While speculative decoding and Mixture-of-Experts (MoE) have been proposed to improve inference efficiency, naively combining them often incurs excessive verification overhead and poor expert reuse, limiting their effectiveness in memory-bound edge settings. In this work, we propose S2-MoE, an efficient self-speculative decoding framework for MoE inference on edge devices. S2-MoE reduces redundant verification through routing-aware adaptive speculative expansion, improves verification efficiency with reuse-aware expert gating, and aligns draft and target execution via shared context. Implemented in llama$.$cpp, S2-MoE achieves up to $5.3\times$ speedup (about $2.0\times$ on average) over standard autoregressive decoding across diverse MoE models and datasets on edge devices. Code is available at https://github.com/angerybob/S2-MoE.