SlimQwen: Exploring the Pruning and Distillation in Large MoE Model Pre-training
This work systematically analyzes large-scale MoE model pruning and distillation, showing pruning outperforms training from scratch, with combined KD and progressive schedules yielding best results.
Key Findings
Methodology
The authors employ structured pruning (width, depth, expert pruning) combined with continual pretraining, comparing from-pruned-initialization versus training from scratch. They introduce partial-preservation expert merging, multi-token prediction (MTP) distillation, and progressive pruning schedules (depth-first, width-first, joint). Experiments on 400B tokens validate the effectiveness, evaluated via benchmarks like MMLU, GSM8K. The approach systematically assesses the impact of different strategies on model performance and convergence.
Key Results
- Pruned models initialized before pretraining outperform from-scratch models by about 11.79 points (73.45 vs. 61.66), recovering 86.5% of teacher performance despite being 3.4× smaller. Different expert pruning methods converge to similar performance after large-scale continual pretraining. The partial-preservation expert merging strategy improves downstream task results. Combining KD with LM loss and multi-token distillation enhances knowledge-intensive task performance. Progressive pruning schedules outperform one-shot compression, leading to a 4× compressed Qwen3-Next-80A3B into a 23A2B model with competitive performance.
- These findings demonstrate that structured pruning offers a better initialization, expert merging balances specialization and consolidation, and staged pruning with distillation ensures smoother training trajectories, enabling effective large-scale MoE model compression.
Significance
This research provides a comprehensive framework for efficient large-scale MoE model compression, addressing practical deployment challenges. By integrating pruning, distillation, and staged schedules, it reduces model complexity and inference costs while maintaining high performance. The methods facilitate resource-efficient deployment in industry, especially in scenarios with limited hardware, and advance the state-of-the-art in sparse model engineering, promoting sustainable AI development.
Technical Contribution
The paper systematically compares various pruning strategies (width, depth, expert), introduces a partial-preservation expert merging technique, and develops multi-token distillation (MTP KD). It demonstrates that progressive pruning schedules outperform one-shot methods, providing a practical pipeline for large-scale MoE compression. The approach bridges the gap between theoretical sparsity and real-world deployment, offering new insights into training dynamics and knowledge transfer in sparse models.
Novelty
This is the first comprehensive study to evaluate multiple expert pruning and merging strategies at pretraining scale, combined with multi-token distillation and progressive schedules. The partial-preservation merging strategy effectively balances expert retention and consolidation, preventing homogenization. These innovations collectively push the frontier of scalable, efficient MoE model compression, surpassing prior work focused mainly on dense models or isolated pruning techniques.
Limitations
- The methods are validated mainly on Qwen3-Next architecture; applicability to other MoE variants remains to be tested. The automatic tuning of scheduling parameters is lacking, requiring manual hyperparameter optimization. Compression may cause information loss impacting extreme tasks, necessitating further refinement of expert selection and merging algorithms.
- Computational costs during large-scale continual pretraining are significant, and the approach's scalability to even larger models or multi-modal settings needs exploration.
Future Work
Future directions include developing adaptive, automated scheduling algorithms, exploring more sophisticated expert merging techniques, and extending the framework to multi-modal models. Additionally, integrating reinforcement learning for dynamic pruning and merging decisions could further improve efficiency and robustness, facilitating broader adoption in industry.
AI Executive Summary
As large language models (LLMs) continue to grow, their deployment faces increasing challenges in computational and storage costs. Mixture-of-Experts (MoE) architectures offer a promising solution by sparsely activating only subsets of experts, enabling scalable expansion. However, this sparsity introduces complexity in model management and efficiency. Traditional dense model compression techniques like pruning and distillation have shown success, but their adaptation to MoE models remains underexplored.
This paper systematically investigates large-scale pretraining strategies for MoE models, focusing on structured pruning, expert merging, knowledge distillation, and progressive scheduling. The authors demonstrate that pruning a pretrained MoE model provides a significantly better initialization than training from scratch, leading to faster convergence and higher downstream performance. They introduce a partial-preservation expert merging strategy that retains the most important experts while merging less critical ones, effectively balancing specialization and consolidation.
Furthermore, the authors develop multi-token prediction (MTP) distillation, which supervises multiple future tokens simultaneously, enhancing knowledge transfer especially on complex tasks. They compare different progressive pruning schedules—depth-first, width-first, and joint—and find all outperform one-shot pruning, providing smoother training trajectories. The combined approach enables compressing a 80A3B model into a 23A2B model with minimal performance loss, validated across diverse benchmarks like MMLU, GSM8K, and Chinese proficiency tests.
Overall, this work offers a practical, scalable framework for efficient MoE model compression, addressing critical challenges in deploying large models in resource-constrained environments. It paves the way for future research on automated, adaptive pruning and merging strategies, extending the benefits of sparse models to broader applications and industries.
Deep Analysis
Background
The evolution of large language models (LLMs) has led to unprecedented performance but also immense resource demands. Dense models like GPT-3 and PaLM have pushed the limits of parameter size, resulting in high computational costs. MoE architectures, introduced by Shazeer et al. (2017), have emerged as a scalable alternative by activating only a subset of experts per token, thus reducing computation while maintaining capacity. Recent works such as Jiang et al. (2024) and Yang et al. (2025a) have demonstrated the effectiveness of MoE in scaling LLMs. Despite these advances, deploying such models remains costly, and model compression becomes essential for practical applications. Existing techniques like structured pruning and knowledge distillation have been successful in dense models, but their adaptation to MoE models involves unique challenges, especially in pruning experts without degrading their specialized knowledge. Prior studies (e.g., Lasby et al., 2025; Xia et al., 2024b) have explored expert pruning and merging but lack systematic evaluation at pretraining scale, particularly in conjunction with continual training and advanced distillation methods.
Core Problem
The core challenge is reducing the size and complexity of large-scale MoE models while preserving their performance. Traditional pruning methods risk losing critical expert knowledge, leading to performance degradation. Moreover, the optimal initialization for subsequent training remains unclear—whether starting from a pruned model or training from scratch. The difficulty lies in balancing expert specialization with consolidation, ensuring that pruning and merging do not homogenize experts and diminish their unique contributions. Additionally, designing effective training strategies post-pruning, such as distillation and scheduling, is crucial for performance recovery. Addressing these issues requires a comprehensive framework that integrates pruning, expert merging, distillation, and staged training, validated on massive datasets like 400B tokens.
Innovation
The paper introduces several key innovations. First, a partial-preservation expert merging strategy retains the top half of experts based on importance scores, merging the rest to prevent homogenization. Second, multi-token prediction (MTP) distillation extends standard knowledge distillation by supervising multiple future tokens simultaneously, improving the training dynamics. Third, the authors systematically compare progressive pruning schedules—depth-first, width-first, and joint—showing all outperform one-shot pruning in terms of convergence and performance recovery. These methods collectively form a practical pipeline for large-scale MoE compression, balancing efficiency and accuracy, and enabling models like Qwen3-Next to be compressed from 80A3B to 23A2B with minimal performance loss.
Methodology
- �� 结构化剪枝：利用激活统计、路由频率、软logits和REAP指标选择剪枝目标，分别进行宽度、深度和专家剪枝。
- �� 专家合并：基于专家相似性和重要性，采用部分保持策略，保留最重要的专家，剩余专家合并，避免专家同质化。
- �� 蒸馏方法：引入多Token预测（MTP）蒸馏，通过监督多个未来Token，增强知识迁移。
- �� 逐步调度：设计深度优先、宽度优先和联合调度，逐步剪枝，确保训练平滑过渡。
- �� 实验验证：在80A3B模型上，进行400B tokens持续预训练，评估多任务性能，比较不同策略的效果。
Experiments
采用Qwen3-Next架构，进行深度、宽度和专家剪枝，结合多Token蒸馏，比较不同调度策略。训练数据为400B tokens，评估指标包括MMLU、GSM8K、BBH等。对比从零训练、剪枝初始化、不同蒸馏方法和调度策略的性能差异。验证策略在大规模持续预训练中的有效性，分析模型收敛速度和任务表现。最终模型压缩到23A2B，性能与原始模型相当。
Results
剪枝预训练模型在多项任务中优于从零训练，平均提升11.79点（73.45 vs. 61.66），恢复到教师模型86.5%的性能。专家合并策略提升下游任务表现，MTP KD结合LM损失在知识密集任务中表现优异。渐进调度在保持性能的同时，优化训练路径。最终模型在多任务评估中表现出色，验证了策略的实用性和有效性。
Applications
该方法适用于大规模语言模型的部署优化，特别在硬件资源有限的环境中实现高效推理。可在工业界的对话系统、内容生成、搜索引擎等场景中应用，降低硬件成本，提升响应速度。未来结合自动调度和专家选择算法，将推动稀疏模型的智能化和自动化发展。
Limitations & Outlook
目前方法主要在Qwen架构验证，泛化到其他MoE模型仍需验证。调度参数需手动调优，缺乏自动化机制。模型压缩可能引起信息丢失，影响极端任务表现。未来需优化专家合并策略和调度算法，提升鲁棒性和适应性。
Plain Language Accessible to non-experts
想象你有一个大工厂，里面有很多不同的工人（专家），每个工人擅长不同的任务。为了节省空间和人力，你决定只用一些最重要的工人，或者把一些工人合并成一个更厉害的工人。刚开始时，所有工人都在工作，但这样很慢也很贵。后来，你只留下最擅长的工人，或者把一些工人合并，既节省空间，又能完成大部分任务。这就像模型剪枝和专家合并，目标是让工厂变得更小、更快，但还保持大部分生产能力。
ELI14 Explained like you're 14
假设你有一个超级大的学校，里面有很多老师（专家），每个老师教不同的科目。学校太大了，管理和维护都很困难。于是，你想把一些老师合并，或者只留下最厉害的老师。开始时，所有老师都在教课，但这样很慢也很累。后来，你决定只留一些最棒的老师，或者把一些老师合并成一个更厉害的老师。这样，学校变得更小、更容易管理，但学生们还是能学到大部分的知识。这就像把大模型变小一样，既节省资源，又能保持学习效果。
Glossary
结构化剪枝 (Structured Pruning)
通过删除模型中的完整结构单元（如层、专家或注意力头）实现模型压缩，减少计算量。
在论文中用于减少模型深度、宽度和专家数量。
知识蒸馏 (Knowledge Distillation)
将大模型的知识转移到小模型中，通过训练使小模型模仿大模型的输出。
结合多Token预测蒸馏提升模型性能。
渐进调度 (Progressive Scheduling)
逐步调整模型结构（如深度、宽度或专家数），平滑过渡到目标架构。
优化剪枝和蒸馏过程中的训练路径。
专家合并 (Expert Merging)
将相似或重要性低的专家合并成一个专家，减少专家数量。
提出部分保持策略，避免专家同质化。
多Token预测蒸馏 (MTP KD)
扩展蒸馏目标到多个未来Token，提高知识迁移效率。
提升模型在知识密集任务中的表现。
Open Questions Unanswered questions from this research
- 1 如何自动优化专家合并和调度策略，提升模型压缩的鲁棒性和效率。
- 2 在不同类型的MoE架构中验证方法的普适性和效果。
Abstract
Structured pruning and knowledge distillation (KD) are typical techniques for compressing large language models, but it remains unclear how they should be applied at pretraining scale, especially to recent mixture-of-experts (MoE) models. In this work, we systematically study MoE compression in large-scale pretraining, focusing on three key questions: whether pruning provides a better initialization than training from scratch, how expert compression choices affect the final model after continued training, and which training strategy is most effective. We have the following findings: First, across depth, width, and expert compression, pruning a pretrained MoE consistently outperforms training the target architecture from scratch under the same training budget. Second, different one-shot expert compression methods converge to similar final performance after large-scale continual pretraining. Motivated by this, we introduce a simple partial-preservation expert merging strategy that improves downstream performance across most benchmarks. Third, combining KD with the language modeling loss outperforms KD alone, particularly on knowledge-intensive tasks. We further propose multi-token prediction (MTP) distillation, which yields consistent gains. Finally, given the same training tokens, progressive pruning schedules outperform one-shot compression, suggesting that gradual architecture transitions lead to better optimization trajectories. Putting it all together, we compress Qwen3-Next-80A3B to a 23A2B model that retains competitive performance. These results offer practical guidance for efficient MoE compression at scale.