UniMoMo: Expert Merging-Based MoE Acceleration for Large Recommendation Models
UniMoMo compresses trained sparse MoE recommendation models via expert merging, achieving speedups of up to 2.21× with minimal performance loss.
Key Findings
Methodology
UniMoMo transforms a trained sparse MoE into a smaller, standard MoE under a specified expert budget by leveraging an offline, calibration-based graph coarsening approach. It evaluates expert responses on unlabeled data using KL divergence to build an affinity graph, then applies a greedy merging algorithm with traffic-aware protection to prevent high-traffic experts from being merged repeatedly. The experts are fused via parameter averaging and linear response correction, ensuring minimal performance degradation. The process requires no online modules, only offline calibration and a brief supervised fine-tuning, enabling seamless deployment at various resource budgets.
Key Results
- Across datasets Amazon Beauty, KuaiRec, and TenRec with 2, 4, and 6 MoE blocks, the compressed models retain 99.92%–102.30% of the original NDCG@10 scores, while achieving speedups of 1.28× to 1.63×. In more aggressive two-expert configurations, performance remains within 98.36%–104.24%, with speedups reaching 2.21×. These results demonstrate the method’s robustness across different depths and budgets.
- Expert response-based similarity and traffic protection effectively prevent performance loss during expert merging, enabling significant model size reduction without sacrificing ranking quality. The experimental results confirm the approach’s practicality in real-world recommendation scenarios.
- The method’s flexibility allows it to adapt trained MoE checkpoints to multiple deployment budgets, facilitating efficient inference in production environments. It offers a scalable solution for large recommendation systems seeking to balance capacity and efficiency.
Significance
This work addresses a critical bottleneck in deploying large sparse MoE recommendation models by providing a systematic, offline expert merging framework. It enables models trained with high capacity to be compressed to smaller, faster variants without retraining from scratch, thus bridging the gap between training and deployment. The approach enhances the practicality of MoE architectures in industry, supporting scalable, resource-efficient recommendation services. It also opens avenues for further research into behavior-aware model compression and adaptive expert management.
Technical Contribution
UniMoMo introduces a novel expert merging paradigm based on functional response similarity and traffic-aware protection, formulated as a graph coarsening problem. It departs from traditional parameter-distance-based methods, leveraging expert response distributions derived from unlabeled calibration data. The method employs KL divergence to quantify expert behavior similarity, constructs an affinity graph, and applies greedy merging with protected high-traffic experts. The expert parameters are fused via response-guided weighted averaging and linear correction, ensuring output fidelity. The entire process is offline, requiring only calibration data and minimal supervised fine-tuning, simplifying integration into existing systems.
Novelty
UniMoMo is the first to formulate expert merging for recommendation MoE models as a traffic-conditioned graph coarsening problem based on expert response behavior. Unlike prior approaches relying solely on parameter proximity or output similarity, it explicitly considers functional response similarity and traffic exposure, enabling more effective and stable expert reduction. Its offline, calibration-based framework offers a practical, deployment-ready solution that preserves model performance across multiple resource budgets.
Limitations
- The effectiveness depends on the representativeness of the calibration set; biased or insufficient data may lead to suboptimal merging and performance degradation.
- In extreme compression scenarios, the model may still experience some performance loss, especially in complex or highly dynamic environments.
- The computational overhead of expert response evaluation and graph coarsening can be significant for very large models, requiring further optimization.
Future Work
Future research could explore adaptive, online expert merging strategies that respond to real-time traffic shifts, as well as multi-layer, multi-scale merging schemes. Integrating self-supervised or reinforcement learning-based fine-tuning could further enhance robustness. Additionally, extending the framework to other modalities and tasks beyond recommendation systems presents promising directions.
AI Executive Summary
Large-scale recommendation systems increasingly rely on sparse mixture-of-experts (MoE) architectures to boost capacity and performance. However, deploying these models efficiently remains challenging due to their large expert banks, which demand significant storage and computational resources. Existing methods primarily focus on optimizing routing or sparse execution, leaving the expert bank size fixed after training. This disconnect limits practical deployment, especially under resource constraints.
UniMoMo introduces a novel offline expert merging framework that transforms a trained sparse MoE into a smaller, standard MoE aligned with a specified expert budget. The core idea is to evaluate experts’ functional responses on unlabeled calibration data, constructing an affinity graph based on response similarity measured via KL divergence. A greedy graph coarsening algorithm merges experts with high similarity while respecting traffic exposure constraints, ensuring high-traffic experts are protected from repeated merging. The merged experts are reconstructed through response-guided parameter averaging and linear correction, preserving model fidelity.
Extensive experiments on Amazon Beauty, KuaiRec, and TenRec datasets demonstrate that UniMoMo achieves near-original performance (within 1-2%) while accelerating inference by up to 2.21×. The compressed models maintain high ranking quality (NDCG@10 ratios of 99.92%–102.30%) across various depths and expert budgets. These results validate the approach’s effectiveness and practicality, enabling large recommendation models to be efficiently deployed under diverse resource constraints.
This work significantly advances the deployment of large sparse MoE models, offering a scalable, offline compression solution that preserves performance and reduces inference costs. It opens new avenues for adaptive expert management and resource-aware model deployment, fostering broader adoption of high-capacity recommendation architectures in industry. Future directions include online adaptive merging, multi-layer schemes, and cross-modal applications, promising further improvements in model efficiency and robustness.
Deep Analysis
Background
近年来，推荐系统在模型规模和复杂度方面持续扩大，从因子分解模型到深度神经网络，再到基于注意力机制的交互层和序列编码器，极大提升了推荐效果。伴随模型容量的增长，稀疏专家（MoE）架构成为提升模型表达能力的重要手段，广泛应用于多任务、多行为场景中。现有研究多集中在模型训练、架构设计和稀疏执行优化，然而训练完成后，如何在保证性能的同时，有效压缩专家数量，满足实际部署的资源限制，仍是难点。传统的剪枝、量化方法在固定参数空间内优化，难以应对专家响应行为的复杂性。专家融合技术如MergeMoE、HC-SMoE等，试图在保持性能的同时减少专家数，但多依赖参数距离或输出相似性，未充分考虑专家在推荐状态下的行为一致性和流量分布。本文提出的UniMoMo，结合专家响应的功能相似性和流量保护，提供一种全新的离线压缩方案，填补了模型部署中的空白。
Core Problem
稀疏MoE模型在训练完成后，如何在保证推荐性能的基础上，将专家数量压缩到有限预算内，是实际应用中的核心难题。现有方法多依赖参数距离或输出相似性，忽视了专家在不同推荐状态下的行为一致性和实际流量分布。若未合理合并，可能导致模型性能下降、推理效率降低，甚至出现流量不均衡的问题。如何在不引入额外在线模块的情况下，利用离线校准数据实现专家合并，成为关键技术难题。这关系到模型的实际部署效率和推荐质量。
Innovation
UniMoMo的创新点主要体现在：1）提出基于专家响应的功能相似性评估机制，避免参数距离误导；2）引入专家流量保护机制，防止高流量专家被频繁合并，确保模型稳定性；3）采用图粗化算法，将专家合并问题转化为专家相似图的簇划分，结合响应的KL散度作为边权，优化合并策略；4）融合专家参数时，利用响应的线性校正确保非线性专家输出的一致性。整个流程无需在线压缩模块，仅依赖离线校准数据，简化部署流程，提升效率。该方法在保持模型性能的同时，大幅度缩减专家数，满足不同预算需求。
Methodology
- �� 利用训练好的稀疏MoE模型和无标签校准集，评估每个专家在共享状态下的响应，计算响应的均值和方差，构建专家响应的高斯模型。
- �� 通过KL散度衡量专家间的行为差异，构建专家相似图，边权反映专家行为的相似程度。
- �� 采用贪心图粗化算法，逐步合并相似度高且未被保护的专家对，形成目标专家数的簇。
- �� 在合并过程中，考虑专家的路由流量，保护高流量专家，避免其被频繁合并。
- �� 利用簇内专家响应的加权平均参数，结合响应的线性校正，融合专家参数，确保输出一致性。
- �� 最后，重建压缩后模型的路由和专家，进行少量微调以恢复性能。
Experiments
在Amazon Beauty、KuaiRec和TenRec数据集上，分别测试2、4、6层MoE模型。对比原始模型与UniMoMo压缩模型的性能变化（NDCG@10、速度提升），评估不同专家预算的效果。多组实验验证不同层数和专家数的鲁棒性，进行参数敏感性分析和消融研究，确保方法的普适性和稳定性。
Results
压缩后模型在所有数据集和层数上，性能几乎保持不变（相对性能99.92%至102.30%），速度提升1.28×至1.63×。在极端压缩（两个专家）下，性能仍达98.36%至104.24%，速度提升最高达2.21×。专家响应的功能相似性评估和流量保护机制，有效避免性能下降，验证了方法的有效性和鲁棒性。
Applications
该方法适用于工业推荐系统中的模型压缩与部署，尤其在需要快速响应和低延迟场景中。通过离线压缩，模型可以在不同硬件平台和预算下灵活部署，降低存储和计算成本，提升用户体验。未来，可结合动态流量调度，进一步优化模型的适应性和鲁棒性。
Limitations & Outlook
依赖校准数据的代表性，若数据偏差大，可能影响合并效果。极端压缩可能导致性能下降，尤其在复杂任务或变化的流量分布中。合并过程计算成本较高，需优化算法以适应大规模模型。未来需探索动态保护机制和自适应微调策略。
Plain Language Accessible to non-experts
想象你经营一家工厂，里面有很多不同的工人（专家），每个工人擅长不同的任务。平时工厂运转正常，但如果需要节省成本，你可能会考虑把一些工人合并，让他们共同完成任务。可是，不能随便合并，否则可能会影响工厂的效率。UniMoMo就像一个聪明的管理者，它会根据工人的工作表现（专家响应）和工厂的订单（推荐状态）来决定哪些工人可以合并，哪些不能。它会先评估每个工人在不同任务中的表现，然后用一种特殊的算法，把表现相似、工作量不高的工人合成一组。这样，工厂就能用更少的人手，依然保持高效率。整个过程不用在工厂里增加新设备，只是在后台做一些调整，最后让工厂继续正常运转，节省成本又不影响质量。
ELI14 Explained like you're 14
想象你在学校里有很多不同的老师（专家），每个老师都擅长不同的科目。有时候，老师们会教很多学生（模型的推荐任务），但如果学校想节省开支，就需要让一些老师合并，减少老师的数量。可是，不能随便合并，否则学生会觉得老师变得不专业或者不懂他们的需求。UniMoMo就像一个聪明的校长，它会观察每个老师的教学风格和学生的反馈（专家响应），判断哪些老师的教学内容很像，可以合成一组。它还会考虑哪些老师每天都很忙（高流量专家），避免把他们合并掉。这样，学校可以用更少的老师，依然让学生学得很好。整个过程不用在课堂上增加新老师，只是在后台调整老师的组合，最后让学校正常运作，既省钱又保证教学质量。
Glossary
稀疏专家模型 (Sparse MoE)
一种模型架构，通过只激活部分专家实现高效推理，减少计算资源。技术上采用路由机制选择专家。
本文中用于提升推荐模型容量和效率的基础架构。
专家合并 (Expert Merging)
将多个专家的参数和行为融合成一个专家，以减小模型规模。技术上包括参数融合和行为匹配。
本文核心技术，用于模型压缩。
图粗化 (Graph Coarsening)
通过合并相似节点形成较大簇的算法，用于简化图结构。应用于专家合并中评估专家相似性。
UniMoMo中的关键算法步骤。
响应相似性 (Response Similarity)
衡量两个专家在相同输入下输出行为的相似程度，采用KL散度等指标。
评估专家行为一致性的重要依据。
受保护机制 (Protection Mechanism)
在合并过程中，避免高流量专家被频繁融合，确保模型稳定性。
UniMoMo中的关键策略。
Open Questions Unanswered questions from this research
- 1 如何在更复杂的动态流量环境中保持专家合并的鲁棒性仍需研究。当前方法主要依赖静态校准集，未来需探索在线自适应机制。
- 2 专家响应的评估指标是否能更好反映实际推荐性能，仍有优化空间。
Abstract
Sparse mixture-of-experts (MoE) layers expand recommendation capacity through conditional computation, yet a trained checkpoint still stores and routes over its full expert bank. We study a deployment problem: convert that checkpoint to a smaller standard MoE under an explicit expert budget, without adding a compression-specific online module. To address this, we introduce UniMoMo, a post-training compression framework formulated as a constrained graph coarsening problem. Rather than relying on parameter distance, UniMoMo groups experts based on their functional similarity, using an unlabeled calibration set to measure how similarly experts respond to shared recommendation states. To prevent performance degradation, we introduce a layer-adaptive protection mechanism that restricts the merging of high-traffic experts based on their routing exposure. Across Amazon Beauty, KuaiRec, and TenRec with 2, 4, and 6 MoE blocks, the final four-expert checkpoints obtain source-relative five-run mean NDCG@10 ratios of 99.92%--102.30% and measured A100 speedups of 1.28$\times$--1.63$\times$. An aggressive two-expert, top-1 operating point obtains ratios of 98.36%--104.24% and speedups of 1.47$\times$--2.21$\times$. These endpoint results evaluate the complete conversion-and-adaptation workflow and show that a trained recommendation MoE can be exported at multiple serving budgets.