UniMixer: A Unified Architecture for Scaling Laws in Recommendation Systems
UniMixer introduces a unified recommendation scaling architecture with parameterized TokenMixer, combining attention, TokenMixer, and FM advantages for improved efficiency.
Key Findings
Methodology
This work proposes transforming rule-based TokenMixer into a learnable parameterized module, constructing a unified feature mixing framework that integrates attention, TokenMixer, and FM. By parameterizing global and local feature interactions, the architecture reduces computational complexity and enhances scalability. The design includes a lightweight UniMixing-Lite module, which further compresses parameters and accelerates inference. Experiments on Kuaishou datasets evaluate AUC, parameter count, and FLOPs, demonstrating stable and superior scaling behavior across model sizes.
Key Results
- At 100 million parameters, UniMixer achieves an AUC of 0.753, outperforming RankMixer by 0.012, with a 15% reduction in FLOPs. Across various scales, the model exhibits consistent scaling curves, confirming its efficiency and robustness.
- UniMixer-Lite maintains an AUC of 0.750 with half the parameters, indicating high parameter efficiency. Offline and online A/B tests show a 3% increase in CTR and conversions compared to baseline architectures.
- Ablation studies reveal that parameterized TokenMixer improves performance by 8%, while parameter compression reduces parameters by 30% and inference time by 20%, with minimal accuracy loss.
Significance
This research provides a theoretical and practical framework for scalable recommendation models, unifying diverse architectures under a common paradigm. It addresses key bottlenecks in model efficiency and scalability, facilitating deployment of larger, more accurate recommenders in real-world systems. The unified approach paves the way for future large-scale recommendation engines that are both performant and resource-efficient, impacting industry and academia alike.
Technical Contribution
The paper introduces a parameterized TokenMixer that generalizes rule-based operations, enabling learnability and optimization. It establishes a unified theoretical framework connecting attention, TokenMixer, and FM, supported by a novel global-local feature interaction scheme. The design reduces computational overhead via optimized pipelines and low-rank approximations. The lightweight UniMixing-Lite further enhances parameter efficiency, making large-scale deployment feasible. These innovations significantly advance the understanding and engineering of scalable recommendation architectures.
Novelty
This is the first work to parameterize rule-based TokenMixer, creating a unified feature interaction framework that bridges attention, TokenMixer, and FM architectures. It introduces a learnable, optimized, and scalable module that surpasses prior isolated designs, offering a comprehensive theoretical foundation and practical efficiency improvements, marking a significant step forward in recommendation model scaling.
Limitations
- The architecture's performance may degrade in extremely sparse or high-dimensional feature spaces, where interaction modeling becomes challenging. Its reliance on parameterization may limit generalization in some scenarios.
- Online deployment and real-time adaptation require further validation, especially under dynamic user behaviors and system constraints.
- While the model reduces complexity, training large-scale models still demands substantial computational resources, and further hardware-aware optimizations are needed.
Future Work
Future directions include extending the framework to multi-modal data, multi-task learning, and reinforcement learning settings. Enhancing model robustness, interpretability, and real-time adaptability remains crucial. Additionally, integrating self-supervised signals and meta-learning could further improve scalability and generalization, pushing recommendation systems toward truly intelligent, autonomous content curation.
AI Executive Summary
The rapid growth of recommendation systems has driven the need for scalable, efficient architectures capable of handling massive data and complex feature interactions. Traditional models like attention-based transformers, TokenMixer, and factorization machines each offer unique strengths but also face limitations in computational cost, flexibility, and scalability. Recognizing these challenges, this paper introduces UniMixer, a unified architecture that parameterizes the core feature mixing operations, enabling seamless integration of these paradigms.
UniMixer transforms rule-based TokenMixer into a learnable, optimized module by introducing parameterization of permutation matrices, supported by Kronecker decompositions, which significantly reduces parameters and computational complexity. The framework unifies the diverse feature interaction mechanisms into a single theoretical model, allowing the model to adaptively learn the most effective mixing patterns during training. An innovative lightweight version, UniMixing-Lite, further compresses parameters by leveraging low-rank approximations and basis matrices, maintaining high performance while reducing resource demands.
Extensive offline and online experiments on Kuaishou datasets demonstrate that UniMixer achieves superior scaling laws. At 100 million parameters, it surpasses existing architectures with an AUC of 0.753, while reducing FLOPs by 15%. The lightweight variant maintains high accuracy with fewer parameters, showing excellent parameter efficiency. Ablation studies confirm that the parameterized TokenMixer and optimized pipelines are key to these improvements.
This work offers a comprehensive solution for large-scale recommendation systems, enabling models to scale efficiently without sacrificing performance. Its unified theoretical foundation bridges current architectural gaps, fostering future research in scalable, resource-efficient recommendation models. The approach has broad industry implications, promising more accurate, faster, and cost-effective personalized content delivery systems.
Deep Analysis
Background
推荐系统经历了从传统协同过滤到深度学习的演变，Transformer的引入极大增强了特征交互能力，但在异构特征和大规模场景中仍面临效率瓶颈。现有架构主要分为注意力机制、TokenMixer和因子分解机，各自解决不同问题，但都存在模型复杂度高、扩展性差的问题。随着数据规模和模型深度不断增长，如何在保证性能的同时实现高效扩展成为核心挑战。此前的研究如HiFormer、FAT、RankMixer等在特定场景取得一定成功，但缺乏统一的理论框架，难以系统性提升模型规模。
Core Problem
当前推荐模型在大规模扩展中面临多重难题，包括高昂的计算成本、参数冗余、交互效果不足以及架构割裂。注意力机制虽表达力强，但复杂度为O(T^2)，难以部署于超大规模场景。TokenMixer虽参数少，但缺乏可学习性，限制适应性。FM模型虽高效，但难以捕获高阶交互。如何融合这些架构的优势，建立统一、可扩展的模型架构，成为亟待解决的问题。
Innovation
本文提出UniMixer架构，核心创新包括• 将规则TokenMixer参数化，增强可学习性；• 设计统一的特征交互框架，融合注意力、TokenMixer和FM的优势；• 引入参数化的全局和局部交互机制，优化计算流程，降低复杂度；• 开发轻量级UniMixing-Lite模块，提升参数效率和推理速度。这些创新突破了传统架构的局限，为模型大规模缩放提供了理论和实践基础。
Methodology
- �� 特征分词：将输入特征按类别划分，嵌入后拼接成向量；• Token化：用线性层将嵌入向量划分为Token，形成输入序列；• UniMixer模块：
- 参数化TokenMixer，将规则操作转化为可学习矩阵Wperm，结合Kronecker分解降低参数；
- 设计全局（WG）和局部（WiB）交互机制，支持多尺度特征融合；
- 采用Sinkhorn-Knopp算法确保参数满足双随机性、稀疏性和对称性；
- 通过残差连接和归一化增强训练稳定性；• 轻量化设计：引入basis矩阵和低秩逼近，进一步压缩参数，提升效率。
Experiments
采用Kuaishou推荐数据集，比较UniMixer与RankMixer、Wukong等架构在不同参数规模（从几千万到亿级）下的性能。指标包括AUC、FLOPs、参数量。设置多组离线训练和线上AB测试，验证模型在CTR、转化率上的提升。还进行了消融实验，分析参数化TokenMixer、UniMixing-Lite的贡献。超参数调优包括学习率、层数、块大小，确保公平比较。
Results
UniMixer在参数为1亿时，AUC达0.753，比RankMixer提升0.012，FLOPs降低15%。UniMixing-Lite在参数减半时仍保持0.750的AUC，显示极高参数效率。多场景下模型表现稳定，验证了缩放规律的普适性。消融实验表明参数化TokenMixer提升8%，模型参数减少30%，推理速度提升20%。
Applications
该架构适用于大规模推荐场景，如电商、内容平台、广告投放。可在现有深度学习基础上快速部署，提升模型性能和效率。未来可结合多模态信息，支持个性化推荐、实时调度等复杂任务，推动行业智能化升级。
Limitations & Outlook
模型在极端稀疏特征或超大规模场景中仍面临交互不足或训练成本高的问题。参数化机制对特征多样性依赖较大，泛化能力有待验证。实际部署中，硬件资源和优化算法仍需改进，未来需解决模型的鲁棒性和可解释性。
Plain Language Accessible to non-experts
想象你在厨房做饭，食材就像推荐系统中的不同特征，有的像蔬菜，有的像调料。传统做法是每次都用不同的配方，效率低还容易出错。现在，厨师发明了一套可以自己调节比例的调料瓶，能根据不同菜肴自动调整用量。这就像UniMixer，把不同的特征交互规则变成可以学习和优化的参数，让模型在不同场景下都能表现得更好。它像一台智能厨师，懂得根据食材变化调整调料比例，做出美味佳肴。这样，推荐系统也能像厨房一样，快速、灵活地应对各种用户需求，提供更精准的内容。
ELI14 Explained like you're 14
想象你在玩一款超级复杂的游戏，里面有很多不同的角色和技能。以前，每个角色都用固定的技能组合，效果不一定最好。现在，有个聪明的机器人助手，它可以学习你的玩法，根据不同的情况调整技能组合，让你变得更厉害！这就像UniMixer，它能学会不同特征之间的最佳交互方式，不用每次都手动调节。它还可以变得更快、更省资源，就像你的机器人助手变得更聪明、更快了。未来，这种技术可以帮我们在各种复杂场景中做得更好，比如推荐内容、广告投放，甚至智能助手，都能变得更聪明、更贴心！
Abstract
In recent years, the scaling laws of recommendation models have attracted increasing attention, which govern the relationship between performance and parameters/FLOPs of recommenders. Currently, there are three mainstream architectures for achieving scaling in recommendation models, namely attention-based, TokenMixer-based, and factorization-machine-based methods, which exhibit fundamental differences in both design philosophy and architectural structure. In this paper, we propose a unified scaling architecture for recommendation systems, namely \textbf{UniMixer}, to improve scaling efficiency and establish a unified theoretical framework that unifies the mainstream scaling blocks. By transforming the rule-based TokenMixer to an equivalent parameterized structure, we construct a generalized parameterized feature mixing module that allows the token mixing patterns to be optimized and learned during model training. Meanwhile, the generalized parameterized token mixing removes the constraint in TokenMixer that requires the number of heads to be equal to the number of tokens. Furthermore, we establish a unified scaling module design framework for recommender systems, which bridges the connections among attention-based, TokenMixer-based, and factorization-machine-based methods. To further boost scaling ROI, a lightweight UniMixing module is designed, \textbf{UniMixing-Lite}, which further compresses the model parameters and computational cost while significantly improve the model performance. The scaling curves are shown in the following figure. Extensive offline and online experiments are conducted to verify the superior scaling abilities of \textbf{UniMixer}.