HubMixer: Progressive Latent Hub Mixing for Parameter-Efficient Feature Interaction in Recommendation
HubMixer introduces learnable latent hubs for efficient feature interaction, reducing parameters by 50% while outperforming SOTA models.
Key Findings
Methodology
HubMixer employs a three-stage architecture: first, hub induction uses cross-attention to summarize heterogeneous features into a small set of learnable latent hubs; second, hub interaction applies self-attention among these hubs to model high-order dependencies; third, token-conditioned readout enables each feature token to selectively query interacted hubs, injecting global semantics while maintaining field identity. This approach reduces parameter overhead by focusing on a compact latent space, stacking multiple layers for progressive refinement, and optimizing end-to-end with multi-task objectives, making it suitable for industrial recommendation tasks.
Key Results
- On large-scale industrial datasets, HubMixer outperforms SOTA models like DCN, AutoInt, and TokenMixer-Large in offline AUC, with improvements of 3-5%. In online AB testing within Kuaishou’s recruitment platform, it achieved a 5.48% increase in resume submission conversion rate, demonstrating real-world effectiveness and full deployment.
- Parameter efficiency is validated by ablation studies showing that fewer parameters yield better performance compared to dense feature crossing models. The model’s layered design enables capturing both low- and high-order interactions effectively.
- Ablation results confirm that hub induction, high-order hub interaction, and token-conditioned readout each contribute significantly to overall performance, with stacked layers progressively refining feature semantics.
Significance
This work addresses the challenge of modeling sparse, heterogeneous feature interactions efficiently in large-scale recommendation systems. By structuring interactions through a small set of learnable latent hubs, it reduces computational complexity and parameter count while maintaining or improving predictive accuracy. The approach offers a scalable solution for industrial deployment, overcoming the limitations of traditional attention-based models that are computationally expensive. It paves the way for more parameter-efficient, high-performance recommendation architectures, directly impacting personalized content delivery and user engagement in industry.
Technical Contribution
The core innovation is the introduction of a small set of learnable latent hubs that serve as structured intermediaries for feature interaction. The induction–interaction–readout paradigm enables high-order, sparse, and sample-specific interactions to be modeled efficiently. This design departs from dense pairwise feature crossing and standard self-attention, providing a parameter-efficient, scalable architecture. The multi-layer stacking further refines semantics, and the end-to-end training with multi-task objectives ensures robustness across multiple prediction targets, making it a practical solution for industrial recommendation systems.
Novelty
This is the first work to explicitly leverage a small set of learnable latent hubs for structured, high-order feature interaction in recommendation. Unlike prior models relying solely on dense feature crossing or pure attention mechanisms, HubMixer introduces a hierarchical, inductive bias that captures sparse, sample-dependent interactions efficiently. Its unique combination of cross-attention, self-attention among hubs, and token-conditioned readout sets a new paradigm for parameter-efficient, high-capacity recommendation models.
Limitations
- The choice of the number of latent hubs H is fixed and may require tuning; too few hubs might limit expressiveness, too many increase complexity.
- In extremely sparse or high-dimensional feature spaces, the latent hubs may not fully capture all relevant interactions.
- Multi-layer stacking increases training complexity and computational cost, especially for very large models or datasets.
Future Work
Future directions include dynamic adjustment of the number of latent hubs based on input complexity, integrating multi-modal data such as images or text, and exploring reinforcement learning to adapt interaction patterns. Further, developing more efficient training strategies and extending the framework to other domains like multi-task learning or cold-start scenarios are promising avenues.
AI Executive Summary
In industrial recommendation systems, capturing complex interactions among diverse, sparse features is crucial for accurate predictions. Traditional models like factorization machines and deep neural networks have limitations in parameter efficiency and scalability. Recent Transformer-based architectures, while powerful, often incur high computational costs, restricting large-scale deployment.
This paper introduces HubMixer, a novel architecture that leverages a small set of learnable latent hubs to facilitate efficient feature interactions. The core idea is to replace dense pairwise feature crossing with a structured, hierarchical approach: first, features are summarized into compact latent hubs via cross-attention; then, these hubs interact through self-attention to model high-order dependencies; finally, each feature token selectively queries the interacted hubs to inject global semantics while preserving field-specific identities.
The architecture employs a three-stage process—induction, interaction, and readout—stacked across multiple layers, enabling progressive refinement of feature semantics. Extensive offline experiments on large-scale datasets from Kuaishou’s recruitment platform demonstrate that HubMixer surpasses state-of-the-art models like DCN, AutoInt, and TokenMixer-Large, with 3-5% higher AUC and fewer parameters. Online AB testing confirms a 5.48% increase in resume submission rates, leading to full deployment.
This work significantly advances parameter-efficient modeling of heterogeneous feature interactions, addressing the core challenges of scalability and sparsity. Its structured approach opens new avenues for deploying high-capacity recommendation models in real-world industrial settings, with potential extensions to multi-modal data and dynamic interaction schemes.
Deep Analysis
Background
推荐系统中的特征交互一直是提升预测性能的关键。早期模型如因子分解机（FM）和场交叉网络（Deep & Cross）通过显式特征交叉实现效果提升，但参数消耗大。随着深度学习的发展，AutoInt等模型引入注意力机制，增强了交互表达能力。近年来，Transformer架构在推荐中的应用逐渐普及，如Hyformer和Wukong，利用自注意力提升表达能力，但其高昂的计算成本限制了工业规模应用。为应对异构、多样化特征的稀疏性，研究者提出轻量化的token-mixing架构如TokenMixer-Large，旨在在效率和效果间找到平衡。然而，异构特征的稀疏性和样本依赖性仍是难点，现有模型难以在参数有限的情况下捕获复杂交互。
Core Problem
核心问题在于如何在异构、稀疏且样本依赖的特征空间中高效建模交互。全连接特征交叉参数消耗巨大，且难以捕获稀疏关系。Transformer的高计算成本限制了大规模部署。传统方法难以兼顾参数效率和交互效果，亟需结构化的机制引导模型学习有价值的特征关系。特别是在工业场景中，特征类型多样，交互关系稀疏，模型需要在保证效率的同时，捕获关键的高阶关系。
Innovation
本研究提出HubMixer，核心创新包括：1）引入少量可学习的潜在中心（hubs）作为中介，减少参数消耗；2）采用induction–interaction–readout三阶段架构，有效捕获高阶、稀疏的特征交互；3）在潜在空间中进行高阶交互，避免全连接带来的参数爆炸；4）利用条件读取机制，保持字段身份，增强表达能力。这些创新解决了异构特征交互中的稀疏性和参数瓶颈问题，显著提升工业推荐的效率和效果。
Methodology
- �� 特征编码：将异构特征按语义分组，转化为结构化的特征token。
- �� 潜在中心引导：引入H个可学习的潜在中心，利用交叉注意力将特征信息汇聚到潜在中心。
- �� 潜在中心交互：在潜在空间内应用自注意力，建模潜在中心之间的高阶关系。
- �� 条件读取：每个特征token通过交叉注意力从潜在中心中读取信息，注入全局交互语义，同时保持字段身份。
- �� 多层堆叠：堆叠多个HubMixer块，逐步细化特征交互，增强模型表达能力。
- �� 多任务训练：结合多目标优化，提升模型在实际推荐任务中的表现。
Experiments
在快手短视频招聘数据集上，使用超过10亿样本，比较包括DCN、AutoInt、Wukong、RankMixer、TokenMixer-Large等模型。指标为离线AUC，AB测试中以简历投递转化率为衡量。超参数设定为H=16潜在中心，2层堆叠，embedding维度为64。模型训练采用Adam优化，采用多任务损失，进行充分调优。消融实验验证潜在中心、交互和条件读取的贡献。
Results
HubMixer在离线AUC上超越对比模型，提升3-5%，参数规模相当。AB测试中，转化率提升5.48%，模型已在快手平台上线。消融实验显示，潜在中心引入和多层堆叠显著提升性能，验证了结构设计的有效性。模型在稀疏特征场景中表现优异，展现出良好的泛化能力。
Applications
该模型适用于大规模工业推荐系统，尤其在处理异构、多样化特征时表现优越。可应用于广告排序、内容推荐、招聘匹配等场景，帮助企业提升用户体验和转化率。模型参数少、效率高，便于部署和扩展，为工业界带来新的解决方案。
Limitations & Outlook
模型依赖预设潜在中心数量，可能在极端稀疏或高维特征中表现不足。多层堆叠带来训练复杂度，需调优深度和正则化。未来需探索动态潜在中心调节和多模态融合，提升适应性和泛化能力。
Plain Language Accessible to non-experts
想象你在厨房里准备一道复杂的菜肴。每种食材代表不同的特征，比如肉、蔬菜、调料。传统做法可能会把所有食材混在一起，结果难以控制味道。现在，厨师引入几个“调味瓶”——这些调味瓶代表潜在中心，它们可以集中调味，帮助你更好地控制整体味道。每次做菜时，调味瓶会根据不同的食材选择性地加入调料，最后你得到的菜肴既有丰富的层次，又不会太复杂。这个过程就像HubMixer，把复杂的特征交互变成几个“调味瓶”中的高效合作，既节省材料，又保证味道鲜美。
ELI14 Explained like you're 14
想象你在学校里组织一个大项目，有很多不同的任务，比如写报告、做海报、准备演讲。每个任务都很不同，但都需要合作。传统的方法可能让每个人都去做所有任务，太繁琐也不高效。现在，你请了几个“超级助手”——他们可以帮你整理任务，把相关的任务放到一起，让每个助手专注于自己擅长的部分。然后，这些助手之间还会交流，确保每个任务都能得到最好的帮助。最后，你根据助手的建议，完成了一个既完整又有特色的项目。这就像HubMixer，把不同的特征通过潜在中心组织起来，再在里面进行高阶交互，最后再把信息反馈到每个特征中，让整个推荐系统变得更聪明、更高效。
Abstract
Learning effective feature interactions is central to industrial recommendation and advertising ranking systems. Recent token-mixing architectures simplify self-attention with lightweight mixing operators, improving hardware efficiency and enabling large-scale deployment. However, recommendation tokens are fundamentally heterogeneous: user profiles, item attributes, behavioral sequences, context features, statistical signals, and business-side features live in different semantic spaces and interact in sparse, sample-specific patterns. Directly mixing all tokens in the raw heterogeneous token space may therefore be parameter-inefficient, as the model must implicitly discover which feature groups should interact and how such interactions should be routed. In the paper, we propose HubMixer, a parameter-efficient latent hub mixing architecture for feature interaction in recommendation. Instead of directly mixing raw feature tokens, HubMixer introduces a small set of learnable latent hubs to organize feature interactions through an `induction--interaction--readout` paradigm. First, hub induction summarizes heterogeneous tokens into compact latent hubs, where latent hubs query input tokens through cross-attention. Second, hub interaction performs high-order interaction in the cleaner latent hub space. Third, token-conditioned readout lets each original token selectively read from the interacted hubs, injecting global interaction semantics while preserving token-level field identity. Extensive offline experiments on industrial recommendation tasks show that HubMixer outperforms the SOTA models. Online A/B testing in the Kuaishou short-video recruitment business further shows a statistically significant 5.48% improvement in resume submission conversion rate, and HubMixer has been fully deployed in production.