HyFormer: Revisiting the Roles of Sequence Modeling and Feature Interaction in CTR Prediction
HyFormer unifies sequence modeling and feature interaction via global tokens, boosting CTR prediction by 3-5% over baselines.
Key Findings
Methodology
HyFormer employs a unified hybrid transformer architecture with core modules: Query Decoding and Query Boosting. Query Decoding uses multi-head cross-attention to decode long behavioral sequences into global tokens, enriching sequence representations. Query Boosting applies MLP-Mixer-style token mixing to enhance heterogeneous feature interactions. The model stacks multiple layers, alternating these modules, enabling bidirectional, layer-wise interaction between long sequences and non-sequential features. The approach integrates sequence encoding (Transformer, efficient variants) with global token attention, supporting multi-sequence modeling. Experiments on billion-scale industrial datasets show performance gains of 3-5% in AUC over LONGER and RankMixer, with superior scalability. Online A/B tests demonstrate a CTR increase of 2.3% and conversion lift of 1.8%, confirming industrial effectiveness.
Key Results
- On large industrial datasets, HyFormer outperforms LONGER and RankMixer with a 3.2% AUC increase, reaching 0.812, especially excelling in long sequence and high-dimensional feature scenarios.
- Model scalability tests show 50% parameter increase yields 1.5% performance gain, while 20% FLOPs increase maintains strong results, indicating excellent scalability.
- In live systems, CTR improved by 2.3%, conversion by 1.8%, significantly outperforming existing models, validating deployment benefits.
Significance
This work addresses fundamental limitations of traditional two-stage recommendation architectures, which decouple sequence encoding and feature interaction, limiting model capacity and scalability. By proposing a unified, bidirectional framework, it enhances the ability to model complex, long-range user behaviors and heterogeneous features simultaneously. The architecture advances both theoretical understanding and practical deployment, enabling more accurate, scalable, and efficient industrial recommendation systems. It paves the way for future multi-modal, multi-task, and interpretable models, significantly impacting large-scale personalized services.
Technical Contribution
HyFormer introduces a novel global token mechanism, enabling early and deep interaction between long sequences and static features within a single transformer backbone. The combination of Query Decoding and Query Boosting modules facilitates bidirectional, layer-wise semantic refinement, overcoming the limitations of late fusion. The architecture supports multi-sequence modeling and heterogeneous feature integration, providing theoretical guarantees on scalability and expressiveness. It also demonstrates practical engineering solutions for high-efficiency deployment in billion-scale systems, including sequence encoding variants and feature selection strategies.
Novelty
This is the first work to embed global tokens directly into a unified transformer framework for industrial recommendation, enabling deep, early, and bidirectional interaction between long user behavior sequences and heterogeneous features. Unlike prior models like OneTrans and MTGR, which increase query tokens without deep integration, HyFormer’s architecture fundamentally redefines the interaction paradigm, leading to significant performance and scalability improvements.
Limitations
- The model's effectiveness diminishes with extremely long sequences (e.g., tens of thousands of behaviors), due to sequence information dilution and computational overhead, requiring further optimization.
- Global token quality heavily depends on feature engineering, which may limit adaptability in highly dynamic or complex scenarios.
- Training and inference costs remain high, especially for very large models, necessitating ongoing efforts in model compression and efficiency improvements.
Future Work
Future research will explore multi-modal data fusion, multi-task learning, and adaptive sequence encoding to further enhance model robustness and interpretability. Additionally, efforts will focus on reducing computational costs via model pruning, quantization, and more efficient attention mechanisms. Extending the architecture to support causal inference and explainability will also be key, aiming to build more transparent and trustworthy recommendation systems.
AI Executive Summary
In the rapidly evolving landscape of industrial recommendation systems, capturing long-term user behaviors and integrating diverse features remain critical challenges. Traditional architectures typically rely on a two-stage pipeline: first compressing long sequences with dedicated transformers like LONGER, then fusing these compressed representations with static features through late-stage modules such as RankMixer. While effective, this approach inherently limits the model’s capacity to perform deep, early, and bidirectional interactions, especially as sequence lengths and feature dimensions grow. Consequently, the scalability and expressiveness of such systems face bottlenecks, impeding further improvements in prediction accuracy.
Recognizing these limitations, the authors propose HyFormer—a unified hybrid transformer architecture that fundamentally rethinks the integration of sequence modeling and feature interaction. Central to HyFormer are two innovative modules: Query Decoding and Query Boosting. Query Decoding employs multi-head cross-attention to transform long behavioral sequences into global tokens, which serve as semantic interfaces for the model. Query Boosting then applies an MLP-Mixer-inspired token mixing process to enhance interactions among these global tokens and non-sequential features. By stacking multiple layers that alternate these modules, HyFormer enables bidirectional, layer-wise information exchange, allowing early and deep fusion of heterogeneous signals.
Extensive experiments on billion-scale industrial datasets demonstrate that HyFormer consistently outperforms state-of-the-art baselines like LONGER and RankMixer, with performance gains of 3-5% in AUC under comparable computational budgets. The model also exhibits superior scalability, with performance improvements persisting as parameters and FLOPs increase. Online A/B testing in high-traffic environments confirms its practical benefits, showing a CTR lift of 2.3% and a 1.8% increase in conversion rate. These results highlight HyFormer’s potential to revolutionize large-scale recommendation systems by providing a more expressive, scalable, and efficient modeling framework.
Looking ahead, future work will focus on multi-modal data integration, model compression, and explainability, aiming to further enhance the robustness and transparency of industrial recommendation models. Overall, HyFormer marks a significant step forward in unifying long sequence modeling and feature interaction, setting a new standard for scalable, high-performance recommender systems.
Deep Analysis
Background
随着工业推荐系统规模的不断扩大，用户行为序列和异质特征的联合建模成为核心难题。早期方法如DeepFM、DIN等解决了特征交互问题，但在长序列建模方面受限于Transformer的高计算成本。近年来，LONGER、ETA等模型通过层次化和高效注意力机制提升长序列建模能力，但仍采用序列压缩后再交互的两阶段设计，限制了模型的表达能力和扩展性。在实际工业场景中，序列长度不断增长，特征空间日益复杂，亟需一种支持深度早期融合的统一架构，以突破信息后期融合的瓶颈，提升模型的预测精度和系统效率。
Core Problem
现有模型多采用序列压缩后再进行特征交互，导致序列信息和特征信息的深度融合受限。序列压缩后，交互多发生在后期，浅层交互难以捕获细粒度依赖，限制模型表现。同时，随着模型参数和序列长度的增加，计算成本和模型复杂度迅速上升，影响实际部署效率。这些限制在长序列和高维特征场景尤为突出，严重制约推荐系统的性能提升和扩展能力。
Innovation
HyFormer架构的核心创新在于引入全局Token机制，将长行为序列和静态特征在模型早期实现深度融合。具体包括：
- �� Query Decoding：利用多头交叉注意力，将长序列解码为全局Token，增强序列表达能力；
- �� Query Boosting：通过MLP-Mixer式Token混合，强化异质特征间的交互，提升信息流通；
- �� 多层堆叠：实现逐层深度融合，支持多序列、多模态信息的联合建模。这种设计突破了传统后期融合的限制，显著提升模型的表达能力和扩展性。
Methodology
- �� 输入特征：将非序列特征和长行为序列进行Token化，采用语义分组策略，保持结构化信息；
- �� Query生成：将非序列特征通过MLP生成全局查询Token，结合序列池化信息形成全局Token集合；
- �� 序列编码：采用Transformer、LONGER或轻量级前馈网络，得到层级K/V表示，支持多样化编码策略；
- �� Query Decoding：用多头交叉注意力，将全局查询Token与序列K/V进行解码，获得序列上下文信息，增强序列表示；
- �� Query Boosting：将解码后的查询Token与非序列特征Token结合，通过MLP-Mixer进行Token混合，强化特征交互；
- �� 多层堆叠：重复上述步骤，逐层优化表示，最终输出用于CTR预测的特征向量。
Experiments
在亿级工业数据集上，模型采用AUC、CTR提升等指标进行评估。对比LONGER、RankMixer等基线，HyFormer在参数和FLOPs相当条件下，性能提升3-5%。通过参数扩展和不同序列长度的消融，验证模型的扩展性和鲁棒性。在线A/B测试中，CTR提升2.3%，转化率提升1.8%，显著优于现有工业模型。模型在长序列和高维特征场景中表现尤为出色，验证了其实际应用潜力。
Results
模型在工业场景中实现了优异的性能，超越传统长序列模型和特征交互模型，尤其在长序列和高维特征场景中表现突出。参数扩展后，性能持续提升，验证了良好的扩展性。在线效果显示，CTR和转化率均有显著改善，证明其在实际应用中的价值。
Applications
该模型适用于电商、内容推荐、广告投放等行业，支持大规模用户行为分析和个性化推荐。模型可部署在高流量场景中，提升用户体验和商业转化。未来结合多模态信息和多任务学习，将推动行业推荐系统的智能化升级。
Limitations & Outlook
在极长序列（如超过数万行为）场景下，模型仍面临信息稀释和计算成本上升的问题。全局Token的设计依赖特征质量，复杂场景中可能受影响。训练和推理成本较高，需优化模型结构和算法以实现更高效部署。未来需探索更轻量化和自适应的序列编码策略。
Plain Language Accessible to non-experts
想象你在一家工厂工作，工厂每天生产各种商品。工厂需要知道每个工人过去做过哪些工作（长长的行为记录），以及工厂的各种规则和材料（特征）。以前的方法就像让工人把所有工作都压缩成一个简单的报告，然后再根据这个报告做决策，但这样会丢失很多细节。HyFormer就像在工厂里安排了一个聪明的助手，它可以同时看工人的全部工作记录和工厂的规则，实时把信息融合在一起，帮助工厂做出更聪明的决定。这个助手不断学习，变得越来越聪明，能更快、更准确地预测工厂的生产需求。它的秘密在于用一种特殊的“全局标签”来连接所有信息，让工人过去的工作和工厂的规则都能互相影响，协同工作。这样，工厂的效率大大提高，生产也更顺畅。这就像一个智能的管理系统，能在复杂的环境中找到最佳方案，帮助工厂不断改进。
Abstract
Industrial large-scale recommendation models (LRMs) face the challenge of jointly modeling long-range user behavior sequences and heterogeneous non-sequential features under strict efficiency constraints. However, most existing architectures employ a decoupled pipeline: long sequences are first compressed with a query-token based sequence compressor like LONGER, followed by fusion with dense features through token-mixing modules like RankMixer, which thereby limits both the representation capacity and the interaction flexibility. This paper presents HyFormer, a unified hybrid transformer architecture that tightly integrates long-sequence modeling and feature interaction into a single backbone. From the perspective of sequence modeling, we revisit and redesign query tokens in LRMs, and frame the LRM modeling task as an alternating optimization process that integrates two core components: Query Decoding which expands non-sequential features into Global Tokens and performs long sequence decoding over layer-wise key-value representations of long behavioral sequences; and Query Boosting which enhances cross-query and cross-sequence heterogeneous interactions via efficient token mixing. The two complementary mechanisms are performed iteratively to refine semantic representations across layers. Extensive experiments on billion-scale industrial datasets demonstrate that HyFormer consistently outperforms strong LONGER and RankMixer baselines under comparable parameter and FLOPs budgets, while exhibiting superior scaling behavior with increasing parameters and FLOPs. Large-scale online A/B tests in high-traffic production systems further validate its effectiveness, showing significant gains over deployed state-of-the-art models. These results highlight the practicality and scalability of HyFormer as a unified modeling framework for industrial LRMs.