WHALE: A Scalable Unified Model for Recommendation with Wukong-HSTU Architecture
WHALE model integrates Wukong and HSTU architectures to enhance performance in large-scale recommendations.
Key Findings
Methodology
WHALE combines Wukong and HSTU architectures in a recursive layer structure for joint modeling of non-sequential and sequential features. Each layer includes a Wukong module, an HSTU module, and an attention-based fusion module, enabling high-order feature interactions to repeatedly extract fine-grained evidence from long user histories.
Key Results
- On large-scale industrial recommendation data, WHALE consistently improves performance in offline experiments and achieves positive gains online, albeit with a modest serving throughput trade-off.
- WHALE outperforms baselines using only HSTU or Wukong in various experimental settings.
- Ablation studies confirm the effectiveness of WHALE's modules, particularly the contribution of the attention-based fusion module.
Significance
WHALE demonstrates how non-sequential and sequential information can be scalably unified in industrial recommendation models, addressing the pain point of insufficient integration in current architectures, with significant implications for academia and industry.
Technical Contribution
WHALE achieves joint modeling of non-sequential and sequential features through a recursive layer structure and attention-based fusion module, offering new engineering possibilities and theoretical guarantees compared to existing SOTA methods.
Novelty
WHALE is the first to unify Wukong and HSTU, two representative architectures, within a single scalable recommendation model design.
Limitations
- WHALE may require significant computational resources, especially when handling ultra-long sequences.
- The complexity of the model increases deployment and maintenance challenges.
Future Work
Future research directions include optimizing WHALE's computational efficiency, exploring more application scenarios, and enhancing model interpretability.
AI Executive Summary
As recommendation systems scale up, existing architectures often model non-sequential and sequential features separately, leading to insufficient integration. The WHALE model combines Wukong and HSTU architectures to provide a scalable unified recommendation architecture that models both non-sequential and sequential features. Each WHALE layer includes a Wukong module, an HSTU module, and an attention-based fusion module, allowing high-order feature interactions to repeatedly extract fine-grained evidence from long user histories. On large-scale industrial recommendation data, WHALE consistently improves performance in offline experiments and achieves positive gains online, albeit with a modest serving throughput trade-off. The successful deployment of WHALE demonstrates its feasibility and practicality in industrial recommendation systems. Future research directions include optimizing computational efficiency, exploring more application scenarios, and enhancing model interpretability.
Deep Analysis
Background
Recommender systems are core components of modern social media platforms, responsible for ranking user-item interactions at scale. As recommendation workloads grow in scale and diversity, increasing model capacity has become crucial for improving quality. Recently, large recommender models have advanced in two directions: non-sequential feature interactions and user behavior history sequence modeling. Wukong and HSTU represent scalable architectures for these directions.
Core Problem
Existing recommendation architectures often model non-sequential and sequential features separately, leading to insufficient integration. This separation limits the model's ability to capture user intent and item semantics, especially in social media ranking, where a candidate item's relevance often depends on connecting the current candidate or context to specific events in the user's behavior history.
Innovation
The WHALE model combines Wukong and HSTU architectures to achieve joint modeling of non-sequential and sequential features. Each WHALE layer includes a Wukong module, an HSTU module, and an attention-based fusion module, allowing high-order feature interactions to repeatedly extract fine-grained evidence from long user histories. This design keeps both backbones active throughout the network and enables progressive Wukong-HSTU exchange.
Methodology
- �� The input layer maps user and item categorical features, numerical features, and user behavior sequences into embeddings.
- �� The Wukong module models high-order interactions of non-sequential features.
- �� The HSTU module models the temporal evolution of user interests.
- �� The fusion module uses Wukong output to query HSTU output, injecting the most relevant sequential signals into non-sequential interaction representation.
Experiments
Experiments use large-scale industrial recommendation data to evaluate WHALE's performance in offline and online settings. Baseline models include those using only HSTU or Wukong. Evaluation metrics include recommendation accuracy and serving throughput. Ablation studies validate the effectiveness of each module.
Results
WHALE consistently improves performance in offline experiments and achieves positive gains online. It outperforms baselines using only HSTU or Wukong. Ablation studies show the attention-based fusion module significantly contributes to performance improvement.
Applications
WHALE can be used in large-scale recommendation systems, particularly on social media platforms. Its design allows handling complex user behaviors and item features without sacrificing performance.
Limitations & Outlook
WHALE may require significant computational resources, especially when handling ultra-long sequences, increasing deployment and maintenance challenges. Future research can optimize computational efficiency and explore more application scenarios.
Plain Language Accessible to non-experts
Imagine you're in a library; WHALE is like a smart librarian. The Wukong module organizes book categories like titles, authors, and topics, which have no temporal order. The HSTU module is like a log of readers' borrowing history, focusing on books borrowed at different times and frequencies. The fusion module acts like the librarian recommending the most suitable books based on the reader's borrowing history. This way, WHALE provides more accurate recommendations in the system.
ELI14 Explained like you're 14
Imagine playing a game; WHALE is like a super assistant. The Wukong module is like the assistant organizing your gear and skills without time order. The HSTU module is like a log of your adventures, focusing on your performance in different levels. The fusion module is like the assistant suggesting the best gear and skills for the current level based on your adventure log. This way, WHALE helps you perform better in the game!
Glossary
Wukong
An architecture for modeling high-order interactions of non-sequential features.
Used for handling interactions among user, item, and context features.
HSTU
An architecture for modeling long user behavior sequences.
Used for capturing temporal evolution and long-range dependencies in user behavior.
Attention-based Fusion Module
A module in WHALE for cross-branch information exchange.
Allows non-sequential interaction representations to selectively extract behavioral evidence from the sequence branch.
Recursive Layer Structure
A design for progressive information exchange through multi-layer structures.
Used in WHALE for joint modeling of non-sequential and sequential features.
Ablation Study
An experimental method for evaluating the contribution of model components by removing them.
Used to validate the effectiveness of each module in WHALE.
Open Questions Unanswered questions from this research
- 1 How to handle longer user behavior sequences without increasing computational resources?
- 2 How to enhance the interpretability of the WHALE model?
Applications
Immediate Applications
Social Media Recommendation
WHALE can optimize content recommendations on social media platforms, enhancing user experience.
Long-term Vision
Personalized Advertising
WHALE can be used for personalized ad recommendations, increasing ad click-through and conversion rates.
Abstract
As scalability becomes increasingly important in recommendation modeling, recent architectures have advanced the modeling of two broad sources of ranking signals along separate paths: non-sequence features, including user, item, context, and cross features; and sequence features from user behavior histories. Wukong and HSTU have emerged as representative scalable backbones for these paths: Wukong scales high-order non-sequence feature-interaction modeling, while HSTU scales long user-behavior sequence modeling. Despite their complementary strengths, practical architectures that combine these two types of feature modeling remain underexplored. We present WHALE, a scalable unified recommendation architecture that jointly models non-sequence and sequence features on top of Wukong and HSTU. Each WHALE layer contains a Wukong module, an HSTU module, and an attention-based fusion module in which Wukong-derived interaction representations query HSTU-derived behavior representations. This design keeps both backbones active throughout the network and enables progressive Wukong-HSTU exchange, allowing high-order feature crosses to repeatedly retrieve fine-grained evidence from long user histories. To make WHALE practical for industrial deployment, we introduce customized Triton kernels and other model-systems co-design techniques to improve training and inference efficiency. On large-scale industrial recommendation data, WHALE achieves consistent gains in offline experiments. Additionally, it delivers positive online gains with a modest serving-throughput trade-off. The method has been deployed in production systems. Overall, WHALE provides a practical example of how these two sources of information can be scalably unified in an industrial recommendation model.