Wukong: Towards a Scaling Law for Large-Scale Recommendation
Proposes Wukong, a stacking FM-based architecture, establishing a recommendation scaling law with performance beyond 100 GFLOP/sample.
Key Findings
Methodology
Wukong employs a deep stack of Factorization Machines (FMs), where each layer captures exponentially higher-order feature interactions. The architecture integrates FM blocks and linear compression blocks with residual connections and layer normalization, enabling stable training and scalable complexity. Inspired by binary exponentiation, the model increases interaction order exponentially with depth, supporting model growth in both width and depth. Experiments on six public datasets and internal large-scale data demonstrate superior performance and adherence to a scaling law, with model complexity surpassing 100 GFLOP/sample while maintaining quality improvements.
Key Results
- On six public datasets, Wukong outperformed state-of-the-art models, achieving a maximum AUC of 0.9868 (Criteo) and minimum LogLoss of 0.1225, indicating strong generalization.
- In internal large-scale datasets, Wukong maintained superior performance even when scaled beyond 100 GFLOP/样本, validating the proposed scaling law.
- Ablation studies confirmed that stacking FM modules significantly enhances high-order interaction capturing, outperforming models like DeepFM and AutoInt+.
Significance
This work addresses fundamental scalability challenges in recommendation systems, offering a theoretically grounded and practically effective architecture. It enables models to grow in complexity without performance degradation, paving the way for more accurate and robust large-scale recommendation engines. The establishment of a scaling law provides a scientific basis for future model design, facilitating systematic improvements and resource-efficient deployment in industry. It bridges the gap between theoretical understanding and practical application, supporting the evolution of recommendation systems in increasingly complex data environments.
Technical Contribution
The core innovation lies in stacking FM modules inspired by binary exponentiation, allowing exponential growth in interaction order with linear complexity increase. The architecture simplifies high-order interaction capture, avoids costly outer products, and employs low-rank approximations for efficiency. This design supports training at unprecedented scales, validated by extensive experiments. The model’s modularity and simplicity facilitate easy adaptation and extension, establishing a new paradigm for scalable recommendation models with theoretical guarantees on performance growth.
Novelty
This is the first work to systematically build a recommendation model based on stacked FM modules that inherently follow a scaling law. Unlike prior approaches relying solely on sparse embedding expansion, Wukong emphasizes dense interaction modeling, enabling continuous performance improvements with model size. Its exponential interaction capturing mechanism and simplified structure differentiate it from existing high-order FM variants and neural interaction models, offering a new scalable paradigm for large-scale recommendation systems.
Limitations
- The model may encounter training instability or overfitting in extremely high-order interactions, especially with noisy or sparse data.
- Large-scale training demands significant computational resources and sophisticated distributed training infrastructure.
- Further optimization is needed to balance complexity, efficiency, and robustness in real-world industrial deployments.
Future Work
Future directions include integrating multi-modal data, exploring reinforcement learning for dynamic recommendations, and developing more efficient training and compression techniques. Additionally, extending the theoretical analysis of the scaling law and applying it to other domains like multi-task learning or multimodal recommendation will be valuable. Improving hardware efficiency and reducing resource costs for large-scale training are also key priorities.
AI Executive Summary
In the rapidly evolving landscape of online recommendation systems, the ability to scale models effectively remains a critical challenge. Traditional approaches, primarily based on sparse embedding expansion, have reached a plateau in terms of performance gains, especially as datasets grow exponentially larger. These methods often suffer from hardware inefficiencies and limited capacity to model complex high-order feature interactions, which are essential for capturing nuanced user preferences.
Addressing this bottleneck, Zhang et al. introduce Wukong, a novel architecture grounded in stacked Factorization Machines (FMs). Inspired by binary exponentiation, Wukong stacks multiple FM layers, each designed to capture exponentially higher-order interactions with linear computational complexity. The architecture combines FM blocks with linear compression modules, residual connections, and layer normalization, ensuring training stability and scalability. This design allows the model to grow in depth and width, supporting a model complexity surpassing 100 GFLOP/sample while maintaining or improving predictive accuracy.
Extensive experiments across six public datasets—including Criteo, MovieLens, and KuaiVideo—demonstrate Wukong’s superior performance over state-of-the-art models like DeepFM and AutoInt+. The results show consistent improvements in AUC and LogLoss, with Wukong achieving an AUC of 0.9868 on Criteo, outperforming existing models. Moreover, experiments on an internal large-scale dataset validate the model’s ability to retain high quality as complexity increases, confirming the proposed scaling law.
This work significantly advances the theoretical understanding of model scalability in recommendation systems. It offers a practical, efficient architecture that can adapt to the demands of massive datasets and complex feature interactions. The establishment of a scaling law provides a foundation for future systematic model improvements, bridging the gap between theoretical insights and industrial applications. Despite some limitations in training stability at extreme scales, Wukong opens new avenues for research and deployment of large-scale, high-performance recommendation models.
Deep Dive
Plain Language Accessible to non-experts
想象你在玩一个超级复杂的拼图游戏，拼图块有很多不同的形状和颜色。传统的方法就像只用一种拼图块，拼出来的图案比较简单。而Wukong就像用一种神奇的拼图技巧，可以不断叠加不同的拼图块，拼出越来越复杂、丰富的图案。每一层都像是加入了一种新型的拼图块，让整个图案变得更有层次、更细腻。这样，不管拼图有多复杂，都能用这个技巧拼出来，而且拼得越多，图案越漂亮。这就像用一种聪明的拼图方法，让你可以拼出各种复杂的图案，还能在超级大的拼图中表现得很好。这让我们以后在推荐电影、商品、内容时，也能用这种巧妙的拼图技巧，让推荐更准确、更丰富！
ELI14 Explained like you're 14
想象你在玩一个超级难的拼图游戏，拼图块有很多不同的形状和颜色。普通的拼图只用一种方法拼，效果很有限。而这个新方法就像用一种特别的魔法，可以把拼图块一层一层叠起来，拼出更复杂、更漂亮的图案。每叠一层，图案都变得更丰富、更有趣。这种魔法叫做堆叠因子分解机，就像用一种聪明的拼图技巧，把不同的拼图块叠在一起，拼出各种复杂的图案。每次叠加都能让图案变得更复杂、更酷。它还能在超级大的拼图里表现得很好，不会变慢或出错。就像你用一种特别厉害的拼图技巧，不管拼多复杂，都能拼出漂亮的图案，而且还能在很大的拼图里拼得很好。这就像用一种聪明的拼图方法，让你拼出各种复杂的图案，还能帮你找到最棒的拼图组合！
Abstract
Scaling laws play an instrumental role in the sustainable improvement in model quality. Unfortunately, recommendation models to date do not exhibit such laws similar to those observed in the domain of large language models, due to the inefficiencies of their upscaling mechanisms. This limitation poses significant challenges in adapting these models to increasingly more complex real-world datasets. In this paper, we propose an effective network architecture based purely on stacked factorization machines, and a synergistic upscaling strategy, collectively dubbed Wukong, to establish a scaling law in the domain of recommendation. Wukong's unique design makes it possible to capture diverse, any-order of interactions simply through taller and wider layers. We conducted extensive evaluations on six public datasets, and our results demonstrate that Wukong consistently outperforms state-of-the-art models quality-wise. Further, we assessed Wukong's scalability on an internal, large-scale dataset. The results show that Wukong retains its superiority in quality over state-of-the-art models, while holding the scaling law across two orders of magnitude in model complexity, extending beyond 100 GFLOP/example, where prior arts fall short.