Scaling Law for Recommendation Models: Towards General-purpose User Representations
Proposed CLUE uses contrastive learning to scale user representations, outperforming task-specific models with significant transferability.
Key Findings
Methodology
CLUE employs a hierarchical Transformer architecture with Item and Service Transformers to encode user behavior logs into semantic-rich representations. It uses a multimodal contrastive loss, inspired by CLIP, to maximize similarity of user embeddings across different services. The training leverages massive datasets—over 50 billion behavior tokens from 11 million users—optimizing task-agnostic objectives. The model’s performance follows a power-law scaling with respect to compute, data, and model size, validated through extensive experiments. The architecture incorporates nonlinear projection layers to enhance embedding quality, enabling effective cross-domain transfer. Evaluation on multiple downstream recommendation tasks demonstrates superior performance over task-specific baselines, with online CTR improvements of 7%. The framework emphasizes the importance of scaling all factors—model size, data volume, sequence length, and batch size—for optimal results.
Key Results
- Pretraining error exhibits a power-law decay with increasing compute, confirming the scaling law in user modeling. Larger models and longer sequences consistently improve downstream task performance.
- CLUE outperforms models like DeepFM, BST, LightGCN, and UserBERT across benchmark datasets, with MRR gains exceeding 6%. Transferability is validated through cross-domain experiments, including company-level tests.
- Online A/B tests show a 7% CTR lift, especially for new and cold-start users, demonstrating strong practical benefits. The model maintains robustness across various tasks and data distributions.
Significance
This work bridges the gap between large-scale pretraining and user behavior modeling, establishing a theoretical and empirical foundation for scaling laws in recommendation systems. It demonstrates that universal user representations can be learned efficiently at scale, enabling better transferability, cold-start handling, and multi-task performance. The approach paves the way for more intelligent, scalable, and adaptable recommendation engines, impacting both academia and industry by reducing reliance on handcrafted features and domain-specific tuning.
Technical Contribution
The paper introduces a hierarchical Transformer-based architecture combined with a multimodal contrastive loss, inspired by CLIP, to learn universal user embeddings. It rigorously validates the power-law relationship between performance and compute, data, and model size, providing a scalable framework for future research. The natural language-based data transformation enables cross-service and cross-platform transfer, addressing ID dependency issues. The method significantly advances the state-of-the-art in large-scale user modeling, offering a flexible, task-agnostic solution with proven industrial relevance.
Novelty
This is the first comprehensive empirical validation of scaling laws in user representation learning within recommendation systems. It innovatively applies multimodal contrastive learning, leveraging natural language descriptions to unify heterogeneous data sources. Unlike prior ID-based embedding methods, CLUE emphasizes the importance of scale factors, demonstrating their combined impact on performance and transferability, thus opening new avenues for scalable, general-purpose user modeling.
Limitations
- High computational cost limits accessibility; training large models requires substantial resources, restricting deployment in resource-constrained environments.
- Dependence on natural language descriptions may introduce semantic biases or inaccuracies, especially in domain-specific contexts.
- In extremely cold-start scenarios with minimal data, the model’s performance may still lag behind specialized methods, necessitating further optimization.
Future Work
Future research will focus on model compression and efficiency improvements to reduce costs. Exploring multi-task and multi-modal training strategies can further enhance adaptability. Extending the framework to incorporate additional modalities like images or audio, and improving robustness in low-data regimes, will be key directions. Additionally, theoretical work on the limits of scaling laws in user modeling remains an open question.
AI Executive Summary
The rapid development of large-scale pretrained models such as BERT and GPT-3 has revolutionized natural language processing and computer vision, inspiring similar pursuits in recommendation systems. Traditional recommendation models rely heavily on ID embeddings, which are limited in their ability to generalize across domains and handle cold-start users. Recognizing this challenge, the authors introduce CLUE, a novel user representation framework based on contrastive learning, inspired by CLIP, that leverages massive behavioral datasets to learn universal, task-agnostic embeddings.
CLUE employs a hierarchical Transformer architecture, with separate modules encoding item descriptions and user behavior sequences into semantic-rich vectors. By maximizing the similarity of user embeddings across different services, CLUE achieves a unified, multi-modal user space. Extensive experiments on 50 billion behavior tokens from 11 million users reveal that the model’s performance follows a power-law scaling law with respect to compute, data, and model size. This validates the theoretical premise that larger models trained on more data with longer sequences can significantly improve downstream recommendation accuracy.
The empirical results demonstrate that CLUE surpasses task-specific baselines such as DeepFM, BST, and LightGCN across multiple benchmark and industrial datasets, with notable improvements in metrics like MRR and CTR. In online A/B tests, CLUE achieves a 7% CTR lift, especially benefiting new and cold-start users, highlighting its practical impact. The study underscores the importance of scaling all factors—model size, data volume, sequence length, and batch size—in tandem to unlock optimal performance.
Overall, this research establishes a foundational understanding of scaling laws in user representation learning, providing a scalable, flexible, and effective approach for industrial recommendation systems. Future directions include model compression, multi-modal extensions, and theoretical investigations into the limits of scaling, promising to further advance the field of intelligent user modeling and personalized recommendation.
Deep Dive
Limitations & Outlook
What gaps remain?
Abstract
Recent advancement of large-scale pretrained models such as BERT, GPT-3, CLIP, and Gopher, has shown astonishing achievements across various task domains. Unlike vision recognition and language models, studies on general-purpose user representation at scale still remain underexplored. Here we explore the possibility of general-purpose user representation learning by training a universal user encoder at large scales. We demonstrate that the scaling law is present in user representation learning areas, where the training error scales as a power-law with the amount of computation. Our Contrastive Learning User Encoder (CLUE), optimizes task-agnostic objectives, and the resulting user embeddings stretch our expectation of what is possible to do in various downstream tasks. CLUE also shows great transferability to other domains and companies, as performances on an online experiment shows significant improvements in Click-Through-Rate (CTR). Furthermore, we also investigate how the model performance is influenced by the scale factors, such as training data size, model capacity, sequence length, and batch size. Finally, we discuss the broader impacts of CLUE in general.