Understanding Scaling Laws for Recommendation Models
This study models recommendation system scaling laws, revealing performance follows a power law plus constant, with data size being the dominant factor.
Key Findings
Methodology
Using large-scale experiments, the study investigates how different resource dimensions—model parameters, data volume, and compute—affect recommendation model performance. It employs the Deep Learning Recommendation Model (DLRM) architecture, varying embedding table sizes and MLP layers, measuring normalized cross-entropy loss. The performance-resource relationship is modeled via a power law plus constant, fitting parameters like β and γ. Multiple scaling schemes, including vertical/horizontal embedding and layer scaling, are compared across datasets such as Criteo and Avito. The analysis includes training vs. test loss, efficiency metrics, and saturation points, providing a comprehensive understanding of scaling behaviors.
Key Results
- Model performance (test loss) scales with resources according to a power law plus constant (αx−β + γ), with β between 0.09 and 0.15, and γ indicating an irreducible error (~0.98). Data size has the strongest impact, while parameter size shows diminishing returns. Vertical embedding scaling outperforms horizontal, and MLP scaling offers better compute efficiency. Overfitting is evident as training loss exhibits a higher power-law exponent (-0.20) compared to test loss (-0.12).
- Across scaling schemes, data scaling maintains similar efficiency (β ≈ 0.10-0.12), whereas parameter scaling efficiency is low in industry-scale models due to saturation. Compute scaling shows consistent efficiency (~0.12-0.15). The results suggest industry models have reached a saturation point for parameter growth, emphasizing data and compute as primary growth avenues.
- Performance gains are more significant when increasing data and compute rather than parameters. For a fixed compute budget, larger models perform worse; smaller models with more data perform better. The study highlights the importance of resource allocation strategies, with implications for hardware investment and system design.
Significance
This work provides a quantitative framework for understanding how recommendation models scale, crucial for designing efficient large-scale systems. By revealing that data and compute are more effective than parameter expansion at industry scale, it guides resource allocation, hardware planning, and future architecture development. The findings address industry challenges such as cost escalation, overfitting, and diminishing returns, offering a pathway for sustainable growth in personalized AI services. The insights also inform long-term infrastructure planning, enabling better prediction of hardware needs and system bottlenecks, thus fostering more cost-effective and scalable recommendation solutions.
Technical Contribution
The paper introduces a formal power law plus constant model to describe recommendation model performance scaling, validated across multiple resource dimensions and scaling schemes. It systematically compares the efficiency of embedding and layer scaling, revealing industry saturation points and emphasizing the dominance of data and compute over parameters. The methodology combines empirical large-scale experiments with theoretical modeling, providing a new quantitative tool for resource-performance analysis. This approach advances the understanding of scaling behaviors, enabling more precise optimization of large-scale recommendation architectures.
Novelty
This is the first comprehensive analysis applying power law plus constant models to recommendation systems, especially emphasizing the role of embedding parameters. Unlike prior work focused on language models, this study demonstrates industry models have entered a saturation regime for parameters, shifting focus to data and compute. It offers a detailed comparison of multiple scaling schemes, providing actionable insights for industry practitioners. The integration of empirical data with theoretical modeling marks a significant step forward in understanding recommendation system scalability.
Limitations
- The analysis is primarily based on CTR prediction tasks and the specific architecture of DLRM, which may limit generalization to other recommendation scenarios or models with different architectures.
- The datasets used (e.g., Criteo, Avito) are industry-specific, and results might vary with different data distributions or system configurations.
- The study does not incorporate multi-task or multi-modal recommendation settings, which could exhibit different scaling behaviors. Future work should explore these dimensions.
Future Work
Future research will focus on developing new architectures that can surpass current saturation points, possibly through hybrid models or multi-task learning. Investigations into multi-modal recommendation scenarios and their scaling laws are planned to verify universality. Additionally, integrating hardware-aware optimization and energy efficiency considerations will be crucial. Exploring adaptive resource allocation strategies based on real-time performance metrics could further enhance scalability and cost-effectiveness. Long-term, the goal is to establish a comprehensive theoretical framework guiding the sustainable growth of AI-driven recommendation systems.
AI Executive Summary
The rapid growth of deep learning in recommendation systems has led to models with hundreds of billions to trillions of parameters, significantly boosting personalization quality. However, this expansion comes with escalating hardware costs and system complexity. Despite these challenges, industry-scale models continue to grow, driven by the need for better accuracy and user experience.
This study systematically investigates the fundamental scaling laws governing recommendation models, focusing on the relationship between model performance and resource investment. Using the DLRM architecture and extensive experiments on datasets like Criteo and Avito, the authors reveal that model performance follows a power law plus a constant, with the performance improvement diminishing as resources increase. Notably, data size emerges as the dominant factor influencing accuracy, whereas parameter scaling shows signs of saturation, indicating a limit to the benefits of simply enlarging models.
The analysis compares various resource scaling schemes, including vertical and horizontal embedding expansion, as well as layer scaling strategies. Results show that vertical embedding scaling outperforms horizontal, and MLP layer scaling offers better compute efficiency than embedding expansion. The study emphasizes that industry models have largely reached a saturation point for parameter growth, making data and compute the primary avenues for future improvements.
These insights have profound implications for system design and resource planning. They suggest that beyond a certain size, increasing parameters yields minimal gains, and efforts should focus on optimizing data quality and computational resources. The findings guide hardware investments, architectural choices, and long-term infrastructure development, ensuring sustainable growth in AI-powered recommendation services. Future directions include exploring new architectures, multi-task learning, and hardware-aware optimization to push beyond current limits and achieve more efficient, scalable recommendation systems.
Deep Dive
Abstract
Scale has been a major driving force in improving machine learning performance, and understanding scaling laws is essential for strategic planning for a sustainable model quality performance growth, long-term resource planning and developing efficient system infrastructures to support large-scale models. In this paper, we study empirical scaling laws for DLRM style recommendation models, in particular Click-Through Rate (CTR). We observe that model quality scales with power law plus constant in model size, data size and amount of compute used for training. We characterize scaling efficiency along three different resource dimensions, namely data, parameters and compute by comparing the different scaling schemes along these axes. We show that parameter scaling is out of steam for the model architecture under study, and until a higher-performing model architecture emerges, data scaling is the path forward. The key research questions addressed by this study include: Does a recommendation model scale sustainably as predicted by the scaling laws? Or are we far off from the scaling law predictions? What are the limits of scaling? What are the implications of the scaling laws on long-term hardware/system development?