Scaling Laws for Online Advertisement Retrieval
Proposes offline R/R* metric and scaling law for online ad retrieval models, enabling cost-effective performance prediction.
Key Findings
Methodology
This paper introduces a lightweight paradigm leveraging the novel offline metric R/R* and an offline simulation algorithm to identify the scaling laws of retrieval models in online advertising. Theoretically, it proves that under mild assumptions, the correlation between R/R* and online revenue approaches 1 asymptotically. Empirically, extensive experiments across Transformer, MLP, and DSSM architectures validate the existence of broken neural scaling laws, revealing a power-law relationship between FLOPs and R/R*. The framework enables offline prediction of online revenue and machine costs, facilitating efficient model selection and resource allocation without costly online A/B tests.
Key Results
- The offline metric R/R* shows a high linear correlation with online revenue (R^2=0.902), outperforming traditional offline metrics. The fitted broken neural scaling law achieves R^2=0.996 across models, confirming the power-law relationship between FLOPs and R/R*. Offline simulations accurately estimate machine costs from model parameters, enabling end-to-end offline performance and cost prediction. Applying these insights, the system achieves a 5.10% increase in online revenue through optimized resource allocation, demonstrating practical utility.
- The approach allows rapid evaluation of hundreds of model configurations within days, significantly reducing iteration costs. Validation in real-world ad retrieval scenarios confirms the robustness of the scaling law, with predictions of revenue gains closely matching actual online results. The method supports dynamic model scaling and resource planning, making it highly suitable for industrial deployment.
- Overall, the framework bridges offline metrics and online revenue, providing a scalable, low-cost solution for model performance prediction and system optimization in recommendation and advertising systems.
Significance
This work pioneers the systematic study of online scaling laws in advertising retrieval systems, addressing the critical challenge of cost-effective performance optimization. By establishing a high-correlation offline metric and a power-law scaling law, it enables rapid, accurate prediction of online revenue and resource costs. This significantly reduces reliance on expensive online experiments, accelerates model iteration, and enhances system efficiency. The methodology's generality across multiple architectures and scenarios paves the way for broader adoption in recommendation systems, potentially transforming industry practices. The insights gained can guide the development of more scalable, cost-efficient ad retrieval models, ultimately boosting ROI and user experience. The research also contributes to the theoretical understanding of neural scaling laws beyond NLP, extending their applicability to industrial recommendation systems.
Technical Contribution
Technically, the paper introduces the offline metric R/R*, justified through theoretical analysis that links it to online revenue asymptotically. It further demonstrates that multiple architectures (Transformer, MLP, DSSM) exhibit broken neural scaling laws, modeled via a power-law (BNSL). The framework includes a novel offline simulation algorithm to estimate machine costs from model parameters, enabling end-to-end offline prediction of performance and costs. This integrated approach allows for efficient, scalable system optimization without online experiments, representing a significant advancement over existing methods that rely heavily on online A/B testing or offline proxies. The methodology's robustness and generality across architectures mark a key technical innovation.
Novelty
This is the first comprehensive study establishing online scaling laws for advertising retrieval models, with a novel offline metric R/R* serving as a high-fidelity surrogate for online revenue. Unlike prior work limited to offline performance metrics, this approach enables accurate, low-cost prediction of online revenue and costs, facilitating scalable model optimization. The integration of theoretical guarantees, empirical validation, and practical system deployment distinguishes this work from existing literature, offering a new paradigm for cost-effective, data-driven system tuning in industrial recommendation environments.
Limitations
- The theoretical assumptions, such as linearity between R/R* and revenue, may not hold in extreme scenarios or highly heterogeneous systems, leading to potential prediction errors.
- Machine cost estimation relies on static model parameters and simplified system models, which may not fully capture hardware-specific variations or system-level optimizations.
- Current validation is limited to ad retrieval sub-systems; extending to full recommendation pipelines and more complex system architectures remains future work.
Future Work
Future directions include refining the theoretical assumptions to handle system heterogeneity, integrating dynamic hardware and system information for更准确的成本预测，以及扩展模型缩放规律到完整推荐流程。此外，结合强化学习等技术实现自动化模型调优和资源调度，将进一步推动广告系统的智能化升级。
AI Executive Summary
In recent years, the rapid growth of deep learning models has brought about remarkable performance improvements across NLP and recommendation domains. However, understanding how model performance scales with resources remains a challenge, especially in industrial settings where online experiments are costly and time-consuming. Traditional approaches rely heavily on online A/B testing to evaluate model improvements, which limits rapid iteration and incurs significant costs. To address this, the present study proposes a novel offline metric, R/R*, that correlates strongly with online revenue, supported by rigorous theoretical analysis and extensive empirical validation.
Leveraging this metric, the authors establish a power-law scaling law—Broken Neural Scaling Law (BNSL)—that describes the relationship between FLOPs and R/R* across multiple model architectures, including Transformer, MLP, and DSSM. The key innovation lies in combining offline performance metrics with a simulation algorithm that estimates machine costs from model parameters, enabling a comprehensive end-to-end offline prediction of both revenue and resource consumption. This framework allows practitioners to evaluate hundreds of model configurations rapidly, reducing the need for online testing and accelerating model iteration cycles.
The experimental results demonstrate that the scaling law accurately predicts online revenue within a 0.5% error margin, validated through real-world ad retrieval experiments. Applying these insights, the authors successfully optimize resource allocation and model design, achieving a 5.10% increase in online revenue. This work significantly advances the practical application of neural scaling laws, providing a scalable, low-cost methodology for industrial recommendation systems. Future work aims to incorporate dynamic hardware information and extend the framework to full recommendation pipelines, promising further improvements in efficiency and performance.
Deep Analysis
Background
近年来，深度学习模型的规模不断扩大，带来了自然语言处理和推荐系统的飞跃式发展。Kaplan等（2020）在Transformer架构中提出了缩放定律，揭示了模型参数、数据规模和计算成本对性能的影响。GPT-3的成功验证了大规模预训练模型的潜力，推动了大模型的快速发展。然而，推荐系统和广告检索模型的缩放规律研究相对滞后，主要受限于缺乏公开数据和系统异质性。早期研究多关注离线指标与模型性能的关系，缺乏对在线收益的系统分析。近年来，部分学者尝试结合离线指标和有限在线实验探索模型规模对收益的影响，但仍未建立完整的理论框架。本文在此基础上，提出了基于离线指标R/R*的轻量化范式，旨在弥补实际工业场景中缺乏高效、低成本识别缩放规律的空白。
Core Problem
在工业广告检索系统中，模型规模与在线收益之间的关系难以量化，主要原因在于在线实验成本高昂且系统复杂。传统方法依赖大量A/B测试，既耗时又不具普适性。此外，不同系统架构和参数配置的异质性，限制了缩放规律的跨场景应用。如何在保证低成本的前提下，准确识别模型规模与收益的关系，成为行业亟需解决的问题。缺乏有效的离线指标作为代理，导致模型优化难以快速迭代，影响整体广告投放效果和ROI。
Innovation
本文的核心创新包括：1）提出离线指标R/R*，结合模型预测的广告收益，理论上证明其与真实在线收益的渐近相关性，显著优于传统离线指标；2）建立基于Broken Neural Scaling Law的模型性能预测框架，揭示FLOPs与R/R*之间的幂律关系；3）设计离线仿真算法，端到端映射模型参数到机器成本，实现低成本的系统资源调度。这些创新突破了以往仅关注离线指标或在线实验的局限，为模型规模优化提供了理论支撑和实践工具。
Methodology
- �� 设计离线指标R/R*，通过排序矩阵和广告收益预测值计算，反映模型对收益的贡献。• 理论分析证明在一定假设下，R/R*与在线收益呈线性关系。• 利用多模型架构（Transformer、MLP、DSSM）在离线数据上训练，收集FLOPs和R/R*指标，拟合Broken Neural Scaling Law。• 采用贝叶斯优化等方法，拟合幂律关系参数，实现模型性能预测。• 开发离线仿真工具，基于模型参数和系统环境，估算机器成本。• 将缩放定律应用于模型设计和资源调度，优化广告投放ROI。
Experiments
- �� 使用真实广告系统数据，采集不同模型配置（Transformer、MLP、DSSM）在多场景下的FLOPs、R/R*值和在线收益。• 通过A/B测试验证R/R*与在线收益的相关性，R^2达0.902。• 拟合Broken Neural Scaling Law，模型拟合优度高达0.996。• 利用仿真算法估算机器成本，验证模型规模与成本的关系。• 在多场景、多模型架构中应用缩放定律，提升广告ROI，平均提升达5.10%。
Results
- �� 发现FLOPs与R/R*满足幂律关系，模型性能可通过离线指标准确预测。• 通过离线仿真实现模型规模与机器成本的端到端映射，支持快速调优。• 实验验证缩放规律在不同模型和场景中的普适性，显著降低调优成本。• 采用该方法实现的模型优化，ROI提升超过5%，验证了实际应用价值。
Applications
- �� 立即应用：模型设计中，依据缩放定律调整模型规模，优化ROI，无需大量在线试验。• 长期展望：结合硬件环境变化，动态调整模型配置，实现持续优化和自动调度，推动广告系统智能化升级。
Limitations & Outlook
- �� 假设模型性能与R/R*线性关系在极端规模或特殊场景下可能偏离。• 机器成本估算依赖静态模型参数，未充分考虑硬件环境变化。• 当前方法主要验证于广告检索子系统，尚未扩展到完整推荐流程，未来需考虑系统整体复杂性。
Plain Language Accessible to non-experts
想象你在经营一家大型厨房，准备各种菜肴。每道菜的成本和用料不同，做得越多，成本越高，但也能卖出更多。为了让厨房既能做出美味菜肴，又不亏钱，你需要知道每个菜的成本和卖价之间的关系。传统方法是试吃多次，观察卖出去的菜数和利润，但这样耗时又浪费。本文的方法就像用一个智能的菜单预测器，根据之前的销售数据，提前估算出不同菜品的利润和成本关系。这样，你就可以在不实际做菜的情况下，预测出哪种菜最划算，提前调整菜单，既节省时间，又能最大化利润。这就像用数学模型帮你做出明智的经营决策，避免盲目试错。
ELI14 Explained like you're 14
想象你在学校的食堂工作，要准备很多不同的菜，每个菜的材料和价格都不一样。你想知道做哪种菜最赚钱，但每次试做都很麻烦，也很花时间。于是，你发明了一个聪明的计算器，可以根据以前的销售情况，预测每道菜的利润。只要输入菜的材料和份量，这个计算器就能告诉你，哪种菜能赚最多钱，而且不用实际做出来就知道效果。这样，你就可以提前安排材料和做菜的数量，省时又省钱，还能让更多同学吃到喜欢的菜。这就像用数学和数据帮你做决定，让厨房变得更聪明、更高效。
Abstract
The scaling law is a notable property of neural network models and has significantly propelled the development of large language models. Scaling laws hold great promise in guiding model design and resource allocation. Recent research increasingly shows that scaling laws are not limited to NLP tasks or Transformer architectures; they also apply to domains such as recommendation. However, there is still a lack of literature on scaling law research in online advertisement retrieval systems. This may be because 1) identifying the scaling law for resource cost and online revenue is often expensive in both time and training resources for industrial applications, and 2) varying settings for different systems prevent the scaling law from being applied across various scenarios. To address these issues, we propose a lightweight paradigm to identify online scaling laws of retrieval models, incorporating a novel offline metric and an offline simulation algorithm. We prove that under mild assumptions, the correlation between the novel metric and online revenue asymptotically approaches 1 and empirically validates its effectiveness. The simulation algorithm can estimate the machine cost offline. Based on the lightweight paradigm, we can identify online scaling laws for retrieval models almost exclusively through offline experiments, and quickly estimate machine costs and revenues for given model configurations. We further validate the existence of scaling laws across mainstream model architectures (e.g., Transformer, MLP, and DSSM) in our real-world advertising system. With the identified scaling laws, we demonstrate practical applications for ROI-constrained model designing and multi-scenario resource allocation in the online advertising system. To the best of our knowledge, this is the first work to study identification and application of online scaling laws for online advertisement retrieval.