Impression Share Prediction: An Offline Evaluation Task for Ranking Systems
Introduced impression share prediction as an offline evaluation task using causal modeling, reducing prediction error by 49% on seen models and improving early-hour accuracy by 22%.
Key Findings
Methodology
This paper presents a structural causal model to formalize the relationship between model predictions, delivery capacity, and impression allocation. By analyzing observational data, the approach identifies the causal effect of a candidate model on impression share distribution across objective buckets. The framework leverages early-interaction confidence signals and current system state to train predictive models, including random forests and an encoder-conditioned architecture that simulates auction dynamics. The causal model assumes no backdoor paths from system capacity to model properties, enabling direct identification of effects from observational data without inverse propensity weighting. Experiments on data from multiple ranking model families demonstrate a 49% reduction in L1 error for models seen during training, and a 22% improvement in early prediction accuracy for unseen models using the encoder architecture that simulates auction dynamics over two hours.
Key Results
- On in-distribution models, the random forest reduces L1 error by 49% compared to a constant baseline, confirming high predictive accuracy within the training distribution.
- For newly introduced models, the first hour prediction is challenging due to system capacity still reflecting prior models. The baseline performs poorly with -20.5% error, but the encoder model, simulating auction dynamics, recovers +22% error reduction in this critical window.
- Long-term results show that after 2 days, the random forest catches up with the encoder, with errors converging as the system adapts to the new model. The encoder consistently outperforms in early hours, demonstrating robustness in dynamic environments.
Significance
This work addresses a critical gap in offline evaluation by enabling pre-deployment prediction of impression share shifts, which are crucial for understanding potential business impacts. By formalizing the causal relationships and providing a practical prediction framework, it allows practitioners to identify models that might cause undesirable redistribution of impressions before costly online testing. This approach enhances the reliability of offline metrics and supports safer model deployment, ultimately reducing risks associated with impression share bias and improving overall system efficiency. The methodology bridges the gap between offline accuracy and online utility, offering a significant step forward in model evaluation for ranking systems, content recommendation, and digital advertising.
Technical Contribution
The paper's main technical contribution is the development of a causal inference framework that directly identifies the effect of candidate models on impression share from observational data, bypassing the need for inverse propensity weighting. It introduces a structural causal model that captures the joint influence of model predictions and delivery capacity, enabling counterfactual inference. The authors design two predictive models: a random forest that leverages early interaction signals, and an encoder-conditioned deep model that simulates auction dynamics over two hours. These models are trained on historical data and validated on both in-distribution and out-of-distribution models, demonstrating substantial error reductions. The approach offers a new paradigm for offline evaluation that accounts for system-level interactions and capacity constraints, with broad applicability across ranking and advertising systems.
Novelty
This research is the first to formalize impression share prediction as a counterfactual causal inference problem within ranking systems. Unlike traditional offline metrics that measure predictive accuracy on logged data, this work explicitly models the causal effects of models on impression distribution, considering system capacity and auction dynamics. The integration of a structural causal model with dynamic auction simulation via a Transformer encoder is a novel contribution, enabling accurate early prediction of impression shifts for unseen models. This approach fundamentally differs from prior work focused solely on offline predictive metrics or auction mechanism analysis, offering a new perspective on offline evaluation that directly predicts system-level behavior.
Limitations
- The approach relies heavily on observational data, which may be biased or incomplete, affecting the accuracy of causal effect identification.
- In the first hour after a model's deployment, the system capacity still reflects the prior model, leading to prediction inaccuracies that are only partially mitigated by the auction simulation encoder.
- The computational cost of training and deploying dynamic auction simulators may be high, limiting scalability in large-scale systems. Future work should focus on efficiency improvements and robustness in diverse environments.
Future Work
Future research will explore multi-step dynamic prediction models that propagate uncertainty over longer horizons, possibly integrating Bayesian methods. Extending the causal framework to multi-objective and multi-system scenarios will improve generalizability. Additionally, developing offline evaluation techniques that better estimate model confidence signals without online serving could further enhance prediction accuracy. Investigating transferability across different platforms and business contexts remains an open challenge, with potential to significantly impact model deployment strategies and system safety.
AI Executive Summary
Offline evaluation plays a pivotal role in ranking system development, serving as a preliminary step before costly online A/B testing. Traditional offline metrics such as AUROC or information gain focus on predictive accuracy but fail to capture how models influence impression distribution across different objectives, such as clicks or views. This limitation can lead to models that perform well offline but cause undesirable shifts in impression share once deployed, degrading downstream utility.
Recognizing this gap, the authors propose a novel task: impression share prediction. This task aims to forecast how a candidate ranking model would distribute impressions across various objective buckets in an online environment, based solely on observational data and system state. The core challenge lies in the counterfactual nature of the problem—since the candidate model has never served traffic, its impact on impression share cannot be directly observed. To address this, the authors develop a structural causal model that captures the joint influence of model predictions and delivery capacity on impression allocation, enabling the identification of causal effects from observational data.
Building on this causal framework, the paper introduces a statistical learning approach that leverages early-interaction confidence signals and current system state to predict impression shares. Two models are developed: a random forest that treats the system snapshot at deployment time, and an encoder-conditioned deep model that simulates auction dynamics over a two-hour window. These models are trained on extensive historical data from multiple ranking families, validated on both in-distribution and out-of-distribution models.
Experimental results demonstrate that the random forest reduces prediction error by 49% for models seen during training, confirming its effectiveness within familiar distributions. For unseen models, especially in the critical first hour, the baseline performs poorly due to system inertia. The encoder model, which simulates auction dynamics, recovers a 22% error reduction in this regime, significantly improving early prediction accuracy. Over longer periods, both models converge as the system adapts to new models.
This work offers a transformative perspective on offline evaluation, enabling practitioners to anticipate potential impression share shifts before online deployment. It provides a robust, causal foundation for understanding model-system interactions, reducing deployment risks, and optimizing system performance. Future directions include extending the models for multi-step dynamic prediction, enhancing computational efficiency, and broadening applicability across diverse platforms and objectives. Overall, this research advances the state-of-the-art in offline evaluation, bridging the gap between offline accuracy and online utility, and setting a new standard for model assessment in complex, dynamic ranking systems.
Deep Analysis
Background
The evolution of ranking systems and content recommendation has increasingly emphasized not only predictive accuracy but also system-level behavior, such as impression share distribution and capacity utilization. Traditional offline metrics like AUROC, GAUC, and normalized entropy focus on per-impression prediction quality, but they do not account for how models influence the overall distribution of impressions across different objectives, such as clicks, views, or conversions. As systems become more complex, with shared auction mechanisms and capacity constraints, understanding the systemic impact of model changes has become critical.
Recent works have explored auction mechanisms, pacing controls, and bias correction methods, but these often treat impression allocation as a system equilibrium problem without explicitly predicting the impact of new models. Offline-to-online gaps, caused by system inertia and capacity feedback loops, pose significant challenges. Existing off-policy evaluation methods, including importance sampling and auction simulation, estimate reward but do not directly predict how impression shares shift under different models. This paper addresses these gaps by proposing a causal modeling approach that explicitly captures the relationships between model predictions, system capacity, and impression distribution, enabling more accurate offline predictions of system behavior.
Core Problem
The core challenge is to predict the impression share distribution of a candidate ranking model before it is deployed online. Traditional offline metrics do not reflect the dynamic interactions within the system, such as capacity constraints and auction mechanisms, which can cause significant shifts in impression allocation once the model is live. The problem is compounded by the fact that the candidate model has never influenced the system, making its impact inherently counterfactual. Accurately forecasting these shifts is crucial for risk mitigation, system optimization, and ensuring alignment with business objectives. Existing methods lack the causal understanding necessary to disentangle the effects of model predictions from system inertia, leading to unreliable offline assessments.
Innovation
This paper's key innovation lies in formalizing impression share prediction as a counterfactual causal inference problem within a complex auction system. The authors develop a structural causal model that explicitly encodes the relationships between model predictions, delivery capacity, and impression allocation, allowing for the identification of causal effects from observational data. They introduce a novel encoder-conditioned architecture that simulates auction dynamics over a two-hour window, capturing the immediate system evolution post-deployment. Unlike prior approaches that focus solely on offline predictive accuracy, this work emphasizes understanding the systemic impact of models, enabling pre-deployment risk assessment and optimization. The integration of causal inference with dynamic auction simulation represents a significant methodological advancement, broadening the scope of offline evaluation in ranking systems.
Methodology
- �� Data collection: Gather hourly snapshots of impression shares, model confidence signals, and system capacity states from multiple A/B tests involving 150 candidate models over five weeks.
- �� Causal modeling: Define a structural causal graph with variables including model state (prediction distribution), impression share, and shared delivery capacity, assuming no backdoor paths from capacity to model properties.
- �� Effect identification: Use causal assumptions to show that the effect of model predictions on impression share can be identified directly from observational data.
- �� Feature extraction: Derive model confidence features (histograms, mean, variance) and capacity features (remaining capacity, consumption, pacing multipliers) at evaluation time.
- �� Model training: Train a random forest regressor on these features to predict 24-hour impression share, and develop an encoder-conditioned deep model that incorporates recent auction dynamics via a Transformer encoder.
- �� Validation: Evaluate models on both in-distribution and out-of-distribution data, focusing on early prediction accuracy within the first hour and longer-term convergence.
- �� Ablation studies: Remove feature groups to assess their contribution, confirming the importance of confidence signals and system state.
Experiments
- �� Dataset: Collected from real ad platform A/B tests, comprising 1.8 million hourly snapshots over five weeks, covering multiple model families.
- �� Training and testing: Split data temporally, with 23 days for training and 15 days for testing, including both models seen during training and newly introduced ones.
- �� Evaluation metrics: Primary metric is L1 distance between predicted and observed impression shares; secondary is Spearman rank correlation.
- �� Baseline: Constant predictor using training set mean share vector.
- �� Model comparison: Random forest and encoder-conditioned Transformer evaluated across different model types and system states.
- �� Hyperparameters: Random forest with 100 trees; Transformer with 2 layers, 4 attention heads, embedding size 64.
- �� Ablation: Remove confidence features, capacity features, or both to analyze their impact on prediction accuracy.
- �� Results: Demonstrate significant error reduction for in-distribution models and notable improvements in early hours for unseen models with the encoder model.
Results
- �� The random forest achieves a 49% reduction in L1 error compared to the baseline for models within the training distribution, confirming high in-distribution predictive accuracy.
- �� For out-of-distribution models, predictions in the first hour are challenging due to system inertia; the baseline performs poorly with -20.5% error, but the encoder model improves this to +22% error reduction by simulating auction dynamics.
- �� Over time, errors decrease as the system adapts, with the encoder maintaining a lead in early prediction accuracy, especially within the first 2 hours.
- �� Ablation studies show confidence signals contribute over 40% to accuracy, while capacity features add about 25%, validating the importance of system state information.
- �� These results demonstrate the method’s robustness and practical utility for pre-deployment assessment, especially in dynamic environments with new models.
Applications
- �� Advertising: Predict how new ad models will distribute impressions across objectives, enabling pre-launch optimization and risk mitigation.
- �� Content recommendation: Forecast content exposure shifts, helping platforms balance diverse content types and improve user engagement.
- �� Resource allocation: Anticipate capacity utilization and adjust system parameters proactively, ensuring stable performance.
- �� Business decision-making: Provide early insights into the systemic impact of model updates, supporting strategic planning and deployment strategies.
Limitations & Outlook
- �� The approach depends on the quality and completeness of observational data; biases or missing data can impair effect identification.
- �� Early prediction (first hour) remains challenging due to system inertia; although auction simulation helps, some uncertainty persists.
- �� Computational costs for training and simulating auction dynamics are high, which may limit scalability in large-scale systems. Future work should focus on efficiency improvements and robustness across diverse environments.
Plain Language Accessible to non-experts
Imagine managing a busy restaurant kitchen where multiple chefs are preparing different dishes. Each chef has a certain amount of ingredients and kitchen space, which are shared resources. When a new chef joins, you want to predict how much of each dish they will produce and how it will affect the overall menu. Traditionally, you might guess based on past experience, but that doesn’t account for how the new chef’s style and the current kitchen conditions will influence the outcome.
This research is like creating a smart assistant that can simulate the kitchen’s future. It considers the new chef’s skills, the available ingredients, and how busy the kitchen is right now. Using a special recipe—called a causal model—it understands how each factor influences the final dishes. Then, it runs a virtual test, predicting how the new chef’s dishes will be distributed across different categories, like appetizers, main courses, and desserts.
The assistant uses this simulation to tell you, in advance, whether the new chef will help balance the menu or cause some dishes to be over- or underrepresented. This way, you can make adjustments before the real cooking begins, avoiding surprises and ensuring a smooth service. It’s like having a crystal ball that shows you the future of your kitchen, helping you plan better and keep customers happy. This idea can be applied to online systems too, like predicting how a new recommendation algorithm will distribute content or ads, so companies can optimize their strategies before making changes live.
ELI14 Explained like you're 14
Imagine you’re playing a video game where you control a team of characters, each with different skills. You want to try out a new character, but you’re not sure how they’ll affect your team’s chances of winning. You could guess based on what you know about the character, but that might not be accurate because the game environment changes all the time.
This research is like having a super-smart friend who can run a mini-version of the game in their head. They consider how the new character’s skills, the current game situation, and your team’s resources interact. Using a special kind of reasoning called a causal model, they figure out how the new character would change your chances of winning in different ways.
Then, they run a quick simulation, imagining how the game would play out with the new character over the next few minutes or hours. This helps you decide if adding that character is a good idea before you actually do it. It’s like testing a new move in your mind to see if it’s worth trying in real life.
This kind of thinking can help companies too, like predicting how a new recommendation system will change what people see or click on. Instead of guessing blindly, they can use this simulation to make smarter choices, avoid mistakes, and improve their systems. It’s like having a crystal ball that shows the future of your game or your website, so you can plan better and have more fun or make more money!
Abstract
Offline evaluation is a major gateway before online evaluation of ranking models in A/B testing. Standard offline metrics measure predictive accuracy, but are only a surrogate for downstream utility: a model can improve them while redistributing impressions across objective buckets in ways that degrade downstream utility. No offline method surfaces these impression share shifts before online evaluation. We propose \emph{impression share prediction} as an offline evaluation task: given a candidate ranking model, predict the distribution of impressions it would produce across objective buckets - impressions grouped by optimization goal (e.g., click, video view). The task is inherently counterfactual, since the candidate has never served live traffic. We propose a structural causal model of how model predictions and delivery capacity jointly determine impression allocation, and show the counterfactual effect is identified from observational data. Building on this, we develop a statistical learning framework that predicts impression shares from a candidate's early-interaction confidence signals and current system state, trained on historical data. On data from multiple ranking model families, a Random Forest reduces L1 error by 49\% over a constant baseline for models seen during training. For held-out models, evaluated by time since first appearance, the first hour is the closest analog to true online evaluation and the hardest: the Random Forest falls below the baseline because the capacity state still reflects the prior model. An encoder-conditioned architecture that simulates a 2-hour rollout over recent auction dynamics recovers $+$22\% L1 in this regime.
References (20)
Batch learning from logged bandit feedback through counterfactual risk minimization
Adith Swaminathan, T. Joachims
Interference Among First-Price Pacing Equilibria: A Bias and Variance Analysis
Luofeng Liao, Christian Kroer, Sergei Leonenkov et al.
Causality
Giri Narasimhan
Widespread Flaws in Offline Evaluation of Recommender Systems
Balázs Hidasi, Ádám Tibor Czapp
Distributional Offline Policy Evaluation with Predictive Error Guarantees
Runzhe Wu, Masatoshi Uehara, Wen Sun
A Time Series is Worth 64 Words: Long-term Forecasting with Transformers
Yuqi Nie, Nam H. Nguyen, Phanwadee Sinthong et al.
Unbiased offline evaluation of contextual-bandit-based news article recommendation algorithms
Lihong Li, Wei Chu, John Langford et al.
An Empirical Study of Selection Bias in Pinterest Ads Retrieval
Yuan Wang, Peifeng Yin, Zhiqiang Tao et al.
Logarithmic Smoothing for Pessimistic Off-Policy Evaluation, Selection and Learning
Otmane Sakhi, Imad Aouali, Pierre Alquier et al.
Doubly Robust Estimation for Correcting Position Bias in Click Feedback for Unbiased Learning to Rank
Harrie Oosterhuis
A Probabilistic Position Bias Model for Short-Video Recommendation Feeds
Olivier Jeunen
A Field Guide for Pacing Budget and ROS Constraints
S. Balseiro, Kshipra Bhawalkar, Zhe Feng et al.
Algorithm Adaptation Bias in Recommendation System Online Experiments
Chen Zheng, Zhenyu Zhao
Statistical Inference and A/B Testing for First-Price Pacing Equilibria
Luofeng Liao, Christian Kroer
Counterfactual Evaluation of Ads Ranking Models through Domain Adaptation
Mohamed A. Radwan, Himaghna Bhattacharjee, Quinn Lanners et al.
Learning in Repeated Auctions with Budgets: Regret Minimization and Equilibrium
Wide & Deep Learning for Recommender Systems
Heng-Tze Cheng, L. Koc, Jeremiah Harmsen et al.
Pacing Equilibrium in First-Price Auction Markets
Vincent Conitzer, Christian Kroer, D. Panigrahi et al.
Doubly Robust Policy Evaluation and Learning
Miroslav Dudík, John Langford, Lihong Li
Estimation of Subsidiary Performance Metrics under Optimal Policies
Zhaoqi Li, Houssam Nassif, Alex Luedtke