Building a User Foundation Model for the Open Web
Proposed a User Foundation Model using self-supervised learning to enhance CTR prediction on the open web, achieving a 2.13% CTR increase.
Key Findings
Methodology
The study introduces a User Foundation Model leveraging self-supervised learning on user browsing histories. The model employs a Transformer encoder, pre-trained with masked language modeling and a sequence-level contrastive objective, then fine-tuned on a click prediction task. The encoder's pre-training pipeline is optimized using an LLM-in-the-loop search, demonstrating significant performance improvements in production settings.
Key Results
- The encoder representation improved RIG by 1.197% on the production bid win-rate model and by 1.354% on the CTR ranker.
- A 7-day live A/B test confirmed a 2.13% CTR increase and a 1.13% eCPC reduction.
- In the absence of historical data, the model achieved a 0.99% RIG improvement in offline tests.
Significance
This research demonstrates the feasibility of User Foundation Models in open web environments, addressing issues of fragmented user identity and lack of historical data. By employing self-supervised learning, the model provides effective user representations even without continuous user histories, enhancing ad recommendation system performance.
Technical Contribution
Technical contributions include applying LLM to optimize the encoder pre-training pipeline in an industrial setting and showcasing the effectiveness of User Foundation Models in open web environments for the first time. The model performs well in both history-absent and history-present scenarios, proving its generalization capability.
Novelty
This study is the first to apply User Foundation Models in open web environments, using self-supervised learning to tackle fragmented user identity, offering a novel solution compared to existing methods.
Limitations
- The model's performance may be affected in scenarios with extremely sparse user history data.
- More validation is needed across different network environments.
Future Work
Future work could explore more complex user behavior modeling and validate the model's generalization across different advertising platforms.
AI Executive Summary
In open web real-time bidding environments, user identities are often fragmented and non-persistent, posing challenges for ad recommendation systems. Existing methods rely on stable user identities and continuous historical data, which are not applicable in open web contexts.
To address this, researchers propose a User Foundation Model using self-supervised learning on user browsing histories. The model employs a Transformer encoder, pre-trained with masked language modeling and a sequence-level contrastive objective, then fine-tuned on click prediction tasks.
Experimental results show significant improvements in CTR and reductions in eCPC in production settings, demonstrating the model's effectiveness in open web environments. This research offers a new solution for ad recommendation systems in data-scarce environments.
Deep Analysis
Background
With the evolution of the internet, ad recommendation systems have achieved significant success in e-commerce and social platforms. However, these systems typically rely on stable user identities and rich historical data. In open web real-time bidding environments, user identities are fragmented and non-persistent, making traditional methods less applicable.
Core Problem
In open web environments, fragmented and non-persistent user identities and lack of historical data make it challenging for ad recommendation systems to effectively model users and provide personalized recommendations. Solving this issue is crucial for improving ad targeting accuracy and user experience.
Innovation
The study proposes a User Foundation Model using self-supervised learning on user browsing histories. The model employs a Transformer encoder, pre-trained with masked language modeling and a sequence-level contrastive objective, then fine-tuned on click prediction tasks. Compared to existing methods, this model provides effective user representations even without continuous user histories.
Methodology
- �� Pre-train a Transformer encoder on user browsing histories.
- �� Use masked language modeling and sequence-level contrastive objectives for pre-training.
- �� Fine-tune on click prediction tasks.
- �� Optimize the encoder pre-training pipeline using LLM.
Experiments
Experiments were conducted in open web real-time bidding environments using various datasets. The model performed well in both production bid win-rate and CTR ranker models, and its effectiveness was confirmed through a 7-day live A/B test.
Results
The model improved RIG by 1.197% on the production bid win-rate model and by 1.354% on the CTR ranker. A 7-day live A/B test confirmed a 2.13% CTR increase and a 1.13% eCPC reduction.
Applications
The model can be used in ad recommendation systems, particularly in open web environments where user identities are fragmented and historical data is scarce. It enhances ad targeting accuracy and user experience.
Limitations & Outlook
The model's performance may be affected in scenarios with extremely sparse user history data. Additionally, more validation is needed across different network environments to ensure the model's generalization capability.
Plain Language Accessible to non-experts
Imagine you're in a library, and the librarian needs to recommend new books based on your borrowing history. But the problem is, you use a different identity each time you visit, so the librarian can't track your full history. Our model is like a smart librarian who learns from each borrowing record, even if you use different identities, to recommend books you'll love.
ELI14 Explained like you're 14
Imagine you're playing a game, and each time you enter, you use a different character. The game needs to recommend gear based on your play style. Our model is like a super-smart AI assistant that learns from each play record, even if you use different characters, to recommend the best gear for you!
Glossary
User Foundation Model
A method for modeling user behavior using self-supervised learning.
Used for user modeling in open web environments.
Self-Supervised Learning
A machine learning method that uses unlabeled data for model training.
Used for pre-training on user browsing histories.
Transformer Encoder
A deep learning model for processing sequential data.
Used for encoding user browsing histories.
Masked Language Modeling
A language model trained by masking parts of the input data.
Used for pre-training the User Foundation Model.
Sequence-Level Contrastive Objective
A training objective that contrasts different sequence representations.
Enhances the model's sequence representation capability.
Open Questions Unanswered questions from this research
- 1 How to improve model performance with extremely sparse user history data?
- 2 How to validate the model's generalization across different network environments?
Applications
Immediate Applications
Ad Recommendation Systems
Can be used to enhance ad targeting accuracy and user experience, especially in open web environments.
Long-term Vision
Cross-Platform User Modeling
Achieve cross-platform user behavior modeling to improve personalized recommendation accuracy.
Abstract
User foundation models have demonstrated strong results in e-commerce and social recommendation, but most industrial deployments assume environments where user identity is stable and persistent. Open-web real-time bidding (RTB) operates on a structurally different data distribution: user identity is fragmented and non-persistent across browsing sessions, and the availability of browsing history depends on user privacy choices. Consequently, a significant portion of traffic carries no historical data, and available records often consist of relatively short, disjointed sessions. As a result, historical signals in this domain are typically represented as aggregated counters and recency buckets, leaving the sequential structure unexploited. To address this limitation, we present a user foundation model that applies self-supervised learning on user browsing histories and show that the learned representation improves multiple downstream production tasks, demonstrating the viability of this approach on the open web. We pre-train a Transformer encoder with masked language modeling and a sequence-level contrastive objective, then fine-tune it on the click prediction task. We optimize the encoder's pre-training pipeline with an LLM-in-the-loop search over a curated catalog of reviewable, code-level edits (lifters), instantiating the LLM-as-optimizer paradigm in an industrial setting. The same encoder representation yields +1.197% RIG on the production bid win-rate model and +1.354% RIG on the production CTR ranker; a 7-day live A/B test confirms +2.13% CTR, -1.13% eCPC (80% CI excluding zero on both metrics).