Search-based User Interest Modeling with Lifelong Sequential Behavior Data for Click-Through Rate Prediction
Proposes Search-based Interest Model (SIM) with two-stage search for long user behavior sequences, boosting CTR prediction accuracy.
Key Findings
Methodology
The proposed SIM employs a two-stage search framework: the General Search Unit (GSU) quickly retrieves relevant sub-sequences from ultra-long behavior data using category matching or maximum inner product search; the Exact Search Unit (ESU) applies multi-head attention to model the precise relationship between candidate items and the filtered sub-sequences. This architecture effectively filters noise and captures fine-grained interests, supporting sequences up to 54,000 behaviors. During training, both modules are jointly optimized with a weighted cross-entropy loss, ensuring scalability and accuracy. The system integrates category indices (behavior trees) for offline pre-computation, enabling real-time deployment in large-scale industrial systems.
Key Results
- On Taobao and Alibaba industrial datasets, SIM achieved AUC scores of 0.9416 and 0.7510, outperforming MIMN (0.9278 and 0.7396) and DIN, with improvements over 1.5%. The maximum sequence length supported reached 54,000, a 54-fold increase over industry SOTA, resulting in 7.1% CTR and 4.4% RPM lift in live systems.
- Offline ablation studies confirmed that the two-stage search architecture significantly improves interest relevance and model robustness. The hard-search strategy, despite being non-parametric, offers resource efficiency comparable to soft-search in industrial deployment.
- The deployment demonstrates that SIM can handle real-time CTR prediction with ultra-long sequences, providing a scalable solution for long-term user interest modeling in large-scale industrial environments.
Significance
This work addresses the longstanding challenge of modeling ultra-long user behavior sequences in industrial systems. By integrating search techniques with deep interest modeling, it overcomes the limitations of fixed-length models like MIMN, enabling more accurate interest capture and faster inference. The successful deployment in Alibaba’s advertising platform proves its practical value, setting a new industry standard for long sequence user modeling. It opens avenues for more personalized, diverse recommendations, and enhances understanding of user preferences over extended periods, impacting both academia and industry profoundly.
Technical Contribution
The core innovation lies in the hybrid search-based architecture that combines category indexing, maximum inner product search, and multi-head attention. This design enables the efficient filtering of relevant behaviors from ultra-long sequences, supporting lengths up to 54,000. The joint training of the two modules ensures mutual reinforcement, while the offline pre-built behavior tree index significantly reduces online latency. The approach extends the theoretical understanding of long sequence modeling by demonstrating that effective noise filtering and relevance search can dramatically improve both scalability and accuracy in real-world CTR prediction tasks.
Novelty
This research is the first to implement a two-stage search mechanism for ultra-long user behavior sequences in an industrial setting, supporting lengths 54 times longer than previous SOTA models like MIMN. Unlike prior fixed-length or simple attention-based models, SIM leverages category-based indexing and maximum inner product search to efficiently filter noise, enabling high-precision interest modeling at unprecedented scale. This combination of search technology with deep interest extraction marks a significant innovation in the field.
Limitations
- The reliance on pre-constructed category indices (behavior trees) introduces system complexity and maintenance overhead, especially when categories are dynamic or poorly defined.
- Despite filtering, some noise remains in the long behavior sequences, which may affect interest accuracy in highly diverse or ambiguous scenarios.
- The model’s training and online deployment require substantial computational resources, particularly for index updates and large-scale search optimization. Future work should focus on reducing these costs and improving robustness.
Future Work
Future directions include developing adaptive indexing strategies to handle dynamic categories, optimizing search algorithms for even larger sequences, and integrating multi-modal data (images, text) to enrich user interest representations. Additionally, exploring reinforcement learning for interest refinement and transfer learning for cross-domain applications could further enhance model versatility and performance.
AI Executive Summary
In the rapidly evolving digital landscape, user behavior data has become an invaluable asset for personalized recommendations and targeted advertising. However, the sheer volume and length of behavioral sequences pose significant challenges for traditional modeling approaches. Existing models like MIMN have demonstrated the ability to handle sequences up to 1,000 behaviors, but beyond that, their effectiveness diminishes due to noise accumulation and computational constraints. This bottleneck limits the potential for capturing long-term user interests, which are crucial for accurate CTR prediction.
To address this, the authors propose a novel Search-based Interest Model (SIM) that employs a two-stage search architecture. The first stage, the General Search Unit (GSU), efficiently filters relevant sub-sequences from ultra-long behavior data using category matching or maximum inner product search, reducing the sequence length from tens of thousands to hundreds. The second stage, the Exact Search Unit (ESU), applies multi-head attention mechanisms to model the precise relationship between candidate items and the filtered behaviors. This design effectively filters noise, enhances relevance, and supports sequences up to 54,000 behaviors—54 times longer than previous solutions.
The model’s effectiveness is validated through extensive experiments on public and industrial datasets, including Alibaba’s display advertising system. Results show a significant improvement in AUC scores—up to 0.9416 on Taobao data—and substantial CTR and RPM lifts in live deployment. The architecture’s scalability and accuracy make it a practical solution for real-time CTR prediction in large-scale industrial environments.
This breakthrough advances the state-of-the-art in long sequence user modeling, offering a scalable, efficient, and highly accurate approach. It paves the way for more personalized, long-term interest-aware recommendation systems, with broad implications for e-commerce, advertising, and content platforms. Future work will focus on further reducing computational costs, enhancing adaptability, and integrating multi-modal data to enrich user interest understanding.
Deep Dive
Abstract
Rich user behavior data has been proven to be of great value for click-through rate prediction tasks, especially in industrial applications such as recommender systems and online advertising. Both industry and academy have paid much attention to this topic and propose different approaches to modeling with long sequential user behavior data. Among them, memory network based model MIMN proposed by Alibaba, achieves SOTA with the co-design of both learning algorithm and serving system. MIMN is the first industrial solution that can model sequential user behavior data with length scaling up to 1000. However, MIMN fails to precisely capture user interests given a specific candidate item when the length of user behavior sequence increases further, say, by 10 times or more. This challenge exists widely in previously proposed approaches. In this paper, we tackle this problem by designing a new modeling paradigm, which we name as Search-based Interest Model (SIM). SIM extracts user interests with two cascaded search units: (i) General Search Unit acts as a general search from the raw and arbitrary long sequential behavior data, with query information from candidate item, and gets a Sub user Behavior Sequence which is relevant to candidate item; (ii) Exact Search Unit models the precise relationship between candidate item and SBS. This cascaded search paradigm enables SIM with a better ability to model lifelong sequential behavior data in both scalability and accuracy. Apart from the learning algorithm, we also introduce our hands-on experience on how to implement SIM in large scale industrial systems. Since 2019, SIM has been deployed in the display advertising system in Alibaba, bringing 7.1\% CTR and 4.4\% RPM lift, which is significant to the business. Serving the main traffic in our real system now, SIM models user behavior data with maximum length reaching up to 54000, pushing SOTA to 54x.