Dual-Stream MLP is All You Need for CTR Prediction
Proposes Dual-Stream MLP with knowledge distillation, achieving state-of-the-art CTR prediction performance with simplified architecture.
Key Findings
Methodology
This paper introduces DS-MLP, comprising a main MLP and a parallel MLP, trained via a two-stage process: distillation and alignment. The teacher model, GDCN, captures complex explicit interactions, guiding the main MLP through knowledge distillation. The parallel MLP learns implicit interactions. Two alignment strategies—hidden state normalization and prediction consistency—are employed to harmonize the outputs. The final model is a simple MLP structure, yet it effectively captures both interaction types, resulting in superior CTR prediction performance on benchmark datasets.
Key Results
- On Criteo, Avazu, and iPinYou datasets, DS-MLP outperformed existing models, with an average AUC improvement of over 0.3%, while maintaining low latency and parameter count, suitable for large-scale deployment.
- Ablation studies confirmed that knowledge distillation enhances explicit interaction learning, and the dual-MLP architecture improves implicit interaction modeling. Alignment strategies effectively balanced the outputs, leading to robust performance.
- In online A/B testing, the model achieved a 15% reduction in response latency and improved click prediction accuracy, demonstrating strong industry applicability.
Significance
This work addresses the core challenge of balancing model complexity and expressive power in CTR prediction. By leveraging a simple yet effective dual-MLP architecture guided by knowledge distillation, it overcomes the limitations of traditional high-order interaction models. The approach offers a scalable, efficient solution that maintains high accuracy, making it highly relevant for real-world recommendation systems. Its novel design paves the way for future research on lightweight, high-performance models that can handle massive feature spaces with ease, impacting both academia and industry.
Technical Contribution
The paper's main contribution is the development of a dual-stream MLP framework that integrates explicit and implicit feature interactions through knowledge distillation and alignment. It introduces a novel training pipeline with a teacher-student paradigm, along with two alignment strategies to ensure output compatibility. This design reduces model complexity while preserving or surpassing state-of-the-art accuracy, providing a new pathway for efficient CTR prediction models. The framework's flexibility allows adaptation to various teacher models, broadening its applicability.
Novelty
This is the first work to embed knowledge distillation within a dual-stream MLP architecture specifically for CTR prediction, focusing on balancing explicit and implicit feature interactions. Unlike previous models that simply sum interaction components, this approach employs alignment strategies to harmonize outputs, ensuring both interaction types are effectively learned. This innovation significantly reduces complexity while maintaining high performance, setting a new benchmark in lightweight CTR modeling.
Limitations
- The effectiveness heavily depends on the quality of the teacher model; if the teacher's performance is subpar, the student model's gains are limited. Additionally, the alignment strategies may face stability issues in extremely high-dimensional or imbalanced data scenarios.
- The current framework focuses on static features; incorporating temporal or multi-modal data remains an open challenge. Computational costs during training, especially for large teacher models, could be further optimized.
- Future work should explore robustness under diverse data distributions and extend to dynamic feature settings.
Future Work
Future directions include integrating multi-modal and temporal features, developing more robust alignment strategies, and exploring self-supervised pretraining to enhance generalization. Additionally, model compression and edge deployment will be prioritized to facilitate real-time, large-scale recommendation systems. Extending the framework to other prediction tasks beyond CTR, such as conversion rate or user engagement modeling, is also promising.
AI Executive Summary
In the rapidly evolving landscape of online advertising and personalized recommendations, accurately predicting user click-through rates (CTR) remains a fundamental challenge. Traditional models like deep cross networks and factorization machines have demonstrated strong capabilities in capturing complex feature interactions, but their high computational costs and tendency to overfit limit practical deployment at scale. To address these issues, this paper introduces DS-MLP, a novel dual-stream architecture built upon simple MLP components, optimized through a sophisticated training pipeline involving knowledge distillation and output alignment.
The core idea is to leverage a powerful teacher model, GDCN, which captures explicit feature interactions via a gated deep cross network, to guide a lightweight student MLP. Simultaneously, a parallel MLP is introduced to learn implicit interactions. Two alignment strategies—normalizing hidden states and aligning predictions—are employed to harmonize the outputs of both streams, ensuring balanced learning. This design allows the final model to effectively represent both explicit and implicit feature relationships without the complexity of traditional high-order interaction models.
Extensive experiments on benchmark datasets such as Criteo, Avazu, and iPinYou validate the effectiveness of DS-MLP. The model consistently surpasses existing state-of-the-art approaches, achieving over 0.3% improvement in AUC, with significantly fewer parameters and lower latency. Its simplicity, efficiency, and scalability make it highly suitable for industrial deployment, especially in large-scale recommendation systems.
This research marks a significant step forward in CTR prediction, demonstrating that a carefully designed, lightweight MLP framework guided by knowledge distillation can match or outperform complex models. The approach opens new avenues for developing efficient, high-performance models capable of handling massive feature spaces, with broad implications for both academia and industry. Future work will focus on extending the framework to multi-modal data, dynamic features, and edge deployment, aiming to further enhance the practical impact of this methodology.
Deep Dive
Abstract
Click-through rate (CTR) prediction holds a pivotal role in online advertising and recommendation systems, where even small improvements can significantly boost revenue. Existing research primarily focuses on designing dual-stream architectures to capture effective complex feature interactions from both explicit and implicit perspectives. However, these approaches are faced with two major challenges: 1) the high complexity of feature interaction learning, which increases computational demands and the overfitting risk, and 2) the imbalance between explicit and implicit modules, where one module's output may dominate the final prediction. To address these issues, in this paper, we propose Dual-Stream MLP (DS-MLP), a novel feature interaction framework for the CTR prediction task. Specially, it leverages knowledge distillation to consolidate the capacity of learning explicit feature interaction into a main MLP network, while a parallel MLP simultaneously captures implicit feature interactions as a complement. To effectively optimize the dual-stream MLP architecture, we further design a specific learning approach with two alignment strategies for enhancing the compatibility of the two MLP components. Experiments demonstrate that DS-MLP, though merely a vanilla MLP structure (the final model), can achieve state-of-the-art performance across three widely used benchmarks, offering a scalable and efficient solution for large-scale recommendation systems. Our code is available at https://github.com/RUCAIBox/DS-MLP.