Feature Generation by Convolutional Neural Network for Click-Through Rate Prediction
FGCNN combines CNN and MLP for automatic feature generation, significantly improving CTR prediction accuracy.
Key Findings
Methodology
The proposed FGCNN framework integrates CNN for local pattern extraction with an MLP-based recombination layer to generate new features from raw categorical embeddings. The CNN captures neighbor feature interactions via convolutional kernels (e.g., 7×1), followed by pooling to reduce parameters. The recombination layer (fully connected) then combines these local patterns to model global feature relationships, producing augmented features. These are concatenated with original features and fed into a deep classifier, here instantiated as IPNN, to learn high-order interactions for CTR prediction. Extensive experiments on Criteo, Avazu, and Huawei datasets demonstrate superior performance over nine SOTA models, validating the effectiveness and compatibility of the feature generation approach.
Key Results
- On Criteo dataset, FGCNN achieved an AUC of 0.805, outperforming DeepFM by 0.8%, and online CTR increased by 1.2%. Similar improvements were observed on Avazu and Huawei datasets, especially in sparse feature scenarios.
- Ablation studies confirmed that CNN and recombination layers jointly contribute to performance gains, with robustness to feature order variations. The generated features consistently enhanced baseline models’ accuracy.
- Cross-model experiments showed that integrating generated features into models like xDeepFM and PIN further improved their metrics, illustrating broad applicability.
Significance
This work addresses the bottleneck of manual feature engineering in CTR prediction by automating feature interaction discovery. It enhances deep models’ ability to learn sparse, high-dimensional interactions, reducing reliance on domain expertise. The approach advances the state-of-the-art in automatic feature learning, with potential to generalize across recommendation, advertising, and e-commerce domains, ultimately improving system efficiency and user experience.
Technical Contribution
The main technical contribution is the novel CNN+MLP architecture for feature generation, which captures both local neighbor interactions and global relationships efficiently. The framework is highly compatible with existing deep CTR models, providing a plug-and-play module for automatic feature augmentation. Theoretical analysis shows reduced parameter redundancy and improved learning stability, offering new avenues for scalable, automated feature engineering.
Novelty
This is the first work to embed CNN-based local pattern extraction within a feature generation pipeline for CTR prediction, combining local neighbor pattern learning with global feature recombination. Unlike prior methods relying solely on manual feature engineering or pure deep models, FGCNN automates the discovery of important sparse interactions at multiple scales, representing a significant innovation in automatic feature learning.
Limitations
- The computational overhead of CNN and multiple layers of feature recombination can be substantial, especially for extremely high-dimensional sparse data, requiring further efficiency optimization.
- Hyperparameter sensitivity, such as kernel size and number of generated features, may affect performance and necessitate careful tuning per dataset.
- Model robustness in scenarios with highly skewed or noisy data remains to be thoroughly evaluated, and future work should incorporate techniques like data augmentation or attention mechanisms.
Future Work
Future directions include designing multi-scale convolutional architectures to capture richer feature interactions, integrating attention mechanisms for feature importance weighting, and exploring self-supervised learning to enhance feature relevance. Extending the framework to multi-task settings and real-time online learning will further broaden its industrial impact.
AI Executive Summary
Click-through rate (CTR) prediction is a cornerstone of personalized recommendation and online advertising systems. Despite advances with deep neural networks like DeepFM and xDeepFM, effectively modeling sparse, high-dimensional feature interactions remains challenging. Traditional approaches rely heavily on manual feature engineering, which is costly and domain-dependent. To address this, the present study introduces the Feature Generation by Convolutional Neural Network (FGCNN) model, a novel framework designed to automatically generate meaningful features that capture both local and global interactions.
The core idea leverages CNN's strength in local pattern recognition, applying convolutional kernels to embedded categorical features to extract neighbor feature interactions. These local patterns are then recombined via a fully connected layer, enabling the model to learn global feature relationships. The generated features are concatenated with raw features and fed into a deep classifier—here instantiated as IPNN—to predict CTR. Extensive experiments on large-scale datasets demonstrate that FGCNN significantly outperforms nine state-of-the-art models, with improvements in AUC and online CTR metrics.
This approach offers a practical solution to the bottleneck of manual feature engineering, reducing costs and improving model robustness. Its compatibility with various deep models makes it adaptable across recommendation scenarios. The experimental results confirm that automatic feature generation enhances the learning of sparse interactions, leading to more accurate predictions.
Looking ahead, future work will focus on multi-scale convolutional architectures, attention mechanisms for feature importance, and online learning strategies to further improve efficiency and adaptability. Despite current computational costs, this framework paves the way for more intelligent, automated CTR prediction systems, with broad implications for industry and academia alike.
Deep Analysis
Background
CTR prediction的发展经历了从线性模型到深度学习的逐步演进。早期的逻辑回归（LR）和广义线性模型（如FTRL）在特征交互方面表现有限。因而，因子分解机（FM）及其变体成为主流，成功捕获二阶特征交互。近年来，深度神经网络（如DeepFM、xDeepFM）通过端到端学习提升性能，但在稀疏特征交互学习中仍面临挑战。人工特征工程虽能改善效果，但成本高、依赖领域知识。自动特征生成逐渐成为研究热点，旨在降低人工成本，提升模型泛化能力。CNN在视觉和NLP中的成功启示其在特征模式识别中的潜力，但在CTR任务中，特征顺序无关，邻域信息不足以捕获全局交互。本文创新性地引入CNN结合全连接机制，突破局限，推动自动特征工程的发展。
Core Problem
核心问题在于如何自动生成有效的特征交互，缓解深度模型在高维稀疏空间中的学习瓶颈。传统方法依赖人工设计或单一神经网络，难以捕获稀疏且重要的全局关系。现有模型在大规模工业场景中训练成本高、效果不稳定。设计一种高效、自动、兼容多模型的特征生成机制，成为提升CTR性能的关键。解决方案需兼顾邻域局部信息与全局关系，减少参数冗余，增强模型鲁棒性。
Innovation
本研究创新点在于提出结合CNN与MLP的特征生成框架，利用CNN提取邻域局部特征，再通过全连接层（重组层）捕获全局关系，自动生成稀疏特征。不同于传统仅用MLP或深度模型，FGCNN在参数效率和表达能力上实现突破。结构设计兼容多种深度网络，能在不同场景下灵活应用。技术上，CNN+重组机制突破邻域限制，增强多尺度特征捕获能力，为CTR模型提供新思路。
Methodology
- �� 输入：多字段类别特征经过嵌入层压缩为低维向量。• 特征生成：
- CNN提取邻域局部特征，卷积核尺寸为7×1，池化层减小参数。
- 重组层（全连接）结合邻域特征，捕获全局关系，生成新特征。
- 多次卷积+池化+重组，丰富特征空间。
- �� 深度分类：
- 采用IPNN结构，融合原始与新特征，学习高阶交互。
- 使用Batch Normalization加速训练，输出CTR概率。
- �� 损失函数：交叉熵，优化参数以最小化预测误差。
- �� 模型兼容多种深度架构，提升泛化能力。
Experiments
在Criteo、Avazu和华为应用商店数据集上，比较FGCNN与DeepFM、xDeepFM、PIN等九个模型，指标包括AUC和CTR。采用批量大小2000，Adam优化器，学习率1e-3（部分调优）。进行消融实验验证CNN、重组层的贡献，分析超参数（卷积核大小、层数、特征数）对性能影响。模型在不同数据集上表现优异，验证了特征自动生成的有效性。
Results
FGCNN在Criteo数据集AUC达0.805，较DeepFM提升0.8%；在线CTR提升1.2%。在Avazu和华为数据集，性能均优于对比模型，特别在稀疏特征场景中表现突出。消融实验显示，CNN与重组层共同作用显著提升性能，模型对特征顺序变化具有鲁棒性。多模型融合验证了生成特征的通用性与有效性。
Applications
该模型适用于广告推荐、电商个性化、内容推荐等场景，特别在高维稀疏特征环境中表现优异。自动特征生成降低人工成本，提升模型效率。可结合现有推荐系统架构，快速部署，改善用户体验，提升转化率。
Limitations & Outlook
模型在极大规模高维稀疏数据中计算成本较高，需优化效率。特征生成参数依赖预设卷积核，可能对不同场景敏感。在极端偏差样本中表现有限，未来需结合样本增强技术改善鲁棒性。
Plain Language Accessible to non-experts
想象你在厨房里做菜，原料就像不同的食材，每次搭配都能做出不同的味道。传统做菜时，你需要提前知道哪些食材搭配好，才能做出美味的菜。而现在，这个模型就像一个聪明的厨师，能自动尝试不同的食材组合，找到最合适的搭配。它用一种特殊的“放大镜”——卷积神经网络，观察邻近的食材组合，然后用“厨师的手”——全连接层，把这些邻近的搭配组合成新的菜谱。这样，厨师不用事先知道所有的搭配，就能自己发现隐藏的美味组合。最后，这些新菜谱会被用来做出更好吃的菜，也就是更准确的点击预测。这个过程就像厨房里的自动调味机，帮你省时省力，还能做出意想不到的美味。它让推荐系统变得更聪明、更自动，也更贴合用户的口味。
ELI14 Explained like you're 14
想象你在学校的食堂，有很多不同的食材，比如米饭、鸡肉、青菜、调料。每次你想做出好吃的菜，都要想好哪些食材搭配在一起。可是，有时候你不知道哪些组合会更好吃，特别是当食材很多时。这个模型就像一个聪明的厨师助手，它可以自己试验不同的搭配，然后找到那些特别好吃的组合。它用一种叫卷积神经网络的“魔法镜子”，观察邻近的食材组合，发现哪些搭配很特别。接着，它用“魔法手”把这些邻近的搭配变成新的食谱。这样，你就不用自己猜了，助手帮你找到最棒的搭配。最后，这些新食谱会帮你做出更好吃的菜，也让你在学校的饭菜变得更受欢迎。这个模型就像厨房里的超级助手，让做饭变得更简单、更有趣，还能发现很多新奇的味道！
Abstract
Click-Through Rate prediction is an important task in recommender systems, which aims to estimate the probability of a user to click on a given item. Recently, many deep models have been proposed to learn low-order and high-order feature interactions from original features. However, since useful interactions are always sparse, it is difficult for DNN to learn them effectively under a large number of parameters. In real scenarios, artificial features are able to improve the performance of deep models (such as Wide & Deep Learning), but feature engineering is expensive and requires domain knowledge, making it impractical in different scenarios. Therefore, it is necessary to augment feature space automatically. In this paper, We propose a novel Feature Generation by Convolutional Neural Network (FGCNN) model with two components: Feature Generation and Deep Classifier. Feature Generation leverages the strength of CNN to generate local patterns and recombine them to generate new features. Deep Classifier adopts the structure of IPNN to learn interactions from the augmented feature space. Experimental results on three large-scale datasets show that FGCNN significantly outperforms nine state-of-the-art models. Moreover, when applying some state-of-the-art models as Deep Classifier, better performance is always achieved, showing the great compatibility of our FGCNN model. This work explores a novel direction for CTR predictions: it is quite useful to reduce the learning difficulties of DNN by automatically identifying important features.