CCFormer: Efficient Cross-Field Interaction and Hierarchical Sequence Compression for Industrial Recommendation at Tencent
CCFormer integrates cross-field attention with hierarchical sequence compression, boosting recommendation accuracy by 3.57% CTR and 1.71% ad revenue, with 2.21× faster training.
Key Findings
Methodology
CCFormer employs a feature-field separated cross-attention mechanism, combining long-sequence subspace token mixing with hierarchical sequence compression. It uses multi-layer interaction blocks to facilitate heterogeneous feature fusion, enhanced by relative temporal-position encoding. The subspace token mixing reduces computational complexity, while the progressive compression enlarges receptive fields. Extensive experiments on public benchmarks and Tencent's industrial data demonstrate superior performance, with online A/B tests showing significant CTR and revenue gains and training speed improvements. The architecture balances rich sequence information with efficiency, suitable for large-scale industrial deployment.
Key Results
- On Taobao and KuaiRec datasets, CCFormer achieved 13.93% and 10.36% AUC improvements, respectively, outperforming SOTA models. On Tencent's industrial dataset, AUC increased from 77.66% to 77.94%, GAUC rose by 0.5%. Online, CTR increased by 3.57%, ad revenue by 1.71%, and training speed by 2.21×. These results validate its effectiveness across scenarios.
- The feature-domain separated cross-attention effectively captures long-term user preferences and short-term behaviors. Subspace token mixing reduces complexity while preserving behavioral richness. Hierarchical compression expands receptive fields across layers, balancing detail and abstraction. Multi-scenario validation confirms the model's robustness and industrial applicability.
- AB testing in video and ad scenarios shows consistent performance boosts, confirming real-world benefits. The model's efficiency gains enable faster training and higher throughput, making it suitable for large-scale deployment. Overall, CCFormer sets a new benchmark for long-sequence recommendation in industry.
Significance
This work addresses the core challenge of long-sequence modeling in industrial recommendation systems, offering a scalable, efficient architecture that preserves rich behavioral signals. It bridges the gap between high-capacity models and real-time constraints, enabling more accurate personalization at scale. The practical deployment at Tencent demonstrates its readiness for industrial use, marking a significant step forward in recommendation research. Its innovative combination of feature separation, token mixing, and hierarchical compression provides a blueprint for future systems aiming to handle ultra-long sequences efficiently, with broad implications for e-commerce, advertising, and content platforms.
Technical Contribution
CCFormer introduces a novel feature-domain separated cross-attention mechanism, enabling efficient heterogeneous feature interaction. It innovatively applies subspace token mixing to reduce long-sequence complexity, coupled with a hierarchical compression strategy that progressively enlarges the receptive field. The model leverages relative temporal-position encoding within subspaces, enhancing sequence understanding without quadratic costs. These design choices collectively improve scalability, accuracy, and training efficiency, setting new standards for industrial recommendation architectures. The integration with Tencent's infrastructure further demonstrates its engineering feasibility and impact.
Novelty
This is the first work to combine feature-domain separated cross-attention with hierarchical sequence compression in an industrial recommendation context. Unlike prior methods that rely solely on sequence truncation or sparse attention, CCFormer preserves full long-term behavioral information through layered compression and local token mixing. Its innovative architecture allows for detailed, scalable modeling of ultra-long sequences, addressing a critical bottleneck in deploying Transformer-based models at scale. This approach significantly advances the state-of-the-art in long-sequence recommendation, especially for large-scale industrial systems.
Limitations
- Despite hierarchical compression, extremely long sequences (beyond tens of thousands of behaviors) may still suffer from information loss or insufficient granularity, especially in highly noisy or redundant data scenarios.
- The model's complexity and hardware requirements remain substantial, potentially limiting deployment on resource-constrained devices or environments.
- Further research is needed to adapt the architecture for multimodal data and dynamic user interests, as current design mainly focuses on behavioral sequences.
Future Work
Future directions include developing adaptive compression strategies that dynamically balance detail and efficiency, integrating multimodal signals such as images and text, and exploring reinforcement learning for interest modeling. Additionally, optimizing the architecture for resource-limited environments and extending its applicability to real-time, multi-task scenarios will be key to broader industrial adoption.
AI Executive Summary
In the realm of industrial recommendation systems, effectively modeling long user behavior sequences remains a significant challenge. Traditional Transformer-based models, while powerful, suffer from quadratic complexity, making them impractical for real-time, large-scale applications. To address this, Tencent researchers developed CCFormer, an innovative architecture that combines cross-field feature interaction with hierarchical sequence compression. This design allows the model to efficiently capture both short-term and long-term user preferences without incurring prohibitive computational costs.
CCFormer’s core innovation lies in its feature-domain separated cross-attention mechanism, which enables heterogeneous feature fields—such as user profiles, historical behaviors, and target items—to interact explicitly. This approach enhances the model’s ability to understand complex user-item relationships. Complementing this, the model employs a subspace token mixing module that partitions long behavior sequences into manageable chunks, applying localized operations to reduce complexity while maintaining behavioral richness.
Furthermore, the hierarchical sequence compression strategy progressively condenses the sequence representation across layers, expanding the receptive field and capturing multi-scale preferences. This layered approach balances detailed short-term signals with abstract long-term interests, significantly reducing computational overhead. Extensive experiments on public benchmarks like Taobao and KuaiRec, as well as Tencent’s large-scale industrial data, demonstrate that CCFormer outperforms existing models, achieving up to 13.93% AUC improvement and notable online performance gains.
In real-world deployment, CCFormer has delivered a 3.57% CTR increase and a 1.71% revenue lift, validating its industrial practicality. Its training speed is over twice as fast as previous baselines, enabling faster iteration and scaling. Looking ahead, future work will explore adaptive compression, multimodal integration, and resource-efficient variants, aiming to further enhance recommendation accuracy and efficiency. Overall, CCFormer marks a significant advancement in long-sequence modeling, bridging the gap between high-capacity models and industrial constraints, and paving the way for smarter, faster recommendation systems.
Deep Dive
Abstract
Recent studies in industrial recommendation systems have demonstrated that sequential recommendation models built upon self-attention can benefit from predictable scaling laws by increasing sequence length and model capacity. However, practical recommender systems impose strict latency and resource constraints, making it challenging to balance computational overhead with fine-grained feature interaction. In this paper, we propose CCFormer, an efficient Transformer backbone that unifies cross-field feature interaction and compressed long-sequence modeling for industrial recommendation. Specifically, CCFormer combines feature-field separated cross attention with long-sequence subspace token mixing to exploit long-term preference signals across heterogeneous feature domains. A hierarchical sequence compression strategy with progressively expanded receptive fields enables efficient long-sequence modeling with reduced information loss. Extensive experiments on two public benchmarks and a large-scale industrial dataset demonstrate that CCFormer consistently outperforms state-of-the-art baselines. Online A/B tests in a video recommendation scenario and an advertising ranking scenario at Tencent further validate its industrial practicality, yielding a 3.57% CTR gain and a 1.71% advertising revenue lift, respectively, while accelerating model training by 2.21x over the strong HSTU baseline. CCFormer has been fully deployed in Tencent's production recommendation system, serving the main traffic of both scenarios.