RankUp: Towards High-rank Representations for Large Scale Advertising Recommender Systems
RankUp enhances representation rank via randomized permutation, multi-embedding, and global integration, boosting large-scale recommendation performance.
Key Findings
Methodology
RankUp integrates randomized permutation splitting, multi-embedding paradigms, global token integration, and cross-domain pre-trained embeddings to address representation collapse in deep recommendation models. Spectral metrics like effective rank evaluate the diversity of token representations across layers. The architecture employs stochastic feature shuffling, multiple embedding tables for features, holistic context via a global token, and external knowledge injection. Multi-task training with stabilized normalization and SwiGLU activations ensures effective optimization. Large-scale experiments on Tencent’s advertising datasets demonstrate significant improvements in GMV and AUC, validating the approach’s effectiveness.
Key Results
- In Tencent’s advertising platform, RankUp achieved GMV increases of 3.41%, 4.81%, and 2.12% across different scenarios, outperforming baseline RankMixer. Online AUC metrics showed consistent gains, especially in deep layers where effective rank remained high. Ablation studies confirmed each mechanism’s contribution, with global token and multi-embedding providing the largest improvements. The spectral effective rank metrics indicated better utilization of latent space, correlating with improved recommendation accuracy.
- Compared to traditional models, RankUp maintained higher spectral effective rank in deep layers, preventing the typical rank oscillation and collapse. Multi-mechanism synergy led to more distinguishable token representations, especially for sparse features. The approach demonstrated robustness across tasks, reducing overfitting and improving generalization in multi-task environments. Results on public datasets further confirmed its superior ability to preserve information in deep architectures.
- Ablation experiments revealed that removing randomized permutation or cross-domain embeddings significantly degraded performance, highlighting their critical roles. The architecture’s ability to sustain high-rank representations in deep layers directly translated into better online metrics and user engagement, confirming its industrial viability.
Significance
This work addresses a fundamental bottleneck in deep recommendation systems: the inability to fully exploit the latent space as models grow deeper. By enhancing the diversity and capacity of token representations, RankUp enables models to learn richer, more distinguishable features, leading to improved accuracy and robustness. Its successful deployment in Tencent’s large-scale advertising system demonstrates practical impact, offering a scalable solution to the longstanding problem of representation collapse. The approach bridges theoretical insights with engineering innovations, pushing the frontier of deep recommendation architecture design and setting a new standard for industrial applications.
Technical Contribution
RankUp introduces a multi-mechanism framework that explicitly enhances latent space diversity. Key innovations include randomized feature permutation, multi-embedding strategies, global contextual tokens, and cross-domain knowledge injection, all integrated within a MetaFormer backbone. The architecture leverages spectral effective rank metrics to guide design choices, ensuring sustained information utilization. The combination of stabilization techniques like PreNorm and SwiGLU activations facilitates deep training. These contributions collectively enable deep models to maintain high-rank representations, significantly advancing the state-of-the-art in industrial recommendation systems.
Novelty
This is the first comprehensive integration of randomized permutation splitting, multi-embedding, and global context mechanisms specifically targeting deep representation collapse in recommendation models. Unlike prior work that mainly focused on feature interaction improvements or parameter scaling, RankUp fundamentally enhances the latent space’s capacity and diversity. Its spectral effective rank evaluation provides a new theoretical perspective, and the combined mechanisms demonstrate superior empirical performance, marking a significant step forward in deep recommendation architecture design.
Limitations
- The increased complexity and parameter count may lead to higher computational costs, limiting deployment in resource-constrained environments.
- While effective in large-scale advertising, the approach’s generalization to other domains or modalities remains to be validated.
- Potential overfitting risks in scenarios with extremely sparse or high-dimensional features need further investigation, and mechanisms for efficiency improvements are necessary.
Future Work
Future research will focus on reducing model complexity and training costs, possibly through pruning or distillation. Integrating self-supervised learning and multi-modal data could further enrich representations. Extending the framework to other recommendation domains, such as e-commerce or content platforms, and exploring explainability and interpretability of high-rank representations are also promising directions.
AI Executive Summary
Deep recommendation systems have become essential in personalized content delivery, yet their effectiveness is often hampered by the phenomenon of representation collapse, especially as models deepen. Traditional architectures like RankMixer and Transformer-based models benefit from increased parameters and layers, but empirical evidence shows that their latent space utilization stagnates or deteriorates in deeper layers. This limits the models’ ability to distinguish subtle differences among users and items, constraining overall recommendation quality.
To address this, the authors propose RankUp, a novel architecture that explicitly enhances the diversity and capacity of token representations. Central to RankUp are mechanisms such as randomized permutation splitting, which decouples feature correlations; multi-embedding paradigms, which provide multiple geometric perspectives for features; global token integration, capturing holistic context; and cross-domain pre-trained embeddings, injecting external knowledge. These components work synergistically to maintain high spectral effective rank across layers, preventing the typical oscillation and collapse observed in deep models.
Extensive experiments on Tencent’s large-scale advertising datasets demonstrate that RankUp significantly outperforms baseline models like RankMixer, achieving GMV improvements of over 3% and online AUC gains. Ablation studies confirm the importance of each mechanism, with the combination yielding the best results. The architecture’s robustness across multiple tasks and scenarios underscores its industrial viability.
This work not only advances the theoretical understanding of representation capacity in deep recommendation models but also offers practical solutions for large-scale deployment. By enabling deeper, more expressive architectures, RankUp paves the way for more accurate, robust, and scalable personalized recommendation systems. Future directions include optimizing computational efficiency, extending to multi-modal data, and exploring interpretability, promising a new horizon for deep learning in industrial recommendation engines.
Deep Dive
Abstract
The scaling laws for recommender systems have been increasingly validated, where MetaFormer-based architectures consistently benefit from increased model depth, hidden dimensionality, and user behavior sequence length. However, whether representation capacity scales proportionally with parameter growth remains unexplored. Prior studies on RankMixer reveal that the effective rank of token representations exhibits a damped oscillatory trajectory across layers, failing to increase consistently with depth and even degrading in deeper layers. Motivated by this observation, we propose RankUp, an architecture designed to mitigate representation collapse and enhance expressive capacity through randomized permutation splitting over sparse features, a multi-embedding paradigm, global token integration and crossed pretrained embedding tokens. RankUp has been fully deployed in large-scale production across Weixin Video Accounts, Official Accounts and Moments, yielding GMV improvements of 3.41%, 4.81% and 2.12%, respectively.