SpecFormer: Mitigating Embedding and Attention Collapse via Spectral-Aware Transformer for Recommendation
SpecFormer introduces spectral softening and spectral-aware attention to mitigate embedding and attention collapse in recommendation Transformers.
Key Findings
Methodology
This paper analyzes the spectral properties of Transformer models in recommendation scenarios, revealing that data heterogeneity and long-tail distributions cause spectral collapse. It proposes a learnable spectral softening module to balance singular values, a spectrum-softened attention mechanism operating in spectral space, and Taylor expansion-based spectral residual position encoding. Theoretically, these components break the vicious cycle of spectral collapse, enabling deeper stacking. The approach is validated through extensive experiments on industrial and public datasets, demonstrating significant improvements in recommendation accuracy and attention effective rank.
Key Results
- On industrial and public datasets, SpecFormer outperforms SOTA models such as OneTrans and RankMixer, with CTR improvements of 1.34% and a 16.72% increase in orders. Attention effective rank increases by 30%, confirming enhanced spectral diversity. Ablation studies show spectral softening and spectral attention are critical for performance gains.
- Compared to baseline Transformers, SpecFormer maintains stable deep stacking, with higher attention rank and richer feature representations, especially on long-tail features and heterogeneous data.
- In real-world deployment, online A/B tests reveal CTR gains of 1.34%, and order volume increases by 16.72%, validating industrial applicability.
Significance
This work addresses the fundamental spectral collapse problem in recommendation Transformers, providing a theoretical and practical framework to enable deep stacking. By leveraging spectral space modeling, it mitigates data heterogeneity effects, enhances feature expressiveness, and pushes the boundary of Transformer scalability in recommendation systems, with broad implications for both academia and industry.
Technical Contribution
The paper introduces a spectral softening module, spectral-aware attention, and Taylor-based spectral residual encoding, which together fundamentally alter the spectral dynamics of the model. Theoretical analysis demonstrates how low-rank spectral structures bias gradients, and the proposed architecture effectively counteracts this, enabling deep model stacking. The engineering design facilitates scalable, stable, and expressive recommendation models.
Novelty
This is the first work to systematically incorporate spectral analysis into Transformer architecture for recommendation, proposing spectral softening and spectral attention mechanisms explicitly designed to address spectral collapse. Unlike prior regularization or frequency domain methods, it dynamically modulates spectral components to improve stability and expressiveness, representing a significant innovation.
Limitations
- The spectral softening parameters are learned, which may be sensitive to data quality and noise, potentially affecting robustness in extreme scenarios.
- Additional spectral modules increase model complexity and computational cost, requiring careful efficiency trade-offs.
- Current validation is limited to CTR prediction; applicability to other recommendation tasks and multi-modal data remains to be explored.
Future Work
Future research will focus on adaptive spectral softening mechanisms, multi-modal feature integration, and robustness enhancements. Extending the approach to multi-task and cross-domain recommendation, as well as optimizing inference efficiency, are promising directions to broaden industrial impact.
AI Executive Summary
Transformer architectures have revolutionized recommendation systems by enabling complex feature interactions. However, in recommendation scenarios, these models often suffer from spectral collapse, where the embedding and attention matrices become dominated by a few principal components. This leads to poor scalability and degraded performance, especially in deep stacks. The core challenge stems from the inherent data heterogeneity and long-tail distributions typical in recommendation data, which induce a severe spectral imbalance. To address this, the authors propose SpecFormer, a spectral-aware Transformer that introduces a learnable spectral softening module, spectrum-softened attention, and Taylor expansion-based spectral residual position encoding. These innovations work together to balance the singular value distribution, model feature interactions in spectral space, and explicitly incorporate spectral biases. Theoretically, the paper demonstrates that low-rank spectral structures cause gradient concentration on dominant components, creating a vicious cycle of collapse. Empirical results on industrial and public datasets show that SpecFormer significantly outperforms existing methods, with CTR improvements of 1.34% and a 16.72% increase in orders, while maintaining stable deep stacking capabilities. Deployment in a real-world platform confirmed its effectiveness, with online A/B tests validating substantial business gains. This work fundamentally advances the understanding of spectral dynamics in recommendation Transformers and offers a scalable, robust architecture for future large-scale recommendation systems. Future directions include adaptive spectral regulation, multi-modal integration, and further robustness enhancements, promising to expand the applicability and performance of spectral-aware models in diverse recommendation contexts.
Deep Analysis
Background
Recommendation systems的演进经历了从传统因子分解模型到深度学习，再到Transformer架构的快速发展。Transformer凭借其强大的特征交互能力，在工业界如Facebook的Deep Learning Recommendation Model（DLRM）和Google的多模态推荐中得到广泛应用。然而，深层Transformer在推荐中面临谱崩溃问题，即特征嵌入和注意力矩阵的谱结构退化，导致模型性能下降。此前的研究多集中在正则化和频域变换，但未从谱结构角度系统性解决崩溃问题。随着数据的异质性和长尾分布的加剧，模型的谱特性逐渐退化，影响特征表达和模型可扩展性，成为制约深层Transformer在推荐中的应用瓶颈。
Core Problem
核心问题在于推荐场景中Transformer的谱崩溃，表现为嵌入退化和注意力低秩。异质数据和长尾分布导致特征的奇异值极度偏斜，少数主导奇异值占据绝大部分能量，抑制次要信息。模型在深层堆叠时，低秩的注意力矩阵限制了特征交互的丰富性，梯度偏向主导谱方向，形成恶性循环。这不仅影响模型性能，也严重限制了模型的深度扩展，成为推荐系统中深层Transformer应用的瓶颈。
Innovation
本文提出谱软化模块、谱空间注意力机制和Taylor展开的谱残差位置编码，创新性地从谱角度缓解崩溃问题。谱软化通过动态调节奇异值分布，平衡谱能量；谱注意力在谱空间建模特征交互，避免单一主导；谱残差编码引入谱偏置，增强模型的谱表达能力。这些设计打破了低秩陷阱，提升深层模型的稳定性和表达能力。理论分析证明低秩谱结构引发的梯度偏向机制，模型堆叠深度显著增加。
Methodology
- �� 输入特征通过分组和投影形成token矩阵X。• 在每层中，H(l)通过奇异值分解（SVD）得到U(l)、Σ(l)、V(l)。• 采用可学习的幂律软化参数τ，将奇异值σ(l)i进行平滑，形成Σ(l)*。• 计算谱软化的Query、Key矩阵，保持Value在原空间。• 在谱空间中用softmax计算注意力，避免谱崩溃。• 引入Taylor展开的谱残差位置编码，提供谱偏置。• 多层堆叠，模型在深层中保持稳定的谱结构，提升性能。
Experiments
在工业平台和公开数据集（如MovieLens、Criteo）上，比较SpecFormer与SOTA模型（如OneTrans、RankMixer）。指标包括CTR、订单转化率和attention有效秩。采用不同层数和参数设置，验证深层堆叠能力。进行消融实验，分析谱软化、谱注意力和位置编码的贡献。训练采用Adam优化，调节学习率，早停策略，确保公平对比。
Results
SpecFormer在工业数据集上CTR提升1.34%，订单增长16.72%，明显优于对比模型。深层模型中，attention的有效秩提升30%，模型稳定性增强。消融实验显示谱软化和谱注意力机制是性能提升的关键。在长尾特征和异质数据中表现尤佳，验证谱结构调节的有效性。
Applications
可应用于大规模工业推荐平台，提升广告点击率和商品转化率。模型支持多样化特征和复杂场景，适合深层堆叠，满足工业性能需求。未来结合多模态信息，拓展多任务和跨域推荐，推动个性化推荐智能升级。
Limitations & Outlook
模型依赖谱参数学习，在极端异质或噪声数据中可能表现不佳。引入额外参数增加复杂度，影响推理效率。当前验证主要在CTR任务，泛化到其他推荐任务和多模态场景仍需验证。未来需优化算法效率和鲁棒性，提升实际应用适应性。
Plain Language Accessible to non-experts
想象你在厨房准备一道复杂的菜肴。每种食材代表不同的特征，厨师（模型）需要把它们混合成美味佳肴。传统厨师用固定方法混合，但如果某些食材特别多或少，他可能只关注主要的几样，就像模型只关注少数特征。这会让菜变得单调。为了让厨师更公平地使用所有食材，他可以用一种特殊的调料（谱软化）平衡各个食材的比例，让每个都能发挥作用。这样做出来的菜肴更丰富、更美味，也能做得更复杂。本文提出的SpecFormer就像给厨师配备了这种“调料”，让他在做菜时能充分利用所有食材，做出更好吃的菜。
ELI14 Explained like you're 14
想象你在学校帮忙做披萨。平时你只用一种酱料和几种配料，但如果只关注主要配料，其他的小配料就会被忽略，就像模型只关注少数几个特征一样。现在，你得到了一瓶神奇的调料，可以让所有配料都变得一样重要，不管是少的还是多的。这样，你做的披萨就会有各种各样的味道，特别丰富。这瓶调料就像论文里的谱软化机制，它让模型在处理不同特征时变得更公平、更全面。这样，模型就能更聪明、更强大，能做出更好的推荐，就像你的披萨变得更好吃一样。
Abstract
Transformer architectures have achieved remarkable success across diverse domains; however, directly applying their standard self-attention mechanism to recommendation often yields suboptimal performance, sometimes even trailing behind well-designed simple recommendation models. In this paper, we reveal that this performance bottleneck stems from severe embedding and attention collapse unique to recommendation scenarios. The heterogeneity and long-tail nature of recommendation data lead to a severe spectral collapse dominated by a few principal singular values. We further theoretically demonstrate that this triggers a vicious cycle in recommendation model's forward and backward propagation, which accelerates embedding and attention collapse and limits the model's scaling capability with increased depth. To address these issues, we propose SpecFormer, a novel Spectral-Aware Transformer designed for mitigating embedding and attention collapse in recommendation. Specifically, SpecFormer introduces 1) a Learnable Spectral Softening module to dynamically smooth the singular values distribution of the input token embeddings; 2) a Spectrum-softened Attention mechanism to model feature interaction under a more uniform spectral distribution space; 3) a Spectral Residual Position Encoding via Taylor expansion of singular values, explicitly providing a spectral inductive bias for feature interactions. Extensive experiments on one industrial and two public datasets demonstrate that SpecFormer significantly outperforms state-of-the-art baselines. Notably, SpecFormer has been successfully deployed in a real-world commercial recommender system and exhibits exceptional scaling capabilities: stacking SpecFormer layers actively improves the attention effective rank and recommendation performance.