UniScale: Synergistic Entire Space Data and Model Scaling for Search Ranking
UniScale combines entire-space data expansion and hierarchical Transformer architecture to boost search ranking performance.
Key Findings
Methodology
UniScale employs ES3 for full-space data augmentation, utilizing hierarchical label propagation and cross-domain searchification to enrich training signals. HHSFT integrates heterogeneous feature tokenization, hierarchical feature interaction, and user interest fusion, effectively modeling complex data distributions. Extensive experiments on large-scale industrial datasets demonstrate significant performance gains, with online A/B tests on Taobao showing 1.70% increase in user purchase rate and 2.04% in GMV, outperforming strong baselines.
Key Results
- In deployment on Taobao, UniScale improved user purchase rate by 1.70% and GMV by 2.04%. Offline evaluation showed a 0.32% increase in AUC at 300M parameters. Data augmentation via ES3 significantly increased training data richness, leading to better generalization. The architecture's innovations, including hierarchical feature interaction and interest fusion, surpassed performance limits of structure-only tuning. Ablation studies confirmed the importance of label propagation and cross-domain searchification.
Significance
This work addresses the fundamental bottleneck in industrial search systems—the limited information capacity of training data—by jointly optimizing data and architecture. It demonstrates that data quality and diversity are crucial for scaling model performance, providing a systematic framework for industrial AI. The approach enhances model robustness, generalization, and business metrics, setting a new standard for large-scale ranking systems.
Technical Contribution
The paper introduces ES3, a full-space data sampling system that mitigates bias and sparsity, and HHSFT, a hierarchical Transformer that captures high-order interactions among heterogeneous features and user interests. This integrated co-design approach enables models to leverage richer data and more expressive architectures, pushing beyond the limits of traditional structure-only tuning. The system is optimized for production deployment with feature pre-hashing, attention kernel fusion, and other efficiency techniques.
Novelty
UniScale uniquely combines full-space data augmentation with a hierarchical Transformer architecture tailored for heterogeneous industrial data. Its hierarchical label propagation and cross-domain searchification mechanisms are novel, addressing biases and data sparsity systematically. This integrated approach is the first to demonstrate significant performance improvements through joint data-architecture co-design in large-scale industrial search, distinguishing it from prior works that focus solely on model scaling or simple data augmentation.
Limitations
- The approach relies on large storage for expanded samples, increasing deployment costs. The complex architecture adds computational overhead, impacting training and inference speed. Its effectiveness on extremely heterogeneous or sparse data distributions remains to be validated. Future work should explore model compression, online updating, and adaptation to diverse industrial scenarios.
Future Work
Future directions include developing more efficient data compression techniques, online learning methods, and multi-modal extensions. Further research will focus on reducing computational costs, improving real-time adaptability, and extending the framework to multi-task and multi-modal industrial applications, broadening its impact.
AI Executive Summary
In the rapidly evolving landscape of industrial search systems, the quest for higher accuracy and user engagement faces a fundamental challenge: the saturation of performance gains from merely increasing model parameters. Traditional approaches, relying heavily on architectural improvements, encounter diminishing returns, especially when data quality and diversity are limited. This paper introduces UniScale, a comprehensive framework that jointly optimizes data and model architecture to unlock the full potential of large-scale ranking models.
The core innovation lies in the ES3 system, which constructs a high-quality, full-space training set by expanding samples within domains through hierarchical label propagation, and across domains via searchification. This process mitigates selection bias, alleviates label sparsity, and addresses cross-domain blind spots, resulting in a richer, more representative dataset. Complementing this, the HHSFT architecture employs heterogeneous feature tokenization, hierarchical feature interaction, and user interest fusion mechanisms. These components enable the model to effectively capture complex data distributions and user preferences, surpassing the performance ceiling of structure-only tuning.
Extensive offline experiments on billion-scale datasets demonstrate that UniScale achieves significant improvements over baseline models, with a 0.32% increase in AUC at 300 million parameters. Online deployment on Taobao's search platform confirms the effectiveness, with a 1.70% rise in user purchase rate and 2.04% increase in GMV. The results validate the importance of co-designing data and architecture, highlighting a new paradigm for industrial AI systems.
This work addresses the longstanding bottleneck of data limitations in large-scale models, emphasizing that richer, more diverse data combined with advanced architectures can break performance plateaus. It offers a scalable, practical solution for industrial applications, with potential extensions into multi-modal and real-time adaptive systems. Despite its success, challenges remain in optimizing storage, computational efficiency, and robustness across diverse scenarios, guiding future research directions.
Deep Analysis
Background
随着推荐和搜索系统在互联网中的普及，模型性能不断依赖于参数规模和数据量的增长。代表性工作如OneRec、HSTU、RankMixer等，推动了大规模预训练和特征交互的发展。然而，参数规模的边际收益逐渐递减，数据的质量和多样性成为瓶颈。传统采样策略如负采样和启发式增强，难以充分利用复杂异质数据，限制了模型的泛化能力。工业场景中，偏差、标签稀疏和异质性问题亟待解决，促使研究转向数据与模型的联合优化。
Core Problem
核心问题在于工业搜索中的数据偏差和标签稀疏，导致模型训练与实际推理存在差距。单纯扩展模型参数无法突破性能瓶颈，数据的多样性和质量不足限制了模型潜能的发挥。如何构建高质量、全空间的训练样本，兼顾异质性和偏差，是提升工业搜索模型性能的关键挑战。同时，模型架构需能有效建模复杂数据分布，避免性能极限。
Innovation
提出ES3系统，通过层级标签传播和跨域搜索化机制，丰富训练样本，缓解偏差和标签稀疏问题。创新HHSFT架构，采用异质特征块化、层级特征交互和用户兴趣融合，有效建模复杂异质数据。结合数据扩展与架构创新，突破只调优模型结构的性能极限，提升工业搜索模型的表达能力和泛化能力。
Methodology
- �� 构建全空间训练集：采用层级标签传播，增强偏差样本；跨域搜索化，将非搜索行为转化为搜索样本。• ES3机制：包括在域内扩展未曝光样本，传播交叉场景信号，生成丰富样本；跨域搜索化，利用相似性和层次策略合成搜索样本。• HHSFT架构：特征块化处理不同语义空间的特征，采用异质注意力机制保持语义信息；多层全局特征交互，融合跨域用户兴趣。• 利用专家融合和门控注意力，动态调节跨域信息流。• 训练过程中采用特征预哈希、注意核融合等优化措施，保证效率。• 实验在淘宝大规模数据集上，比较不同采样策略、架构变体，验证性能提升。
Experiments
在淘宝搜索平台进行线上线下验证，使用亿级用户行为数据，指标包括AUC、转化率和GMV。对比基线模型，验证ES3和HHSFT的贡献。进行消融实验，分析标签传播和搜索化机制的作用。调优超参数如样本比例、层数和注意力头数，确保模型稳定性和性能。通过A/B测试，验证模型在实际场景中的效果，确保提升的商业价值。
Results
UniScale模型在淘宝平台上线后，用户购买转化率提升1.70%，GMV增长2.04%，优于传统模型。离线AUC指标提升0.32%，在300M参数规模下表现优异。数据扩展策略显著丰富训练样本，模型泛化能力增强。架构创新突破性能瓶颈，验证数据-模型协同优化的有效性。消融实验显示层级标签传播和搜索化机制是性能提升的关键因素。
Applications
该方法适用于工业搜索、推荐系统和广告投放等场景，尤其在数据偏差严重、异质性高的环境中表现优越。可作为大规模模型训练的标准流程，提升模型的泛化能力和商业指标。未来可结合多模态信息和实时在线学习，推动行业智能化升级。
Limitations & Outlook
当前方法依赖大量存储资源存放扩展样本，增加部署成本。模型复杂度提升带来训练和推理时间的增加，需优化效率。对极端异质数据分布的适应性尚未充分验证，未来需探索更高效的模型压缩和在线更新策略。
Plain Language Accessible to non-experts
想象你在一家大型工厂里，生产各种不同的产品。过去，工厂只用一种简单的机器，生产效率受到限制。现在，工厂引入了更聪明的机器（模型），但这些机器只能根据已有的生产数据做决策。问题是，工厂的原材料（数据）不够丰富，导致新产品的质量难以提升。为了改善这个问题，工厂开始收集更多不同类型的原材料（数据扩展），并用更先进的机器（架构）来处理这些材料。通过让机器学习不同材料的特性（特征交互），以及了解工厂中不同工序的关系（兴趣融合），工厂的生产效率大大提高。这个过程就像用更丰富的材料和更聪明的机器一起工作，最终让工厂生产出更好、更快的产品。
ELI14 Explained like you're 14
想象你在一个大厨房里做饭，平时只用一种食材和简单的厨具，做出来的菜也就一般。现在，你开始用各种新鲜的食材，还用不同的厨具组合，试图做出更好吃的菜。这就像在用更丰富的原料和更聪明的厨具来提升菜的质量。你还发现，把不同食材的味道结合起来，能做出新奇的味道（特征交互）；而且，了解不同食材的搭配和厨具的用法（兴趣融合），让菜变得更丰富。通过不断试验和学习，你的厨艺变得更厉害，做出更受欢迎的菜。这就像论文中的方法，用丰富的数据和创新的架构，让搜索系统变得更聪明、更强大。
Abstract
Recent advances in Large Language Models (LLMs) have inspired a surge of scaling research in industrial search, advertising, and recommendation systems. However, existing approaches focus mainly on architectural improvements, overlooking the critical synergy between data and architecture design. We observe that scaling model parameters alone exhibits diminishing returns, and that the performance degradation caused by complex heterogeneous data distributions is often irrecoverable through model design alone. In this paper, we propose UniScale, a novel co-design framework that jointly optimizes data and architecture to unlock the full potential of model scaling. UniScale includes two core parts: (1) ES$^3$ (Entire-Space Sample System), a high-quality data scaling system that expands the training signal beyond conventional sampling strategies through intra-domain expansion with hierarchical label attribution and cross-domain searchification; and (2) HHSFT (Heterogeneous Hierarchical Sample Fusion Transformer), a novel architecture that effectively models the complex heterogeneous distribution of scaled data via Heterogeneous Hierarchical Feature Interaction and Entire Space User Interest Fusion, thereby surpassing the performance ceiling of structure-only model tuning. Extensive experiments on large-scale industrial datasets demonstrate that UniScale achieves significant improvements and exhibits clear scaling trends. Online A/B tests on a real-world E-commerce search platform confirm that UniScale consistently outperforms strong production baselines, achieving 1.70% and 2.04% increases in user purchase and Gross Merchandise Value (GMV).