Efficiency and Effectiveness of SPLADE Models on Billion-Scale Web Document Title
Proposes SPLADE and Extended-SPLADE models with pruning strategies for billion-scale web document retrieval, balancing effectiveness and efficiency.
Key Findings
Methodology
This study employs sparse lexical models SPLADE and its extension, combined with document-centric pruning, top-k query term selection, and boolean term thresholding to optimize large-scale web retrieval. Training uses 260 million query-document pairs, leveraging a pretrained Korean BERT backbone, with batch negative sampling loss. The extended model supports custom vocabularies, trained on 600 million web documents. Evaluation metrics include FLOPS, latency, and semantic similarity scores (SSS@10). The approach systematically analyzes trade-offs between retrieval effectiveness and computational cost across datasets from tens of millions to billions of titles.
Key Results
- On billion-scale data, Extended-SPLADE achieves SSS@10 of 0.7642, surpassing BM25, with a 30% reduction in FLOPS and 40% latency decrease. Pruning strategies reduce latency over 50%, maintaining over 95% effectiveness. Short queries (1.5 words) see latency increase from 1.96s to 2.39s, validating robustness across query lengths. Combining multiple pruning strategies yields near BM25 latency with high semantic relevance, demonstrating practical deployment potential.
- In long query scenarios, SPLADE outperforms BM25 in effectiveness but has higher latency (~1.9×). Short queries reveal greater latency for SPLADE (~4.75s), while extended models reduce this to 2.39s, confirming the benefit of pruning. Effectiveness metrics remain high after pruning, with minimal performance loss, confirming the models' suitability for real-time large-scale search.
- Multi-strategy pruning (document, query, threshold) effectively balances efficiency and effectiveness. Configurations like qk7-dk10 achieve latency close to BM25 while preserving high semantic scores, validating the combined approach for industrial deployment in search engines handling billions of documents.
Significance
This work advances sparse retrieval methods, demonstrating their viability in billion-scale web search. By integrating pruning techniques, it addresses the core challenge of balancing high retrieval quality with computational efficiency. The findings provide a practical framework for deploying scalable, semantic-aware search engines, reducing hardware costs and response times. It bridges the gap between research and real-world application, paving the way for next-generation large-scale retrieval systems that are both fast and accurate.
Technical Contribution
The paper introduces a novel combination of sparse lexical models with multi-level pruning strategies, including document-level, query-level, and threshold-based filtering. It employs FLOPS regularization to control model sparsity and optimize training. The extended vocabulary support enhances flexibility, while pruning significantly reduces inference latency. These innovations collectively enable high-accuracy retrieval on billion-scale datasets with manageable computational costs, marking a step forward in scalable sparse retrieval research.
Novelty
This is the first work to combine multi-strategy pruning with support for extended vocabularies in sparse lexical models, achieving a practical balance between effectiveness and efficiency at billion-scale. Unlike prior dense or single-strategy sparse models, this approach integrates semantic similarity scoring with flexible vocabulary support, addressing the scalability bottleneck in large-scale retrieval. The comprehensive evaluation demonstrates its superiority over traditional methods in real-world scenarios.
Limitations
- The models' performance drops slightly with extremely short queries, as pruning may remove relevant semantic cues. Parameter tuning for pruning strategies remains complex and dataset-dependent.
- Despite efficiency gains, training and inference still require substantial hardware resources, limiting deployment in resource-constrained environments.
- The approach relies on predefined thresholds and fixed pruning parameters, which may not adapt well to dynamic query distributions or evolving datasets.
Future Work
Future directions include developing adaptive pruning mechanisms that dynamically adjust parameters based on query context, integrating multi-modal data (images, videos) for richer retrieval, and exploring lightweight architectures to reduce hardware dependency. Additionally, extending the models to multilingual and cross-lingual retrieval scenarios will broaden their applicability.
AI Executive Summary
In the era of information explosion, web search engines face the challenge of retrieving relevant documents from billions of titles rapidly and accurately. Traditional keyword-based methods like BM25, while efficient, often lack semantic understanding, especially for complex queries. Deep learning models, such as dense vector retrieval, improve semantic matching but are computationally expensive at scale. Sparse lexical models like SPLADE have emerged as a promising middle ground, capturing semantic nuances with manageable costs.
This study systematically compares BM25, SPLADE, and its extended variant, Extended-SPLADE, on datasets ranging from tens of millions to billions of web titles. The core innovation lies in integrating multi-level pruning strategies—document-level static pruning, top-k query term selection, and boolean term thresholding—to significantly reduce inference latency while maintaining high retrieval effectiveness. The experiments demonstrate that Extended-SPLADE, with these optimizations, achieves a balanced performance: SSS@10 reaches 0.7642, outperforming BM25, with FLOPS reduced by nearly 30% and latency cut by over 40%.
Furthermore, the analysis across different query lengths reveals that while sparse models incur higher latency for short queries, pruning strategies mitigate this issue effectively. Combining multiple techniques results in near-BM25 latency with high semantic relevance, validating the approach's practicality for real-world large-scale search engines. These findings mark a significant step toward deploying efficient, semantic-aware retrieval systems capable of handling billions of documents, with broad implications for industry and academia. Despite some limitations in extremely short query scenarios and hardware demands, the proposed framework opens avenues for future adaptive, multi-modal, and multilingual retrieval research, promising more intelligent and scalable search solutions.
Deep Analysis
Background
信息检索技术经历了从关键词匹配到深度语义理解的演变。早期模型如BM25依赖词频和逆文档频率，效果有限。深度学习引入后，密集向量检索（如DPR）提升了语义捕获能力，但计算成本高昂。稀疏表示模型如SPLADE结合词汇扩展和稀疏编码，兼顾效率与效果，成为研究热点。近年来，面对超大规模数据，如何实现高效检索成为核心难题。此前研究多集中在模型优化和索引压缩，但在实际部署中仍受延迟和硬件限制困扰。
Core Problem
大规模网页标题检索面临两大瓶颈：一是模型在复杂语义理解上的不足，二是计算资源消耗过大。传统稠密模型虽效果优异，但难以满足实时性需求。稀疏模型虽具潜力，但在亿级数据场景下的效率仍待提升。如何在保证检索效果的同时，降低延迟和资源消耗，是当前亟需解决的问题。
Innovation
本文提出支持扩展词表的稀疏模型，结合多层次剪枝策略，创新点包括：1）引入文档级和查询级剪枝，有效减少无关信息；2）采用布尔阈值控制匹配严格度，平衡效果与效率；3）结合FLOPS正则化，优化模型训练。此方案区别于传统单一稀疏模型，兼顾大规模场景的实用性与效果，突破了模型在超大数据集上的性能瓶颈。
Methodology
- �� 训练数据：260百万对（查询-文档），利用预训练BERT基础，采用批内负采样。• 模型结构：支持词汇扩展的SPLADE，扩展版本支持自定义词表。• 损失函数：采用FLOPS正则化，控制模型稀疏性。• 剪枝策略：文档剪枝（只保留重要词）、top-k词筛选（只处理影响最大的词）、布尔阈值（设定最小匹配比例）。• 训练流程：预训练+微调，结合多策略优化。• 评估指标：FLOPS、延迟、SSS@10，覆盖不同查询长度。• 实验环境：亿级网页标题数据库，硬件配置高性能GPU集群。
Experiments
设计包括在小（2000万标题）和大（9亿标题）数据集上对比BM25、SPLADE、扩展SPLADE的检索效果与效率。调优剪枝参数，验证不同k值的影响。采用多查询长度测试，确保模型鲁棒性。指标包括MRR@10、SSS@10、延迟时间。通过参数敏感性分析，验证剪枝策略的效果，确保模型在实际大规模场景中的适应性。
Results
扩展SPLADE在亿级数据集上，SSS@10达0.7642，优于BM25，提升15%；FLOPS降低约30%，延迟缩短至原模型的60%。剪枝策略使延迟降低超过50%，同时保持95%以上的检索效果。短查询（1.5词）时，延迟由1.96秒升至2.39秒，验证稀疏模型在不同场景中的适应性。多策略结合实现了接近BM25的延迟和高语义匹配，验证模型在实际应用中的实用性。
Applications
该模型适用于大规模搜索引擎、企业知识库、电子商务商品检索等场景，能在保证检索效果的同时，显著降低硬件成本和响应时间。对实时性要求高的应用尤为关键。未来可结合用户行为数据，优化个性化推荐和动态剪枝策略，提升用户体验。
Limitations & Outlook
模型在极短查询（如1-2词）场景下表现仍有限，剪枝参数需调优，可能影响效果。高效剪枝依赖预定义参数，泛化能力有限。训练和推理仍需较大硬件资源，限制了普及。未来需探索自适应剪枝和多模态融合技术，以增强模型鲁棒性和适应性。
Plain Language Accessible to non-experts
想象你在一个巨大的图书馆里找书。传统方法就像用关键词搜索，找到的书可能很多，但不一定是你真正想要的。稀疏表示模型像是给每本书贴标签，只有重要的关键词会被记住，这样可以快速筛选出相关书籍。扩展模型就像是给标签库增加更多词汇，让搜索更灵活。为了让搜索更快，图书馆会把不重要的标签去掉，或者只关注最关键的几个词。这样一来，即使书很多，找到你想要的也变得更快更准。这就像在海量信息中用聪明的方法找到宝藏，不仅快，还很准。
Abstract
This paper presents a comprehensive comparison of BM25, SPLADE, and Expanded-SPLADE models in the context of large-scale web document retrieval. We evaluate the effectiveness and efficiency of these models on datasets spanning from tens of millions to billions of web document titles. SPLADE and Expanded-SPLADE, which utilize sparse lexical representations, demonstrate superior retrieval performance compared to BM25, especially for complex queries. However, these models incur higher computational costs. We introduce pruning strategies, including document-centric pruning and top-k query term selection, boolean query with term threshold to mitigate these costs and improve the models' efficiency without significantly sacrificing retrieval performance. The results show that Expanded-SPLADE strikes the best balance between effectiveness and efficiency, particularly when handling large datasets. Our findings offer valuable insights for deploying sparse retrieval models in large-scale search engines.