SPEAR: Selection-aware Personalized End-to-end Adaptive Rewriting and Retrieval for Community Search
SPEAR enhances community search with selection-aware adaptive rewriting, boosting semantic similarity by 18.2%.
Key Findings
Methodology
SPEAR employs a dual-embedding backbone, multiplicative gating aggregator, and dynamic rewrite selector for personalized end-to-end adaptive rewriting and retrieval. The dual-embedding backbone uses auxiliary loss and gradient isolation to protect recall-side semantics, the multiplicative gating aggregator eliminates generic-word shortcuts, and the dynamic rewrite selector generates request-specific rewrite weights and user-query-conditioned scale and bias terms.
Key Results
- In 100K industrial search sessions, SPEAR improved rewrite semantic similarity@10 by 18.2% and click recall@10 by 99.5%.
- In online A/B testing, query-view CTR increased by 0.259%, and average reading depth by 0.733%.
- The SPEAR system is fully deployed on Dewu's community search platform, with open-source code available.
Significance
SPEAR addresses the structural misalignment between rewrite quality and retrieval effectiveness in traditional systems by establishing direct supervision between rewriting and retrieval. This approach not only enhances semantic similarity but also significantly boosts user engagement, offering substantial academic and industrial value.
Technical Contribution
SPEAR introduces dual-embedding isolation and multiplicative gating to prevent CTR-driven semantic degradation and implements end-task supervision for rewrite selection through a dynamic rewrite selector. These technical breakthroughs result in significant performance improvements in personalized retrieval.
Novelty
SPEAR is the first to introduce dual-embedding isolation and multiplicative gating in personalized retrieval, addressing the generic-word dominance effect. Compared to existing path models, SPEAR achieves higher user engagement while maintaining semantic fidelity.
Limitations
- SPEAR may underperform with long-tail queries due to fewer rewrite candidates.
- The system's reliance on user behavior data may introduce bias.
Future Work
Future research could explore improving SPEAR's performance without extensive user behavior data and further optimizing the rewrite selector for more complex query scenarios.
AI Executive Summary
In e-commerce search, query reformulation is crucial for bridging user intent and retrieval effectiveness. However, existing systems often optimize rewrite quality and retrieval effectiveness separately, leading to structural misalignment. SPEAR addresses this issue by introducing a selection-aware personalized end-to-end adaptive rewriting and retrieval framework. Its core technologies include a dual-embedding backbone, multiplicative gating aggregator, and dynamic rewrite selector, ensuring rewrites remain faithful to user intent while effectively enhancing retrieval.
Experimental results show that SPEAR significantly improves rewrite semantic similarity and click recall in offline evaluations. In online tests, user engagement also sees substantial improvement. SPEAR's deployment demonstrates its practical value, bringing deeper user engagement and higher retrieval precision to community search platforms.
Despite SPEAR's strong performance in many areas, it still faces limitations in handling long-tail queries and reliance on user behavior data. Future research directions include optimizing the rewrite selector for more complex query scenarios and improving system performance in data-scarce environments.
Deep Analysis
Background
With the rapid development of e-commerce, query reformulation has become a key technology for improving search engine performance. Traditional methods often rely on simple synonym expansion, while modern systems have evolved into multi-path recall mechanisms. However, these systems face structural misalignment between rewrite quality and retrieval effectiveness, leading to deviations from user intent.
Core Problem
Existing path models perform well in personalized recommendations but tend to favor generic rewrites in search, causing semantic drift. The challenge is to enhance retrieval effectiveness while maintaining fidelity to user intent.
Innovation
SPEAR addresses these issues through the following innovations: 1) Dual-embedding isolation to protect semantic structure; 2) Multiplicative gating to eliminate generic-word shortcuts; 3) Dynamic rewrite selector for request-specific rewrite weights and calibration.
Methodology
- �� Dual-embedding backbone: Protects recall-side semantics through auxiliary loss and gradient isolation.
- �� Multiplicative gating aggregator: Ensures rewrites score high only with high confidence and relevance.
- �� Dynamic rewrite selector: Generates request-specific rewrite weights and calibration terms.
Experiments
Experiments were conducted on 100K industrial search sessions using semantic similarity@10 and click recall@10 as primary metrics. The baseline system was the existing production system, with SPEAR showing significant improvements across all metrics.
Results
SPEAR improved rewrite semantic similarity@10 by 18.2% and click recall@10 by 99.5% in offline evaluations. In online testing, query-view CTR increased by 0.259%, and average reading depth by 0.733%.
Applications
SPEAR is fully deployed on Dewu's community search platform, significantly enhancing user engagement and retrieval precision. Its methods are applicable to scenarios requiring high-precision and personalized retrieval.
Limitations & Outlook
SPEAR's performance with long-tail queries and in data-scarce environments needs improvement. Future research could explore optimizing system performance in these scenarios.
Plain Language Accessible to non-experts
Imagine you're shopping in a supermarket, and SPEAR is like a smart shopping assistant. It not only recommends products based on your shopping list but also adjusts according to your shopping habits and preferences. For instance, if you want to buy a bottle of milk, SPEAR will recommend the best product based on your brand preference and price sensitivity, rather than simply suggesting the best-selling brand. This way, you can find products that meet your needs while saving time and effort.
ELI14 Explained like you're 14
Imagine you're playing a game, and SPEAR is like a super helper. You need to find treasures hidden on the map, and SPEAR gives you suggestions based on your gaming style and preferences. For example, if you like adventure, it will recommend exploring mysterious places; if you like collecting, it will tell you where to find rare items. This way, you can find treasures faster and enjoy the game more!
Glossary
Dual-Embedding Backbone
A technique for protecting recall-side semantic structure using auxiliary loss and gradient isolation.
Used in SPEAR to prevent CTR-driven semantic degradation.
Multiplicative Gating Aggregator
A mechanism ensuring rewrites score high only with high confidence and relevance.
Used to eliminate generic-word shortcuts.
Dynamic Rewrite Selector
A component generating rewrite weights and calibration terms based on requests.
Used in SPEAR for request-specific rewrite selection.
Semantic Similarity
A metric measuring the semantic consistency between rewrites and the original query.
Used to evaluate SPEAR's rewrite effectiveness.
Click Recall
A metric measuring the correlation between user clicks and retrieval results.
Used to evaluate SPEAR's retrieval effectiveness.
Open Questions Unanswered questions from this research
- 1 How to improve SPEAR's performance in data-scarce environments? Current methods' reliance on user behavior data may introduce bias.
- 2 How to optimize the rewrite selector for handling long-tail queries? These queries have fewer rewrite candidates.
Applications
Immediate Applications
E-commerce Search Optimization
SPEAR can enhance search precision and user experience on e-commerce platforms, suitable for scenarios requiring high-precision retrieval.
Long-term Vision
Intelligent Recommendation Systems
SPEAR's technology can be used to develop smarter recommendation systems, improving the quality of personalized services and user satisfaction.
Abstract
Query reformulation bridges user intent and retrieval in e-commerce search, yet production systems optimize rewrite quality and retrieval effectiveness separately, leaving the two stages structurally misaligned. Path-based architectures unify them end-to-end but were designed for personalization, where relevance is not an explicit constraint-search additionally requires the rewrite to remain faithful to the user's stated query intent. Transplanted directly, these models learn a shortcut we term the generic-word dominance effect: they favor generic rewrites that score well on paths but drift from query intent. To address this, we propose SPEAR (Selection-aware Personalized End-to-end Adaptive Rewriting and Retrieval), which integrates three components that each target one failure mode: (1) a dual-embedding backbone with auxiliary loss and gradient isolation that shields recall-side semantics from being eroded by CTR-driven ranking signals; (2) a multiplicative gating aggregator that lets a rewrite score high only when both its confidence and item relevance are strong, eliminating the generic-word shortcut; (3) a Dynamic Rewrite Selector that jointly generates request-specific rewrite weights and user-query-conditioned scale and bias terms, allowing both rewrite preference and relevance calibration to adapt to each request. Offline evaluation on 100K held-out industrial search sessions shows that the proposed framework improves rewrite semantic similarity@10 by +18.2 and click recall@10 by +99.5 over the production baseline. In online A/B testing, SPEAR achieves +0.259 in query-view CTR and +0.733 in average reading depth, confirming that improved rewrite selection translates into stronger retrieval and deeper user engagement. The proposed SPEAR system has been fully deployed in Dewu's community search platform since 2025. Our code is available at https://github.com/mallocagi1-cell/spear.