SAM-D2Q: Aligning Multimodal Doc2Query with Search Demand and Conversion for E-commerce
SAM-D2Q enhances e-commerce search with multimodal document expansion, boosting GMV by 3.38%.
Key Findings
Methodology
SAM-D2Q framework comprises three stages: task-adapted multimodal supervised fine-tuning, multimodal data augmentation, and reinforcement-learning-based preference alignment. It optimizes pseudo-query generation using visual signals and business objectives.
Key Results
- Result 1: Offline experiments show SAM-D2Q outperforms traditional Doc2Query methods in retrieval performance, boosting GMV by 3.38%.
- Result 2: Online tests increase Pay Count by 2.27%.
- Result 3: Multimodal data augmentation improves perception of visual attributes.
Significance
This study optimizes e-commerce search by integrating multimodal information with business objectives, addressing shortcomings in semantic relevance and commercial value of traditional methods, significantly enhancing retrieval performance and business metrics.
Technical Contribution
Introduces an industrial multimodal Doc2Query framework, combining visual signals and business objectives optimization, offering new engineering possibilities and theoretical guarantees.
Novelty
First to implement multimodal Doc2Query under Boolean retrieval constraints, where expansion terms must provide recall gain and business utility.
Limitations
- Limitation 1: The model may overfit on low-quality data, affecting the validity of expansion terms.
- Limitation 2: Recognition of visual attributes depends on image quality.
Future Work
Future work may explore more complex visual signal processing and broader business objective optimization to further enhance model performance.
AI Executive Summary
E-commerce search often suffers from vocabulary mismatch between user queries and product titles. Traditional Doc2Query methods generate pseudo-queries to expand documents but are text-only, failing to optimize for business objectives. The SAM-D2Q framework integrates multimodal information with business objectives to optimize e-commerce search, significantly enhancing retrieval performance and business metrics.
SAM-D2Q comprises three stages: task-adapted multimodal supervised fine-tuning, multimodal data augmentation, and reinforcement-learning-based preference alignment. By optimizing pseudo-query generation using visual signals and business objectives, offline experiments show it outperforms traditional methods, while online tests show significant improvements in business metrics like GMV and Pay Count.
This study addresses shortcomings in semantic relevance and commercial value of traditional methods by integrating visual signals and business objectives optimization, providing a new solution for e-commerce search. Future work may explore more complex visual signal processing and broader business objective optimization to further enhance model performance.
Deep Analysis
Background
With the rapid growth of e-commerce, online shopping has become an integral part of daily life for billions of users worldwide. E-commerce search engines serve as the key gateway connecting user intent with massive product inventories. Although neural retrieval methods have made significant progress, keyword-based retrieval remains the backbone of large-scale e-commerce search systems due to its low latency, high interpretability, and robustness for cold-start products.
Core Problem
E-commerce search suffers from vocabulary mismatch, where user queries and product descriptions may express the same intent using different terms. Traditional Doc2Query methods mainly optimize semantic relevance based on textual data, overlooking critical business signals such as search Page Views and Conversion Rate.
Innovation
The SAM-D2Q framework optimizes e-commerce search by integrating multimodal information with business objectives. It generates expanded queries grounded in product content and valuable for downstream commercial performance by integrating visual signals, multimodal data synthesis, and business-aligned optimization.
Methodology
- �� Task-adapted multimodal supervised fine-tuning: Enhances vision-language understanding of product titles, images, and user queries.
- �� Multimodal data augmentation: Improves perception of key visual attributes and expansion coverage.
- �� Reinforcement-learning-based preference alignment: Encourages the model to generate pseudo-queries that better match user intent and commercial value.
Experiments
Experimental design includes offline experiments and online A/B testing. Offline experiments show SAM-D2Q outperforms traditional methods in retrieval performance, while online tests show significant improvements in business metrics like GMV and Pay Count.
Results
Offline experiments show SAM-D2Q outperforms traditional Doc2Query methods in retrieval performance, boosting GMV by 3.38%. Online tests increase Pay Count by 2.27%. Multimodal data augmentation improves perception of visual attributes.
Applications
SAM-D2Q can be directly applied to e-commerce search systems, enhancing the match between user queries and product titles, increasing business metrics like GMV and Pay Count.
Limitations & Outlook
The model may overfit on low-quality data, affecting the validity of expansion terms. Recognition of visual attributes depends on image quality. Future work may explore more complex visual signal processing and broader business objective optimization.
Plain Language Accessible to non-experts
Imagine you're shopping in a mall, looking for the perfect red dress. The store's tags might not clearly state 'red dress,' but you can find it by observing the color and style of the products. Similarly, SAM-D2Q analyzes product images and titles to generate queries that better match user search intent, making search results more aligned with user needs.
ELI14 Explained like you're 14
Hey there! Imagine you're playing a treasure hunt game. You have a map with some clues, but they might not be detailed enough. SAM-D2Q is like a super helper that uses the clues and pictures on the map to find more treasures, making your game more fun!
Glossary
Doc2Query
A method for document expansion by generating pseudo-queries.
Used to address vocabulary mismatch in e-commerce search.
GMV
A metric measuring the sales volume on an e-commerce platform.
Used to evaluate the commercial effect of SAM-D2Q.
Multimodal
Technology that combines multiple data types (e.g., text and images).
Used to enhance vision-language understanding in e-commerce search.
Reinforcement Learning
A machine learning method that optimizes model behavior through reward mechanisms.
Used to optimize the commercial value of pseudo-query generation.
Preference Alignment
The process of making model-generated results more aligned with user intent and business objectives.
Used to enhance the commercial utility of pseudo-queries.
Open Questions Unanswered questions from this research
- 1 How to maintain the validity of expansion terms on low-quality data?
- 2 How to further optimize visual signal processing to enhance model performance?
Applications
Immediate Applications
E-commerce Search Optimization
Optimizes e-commerce search by integrating multimodal information with business objectives, enhancing the match between user queries and product titles.
Long-term Vision
Intelligent Shopping Assistant
Provides a smarter shopping experience by optimizing multimodal information and business objectives.
Abstract
E-commerce search often suffers from vocabulary mismatch between user queries and merchant-authored product titles, since short titles cannot fully cover diverse user expressions or visual product attributes. Although Doc2Query alleviates this issue by generating pseudo-queries for document expansion, traditional methods are text-only and not optimized for e-commerce business objectives. As a result, they may produce semantically plausible but commercially ineffective expansions and miss key attributes present in product images. To this end, we propose E-commerce Search-Aligned Multimodal Doc2Query (SAM-D2Q), a business-aligned multimodal document expansion framework for e-commerce search under Boolean retrieval constraints. SAM-D2Q consists of three stages: (1) task-adapted multimodal supervised fine-tuning to enhance vision-language understanding of product titles, images, and user queries; (2) multimodal data augmentation to improve perception of key visual attributes and expansion coverage; and (3) reinforcement-learning-based preference alignment toward search business objectives, encouraging the model to generate pseudo-queries that better match user intent and commercial value. Offline experiments show that SAM-D2Q substantially improves retrieval performance over traditional Doc2Query methods. Deployed in the AliExpress production search system, SAM-D2Q improves online business metrics, increasing GMV by +3.38% and Pay Count by +2.27%.