Document Similarity Enhanced IPS Estimation for Unbiased Learning to Rank

The evolution of learning to rank (LTR) models has transformed search and recommendation systems, with methods like RankSVM, LambdaRank, and neural models improving relevance. However, training data derived from user clicks suffers from inherent biases, notably position bias, where higher-ranked documents are disproportionately clicked regardless of relevance. To address this, approaches like Inverse Propensity Scoring (IPS) and Doubly Robust (DR) estimators have been developed, aiming to debias click data. Despite success, these methods primarily rely on positional information, ignoring content relevance. Recent advances in deep learning enable rich content representations, yet their integration into bias correction remains limited. This context underscores the need for content-aware bias mitigation techniques.

The core challenge lies in accurately estimating document relevance from biased click data. Position bias skews the observed interactions, making low-ranked relevant documents underrepresented. Existing IPS-based methods correct for position bias but fail to consider content similarity, leading to residual bias, especially for documents that are relevant but rarely clicked due to their position. This results in suboptimal ranking models that favor popular or highly-ranked content, reducing fairness and relevance. Addressing this gap requires a method that leverages content similarity to better estimate the true relevance of low-ranked documents, thus improving the bias correction process.

This paper proposes IPSsim, a novel bias correction framework that incorporates content similarity into IPS estimation. Key innovations include: 1) defining a similarity measure between documents using cosine, Euclidean, or Manhattan distances; 2) computing an average similarity score between each document and the top-ranked relevant set; 3) integrating this score into the IPS formula via a weighted parameter α, balancing content relevance and position bias correction. This approach enables the model to assign higher importance to low-ranked documents that are content-wise similar to relevant high-ranked documents, effectively reducing residual bias. The method is flexible, allowing different similarity metrics, and is validated through extensive experiments.

• �� Collect historical rankings h and click data c for each query. • Compute similarity sim(i, im) between each document i and top-ranked relevant documents T using cosine, Euclidean, or Manhattan distances. • Calculate Avgsim(i, T) as the average similarity to T. • Adjust IPS weights by combining original propensity scores with Avgsim(i, T), modulated by parameter α. • Formulate IPSsim as a weighted sum of position-based and content-based components. • Tune α via validation, and apply the model to simulate user clicks with varying top-n and click counts. • Compare IPSsim against baseline estimators like IPS, DR, and MIPS, analyzing NDCG improvements.

Experiments utilize ISTELLA-S and MSLTR-WEB30K datasets, simulating user clicks with different click volumes (10^4, 10^6, 10^8) and top-n (10, 30, 50). Baselines include IPS, DR, MIPS, and variants with content similarity. Parameters such as similarity measure and α are optimized on validation sets. Performance is evaluated using NDCG over five independent runs, with statistical significance tests (p<0.05). The analysis examines the impact of similarity metrics, click volume, and top-n on bias mitigation and ranking quality. Ablation studies confirm the robustness of the content similarity integration.

Results show IPSsim achieves about 3% NDCG improvement over DR at n=50, 106 clicks on ISTELLA-S, with statistical significance. Across datasets, all three similarity measures (Cosine, Euclidean, Manhattan) yield comparable gains, with Cosine slightly better. Increasing click volume enhances bias correction effectiveness. In top-n≥30 scenarios, IPSsim consistently outperforms traditional IPS and content-agnostic methods, demonstrating its capacity to leverage content relevance for bias mitigation. The approach remains stable across different similarity metrics, confirming its robustness.

This method can be deployed in search engines, online recommendation systems, and personalized content delivery platforms, especially where click data is biased or sparse. By incorporating content similarity, systems can better identify relevant but low-ranked documents, improving fairness and user satisfaction. It also opens avenues for integrating deep content embeddings, enabling more nuanced bias correction in complex multimedia environments. The approach is suitable for large-scale online systems with sufficient computational resources.

The reliance on predefined similarity metrics may limit adaptability to highly diverse or noisy content. Computational overhead for similarity calculations can hinder real-time deployment. The method's effectiveness in extremely sparse click scenarios remains uncertain. Further research is needed to optimize similarity computation and extend online applicability, especially under dynamic content and user behavior changes.