Understanding the Effects of the Baidu-ULTR Logging Policy on Two-Tower Models

Learning to rank has evolved from classical IR methods like BM25 and TF-IDF to deep neural models, with ULTR (Unbiased Learning to Rank) emerging to address bias issues. Two-tower architectures, combining relevance and bias estimation, have gained industrial prominence due to their simplicity and effectiveness. Prior work mainly relied on simulation or small datasets, leaving the real impact of logging bias in large-scale, real-world data less understood. Recent datasets like Baidu ULTR provide an opportunity to empirically validate these assumptions, but the actual influence of bias remains uncertain.

The core challenge lies in the fact that logging policies used to collect click data may correlate relevance with position bias, violating the independence assumption crucial for unbiased learning. This confounding can lead models to learn shortcuts, relying on biased features rather than true relevance signals. While theoretical conditions for bias impact are well-understood, empirical validation on large-scale real-world data is scarce. The question remains whether bias significantly degrades model performance or if models are inherently robust, and how to effectively mitigate bias when it does matter.

This work innovates by: 1) estimating the logging policy bias strength directly from large-scale Baidu ULTR data using a LambdaMART model, 2) systematically comparing bias correction techniques (Dropout and backdoor adjustment) within two-tower models, 3) revealing that despite bias conditions being met, the impact on model performance is minimal, and 4) highlighting the discrepancy between click-based and expert annotations, which suggests that user clicks may not fully reflect true relevance. These insights challenge existing assumptions and provide a more nuanced understanding of bias effects in real-world systems.

• �� Data collection: Baidu ULTR dataset with 4.8 million query-document pairs, including BERT embeddings and classical features.
• �� Bias estimation: Train LambdaMART to predict logged positions, validate bias strength via nDCG@10.
• �� Model training: Develop relevance and bias towers, incorporate bias correction methods (Dropout, backdoor adjustment).
• �� Performance evaluation: Use expert annotations (380k pairs) and click data, compare models across bias intensities.
• �� Analysis: Group queries by estimated bias strength, assess performance variations, validate bias impact.
• �� Statistical validation: Confidence intervals for metrics, ablation studies to isolate effects.

Experiments involve training LambdaMART to estimate logging bias, validating bias strength with high accuracy (nDCG@10 0.933). Two-tower models are trained with and without bias correction, evaluated on expert-labeled test sets. Results show no performance gain from bias correction, with slight declines observed. The analysis includes grouping queries by bias strength, revealing no significant performance degradation on well-ranked queries, challenging the assumption that bias harms model effectiveness. The comparison with expert models underscores the robustness of the two-tower architecture in real-world data.

The LambdaMART model accurately predicts logged positions (nDCG@10 0.933), confirming bias presence. Two-tower models outperform naive models but do not benefit from bias correction techniques; performance remains stable or slightly declines. Expert models trained on annotations outperform click-trained models significantly (nDCG@10 0.539 vs. 0.375), indicating user click bias may not be as impactful as assumed. The results suggest that bias, while theoretically present, does not substantially impair model effectiveness in this setting.

Findings inform industrial ranking system design, indicating that aggressive bias correction may be unnecessary in some scenarios. Practical applications include search engines, recommendation systems, and online advertising, where robustness to bias is critical. The methodology for bias estimation and correction can be integrated into existing pipelines to improve fairness and relevance without risking performance loss. Additionally, insights into user behavior versus expert relevance can guide data collection and annotation strategies.

The study focuses on two specific bias correction methods, excluding others like gradient reversal. Bias estimation relies on predictive models, which may introduce errors. Results are specific to Baidu ULTR and may not generalize across platforms. The discrepancy between user clicks and true relevance warrants further investigation, and computational costs for large-scale bias estimation remain high. Future work should explore broader bias mitigation techniques and cross-platform validation.