Unidentified and Confounded? Understanding Two-Tower Models for Unbiased Learning to Rank

The evolution of learning to rank has transitioned from explicit relevance feedback to leveraging implicit signals like clicks. Early models focused on position bias correction, exemplified by cascade models and pairwise approaches. Two-tower neural architectures, such as Deep Relevance Matching Models (DRMM) and neural click models, have become industry standards due to their scalability and ability to model complex bias-relevance interactions. Despite these advances, the challenge of bias amplification persists, especially when models are trained on production logs collected under deterministic policies. Prior works like Guo et al. (2016) and Zhuang et al. (2020) addressed bias correction but overlooked the impact of data collection strategies on model identifiability and bias amplification. Recent studies highlight that stronger logging policies, while improving relevance, can inadvertently entrench biases, leading to performance deterioration—a phenomenon requiring deeper theoretical understanding.

The core issue lies in the inherent unidentifiability of two-tower models under certain data collection schemes, especially deterministic logging policies that lack document swaps or feature diversity. When models are trained on such biased data, parameters related to relevance and bias become entangled, causing bias amplification and degraded ranking performance. This problem is compounded by the fact that existing models assume certain independence conditions that are violated in practice. The industry faces a pressing need to understand the precise conditions for parameter recovery, how logging policies influence bias propagation, and how to design data collection and modeling strategies that ensure robust, unbiased ranking. Addressing these issues is critical for deploying fair and reliable systems.

This work introduces a rigorous theoretical framework for analyzing the identifiability of two-tower models, emphasizing the roles of document swaps and feature overlap. It extends prior models by formalizing the conditions under which parameters can be uniquely recovered, using graph connectivity and feature support overlap. The authors propose a novel sample weighting scheme that adjusts for bias amplification caused by logging policies, supported by theoretical guarantees and simulation validation. Unlike previous approaches that treat bias correction heuristically, this work integrates model structure, data collection, and bias mitigation into a unified framework, offering both theoretical insights and practical tools for industry deployment.

• �� Formalize the probabilistic structure of additive two-tower models, defining relevance and bias logits. • Use graph theory to model position relationships, analyzing how random document swaps connect position nodes and influence parameter shifts. • Derive conditions for identifiability, showing that position swaps or feature overlap ensure unique parameter recovery. • Develop a propensity-based sample weighting scheme that compensates for bias amplification, adjusting click likelihoods based on logging policy probabilities. • Conduct simulation experiments on datasets like MSLR30K, varying logging policies, feature overlaps, and model misspecification levels. • Evaluate models using ranking metrics (NDCG, MAP) and parameter estimation errors, comparing weighted vs. unweighted approaches to validate theoretical predictions.

Simulations involve generating click data under different logging policies—deterministic, randomized, and mixed—on the MSLR30K dataset. The experiments test the impact of document swaps and feature overlaps on parameter identifiability. They evaluate the effectiveness of the proposed sample weighting in reducing bias amplification, measuring improvements in ranking metrics and parameter estimation accuracy. Multiple runs ensure statistical significance. The experiments also compare models trained with and without weighting across scenarios with varying degrees of bias and model misspecification, illustrating the robustness of the approach.

Results show that when position swaps or feature overlaps are present, parameter estimation errors decrease by over 20%, and NDCG scores improve by approximately 5%. Without these conditions, models suffer from bias amplification, with performance dropping by up to 10%. The sample weighting method consistently reduces bias effects, leading to more stable rankings and accurate relevance estimation. Empirical data confirms that bias amplification correlates with logging policy strength, and the proposed method effectively mitigates this, validating the theoretical framework. These findings demonstrate that careful data collection and weighting strategies are crucial for unbiased, robust models.

The insights apply directly to search engines, recommendation systems, and online advertising platforms, guiding data collection policies and model training procedures. Implementing randomized document swaps or ensuring feature overlap can enhance model identifiability. The proposed weighting scheme can be integrated into existing training pipelines to improve bias robustness. Long-term, these strategies promote fairer content ranking, reduce bias propagation, and improve user trust. They also inform industry standards for logging policies and data collection practices, fostering more equitable information access across diverse applications.

The analysis assumes feature continuity and graph connectivity, which may not hold in high-dimensional sparse data. The effectiveness of the weighting scheme depends on accurate logging policy probabilities, which can be challenging in real-world settings. The experiments are primarily simulation-based, and real-world bias mechanisms may be more complex, requiring further validation. Additionally, the approach introduces hyperparameters that need tuning, potentially increasing computational costs and complexity. Future work should address these limitations by developing adaptive methods and validating on large-scale industrial data.