PosIR: Position-Aware Heterogeneous Information Retrieval Benchmark
PosIR employs length-controlled bucketing and span-based relevance to diagnose position bias across 10 languages, revealing that models favor early content, especially in long documents.
Key Findings
Methodology
PosIR adopts a length-controlled bucketing strategy, grouping queries by positive document length to analyze positional effects within each bucket. It employs span-based relevance annotations generated via LLMs, combined with precise reference spans, to isolate position bias from length effects. The framework integrates position sensitivity indices (PSI) and gradient-based saliency analysis to quantify and interpret internal model mechanisms. This comprehensive approach enables systematic diagnosis of biases across multilingual, multi-domain datasets, leveraging algorithms like span extraction, relevance contrast, and attention mechanism analysis.
Key Results
- Models exhibit significant performance degradation on documents exceeding 1536 tokens, with correlation to short-text benchmarks dropping sharply. Most models show primacy bias, favoring early document segments, while some unexpectedly favor recent content. Gradient analysis uncovers two distinct internal attention behaviors correlating with these biases, confirming their mechanistic basis. These biases intensify with document length and are consistent across languages and domains, highlighting a pervasive issue in current models.
- PosIR reveals that long, multilingual documents pose challenges not captured by existing benchmarks, exposing limitations in current short-text evaluation paradigms. The framework's detailed bias quantification and mechanistic insights provide a foundation for targeted improvements.
- The study demonstrates that position bias is widespread and exacerbated in long-text, multilingual retrieval, emphasizing the need for bias-robust models. PosIR's standardized, fine-grained diagnostic tools enable researchers to systematically evaluate and mitigate such biases, advancing the development of fairer, more reliable retrieval systems.
Significance
This work addresses a critical gap in understanding position bias in deep retrieval models, especially in long, multilingual contexts. By providing a standardized, comprehensive benchmark, PosIR facilitates systematic diagnosis and comparison across models, fostering improvements in fairness and robustness. Its insights into internal mechanisms open avenues for bias mitigation strategies, impacting both academia and industry. As retrieval systems become integral to information access, ensuring their fairness and effectiveness across diverse scenarios is vital. PosIR's methodology and datasets set a new standard for bias analysis, promising to influence future research directions and practical applications in search engines, question answering, and knowledge retrieval.
Technical Contribution
The key technical innovations include the length-controlled bucketing strategy that isolates positional effects, the use of span-based relevance annotations generated via LLM prompting, and the introduction of position sensitivity indices (PSI) combined with gradient saliency mechanisms to interpret internal attention behaviors. These methods collectively enable a systematic, quantitative diagnosis of position bias across diverse languages and domains. The framework's ability to disentangle length effects from positional bias and its mechanistic insights into internal attention behaviors represent significant advancements over prior short-text, monolingual evaluations, offering a new paradigm for bias analysis and model improvement.
Novelty
This is the first comprehensive, multilingual benchmark explicitly designed to diagnose position bias in long-text retrieval scenarios. Its core innovations—length-controlled bucketing, span-based relevance annotation, and mechanistic analysis via gradient saliency—distinguish it from existing benchmarks like MS MARCO or BEIR, which lack fine-grained, position-aware relevance labels and multi-language coverage. PosIR's systematic approach and detailed internal mechanism exploration set a new standard in bias diagnosis, enabling targeted model improvements.
Limitations
- 尽管PosIR采用多重验证机制，但生成的跨度标注仍可能存在误差，影响偏差分析的精确性。
- 目前评估集中在嵌入模型，尚未涵盖端到端检索系统，未来需扩展。
- 偏差根源尚未完全解析，模型内部机制的深入研究仍需进行，尤其在跨模态和多任务场景中。
Future Work
未来将结合模型内部注意力机制的可解释性研究，深入分析偏差形成的深层次原因。计划扩展多模态、多任务场景下的偏差诊断，推动偏差鲁棒模型的设计。同时，结合用户行为数据，优化偏差校正策略，提升实际应用中的公平性和效果。
AI Executive Summary
In real-world information retrieval, the relevance of a document to a query often depends not just on semantic content but also on the position of that content within the document. Traditional benchmarks mainly evaluate short texts or monolingual data, neglecting the complex biases that emerge in long, multilingual documents. PosIR introduces a comprehensive, standardized benchmark that addresses this gap by systematically diagnosing position bias across ten languages and thirty-one domains. Its core innovation lies in a length-controlled bucketing approach, which groups queries by positive document length, allowing precise analysis of positional effects without confounding length-related performance degradation.
Using span-based relevance annotations generated via large language models, PosIR associates each query with a specific reference span within the document. This fine-grained relevance labeling enables detailed analysis of how the physical location of information influences retrieval performance. The framework incorporates position sensitivity indices (PSI) and gradient-based saliency analysis, revealing that most models exhibit a primacy bias—favoring early content—and that this bias intensifies with document length. Interestingly, some models show an unexpected recency bias, favoring recent content, which is mechanistically linked to different internal attention behaviors.
Extensive experiments across 10 state-of-the-art embedding models demonstrate that performance on long documents (>1536 tokens) correlates poorly with short-text benchmarks like MMTEB, exposing significant limitations in current evaluation paradigms. The findings highlight the need for bias-aware model development, especially for long, multilingual retrieval tasks. PosIR's standardized, multi-domain datasets and mechanistic insights provide a valuable diagnostic tool for researchers aiming to build more robust, fair retrieval systems. Future work will focus on integrating internal attention analysis, expanding to multimodal data, and developing bias mitigation strategies, ultimately advancing the fairness and effectiveness of real-world retrieval applications.
Deep Dive
Plain Language Accessible to non-experts
想象你在图书馆找一本书，书的内容散布在不同章节。传统的搜索就像只看前几页，忽略了后面的重要信息。而PosIR就像给每一段内容贴上标签，告诉你它在书中的具体位置，然后帮你找到最相关的部分。不管书多长，重要信息都能被公平地找到，就像老师公平对待每一部分内容一样。它用一种特别的方法，把书的每一部分都标记得很清楚，确保搜索系统不会只喜欢前面几章，而忽略了中间或后面的内容。这样，搜索结果更全面、更公平，也更贴近我们实际的需要。
ELI14 Explained like you're 14
想象你在学校图书馆找资料，很多时候你会发现，老师喜欢让你看书的前几页，因为觉得那里最重要，但其实有些关键内容藏在后面。PosIR就像给每一段内容贴标签，告诉你它在书中的具体位置，然后帮你找到最相关的部分。它还会告诉你，模型是不是只喜欢看前面几页，还是也会关注后面。这样一来，无论书多长，重要信息都能被公平找到，就像老师公平对待每一部分内容一样。这个方法让搜索变得更智能、更公平，也更贴近我们实际需要的全面信息。
Abstract
In real-world documents, the information relevant to a user query may reside anywhere from the beginning to the end. This makes position bias -- a systematic tendency of retrieval models to favor or neglect content based on its location -- a critical concern. Although recent studies have identified such bias, existing analyses focus predominantly on English, fail to disentangle document length from information position, and lack a standardized framework for systematic diagnosis. To address these limitations, we introduce PosIR (Position-Aware Information Retrieval), the first standardized benchmark designed to systematically diagnose position bias in diverse retrieval scenarios. PosIR comprises 310 datasets spanning 10 languages and 31 domains, with relevance tied to precise reference spans. At its methodological core, PosIR employs a length-controlled bucketing strategy that groups queries by positive document length and analyzes positional effects within each bucket. This design strictly isolates position bias from length-induced performance degradation. Extensive experiments on 10 state-of-the-art embedding-based retrieval models reveal that: (1) retrieval performance on PosIR with documents exceeding 1536 tokens correlates poorly with the MMTEB benchmark, exposing limitations of current short-text evaluations; (2) position bias is pervasive in embedding models and even increases with document length, with most models exhibiting primacy bias while certain models show unexpected recency bias; (3) as an exploratory investigation, gradient-based saliency analysis further uncovers two distinct internal mechanisms that correlate with these positional preferences. We hope that PosIR can serve as a valuable diagnostic framework to advance the development of position-robust retrieval systems.