As It Was: Aligning LLM Search Evaluation with Historical User Preferences
Behavior-grounded LLM judge using QRI cards improves relevance alignment by 5% overall and 91% in disagreement cases.
Key Findings
Methodology
This work introduces a behavior-grounded evaluation framework for LLMs, integrating Query-Relevance-Impressions (QRI) cards that summarize historical user interactions. Using a large-scale Spotify music search dataset, the approach employs debiased relevance estimates via inverse propensity scoring (IPS), combined with prompt engineering to incorporate behavioral evidence. The evaluation compares a baseline semantic-only model with the proposed behavior-enhanced model across multiple metrics, including Spearman 𝜌 and Kendall 𝜏, on interaction logs, human annotations, and live A/B tests. The framework effectively resolves ambiguity, calibrates near-miss cases, and improves ranking sensitivity, demonstrating robustness across multilingual scenarios.
Key Results
- The behavior-grounded model improves overall Spearman 𝜌 by approximately 5%, with a 91% relative increase in disagreement cases, indicating better alignment with user preferences.
- In a multilingual human-annotated dataset, relevance correlation increased by 15%, confirming cross-lingual robustness.
- In live A/B testing, the grounded judge showed higher agreement with actual user preferences, validating practical applicability.
Significance
This research advances automatic search evaluation by systematically integrating user behavior signals, addressing the limitations of pure semantic models. It enhances the reliability, interpretability, and user-alignment of large-scale relevance judgments, crucial for deploying trustworthy AI in real-world search systems. The approach is especially valuable for ambiguous or long-tail queries, where semantic cues alone are insufficient, paving the way for more personalized and accurate retrieval experiences.
Technical Contribution
The key innovation lies in embedding lightweight, auditable behavioral evidence via QRI cards into LLM prompts, combined with IPS bias correction, to improve relevance judgments. This method enhances model robustness against ambiguity and bias, supports multi-language deployment, and offers a scalable, interpretable evaluation mechanism that can be integrated into production systems, setting a new standard for behavior-aware AI evaluation.
Novelty
This is the first systematic integration of user interaction logs as explicit, auditable evidence within LLM-based search evaluation. Unlike prior work focusing solely on semantic similarity or external retrieval, this approach leverages historical behavioral data to resolve ambiguity and calibrate relevance, offering a novel, practical solution for real-world, large-scale search assessment.
Limitations
- The approach relies on sufficient historical interaction data; in cold-start or sparse data scenarios, effectiveness diminishes.
- Bias correction depends on accurate click models; inaccuracies may affect relevance estimates.
- Real-time deployment involves computational and storage overheads for maintaining and updating behavioral logs.
Future Work
Future research will explore integrating richer behavioral signals such as dwell time and secondary interactions, developing adaptive bias correction techniques, and extending the framework to other domains like web search and recommendation systems. Additionally, efforts will focus on optimizing computational efficiency and scalability for real-time applications.
AI Executive Summary
In the rapidly evolving landscape of search systems, traditional manual evaluation methods struggle to keep pace with model updates and increasing query diversity. To address this, recent advances have turned to large language models (LLMs) for automated relevance assessment, but these often rely solely on semantic similarity or world knowledge, which can diverge from actual user preferences, especially for ambiguous or long-tail queries.
This paper introduces a novel behavior-grounded evaluation framework that incorporates user interaction history through Query-Relevance-Impressions (QRI) cards. These cards summarize how users have historically engaged with similar queries and results, providing empirical evidence that complements semantic reasoning. The approach employs debiased relevance estimates via inverse propensity scoring (IPS) to mitigate bias inherent in interaction logs, ensuring more accurate behavioral signals.
Experiments conducted on Spotify’s music search data demonstrate that the behavior-grounded model outperforms traditional semantic-only models, with a 5% increase in overall relevance ranking correlation and a 91% improvement in disagreement cases. Multilingual evaluations further confirm a 15% boost in alignment with human judgments. In live A/B tests, the grounded evaluator shows higher consistency with actual user preferences, validating its practical utility.
These findings highlight the importance of integrating lightweight, auditable behavioral evidence into LLM evaluation processes. The method effectively resolves ambiguity, calibrates near-miss cases, and enhances ranking sensitivity, especially in complex scenarios. While challenges remain in cold-start situations and computational costs, this work paves the way for more reliable, user-aligned search evaluation systems, with broad implications for industry and academia alike.
Overall, this research marks a significant step toward more trustworthy AI-driven search systems, emphasizing the value of behavioral signals in bridging the gap between model judgments and real-world user preferences.
Deep Analysis
Background
随着搜索引擎和推荐系统的快速发展，评估技术也在不断演进。传统依赖人工标注的评估方法难以满足大规模、多语种、多场景的需求。近年来，LLMs在自动相关性评估中展现潜力，但其单纯基于语义和世界知识的判断在模糊或多义场景中偏离用户实际偏好。行为信号（如点击、停留时间）被视为潜在的宝贵证据，但尚未充分融合到LLM评估中。现有研究多关注模型的可解释性和可靠性，尝试结合外部证据或偏差校正技术以提升性能，但缺乏系统性整合用户行为的机制。本文在此背景下提出行为基础的LLM评判框架，旨在弥补语义推理的不足，增强模型的实用性和可信度。
Core Problem
核心问题在于，纯语义的LLM评判在面对模糊、长尾或多义查询时，容易偏离用户真实偏好，导致评估结果不可靠。传统方法缺乏对用户实际交互行为的利用，无法有效解决多义性和偏差问题。此外，现有评估机制难以在大规模、多语种环境中实现快速、可审计的自动化，限制了其在实际系统中的应用。如何结合用户行为数据，设计一种既能保持语义推理优势，又能反映用户偏好的评估机制，成为亟待解决的关键问题。
Innovation
本研究的创新点在于引入QRI卡，将用户历史交互数据作为轻量级、可审计的证据，结合逆概率评分（IPS）校正偏差，增强LLM的相关性判断能力。通过在模型提示中融入行为证据，模型在模糊场景中表现更贴近用户偏好。与传统纯语义模型不同，本文实现了行为证据的系统性整合，提升了模型的鲁棒性和可解释性。该方法还在多语种、多场景中验证了其广泛适用性，为大规模搜索系统的自动评估提供了新思路。
Methodology
- �� 构建QRI卡：从搜索日志中提取历史查询与结果交互，采用IPS校正偏差，生成每个结果的行为证据。• 结合模型提示：在LLM评判中加入QRI信息，作为支持性证据，辅助判定模糊或争议查询。• 评估机制：在Spotify音乐搜索中，比较纯语义（P）与行为基础（BG）两种评判，利用真实用户交互数据和人工标注数据进行验证。• 逆概率评分：校正点击偏差，确保行为信号的真实性。• 过滤近似重复查询：避免信息泄露，确保评估的公平性。• 结合多语言数据：验证方法在多语种环境中的有效性。• 统计分析：使用Spearman和Kendall相关系数评估模型与用户偏好的匹配程度。
Experiments
采用Spotify真实搜索日志，抽取约5000个查询，构建重构SERP，评估6000个样本。比较纯语义模型与行为基础模型的相关性排名，分析争议案例和多语种表现。采用偏差校正的点击数据作为行为证据，结合模型提示生成评判。通过实际A/B测试，采集用户偏差，验证模型的实际效果。指标包括Spearman 𝜌、Kendall 𝜏及在线偏好一致性，进行统计检验，确保结果可靠。
Results
行为基础模型在整体排名中提升相关性指标约5%，在争议案例中提升达91%，显著优于纯语义模型。多语种数据中，相关性提升15%。在A/B测试中，行为模型与用户偏好一致性提高，偏差显著减少，验证了其实际应用价值。这些结果表明，结合用户行为信号能有效弥补语义推理的不足，提升搜索评估的准确性和信任度。
Applications
该方法可广泛应用于搜索引擎、推荐系统、内容过滤等场景，尤其适合多语种、多样化查询环境。通过引入行为证据，系统能更贴近用户实际需求，提升个性化体验。未来还可结合更多行为指标（如停留时间、二次点击）进一步优化评估效果。
Limitations & Outlook
模型在极端冷启动场景下表现有限，行为数据稀疏时难以充分利用。偏差校正依赖于点击模型的准确性；实时应用中，行为数据存储和更新成本较高。未来需结合多模态信号，提升模型的泛化能力和鲁棒性。
Plain Language Accessible to non-experts
想象你在一家工厂工作，工厂里有很多不同的机器，每台机器都在生产不同的产品。有时候，工厂需要判断某台机器是否正常工作，但只看机器的外表（就像用语义判断），可能会误判。于是，工厂还会查看工厂的历史记录，比如这台机器过去的生产数据、维修记录（就像用户的交互行为），用这些信息作为证据，帮助判断是否正常。这样，工厂的判断就更靠谱，也更贴近实际情况。这个方法就是用用户过去的行为数据，帮助AI更准确地评估搜索结果的相关性，就像工厂用历史记录来判断机器状态一样。
ELI14 Explained like you're 14
想象你在学校里，有很多不同的老师在给你评分。有时候，老师只看你的答案是不是对的（像用语义判断），但有时候他们还会看看你平时的表现，比如你平时喜欢做什么、参加了哪些活动（就像用户的交互行为）。如果你平时经常参加篮球比赛，老师可能会觉得你对体育题目更感兴趣。这样，老师的评分就会更贴近你的真实兴趣。这个方法就是让AI也参考用户过去的行为，帮助它更准确地判断搜索结果是不是符合用户的偏好，就像老师用你的表现来打分一样。
Glossary
QRI卡 (Query-Relevance-Impressions Card)
一种总结用户历史交互的证据卡，包含查询、相关性估计和曝光量，用于增强评判的可靠性。
本文中用以作为行为证据支持的关键工具。
逆概率评分 (Inverse Propensity Scoring, IPS)
一种校正偏差的统计方法，通过调整点击偏差，获得更真实的相关性估计。
用于校正用户交互数据中的偏差，确保行为信号的真实性。
Spearman 𝜌 (斯皮尔曼相关系数)
衡量两个变量排序一致性的非参数指标，值范围[-1,1]。
用于评估模型与用户偏好排序的相关性。
Kendall 𝜏 (肯德尔秩相关系数)
衡量两个排序之间一致性的统计指标，值范围[-1,1]。
作为相关性评估的补充指标。
重构SERP (Recomposed Search Engine Results Page)
通过重新组合真实搜索结果，模拟不同质量配置的搜索页面。
用于实验中测试模型在不同排序下的表现。
Open Questions Unanswered questions from this research
- 1 如何进一步融合多模态行为信号（如停留时间、二次点击）以提升评估鲁棒性仍待探索。
- 2 模型在极端冷启动场景下的表现不足，需设计更有效的少样本学习策略。
Abstract
Large-scale search systems evolve faster than human quality assurance can scale, especially for long-tail intents and multilingual queries. LLM-as-a-judge approaches provide a scalable alternative for evaluating the relevance of search engine result pages (SERPs), but judgments based solely on semantic similarity or world knowledge can drift from actual user preferences, particularly for ambiguous queries. We introduce a behavior-grounded LLM judge that augments each SERP item with a lightweight and auditable behavioral prior in the form of a Query-Relevance-Impressions (QRI) card. Each card summarizes how users have historically interacted with similar queries and results, providing compact empirical evidence that the judge can cite to resolve ambiguity and make more consistent relevance judgments while still relying on semantic reasoning. In a large-scale music search evaluation at Spotify, using relevance estimates derived from historical user interactions across 6,000 recomposed SERPs, the behavior-grounded judge achieves stronger alignment with user preferences, improving Spearman rank correlation by approximately 5% overall and yielding a 91% relative improvement on disagreement cases. On a multilingual human-judged dataset spanning five languages, grounding further increases correlation with human relevance judgments by 15%. Importantly, when evaluated against outcomes from a live A/B test, the grounded judge shows consistently higher alignment with the observed winning model. While absolute alignment remains moderate, these findings demonstrate that lightweight behavioral grounding can improve the reliability and practical usefulness of LLM-based evaluation in real-world search systems.