Evaluating Scoring Bias in LLM-as-a-Judge
Defines scoring bias, identifies three types—Rubric order, Score ID, reference answer bias—and proposes a quantification framework.
Key Findings
Methodology
This paper formally defines scoring bias, identifying three novel types: rubric order bias, score ID bias, and reference answer score bias. A comprehensive framework with multi-faceted metrics—stability, accuracy, and tendency—is developed, alongside an automatic data synthesis pipeline to generate tailored evaluation datasets. Experiments on four benchmarks (BiGGen, FLASK, MT, Vicuna) utilize GPT-4.1 as the gold standard, analyzing bias impacts by perturbing scoring prompts (e.g., rubric order change, score ID replacement, reference score attachment). The framework quantifies how these perturbations affect model scores, revealing widespread biases across models of varying strength.
Key Results
- All tested models exhibit significant scoring bias; for example, BiGGen shows a flip rate of 45.54% and MAD of 0.5604 under certain perturbations. GPT-4.1's score correlation with gold scores drops from 0.6048 to 0.4863 with bias. Larger models like GPT-4o demonstrate lower bias (FR<25%, MAD<0.3), but still show sensitivity under specific perturbations.
- Different bias types influence models’ scoring tendencies distinctly; heatmaps reveal shifts in score distributions. Some biases, such as Roman numeral ID, can unexpectedly improve correlation in certain models, indicating complex bias effects. The results highlight the importance of bias mitigation for evaluation robustness.
- Bias impacts not only stability but also models’ scoring preferences, emphasizing the need for debiasing strategies to ensure fair and reliable assessments.
Significance
This work pioneers systematic quantification of scoring bias in LLM-based evaluation, revealing that subtle prompt modifications induce significant score fluctuations. It addresses a critical gap between theoretical understanding and practical deployment, providing tools to improve evaluation fairness and robustness. The findings have broad implications for deploying automated evaluators in education, content moderation, and other domains requiring standardized, trustworthy assessments. By identifying bias sources, the study paves the way for designing bias-resistant prompts and models, ultimately advancing the reliability of AI-driven evaluation systems.
Technical Contribution
The paper introduces a formal definition of scoring bias, identifying three new types—rubric order bias, score ID bias, and reference answer bias—and develops a multi-metric evaluation framework. The automatic data synthesis pipeline enables large-scale bias analysis, facilitating systematic quantification. Experiments across multiple models and datasets demonstrate the pervasiveness of biases and their impact on evaluation stability and fairness. This work provides foundational tools for bias analysis, guiding future research on bias mitigation and evaluation robustness.
Novelty
This is the first comprehensive formalization of scoring bias in LLM evaluation, focusing on prompt components beyond evaluation targets. The identification of three new bias types—particularly reference answer score bias—is novel. The automatic data synthesis pipeline for bias quantification is an innovative approach, enabling large-scale, controlled experiments. Unlike prior work that mainly analyzed comparative biases, this study emphasizes scoring stability and fairness, offering new insights into prompt design and evaluation reliability.
Limitations
- The experiments primarily use GPT-4.1, and bias behavior may vary with other models or architectures, limiting generalizability.
- The scope of bias types is limited; other potential biases like context sensitivity or cultural bias remain unexplored.
- Data synthesis relies on generative models, which may introduce noise or additional biases, affecting measurement accuracy.
Future Work
Future research will focus on developing bias mitigation techniques, such as prompt optimization and model calibration, to enhance robustness. Extending the bias taxonomy to include contextual and cultural biases is crucial. Additionally, integrating human-in-the-loop approaches and exploring multi-task, multi-modal evaluation settings will further improve fairness and reliability. The goal is to establish standardized benchmarks and tools for bias detection and correction, fostering trustworthy AI evaluation systems.
AI Executive Summary
The rapid adoption of large language models (LLMs) for automated evaluation tasks has raised critical questions about the fairness and stability of their scoring mechanisms. While prior research has identified biases in comparative judgments—such as position bias or length bias—less attention has been paid to biases inherent in absolute scoring systems. This gap is significant because scoring-based evaluation is increasingly used in industrial applications for its simplicity and interpretability.
This paper makes a pioneering contribution by formally defining scoring bias and identifying three novel types: rubric order bias, score ID bias, and reference answer score bias. These biases originate from subtle prompt variations—such as changing the order of scoring criteria, replacing score identifiers with letters or Roman numerals, or attaching specific reference scores—that can cause substantial fluctuations in model scores. To systematically analyze these effects, the authors develop a comprehensive evaluation framework, incorporating multi-dimensional metrics including stability (flip rate, MAD), accuracy (correlation coefficients), and scoring tendency (distribution shifts). An automatic data synthesis pipeline is introduced to generate tailored datasets, enabling large-scale bias quantification.
Experiments across four benchmark datasets—BiGGen, FLASK, MT, and Vicuna—demonstrate that even state-of-the-art models like GPT-4.1 are susceptible to these biases. Results show that bias effects vary with model strength, with larger models generally exhibiting better robustness (FR<25%, MAD<0.3), yet still affected under certain perturbations. The analysis reveals that biases influence not only score consistency but also the models’ scoring preferences, which can lead to unfair evaluations.
The significance of this work lies in its systematic approach to bias quantification, providing essential tools for designing more robust scoring prompts. It highlights the necessity of bias mitigation strategies to ensure fair, trustworthy AI evaluation systems, especially as these tools become integral to education, content moderation, and enterprise decision-making. Future directions include bias reduction techniques, expanding bias taxonomy, and multi-modal evaluation frameworks, aiming to foster more reliable and equitable AI assessment methods.
Deep Analysis
Background
近年来，随着大规模语言模型（LLM）在自然语言处理中的突破，其在自动评估领域的应用逐渐增多。早期研究如OpenAI的InstructGPT和GPT-4，已展现出在内容生成和评估任务中的潜力。相关工作包括Prompt设计优化（如Few-shot Learning、Chain-of-Thought）、多模型协作（如Feedback机制）以及偏差分析（如位置偏差、长度偏差）。然而，现有偏差研究多集中在比较任务（如二元排序、偏好判断），对评分任务中的偏差缺乏系统定义与量化，限制了模型在工业场景中的应用效果。随着评分机制逐渐成为工业标准，理解其偏差来源、影响及缓解策略成为亟待解决的问题。
Core Problem
评分任务中，模型的评分结果易受提示设计细节影响，导致不稳定和偏差。具体表现为：微小扰动（如调整Rubric顺序、替换Score ID或添加参考答案）会引起评分显著变化。这种敏感性削弱了自动评估的可信度，阻碍了其在标准化、规模化场景中的应用。核心问题在于：如何定义、量化并缓解这些偏差，确保模型评分的稳定性和公平性。这不仅关系到模型的实际应用效果，也影响自动评估体系的科学性和可信度。
Innovation
本研究的创新点在于：
1）正式定义评分偏差，突破以往偏差只关注比较任务的局限；
2）识别三类新型偏差（Rubric顺序偏差、Score ID偏差、参考答案评分偏差），填补了评分任务偏差研究空白；
3）设计多维指标体系（稳定性、准确性、偏好性）和自动数据合成管道，系统化量化偏差影响；
4）在四个公开基准上实证验证偏差普遍存在，揭示偏差对模型鲁棒性和评分偏好的深远影响。这些创新为自动化评估的公平性和可靠性提供了理论基础和实践工具。
Methodology
- �� 设计评分偏差的正式定义，明确偏差类型及其引入机制。
- �� 构建多维指标体系：
- 稳定性指标（如翻转率、MAD）评估评分一致性；
- 准确性指标（如相关系数）衡量偏差对黄金评分的影响；
- 倾向性指标（评分分布）分析模型偏好。
- �� 开发自动数据合成管道：
- 通过扰动评分提示（Rubric顺序、Score ID、参考答案评分）生成多样化样本；
- 利用GPT-4.1和GPT-4o生成不同偏差场景下的评分数据。
- �� 在四个公开基准（BiGGen、FLASK、MT、Vicuna）上进行实验，比较不同模型（GPT-4o、DeepSeek-V3、Qwen系列、Mistral）在偏差下的表现。
- �� 采用多轮评测和统计分析，验证偏差的普遍性和影响机制。
Experiments
采用四个公开评分基准，分别为BiGGen（2780样本）、FLASK（2001样本）、MT（320样本）和Vicuna（320样本）。在每个数据集上，使用GPT-4.1作为黄金标准，进行多轮评分（三次投票取众数）。引入扰动类型包括Rubric顺序变换、Score ID替换（阿拉伯数字、字母、罗马数字）以及参考答案评分附加。评估指标包括翻转率（FR）、平均绝对偏差（MAD）、相关系数（ρ、r）和评分偏好分布。通过对比偏差前后模型评分的一致性，分析偏差引起的变化，验证偏差的普遍存在和影响程度。
Results
所有模型在不同偏差条件下均表现出显著评分变化，偏差最大时BiGGen基准中FR达45.54%，MAD达0.5604。GPT-4.1模型偏差影响较小（FR<25%，MAD<0.3），而Qwen3-8B在某些扰动下偏差显著（FR超50%）。偏差类型对模型评分倾向影响明显，偏差引起的评分分布偏移在热图中清晰呈现。偏差还能在某些情况下改善模型与黄金评分的相关性（如Roman数字ID偏差提升ρ值），但总体偏差仍降低模型稳定性。实验验证了偏差在工业应用中的潜在风险，强调偏差缓解的重要性。
Applications
本研究为自动化评估系统提供偏差检测与校正工具，适用于教育、内容审核、问答系统等场景。通过优化提示设计，减少偏差影响，提升模型评分的公平性和稳定性。未来可结合偏差分析结果，开发偏差缓解算法，推动自动评估在大规模应用中的普及。
Limitations & Outlook
研究主要基于特定模型（如GPT-4.1），偏差表现可能受模型架构和训练数据影响，泛化性有限。偏差类型虽丰富，但未涵盖所有潜在偏差（如语境敏感性偏差），未来需扩展。自动数据合成依赖模型生成，可能引入噪声，影响偏差量化准确性。偏差缓解策略仍需深入研究，未来应结合人类反馈优化提示设计。
Plain Language Accessible to non-experts
想象你在餐厅点菜，菜单上有各种菜品。服务员（模型）会根据你的点单给出评价，但有时他们的评价会受到一些小细节的影响，比如菜单的排序、菜名的编号，甚至你是否提到一道特别喜欢的菜。这些细节会让服务员给出的评分偏离实际的菜品质量。研究发现，类似的偏差在智能评分模型中也存在，比如菜单（提示）中菜品的顺序或编号会影响评分结果。这个问题就像餐厅里的评分一样，如果不注意，评分就可能不公平或不稳定。通过分析这些偏差，科学家希望让评分更公平、更一致，就像让餐厅的评价变得更客观一样。
ELI14 Explained like you're 14
想象你在学校里参加考试，老师会给你打分。可是，有时候老师的评分会受到一些小细节的影响，比如题目的顺序、题号的编号，甚至是不是你提到的某个关键词。这些细节可能让老师对你的答案打分不公平或不一致。科学家们发现，类似的问题也存在于用电脑自动评分的系统中。比如，改变题目的顺序或用不同的编号，电脑给出的分数可能会有很大变化。这就像你在考试时，老师因为题目排布不同，评分也会不同一样。为了让评分更公平、更可靠，研究人员分析了这些偏差的来源，并设计了方法来减少它们的影响。这样，未来的自动评分系统就能像一个公平的老师一样，给出一致、客观的评价。
Abstract
The "LLM-as-a-Judge" paradigm, using Large Language Models (LLMs) as automated evaluators, is pivotal to LLM development, offering scalable feedback for complex tasks. However, the reliability of these judges is compromised by various biases. Existing research has heavily concentrated on biases in comparative evaluations. In contrast, scoring-based evaluations-which assign an absolute score and are often more practical in industrial applications-remain under-investigated. To address this gap, we undertake the first dedicated examination of scoring bias in LLM judges. We shift the focus from biases tied to the evaluation targets to those originating from the scoring prompt itself. We formally define scoring bias and identify three novel, previously unstudied types: rubric order bias, score ID bias, and reference answer score bias. We propose a comprehensive framework to quantify these biases, featuring a suite of multi-faceted metrics and an automatic data synthesis pipeline to create a tailored evaluation corpus. Our experiments empirically demonstrate that even the most advanced LLMs suffer from these substantial scoring biases. Our analysis yields actionable insights for designing more robust scoring prompts and mitigating these newly identified biases.