Large Language Models are not Fair Evaluators
Proposed a calibration framework to address positional bias in LLM evaluations, improving human alignment by 14.3%.
Key Findings
Methodology
The framework includes: 1) Multiple Evidence Calibration (MEC) to generate multiple evaluation evidences; 2) Balanced Position Calibration (BPC) averaging scores across swapped positions; 3) Human-in-the-Loop Calibration (HITLC) using Balanced Position Diversity Entropy (BPDE) to identify cases needing human intervention.
Key Results
- Vicuna-13B's win rate fluctuated from 2.5% to 82.5% under ChatGPT evaluation, showing significant positional bias.
- MEC and BPC improved GPT-4's human alignment by 9.8% and ChatGPT's by 14.3%.
- HITLC achieved 71.3% ChatGPT-human alignment with only 20% human intervention, reducing cost by 39%.
Significance
This study reveals critical flaws in using LLMs as evaluators and proposes a calibration framework that significantly improves evaluation fairness, paving the way for more reliable automated evaluation methods.
Technical Contribution
First to systematically identify positional bias in LLM evaluations and propose a combined calibration framework leveraging MEC, BPC, and HITLC to enhance reliability.
Novelty
Introduced BPDE-based human intervention strategy, combined with MEC and BPC, to address evaluation bias innovatively.
Limitations
- The framework requires additional computational resources, especially for MEC sampling.
- BPDE threshold selection is sensitive and requires optimization.
- Experiments are limited to the Vicuna Benchmark and need broader validation.
Future Work
Future work includes exploring more efficient calibration methods, extending to diverse tasks, and optimizing BPDE parameters to minimize human intervention.
AI Executive Summary
Large Language Models (LLMs) like GPT-4 are increasingly used as evaluators for text generation quality. However, this study uncovers significant positional bias in such evaluations. Simply swapping the order of candidate responses can drastically alter results, leading to unfair assessments. For instance, Vicuna-13B's win rate under ChatGPT evaluation ranged from 2.5% to 82.5%, highlighting the issue.
To address this, the authors propose a three-part calibration framework: 1) Multiple Evidence Calibration (MEC) stabilizes results by generating multiple evaluation evidences; 2) Balanced Position Calibration (BPC) averages scores across swapped positions to eliminate bias; 3) Human-in-the-Loop Calibration (HITLC) uses Balanced Position Diversity Entropy (BPDE) to identify cases requiring human intervention. Experiments show these methods significantly improve fairness, with GPT-4 and ChatGPT achieving 9.8% and 14.3% better alignment with human judgments, respectively.
While the framework requires additional computational resources, it effectively reduces human intervention costs. Future research could optimize BPDE thresholds, explore more efficient methods, and validate the framework across diverse tasks, setting the stage for fairer automated evaluation systems.
Deep Analysis
Background
LLMs have shown remarkable performance in text generation, but evaluating their output remains challenging. Traditional metrics like BLEU and ROUGE are insufficient, and model-based metrics like BERTScore have limitations. Recently, LLMs themselves are used as evaluators, but their reliability is underexplored.
Core Problem
The study identifies significant positional bias in LLM evaluations. Simply swapping the order of candidate responses can drastically alter results, leading to unfair assessments and potential exploitation.
Innovation
The proposed framework includes: 1) MEC to stabilize results by generating multiple evidences; 2) BPC to eliminate bias by averaging scores across swapped positions; 3) HITLC to involve human intervention based on BPDE.
Methodology
- �� MEC: Generate multiple evaluation evidences and average scores.
- �� BPC: Swap candidate positions and average scores.
- �� HITLC: Use BPDE to identify cases needing human intervention and combine human and model evaluations.
Experiments
Experiments used the Vicuna Benchmark with 80 questions across 9 categories. GPT-4 and ChatGPT were evaluated as assessors, and the effectiveness of calibration strategies was compared using metrics like human alignment and conflict rate.
Results
MEC and BPC improved GPT-4's human alignment by 9.8% and ChatGPT's by 14.3%. HITLC achieved 71.3% ChatGPT-human alignment with only 20% human intervention, reducing costs by 39%.
Applications
The framework can improve fairness in dialogue systems, content generation, and educational evaluation tools, especially in scenarios requiring high reliability.
Limitations & Outlook
The framework requires additional computational resources, especially for MEC sampling. BPDE threshold selection is sensitive and needs further optimization. Experiments are limited to the Vicuna Benchmark.
Plain Language Accessible to non-experts
Imagine you're judging two chefs' dishes, but the order in which they're served affects your judgment. This is like GPT-4's positional bias when evaluating text. The study proposes three solutions: 1) Ask chefs to present their dishes multiple times (MEC); 2) Swap the serving order and average the scores (BPC); 3) Call in a food critic for disputed cases (HITLC). These methods make the judging process fairer.
ELI14 Explained like you're 14
Imagine you and your friend are competing in a writing contest, but the teacher always favors the first essay! That's like GPT-4's positional bias. The researchers fixed this with three tricks: 1) Make the teacher score multiple times and average it; 2) Swap the essay order and score again; 3) Call another teacher to help when it's tricky. Fairer scores, right? Cool stuff!
Glossary
Multiple Evidence Calibration (MEC)
Generates multiple evaluation evidences to stabilize results.
Used to reduce instability in single evaluations.
Balanced Position Calibration (BPC)
Averages scores across swapped positions to eliminate bias.
Addresses LLM sensitivity to response order.
Balanced Position Diversity Entropy (BPDE)
Measures diversity in evaluation results to identify biased cases.
Used in HITLC to select samples for human intervention.
Vicuna Benchmark
An evaluation benchmark with 80 questions across 9 categories.
Used to assess LLM performance.
Conflict Rate
Proportion of conflicting results due to positional changes.
Quantifies the impact of positional bias.
Open Questions Unanswered questions from this research
- 1 Unclear impact of BPDE parameter tuning on results.
- 2 Framework's generalizability to more complex tasks needs validation.
Applications
Immediate Applications
Dialogue System Optimization
Improves fairness in chatbot evaluations, enhancing user experience.
Educational Assessment Tools
Ensures reliability in automated grading systems.
Long-term Vision
Universal Evaluation Framework
Develops cross-domain fair evaluation methods, standardizing AI assessments.
Abstract
In this paper, we uncover a systematic bias in the evaluation paradigm of adopting large language models~(LLMs), e.g., GPT-4, as a referee to score and compare the quality of responses generated by candidate models. We find that the quality ranking of candidate responses can be easily hacked by simply altering their order of appearance in the context. This manipulation allows us to skew the evaluation result, making one model appear considerably superior to the other, e.g., Vicuna-13B could beat ChatGPT on 66 over 80 tested queries with ChatGPT as an evaluator. To address this issue, we propose a calibration framework with three simple yet effective strategies: 1) Multiple Evidence Calibration, which requires the evaluator model to generate multiple evaluation evidence before assigning ratings; 2) Balanced Position Calibration, which aggregates results across various orders to determine the final score; 3) Human-in-the-Loop Calibration, which introduces a balanced position diversity entropy to measure the difficulty of each example and seeks human assistance when needed. We also manually annotate the "win/tie/lose" outcomes of responses from ChatGPT and Vicuna-13B in the Vicuna Benchmark's question prompt, and extensive experiments demonstrate that our approach successfully mitigates evaluation bias, resulting in closer alignment with human judgments. We release our code and human annotation at \url{https://github.com/i-Eval/FairEval} to facilitate future research.