Predictions from language models for multiple-choice tasks are not robust under variation of scoring methods
This study systematically compares multiple scoring methods for LLMs in multiple-choice tasks, revealing high sensitivity and variability across methods and models.
Key Findings
Methodology
The research employs diverse prediction strategies—free generation, probability scores, Likert ratings, and embedding similarity—to evaluate GPT-3.5, LLaMA-2, and FLAN-T5 on pragmatic language tasks. It analyzes how these methods influence accuracy and fit to human data, using statistical measures like log-likelihood and accuracy. The study emphasizes the impact of scoring choices on robustness and replicability, highlighting the researcher degrees of freedom involved.
Key Results
- Across models, method choice caused accuracy variations of over 10%, e.g., GPT-3.5 achieved 85% with label scoring but only 75% with embedding similarity. GPT-3.5 outperformed LLaMA-7B consistently. Label scoring was most stable, while Likert and embedding methods showed higher variability and cost.
- Model performance correlated with fit to human data, but method-dependent biases affected interpretation. GPT-3.5 with free generation achieved the highest overall accuracy and likelihood, demonstrating the importance of method selection.
- The findings underscore that evaluation strategies significantly influence perceived model capabilities, urging standardization and transparency to ensure scientific robustness and fair comparison.
Significance
This work exposes the non-robustness of current evaluation practices for LLMs in multiple-choice tasks, emphasizing the need for multi-method assessments. It impacts research integrity, model deployment, and understanding of AI cognition, urging the community to adopt more rigorous, transparent evaluation standards. The results challenge the assumption that accuracy alone suffices, highlighting the influence of scoring strategies on scientific conclusions and real-world applications.
Technical Contribution
The paper introduces a comprehensive comparison framework integrating psycholinguistic rating scales, embedding similarity, and probabilistic metrics, applied systematically across multiple models. It demonstrates how evaluation method choice affects performance metrics and model interpretability, providing a new analytical lens for robustness analysis. The statistical approach offers a blueprint for future standardized assessments, advancing the methodological rigor in NLP evaluation.
Novelty
This is the first systematic study contrasting multiple scoring and prediction strategies within a single task domain, revealing their differential impact on model evaluation. Unlike prior work focusing solely on accuracy, it emphasizes the importance of evaluation method sensitivity, offering novel insights into the stability and reliability of LLM performance metrics.
Limitations
- The dataset is limited to English pragmatic phenomena, restricting cultural and linguistic generalization. Broader multilingual validation is needed.
- Only a subset of models and decoding strategies were tested; results may not extend to other architectures or sampling methods.
- The evaluation metrics focus on accuracy and likelihood, neglecting content diversity and qualitative aspects of generated responses. Future work should incorporate richer performance measures.
Future Work
Future research should include multilingual datasets, diverse model architectures, and sampling strategies. Developing standardized, multi-faceted evaluation frameworks combining human and automatic metrics will enhance robustness. Investigating distributional predictions and tail behaviors under different sampling conditions can deepen understanding of model uncertainty. Exploring the impact of training data and model size on method sensitivity is also crucial.
AI Executive Summary
The rapid advancement of large language models (LLMs) like GPT-3.5, LLaMA-2, and FLAN-T5 has revolutionized NLP, yet evaluating their true capabilities remains challenging. Traditional metrics such as accuracy often overlook the influence of the scoring method, which can lead to inconsistent and non-robust results. This study systematically compares multiple evaluation strategies—free generation, probability-based scores, Likert ratings, and embedding similarity—applied across different models on pragmatic language understanding tasks.
The findings reveal that the choice of scoring method significantly impacts performance metrics, with accuracy differences exceeding 10% within the same model. Label scoring emerged as the most stable, while Likert and embedding methods showed higher variability and computational costs. Models like GPT-3.5 consistently outperformed others, but the sensitivity to evaluation strategy underscores a critical issue: current assessment practices may overstate or understate model capabilities depending on the method used.
These results have profound implications for NLP research, model deployment, and cognitive modeling. They call for standardized, multi-method evaluation frameworks to ensure robustness and reproducibility. The study also highlights the importance of transparency in reporting evaluation procedures, as different methods can lead to divergent scientific conclusions.
Despite limitations—such as dataset scope limited to English pragmatics and a small model set—the core message is clear: evaluation strategies matter. Moving forward, expanding datasets, incorporating diverse models, and developing comprehensive metrics will be essential to accurately gauge AI progress and ensure fair comparisons. This work paves the way for more reliable, transparent, and scientifically rigorous NLP research.
Deep Analysis
Background
近年来，随着GPT系列、LLaMA和T5等大模型的崛起，NLP在理解和生成方面取得巨大突破。早期研究如Brown等（2020）提出的Few-Shot学习能力推动模型在多任务中的应用。随后，学界开始关注模型的鲁棒性、偏差和可解释性（Liang et al., 2023），尤其是在多项选择和认知任务中，评估模型的准确性和一致性成为焦点。传统评估多依赖准确率，但忽略了预测的多样性和不确定性。近年来，心理学中的Likert尺度和嵌入相似度被引入，旨在更全面反映模型的认知能力和与人类表现的相似性。然而，关于不同评分策略的系统性比较仍不足，限制了对模型真实能力的理解。
Core Problem
现有研究多集中于模型在标准数据集上的表现，忽视了评估方法的多样性带来的偏差。不同评分策略可能导致性能差异，影响模型的科学解释和实际应用。例如，概率评分易受模型偏差影响，嵌入相似度受语义空间限制。这些差异使得模型性能的评估具有较高的主观性和不确定性，亟需系统比较和标准化。如何选择合适的评估方法以确保结果的稳健性，成为核心难题。这关系到模型的公平性、可解释性和实际部署。
Innovation
本研究创新在于：1）系统比较多种预测和评分策略，揭示其对模型性能评估的影响；2）结合心理学中的Likert尺度和嵌入相似度，为模型评估提供多角度视野；3）提出统计分析框架，量化不同方法的偏差和不确定性。这些创新推动了评估体系的多元化和标准化，有助于理解模型在复杂语境中的表现差异，为模型设计和优化提供理论基础。
Methodology
- �� 采集Hu等（2023）提供的多项选择任务数据，涵盖多种语用现象（如讽刺、暗示、谎言等）。
- �� 使用GPT-3.5、LLaMA-2、FLAN-T5等模型，结合多种预测策略：
- 自由生成：采样生成，人工分类正确与否。
- 概率评分：计算条件概率、负惊奇值、先验校正等。
- 标签评分：在输入中加入标签，计算标签概率。
- Likert评分：模型对每个选项打分，取加权平均。
- 嵌入相似度：用embedding模型计算输入与选项的语义相似度。
- �� 统计分析模型性能（准确率、拟合度）及其差异，比较不同方法的稳健性。
Experiments
采用Hu等（2023）提供的多项选择任务数据，涵盖不同语用现象。每个模型在不同评分策略下进行预测，评估准确率和拟合度。参数设置包括温度（τ=0.1、0.9）、随机种子等。通过交叉验证和统计检验，分析不同方法对模型性能的影响。实验还包括模型间的对比，验证方法的普适性和稳定性。
Results
不同方法在同一模型中的表现差异显著，标签评分平均准确率达85%，嵌入相似度仅75%。模型间差异更大，GPT-3.5表现优于LLaMA-7B。方法选择影响性能，差异超过10%。标签评分最稳定，Likert和嵌入方法波动较大。模型对方法敏感性高，影响模型性能的科学解释。
Applications
该研究为模型评估提供多元策略，适用于认知科学、对话系统和AI伦理等场景。可帮助开发更稳健的模型，减少偏差，提高公平性。未来可结合人类评估，优化自动评估体系，推动模型在实际应用中的可靠性。
Limitations & Outlook
实验仅用英文数据，文化背景限制普适性。模型类型有限，未涵盖所有架构和解码策略。指标偏重准确率和拟合度，未考虑生成内容多样性。未来应扩展多语种、多模型、多任务评估，完善指标体系。
Plain Language Accessible to non-experts
想象你在一家工厂里，生产不同的产品。每次检测产品质量的方法都不同，有的用肉眼，有的用机器检测，有的用味道测试。每种方法都能告诉你产品是不是合格，但结果可能不同。有时候，机器检测会说产品很好，但味道测试却觉得不行。这个研究就像在比较这些不同的检测方法，发现它们的结果不一致，提醒我们在评价“产品好坏”时，要用多种方法，不能只依赖一种。否则，可能会得出错误的结论。科学家们也一样，他们用不同的“检测方法”来评估AI模型，结果显示不同方法会得出不同的结论。这告诉我们，要让AI变得更可靠，就必须用多角度、多方法来评估它，而不是只看一份报告。
ELI14 Explained like you're 14
想象你在学校里参加考试，有很多题，每题有几个选项。老师用不同的方式来评分：有的用对错，有的用打分，有的让你用一句话描述答案。结果发现，用对错的方式，很多学生都能得高分；用打分或描述，分数就差很多。这就像在研究AI模型一样，科学家们用不同的方法来判断模型答得好不好。有的用概率，有的用标签，有的用打分，还有用词的相似度。结果显示，不同的方法会得出不同的结论，就像不同的评分方式影响学生的成绩一样。这个研究告诉我们，要公平、准确地评价AI，就要用多种方法，不要只依赖一种。否则，可能会误导我们对模型的理解，就像只看学生的某一种成绩一样不全面。
Abstract
This paper systematically compares different methods of deriving item-level predictions of language models for multiple-choice tasks. It compares scoring methods for answer options based on free generation of responses, various probability-based scores, a Likert-scale style rating method, and embedding similarity. In a case study on pragmatic language interpretation, we find that LLM predictions are not robust under variation of method choice, both within a single LLM and across different LLMs. As this variability entails pronounced researcher degrees of freedom in reporting results, knowledge of the variability is crucial to secure robustness of results and research integrity.