InFoBench: Evaluating Instruction Following Ability in Large Language Models
Introduces DRFR metric and InFoBench benchmark, significantly improving LLM instruction-following evaluation reliability.
Key Findings
Methodology
This paper proposes Decomposed Requirements Following Ratio (DRFR), which dissects complex instructions into multiple binary criteria, quantifying adherence at a granular level. InFoBench comprises 500 instructions and 2250 decomposed questions across content, linguistic, style, format, and numeric constraints, enabling comprehensive assessment. Experiments compare DRFR with traditional scoring, validating higher reliability via inter-annotator agreement (Fleiss’ Kappa). The study also employs GPT-4 as an automatic annotation tool, demonstrating high accuracy (89%) and cost-efficiency, validated through multi-source annotations (experts, crowd, GPT-4). Results reveal performance gaps in complex scenarios across six advanced models, especially in numerical and language understanding.
Key Results
- DRFR achieved a Fleiss’ Kappa of 0.493 on the Hard Set, outperforming traditional direct scoring (0.302), indicating superior consistency and reliability.
- GPT-4-based annotation yielded 89% accuracy at a fraction of expert cost ($5.40 vs. $472), confirming its viability for large-scale evaluation.
- Evaluation of six models showed significant progress overall, yet notable deficiencies remain in handling complex instructions, particularly in numerical reasoning and linguistic nuances, highlighting areas for future improvement.
Significance
This work addresses critical gaps in LLM evaluation by providing a scalable, interpretable, and fine-grained assessment framework. The DRFR metric enables detailed analysis of instruction adherence, facilitating targeted model improvements. InFoBench’s diverse instructions ensure robustness across domains, supporting industry and academia in developing safer, more reliable AI systems. Automating annotation with GPT-4 reduces costs and accelerates research, making large-scale evaluation feasible. Overall, this approach advances the understanding of LLM capabilities, guiding future innovations in instruction-following and model robustness.
Technical Contribution
The core innovation lies in decomposing complex instructions into multiple binary constraints, enabling precise adherence measurement through DRFR. The benchmark’s design ensures coverage of diverse domains and constraints, while the multi-source annotation strategy validates the automatic evaluation’s accuracy and efficiency. The integration of GPT-4 for automated scoring demonstrates a practical, scalable solution, setting a new standard for large-scale, detailed model assessment. This framework enhances interpretability and provides a foundation for future research on instruction robustness and model alignment.
Novelty
This is the first work to formalize DRFR as a fine-grained, decomposed metric for instruction adherence, moving beyond traditional holistic scores. The comprehensive InFoBench dataset and multi-source validation, especially leveraging GPT-4 for automatic annotation, represent significant innovations. These contributions collectively establish a new paradigm for scalable, interpretable evaluation of large language models’ instruction-following capabilities, addressing limitations of prior methods that lacked granularity and automation.
Limitations
- DRFR relies on binary judgments, which may oversimplify nuanced or subjective constraints, potentially missing partial compliance signals. Future work could incorporate fuzzy matching or weighted criteria.
- Current evaluation focuses on text-based instructions; multi-modal or multi-turn complex instructions remain challenging, requiring further extension.
- Automated annotation, while efficient, may produce errors in borderline cases, necessitating hybrid approaches combining AI and human review for critical applications.
Future Work
Future research will expand DRFR to multi-modal and multi-turn instructions, improving robustness in real-world scenarios. Developing adaptive, context-aware annotation techniques and integrating reinforcement learning could further enhance evaluation accuracy. Additionally, establishing standardized benchmarks and metrics for multi-domain instruction adherence will support broader community adoption, ultimately driving the development of more reliable, aligned large language models.
AI Executive Summary
The rapid advancement of large language models (LLMs) such as GPT-3, GPT-4, and others has revolutionized natural language processing, enabling unprecedented capabilities in text generation and understanding. However, evaluating their ability to accurately follow user instructions remains a significant challenge. Traditional assessment methods, including A/B testing and overall scoring, suffer from scalability issues and lack interpretability, especially when dealing with complex, multi-faceted instructions. To address this, the present study introduces a novel evaluation framework centered on the Decomposed Requirements Following Ratio (DRFR) metric, which dissects instructions into multiple specific criteria, each representing a distinct requirement.
Complementing this, the authors develop InFoBench, a comprehensive benchmark dataset comprising 500 diverse instructions and 2250 decomposed questions, covering five key constraint categories: content, linguistic, style, format, and numeric. This dataset is designed to evaluate models across a broad spectrum of real-world scenarios, from simple to highly complex tasks. The core innovation lies in decomposing instructions into binary questions, enabling precise measurement of adherence at a granular level.
Experimental results demonstrate that DRFR outperforms traditional direct scoring in terms of reliability, with Fleiss’ Kappa reaching 0.493 on the Hard Set, compared to 0.302 for conventional methods. Furthermore, the study validates GPT-4 as an effective, cost-efficient automatic annotator, achieving 89% accuracy while significantly reducing evaluation costs. The evaluation of six advanced models reveals that, despite notable progress, models still struggle with complex instructions, especially in numerical reasoning and nuanced language understanding.
This work offers a scalable, interpretable, and detailed assessment framework that can guide future model development, ensuring safer and more reliable AI systems. The combination of DRFR and InFoBench sets a new standard for instruction-following evaluation, with broad implications for both research and industry, fostering the development of more capable and trustworthy language models.
Deep Analysis
Background
近年来，随着GPT-3、BERT等模型的问世，NLP领域迎来了大模型的快速发展。这些模型在文本生成、理解和推理方面表现卓越，但在实际应用中，模型是否能准确遵循用户指令仍是核心难题。传统评估方法如A/B测试、整体评分和Elo评级，虽然在一定程度上反映了模型性能，但存在可扩展性差、解释性不足的问题，难以细致分析模型在复杂、多约束场景下的表现。尤其是在多任务、多领域应用中，模型的指令遵循能力成为限制其推广的瓶颈。为此，学界和工业界亟需一种既能细粒度评估，又具备良好扩展性的工具，以推动模型的持续优化。
Core Problem
现有评估体系多依赖主观评分或单一指标，难以全面反映模型在多维约束条件下的遵循情况。传统方法缺乏对复杂指令多方面表现的细致分析，无法满足实际应用中对模型可靠性和安全性的高要求。此外，缺少自动化、可扩展的评估工具限制了大模型的快速迭代。如何设计一种既能细粒度衡量，又能自动化执行的评估体系，成为行业亟待解决的问题。
Innovation
本文的主要创新包括：1）提出DRFR指标，将复杂指令拆解为多项二元判定问题，提升评估的细粒度和可靠性；2）构建InFoBench基准，涵盖多领域、多约束类别，确保评估的多样性和代表性；3）引入GPT-4作为自动标注工具，显著降低成本，提升效率，验证了自动化评估的可行性。这些创新共同推动了指令遵循能力的量化研究，为模型优化提供了科学依据。
Methodology
- �� 指令拆解：将复杂指令分解为多项二元判定问题，每个问题对应一个具体约束。• 标注机制：结合专家、众包和GPT-4多源标注，确保标注的多样性和可靠性。• DRFR计算：用公式
drfr=∑_{i,j} r_{i,j} / \sum_{i} m_i，衡量模型对所有拆解标准的整体遵循比例。• 约束类别：内容、语言、风格、格式、数字五类，确保评估的多维度。• 数据集构建：设计Easy和Hard两套指令集，覆盖不同复杂度和领域。• 自动标注：采用多轮GPT-4提示，结合上下文信息，自动评估模型输出。• 可靠性验证：通过与专家评分的对比，验证指标的稳定性和一致性。
Experiments
- �� 数据：使用InFoBench的500条指令，涵盖多个领域和约束类型。• 模型：评估GPT-3.5、GPT-4、Claude-v1、Alpaca-7B、Vicuna-13B等六个模型。• 评估方法：比较DRFR与传统直接评分（DS），采用多评估者一致性指标（Fleiss’ Kappa）。• 标注源：专家、众包和GPT-4，分析成本、效率和准确性。• 实验设计：多轮标注、交叉验证，确保评估的稳健性。
Results
- �� DRFR在Hard Set中的Fleiss’ Kappa达0.493，高于传统评分0.302，显示其更高的评估一致性。• GPT-4自动标注准确率达89%，成本仅为专家的1/80，验证了其作为自动评估工具的有效性。• 六模型中，闭源模型表现优异，但在复杂指令场景中仍存在明显差距，特别是在数值和语言理解方面，提示未来需加强多模态和推理能力。
Applications
- �� 立即应用：该指标和基准可用于模型开发中的性能调优，帮助研究者识别模型在特定约束下的不足。• 长期愿景：推动构建更智能、更安全的自然语言系统，提升模型在实际场景中的可靠性和适应性，为智能助手、自动客服等行业提供坚实基础。
Limitations & Outlook
- �� 指标主要基于二元判定，可能忽略模糊或主观要求的细微差异。• 在多模态、多任务环境中表现仍有限，需结合多模态信息优化。• 自动标注虽高效，但在边界场景存在误判风险，需结合人工校验提升准确性。
Plain Language Accessible to non-experts
想象你在厨房做饭，指令就像食谱。复杂的菜肴需要你按照不同步骤操作，比如切菜、调味、烹饪。每个步骤都要符合要求，否则菜就做不好。这个研究就像给厨师设计一份详细的菜谱，确保每个步骤都正确完成。DRFR就像检查每个步骤是否做到位，拆解复杂任务成简单任务逐一确认。InFoBench则像准备了各种不同难度的菜谱，从简单到复杂，帮助评估厨师的水平。用GPT-4自动检测厨师的表现，就像用智能助手帮忙评分。整体目标是让厨师（模型）学会做出符合要求的菜肴，确保每一道菜都好吃、符合标准。这种方法让评估变得更细致、更科学，也更节省时间和成本。
ELI14 Explained like you're 14
想象你在学校里参加一个烹饪比赛，裁判会根据菜的味道、摆盘、创新等方面打分。以前，裁判只会给一个总分，但这样不够公平，也不能告诉你哪方面做得好或不好。现在，评委们设计了一套详细的规则，把每个方面拆开，比如味道、颜色、创意，然后逐项打分。这样一来，就能清楚知道哪方面需要改进。这个研究就像用这种详细的评分方法，帮助模型（就像厨师）学会按照指令做菜。用AI（像GPT-4）自动帮忙评分，不仅快，还能节省很多钱。结果显示，这种方法比传统的评分更稳定、更准确。虽然还存在一些不足，比如在特别复杂的菜肴上还需要改进，但未来可以让模型变得更聪明、更可靠，就像一个厨艺大师一样！
Abstract
This paper introduces the Decomposed Requirements Following Ratio (DRFR), a new metric for evaluating Large Language Models' (LLMs) ability to follow instructions. Addressing a gap in current methodologies, DRFR breaks down complex instructions into simpler criteria, facilitating a detailed analysis of LLMs' compliance with various aspects of tasks. Alongside this metric, we present InFoBench, a benchmark comprising 500 diverse instructions and 2,250 decomposed questions across multiple constraint categories. Our experiments compare DRFR with traditional scoring methods and explore annotation sources, including human experts, crowd-sourced workers, and GPT-4. The findings demonstrate DRFR's higher reliability and the effectiveness of using GPT-4 as a cost-efficient annotator. The evaluation of several advanced LLMs using this framework reveals their strengths and areas needing improvement, particularly in complex instruction-following. This study contributes a novel metric and benchmark, offering insights for future LLM development and evaluation.