Ask, Don't Judge: Binary Questions for Interpretable LLM Evaluation and Self-Improvement
BINEVAL decomposes evaluation into binary questions, enabling interpretable, multi-dimensional assessment with superior correlation to human judgments.
Key Findings
Methodology
BINEVAL employs a meta-prompt to convert task prompts into fine-grained binary questions across evaluation dimensions. An evaluator independently answers each question, producing transparent verdicts with explanations. These answers are aggregated into per-dimension and overall scores. The framework integrates cross-model and self-prompt optimization by analyzing disagreements and extracting lessons for iterative prompt refinement. This approach leverages specific algorithms such as question generation (FLLM), independent evaluation functions (fE), and a two-phase prompt update process, validated on datasets like SummEval, Topical-Chat, and QAGS, demonstrating superior performance over baselines like UniEval and G-Eval.
Key Results
- On SummEval, BINEVAL (Claude) achieved a Spearman correlation of 0.655, surpassing previous methods, especially excelling in factual consistency with a correlation of 0.655, effectively distinguishing borderline outputs from flawed ones.
- In Topical-Chat, BINEVAL (Claude) reached 0.632 correlation, outperforming other evaluation methods, indicating robustness in subjective dialogue quality assessment.
- On QAGS, BINEVAL (Claude) demonstrated the highest factual consistency correlation of 0.655, confirming the effectiveness of binary decomposition in factual verification tasks.
Significance
This work advances automatic evaluation by transforming opaque scalar scores into transparent, diagnostic multi-dimensional scores. It addresses core issues of bias, lack of interpretability, and poor discrimination in existing metrics, facilitating better model debugging, prompt tuning, and iterative development. Its task-agnostic, training-free design makes it highly adaptable for industry-scale deployment, fostering trustworthy and explainable AI systems.
Technical Contribution
The paper introduces a novel framework combining meta-prompt-driven binary question generation with independent evaluation and multi-dimensional scoring. It innovates by enabling cross-model and self-prompt optimization based on disagreement analysis, without requiring task-specific training data. The approach provides theoretical guarantees of interpretability and robustness, validated through extensive experiments, setting a new standard for automated, explainable NLP evaluation.
Novelty
This is the first systematic application of atomic binary question decomposition for evaluation metrics, integrating it with iterative prompt refinement. Unlike prior holistic or single-judgment methods, this approach offers granular, explainable feedback, significantly improving discrimination and correlation with human judgments, marking a fundamental shift in evaluation paradigm.
Limitations
- The binary question approach struggles with subjective or abstract qualities like creativity or humor, where simple yes/no answers are insufficient to capture nuance.
- The method heavily depends on the quality of generated questions; poorly designed questions can mislead evaluation.
- Computational costs increase with the number of questions, and prompt design remains partly manual, limiting scalability in some scenarios.
Future Work
Future directions include integrating multi-modal inputs for comprehensive evaluation, developing adaptive question generation to better handle subjective tasks, and exploring reinforcement learning techniques for automated prompt refinement. Extending the framework to real-time evaluation and incorporating human-in-the-loop feedback are also promising avenues.
AI Executive Summary
The rapid development of large language models (LLMs) has revolutionized natural language processing, enabling fluent and contextually rich outputs across diverse tasks. However, evaluating these outputs remains a significant bottleneck. Human assessments, while accurate, are slow and costly, making large-scale deployment impractical. Automated metrics like ROUGE, BLEU, and BERTScore offer convenience but often fail to capture semantic correctness, factuality, or nuanced quality aspects. Recent efforts have turned to using LLMs themselves as evaluators, but these models tend to produce opaque scores, suffer from biases, and lack interpretability.
In response, this paper introduces BINEVAL, a novel evaluation framework that decomposes complex criteria into atomic binary questions. Using a meta-prompt, the system generates targeted questions for each evaluation dimension, which are answered independently by an LLM, producing transparent verdicts with explanations. These verdicts are aggregated into multi-dimensional scores, providing a detailed and interpretable assessment of output quality. This approach not only aligns more closely with human judgments but also improves the discrimination between borderline and flawed outputs.
Extensive experiments on benchmarks such as SummEval, Topical-Chat, and QAGS demonstrate that BINEVAL outperforms traditional metrics and existing LLM-based evaluators, achieving higher correlation scores and better distribution matching. The framework further supports iterative prompt optimization through disagreement analysis, enabling continuous self-improvement of both evaluators and generators. This dual capability enhances evaluation robustness and facilitates model fine-tuning without task-specific training.
Overall, BINEVAL offers a task-agnostic, training-free, and highly interpretable evaluation paradigm that addresses key limitations of prior methods. Its practical diagnostic and optimization utilities make it a valuable tool for advancing trustworthy, transparent, and scalable NLP systems. Future work aims to extend this approach to multi-modal assessment and real-time evaluation, further broadening its impact.
Deep Analysis
Background
随着GPT、PaLM等大规模预训练模型的崛起，模型生成能力显著提升，但评估方法仍面临挑战。传统指标如ROUGE、BLEU主要依赖词汇重叠，难以衡量语义一致性。Embedding-based指标如BERTScore引入语义匹配，但仍不足以反映事实正确性。近年来，基于LLMs作为评判者的方法逐步发展，如G-Eval、UniEval等，尝试用模型自身进行多维度评估，但存在偏差和不透明的问题。多维度评估（如信实性、连贯性）逐渐成为趋势，但其实现依赖复杂的规则或训练数据，限制了泛用性。Atomic decomposition（原子分解）在事实验证中取得一定成功，但尚未系统性应用于指标拆解。Prompt优化逐渐成为提升模型性能的关键手段，但缺乏有效的自动化机制。整体来看，评估的透明性、可解释性和鲁棒性仍是亟待解决的问题。
Core Problem
现有评估方法多为单一分数或模糊判定，难以提供细粒度、可诊断的反馈，限制了模型调试和Prompt优化的效率。人类评估虽准确，但成本高、速度慢，难以满足大规模应用需求。自动指标虽便捷，但缺乏可解释性，且在事实验证和主观评价中表现不足。模型作为评判者存在偏差、偏见和不一致性，影响评估的可靠性。如何设计一种既具透明度又能细粒度反映输出质量的评估体系，成为当前的核心难题。特别是在开放域、多任务场景下，单一指标难以兼顾多维度需求，亟需一种结构化、可解释的评估框架。
Innovation
本研究提出基于元提示的二元问题生成机制，将复杂评价指标拆解为多项可单独回答的二元问题，提升评估的透明度和细粒度。引入模型独立回答机制，确保评估结果的客观性。结合多维度评分体系，能更准确反映不同质量维度。提出跨模型和自我提示优化算法，通过模型间的争议细节指导Prompt改进，提升评估一致性和鲁棒性。该方法无需任务特定训练，具有极强的泛用性，适用于多种任务和模型，推动了自动化、可解释评估的技术发展。
Methodology
- �� 输入任务描述（T）：定义生成任务（如摘要、对话）
- �� 元提示（M）生成：将T拆解为多个评价维度（如连贯性、事实性）
- �� 二元问题生成（Q = FLLM(T; M)）：每个维度对应一组二元问题（如‘输出是否包含关键信息？’）
- �� 独立评估（fE(x,y,qi)）：模型回答每个问题，输出“是”或“否”，附带解释
- �� 评分整合：根据回答计算每个维度和整体得分
- �� 跨模型优化：分析争议，提取改进建议，更新提示
- �� 自我优化：模型基于自身评估调整Prompt，持续改进
- �� 迭代循环：重复上述步骤，提升评估一致性和准确性
Experiments
在SummEval、Topical-Chat和QAGS上验证，使用GPT-OSS-120B和Claude Sonnet 4作为评估模型。指标包括Spearman、Kendall和Pearson相关系数。对比传统指标和基于LLMs的评估，进行AB测试和提示优化。采用不同的二元问题设计，验证其对评估准确性和鲁棒性的影响。还进行跨模型和自我提示的迭代优化，观察其对评估一致性和细粒度的提升。实验设置确保多任务、多模型的广泛适应性。
Results
BINEVAL在SummEval上相关系数最高达0.655，优于UniEval和G-Eval，特别在事实一致性方面表现优异（相关系数0.655）；在Topical-Chat中，相关性达0.632，优于对比方法；在QAGS中，事实验证相关性最高（0.655），验证了二元问题拆解在提升评估精度方面的有效性。提示优化实验显示，跨模型和自我优化均能显著改善评估相关性，平均提升约0.07-0.09，验证其有效性。
Applications
该框架可广泛应用于模型调试、Prompt优化、自动化评估体系建设。适合大规模生成任务的快速质量检测，特别在事实验证、对话系统和摘要生成中表现优越。未来可结合多模态信息，支持多任务、多模态的评估需求，推动AI透明化和可控性。
Limitations & Outlook
当前方法在处理高度主观或抽象的评价维度（如创造性、幽默感）时效果有限，因二元问题难以捕捉复杂语义。对模型的依赖较大，评估结果受模型理解能力影响。提示设计仍需人工干预，自动化程度不足，未来需优化生成策略以增强鲁棒性。
Plain Language Accessible to non-experts
想象你在一家工厂里，生产线上的每个环节都很重要。以前，我们只关注最终产品的整体质量，但这样太模糊。现在，我们把整个生产过程拆成很多小步骤，比如检查包装是否完整、颜色是否正确、标签是否贴好。每个小步骤用一个简单的“是”或“否”来判断。这样一来，工厂管理者可以清楚知道哪个环节出了问题，哪里需要改进。BINEVAL的想法也是一样，它把复杂的评价任务拆成很多简单的“是”或“否”问题，每个问题都能清楚回答，最后组合成一份详细的报告。这让评估变得更透明、更容易诊断，也更方便优化整个系统。
ELI14 Explained like you're 14
想象你在学校里，老师要给你打分。以前，老师只会给你一个总分，比如80分或90分，但你不知道哪里做得好，哪里需要改进。现在，老师用了一种新方法，把你的表现拆成很多小问题，比如“你是否按时完成作业？”、“你的回答是否完整？”、“你的表达是否清楚？”每个问题都只需要回答“是”或“否”。这样，老师可以很清楚地知道你在哪些方面表现不错，哪些地方需要努力。BINEVAL也是这样，它把复杂的评价变成很多简单的“是”或“否”问题，最后把答案组合成一份详细的评估报告。这让评价变得更公平、更透明，也方便你自己知道哪里可以改进。
Glossary
Meta-prompt (元提示)
一种引导模型生成具体任务细节的提示，用于拆解任务为二元问题。
用于生成细粒度评估问题的关键机制。
Binary question (二元问题)
只需回答“是”或“否”的问题，用于细粒度评估。
核心评估单元，拆解复杂指标。
Multi-dimensional scores (多维评分)
结合多个二元问题的回答，形成多方面的输出质量指标。
实现输出的全面、可解释评估。
Cross-model prompt update (跨模型提示更新)
利用不同模型之间的争议细节，优化评估提示。
提升评估一致性和鲁棒性的方法。
Self-prompt update (自我提示更新)
模型基于自身评估反馈调整生成提示，持续改进。
实现模型自我优化的机制。
Open Questions Unanswered questions from this research
- 1 如何进一步提升二元问题在主观性强任务中的表现？当前方法在多模态评估中的适应性如何？未来是否能结合人类反馈实现更智能的提示优化？
Applications
Immediate Applications
模型调试与Prompt优化
通过二元问题详细诊断模型输出缺陷，快速调整Prompt，提高生成质量。
自动化评估体系建设
为大规模模型提供透明、可解释的评估工具，减少人工成本，提升效率。
Long-term Vision
多模态、多任务评估平台
结合视觉、声音等多模态信息，构建全场景、多任务的自动评估体系，推动AI透明化。
Abstract
Evaluating LLM outputs remains a major bottleneck in NLP: human evaluation is expensive and slow, lexical metrics correlate poorly with human judgments on open-ended generation, and holistic LLM judges often produce opaque scores that are hard to debug. We propose BINEVAL, a framework that decomposes evaluation criteria into atomic binary questions and aggregates the resulting verdicts into interpretable, multi-dimensional scores. Given a task prompt, a meta-prompt generates fine-grained evaluation questions, and an LLM answers them independently for each output, yielding transparent question-level feedback together with calibrated overall scores. This decomposition makes evaluation easier to inspect, easier to diagnose, and directly usable for prompt improvement. Across SummEval, Topical-Chat, and QAGS, BINEVAL matches or outperforms strong baselines including UniEval and G-Eval, with especially strong results on factual consistency benchmarks such as QAGS. Beyond competitive correlation with human judgments, BINEVAL better matches human score distributions and avoids the ceiling effects common in prior LLM judges, leading to better discrimination between borderline and clearly flawed outputs. We further show that the same question-level feedback supports iterative prompt optimization, improving evaluator prompts on summarization and generation prompts on IFBench under both self-update and cross-model update settings. Overall, BINEVAL provides a task-agnostic, training-free, and interpretable evaluation framework that combines strong empirical performance with practical diagnostic and optimization value.