Foundational Autoraters: Taming Large Language Models for Better Automatic Evaluation

The rapid development of large language models (LLMs) has revolutionized NLP, enabling applications from chatbots to code generation. However, evaluating their performance remains a bottleneck. Traditional metrics like BLEU, ROUGE, and BERTScore are limited in capturing semantic nuances. Recent efforts have used LLMs themselves as evaluators, but these often inherit biases, favor responses similar to their own, and lack fairness. Human evaluations, though accurate, are costly, subjective, and inconsistent across studies. The challenge is to develop scalable, unbiased, and generalizable automatic evaluation methods that can adapt to diverse tasks and data sources. Prior work on task-specific metrics and model-based evaluators has shown promise but falls short in flexibility and fairness. This context motivates the creation of FLAMe, leveraging open human judgment datasets to train a universal auto-evaluator capable of handling multiple evaluation scenarios reliably.

Current automatic evaluation methods face multiple issues: reliance on proprietary data, limited generalization, bias amplification, and high costs of human annotation. Model-based evaluators often favor their own outputs, leading to unfair assessments. The lack of standardized, publicly available datasets hampers reproducibility and fairness. Moreover, models trained on narrow datasets struggle to adapt to new tasks or domains. Addressing these issues requires a new approach that can utilize openly licensed human judgments, standardize diverse evaluation tasks, and produce models with broad applicability, fairness, and efficiency. The core problem is how to build such a universal, unbiased, and scalable auto-evaluator that can serve both research and industry needs.

The paper's key innovations include: 1) Curating a large, diverse dataset of 102 tasks with over 5.3 million human judgments from open sources, ensuring transparency and reproducibility. 2) Standardizing all tasks into a unified text-to-text format, enabling effective multi-task learning. 3) Training a base instruction-tuned PaLM-2-24B model via supervised multi-task fine-tuning, resulting in FLAMe, a versatile auto-evaluator with strong generalization. 4) Developing reward modeling and tail-patch fine-tuning techniques to optimize performance on specific tasks with significantly less data. 5) Demonstrating that open data can rival proprietary datasets, reducing dependence on costly annotations and proprietary models. These innovations collectively enable a scalable, fair, and high-performing automatic evaluation framework.

• �� Data collection: Extract human judgments from permissively licensed datasets across HuggingFace and GitHub, ensuring diversity. • Data standardization: Review research, consult authors, extract evaluation fields, craft detailed task definitions and instructions. • Format conversion: Reformat all tasks into a text-to-text structure with input, context, and evaluation blocks. • Model training: Fine-tune PaLM-2-24B on the multi-task dataset for 30K steps, using example-proportional mixture weights capped at 216 per task. • Downstream fine-tuning: Further fine-tune on specific evaluation datasets (e.g., reward modeling) with 50-5000 steps, applying tail-patch strategies to optimize task-specific performance. • Evaluation: Test on RewardBench, LLM-AggreFact, and bias benchmarks, comparing accuracy, bias, and response quality across models.

Experiments involve training FLAMe on curated datasets covering quality, safety, reasoning, coding, and factuality. Baselines include GPT-4, Claude-3, and Llama-3. Metrics focus on accuracy, bias, and generalization. Ablation studies compare multi-task training, reward modeling, and tail-patch fine-tuning. Results show FLAMe achieves 87.8% accuracy on RewardBench, outperforming proprietary models. Bias analysis indicates reduced unfairness, and the model effectively re-ranks high-quality responses. The experiments validate the benefits of large-scale open data, multi-task training, and efficient fine-tuning strategies.

FLAMe-24B surpasses GPT-4-0125 and GPT-4o on RewardBench with 87.8% accuracy, a notable improvement over previous models. Tail-patch fine-tuning achieves comparable performance with only 1/25th of the data, demonstrating high data efficiency. Bias evaluations show FLAMe is significantly less biased than proprietary LLMs, especially on the CoBBLEr benchmark. The model generalizes well across 12 benchmarks, excelling in bias detection, response ranking, and multiple quality attributes, confirming the robustness of multi-task training and data standardization.

This model can be used for automated content moderation, AI safety assessment, bias detection, and response ranking in chatbots and virtual assistants. Its ability to evaluate diverse tasks with minimal data makes it suitable for industry deployment, reducing reliance on costly human evaluations. In the long term, it can serve as a foundation for developing fairer, more transparent AI systems, supporting regulatory compliance and user trust. Additionally, it can be integrated into AI development pipelines for continuous model improvement and benchmarking.

Despite its strengths, FLAMe relies on the quality of curated human judgments, which may contain biases or inconsistencies. Its performance on highly specialized or emerging tasks remains untested. Computational costs for training and fine-tuning, though lower than proprietary models, still pose barriers for some users. Future work should focus on multi-modal capabilities, bias mitigation, and enhancing interpretability to broaden applicability and trustworthiness.