Large Language Models Are State-of-the-Art Evaluators of Translation Quality

The evolution of machine translation (MT) has transitioned from rule-based systems to neural models, dramatically improving fluency and accuracy. Traditional metrics like BLEU, METEOR, and TER rely on lexical overlap, which often fails to reflect semantic adequacy, especially for high-quality translations. Neural metrics such as COMET and BLEURT introduced learned representations, improving correlation with human judgments. However, these still depend on reference translations and require task-specific training or fine-tuning. Recently, large pre-trained models like GPT-3 and GPT-4 have demonstrated exceptional language understanding and generation, suggesting their potential as universal evaluators. Despite this, systematic validation of GPTs for translation quality assessment remains limited, especially in zero-shot scenarios, which are crucial for scalability and multilingual applications.

The core challenge is to develop an automatic, scalable, and accurate translation quality assessment method that does not rely heavily on reference translations or extensive training. Existing metrics often struggle with semantic adequacy, fluency, and cross-lingual consistency. Human evaluation, though accurate, is impractical for large-scale deployment due to high costs and time constraints. The difficulty lies in designing prompts that effectively leverage GPT's capabilities to produce meaningful quality scores or classifications, while ensuring consistency and robustness across languages and domains. Additionally, aligning model outputs with human judgments at both system and segment levels remains a significant hurdle, especially given the discrete nature of many scoring schemes.

The paper introduces GEMBA, a novel prompt-based evaluation framework utilizing GPT-3.5 and GPT-4 models. It innovates by: 1) designing multiple prompt variants tailored for scoring and classification tasks, 2) enabling zero-shot assessment without model fine-tuning, 3) aggregating segment-level scores into system-level metrics, and 4) validating performance on the WMT22 MQM dataset. Unlike prior metrics, GEMBA exploits GPT's natural language understanding and generative abilities to directly produce quality estimates, bypassing complex feature engineering. The approach’s flexibility allows it to operate both with and without reference translations, broadening its applicability. The experimental results establish new state-of-the-art performance, demonstrating the potential of large models as universal evaluators.

• �� Input: source segments, candidate translations, optional reference translations, model choice, prompt template.
• �� Prompt design: four variants (DA, SQM, stars, classes), customized for each task and mode.
• �� Model inference: GPT-3.5/4 generates responses to prompts, producing scores or labels.
• �� Output parsing: extract numerical scores, handle invalid answers by re-prompting with adjusted temperature.
• �� Score aggregation: average segment scores for system-level metrics.
• �� Evaluation: system accuracy via pairwise comparison, segment correlation via Kendall’s Tau.
• �� Experiments: conducted on WMT22 MQM data across three language pairs, comparing against baselines like BLEURT, COMET.
• �� Hyperparameters: prompt variations, temperature settings, model versions, evaluated for robustness.

The evaluation used WMT22 MQM annotations for English-German, English-Russian, and Chinese-English, totaling over 100,000 segments. The metrics compared include system-level accuracy and segment-level Kendall’s Tau. Baselines such as BLEURT, COMET, and other neural metrics were included. The models tested ranged from GPT-2 to GPT-4, with prompt variants optimized for each task. Hyperparameters like prompt wording, temperature, and response parsing strategies were systematically varied. The experiments aimed to quantify the correlation with human judgments, assess the impact of model scale, and evaluate the zero-shot capability of GPT-based metrics across languages and domains.

GEMBA-GPT4-DA achieved 89.8% system accuracy, outperforming all baseline metrics, with the no-reference version reaching 87.6%. The results demonstrate that larger models, especially GPT-4, significantly improve evaluation quality. The metrics maintained high correlation at the system level, with GPT-4 surpassing previous neural metrics. Segment-level Kendall’s Tau scores, while slightly lower, still showed strong alignment with human judgments. The experiments confirmed that prompt design and model scale are critical factors, with the most constrained prompts yielding the best performance. These findings validate the feasibility of using large language models for high-quality, scalable translation evaluation.

This approach can be integrated into automated MT pipelines for real-time quality monitoring, model selection, and fine-tuning. It is suitable for multilingual, large-scale translation systems, reducing reliance on costly human assessments. The method supports both reference-based and reference-free evaluation, making it versatile for diverse scenarios. Industries such as localization, content moderation, and international communication can benefit from rapid, accurate quality metrics, enabling continuous improvement and deployment of high-quality translation models.

Despite strong system-level results, segment-level correlations are affected by discrete scoring and ties, limiting fine-grained analysis. The computational cost of large GPT models restricts real-time deployment in resource-constrained environments. The validation dataset is limited to high-resource languages, and performance on low-resource or under-represented languages remains uncertain. Additionally, potential data contamination from training data cannot be fully ruled out, necessitating further validation for broader generalization.