BLEURT: Learning Robust Metrics for Text Generation

Natural language generation (NLG) has experienced rapid growth, driven by neural models like Transformer architectures. Early evaluation metrics such as BLEU and ROUGE relied on lexical overlap, which inadequately captured semantic quality. Recent deep learning-based metrics like BEER, RUSE, and ESIM improved correlation with human judgments but still faced challenges in generalization and robustness, especially under domain shifts and limited data. The need for more reliable, scalable, and domain-agnostic evaluation methods has become urgent, prompting exploration of BERT-based models and synthetic data augmentation techniques to enhance evaluation fidelity.

Current automatic evaluation metrics often fail to align with human judgments, particularly when data is scarce or distributions shift over time. Traditional metrics lack semantic sensitivity, leading to unreliable assessments of translation or generation quality. Deep learning models, while promising, require extensive labeled data and struggle with out-of-domain scenarios. This creates a bottleneck for rapid development and deployment of NLP systems, necessitating a robust, scalable, and domain-adaptive evaluation framework that can operate effectively with limited human annotations.

This work introduces a multi-faceted pretraining approach for BERT-based evaluation models, leveraging large-scale synthetic sentence pairs generated via masking, backtranslation, and word deletion. It combines multiple signals—BLEU, ROUGE, BERTscore, translation likelihood, and textual entailment—within a multi-task learning framework, enabling the model to learn nuanced quality distinctions. Unlike prior methods that rely solely on embeddings or handcrafted rules, BLEURT's end-to-end training captures complex language phenomena, offering a balanced trade-off between expressivity and robustness. This approach significantly enhances generalization, especially in low-resource and out-of-distribution scenarios.

• �� Generate synthetic sentence pairs from Wikipedia using BERT masking, backtranslation, and word deletion techniques, creating diverse variations.
• �� Extract multiple pretraining signals: BLEU, ROUGE, BERTscore (precision, recall, F-score), translation probabilities, and textual entailment labels.
• �� Design a multi-task loss function combining regression and classification objectives, optimizing the model to predict human-like scores.
• �� Pretrain BERT on these synthetic pairs with multi-task learning, capturing lexical, syntactic, and semantic differences.
• �� Fine-tune the pretrained model on limited human ratings for specific tasks like translation quality and data-to-text evaluation.
• �� Evaluate performance on benchmarks such as WMT Metrics Shared Task and WebNLG, analyzing robustness under data skew and domain shifts.

The experiments utilized datasets from the WMT Metrics Shared Task (2017-2019), with training on previous years' data and testing on subsequent years. Multiple BLEURT variants, based on BERT-large and BERT-base, were trained with and without synthetic pretraining. Evaluation metrics included Kendall Tau and Pearson correlation with human ratings. Additional synthetic datasets simulated quality drift scenarios by skewing training data distributions. Ablation studies assessed the contribution of each pretraining signal. Hyperparameters included batch size 32, learning rate 1e-5, pretraining steps 800,000, and fine-tuning steps 40,000, ensuring thorough training.

BLEURT consistently outperformed baseline metrics, achieving up to 7.4-point improvements in correlation scores across all years. Pretraining significantly boosted robustness, maintaining high correlation even under severe data skew (skew factor up to 3.0). On WebNLG, BLEURT demonstrated rapid adaptation with limited data, outperforming traditional metrics like BLEU, METEOR, and BERTscore. Ablation results confirmed that synthetic pretraining and multi-task signals were critical for optimal performance, especially in low-resource and out-of-domain settings.

BLEURT can be deployed for automatic evaluation in machine translation, summarization, dialogue systems, and content generation, reducing reliance on costly human judgments. Its robustness makes it suitable for real-time quality monitoring and system tuning across diverse domains. The model's transferability enables quick adaptation to new tasks with minimal data, accelerating development cycles and improving system reliability in industrial NLP pipelines.

Despite its strengths, BLEURT's computational cost remains high due to large-scale pretraining. Its performance may decline in languages or domains with limited synthetic data generation capabilities. Extreme distribution shifts or highly specialized content could challenge its robustness. Future work should focus on efficiency improvements, multilingual extension, and domain-specific adaptation to broaden applicability.