Learning Compact Metrics for MT

Multilingual NLP has seen rapid advances with models like BERT, XLM-R, and mT5, enabling cross-lingual transfer and multi-task learning. Evaluation metrics evolved from lexical overlap measures such as BLEU to learned metrics like COMET and BLEURT, which utilize deep pre-trained models to better reflect human judgment. However, as models grow larger to accommodate more languages, they face scalability issues, especially in deployment scenarios with limited hardware resources. The 'curse of multilinguality' describes the phenomenon where increasing languages in a fixed-capacity model leads to performance degradation, limiting the effectiveness for low-resource languages. Researchers have sought methods to compress models without sacrificing accuracy, but balancing size and performance remains challenging.

Current multilingual evaluation models are hampered by their large size, which hampers deployment efficiency and scalability. The capacity bottleneck restricts the number of languages supported and diminishes performance in low-resource scenarios. Moreover, training large models is computationally expensive, limiting accessibility for many researchers and practitioners. Addressing these issues requires developing methods to retain high evaluation accuracy while significantly reducing model parameters, enabling practical deployment in diverse settings. The core challenge lies in effectively transferring knowledge from large models to smaller ones without losing critical semantic understanding, especially across many languages.

This work introduces a hybrid approach combining synthetic data generation and multi-model knowledge distillation to create a lightweight yet high-performing multilingual evaluation metric. Key innovations include:

1) Synthetic data creation using Wikipedia perturbations, simulating translation errors across 13 languages, thus enriching training data without requiring additional labeled datasets.

2) Training a large teacher model (RemBERT-32) on this synthetic corpus to serve as a high-quality reference.

3) Applying pre-trained distillation to transfer knowledge from the teacher to smaller student models (RemBERT-3/6/12), drastically reducing parameters.

4) Employing multi-language fine-tuning and clustering strategies to improve transferability and performance in low-resource languages.

These innovations collectively address the capacity bottleneck, enabling efficient, scalable evaluation models.

• �� Synthetic data generation: Use Wikipedia perturbations—back-translation, word substitution with mBERT, and random deletions—to produce 80 million sentence pairs in 13 languages.
• �� Teacher training: Fine-tune RemBERT-32 on synthetic corpus, optimizing for correlation with human ratings.
• �� Knowledge distillation: Run teacher model on unlabeled data, generate pseudo-labels, then train smaller models (RemBERT-3/6/12) to mimic teacher outputs.
• �� Multi-model training: Cluster languages based on linguistic similarity, train dedicated models for each cluster, and transfer knowledge via distillation.
• �� Fine-tuning: Use WMT metrics data to adapt models for evaluation tasks, validating cross-lingual transfer.
• �� Speed optimization: Implement length-based batching and model distillation to improve inference speed, enabling deployment in real-world scenarios.

The experiments utilize WMT 2020 metrics shared task data, comprising 479k triplets across 12 languages. Models are trained with Adam optimizer, learning rate schedules, and evaluated using DaRR correlation scores. Baselines include COMET and BLEURT-EXTENDED. Synthetic data augmentation and multi-model distillation are systematically tested, with hyperparameters tuned for optimal performance. Additional ablation studies compare different synthetic perturbation strategies, model sizes, and fine-tuning configurations. Cross-lingual transfer is assessed through zero-shot evaluations on unseen languages, demonstrating robustness and scalability of the proposed approach.

The distilled models achieve 92.6% of the teacher model’s performance, surpassing baseline fine-tuning by 10.5%. Across languages, they outperform SOTA metrics like COMET and BLEURT-EXTENDED, with improvements of up to 4% in DaRR scores. Synthetic data and multi-model distillation significantly boost low-resource language performance, especially in zero-shot settings. Speed tests show that the compressed models process 4.8 tuples/sec without GPU, representing a 2.5-3x speedup over the original large model. These results confirm that the proposed framework effectively balances model size, transferability, and accuracy.

The lightweight models can be deployed for real-time machine translation evaluation, quality estimation in multilingual systems, and low-resource language assessment. They are suitable for integration into translation pipelines, enabling faster and more cost-effective quality control. The approach also benefits research in low-resource NLP, facilitating the development of evaluation metrics for underrepresented languages. Future applications include expanding to other NLP tasks such as summarization and question answering, where model efficiency is critical for large-scale deployment.

While synthetic data improves robustness, it may not fully replicate real translation errors, limiting generalization. The multi-model training process increases development complexity and resource consumption. The reliance on large teacher models for initial training remains costly, and extending coverage to extremely low-resource languages requires additional data collection and adaptation. Future work should focus on further reducing training costs, improving synthetic data realism, and exploring unsupervised approaches to enhance scalability.