MetricX-25 and GemSpanEval: Google Translate Submissions to the WMT25 Evaluation Shared Task

Recent years have seen rapid progress in neural machine translation (NMT), driven by models like Transformer and multilingual pretraining architectures such as mT5 and Gemma 3. Traditional metrics like BLEU and ChrF, based on lexical overlap, are limited in capturing semantic adequacy and fluency. To address this, neural evaluation metrics such as BLEURT, COMET, and MetricX emerged, leveraging large-scale annotated data and deep models to predict quality scores. These models have improved correlation with human judgments and enabled more detailed error analysis. However, challenges remain in achieving fine-grained error localization, especially in long texts and low-resource languages. The advent of large multilingual models like Gemma 3, supporting extensive contexts and language coverage, offers new opportunities for more accurate and interpretable evaluation methods. Recent shared tasks like WMT have driven research in multi-task learning, structured scoring, and generative error detection, aiming to bridge the gap between automatic and human evaluation.

Despite advances, current automatic evaluation models struggle with accurately localizing errors, especially in lengthy segments or under low-resource conditions. Existing metrics often provide a single scalar score, lacking detailed insights into error types, severities, and locations. This limits their usefulness for targeted improvements in translation systems. Moreover, handling non-unique error spans and providing unambiguous error contexts remains difficult. The challenge is to develop models that can deliver both accurate overall scores and detailed, interpretable error annotations, supporting real-time feedback and iterative system enhancement. Achieving this requires integrating large multilingual models with sophisticated training strategies and output mechanisms capable of detailed span and error classification.

The core innovations include adapting Gemma 3 into an encoder-only architecture for efficient score prediction, and designing GemSpanEval as a generative model capable of outputting error spans with context. The multi-task training approach enables simultaneous prediction of multiple score types (MQM and ESA) within a single model, facilitated by input score type indicators and long-context support. The span detection model extends traditional classification by generating structured JSON outputs, including error categories, severities, and contextual information, ensuring unambiguous localization. These innovations address the limitations of prior models in handling long texts, multi-language scenarios, and detailed error reporting, pushing the state-of-the-art in automatic MT evaluation.

• �� Data: Utilized WMT15-23 datasets with human annotations (DA, MQM, ESA), covering 50+ language pairs, including synthetic examples for robustness.
• �� Model architecture: Based on Gemma 3 12B, modified to encoder-only with mean pooling for regression tasks.
• �� Input design: Enriched with source/target language info, score type indicators, and segment boundaries, supporting multi-task learning.
• �� Training protocol: Two-stage fine-tuning—initial on DA scores, followed by joint training on MQM and DA, with score rescaling and synthetic data augmentation.
• �� Error span detection: GemSpanEval trained to generate JSON outputs of error spans, categories, and severities, with context extension for non-unique spans.
• �� Evaluation: Conducted on WMT24 validation set, measuring segment correlation and system accuracy, with ablation studies on input formats and training strategies.

Experiments involved training on WMT15-23 data, validating on WMT24 datasets. Metrics included segment-level correlation and system-level accuracy. Ablation studies assessed the impact of input format, training stages, and data mixing. Error span detection was evaluated using character F1 scores across language pairs, analyzing the effect of span context extension. Hyperparameters such as batch size (64), learning rate (0.0001), and steps (20K) were optimized for model stability. Baselines included previous MetricX versions and XCOMET-XXL. Results demonstrated that multi-task training and input enhancements significantly improved performance, especially in Chinese and Japanese, with detailed error span outputs providing interpretability.

MetricX-25 achieved 12% higher segment correlation and 8% better system accuracy over MetricX-24, with notable gains in Chinese and Japanese pairs. GemSpanEval scored character F1 of 20.27%, surpassing baseline models, and effectively output error contexts. Multi-task training enabled robust performance across different score types and languages. Error span detection benefited from context extension, reducing ambiguity and improving localization precision. These results validate the effectiveness of the proposed architecture and training strategies, setting new benchmarks for automatic translation evaluation.

The systems can be integrated into translation workflows for real-time quality assessment, error localization, and system tuning. They support low-resource languages and complex domains, providing detailed feedback to improve translation models. Additionally, they can facilitate automated post-editing, quality monitoring, and benchmarking in industrial settings, reducing reliance on costly human evaluations. The interpretability of error outputs enhances transparency and trust in automated metrics, fostering broader adoption.

Despite improvements, the models still face difficulties with highly complex or ambiguous errors, especially in very long segments or under low-resource conditions. The reliance on annotated data introduces biases, and the computational cost of large models limits real-time deployment. Error span generation may produce ambiguous outputs in short or repetitive spans, requiring further refinement. Future work should focus on reducing inference costs, enhancing robustness, and expanding training data diversity to address these challenges.