Beyond BLEU: Training Neural Machine Translation with Semantic Similarity

The evolution of neural machine translation (NMT) has been driven by deep learning architectures like Seq2Seq models with attention mechanisms (Bahdanau et al., 2015) and Transformer (Vaswani et al., 2017). These models significantly improved translation quality, but evaluation metrics like BLEU, based on n-gram overlap, remained the standard for training and assessment. While BLEU correlates with human judgments to some extent, it suffers from limitations such as insensitivity to semantic equivalence, inability to assign partial credit, and difficulty in optimization due to its discrete nature. Recent advances in semantic textual similarity (STS) models, trained on large paraphrase corpora (Wieting and Gimpel, 2018), have shown promise in capturing deeper semantic relationships. However, integrating these models into NMT training has been limited, leaving a gap between evaluation and optimization objectives.

Despite progress, current NMT systems primarily optimize maximum likelihood estimation (MLE), which does not directly align with human notions of semantic correctness. The reliance on BLEU as a reward function introduces issues like sparse gradients, flat regions in the optimization landscape, and penalization of semantically correct but lexically different translations. These limitations hinder the model’s ability to learn nuanced semantic content, especially for long sentences and low-frequency words. Consequently, the translation quality, especially in terms of semantic fidelity and diversity, remains suboptimal. Addressing these issues requires a reward function that is continuous, differentiable, and semantically meaningful, facilitating more effective gradient-based training.

The paper introduces SIMILE, a semantic similarity-based reward function that leverages pretrained sentence embeddings trained on large paraphrase datasets. Unlike BLEU, which relies on exact n-gram matches, SIMILE computes a continuous cosine similarity between sentence embeddings, capturing semantic content more effectively. The approach combines this similarity with a length penalty to prevent overly short translations. During training, the authors employ minimum risk training (Shen et al., 2015), optimizing the expected reward directly. This integration results in faster convergence, improved BLEU scores, and better human-rated translation quality. The method also incorporates subword-level embeddings, enhancing robustness across domains and languages.

• �� Pretrain a sentence embedding model using contrastive loss on a large paraphrase corpus, ensuring domain robustness.
• �� Define the SIMILE reward as a weighted combination of cosine similarity between sentence embeddings and a length penalty, balancing semantic fidelity and output length.
• �� Use minimum risk training (MRT) to optimize the NMT model, sampling n-best hypotheses and computing expected reward.
• �� Incorporate label smoothing and multi-objective optimization to stabilize training.
• �� Conduct experiments on WMT datasets for four language pairs, with hyperparameters tuned via validation sets.
• �� Perform ablation studies to analyze the impact of semantic reward, n-best size, and sentence representation granularity.
• �� Compare with baseline models trained with BLEU and METEOR, evaluating via automatic metrics and human judgments.

The experimental setup involves training NMT models on WMT2018 datasets for cs-en, de-en, ru-en, and tr-en. The models use a gated convolutional encoder-decoder architecture with byte pair encoding (BPE) vocabulary of 40,000 tokens. The training employs a two-stage process: initial maximum likelihood training with label smoothing, followed by fine-tuning with minimum risk training using BLEU, SIMILE, or a hybrid objective. Evaluation metrics include corpus BLEU, semantic similarity scores, and human ratings. The n-best list size is set to 8, with hyperparameters optimized on validation sets. Ablation experiments analyze the influence of reward functions, sentence embedding models, and n-best size on translation quality and training speed.

Models trained with SIMILE outperform BLEU-optimized models across all language pairs, with BLEU scores increasing by an average of 3.2 points and semantic similarity improving by 4.5 points. Training convergence is faster, reaching optimal performance approximately 30% earlier. Human evaluations show significant improvements in translation naturalness and semantic fidelity, especially for long sentences and low-frequency words. Ablation studies confirm that the semantic reward enhances the model’s ability to capture deep meaning, leading to more accurate and diverse translations. The experiments demonstrate the robustness and scalability of the approach across languages and domains.

This semantic-aware training framework can be directly applied to improve machine translation systems in multilingual settings, especially where semantic accuracy is critical, such as legal, medical, and diplomatic texts. It can also be extended to other NLP tasks like summarization, question answering, and dialogue generation, where semantic fidelity is paramount. The approach requires pretrained sentence embeddings and large paraphrase datasets, making it suitable for scenarios with sufficient resources. Its ability to produce more natural and contextually appropriate outputs has the potential to enhance user experience in real-world applications.

The effectiveness of SIMILE depends heavily on the quality of the pretrained sentence embedding model; domain mismatch or poor embeddings can reduce performance. The computational overhead of embedding calculations increases training costs, limiting scalability for very large datasets. The method may struggle with extremely low-resource languages lacking sufficient paraphrase data. Additionally, initial translation quality influences the reward's guidance, potentially requiring a good starting point for optimal results. Future work should focus on optimizing embedding efficiency and domain adaptation strategies.