Saliency-driven Word Alignment Interpretation for Neural Machine Translation

The evolution of NMT has seen attention mechanisms (Bahdanau et al., 2014; Vaswani et al., 2017) become central, enabling end-to-end translation. Despite high performance, the interpretability of internal alignments remains limited. External tools like GIZA++ and fast-align, developed during SMT era, provide post-hoc alignments but cannot reflect the internal decision process. Recent efforts focus on understanding attention distributions, but these are often noisy and unreliable, especially in Transformer models. Techniques like Layer-wise Relevance Propagation have been explored, yet they face mathematical and practical challenges. The need for a model-internal, stable, and interpretable alignment explanation remains urgent, motivating the development of gradient-based saliency methods.

The core challenge is that attention weights, commonly used for interpretability, are noisy and do not reliably indicate source contribution to target predictions. External aligners, while accurate, are separate from the model and cannot reflect real-time internal decisions. This disconnect hampers error analysis, model debugging, and trustworthiness. Developing an internal, stable, and model-agnostic explanation method that accurately captures source-target word relationships during translation is critical. The difficulty lies in designing a metric that correlates well with true alignments, is computationally feasible, and adaptable across architectures.

The key innovation is using the gradient of the output probability with respect to source word embeddings as a saliency measure. This derivative quantifies how much a small change in a source word affects the target prediction, directly reflecting contribution. Incorporating SmoothGrad stabilizes these gradients, reducing noise and improving interpretability. Unlike attention weights, which are computed before prediction, this method dynamically captures source influence at the prediction stage, providing a more faithful explanation. Its model-agnostic nature allows application across diverse architectures without retraining or structural changes, making it a versatile interpretability tool.

• �� Input a trained NMT model and source sentence, compute target word probability. • Calculate the partial derivative of this probability with respect to each source word embedding, yielding a saliency score. • Apply SmoothGrad by adding Gaussian noise to embeddings, averaging gradients over multiple samples for stability. • Convert saliency scores into hard alignments by selecting maximum contributors. • Use the method in both offline analysis and real-time decoding, without modifying the model. • Validate on datasets like WMT and IWSLT, comparing AER with external tools and baselines, tuning hyperparameters for optimal performance.

The experimental setup involves training attention-based RNN, convolutional, and Transformer models on multilingual datasets, including German-English, English-French, and Romanian-English. Using standard evaluation metrics like AER, the methods are tested under force decoding—where models generate translations constrained by human references—and free decoding—where models produce their own outputs. Hyperparameters for SmoothGrad (noise level, sample size) are tuned on a subset. The comparison includes attention weights, external aligners, and the proposed saliency method with and without smoothing. Results are analyzed for alignment quality, robustness, and resource efficiency, with ablation studies confirming the contribution of each component.

The saliency method with SmoothGrad achieves the lowest AER across models: Transformer from 53.4% to 36.4%, outperforming attention-based and external aligners. In low-resource settings, it effectively uncovers implicit alignments, especially in Romanian-English. The method's robustness is validated through ablation, showing that smoothing significantly improves stability. Compared to GIZA++, the approach is competitive, often surpassing fast-align, especially in online scenarios. These results demonstrate the method’s ability to extract meaningful, stable alignments directly from model gradients, confirming that neural models learn internal alignment representations.

The technique can be integrated into translation systems for real-time interpretability, error diagnosis, and model debugging. It benefits human translators by providing transparent alignment cues, improving trust and efficiency. In industry, it supports development of more explainable AI translation tools, especially in low-resource or domain-specific contexts. Long-term, the approach could be extended to multi-lingual models, multi-task learning, and integrated into training objectives to enhance alignment learning and interpretability, fostering more transparent and controllable neural translation systems.

Gradient-based saliency may be less reliable in highly non-linear or saturated models, where derivatives vanish or become unstable. The computational cost increases with multiple gradient computations, limiting scalability for very large datasets. Handling multiple instances of the same source word requires separate embedding copies, adding complexity. Effectiveness on extremely low-resource or morphologically rich languages needs further validation. Future work should explore multi-layer gradients and more efficient approximation techniques to address these issues.