Analyzing Context Contributions in LLM-based Machine Translation

The evolution of neural machine translation (NMT) has been driven by the advent of transformer architectures (Vaswani et al., 2017), which replaced traditional RNN-based models with attention mechanisms. Early models like Transformer-based NMT achieved significant improvements in translation quality, especially on large datasets such as WMT. Subsequent developments introduced pretraining strategies (e.g., BERT, GPT) and large-scale multilingual models (e.g., mBERT, mT5), enabling zero-shot and few-shot learning capabilities. Despite these advances, interpretability remains limited; most analyses focus on attention weights or output metrics, which do not fully explain internal decision processes. Recent efforts have explored attribution methods, but their application to multi-language LLMs in MT is still nascent. This context underscores the need for detailed, layer-wise interpretability tools to understand how models utilize context, especially in complex scenarios involving multiple examples and languages.

While LLMs demonstrate impressive translation performance, their internal mechanisms—particularly how they leverage different parts of the input context—are poorly understood. Key issues include the dominance of source tokens, positional biases favoring early examples, and the emergence of hallucinations. These problems hinder model transparency and safety, especially when models generate inaccurate or biased outputs. Existing interpretability methods lack the granularity to dissect token-level contributions across layers, limiting insights into how information flows and biases form. Addressing these gaps is crucial for developing trustworthy, controllable translation systems that can reliably handle diverse languages and contexts.

This work introduces ALTI, a layer-wise attribution technique that quantifies token contributions across transformer layers, providing a fine-grained view of information flow. It extends beyond attention weights, capturing causal influence and enabling part-level aggregation of contributions from source text, few-shot examples, and prefixes. The approach reveals persistent positional biases, with early examples exerting disproportionate influence, and identifies anomalous contribution patterns associated with hallucinations. By integrating bias detection with contribution analysis, the method offers a novel diagnostic tool for model interpretability and safety. This comprehensive framework surpasses prior attention-based explanations, offering actionable insights for bias mitigation and model debugging in multilingual MT.

• �� Implement ALTI to compute token-to-token contribution matrices at each transformer layer, capturing the influence of input tokens on outputs. • Aggregate token contributions into part-level scores, separating source text, few-shot examples (source and target sides), and prefixes. • Analyze contribution patterns across different example positions, comparing models before and after fine-tuning, to identify positional biases. • Conduct experiments on WMT22 datasets across multiple language pairs, with 5-shot prompts, shuffling examples to eliminate relevance bias. • Detect anomalies by identifying samples with unusually low source contributions, correlating these with hallucination cases. • Track contribution evolution during translation, dividing generated sequences into bins, to observe how reliance on context shifts over time.

The experiments utilize WMT22 test sets for en-de, de-en, en-ru, and ru-en, employing LLaMA-2, TOWER, and TOWERINSTRUCT models. Five-shot prompts are used, with examples shuffled randomly. ALTI computes contribution matrices at each layer, which are then aggregated to part-level scores. The analysis compares contribution ratios across example positions, evaluates the impact of task descriptions, and tests the effect of copying test samples at the end of context. Anomalous samples with low source contributions are examined to identify hallucinations. The study also investigates how contributions evolve during the translation process by dividing generated tokens into ten segments, revealing consistent source dominance early on and decreasing influence later.

The results confirm that source tokens in few-shot examples dominate contributions, especially at early positions, with source contribution ratios exceeding 60%. Positional bias is robust across models and languages, but fine-tuning reduces its strength. Introducing task descriptions or copying test samples at the end diminishes positional bias, yet does not eliminate it. Anomalous cases with low source contributions often correspond to hallucinations, where the model copies previous translations. During generation, source influence decreases while target prefix influence increases, aligning with classical MT dynamics. These findings demonstrate the effectiveness of attribution-based diagnostics in understanding and mitigating biases.

The attribution framework can be integrated into model debugging pipelines, enabling real-time bias detection and correction. It supports development of safer translation systems by identifying and addressing hallucinations and biases. The methodology can guide training strategies to reduce positional biases and improve interpretability, fostering trust in multilingual NLP applications. Additionally, the insights can inform model architecture design, promoting more balanced and content-aware information flow, ultimately enhancing translation quality and safety in industry deployments.

The computational cost of ALTI limits scalability, restricting analysis to smaller datasets and models. Results are primarily validated on WMT22 and specific models, raising questions about generalization. The attribution method captures correlation rather than causation, and may not fully disentangle complex biases. The detection of hallucinations relies on contribution anomalies, which may not cover all failure modes. Further research is needed to develop bias mitigation strategies and extend the approach to larger models and diverse tasks.