Improving Translation Faithfulness of Large Language Models via Augmenting Instructions

The evolution of pre-trained language models (PLMs) like BERT, GPT, and T5 has transformed NLP, enabling multi-task learning and zero-shot capabilities. Despite these advances, models still face issues like hallucinations and fidelity loss in tasks such as machine translation. Instruction tuning emerged as a promising approach, with datasets like FLAN and Super-NaturalInstructions expanding models' task understanding. However, these methods mainly focus on short instructions and local attention, leaving the challenge of maintaining instruction focus in long texts unaddressed. Recent studies highlight the importance of global attention mechanisms and adversarial training to improve trustworthiness. Yet, the integration of these strategies for open-source models remains limited, especially in translation fidelity and instruction retention.

The core challenge is instruction forgetting during decoding, especially in causal language models with local attention biases. This leads to hallucinations, unfaithful translations, and reduced trustworthiness. In long or complex texts, models tend to focus on nearby tokens, neglecting the instruction context, which causes deviations from expected outputs. Addressing this requires enhancing the model’s ability to maintain global instruction awareness and resist typical translation errors, a task complicated by the inherent limitations of attention mechanisms and training data constraints.

This work introduces SWIE, which employs segment-level weighted embeddings via parameterized adapters, enabling the model to maintain a global view of instructions across multiple layers. It also develops OVERMISS, a dataset of contrastive negative samples generated automatically using GPT-3.5, simulating typical translation errors like over-translation and miss-translation. Combining these innovations, the approach systematically improves instruction focus and translation fidelity. Unlike prior methods, which mainly rely on data scaling or local attention, this framework explicitly targets the global instruction attention problem and models common errors adversarially, leading to more faithful translations.

• �� Segment tokens into instruction, input, and response segments, assigning segment IDs.
• �� Use parameterized adapters at middle layers of the model to fuse instruction representations with global hidden states, applying segment-level weights.
• �� Construct OVERMISS dataset by prompting GPT-3.5 to generate negative samples mimicking over-translation and miss-translation errors, evaluated with awesome-align for quality.
• �� Employ contrastive learning, combining positive (correct) and negative (erroneous) samples, to fine-tune the model’s attention and translation fidelity.
• �� Fine-tune models on multi-task datasets, gradually integrating SWIE and OVERMISS, with multi-stage curriculum learning.
• �� Evaluate on WMT22 and Flores-200, measuring BLEU, COMET, and faithfulness metrics, including ablation studies to analyze attention shifts and error reduction.

Experiments involve fine-tuning BLOOMZ-3b and LLaMA-7b on datasets like WMT17-20, FLORES-200, and the newly constructed OVERMISS. Evaluation metrics include BLEU, COMET, and word alignment-based faithfulness scores. The training employs multi-stage curriculum learning, with hyperparameters aligned with prior work (e.g., 8 V100 GPUs for 3B models). Ablation studies compare models with and without SWIE and OVERMISS, analyzing attention distributions and translation quality across directions such as De→En, En→De, En→Zh. Long-text translation is tested via concatenation of multiple sentences, demonstrating robustness. Zero-shot performance is assessed on directions like Uk→En, Fr→De, showing consistent improvements.

Results show SWIE boosts BLEU from 0.19 to 0.51, especially in long-text and zero-shot scenarios. OVERMISS improves BLEU from 0.69 to 3.12 on WMT22, with an average COMET increase of 0.48%. Combining both methods yields BLEU scores up to 0.56 in English-German translation, with notable improvements in word alignment faithfulness. Ablation confirms SWIE enhances instruction attention ratios, especially in middle layers, reducing instruction neglect. The combined approach outperforms baselines across multiple datasets and directions, validating its effectiveness in improving translation fidelity and trustworthiness.

This framework can be directly applied to enhance machine translation systems, especially for low-resource languages and long-form content requiring high fidelity. It is suitable for integration into commercial translation platforms, AI assistants, and content moderation tools. Long-term, the techniques could extend to multi-modal AI systems, improving instruction understanding in visual and speech tasks, fostering more trustworthy AI interactions across industries.

Current methods are less effective in extremely long or multi-modal inputs, where attention distribution remains challenging. Negative sample quality depends on automatic GPT-3.5 prompts, which may introduce bias. Computational costs increase with multi-layer fusion and adversarial training, limiting scalability. Further research is needed to adapt these techniques for diverse modalities and real-world applications, addressing issues like data quality, efficiency, and robustness.