Multilingual Denoising Pre-training for Neural Machine Translation
Proposes mBART, a multilingual Seq2Seq denoising pretraining, boosting low-resource MT BLEU by up to 12 points, with broad transfer capabilities.
Key Findings
Methodology
This work employs a BART-based denoising autoencoder trained on large-scale monolingual corpora across 25 languages (CC25). The model architecture features 12-layer Transformer encoder and decoder, trained with sentence permutation and span masking (Poisson distribution, 35%). The training maximizes the likelihood of reconstructing original texts from noisy inputs, enabling the model to learn language-agnostic representations. Post pretraining, the model is fine-tuned directly on supervised and unsupervised MT tasks, without task-specific modifications. The approach leverages cross-lingual transfer, especially benefiting low-resource pairs, and demonstrates robustness in zero-shot scenarios.
Key Results
- In low-resource MT (<10M sentence pairs), mBART initialization improves BLEU scores by up to 12 points, notably on language pairs like Vi-En, Tr-En, and noisy alignments such as En-Hi. Combining with back-translation further enhances performance, setting new state-of-the-art on WMT16 Romanian-English. For document-level MT, gains reach 5.5 BLEU points. In unsupervised MT, especially for distant language pairs like Nepali-English, the model achieves a 9.5 BLEU increase, indicating strong cross-lingual generalization. In high-resource settings (>25M pairs), benefits are marginal or slightly negative, suggesting diminishing returns with abundant data.
- In zero-shot transfer, fine-tuning on one language pair enables translation on others without additional training, demonstrating the model’s universal language understanding. The experiments validate that pretraining on multiple languages enhances low-resource and zero-resource translation, with consistent improvements across benchmarks.
Significance
This research marks a significant advancement by enabling end-to-end pretraining of a full Seq2Seq model across multiple languages, addressing the bottleneck of reliance on parallel corpora. It bridges the gap between monolingual data abundance and multilingual translation needs, especially benefiting low-resource languages. The ability to transfer knowledge across unseen language pairs opens new horizons for inclusive language technology, reducing dependence on costly bilingual datasets. The approach also simplifies the pipeline, as a single pretrained model can be adapted to various translation tasks, fostering rapid deployment in real-world applications. Overall, it pushes the frontier of multilingual NLP, making high-quality translation accessible for underrepresented languages and complex scenarios.
Technical Contribution
The core technical innovation is the development of mBART, a fully autoregressive Seq2Seq model trained with a novel multilingual denoising objective. Unlike prior works focusing on encoder-only or decoder-only pretraining, mBART jointly trains both components, capturing bidirectional and autoregressive features. The training involves sentence permutation and span masking, with a carefully designed noise function, to promote robust cross-lingual representations. The large-scale training on CC25 demonstrates that shared parameters across 25 languages facilitate transfer learning, especially in low-resource settings. The model’s architecture and training regimen are optimized for stability and efficiency, enabling effective fine-tuning on diverse MT benchmarks. The work also introduces extensive ablation studies to analyze factors influencing transferability, such as language similarity and data size.
Novelty
This work is the first to pretrain a complete Seq2Seq model in a multilingual setting using a denoising autoencoder objective. It departs from previous methods that pretrain only parts of the model or rely solely on bilingual data, instead leveraging a unified training framework across many languages. The innovation lies in the end-to-end training paradigm, which allows the model to learn language-agnostic features and transfer knowledge seamlessly. This approach significantly improves low-resource translation quality and enables zero-shot transfer, setting a new standard for multilingual pretraining.
Limitations
- The pretraining process is computationally intensive, requiring extensive GPU resources (~2.5 weeks on 256 V100 GPUs), limiting accessibility for smaller research groups.
- In high-resource scenarios, the marginal gains diminish, and sometimes performance slightly declines, indicating a saturation point.
- The model’s transferability to languages with very different scripts or minimal lexical overlap remains limited, especially for unseen languages with disjoint vocabularies.
- Further research is needed to optimize training efficiency, reduce costs, and improve zero-shot transfer for typologically distant languages.
Future Work
Future directions include developing more efficient training algorithms to reduce computational costs, expanding pretraining to include more diverse and low-resource languages, and enhancing zero-shot transfer capabilities. Integrating multimodal data and multi-task learning could further improve language understanding and generation. Exploring adaptive vocabulary strategies and dynamic model architectures may also address transfer limitations for distant languages. Ultimately, these efforts aim to democratize multilingual NLP, making high-quality translation accessible worldwide.
AI Executive Summary
Multilingual machine translation has long been constrained by the scarcity of parallel data, especially for low-resource languages. Traditional approaches rely heavily on bilingual corpora, which are expensive and limited in scope. Recent advances in self-supervised pretraining, such as BERT and GPT, have revolutionized single-language NLP tasks but have yet to be fully exploited in multilingual MT. Addressing this gap, the paper introduces mBART, a novel end-to-end Seq2Seq model trained with a multilingual denoising objective across 25 languages. The core idea is to corrupt monolingual texts by masking spans and permuting sentences, then train the model to reconstruct the original, thereby learning universal language representations. This pretraining enables the model to be fine-tuned directly on supervised and unsupervised MT tasks, with remarkable improvements, especially in low-resource scenarios. Experiments on benchmarks like WMT and FLoRes demonstrate BLEU gains up to 12 points in low-resource MT, surpassing existing models and setting new state-of-the-art results. The model also exhibits impressive zero-shot transfer capabilities, translating between language pairs unseen during training. The approach simplifies the multilingual translation pipeline, reduces reliance on parallel data, and broadens access to high-quality translation for underrepresented languages. Despite high computational costs, the benefits in transferability and performance make mBART a significant step forward. Future work will focus on efficiency, expanding language coverage, and enhancing zero-shot capabilities, aiming to democratize multilingual NLP and facilitate global communication.
Deep Dive
Abstract
This paper demonstrates that multilingual denoising pre-training produces significant performance gains across a wide variety of machine translation (MT) tasks. We present mBART -- a sequence-to-sequence denoising auto-encoder pre-trained on large-scale monolingual corpora in many languages using the BART objective. mBART is one of the first methods for pre-training a complete sequence-to-sequence model by denoising full texts in multiple languages, while previous approaches have focused only on the encoder, decoder, or reconstructing parts of the text. Pre-training a complete model allows it to be directly fine tuned for supervised (both sentence-level and document-level) and unsupervised machine translation, with no task-specific modifications. We demonstrate that adding mBART initialization produces performance gains in all but the highest-resource settings, including up to 12 BLEU points for low resource MT and over 5 BLEU points for many document-level and unsupervised models. We also show it also enables new types of transfer to language pairs with no bi-text or that were not in the pre-training corpus, and present extensive analysis of which factors contribute the most to effective pre-training.