Pre-training via Paraphrasing
MARGE introduces a retrieval-based pretraining method for multilingual multi-document paraphrasing, achieving strong zero-shot performance on translation and summarization.
Key Findings
Methodology
MARGE employs a retrieval-augmented sequence-to-sequence framework, where the model retrieves related documents in multiple languages, computes relevance scores, and biases cross-attention during decoding. The relevance scores are learned via a shared encoder, enabling joint optimization of retrieval and reconstruction. The model encodes each retrieved document separately, then combines information through a relevance-weighted attention mechanism, facilitating paraphrasing, translation, and multi-document summarization without task-specific fine-tuning. The training maximizes the likelihood of target texts conditioned on retrieved evidence, forming an end-to-end self-supervised learning paradigm that captures multiple linguistic tasks.
Key Results
- On zero-shot translation, BLEU scores reached up to 35.8 for document translation without any task-specific fine-tuning, outperforming traditional masked language models. Cross-lingual retrieval on BUCC and Tatoeba datasets surpassed other unsupervised models by nearly 10 points, demonstrating effective cross-lingual semantic matching. In summarization, ROUGE-L scores exceeded baselines across multiple languages, and in paraphrase detection, state-of-the-art accuracy was achieved on PAWS-X. The model also performed competitively on multilingual question answering tasks, confirming its broad applicability.
- In unsupervised document translation, BLEU scores for German-English reached 35.8, and for Arabic-English 19.9, significantly better than previous methods. Fine-tuning on supervised datasets further improved performance, matching or exceeding baselines like mBART. The model's ability to retrieve relevant evidence across languages contributed to its success in cross-lingual tasks, validating the retrieval-augmented pretraining approach. Ablation studies confirmed that relevance bias and joint training were critical for performance gains.
- Overall, the results demonstrate that integrating retrieval into pretraining enhances the model's capacity for multilingual understanding, zero-shot transfer, and multi-task learning, reducing reliance on large task-specific datasets and enabling versatile deployment across NLP applications.
Significance
This work advances the paradigm of pretraining by integrating retrieval mechanisms, enabling models to better understand and generate multilingual content across diverse tasks. It addresses the limitations of traditional masked language models in zero-shot generalization, especially for complex tasks like multi-document summarization and cross-lingual translation. The end-to-end training of retrieval and generation components paves the way for more adaptable, scalable, and resource-efficient models, bridging gaps between supervised and unsupervised learning in multilingual NLP. The approach holds promise for real-world applications such as multilingual information retrieval, cross-lingual question answering, and automated content generation, with potential to significantly reduce the need for task-specific labeled data.
Technical Contribution
The key technical innovation lies in jointly learning relevance scoring and sequence generation within a unified transformer-based framework, where relevance scores bias cross-attention during decoding. The relevance encoder, shared with the reconstruction model, enables efficient similarity computation, while the biasing mechanism allows the model to focus on pertinent evidence dynamically. This end-to-end approach contrasts with prior two-stage retrieval-then-generation systems, offering improved efficiency and coherence. The model scales to nearly 10^9 parameters, demonstrating the feasibility of large-scale, multi-task multilingual pretraining that captures semantic, syntactic, and contextual nuances across languages and tasks.
Novelty
This research is the first to integrate a retrieval mechanism directly into a multilingual sequence-to-sequence pretraining objective, enabling the model to learn paraphrasing, translation, and summarization jointly. Unlike traditional MLMs, which rely solely on masking, MARGE leverages related documents retrieved in multiple languages, emphasizing semantic consistency over lexical overlap. Its end-to-end training paradigm and relevance-biased attention mechanism represent a significant departure from existing models, providing a unified framework for multi-task multilingual NLP without task-specific fine-tuning.
Limitations
- The model's performance on low-resource languages remains limited due to data scarcity and retrieval biases. Its reliance on offline indexing and thresholding introduces latency and scalability challenges, especially in real-time applications. The large parameter count (~960M) demands substantial computational resources, hindering deployment in resource-constrained environments. Additionally, the model's ability to handle non-Latin scripts and dialectal variations needs further improvement. Future work should focus on model compression, dynamic retrieval strategies, and expanding training data for underrepresented languages.
Future Work
Future directions include integrating multimodal data (images, audio) to enrich representations, developing dynamic retrieval mechanisms for real-time applications, and exploring low-resource language adaptation. Further research could also investigate more efficient model architectures to reduce computational costs, as well as extending the framework to tasks like dialogue generation and cross-modal reasoning. Enhancing the retrieval component with learned indexing and adaptive thresholds will improve scalability and robustness, broadening the model's applicability in industrial NLP systems.
AI Executive Summary
The field of multilingual NLP has long grappled with the challenge of creating models that generalize across tasks, languages, and data scarcity. Traditional pretraining methods like masked language modeling (MLM) have achieved remarkable success but remain limited in zero-shot transfer, especially for complex tasks such as multi-document summarization, translation, and cross-lingual retrieval. Addressing these limitations, this study introduces MARGE, a novel retrieval-augmented sequence-to-sequence framework that jointly learns relevance scoring and text reconstruction across multiple languages.
MARGE's core innovation lies in retrieving related documents in various languages, computing relevance scores, and biasing cross-attention during decoding to enhance the reconstruction of target texts. This approach effectively captures semantic equivalence, paraphrasing, and translation relationships without task-specific fine-tuning. The model is trained end-to-end, optimizing relevance and generation jointly, which leads to robust zero-shot performance across diverse NLP tasks.
Experimental results demonstrate that MARGE achieves BLEU scores up to 35.8 in document translation, surpassing previous unsupervised models. It also excels in cross-lingual sentence retrieval, question answering, and summarization, often outperforming supervised baselines. These findings validate the hypothesis that integrating retrieval mechanisms into pretraining significantly enhances multilingual understanding and transferability.
The broader impact of this work is profound: it paves the way for more versatile, resource-efficient models capable of tackling multiple tasks with minimal supervision. By reducing dependence on task-specific data, MARGE offers a scalable solution for real-world multilingual applications, including automated translation, cross-lingual information retrieval, and content summarization.
Despite its strengths, the model faces challenges such as high computational costs, limited low-resource language support, and reliance on offline indexing. Future research will focus on model compression, dynamic retrieval, and multimodal extensions, aiming to make such powerful models more accessible and adaptable for industry deployment and further scientific exploration.
Deep Dive
Abstract
We introduce MARGE, a pre-trained sequence-to-sequence model learned with an unsupervised multi-lingual multi-document paraphrasing objective. MARGE provides an alternative to the dominant masked language modeling paradigm, where we self-supervise the reconstruction of target text by retrieving a set of related texts (in many languages) and conditioning on them to maximize the likelihood of generating the original. We show it is possible to jointly learn to do retrieval and reconstruction, given only a random initialization. The objective noisily captures aspects of paraphrase, translation, multi-document summarization, and information retrieval, allowing for strong zero-shot performance on several tasks. For example, with no additional task-specific training we achieve BLEU scores of up to 35.8 for document translation. We further show that fine-tuning gives strong performance on a range of discriminative and generative tasks in many languages, making MARGE the most generally applicable pre-training method to date.