Larger-Scale Transformers for Multilingual Masked Language Modeling
Scaling XLM-R to 3.5B and 10.7B parameters significantly improves cross-lingual understanding, surpassing previous models on benchmarks.
Key Findings
Methodology
The study employs Transformer-based multilingual masked language models (MLM) trained on the CC100 dataset (167B tokens, 100 languages). Two models, XLM-RXL (3.5B parameters, 36 layers, H=2560) and XLM-RXXL (10.7B, 48 layers, H=4096), utilize SentencePiece tokenization, full softmax, and model parallelism (tensor parallelism). Training involves 500,000 updates with batch size 2048, sequence length 512, optimized for MLM objectives. Evaluation includes cross-lingual transfer on XNLI, question answering on MLQA and XQuad, and monolingual English tasks on GLUE, using zero-shot and fine-tuning strategies.
Key Results
- XLM-RXXL achieves 83.1% average accuracy on XNLI, outperforming the previous XLM-R large model by 2.2%, with notable gains in low-resource languages like Vietnamese (+8%) and Hindi (+7%).
- On English GLUE tasks, XLM-RXXL surpasses RoBERTa-Large by 0.3%, demonstrating competitive monolingual performance while supporting 99 additional languages.
- In cross-lingual question answering (MLQA, XQuad), F1 and EM scores improve by 4.4 and 5.5 points respectively, setting new state-of-the-art results, especially in Arabic, Russian, and Greek.
Significance
This work confirms that increasing model capacity enhances multilingual transfer, effectively serving both high-resource and low-resource languages. It addresses the bottleneck of limited model size, paving the way for more inclusive NLP systems that can handle diverse languages with high accuracy, thus broadening the scope of multilingual AI applications.
Technical Contribution
The paper introduces a large-scale Transformer architecture with advanced model parallelism, optimized training procedures, and balanced data sampling. It demonstrates that scaling parameters to over 10 billion yields substantial performance gains across multiple benchmarks, establishing a new standard for multilingual models. The approach balances computational efficiency with model capacity, enabling practical deployment of massive models.
Novelty
This is the first systematic scaling of XLM-R to 10.7 billion parameters, showing that larger models can simultaneously excel in high-resource language tasks and low-resource transfer scenarios. It differs from prior work by emphasizing the synergy between model size and training data, achieving state-of-the-art results without sacrificing multilingual coverage.
Limitations
- Training such large models requires enormous computational resources, limiting accessibility for smaller research groups.
- Despite improvements, performance on extremely low-resource languages remains suboptimal, indicating the need for better data or adaptation techniques.
- Inference speed and deployment efficiency are impacted by the massive parameter count, posing challenges for real-time applications.
Future Work
Future directions include model sparsification, knowledge distillation, and efficient inference techniques to reduce computational costs. Further research will explore multilingual multi-task learning, domain adaptation, and extending capabilities to low-resource and endangered languages, aiming for more inclusive NLP systems.
AI Executive Summary
The rapid evolution of multilingual NLP has seen models like mBERT and XLM-R make significant strides in cross-lingual transfer. However, their capacity limitations restrict performance, especially for low-resource languages. To address this, recent efforts have scaled models to hundreds of billions of parameters, exemplified by GPT-3 and mT5, which demonstrate that larger models can unlock new performance heights.
Building on this trend, this study scales XLM-R to 3.5 billion (XLM-RXL) and 10.7 billion parameters (XLM-RXXL). These models leverage Transformer architecture with deep layers, large hidden sizes, and advanced model parallelism, trained on the CC100 dataset encompassing 167 billion tokens across 100 languages. The training process involves 500,000 updates, optimized for MLM objectives, with a focus on balancing multilingual data sampling and training stability.
Evaluation results are compelling. On the cross-lingual NLI benchmark XNLI, XLM-RXXL achieves 83.1% accuracy, surpassing the previous XLM-R large model by 2.2%. Notably, in low-resource languages like Vietnamese and Hindi, performance improvements exceed 8 percentage points. In monolingual English tasks, XLM-RXXL outperforms RoBERTa-Large by 0.3%, demonstrating that larger capacity does not compromise high-resource language performance.
Furthermore, in question answering benchmarks MLQA and XQuad, the models set new state-of-the-art with F1 and EM scores improving by 4.4 and 5.5 points respectively. These results highlight the potential of massive multilingual models to serve both high-resource and low-resource languages effectively, broadening NLP applications globally.
The research underscores that increasing model size is a promising pathway to universal language understanding. Nonetheless, the high computational costs and slower inference speeds pose practical challenges. Future work will focus on model compression, efficient inference, and extending capabilities to underrepresented languages, aiming to democratize access to advanced NLP technologies.
Deep Dive
Abstract
Recent work has demonstrated the effectiveness of cross-lingual language model pretraining for cross-lingual understanding. In this study, we present the results of two larger multilingual masked language models, with 3.5B and 10.7B parameters. Our two new models dubbed XLM-R XL and XLM-R XXL outperform XLM-R by 1.8% and 2.4% average accuracy on XNLI. Our model also outperforms the RoBERTa-Large model on several English tasks of the GLUE benchmark by 0.3% on average while handling 99 more languages. This suggests pretrained models with larger capacity may obtain both strong performance on high-resource languages while greatly improving low-resource languages. We make our code and models publicly available.