InfoXLM: An Information-Theoretic Framework for Cross-Lingual Language Model Pre-Training

The evolution of multilingual NLP has seen models like mBERT, XLM, and XLM-R achieve cross-lingual transfer through masked language modeling and translation-based tasks. These models have demonstrated surprising effectiveness but face limitations in low-resource languages and semantic alignment. Recent theoretical insights from information theory suggest that mutual information maximization can serve as a unifying principle to improve semantic consistency across languages. However, prior work lacks a systematic framework that explicitly maximizes mutual information at multiple granularities, especially at the sequence level. This gap motivates the development of a comprehensive, theoretically grounded approach to cross-lingual pretraining, integrating multiple tasks under a common information-theoretic objective.

The core challenge is to learn universal semantic representations that transfer effectively across languages, particularly when parallel data is scarce. Existing models often rely on monolingual or bilingual data, leading to suboptimal alignment and transferability. Moreover, the lack of a unified theoretical framework makes it difficult to understand and improve the underlying mechanisms. The problem is compounded by the need to balance multiple objectives—masked token prediction, translation modeling, and semantic alignment—within a single training paradigm. Addressing these issues requires a principled approach that explicitly maximizes the shared information between different language views, ensuring robust cross-lingual transfer.

This work introduces a unified information-theoretic framework that explicitly maximizes mutual information across multiple granularities and views. Key innovations include: 1) formalizing cross-lingual pretraining as mutual information maximization, 2) proposing XLCO, a sequence-level contrastive task leveraging momentum encoders and mixup augmentation, 3) integrating MLM and TLM with XLCO in a multi-task setting, and 4) providing theoretical bounds that guarantee tighter mutual information estimates. These innovations enable the model to learn language-agnostic semantic representations more effectively, especially in low-resource scenarios, and provide a clear conceptual foundation for future cross-lingual models.

• �� Define a mutual information maximization objective across multilingual texts at different granularities.
• �� Use MLM to maximize I(c1; x1), where c1 is context and x1 is masked token.
• �� Employ TLM to maximize I(c1, c2; x1) across translation pairs, implicitly aligning cross-lingual contexts.
• �� Introduce XLCO, which maximizes sequence-level mutual information I(c1; c2) using contrastive learning with InfoNCE loss.
• �� Implement momentum contrast encoders (gθQ, gθK) to efficiently sample negative examples, updating the key encoder via momentum.
• �� Use mixup augmentation by concatenating sentence pairs from different corpora, encouraging the model to learn sentence boundaries and order.
• �� Conduct contrastive learning on the most universal layer ([CLS] token at layer 8 or 12), promoting language-invariant representations.
• �� Train jointly on monolingual and parallel corpora, balancing data via multilingual sampling.
• �� Optimize the combined loss (MLM + TLM + XLCO) with equal weights, ensuring multi-task synergy.

The pretraining uses CC-100, MultiUN, WikiMatrix datasets, covering 94 languages with 42GB of parallel data. Models are Transformer-based, with 12 and 24 layers, trained for 150K and 200K steps respectively. Hyperparameters include batch size 2048, learning rate 0.0002/0.0001, warmup steps 10K, and momentum coefficients 0.9999/0.999. Evaluation on XNLI, MLQA, and Tatoeba benchmarks shows consistent improvements over baselines like XLM-R and mBERT. Ablation studies confirm the contributions of XLCO and contrastive layers, demonstrating their importance for cross-lingual transfer. The models exhibit strong performance in low-resource and distant language pairs, validating the theoretical framework.

INFOXLM achieves an average accuracy of 76.5% on XNLI, outperforming XLM-R by 1.5%. On MLQA, F1 scores improve by 2-3 points. In sentence retrieval, accuracy on 14 parallel pairs exceeds 77%, with an 8% gain over XLM-R. The sequence-level contrastive task (XLCO) notably enhances translation pair alignment, especially in zero-shot transfer. Larger models further boost performance, confirming the scalability and robustness of the approach. Ablation results show that removing XLCO reduces transferability, highlighting its critical role.

The model is suitable for multilingual question answering, content classification, and cross-lingual retrieval, especially in low-resource languages. It can be integrated into multilingual search engines, virtual assistants, and content moderation tools, providing accurate semantic understanding across languages. The framework also supports building multilingual knowledge bases and improving machine translation quality, especially for unseen language pairs. Future applications include low-resource language support and multi-modal integration.

Despite strong results, the approach depends heavily on large-scale parallel data, limiting applicability to low-resource languages. Computational costs are high due to contrastive mechanisms and large model sizes. The framework's effectiveness diminishes when language pairs are extremely distant or data is extremely sparse. Future work should focus on reducing data dependency, improving efficiency, and extending to unsupervised or semi-supervised settings to broaden real-world impact.