MLQA: Evaluating Cross-lingual Extractive Question Answering

The evolution of NLP has seen rapid progress in monolingual question answering, exemplified by datasets like SQuAD, which have driven innovations in deep learning architectures such as BiDAF and BERT. However, extending these successes to multiple languages faces significant hurdles due to data scarcity, especially in low-resource languages. Existing multilingual datasets like XNLI and XQuAD have made strides but are limited in scale and parallelism. Building high-quality, large-scale multilingual QA datasets remains challenging because manual annotation is costly and time-consuming, and existing methods often rely on machine translation, which can introduce noise. MLQA addresses these issues by leveraging Wikipedia’s multilingual content, applying advanced sentence alignment techniques, and combining crowdsourcing with professional translation to create a high-quality, scalable dataset that enables comprehensive evaluation of cross-lingual models.

The core challenge in cross-lingual QA is the lack of large, high-quality, parallel datasets that allow for effective transfer learning and model evaluation across languages. Existing datasets are either limited in scope, lack parallelism, or are constructed via translation, which can introduce inconsistencies and noise. This hampers the ability to accurately assess how well models trained in one language can perform in others, especially in realistic settings involving diverse linguistic structures and scripts. Addressing this gap is crucial for developing truly multilingual NLP systems that can operate effectively in global contexts.

MLQA introduces several key innovations: 1) a pipeline that employs LASER for automatic multi-sentence alignment, enabling extraction of N-way parallel sentences from Wikipedia at scale; 2) a hybrid annotation process combining crowdsourcing for question generation and professional translation for answer annotation, ensuring high quality and linguistic diversity; 3) the formulation of cross-lingual transfer (XLT) and generalized transfer (G-XLT) tasks for comprehensive evaluation; 4) leveraging pre-trained multilingual models like XLM to assess zero-shot transfer capabilities. These innovations collectively enable the creation of a large, high-quality, multi-language dataset that surpasses previous efforts in scale, diversity, and realism.

• �� Use LASER to automatically detect N-way parallel sentences across multiple languages from Wikipedia, focusing on high-quality sentence alignment.
• �� Extract paragraphs containing aligned sentences, then crowdsource question formulation within the English sentences, ensuring answer spans are within the aligned sentence.
• �� Employ professional translators to translate questions into target languages and annotate answer spans in each language context.
• �� Construct a dataset with over 4万 QA pairs, ensuring high parallelism and linguistic diversity across 7 languages.
• �� Design two evaluation tasks: cross-lingual transfer (XLT), where models trained in English are tested on other languages; and generalized transfer (G-XLT), involving different language pairs for questions and contexts.
• �� Implement baseline models using multilingual BERT and XLM, perform zero-shot transfer experiments, and compare with machine translation-based approaches.

The experiments utilize Wikipedia as the textual source, with models trained on English SQuAD and evaluated on MLQA’s multilingual test sets. Baselines include multilingual BERT and XLM, tested in zero-shot settings. The evaluation metrics are F1 and Exact Match (EM). Different training scenarios are tested, including monolingual, multi-lingual, and translation-augmented models. Ablation studies examine the impact of data size, alignment quality, and model architecture. Hyperparameters are tuned via grid search, and results are compared across languages and transfer tasks to analyze performance gaps and model robustness.

XLM achieves the best zero-shot transfer, with an average F1 score of around 55%, but this is substantially lower than in English (over 75%). Machine translation baselines improve F1 by approximately 10%, yet still lag behind direct multilingual training. Multilingual models trained jointly on MLQA data outperform single-language models in transfer tasks, with an F1 increase of about 10%. These results demonstrate that high-quality parallel data significantly benefits cross-lingual transfer, but current models still face substantial challenges in closing the performance gap.

MLQA can be used to develop multilingual virtual assistants, improve cross-lingual information retrieval, and enhance global knowledge bases. Its evaluation framework helps industry and academia benchmark multilingual QA systems, guiding the design of models that can operate effectively across diverse languages, especially in multilingual customer support, international search engines, and cross-cultural information access.

Despite its scale, MLQA is limited to Wikipedia, which may not reflect the complexity of informal language or specialized domains. Alignment errors from LASER can introduce noise, especially for distant or low-resource languages. Model performance in cross-lingual transfer remains below monolingual levels, indicating the need for further research in model robustness, domain adaptation, and low-resource language support. Future work should address these issues to realize truly universal multilingual QA systems.