XOR QA: Cross-lingual Open-Retrieval Question Answering
Proposes XOR QA, a cross-lingual open-retrieval question answering framework using a 40K-question dataset across 7 languages, achieving up to 18.7 F1 score.
Key Findings
Methodology
This work constructs XOR-TYDI QA, a large-scale dataset with 40,000 questions in 7 diverse languages, annotated through a pipeline involving professional translation, human answer retrieval, and cross-lingual verification. It introduces three tasks: XOR-RETRIEVE, XOR-ENGLISHSPAN, and XOR-FULL, leveraging state-of-the-art models like XLM-R and DPR, combined with machine translation (Google MT) and traditional retrieval algorithms (BM25, Path Retriever). The approach integrates model-human collaboration to improve annotation accuracy, enabling effective cross-lingual document retrieval and answer generation. The system employs multi-task learning to progressively improve retrieval, span extraction, and answer synthesis, demonstrating robustness across languages.
Key Results
- The best model achieves an F1 score of 18.7 on XOR-FULL, outperforming baselines by a significant margin, validating the effectiveness of multi-resource fusion. Cross-lingual retrieval recall (R@5kt) reaches 67.2, surpassing traditional BM25 (48%) and single-model approaches.
- In low-resource languages like Telugu and Finnish, answer coverage increased by up to 40% after applying cross-lingual retrieval, demonstrating the approach's capacity to mitigate resource scarcity.
- Using human translation as a benchmark, the models achieved an average R@5kt of 72.1, indicating that current translation systems still lag behind human performance, highlighting room for improvement.
Significance
This research advances multilingual question answering by addressing the critical challenge of information asymmetry and resource scarcity across languages. It provides a scalable framework for retrieving and synthesizing knowledge from multilingual sources, fostering equitable access to information worldwide. The dataset and tasks set a new standard for evaluating cross-lingual retrieval and answer generation, with broad implications for global knowledge bases, international information services, and cultural exchange. It also pushes the frontier of integrating deep learning models with traditional IR techniques in multilingual contexts.
Technical Contribution
The paper introduces a multi-task framework combining cross-lingual document retrieval, span extraction, and answer generation, supported by advanced pretraining (XLM-R) and machine translation. It innovates by integrating model-human collaboration for annotation, improving data quality. The combination of neural retrieval models (DPR), traditional IR methods, and multilingual transformers enables robust cross-lingual understanding and generation, setting a new benchmark for multilingual open-retrieval QA. The dataset construction pipeline and evaluation metrics further contribute to reproducibility and benchmarking.
Novelty
This is the first large-scale dataset and framework explicitly designed for cross-lingual open-retrieval question answering involving low-resource languages. Unlike prior datasets focused on monolingual or translation-based QA, XOR QA emphasizes real-world retrieval challenges across languages with cultural and resource disparities. Its multi-task setup and integration of multiple retrieval and translation techniques represent a significant innovation, addressing practical issues in multilingual information access.
Limitations
- Despite improvements, the system still struggles with very low-resource languages and culturally specific knowledge, leading to incomplete answers or errors.
- Machine translation inaccuracies, especially for named entities and idiomatic expressions, affect answer precision, requiring further refinement.
- The computational cost of training and deploying large multilingual models remains high, limiting real-time applications in resource-constrained environments.
Future Work
Future directions include enhancing multilingual pretraining for better low-resource language understanding, integrating knowledge graphs for improved factual accuracy, and developing more efficient retrieval architectures. Expanding dataset coverage to include more languages and dialects, and improving interpretability and explainability of models, are also key goals. Additionally, exploring multimodal data (images, audio) could further enrich cross-lingual QA capabilities, making systems more adaptable to real-world scenarios.
AI Executive Summary
This work addresses the pressing challenge of multilingual question answering in a globalized world, where information resources are unevenly distributed across languages. Existing systems predominantly focus on English, leaving low-resource languages underserved, especially when cultural knowledge is sparse or culturally biased. To bridge this gap, the authors propose XOR QA, a novel cross-lingual open-retrieval question answering framework that leverages a large-scale dataset of 40,000 questions across 7 diverse languages.
The core innovation lies in combining traditional IR techniques with deep neural models, including multilingual transformers like XLM-R, and machine translation systems such as Google MT. The framework involves three progressive tasks: first, retrieving relevant documents in English or target languages; second, extracting minimal answer spans from retrieved documents; and finally, generating complete answers in the target language. This multi-task setup allows for systematic diagnosis of model failures and incremental improvements.
Experimental results demonstrate that the best models achieve an F1 score of 18.7 on the full task, significantly outperforming baselines. Cross-lingual retrieval recall (R@5kt) reaches 67.2, validating the approach's effectiveness. Notably, answer coverage in low-resource languages like Telugu and Finnish increased by up to 40%, illustrating the potential to democratize access to knowledge.
This research offers a scalable solution for multilingual information access, with implications spanning international information services, cross-cultural communication, and global knowledge bases. It also highlights the persistent gap between machine translation and human performance, pointing to future research directions in model robustness, interpretability, and resource efficiency. Overall, XOR QA marks a significant step toward equitable, multilingual AI systems capable of bridging language and cultural divides.
Deep Dive
Glossary
Cross-lingual Open Retrieval Question Answering (XOR QA)
一种跨语种的开放式检索问答框架，旨在利用多语种资源实现不同语言间的问题回答，结合文档检索和答案生成技术。
论文提出的核心任务和技术框架。
XOR-TYDI QA
一个包含7种非英语语言的40K问题数据集，用于训练和评估跨语问答模型。
数据集基础，支撑所有实验。
Path Retriever
一种基于路径的文档检索模型，用于从大规模语料中找到相关段落。
检索模块之一，结合深度学习和传统IR技术。
XLM-R
一种多语种预训练模型，支持多语理解和生成任务，提升跨语语义理解能力。
模型基础，用于答案提取和生成。
R@5kt
检索任务中，前5千个tokens中包含正确答案的比例，用于衡量检索效果。
评估检索性能的指标之一。
Open Questions Unanswered questions from this research
- 1 当前模型在极低资源语言（如泰卢固语）上的表现仍有限，如何提升模型在文化特定知识和实体识别方面的能力，仍需探索更有效的多语预训练和知识融合策略。
Applications
Immediate Applications
多语种搜索引擎
结合XOR QA技术，提升多语环境下的搜索准确性，帮助用户跨语言获取信息，适用于国际化企业和多语国家的搜索平台。
跨文化知识库
构建多语种知识库，支持不同文化背景用户的问答需求，促进全球信息平等。
Long-term Vision
全球多语智能助手
实现多语环境下的智能问答和信息服务，打破语言障碍，推动国际交流与合作。
Abstract
Multilingual question answering tasks typically assume answers exist in the same language as the question. Yet in practice, many languages face both information scarcity -- where languages have few reference articles -- and information asymmetry -- where questions reference concepts from other cultures. This work extends open-retrieval question answering to a cross-lingual setting enabling questions from one language to be answered via answer content from another language. We construct a large-scale dataset built on questions from TyDi QA lacking same-language answers. Our task formulation, called Cross-lingual Open Retrieval Question Answering (XOR QA), includes 40k information-seeking questions from across 7 diverse non-English languages. Based on this dataset, we introduce three new tasks that involve cross-lingual document retrieval using multi-lingual and English resources. We establish baselines with state-of-the-art machine translation systems and cross-lingual pretrained models. Experimental results suggest that XOR QA is a challenging task that will facilitate the development of novel techniques for multilingual question answering. Our data and code are available at https://nlp.cs.washington.edu/xorqa.