Polyglot Teachers: Evaluating Language Models for Multilingual Synthetic Data Generation
Proposes Polyglot Score to evaluate 10 multilingual teacher models; finds data quality outweighs model size in effectiveness.
Key Findings
Methodology
The study evaluates 10 models of varying sizes and architectures, generating over 140,000,000 synthetic prompt-response pairs across six typologically diverse languages. It combines intrinsic data quality metrics—such as diversity, fluency, and length—with extrinsic student task performance (cultural understanding, math reasoning, chat). Using principal component analysis (PCA), key features are extracted and used in linear regression models to predict student performance, validating the importance of data quality over model scale. The evaluation involves three data generation methods—generate, translate, respond—and assesses their impact on data quality and downstream task performance.
Key Results
- Gemma 3 27B and Aya Expanse 32B models consistently rank among the top teachers with average Polyglot Scores of 0.726 and 0.706, respectively, surpassing larger models like Llama 3.1 70B (0.140), indicating size is not the sole predictor of effectiveness.
- Intrinsic data features—prompt diversity, response fluency, and length—explain over 93.3% of variance in data quality metrics and can predict student task performance (R²=0.664).
- Matching model families (e.g., Gemma-Gemma) significantly improves synthetic data quality, especially for low-resource languages, with translation and response methods outperforming pure generation in resource-scarce settings.
Significance
This work shifts the paradigm from size-centric evaluation to data-centric assessment, emphasizing data quality as the key to effective multilingual training. The Polyglot Score provides a comprehensive, quantifiable measure of teacher effectiveness, facilitating better model selection and resource allocation, especially for underrepresented languages. It addresses critical issues of data scarcity, bias, and fairness, paving the way for more inclusive multilingual AI systems with broad industry and academic impact.
Technical Contribution
The paper introduces a novel evaluation framework combining intrinsic data quality metrics and extrinsic task performance into a unified Polyglot Score. It demonstrates that data features—diversity, fluency, length—are highly predictive of downstream success, independent of model size. The methodology includes PCA-based feature extraction and linear regression, providing a scalable, cost-effective approach for model assessment. Additionally, the study validates the effectiveness of model family matching in data generation, advancing knowledge distillation techniques for multilingual settings.
Novelty
This is the first comprehensive evaluation of multilingual teacher models using a combined intrinsic-extrinsic metric system, with the Polyglot Score as a core innovation. It challenges the conventional focus on model size, emphasizing data quality and model-family alignment. The systematic comparison across diverse languages and generation methods offers new insights into optimizing synthetic data for low-resource languages, representing a significant advancement in multilingual NLP research.
Limitations
- The evaluation relies on predefined datasets and may not fully capture the nuances of extremely low-resource or underrepresented languages, limiting generalizability.
- Model family matching, while effective, depends on the availability of similar architectures and pretraining data, which may not always be feasible.
- Large-scale data generation and fine-tuning require substantial computational resources, restricting rapid deployment or real-time applications.
Future Work
Future research will explore adaptive data generation strategies incorporating active learning and human feedback to further improve low-resource language performance. Extending evaluation to more languages, especially underrepresented ones, will enhance fairness. Integrating multimodal data and exploring more efficient model architectures could reduce costs and improve scalability. Additionally, developing automated tools for real-time teacher model assessment will facilitate broader adoption.
AI Executive Summary
The rapid development of multilingual AI models has raised critical questions about how to effectively select and utilize teacher models for synthetic data generation. Traditionally, larger models have been assumed superior, but this approach overlooks the nuanced qualities of data and their impact on downstream performance. This research introduces the Polyglot Score, a comprehensive metric that combines intrinsic data quality indicators—such as diversity, fluency, and length—with extrinsic evaluations of student task performance across six diverse languages.
By evaluating ten state-of-the-art models, including Gemma 3 27B and Aya Expanse 32B, the study reveals that model size alone does not determine effectiveness. Instead, data quality features explain over 93% of the variance in intrinsic metrics and can predict student success with high accuracy. The findings emphasize the importance of data-centric approaches, advocating for model-family matching strategies and tailored data generation methods—such as translation and response—to optimize low-resource language performance.
This work has significant implications for the future of multilingual NLP. It offers a scalable, cost-effective framework for assessing teacher models, guiding practitioners toward more equitable and efficient model training. The insights gained challenge prevailing assumptions, promoting a shift from size-based to quality-based evaluation paradigms. Ultimately, the research advances the goal of inclusive, high-performance multilingual AI, addressing data scarcity and bias issues that have long hindered progress in underrepresented languages.
Deep Analysis
Background
Multilingual NLP的发展经历了从单语模型到多语迁移、知识蒸馏的演变。代表性工作如Google的mT5、Facebook的XLM-R和OpenAI的GPT系列，推动了多语模型的快速发展。然而，模型在高资源语种表现优异，但在低资源语种中仍面临数据不足、偏差和公平性挑战。合成数据和知识蒸馏成为缓解数据瓶颈的关键技术，但缺乏系统性评估教师模型多语能力的指标。近年来，模型微调和多任务学习提升了性能，但在多语任务的泛化和公平性方面仍有不足。本文基于此背景，提出多维数据质量指标和Polyglot Score，旨在优化模型选择和数据生成策略，推动多语公平与实用。
Core Problem
当前多语模型训练中，教师模型多依赖规模和单一性能指标，忽视了在非英语语种中的能力差异。这导致合成数据质量参差不齐，影响学生模型的多语能力。尤其在低资源语种，数据不足和偏差严重，限制了模型的公平性和泛化能力。如何科学评估教师模型的多语能力，优化数据生成策略，成为亟待解决的核心问题。传统方法缺乏统一量化指标，难以系统比较不同模型的多语表现，也难以指导实际应用中的模型选择。
Innovation
本文创新点在于提出Polyglot Score（PG-SCORE），结合intrinsic数据质量指标（如多样性、流畅度）与extrinsic学生性能，系统评估教师模型多语能力。引入多维特征分析（如主成分分析）揭示数据特征对学生性能的影响，突破以模型规模为唯一指标的局限。提出模型家族匹配策略，有效提升低资源语种的合成数据质量。研究首次系统比较不同生成方法（生成、翻译、响应）在多语环境中的效果，为多语知识蒸馏提供新思路。这些创新推动多语模型训练的科学化和实用化。
Methodology
- �� 采集10个不同架构和规模的语言模型作为教师，包括Gemma、Llama、Qwen等。
- �� 构建多语种种子数据集，结合公开多语指令调优数据。
- �� 采用三种合成方法：生成（prompt-response采样）、翻译（英语到目标语翻译后生成）、响应（直接用模型响应prompt）。
- �� 利用Llama-EmbedNemotron-8B嵌入文本，计算多样性指标；用M-Prometheus奖励模型评估响应流畅度；计算困惑度（PPL）衡量自然度。
- �� 结合intrinsic指标（多样性、流畅度、长度）归一化后求平均，形成intrinsic数据质量分。
- �� 以合成数据训练学生模型（如OLMo 3 7B），在多任务（文化理解、数学、对话）上评估性能提升，计算外部性能指标（PGR）。
- �� 通过主成分分析提取关键特征，建立线性回归模型预测学生性能，验证指标有效性。
Experiments
在6种语系（阿拉伯语、德语、西班牙语、印尼语、日语、捷克语）上，评估10个教师模型的合成数据效果。每对模型-语种生成10,500个prompt-response样本，重复三次以控制随机性。微调学生模型（OLMo 3 7B）后，使用多任务评估（文化、对话、数学）测定性能提升。比较不同模型规模、架构和生成方法的效果，分析数据特征对性能的影响。实验还包括不同模型家族匹配策略和低资源语种的效果验证，确保结果的广泛适用性。
Results
模型规模与效果无显著相关，Gemma 3 27B和Aya Expanse 32B在所有语种中表现优异，平均PG-SCORE分别为0.726和0.706，优于70B规模的Llama 3.1。数据多样性和响应流畅度特征能解释93.3%以上的质量变异，且能有效预测学生模型在多任务上的表现（R²=0.664）。模型家族匹配显著提升合成效果，低资源语种中，响应和翻译方法优于纯生成策略。不同生成方法在高资源语种（如德语）中效果最佳，而在低资源语种（如阿拉伯）中，响应和翻译更优。这些结果验证了数据质量优于模型规模的假设，为多语模型训练提供了科学依据。
Applications
该方法可用于多语模型开发中的教师模型筛选、合成数据生成和性能优化，特别适合低资源语种的模型训练。企业和研究机构可借助Polyglot Score评估不同模型的多语能力，优化训练流程，减少成本。未来还可结合主动学习和人类反馈，提升低资源语种的表现，实现更公平的多语技术生态。
Limitations & Outlook
研究依赖预定义数据集，低资源语种表现仍有限，模型匹配策略受限于模型家族和预训练数据差异。实验成本较高，规模大，难以快速推广到所有语种和模型架构。未来需探索更低成本、更普适的评估和生成策略，以实现广泛应用。
Plain Language Accessible to non-experts
想象你在厨房里做饭，厨师（模型）负责准备各种菜肴（多语任务）。大厨（大模型）虽然看起来厉害，但不一定每道菜都做得好，尤其是一些少见的菜（低资源语种）。为了让厨师做得更好，你可以用一些“调味料”和“食谱”来训练它（合成数据）。不同的厨师用不同的食谱，有的用英语翻译，有的用已有的菜谱改编。你还会观察厨师做菜的速度、味道和多样性（数据质量指标），并用这些信息决定哪个厨师最适合做某个菜肴。最终，你希望找到既能做出好菜，又能满足不同口味的厨师。这个过程就像研究中的模型评估，目标是用最少的资源，做出最美味的菜肴（高质量多语模型），让每个人都能享受美食（多语任务的成功）。
ELI14 Explained like you're 14
想象你在学校里，有很多不同的老师（模型），他们教不同的科目（语言和任务）。有的老师很大很厉害，但不一定会教所有科目，特别是一些少见的科目（低资源语言）。所以，老师的能力不仅仅看他有多大，还要看他教得好不好。为了让老师教得更好，你可以用一些方法，比如让老师模仿已有的教材（合成数据），或者把教材从英语翻译成其他语言，再让老师讲。然后，你观察老师讲课的内容丰富不丰富、是否流畅、是否多样（数据质量指标），用这些信息决定哪个老师最适合教某个科目。研究发现，老师的大小不是最重要的，关键在于他用的教材和讲课方式。这样一来，不管老师多大，只要教得好，学生就能学得更好，尤其是在一些资源少的科目上。这就像让不同的老师用不同的方法教书，找到最合适的那一个，学生们都能学得开心又有效。
Abstract
Synthesizing supervised finetuning (SFT) data from language models (LMs) to teach smaller models multilingual tasks has become increasingly common. However, teacher model selection is often ad hoc, typically defaulting to the largest available option, even though such models may have significant capability gaps in non-English languages. This practice can result in poor-quality synthetic data and suboptimal student downstream performance. In this work, we systematically characterize what makes an effective multilingual teacher. We measure intrinsic measures of data quality with extrinsic student model performance in a metric we call Polyglot Score; evaluating 10 LMs across 6 typologically diverse languages, generating over 1.4M SFT examples and training 240 student models. Among the models tested, Gemma 3 27B and Aya Expanse 32B emerge as consistently effective teachers across different student base model families. Further analyses reveal that model scale alone does not significantly predict teacher effectiveness; instead, data qualities such as prompt diversity, length, and response fluency capture over 93.3% of variance in intrinsic data quality and predict student performance. Finally, we provide practical recommendations, including matching the model families of teacher-student pairs and translating from or responding to existing prompts, which can yield improvements for less-resourced languages. We hope that our work advances data-centric research in multilingual synthetic data and LM development.