Multilingual Embedding Probes Fail to Generalize Across Learner Corpora
This study evaluates multilingual embedding models' ability to encode general proficiency, revealing poor cross-corpus transfer with performance dropping from QWK≈0.7 to 0.38.
Key Findings
Methodology
Seven models (0.3-8B parameters) were analyzed using five probe architectures (linear, ordinal, MLP, classifier) across nine datasets in seven languages. Probes trained on in-distribution data achieved QWK≈0.7, outperforming surface features (QWK≈0.45). Cross-corpus testing revealed performance collapse to QWK≈0.38, with residual analysis indicating predictions converged to uniform labels, suggesting learned representations capture corpus-specific distributional properties rather than an abstract proficiency dimension.
Key Results
- Probes in in-distribution tests achieved QWK≈0.7, especially in middle layers, significantly better than surface baselines. However, in out-of-distribution tests, performance dropped to QWK≈0.38, indicating poor transferability.
- Larger models (up to 8B) showed marginally better in-distribution performance, but cross-corpus transfer remained weak, with no consistent layer or architecture outperforming others.
- Residual analysis confirmed that out-of-distribution predictions tend toward uniform distribution, implying learned mappings are corpus-specific rather than general proficiency representations.
Significance
These findings challenge the assumption that current multilingual embeddings encode transferable, language-general proficiency. The results highlight the impact of dataset biases and distributional confounds, suggesting that existing models are limited in their ability to generalize proficiency across diverse corpora. This has profound implications for developing robust, proficiency-aware language technologies, emphasizing the need for more balanced, diverse training data and advanced representation learning strategies.
Technical Contribution
The study systematically compares multiple models and probe architectures in multilingual, multi-corpus settings, revealing that learned representations predominantly encode corpus-specific features rather than abstract proficiency. It introduces residual analysis as a diagnostic tool to assess transferability, providing new insights into the limitations of current embedding-based proficiency modeling and guiding future research toward more generalizable representations.
Novelty
This is the first comprehensive evaluation of multilingual embedding models' capacity for cross-corpus proficiency transfer, employing diverse probe architectures and large-scale datasets. It uncovers fundamental limitations in current approaches, emphasizing the importance of distributional factors and proposing diagnostic methods to assess transferability.
Limitations
- The reliance on CEFR labels limits the scope to a single proficiency dimension, neglecting other linguistic or pragmatic skills.
- Performance heavily depends on dataset diversity; severe biases reduce generalization, requiring more balanced corpora.
- Model size and architecture influence results, but further exploration with more advanced models and training paradigms is needed.
Future Work
Future research should focus on constructing more balanced, diverse datasets to reduce confounds, exploring multi-task and multi-modal training to enhance generalization, and developing theoretical frameworks for abstract proficiency representations. Additionally, integrating dynamic, context-aware models could improve transferability across unseen corpora.
AI Executive Summary
Multilingual embedding models hold promise for automated proficiency assessment but face significant challenges in cross-corpus transferability. This study systematically evaluates seven models, ranging from 0.3B to 8B parameters, using five probe architectures across nine diverse datasets. In in-distribution tests, models achieved high performance (QWK≈0.7), especially in middle layers, indicating they can encode proficiency effectively within familiar data. However, when tested on unseen corpora, performance collapsed to QWK≈0.38, revealing a stark inability to generalize. Residual analysis demonstrated that out-of-distribution predictions tend toward uniform labels, suggesting the learned representations primarily capture corpus-specific distributional features like topic, language, or rating methodology rather than an abstract proficiency dimension. These findings challenge the assumption that current multilingual embeddings inherently encode transferable language proficiency, highlighting the importance of dataset diversity and distributional considerations in developing robust proficiency models. The results underscore the need for more balanced, multi-source training data and advanced representation learning strategies to achieve true cross-corpus generalization, vital for scalable, proficiency-aware language technology. Overall, this work provides critical insights into the limitations of existing embedding-based approaches and charts a course for future research aimed at overcoming these barriers.
Deep Analysis
Background
随着大规模预训练语言模型的发展，其在多语能力评估中的潜力逐渐被认识。早期如BERT、XLM-R等模型在语义理解、句法分析中表现优异，但在能力迁移方面仍存局限。近年来，探针技术被引入，用于分析模型内部表征，试图解码能力信息，但多集中在单语或单语料场景。多语能力评估面临多样性与偏差问题，如何让模型学习到跨语料、跨任务的通用能力，成为核心难题。现有方法多依赖表面特征或单一任务，缺乏系统验证其迁移能力。本文通过多模型、多架构、多语料的实验设计，试图揭示预训练模型能力表示的本质，推动多语能力评估的理论基础。
Core Problem
核心问题在于，预训练多语嵌入模型是否编码了跨语料的通用能力表示。尽管在同分布数据上表现良好，但迁移测试中性能显著下降，说明模型可能只学习了语料特有的分布特征（如话题、任务类型），而非能力本身。这限制了模型在实际多语环境中的应用，影响其在多语能力评估、个性化学习等场景中的实用性。解决这一问题需要深入分析模型内部表征的迁移能力，理解其局限性。
Innovation
本研究的创新在于：1）系统性比较多模型、多架构在多语料上的迁移表现，揭示模型未学习到抽象能力的根本原因；2）结合残差分析，明确模型偏向语料特有的分布特征；3）提出跨语迁移性能下降主要源于数据偏差，强调平衡多语料的重要性。这些突破为能力表示的理论研究提供了新视角，推动多语能力评估的科学发展。
Methodology
- �� 采集七个不同规模（0.3B-8B参数）预训练模型（如Qwen3、Gemma、Llama）• 提取每个模型五个隐藏层的激活状态，作为能力表征基础• 设计五种探针（线性回归、序数回归、多层感知机等）在九个多语料库中训练预测CEFR等级• 采用在每个语料上训练、验证在相同分布和留一法迁移场景• 比较表面特征（如文本长度）与激活的预测性能• 进行残差分析，检测迁移性能下降的根源• 评估不同模型规模、架构变化对迁移能力的影响。
Experiments
- �� 使用UniversalCEFR数据集，涵盖7种语言、9个语料库、不同CEFR级别• 训练在每个语料上，验证在相同分布和留一法迁移场景• 采用QWK指标，比较不同模型和探针架构性能• 设计多层探针，分析中间层代表能力的效果• 通过残差分析，揭示迁移性能下降的原因• 评估模型规模、架构变化对迁移能力的影响。
Results
- �� 在同分布验证中，探针在中间层达成QWK≈0.7，优于表面特征（QWK≈0.45）• 跨语料迁移时，性能大幅下降，平均QWK降至0.38，表现出迁移能力不足• 模型越大，迁移性能略有提升，但差异不显著• 残差分析显示迁移预测趋向随机，未学习到抽象能力。
Applications
- �� 可用于多语能力评估，辅助教育、语言学习平台实现个性化教学• 需确保训练语料多样化，减少偏差，提升模型迁移能力• 未来可结合多模态信息，增强能力表示的稳健性。
Limitations & Outlook
- �� 仅依赖CEFR标签，未考虑其他能力维度或多模态信息• 语料偏差严重，影响迁移性能• 模型规模和架构有限，未来需探索更复杂模型和训练策略。
Plain Language Accessible to non-experts
想象你在一家工厂里，工人们每天都在做不同的任务。有的工人擅长装配，有的擅长焊接，但每个人都用不同的工具和流程。工厂想让每个工人都能学会所有任务，但实际上，每个任务的工具和流程都不一样，工厂的系统（模型）试图通过观察工人的表现（激活）来判断他们的技能水平（能力）。然而，研究发现，工厂的系统在熟悉的任务（语料）中表现很好，能准确判断工人的技能，但当遇到新任务（不同语料）时，判断就变得很差。原因是系统只记住了每个任务的特殊工具和流程，而没有学到真正的技能（抽象能力）。这就像你只记住了某个工厂的操作流程，却不知道怎么在别的工厂用相同的技能。
Abstract
Do multilingual embedding models encode a language-general representation of proficiency? We investigate this by training linear and non-linear probes on hidden-state activations from seven embedding models (0.3-8B) to predict CEFR proficiency levels from learner texts across nine corpora and seven languages. We compare five probing architectures against a baseline trained on surface-level text features. Under in-distribution evaluation, probes achieve strong performance (Quadratic Weighted Kappa $\approx0.7$), substantially outperforming the surface baseline, with middle layers consistently yielding the best predictions. However, in cross-corpus evaluation performance collapses across all probe types and model sizes. Residual analysis reveals that out-of-distribution probes converge towards predicting uniformly distributed labels, indicating that the learned mappings capture corpus-specific distributional properties (topic, language, task type, rating methodology) rather than an abstract, transferable proficiency dimension. These results suggest that current multilingual embeddings do not straightforwardly encode language-general proficiency, with implications for representation-based approaches to proficiency-adaptive language technology.