ALEE: Any-Language Evaluation of Embeddings via English-Centric Minimal Pairs
ALEE framework uses AMR-based minimal pairs to evaluate 275+ languages' embedding models, revealing performance gaps related to resources and linguistic phenomena.
Key Findings
Methodology
ALEE employs Abstract Meaning Representation (AMR) to generate controlled semantic minimal pairs by applying rule-based edits such as polarity negation, role swap, antonym, and hypernym substitution. These AMR graphs are used to produce sentence variants with precise semantic shifts. Parallel corpora link original and modified sentences across languages. Embedding models are evaluated via similarity scores (e.g., LaBSE, MPNet, E5) on these pairs, with NLI models verifying semantic differences. The framework supports 275+ languages, analyzing performance variations influenced by training data size, subword fragmentation, and text length. Large-scale experiments reveal persistent gaps in cross-lingual semantic understanding, especially for low-resource languages and complex phenomena.
Key Results
- Models perform best on polarity negation (~85% accuracy), but struggle with role swap and antonym replacement (~60-70%). Low-resource languages show 30% lower TACC compared to high-resource ones. Longer texts and multi-sentence inputs decrease performance by ~15%. Performance correlates strongly with training data volume and subword fragmentation. LaBSE and multilingual E5 models outperform smaller counterparts, indicating the importance of pretraining data. These results highlight ongoing challenges in capturing nuanced cross-lingual semantics.
- Performance varies significantly across languages, with resource-rich languages achieving higher accuracy. The experiments demonstrate that models trained on larger datasets and with less subword fragmentation better distinguish subtle semantic differences. The analysis of Romansh dialects shows that written prevalence and tokenizer coverage directly impact model performance, emphasizing the importance of linguistic resource availability for low-resource languages.
Significance
This work advances the evaluation of multilingual embeddings by providing a dynamic, scalable benchmark that captures fine-grained semantic distinctions across diverse languages. Unlike static datasets, ALEE enables targeted diagnostics, revealing specific weaknesses in models’ cross-lingual understanding. It addresses the critical need for fair and comprehensive evaluation in low-resource settings, guiding future development of more robust, inclusive multilingual models. The framework’s ability to diagnose phenomena like polarity, argument roles, and lexical opposites offers valuable insights for both academia and industry, fostering progress toward truly universal language understanding systems.
Technical Contribution
ALEE innovatively combines AMR-based semantic manipulation with parallel corpora to generate diverse, controlled minimal pairs across 275+ languages. Its automated pipeline allows scalable, dynamic evaluation of models’ sensitivity to specific semantic phenomena. The framework integrates multiple perturbation types, verified via NLI filtering, and extends to paragraph-level texts through iterative sentence manipulation. This approach enables fine-grained, phenomenon-specific diagnostics that surpass traditional static benchmarks, providing a new tool for interpretability and robustness analysis of multilingual embeddings.
Novelty
This is the first framework to leverage AMR for cross-lingual minimal pair generation at scale, covering over 275 languages. Unlike prior static datasets, ALEE dynamically creates targeted semantic contrasts, supporting low-resource and dialectal varieties. Its integration of semantic graph manipulations with parallel corpora and NLI validation offers a novel, scalable approach for detailed cross-lingual semantic evaluation, filling a significant gap in current multilingual NLP assessment tools.
Limitations
- 依赖高质量平行语料，低资源语种覆盖有限，可能影响评估的全面性。
- AMR操控在复杂句和长段落中效果有限，可能引入语义偏差或不自然的变体。
- 模型性能受训练数据分布和子词切分影响较大，未充分考虑模型结构差异对评估结果的影响。
Future Work
未来将结合多模态语义资源（如FrameNet、PropBank），丰富极小对的多样性。探索无监督或少监督的极小对生成技术，提升低资源语种适应性。引入人类评估和多模态信息，增强评估的全面性。优化AMR操控流程，提高生成质量和效率，扩展到更多语种和复杂语义现象。
AI Executive Summary
在多语种自然语言处理领域，评估模型的跨语义理解能力一直是核心难题。传统基准多为静态、单语种，难以反映实际多语环境中的表现差异。本文提出ALEE框架，利用AMR的结构化语义表示，自动生成跨语种极小对，涵盖275+语种。通过在平行语料中施加多种语义变换（如极性否定、角色交换、反义词替换、上位词替换），评估模型在不同语种中的敏感性。实验结果显示，模型在低资源语种和复杂语义现象上表现不足，性能与训练资源、子词碎片化密切相关。ALEE的创新在于结合AMR的结构化表示，实现动态、多现象、多层次的语义诊断，突破静态评测的局限。未来，该框架有望结合更多语义资源，支持低资源语种，推动多语种自然语言理解的公平性与鲁棒性。整体而言，ALEE为多语种语义评估提供了全新工具和视角，有助于构建更智能、更包容的多语种AI系统。
Deep Analysis
Background
多语种嵌入模型的研究经历了从词向量对齐到深层预训练模型的演变。早期如MUSE、FastText主要关注词级对齐，随后BERT、XLM系列提升了跨语义理解能力。然而，现有评估多依赖静态数据集（如XNLI、SemEval），在高资源语种表现优异，但低资源语种缺乏敏感性。近年来，极小对（minimal pairs）成为细粒度诊断工具，Li等（2025）提出的Sentence Smith利用AMR实现动态极小对生成，为语义敏感性检测提供新思路。尽管如此，现有方法多局限于少数高资源语种，难以覆盖多语环境，且缺乏对复杂语义现象的系统评估。
Core Problem
当前多语种嵌入模型在捕捉细粒度语义差异方面表现不足，尤其在低资源语种和复杂语义变化中。静态数据集无法动态反映模型实际表现，且难以区分模型对不同语义现象的敏感度。缺乏一种支持多语、多现象、可扩展的评估框架，限制了模型的优化方向。如何设计一种自动化、支持多语种、可调控语义变换的评估方法，成为亟待解决的核心问题。
Innovation
本文提出ALEE框架，结合AMR的结构化语义表示，通过规则化的语义操控（极性否定、角色交换、反义词、上位词）生成极小对。其核心创新在于：1）利用AMR图的结构化信息，确保变体在语义上的受控差异；2）在平行语料中施加变换，实现跨语种极小对的自动生成；3）支持多种语义现象，覆盖多维度差异；4）支持多文本长度，从单句到段落，提升实用性。这些创新使得ALEE在多语种、多现象、多层次的语义诊断中具有明显优势。
Methodology
- �� 解析英语句子为AMR图，提取语义结构。
- �� 在AMR图中应用规则（如添加:polarity-，交换:ARG0和:ARG1、替换反义词、上位词）生成变体。
- �� 利用AMR到文本的生成模型，输出语义变体句子。
- �� 将变体句与原句配对，结合平行语料中的目标语种句子，形成极小对。
- �� 通过预训练模型（如LaBSE、mpnet、E5）计算相似性，评估模型对极小对的区分能力。
- �� 采用NLI模型验证极小对的语义差异，过滤无效对。
- �� 扩展到段落，采用多轮分句、逐句操控、验证，处理长文本。
- �� 在多语种平行数据（如WMT24++、BOUQuET）上进行大规模评估，分析性能差异。
Experiments
使用FLORES-200、WMT24++和BOUQuET三大平行语料库，涵盖275+语种。评估模型包括LaBSE、mpnet、E5系列等。指标采用Triplet Accuracy（TACC），衡量模型在极小对中的表现。对不同语种、文本长度和语义变换类型进行分析，比较模型在高低资源语种的差异。还通过子词碎片化、预训练和微调数据分布等因素，探讨性能差异的根源。实验设计包括多轮变换验证、人工质量控制和统计分析。
Results
模型在极性否定上表现较好，准确率达85%以上，但角色交换和反义词替换的准确率仅在60-70%。低资源语种平均TACC低于高资源语种30%以上。长文本和多句段落的性能明显下降，平均下降幅度达15%。预训练数据规模与性能呈正相关，子词碎片化也显著影响表现。不同模型中，LaBSE和E5-instruct表现优异，验证了多语种预训练的重要性。这些结果揭示了模型在多语种、多现象语义理解上的不足。
Applications
该框架可用于多语种模型的诊断与改进，帮助开发者识别模型在低资源语种和复杂语义现象上的弱点。适用于跨语种信息检索、机器翻译、语义匹配等场景。未来可结合人类评估，优化极小对生成策略，提升模型的语义敏感性和公平性。
Limitations & Outlook
依赖高质量平行语料，低资源语种覆盖有限。AMR操控在长句和复杂句中效果有限，可能引入语义偏差。模型性能受训练数据分布影响大，未充分考虑模型结构差异。未来需结合多模态信息和人类反馈，提升评估的全面性和鲁棒性。
Plain Language Accessible to non-experts
想象你在厨房做饭，每个菜都需要不同的调料。模型就像厨师，要知道不同调料的作用，才能做出美味的菜。ALEE就像用特殊的调料（AMR）调出不同的味道（语义变化），然后测试厨师（模型）是否能分辨出这些不同的味道。通过不断调整调料，观察厨师的反应，帮助改进厨艺（模型的语义理解）。这就像用不同的调料组合，检验厨师（模型）在不同菜肴（语种）中的不足。这种方法既科学又有趣，也能帮助我们找到厨师（模型）在不同菜肴（语种）中的不足。
ELI14 Explained like you're 14
想象你在学校的厨房里帮忙做饭，每次你都要用不同的调料试试，看厨师（模型）能不能分辨出这些调料的不同。有时候你会用盐代替糖，有时候把鸡肉换成牛肉，或者用不同的香料。这就像给句子加点特别的“调料”——比如变成否定句、换角色、用反义词。然后你问厨师：这道菜变了吗？他能不能发现这些变化？ALEE就是用这种办法，自动帮我们做很多不同的“菜”，测试模型是不是能理解这些细微的差别。这样，我们就知道模型在多语种和复杂语义上是不是聪明，能不能像人一样理解不同的“味道”。
Abstract
Text embeddings are standard for semantic similarity tasks, yet their evaluation remains an open challenge. Current benchmarks are static, cover only a limited set of languages, are often domain-specific, susceptible to overfitting, and poorly representative of low-resource languages. To address these limitations, we introduce ALEE, a framework that extends Sentence Smith (Li et al., 2025) to the cross-lingual and paragraph level. ALEE uses Abstract Meaning Representations (AMR) to generate English minimal pairs with controlled, fine-grained semantic shifts, which are paired with translations in target languages. This approach enables targeted diagnostics for models in any language with English parallel data. We conduct a large-scale empirical study across a diverse set of embedding models and 275+ languages spanning three parallel datasets. On ALEE, performance varies substantially across languages, text lengths, and linguistic phenomena, exposing persistent gaps in cross-lingual semantic representation that track language prevalence in training resources and subword tokenization. We release ALEE at https://github.com/Andrian0s/any-lang-embed-eval