MTEB-BR: A Text Embedding Benchmark for Brazilian Portuguese
Introduces MTEB-BR, a benchmark with 22 native Portuguese tasks, evaluating 93 models using rigorous statistical analysis to distinguish capability tiers.
Key Findings
Methodology
This study constructs a native, translation-free task suite covering classification, similarity, clustering, retrieval, and reranking across seven categories, sourced solely from Brazilian Portuguese data. Evaluating 93 models—from 23 million to 27 billion parameters, including open-source and commercial APIs—employs advanced statistical techniques: bootstrap confidence intervals, paired bootstrap significance tests, item response theory (IRT) discrimination analysis, and cross-leaderboard correlation. These methods enable precise differentiation of model capabilities, revealing approximately a dozen distinct tiers. The top six models are statistically indistinguishable, emphasizing the challenge of fine-grained ranking. The benchmark’s design ensures robustness and relevance for low-resource language evaluation, providing a comprehensive, scientifically grounded framework.
Key Results
- The benchmark effectively separates around 12 model tiers, with the top model Gemini-Embedding-001 achieving a mean score of 0.682 across 22 tasks, outperforming most peers. Statistical analysis shows 78.7% of model pairs are distinguishable at a 95% confidence level, with the top six models converging within a narrow margin. The correlation between the Portuguese-specific benchmark and the multilingual leaderboard is moderate (Spearman ρ=0.75), indicating the importance of native evaluation. Notably, an open licensed, self-hostable model reaches the top tier, demonstrating high-quality Portuguese embeddings without commercial API reliance.
- The detailed statistical layer confirms the robustness of the model ranking, with the ability to resolve most model differences except within the top six. Retrieval tasks exhibit the highest discriminative power, while clustering tasks are less effective at differentiating models. The results highlight the subtle performance gaps among leading models, emphasizing the need for more challenging tasks to further discriminate capabilities. The findings support the use of native benchmarks for accurate assessment of language-specific model performance.
- Overall, the benchmark provides a nuanced, reliable ranking of models, with multiple statistical measures validating the distinctions. It reveals that model performance converges at the top, yet still offers meaningful differentiation across the broader spectrum. This comprehensive evaluation framework advances the field by establishing a rigorous standard for low-resource language embedding assessment, fostering development of more capable and culturally relevant models.
Significance
This work addresses a critical gap in NLP evaluation for Portuguese, a language spoken by over 200 million people. Existing multilingual benchmarks often rely on translated data or limited tasks, which can obscure true model capabilities in native contexts. By creating a dedicated, statistically rigorous benchmark, this study enables more accurate assessment and comparison of models tailored to Portuguese. It empowers researchers and practitioners to select and optimize models based on native performance, reducing reliance on translation-based proxies. The approach also sets a precedent for evaluating other low-resource languages, promoting more inclusive and representative NLP research. Ultimately, this work accelerates the development of high-quality, culturally adapted language models, fostering better AI applications in education, healthcare, legal, and social domains within Portuguese-speaking communities.
Technical Contribution
The study's key technical innovations include the construction of a native, translation-free task suite specifically for Portuguese, ensuring authentic language evaluation. It integrates multiple advanced statistical tools—bootstrap confidence intervals, paired significance tests, and item response theory (IRT)—to rigorously quantify model differences at both task and instance levels. The use of IRT provides a nuanced understanding of task discrimination power, while cross-leaderboard correlation analysis assesses the consistency of model rankings across different evaluation frameworks. The comprehensive evaluation of 93 diverse models, from open-source to commercial APIs, across a broad parameter spectrum, offers a detailed capability landscape. This methodological rigor and dataset diversity set new standards for low-resource language model assessment, providing a replicable framework adaptable to other languages.
Novelty
This research is the first to establish a native, translation-free benchmark for Portuguese, explicitly excluding machine-translated datasets to avoid artifacts. It innovatively combines multiple statistical analysis layers—bootstrap confidence intervals, paired significance, and IRT discrimination—to achieve fine-grained differentiation of model capabilities. Unlike prior benchmarks that primarily focus on scope, this work emphasizes statistical robustness and native language authenticity, offering deeper insights into models’ true proficiency in Portuguese. The integration of these methods creates a more reliable, interpretable ranking system, setting a new standard for low-resource language evaluation.
Limitations
- While the benchmark effectively distinguishes most models, the top six models are statistically indistinguishable, limiting precise ranking at the highest level. The dataset size for some tasks is relatively small, which may affect confidence interval accuracy. The evaluation focuses solely on embedding quality, not end-to-end task performance, thus not capturing real-world application effectiveness. The statistical complexity may hinder accessibility for non-expert users, requiring further simplification or tooling. Additionally, the benchmark currently emphasizes static tasks; incorporating generative and multi-modal tasks could provide a more comprehensive assessment in future work.
Future Work
Future directions include expanding the task suite to cover more real-world applications such as question answering and generation tasks, integrating multi-modal data for richer evaluation, and analyzing training data biases to understand model limitations. Developing lightweight, user-friendly statistical tools will facilitate broader adoption. Extending the benchmark to include end-to-end task performance metrics and exploring domain-specific models will enhance practical relevance. Additionally, applying this framework to other low-resource languages can promote inclusive NLP development. Ultimately, the goal is to establish a comprehensive, scalable, and interpretable evaluation ecosystem that drives continuous improvement in language models tailored for diverse linguistic and cultural contexts.
AI Executive Summary
Deep Dive
Plain Language Accessible to non-experts
想象你在一家工厂里，工厂每天都在生产各种不同的产品。有的产品需要用不同的材料，有的用不同的机器制造。为了确保每个产品都符合标准，工厂会用一套检测系统，快速判断每个产品的质量。这个检测系统会设计很多不同的测试，就像老师出题考你一样。每个测试都用来检查产品的某个特性，比如外观、耐用性或功能。通过这些测试，工厂可以知道哪个生产线的产品更好，哪个还需要改进。这个研究就像是在设计这样一套检测系统，专门用来评估用葡萄牙语写的“产品”——也就是模型的能力。它帮助大家公平、科学地比较不同模型，找到最适合葡萄牙语环境的“优质产品”。
ELI14 Explained like you're 14
想象你在学校参加考试，老师出很多不同类型的题目，测试你的数学、语文、科学能力。每个题目代表你在某个方面的表现。有些题很难，有些题很简单。你有很多朋友，有的擅长数学，有的擅长语文。老师想知道谁最厉害，但每个人的强项不同。于是，老师用一种特别的评分方法，既看每个题目的表现，也考虑每个人的整体水平。这个方法用一些数学技巧，确保每个题目的难度都被公平对待，最后得出每个人的排名。这个研究就像是用这样的方法，来衡量不同的模型在理解葡萄牙语方面的能力，确保评估公平、科学，帮助开发出更聪明、更贴近真实使用场景的模型。
Abstract
Text embeddings for Portuguese have no dedicated benchmark: evaluation rests on translated corpora such as English MS MARCO or on thin multilingual coverage, with native tasks scattered and unconsolidated. We introduce MTEB-BR, a benchmark of 22 native Brazilian-Portuguese tasks across seven categories (classification, multilabel classification, pair classification, semantic textual similarity, clustering, retrieval, and reranking), admitting only data created or found in Portuguese and excluding translations by construction. We evaluate 93 models spanning 23M to 27B parameters: 73 open-weight and 20 closed commercial APIs. Alongside the leaderboard we report a statistical layer for every headline comparison: per-task bootstrap confidence intervals, paired-bootstrap significance, a task- and instance-level discrimination analysis (how sharply each task separates models) adapted from Item Response Theory, and a cross-leaderboard correlation. Three findings stand out. The benchmark cleanly separates about a dozen tiers of models, though the top six are statistically too close to order. An openly licensed, self-hostable model reaches that leading tier, so strong Portuguese embedding quality does not require a commercial API. And a model's rank on the global multilingual leaderboard predicts its Portuguese rank only moderately (Spearman rho = 0.75 over 55 shared models; one model ranks 3rd there and 49th here), so a native benchmark measures something the multilingual boards do not. We release every task, our code, and a public leaderboard, so practitioners can choose Portuguese embedding models on native evidence.