核心发现
方法论
采用系统翻译和筛选，利用GPT-4 mini模型将8个高质量英文基准数据集转为孟加拉语，结合人工验证确保翻译质量。基于此，评估了10个开源多语模型在不同任务上的表现，分析其错误模式和Token化行为。通过准确率、响应错误率和模型一致性指标，比较模型在英语与孟加拉语中的性能差异，特别关注模型大小和架构对表现的影响。
关键结果
- 模型在孟加拉语中的平均准确率明显低于英语，尤其在较小模型和Mistral家族中差距显著，最低模型准确率仅为0.048（Mistral 7B），最高为0.966（Qwen 72B）; 大模型表现更稳定，DeepSeek架构表现出较强鲁棒性，部分模型在孟加拉语表现甚至优于英语。
- Tokenization分析显示，孟加拉语输入的Token数明显高于英语，平均每词Token数增加了30%以上，且Token效率与模型性能呈反比关系，过度Token化导致性能下降。模型在处理复杂多字符字形时，表现出Token碎片化严重的问题，影响理解和推理能力。
- 基准数据集的翻译质量对评估结果影响显著，GPT-4 mini模型在保持语义一致性方面优于其他翻译工具，人工验证确保了数据的可靠性。这为低资源语言的基准构建提供了可行路径，也揭示了翻译质量对模型评估的关键作用。
研究意义
本研究填补了孟加拉语NLP评测标准缺失的空白，提供了系统化的多任务基准数据，为未来模型优化和多语种研究奠定基础。通过分析模型在低资源语境下的表现差异，揭示了模型规模、Token化策略对多语言性能的影响，为多语种模型设计提供实证依据。这对于推动低资源语言的技术普及、实现语言平等具有重要意义，也为多语言模型的公平性和鲁棒性研究提供新视角。
技术贡献
提出了基于GPT-4 mini的高质量翻译流程，构建了8个孟加拉语基准数据集，公开发布数据和工具链。系统评估了10个开源多语模型在多任务上的表现，结合Token化分析，揭示模型大小与Token效率的关系。创新在于结合自动翻译与人工验证，确保低资源语言的评测数据质量，首次系统性比较不同模型在孟加拉语中的性能差异，为多语模型优化提供实证依据。
新颖性
首次系统性构建孟加拉语多任务评测基准，采用GPT-4 mini进行高质量翻译，结合详细错误分析，揭示Token化效率与模型性能的关系。不同于以往仅依赖机器翻译或单一任务评估的方法，本研究强调数据质量与模型架构的结合，提供了低资源语种模型性能的全景视图，具有较强的创新性和实用价值。
局限性
- 翻译过程依赖GPT-4 mini模型，可能存在语义偏差或文化差异，影响评测数据的代表性和普适性。
- 模型评估仅涵盖公开开源模型，未涉及商业或专有模型，限制了结果的全面性。
- Token化分析主要基于静态指标，未充分考虑模型在实际推理中的动态Token处理行为，未来需结合端到端任务评估。
未来方向
未来将扩展更多低资源语言的基准数据集，结合多模态信息提升模型鲁棒性。探索自监督预训练与微调策略，优化Token化机制以适应复杂字符体系。此外，将引入人类评估与用户反馈，完善多语模型的公平性和实用性，推动多语言NLP的广泛应用。
AI 总览摘要
本研究系统性构建了孟加拉语多任务评测基准，填补了该语种在NLP领域的空白。通过将8个英文高质量数据集翻译成孟加拉语，结合人工验证，确保了数据的高质量和代表性。随后，评估了10个开源多语模型在不同任务上的表现，发现模型在孟加拉语中的平均准确率显著低于英语，尤其在较小模型和特定架构中差距明显。分析显示，Token化效率与模型性能呈反比关系，过度Token化导致理解困难，影响推理效果。值得注意的是，DeepSeek架构表现出较强的跨语言鲁棒性，部分模型甚至在孟加拉语中优于英语。这些发现强调了模型规模和Token策略在多语言环境中的重要性，为未来低资源语种模型优化提供了实证依据。研究还揭示了翻译质量对评测结果的影响，采用GPT-4 mini模型实现高质量翻译，为低资源语种基准构建提供了新思路。整体而言，该工作推动了多语言NLP的发展，促进了低资源语言的技术普及，为实现语言平等提供了基础。未来，研究将继续扩展多语种基准，优化Token化机制，结合多模态信息，提升模型的鲁棒性和实用性。
深度解读
原文摘要
Bengali is an underrepresented language in NLP research. However, it remains a challenge due to its unique linguistic structure and computational constraints. In this work, we systematically investigate the challenges that hinder Bengali NLP performance by focusing on the absence of standardized evaluation benchmarks. We then evaluated 10 recent open source Large Language Models (LLMs) in 8 of the translated datasets and performed a comprehensive error analysis to pinpoint their primary failure modes. Our findings reveal consistent performance gaps for Bengali compared to English, particularly for smaller models and specific model families like Mistral. We also identified promising robustness in certain architectures, such as DeepSeek, that maintain more stable performance across languages. Our analysis reveals an inverse relationship between tokenization efficiency and LLM accuracy where models tend to perform worse when inputs are excessively tokenized, whereas more efficient \& concise tokenization results in improved performance. These findings highlight critical areas where current models fall short and underscore the need for improved dataset quality and evaluation methodologies tailored to multilingual contexts. This work will catalyze further research on NLP for underrepresented languages, helping to democratize access to advanced language technologies worldwide. The code and dataset used in this research is publicly available at https://github.com/BengaliAI/bn-llm-benchmark.
参考文献 (20)
BanglaByT5: Byte-Level Modelling for Bangla
Pramit Bhattacharyya, Arnab Bhattacharya
BEnQA: A Question Answering Benchmark for Bengali and English
Sheikh Shafayat, H. Hasan, Minhajur Rahman Chowdhury Mahim 等
A Large Multi-target Dataset of Common Bengali Handwritten Graphemes
Samiul Alam, Tahsin Reasat, Asif Sushmit 等
How Good is Your Tokenizer? On the Monolingual Performance of Multilingual Language Models
Phillip Rust, Jonas Pfeiffer, Ivan Vulic 等
Language Model Tokenizers Introduce Unfairness Between Languages
Aleksandar Petrov, Emanuele La Malfa, Philip H. S. Torr 等
Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formatting
Melanie Sclar, Yejin Choi, Yulia Tsvetkov 等
Do All Languages Cost the Same? Tokenization in the Era of Commercial Language Models
Orevaoghene Ahia, Sachin Kumar, Hila Gonen 等
Judging LLM-as-a-judge with MT-Bench and Chatbot Arena
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng 等
Large Language Models Are Not Robust Multiple Choice Selectors
Chujie Zheng, Hao Zhou, Fandong Meng 等
Association for Computational Linguistics
G. Hirst
Rethinking Tokenization for Rich Morphology: The Dominance of Unigram over BPE and Morphological Alignment
Saketh Reddy Vemula, Sandipan Dandapat, D. Sharma 等
BanglaBERT: Language Model Pretraining and Benchmarks for Low-Resource Language Understanding Evaluation in Bangla
Abhik Bhattacharjee, Tahmid Hasan, Kazi Samin Mubasshir 等
BanglaGPT: A Generative Pretrained Transformer-Based Model for Bangla Language
Md. Shahidul Salim, Hasan Murad, Dola Das 等
BenLLM-Eval: A Comprehensive Evaluation into the Potentials and Pitfalls of Large Language Models on Bengali NLP
M. Kabir, Mohammed Saidul Islam, Md Tahmid Rahman Laskar 等
IndicLLMSuite: A Blueprint for Creating Pre-training and Fine-Tuning Datasets for Indian Languages
Mohammed Safi Ur Rahman Khan, Priyam Mehta, A. Sankar 等
Khayyam Challenge (PersianMMLU): Is Your LLM Truly Wise to The Persian Language?
Omid Ghahroodi, Marzia Nouri, M. V. Sanian 等
Bridging the Bosphorus: Advancing Turkish Large Language Models through Strategies for Low-Resource Language Adaptation and Benchmarking
Emre Can Acikgoz, Mete Erdogan, Deniz Yuret
xFinder: Large Language Models as Automated Evaluators for Reliable Evaluation
Qingchen Yu, Zifan Zheng, Shichao Song 等
Testing prompt engineering methods for knowledge extraction from text
Fina Polat, Ilaria Tiddi, Paul Groth
被引用 (11)
Quantization Effects on Bangla Language Understanding in Large Language Models: A Systematic Evaluation
Creating Multilingual Mental Health Dialogue Datasets: Limits of Persona-Based Localization via Nationality and Language
Recent Advances in Bangla NLP: A Survey of Few-Shot, Zero-Shot, and Transfer Learning Techniques
Structural Silence: When AI Infrastructure Fails Speakers of Underrepresented Languages
Do Small Models Use the Law You Give Them? Context-Injected Fine-Tuning for Legal QA in Bangladesh
BenHalluEval: A Multi-Task Hallucination Evaluation Framework for Large Language Models on Bengali
Cross-Lingual Sentiment Misalignment: Auditing Multilingual Language Models for Inversion Risk, Dialectal Representation, and Affective Stability
UrduBench: An Urdu Reasoning Benchmark using Contextually Ensembled Translations with Human-in-the-Loop
GanitLLM: Difficulty-Aware Bengali Mathematical Reasoning through Curriculum-GRPO
PESCO-BERT: An Efficient Prompt-Based Contrastive Learning for Bangla News Classification
CUET_Expelliarmus at BLP2025 Task 2: Leveraging Instruction Translation and Refinement for Bangla-to-Python Code Generation with Open-Source LLMs