Evaluating LLMs' Multilingual Capabilities for Bengali: Benchmark Creation and Performance Analysis
Constructed Bengali multilingual benchmarks, evaluated 10 open-source LLMs, found size and tokenization impact performance, with larger models showing more robustness.
Key Findings
Methodology
The study employed systematic translation of 8 high-quality English datasets into Bengali using GPT-4 mini, followed by manual validation to ensure translation fidelity. Based on these datasets, ten open-source multilingual LLMs were evaluated across various tasks, analyzing their error modes and tokenization behaviors. Metrics such as accuracy, response error rate, and model consistency were used to compare performance in English and Bengali, with particular focus on model size and architecture effects. The evaluation framework combined automated metrics and human-like judgment via GPT-based evaluators, providing comprehensive insights into language-specific performance gaps.
Key Results
- Models exhibited significantly lower average accuracy in Bengali compared to English, especially smaller models and Mistral family, with the lowest accuracy at 0.048 (Mistral 7B) and highest at 0.966 (Qwen 72B). Larger models demonstrated more stable performance, with DeepSeek architecture showing notable robustness. Some models even outperformed English prompts in structured tasks, indicating the influence of translation quality and tokenization strategies.
- Tokenization analysis revealed that Bengali inputs produce approximately 30% more tokens per word than English, with token efficiency inversely correlated with model performance. Complex multi-character glyphs in Bengali cause token fragmentation, impairing model understanding and reasoning. Efficient tokenization correlates with higher accuracy, emphasizing the importance of optimized subword segmentation.
- High-quality translation using GPT-4 mini was crucial for reliable evaluation, as it preserved semantic integrity better than other machine translation tools. This approach demonstrates a practical pathway for creating benchmarks in low-resource languages, highlighting the critical role of translation fidelity in model assessment. The publicly released datasets and tools facilitate future research in underrepresented languages.
Significance
This work addresses the critical gap in standardized benchmarks for Bengali NLP, providing a comprehensive evaluation framework that enables fair comparison of models across languages. By analyzing the impact of model size and tokenization, it offers empirical evidence guiding model design for low-resource languages. The findings promote equitable access to advanced NLP technologies, fostering inclusive language AI development. The methodology and datasets serve as a foundation for future multilingual research, encouraging community efforts to improve low-resource language NLP capabilities and ensuring broader linguistic diversity in AI applications.
Technical Contribution
The study introduced a systematic pipeline combining GPT-4 mini translation and manual validation to produce high-quality Bengali datasets. It evaluated ten open-source multilingual LLMs across multiple tasks, integrating detailed tokenization analysis to reveal the relationship between token efficiency and model performance. The research also provided a comparative analysis of model robustness across languages, highlighting architecture-specific strengths like DeepSeek. These contributions advance the state-of-the-art in low-resource multilingual evaluation, emphasizing data quality and tokenization strategies as key factors for performance improvements.
Novelty
This is the first comprehensive effort to systematically translate and evaluate multiple NLP benchmarks in Bengali, utilizing GPT-4 mini for high-quality translation and combining automated and manual validation. Unlike prior works that rely solely on machine translation or limited datasets, this study emphasizes data quality, detailed tokenization analysis, and cross-lingual performance comparison. Its innovative approach provides a holistic view of low-resource language model capabilities, setting a new standard for multilingual benchmark development and evaluation.
Limitations
- The translation process depends heavily on GPT-4 mini, which may introduce semantic biases or cultural inaccuracies, potentially affecting the representativeness of the datasets.
- Evaluation is limited to open-source models, excluding commercial or proprietary models, which may perform differently in real-world applications.
- Tokenization analysis focuses on static metrics and does not fully capture dynamic behaviors during inference, suggesting the need for more integrated evaluation methods.
Future Work
Future research will expand to include more low-resource languages, develop adaptive tokenization techniques tailored to complex scripts, and incorporate human-in-the-loop evaluation for better semantic fidelity. Exploring multi-modal data integration and fine-tuning strategies will further enhance model robustness. Additionally, establishing community-driven benchmarks and expanding dataset diversity will promote inclusive NLP development for underrepresented languages.
AI Executive Summary
This research presents a pioneering effort to establish standardized multilingual benchmarks for Bengali, a language historically underrepresented in NLP. By translating eight high-quality English datasets into Bengali using GPT-4 mini and validating the translations through manual review, the study ensures data reliability. The evaluation of ten open-source multilingual large language models across diverse tasks reveals a consistent performance gap between Bengali and English, with smaller models and certain architectures like Mistral showing significant weaknesses. Notably, larger models such as Qwen 72B and DeepSeek demonstrate more stable performance, highlighting the importance of model scale and architecture in cross-lingual robustness.
A key insight from the analysis is the inverse relationship between tokenization efficiency and model accuracy. Bengali inputs tend to generate more tokens per word, especially with complex glyphs, which hampers model understanding and reasoning. Efficient tokenization strategies, therefore, are critical for improving low-resource language performance. The study also underscores the importance of translation quality; GPT-4 mini's high-fidelity translations enable more accurate benchmarking, setting a new standard for low-resource language evaluation.
Overall, this work significantly advances the field by providing high-quality datasets, comprehensive evaluation metrics, and detailed analysis of model behaviors in Bengali. It offers practical guidance for developing more inclusive NLP models that can serve diverse linguistic communities. Future directions include expanding benchmarks to other low-resource languages, refining tokenization methods, and integrating multi-modal data to further boost model robustness and applicability. This research lays a solid foundation for democratizing NLP technology across languages, fostering a more equitable AI landscape.
Deep Dive
Abstract
Bengali is an underrepresented language in NLP research. However, it remains a challenge due to its unique linguistic structure and computational constraints. In this work, we systematically investigate the challenges that hinder Bengali NLP performance by focusing on the absence of standardized evaluation benchmarks. We then evaluated 10 recent open source Large Language Models (LLMs) in 8 of the translated datasets and performed a comprehensive error analysis to pinpoint their primary failure modes. Our findings reveal consistent performance gaps for Bengali compared to English, particularly for smaller models and specific model families like Mistral. We also identified promising robustness in certain architectures, such as DeepSeek, that maintain more stable performance across languages. Our analysis reveals an inverse relationship between tokenization efficiency and LLM accuracy where models tend to perform worse when inputs are excessively tokenized, whereas more efficient \& concise tokenization results in improved performance. These findings highlight critical areas where current models fall short and underscore the need for improved dataset quality and evaluation methodologies tailored to multilingual contexts. This work will catalyze further research on NLP for underrepresented languages, helping to democratize access to advanced language technologies worldwide. The code and dataset used in this research is publicly available at https://github.com/BengaliAI/bn-llm-benchmark.
References (20)
BanglaByT5: Byte-Level Modelling for Bangla
Pramit Bhattacharyya, Arnab Bhattacharya
BEnQA: A Question Answering Benchmark for Bengali and English
Sheikh Shafayat, H. Hasan, Minhajur Rahman Chowdhury Mahim et al.
A Large Multi-target Dataset of Common Bengali Handwritten Graphemes
Samiul Alam, Tahsin Reasat, Asif Sushmit et al.
How Good is Your Tokenizer? On the Monolingual Performance of Multilingual Language Models
Phillip Rust, Jonas Pfeiffer, Ivan Vulic et al.
Language Model Tokenizers Introduce Unfairness Between Languages
Aleksandar Petrov, Emanuele La Malfa, Philip H. S. Torr et al.
Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formatting
Melanie Sclar, Yejin Choi, Yulia Tsvetkov et al.
Do All Languages Cost the Same? Tokenization in the Era of Commercial Language Models
Orevaoghene Ahia, Sachin Kumar, Hila Gonen et al.
Judging LLM-as-a-judge with MT-Bench and Chatbot Arena
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng et al.
Large Language Models Are Not Robust Multiple Choice Selectors
Chujie Zheng, Hao Zhou, Fandong Meng et al.
Association for Computational Linguistics
G. Hirst
Rethinking Tokenization for Rich Morphology: The Dominance of Unigram over BPE and Morphological Alignment
Saketh Reddy Vemula, Sandipan Dandapat, D. Sharma et al.
BanglaBERT: Language Model Pretraining and Benchmarks for Low-Resource Language Understanding Evaluation in Bangla
Abhik Bhattacharjee, Tahmid Hasan, Kazi Samin Mubasshir et al.
BanglaGPT: A Generative Pretrained Transformer-Based Model for Bangla Language
Md. Shahidul Salim, Hasan Murad, Dola Das et al.
BenLLM-Eval: A Comprehensive Evaluation into the Potentials and Pitfalls of Large Language Models on Bengali NLP
M. Kabir, Mohammed Saidul Islam, Md Tahmid Rahman Laskar et al.
IndicLLMSuite: A Blueprint for Creating Pre-training and Fine-Tuning Datasets for Indian Languages
Mohammed Safi Ur Rahman Khan, Priyam Mehta, A. Sankar et al.
Khayyam Challenge (PersianMMLU): Is Your LLM Truly Wise to The Persian Language?
Omid Ghahroodi, Marzia Nouri, M. V. Sanian et al.
Bridging the Bosphorus: Advancing Turkish Large Language Models through Strategies for Low-Resource Language Adaptation and Benchmarking
Emre Can Acikgoz, Mete Erdogan, Deniz Yuret
xFinder: Large Language Models as Automated Evaluators for Reliable Evaluation
Qingchen Yu, Zifan Zheng, Shichao Song et al.
Lexically Grounded Subword Segmentation
Jindvrich Libovick'y, JindΕich Helcl
Testing prompt engineering methods for knowledge extraction from text
Fina Polat, Ilaria Tiddi, Paul Groth
Cited By (11)
Quantization Effects on Bangla Language Understanding in Large Language Models: A Systematic Evaluation
Creating Multilingual Mental Health Dialogue Datasets: Limits of Persona-Based Localization via Nationality and Language
Recent Advances in Bangla NLP: A Survey of Few-Shot, Zero-Shot, and Transfer Learning Techniques
Structural Silence: When AI Infrastructure Fails Speakers of Underrepresented Languages
Do Small Models Use the Law You Give Them? Context-Injected Fine-Tuning for Legal QA in Bangladesh
BenHalluEval: A Multi-Task Hallucination Evaluation Framework for Large Language Models on Bengali
Cross-Lingual Sentiment Misalignment: Auditing Multilingual Language Models for Inversion Risk, Dialectal Representation, and Affective Stability
UrduBench: An Urdu Reasoning Benchmark using Contextually Ensembled Translations with Human-in-the-Loop
GanitLLM: Difficulty-Aware Bengali Mathematical Reasoning through Curriculum-GRPO
PESCO-BERT: An Efficient Prompt-Based Contrastive Learning for Bangla News Classification
CUET_Expelliarmus at BLP2025 Task 2: Leveraging Instruction Translation and Refinement for Bangla-to-Python Code Generation with Open-Source LLMs