BenLLMEval: A Comprehensive Evaluation into the Potentials and Pitfalls of Large Language Models on Bengali NLP
This study evaluates GPT-3.5, LLaMA-2-13b-chat, and Claude-2 on seven Bengali NLP tasks in zero-shot settings, revealing significant performance gaps compared to fine-tuned models.
Key Findings
Methodology
This research employs the BenLLM-Eval benchmark, which encompasses seven critical NLP tasks in Bengali, including text summarization, question answering, paraphrasing, natural language inference, transliteration, text classification, and sentiment analysis. Zero-shot evaluation is conducted using three prominent LLMs: GPT-3.5, LLaMA-2-13b-chat, and Claude-2, with carefully crafted prompts tailored for each task. Performance metrics such as ROUGE, BLEU, accuracy, and F1 are used, complemented by human-in-the-loop assessments to ensure fairness. The evaluation also incorporates task contamination detection mechanisms—using example extraction and membership inference—to verify that models did not have prior exposure to test data. This comprehensive approach provides a nuanced understanding of the models’ capabilities and limitations in a low-resource language context.
Key Results
- In text summarization, GPT-3.5 and Claude-2 achieved ROUGE-1 scores of 20.19 and 20.79 respectively, which are below the SOTA fine-tuned mT5 model (28.32). LLaMA-2-13b-chat performed poorly with a ROUGE-1 of only 0.41, often producing summaries in English; translation of outputs improved scores significantly. In question answering, GPT-3.5 attained an F1 of 78.67, close to SOTA, but with low exact match accuracy (44.85%), indicating diverse but imprecise responses. Claude-2 scored similarly, while LLaMA-2-13b-chat failed to generate meaningful answers.
- Natural Language Inference results showed GPT-3.5 outperforming others with 52.71% accuracy on BNLI, yet still far from the 87.6% of fine-tuned BanglaBERT. LLaMA-2-13b-chat struggled with neutral relations, and Claude-2 underperformed on entailment. In transliteration, GPT-3.5 achieved the lowest character error rate (CER) of 18.1% and sentence error rate (WER) of 60.6%, outperforming other LLMs but still far from state-of-the-art models. Text classification results revealed GPT-3.5 and Claude-2 with roughly 48% accuracy, much lower than the 87.6% of XLM-R, emphasizing the importance of prompt design. Sentiment analysis showed GPT-3.5 reaching 90.20% accuracy on IndicSentiment, setting new benchmarks, with Claude-2 close behind.
- Task contamination analysis suggested that GPT-3.5 might have seen some test data during training, but overall, the evaluation was robust. Other models showed minimal evidence of prior exposure, validating the results. Overall, the study highlights the significant performance gap between large closed-source models and fine-tuned models in low-resource Bengali NLP tasks, emphasizing the need for further research into multilingual training and data augmentation.
- This is the first comprehensive evaluation of large language models on Bengali, providing critical insights into their capabilities and limitations. The findings serve as a benchmark for future research, guiding the development of more effective multilingual models tailored for low-resource languages. The study underscores the importance of diverse training data, prompt engineering, and evaluation strategies to improve LLM performance in underrepresented languages.
Significance
This research underscores the challenges faced by large language models when applied to low-resource languages like Bengali, which has over 300 million native speakers. The findings highlight the necessity of expanding multilingual training datasets and developing language-specific adaptation techniques. By systematically evaluating models across multiple NLP tasks, the study offers valuable benchmarks that inform both academia and industry about the current state and future directions of multilingual NLP. It also emphasizes that zero-shot capabilities, while promising, are insufficient for many practical applications in underrepresented languages, calling for more targeted data collection, model fine-tuning, and prompt optimization. The broader impact lies in fostering equitable AI development, ensuring that speakers of diverse languages benefit from advances in NLP technology.
Technical Contribution
This work introduces the BenLLM-Eval framework, integrating multi-task performance assessment with rigorous contamination detection, thus setting a new standard for evaluating LLMs in low-resource languages. The design of tailored prompts for each task ensures consistency and fairness, while the contamination analysis—via example extraction and membership inference—addresses concerns about data leakage. The comparative analysis of GPT-3.5, LLaMA-2-13b-chat, and Claude-2 provides a detailed understanding of their strengths and weaknesses in Bengali, revealing that open-source models lag significantly behind closed-source counterparts in zero-shot scenarios. The methodology and insights contribute to the broader field of multilingual NLP, guiding future model training and evaluation strategies.
Novelty
This is the first systematic evaluation of large language models’ zero-shot performance specifically in Bengali, a low-resource language. The study’s novelty lies in its comprehensive, multi-task benchmark combined with contamination detection, providing a nuanced understanding of model capabilities beyond traditional metrics. Unlike prior work focusing mainly on English or European languages, this research emphasizes the challenges and opportunities in underrepresented languages, offering a new perspective on the generalizability and fairness of LLMs across diverse linguistic contexts.
Limitations
- The training datasets for GPT-3.5 and Claude-2 are proprietary, making it impossible to confirm whether the models have seen test data, which could bias the evaluation results. Although contamination detection was employed, absolute certainty remains elusive.
- The evaluation is limited to zero-shot settings; the potential improvements from fine-tuning or few-shot learning are not explored, which could significantly enhance performance.
- The quality and quantity of Bengali training data remain limited, constraining the models’ ability to learn language-specific nuances, especially for complex tasks like NLI and summarization.
- The study only considers three models, and future work should include more open-source and commercial models to broaden understanding.
- Automated metrics like ROUGE and BLEU, while useful, do not fully capture the qualitative aspects of language understanding, necessitating human evaluations for comprehensive assessment.
Future Work
未来将扩大评估范围，涵盖更多低资源语言和任务类型，探索多语言联合训练和微调策略。计划引入更丰富的提示工程技术，提升模型在孟加拉语中的表现。还将结合人类反馈机制，优化模型生成的真实性和可靠性。此外，研究将关注模型在实际应用中的适应性，如孟加拉语聊天机器人和内容生成系统，推动多语言AI的普及和公平性发展。
AI Executive Summary
In recent years, large language models (LLMs) have revolutionized natural language processing, demonstrating remarkable capabilities in understanding and generating human language. These models, trained on vast datasets, have achieved state-of-the-art results across a wide range of NLP tasks, primarily in English. However, their performance in low-resource languages like Bengali remains under-explored, despite the language’s significant global speaker base of over 300 million native speakers.
This study introduces BenLLM-Eval, a comprehensive benchmark designed to evaluate the zero-shot performance of prominent LLMs—GPT-3.5, LLaMA-2-13b-chat, and Claude-2—across seven diverse Bengali NLP tasks. These tasks include text summarization, question answering, paraphrasing, natural language inference, transliteration, text classification, and sentiment analysis. The evaluation employs carefully crafted prompts to ensure consistency and fairness, supplemented by human-in-the-loop assessments to verify the accuracy of responses.
Results reveal that, in some cases, GPT-3.5 and Claude-2 perform on par with or better than current fine-tuned models, especially in sentiment analysis and summarization. However, in most tasks, especially for open-source models like LLaMA-2-13b-chat, performance is significantly lower, with many outputs being inaccurate or in English. For example, LLaMA-2-13b-chat’s ROUGE-1 score in summarization was only 0.41, and its outputs often lacked Bengali language fidelity. In question answering, GPT-3.5 achieved an F1 score of 78.67, but with low exact match accuracy, indicating diverse responses that are not always precise.
The evaluation also uncovered issues related to task contamination, where some models might have been exposed to test data during training, especially GPT-3.5. Despite these challenges, the study provides valuable insights into the current limitations of large models in low-resource settings. It emphasizes the need for more diverse multilingual training data, improved prompt engineering, and targeted fine-tuning to bridge the performance gap.
Overall, this work marks a significant step toward understanding and improving LLMs for underrepresented languages. It offers a benchmark for future research, highlighting the importance of inclusive AI development. The findings suggest that while large models hold promise, substantial work remains to adapt them effectively for languages like Bengali, ensuring equitable access to advanced NLP technologies for all language communities.
Deep Analysis
Background
近年来，大规模预训练语言模型（LLMs）在自然语言处理（NLP）领域引发了革命性变革。从早期的Word2Vec、GloVe到Transformer架构的出现，再到BERT、GPT系列的广泛应用，模型的规模和能力不断提升。这些模型通过在海量多语种、多任务数据上的训练，展现出强大的语言理解和生成能力。代表性工作包括OpenAI的GPT-3、Google的T5、Meta的LLaMA等，它们在文本生成、问答、摘要等任务中取得了突破性进展。然而，绝大多数研究集中在英语和少数资源丰富的语言，对于像孟加拉语这样资源匮乏的语言，模型表现仍然有限，缺乏系统性评估和优化。近年来，针对低资源语言的研究逐渐增多，但仍面临数据不足、模型偏差和公平性等挑战。
Core Problem
孟加拉语作为世界第六大使用人口的语言，拥有超过3亿母语使用者，但在NLP资源方面严重不足。现有的预训练模型如BanglaBERT、BanglaT5等，虽然在特定任务上取得一定进展，但大多依赖微调，且在零-shot或少-shot场景下表现有限。由于缺乏大规模、多任务、多场景的评估体系，难以全面理解模型在实际应用中的能力和局限性。此外，低资源环境下模型的泛化能力不足，难以满足实际需求，如自动翻译、内容生成、聊天机器人等。如何在有限资源条件下提升模型性能，成为亟待解决的问题。
Innovation
本研究的核心创新在于提出了BenLLM-Eval评估框架，结合多任务、多指标的性能评估机制，首次系统性地评估了GPT-3.5、LLaMA-2-13b-chat和Claude-2在孟加拉语中的零-shot能力。具体创新包括：• 设计了针对每个任务的专用提示（prompt），确保评估的公平性和一致性；• 引入任务污染检测机制，通过示例提取和成员推断，验证模型是否提前见过测试数据；• 综合多任务、多指标评估，提供全面的模型能力画像。这些创新为低资源语言模型评估提供了新的方法论，也为未来多语言、多任务模型的开发提供了数据基础。
Methodology
- �� 任务定义：选择文本摘要、问答、改写、自然语言推理、转写、文本分类和情感分析七个关键任务。• 数据集准备：采用XL-Sum、SQuAD_Bangla、IndicParaphrase、BNLI、Dakshina、Soham新闻分类、IndicSentiment和SentNoB等公开数据集。• 提示设计：为每个任务设计专门的提示（prompt），确保模型理解任务目标。• 模型选择：使用GPT-3.5、LLaMA-2-13b-chat和Claude-2，进行零-shot推理。• 评估指标：采用ROUGE、BLEU、准确率、F1等指标，结合人工评估确保结果可靠。• 任务污染检测：利用示例提取（TEE）和成员推断（Membership Inference）机制，检测模型是否提前见过测试数据。• 实验流程：输入任务提示和测试样本，模型生成响应，自动和人工评估性能，记录结果并分析。
Experiments
实验设计包括多任务、多数据集的评估，确保模型在不同场景下的表现。每个任务都设计了标准化的prompt，确保公平性。采用自动指标（ROUGE、BLEU、准确率、F1）和人工验证相结合的方法，评估模型输出的质量。对比微调的SOTA模型，分析零-shot模型的优势和不足。引入任务污染检测，确保模型未提前见过测试数据。通过对模型输出的详细分析，揭示其在低资源环境中的表现差异和潜在偏差。实验还包括对模型输出的语言分析，验证其在孟加拉语中的语言忠实度和语义准确性。
Results
实验结果显示，GPT-3.5在多个任务中表现优异，尤其在情感分析和摘要任务中，达到了或超过了微调模型的性能，但在问答和推理任务中仍存在明显差距。LLaMA-2-13b-chat表现极差，许多输出为英语，且在关键任务中的准确率远低于预期。Claude-2在问答和推理任务中表现较好，但在摘要和转写任务中仍有提升空间。模型在任务污染检测中表现出一定的潜在泄露风险，尤其是GPT-3.5，提示未来需要加强数据隐私保护。整体来看，模型在孟加拉语中的零-shot能力有限，表现受限于训练数据的多样性和质量。
Applications
本研究的评估结果为孟加拉语自动化应用提供了理论基础，包括智能聊天机器人、内容生成、自动摘要和多语言翻译系统。通过优化提示设计和微调策略，可以提升模型在实际场景中的表现。未来，结合人类反馈和持续学习，模型有望实现更高的准确性和鲁棒性，为孟加拉语用户提供更智能、更贴心的服务。此外，研究成果也可推广到其他低资源语言，推动多语言AI的公平发展。
Limitations & Outlook
模型训练数据的保密性限制了对训练集的完全了解，存在潜在的任务污染风险。评估仅在零-shot场景下进行，未充分利用微调或少-shot策略，未来应结合多样化训练方式。模型在孟加拉语中的表现受限于训练数据的质量和规模，尤其在复杂任务如推理和摘要中表现不足。实验只涉及三款模型，未来应引入更多开源和商业模型进行对比。自动指标虽有参考价值，但难以全面反映语言理解能力，需结合人类评价进行优化。未来还需探索多任务联合训练、提示工程改进和多模态融合等方向，以提升模型的实用性和公平性。
Plain Language Accessible to non-experts
想象一下，你在一家大型厨房里工作，负责准备各种菜肴。每次你需要做一道新菜时，你可以从食谱、经验和厨房里的其他厨师那里学习。大规模语言模型（LLMs）就像这个厨房里的厨师，它们通过阅读大量的书籍、文章和对话，学会了如何“做菜”。但问题是，这些厨师主要是在英语的厨房里学会的，孟加拉语的菜谱很少，厨师们在这方面就显得不那么擅长。
这项研究就像是在测试不同厨师（模型）在孟加拉语厨房里的表现。研究者给他们一些任务，比如写摘要、回答问题、改写句子、判断句子之间的关系、把词从拉丁字母转成孟加拉字母、分类新闻、分析情感等。每个任务都像一道菜，研究者用特别设计的“提示”告诉厨师们要做什么，然后观察他们的表现。
结果发现，虽然一些厨师（如GPT-3.5）在某些菜肴上表现不错，甚至比专门训练过的厨师还好，但大多数厨师在孟加拉语菜肴上都表现不好，尤其是开源的厨师（LLaMA-2-13b-chat），表现得非常差。有的厨师甚至会用英语回答，完全不符合孟加拉语的要求。研究还发现，有些厨师可能在训练中偷偷看过测试菜谱，这会影响评估的公平性。
这就像是在厨房里，厨师们需要不断学习和练习，才能做出美味的菜肴。研究提醒我们，要让这些“厨师”在孟加拉语厨房里表现得更好，就需要提供更多的孟加拉语食谱和训练材料，同时设计更聪明的提示，帮助他们理解任务。未来，还要开发更智能的厨师，让他们能在不同的厨房里都能做出好菜，为孟加拉语使用者带来更多便利。
ELI14 Explained like you're 14
想象你在学校里，有很多不同的老师教你不同的科目。有的老师用英语讲课，有的用孟加拉语。有时候，你需要用孟加拉语写作文、回答问题或者总结一篇文章。可是，如果老师只会用英语教你，你就很难学会用孟加拉语做这些事情。这就像是大规模的语言模型（LLMs），它们通过阅读很多书和文章学会了怎么“说话”。但大部分模型都是用英语训练的，孟加拉语就像是少有人问津的那门课，模型在这方面就不太擅长。
这项研究就像是在测试几位“超级老师”——GPT-3.5、LLaMA-2-13b-chat和Claude-2，在孟加拉语环境下的表现。研究者给他们一些任务，比如写文章摘要、回答问题、改写句子、判断句子之间的关系、把词从拉丁字母转成孟加拉字母、分类新闻、分析情感等。每个任务都像是不同的考试题，研究者用特别设计的提示（prompt）告诉他们要做什么，然后观察他们的答案。
结果显示，虽然GPT-3.5在某些任务上表现不错，甚至比一些专门训练的模型还好，但大多数模型在孟加拉语任务中表现很差。有的模型甚至用英语回答，完全没有用孟加拉语。研究还发现，有些模型可能在训练时偷偷看过测试题，这样会影响公平性。总的来说，要让这些“老师”在孟加拉语的课堂上表现得更好，就需要给他们更多的孟加拉语资料和更聪明的提示，让他们理解任务的意思。未来，还要开发更厉害的模型，让他们在不同的语言环境中都能表现出色，帮助更多孟加拉语的用户。
Abstract
Large Language Models (LLMs) have emerged as one of the most important breakthroughs in NLP for their impressive skills in language generation and other language-specific tasks. Though LLMs have been evaluated in various tasks, mostly in English, they have not yet undergone thorough evaluation in under-resourced languages such as Bengali (Bangla). To this end, this paper introduces BenLLM-Eval, which consists of a comprehensive evaluation of LLMs to benchmark their performance in the Bengali language that has modest resources. In this regard, we select various important and diverse Bengali NLP tasks, such as text summarization, question answering, paraphrasing, natural language inference, transliteration, text classification, and sentiment analysis for zero-shot evaluation of popular LLMs, namely, GPT-3.5, LLaMA-2-13b-chat, and Claude-2. Our experimental results demonstrate that while in some Bengali NLP tasks, zero-shot LLMs could achieve performance on par, or even better than current SOTA fine-tuned models; in most tasks, their performance is quite poor (with the performance of open-source LLMs like LLaMA-2-13b-chat being significantly bad) in comparison to the current SOTA results. Therefore, it calls for further efforts to develop a better understanding of LLMs in modest-resourced languages like Bengali.
References (20)
XL-Sum: Large-Scale Multilingual Abstractive Summarization for 44 Languages
Tahmid Hasan, Abhik Bhattacharjee, Md Saiful Islam et al.
A Systematic Study and Comprehensive Evaluation of ChatGPT on Benchmark Datasets
Md Tahmid Rahman Laskar, M Saiful Bari, Mizanur Rahman et al.
SentNoB: A Dataset for Analysing Sentiment on Noisy Bangla Texts
Khondoker Ittehadul Islam, Sudipta Kar, Md. Saiful Islam et al.
Language Models are Few-Shot Learners
Tom B. Brown, Benjamin Mann, Nick Ryder et al.
IndicXTREME: A Multi-Task Benchmark For Evaluating Indic Languages
Sumanth Doddapaneni, Rahul Aralikatte, Gowtham Ramesh et al.
A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference
Adina Williams, Nikita Nangia, Samuel R. Bowman
IndicNLG Benchmark: Multilingual Datasets for Diverse NLG Tasks in Indic Languages
Aman Kumar, Himani Shrotriya, P. Sahu et al.
BanglaBERT: Language Model Pretraining and Benchmarks for Low-Resource Language Understanding Evaluation in Bangla
Abhik Bhattacharjee, Tahmid Hasan, Kazi Samin Mubasshir et al.
Extract with Order for Coherent Multi-Document Summarization
Mir Tafseer Nayeem, Yllias Chali
BanglaRQA: A Benchmark Dataset for Under-resourced Bangla Language Reading Comprehension-based Question Answering with Diverse Question-Answer Types
Syed Mohammed Sartaj Ekram, Adham Arik Rahman, Mangera Altaf et al.
iNLPSuite: Monolingual Corpora, Evaluation Benchmarks and Pre-trained Multilingual Language Models for Indian Languages
Divyanshu Kakwani, Anoop Kunchukuttan, S. Golla et al.
[Workshop].
Susann Fischer, M. Marchis, Mario Navarro
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Jacob Devlin, Ming-Wei Chang, Kenton Lee et al.
Conference on Empirical Methods in Natural Language Processing EMNLP 2016
Zita Marinho, A. Martin, Shay B. Cohen et al.
Learning from 26 Languages: Program Management and Science in the Babel Program
M. Harper
Paraphrastic Fusion for Abstractive Multi-Sentence Compression Generation
Mir Tafseer Nayeem, Yllias Chali
Abstractive Unsupervised Multi-Document Summarization using Paraphrastic Sentence Fusion
Mir Tafseer Nayeem, Tanvir Ahmed Fuad, Yllias Chali
XNLI: Evaluating Cross-lingual Sentence Representations
Alexis Conneau, Guillaume Lample, Ruty Rinott et al.
RoBERTa: A Robustly Optimized BERT Pretraining Approach
Yinhan Liu, Myle Ott, Naman Goyal et al.
Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism
M. Shoeybi, M. Patwary, Raul Puri et al.
Cited By (20)
BdSentiLLM: A Novel LLM Approach to Sentiment Analysis of Product Reviews
bnContextQA: Benchmarking Long-Context Question Answering and Challenges in Bangla
ConVerSum: A Contrastive Learning-Based Approach for Data-Scarce Solution of Cross-Lingual Summarization Beyond Direct Equivalents
BeliN: A Novel Corpus for Bengali Religious News Headline Generation using Contextual Feature Fusion
TituLLMs: A Family of Bangla LLMs with Comprehensive Benchmarking
Performance Evaluation of Large Language Models in Bangla Consumer Health Query Summarization
Empowering Regional Language: NLP-driven Conversion from Rangpur Dialect to Standard Bengali Language
Mitigating Extrinsic Gender Bias for Bangla Classification Tasks
DiaFrame: A Framework for Understanding Bengali Dialects in Human-AI Collaborative Creative Writing Spaces
Better to Ask in English: Evaluation of Large Language Models on English, Low-resource and Cross-Lingual Settings
Do Large Language Models Speak All Languages Equally? A Comparative Study in Low-Resource Settings
A Systematic Survey and Critical Review on Evaluating Large Language Models: Challenges, Limitations, and Recommendations
Too Late to Train, Too Early To Use? A Study on Necessity and Viability of Low-Resource Bengali LLMs
Decoding the Diversity: A Review of the Indic AI Research Landscape
Unraveling the Dominance of Large Language Models Over Transformer Models for Bangla Natural Language Inference: A Comprehensive Study
Harnessing Large Language Models Over Transformer Models for Detecting Bengali Depressive Social Media Text: A Comprehensive Study
Generation, Analysis, and Detection of LLM Manipulated Bangla News
TeamB2B at BLP-2025 Task 2: BanglaForge: LLM Collaboration with Self-Refinement for Bangla Code Generation
AlphaBorno at BLP-2025 Task 2: Code Generation with Structured Prompts and Execution Feedback
The Anoa-L01 Benchmark: Prompt-Based Zero-Shot Evaluation for Sulawesi’s Regional Languages Detection in LLMs