BanglaBERT: Language Model Pretraining and Benchmarks for Low-Resource Language Understanding Evaluation in Bangla

Bangla, with over 300 million speakers, is among the most spoken languages worldwide, yet it remains underrepresented in NLP research due to scarce high-quality corpora and dedicated models. Early NLP efforts focused on rule-based and shallow machine learning approaches for tasks such as POS tagging and sentiment analysis, often limited by small datasets. The advent of deep learning and transformer architectures, exemplified by BERT, revolutionized NLP but heavily relied on large-scale pretraining data. For Bangla, the limited size of existing corpora, such as Wikipedia dumps (~650MB), hindered the training of effective language models. Recent efforts introduced multilingual models like mBERT and XLM-R, but their performance on Bangla-specific tasks was suboptimal, especially in low-resource settings. The lack of comprehensive benchmarks further impeded progress, making it difficult to evaluate and compare models systematically. This backdrop underscores the necessity for a dedicated Bangla corpus, a tailored model architecture, and a unified evaluation framework to propel Bangla NLP forward.

The core challenge lies in the scarcity of large-scale, high-quality Bangla datasets suitable for pretraining deep language models. Existing corpora are limited in size and diversity, restricting the models' ability to learn rich language representations. Multilingual models, while helpful, are often too large and computationally demanding, with diminished performance on Bangla-specific tasks. Moreover, the absence of standardized benchmarks hampers objective evaluation and comparison of models. Addressing these issues requires collecting extensive Bangla web data, designing efficient pretraining strategies, and establishing comprehensive evaluation protocols to facilitate progress in low-resource language NLP.

This work introduces several key innovations: 1) Construction of Bangla2B+ corpus—27.5GB of high-quality web data curated from 110 websites, enabling large-scale pretraining. 2) Adoption of ELECTRA's RTD objective, which improves training efficiency by replacing MLM with a discriminative task, reducing computational costs. 3) Development of BanglaBERT, a 110M parameter model tailored for Bangla, outperforming existing multilingual models on multiple tasks. 4) Creation of BanglishBERT, a bilingual model trained jointly on Bangla and English, supporting zero-shot transfer. 5) Establishment of BLUB, a multi-task benchmark covering sentiment, NLI, NER, and QA, providing a systematic evaluation platform for Bangla NLP. These innovations collectively address data scarcity, efficiency, and evaluation gaps in Bangla NLP.

• �� Data Collection: Crawled 110 popular Bangla websites, including news, blogs, social media, and encyclopedic sources, to assemble a diverse corpus of 27.5GB of text. • Data Cleaning: Removed duplicates, non-textual content, and filtered out non-Bangla pages using a language classifier. • Vocabulary Construction: Trained a 32k WordPiece subword vocabulary to handle code-switching and romanized content. • Preprocessing: Tokenized texts, segmented into sequences of up to 512 tokens, and created 7.18 million samples totaling 2.18 billion tokens. • Pretraining: Employed ELECTRA's RTD objective, training a generator and discriminator jointly, with the discriminator used for downstream fine-tuning. • Model Architecture: Built a 12-layer Transformer encoder with 768 hidden units, 12 attention heads, totaling 110M parameters. • Cross-lingual Model: Trained BanglishBERT on combined Bangla and English corpora with shared vocabulary, upsampling Bangla data to balance languages. • Downstream Tasks: Curated datasets for sentiment analysis, NLI, NER, and QA, translated and annotated to ensure quality, and performed supervised fine-tuning and evaluation.

The experimental setup involved fine-tuning BanglaBERT and BanglishBERT on four downstream tasks within the BLUB benchmark. Each model was trained for 3-20 epochs with batch size 32, tuning learning rates between 2e-5 and 5e-5 based on validation performance. The evaluation metrics included accuracy, macro-F1, micro-F1, and exact match (EM). Baseline comparisons involved models like mBERT, XLM-R (base and large), IndicBERT, and sahajBERT. Zero-shot transfer experiments assessed the models' ability to generalize from English to Bangla without task-specific fine-tuning. Sample efficiency was tested by limiting training data to fewer than 1,000 samples, demonstrating the models' robustness in data-scarce environments. The results were statistically validated using bootstrap sampling to confirm significance.

Results show that BanglaBERT achieved a BLUB score of 77.09, outperforming mBERT (70.41) and XLM-R (72.82), across all tasks. In sentiment classification, it reached 72.89% accuracy; in NLI, 82.80% accuracy; in NER, 77.78% micro-F1; and in QA, 72.63% EM. BanglishBERT demonstrated strong cross-lingual transfer, surpassing mBERT and XLM-R (base) by 15.8 and 10.8 points respectively. Sample efficiency tests indicated that with fewer than 1k samples, BanglaBERT outperformed XLM-R (large) by 2-9% on sentiment and NLI tasks, confirming its suitability for low-resource scenarios. These findings validate the effectiveness of the proposed corpus, architecture, and training strategy.

The models can be deployed in real-world applications such as Bangla social media monitoring, sentiment analysis for marketing, customer service chatbots, and information retrieval systems. They enable rapid development of NLP tools in Bangla, even with limited labeled data, facilitating industry adoption. Additionally, the zero-shot transfer capability allows leveraging English-trained models for Bangla tasks, reducing the need for extensive annotated datasets. Future applications include multilingual virtual assistants, educational tools, and automated content moderation, all benefiting from the models' efficiency and accuracy.

Despite promising results, the models are trained on web data that may contain biases, offensive content, or noise, affecting fairness and safety. Their performance on dialectal, regional, or domain-specific language remains to be validated, requiring further fine-tuning. Computational costs, although optimized, are still significant, posing challenges for deployment on edge devices. The current benchmark covers only four tasks; expanding to more complex NLP tasks like dependency parsing or semantic role labeling is necessary. Future work should focus on bias mitigation, domain adaptation, and model compression to enhance practical usability.