TituLLMs: A Family of Bangla LLMs with Comprehensive Benchmarking

Recent advances in Transformer-based models like GPT, BERT, and Llama have significantly advanced NLP, primarily for high-resource languages. However, low-resource languages such as Bangla lack sufficient digital corpora, limiting the development of effective models. Prior efforts include Zehady et al. (2024), who fine-tuned Llama on 12.4 million news articles, but their dataset remains small. The challenge lies in data scarcity, vocabulary limitations, and cultural knowledge gaps. Addressing these issues requires innovative data collection, synthetic data augmentation, and vocabulary expansion strategies. This research aims to build a dedicated Bangla language model, leveraging multi-source data and novel translation techniques, to bridge the gap between high-resource and low-resource NLP.

The core problem is the scarcity of large-scale, high-quality digital corpora for Bangla, which hampers the training of robust language models. Existing models rely heavily on transfer learning from multilingual models, but their performance remains suboptimal due to limited vocabulary coverage and cultural knowledge. Additionally, there is a lack of comprehensive benchmarks to evaluate model capabilities specifically in Bangla. Overcoming these bottlenecks involves developing extensive, culturally relevant datasets, optimizing vocabulary efficiency, and designing evaluation benchmarks that reflect real-world tasks. The challenge is further compounded by the need for models to understand complex reasoning and world knowledge within limited data environments.

This work introduces several innovations: 1) Vocabulary expansion through training multiple tokenizers with different sizes and merging them into the base model, significantly reducing tokens per word (TPW). 2) Multi-source data collection pipeline, including web scraping, book digitization with OCR, and synthetic data generation via translation and speech recognition, to create a diverse corpus of 37 billion tokens. 3) The Expressive Semantic Translation (EST) method, which refines translation quality for benchmark data, ensuring cultural and contextual relevance. 4) Development of five comprehensive benchmark datasets covering educational, commonsense, and world knowledge tasks, tailored for Bangla. These innovations collectively enable the creation of a high-performing, culturally adapted low-resource language model.

• �� Data collection: Query Common Crawl via Amazon Athena to extract Bangla web data, followed by filtering based on language and domain-specific rules, then deduplication using MinHash.
• �� Book digitization: Collect open-source Bangla books, apply OCR (Google OCR and Tesseract), and train document segmentation models (YOLO) to remove non-text elements.
• �� Synthetic data: Generate conversational data via crawled topics, translate English texts using NLLB-600M, and transcribe speech data with Bangla ASR.
• �� Vocabulary expansion: Train multiple tokenizers (32K-96K tokens), merge with Llama-3.2, optimize for token efficiency.
• �� Model training: Fine-tune Llama-3.2 models on the assembled dataset using the LlamaFactory platform, with context length 4096, one epoch (~1750 GPU hours).
• �� Evaluation: Develop five benchmark datasets, including Bangla MMLU, BoolQ, CommonsenseQA, OpenBookQA, and PIQA, for comprehensive performance assessment.

The experiments involve training two models (1B and 3B parameters) with different vocabulary sizes, then evaluating on five benchmark datasets. The datasets cover multiple domains: education (Bangla MMLU), commonsense reasoning (CommonsenseQA), factual knowledge (OpenBookQA), and physical reasoning (PIQA). Performance metrics include accuracy in zero-shot and few-shot settings, with comparisons against baseline models like Llama-3.2, Qwen, and Gemma. Ablation studies analyze the impact of vocabulary size, synthetic data, and training epochs. The evaluation setup ensures robustness across diverse tasks, with hyperparameters tuned for low-resource adaptation.

TituLLMs-3B achieved 0.25 accuracy on Bangla MMLU in 0-shot, outperforming prior multilingual models. In BoolQ, it scored 0.53 in 0-shot, indicating strong question-answering ability. The PIQA score reached 0.60, demonstrating effective physical commonsense reasoning. Vocabulary expansion reduced tokens per word (TPW), improving comprehension efficiency. Synthetic data, especially translation and transcription, significantly enhanced knowledge coverage. Although performance is promising, the models still lag in complex world knowledge tasks, indicating the need for larger datasets and multi-epoch training.

The models can be used for automated question answering, content generation, translation, and educational tools in Bengali-speaking regions. Their cultural and linguistic adaptation makes them suitable for local content creation, virtual assistants, and language learning applications. Future integration with multimodal data (images, speech) could further enhance their utility, enabling more natural human-computer interactions in low-resource settings.

The models were trained for only one epoch, limiting knowledge depth. Data scarcity restricts coverage of complex world facts. The models show room for improvement in reasoning and long-text understanding. Future work should include multi-epoch training, larger datasets, and multi-modal integration to address these limitations.