Unraveling the Dominance of Large Language Models Over Transformer Models for Bangla Natural Language Inference: A Comprehensive Study

Recent advances in NLP have been driven by pre-trained models like BERT, ALBERT, and mBERT, which have achieved state-of-the-art results in many languages. However, low-resource languages such as Bangla face challenges due to limited annotated datasets and linguistic diversity. Prior works (e.g., Mohsinul et al.) evaluated Bengali-specific models, but their scope was limited to certain tasks like sentiment analysis and summarization. Large language models (LLMs) like GPT-3.5 Turbo and Gemini 1.5 Pro, trained on massive multilingual corpora, show promise for zero-shot and few-shot learning, yet their performance in Bengali NLP remains underexplored. This study aims to fill this gap by systematically assessing LLMs on the NLI task, a core component of natural language understanding, in the Bengali context.

The core challenge lies in enabling low-resource languages like Bangla to benefit from the advances in LLMs. Traditional models require extensive labeled data for fine-tuning, which is scarce for Bengali. Moreover, LLMs tend to hallucinate or generate unreliable outputs, especially in languages with limited training data. The key issues include poor zero-shot performance, high hallucination rates, and difficulty in capturing nuanced linguistic features. Addressing these problems is crucial for deploying reliable NLP systems in Bengali, impacting applications like automated translation, sentiment analysis, and information retrieval in underserved communities.

This work introduces a novel evaluation framework combining zero-shot and few-shot learning for Bengali NLI, leveraging prompt engineering tailored to Bengali syntax and semantics. It compares the performance of GPT-3.5 Turbo and Gemini 1.5 Pro against Bengali pretrained models, revealing that large models excel in few-shot scenarios, surpassing fine-tuned models. The study also analyzes hallucination mechanisms, proposing prompt design strategies to mitigate false outputs. Additionally, it pioneers the use of automated prompt optimization techniques for low-resource languages, setting a new benchmark for multilingual NLP research.

• �� Data preprocessing: Normalize Bengali text, handle Unicode, remove extraneous spaces, ensure input consistency.
• �� Model setup: Load GPT-3.5 Turbo, Gemini 1.5 Pro, and Bengali PLMs like BanglaBERT.
• �� Training: Use AdamW optimizer, set learning rate to 0.001, batch size 16, for 15 epochs.
• �� Prompt design: Create task-specific prompts for NLI, incorporating premise-hypothesis pairs.
• �� Evaluation: Measure accuracy, F1, precision, recall across zero-shot and few-shot (5, 10, 15) scenarios.
• �� Analysis: Examine hallucination frequency, prompt effectiveness, and model robustness.

Experiments utilize the Bengali subset of XNLI with 4,895 test samples. Models are evaluated under zero-shot and few-shot conditions, with 5, 10, and 15 examples per prompt. Hyperparameters are standardized for fairness. Multiple runs ensure stability. The focus is on comparing large models’ transfer learning ability against Bengali-specific models, analyzing performance metrics and hallucination rates. The study also tests different prompt formulations to optimize inference accuracy, providing comprehensive insights into model capabilities and limitations in low-resource settings.

GPT-3.5 Turbo achieves 92.05% accuracy in few-shot, outperforming BanglaBERT (82.04%). Gemini 1.5 Pro scores 91.46%, but with higher hallucination rates. Few-shot learning boosts performance dramatically, surpassing fine-tuned models. Hallucination issues persist, especially in complex inference, indicating the need for better prompt strategies. Results confirm that large models can effectively transfer knowledge to Bengali, but reliability issues remain, requiring further research.

The findings support deploying LLMs for Bengali NLP tasks such as automated customer support, educational tools, and multilingual chatbots. These models enable high-quality inference with minimal labeled data, reducing resource barriers. Long-term, integrating multimodal data and developing robust prompt techniques could revolutionize low-resource language AI, making technology accessible to underserved populations and fostering linguistic diversity in AI applications.

Hallucination remains a major challenge, affecting trustworthiness. Limited Bengali training data constrains model understanding of linguistic nuances. Variability in inference quality suggests the need for advanced prompt engineering. Computational costs of large models are high, limiting real-time deployment. Future work must address factual accuracy, dataset expansion, and model interpretability to ensure practical usability.