Quantization Effects on Bangla Language Understanding in Large Language Models: A Systematic Evaluation
Quantization effects on Bangla language: GPTQ-Int8 and GPTQ-Q8 maintain high accuracy on Bangla benchmarks.
Key Findings
Methodology
The study evaluates three LLM families (Qwen-2.5-7B, LLaMA-3.1-8B, GPT-OSS-20B) using post-training quantization across five Bangla NLU benchmarks. Three quantization formats (GPTQ-Int8, GPTQ-Q8, GGUF-W8A16) are compared using zero-shot evaluation.
Key Results
- GPT-OSS loses up to 57.35% accuracy on reasoning tasks under GGUF-W8A16.
- Qwen and LLaMA remain stable under GPTQ, with some quantized versions outperforming full precision.
- BoolQ-BN remains stable across all models and formats.
Significance
This study is the first to systematically evaluate the impact of quantization formats on Bangla NLU, filling a gap in low-resource language research. It shows that choosing the right architecture and quantization method is more important than bit width alone.
Technical Contribution
Provides the first comparative analysis of three quantization formats on Bangla NLU tasks, revealing performance differences across model families and offering practical guidance for deploying models in low-resource languages.
Novelty
First systematic comparison of quantization formats for Bangla NLU, highlighting significant performance differences in reasoning tasks across model families.
Limitations
- Only INT8-range quantization formats were evaluated, excluding INT4.
- Inference latency and memory usage were not measured.
- No quantized Bangla-native models were included.
Future Work
Future work could explore INT4 quantization, Bangla-native model quantization, and quantization-aware training to enhance robustness.
AI Executive Summary
This study explores the effects of post-training quantization on Bangla language understanding, particularly in low-resource language settings. By evaluating three LLM families (Qwen-2.5-7B, LLaMA-3.1-8B, GPT-OSS-20B) with quantization, it finds that GPTQ-Int8 and GPTQ-Q8 formats perform well on Bangla benchmarks, while GGUF-W8A16 performs poorly on reasoning tasks. The study shows that selecting the right model architecture and quantization method is more crucial than merely adjusting bit width.
Experimental results indicate that Qwen and LLaMA maintain accuracy post-quantization, whereas GPT-OSS suffers significant accuracy drops in reasoning tasks. BoolQ-BN remains stable across all models and formats, suggesting lower sensitivity to quantization in comprehension tasks.
This research provides practical guidance for deploying models in low-resource languages, emphasizing the critical impact of architecture and quantization method selection on model performance. Future research could explore lower bit-width quantization formats and quantization-aware training for Bangla-native models.
Deep Analysis
Background
As large language models become widely used in NLP tasks, their computational and storage demands increase. This is particularly challenging in low-resource languages like Bangla, where computational resources are limited. Post-training quantization is an effective model compression technique that reduces computational resource consumption without significantly degrading model performance.
Core Problem
Current quantization research focuses primarily on English benchmarks, leaving the impact on low-resource languages like Bangla unclear. Bangla's complex morphology and character composition rules may lead to representation degradation during quantization, affecting model performance.
Innovation
This study is the first to systematically evaluate the impact of quantization formats on Bangla NLU, revealing significant performance differences in reasoning tasks across model families. By comparing three quantization formats, the study provides practical guidance for deploying models in low-resource languages.
Methodology
- �� Select three LLM families: Qwen-2.5-7B, LLaMA-3.1-8B, GPT-OSS-20B.
- �� Use three quantization formats: GPTQ-Int8, GPTQ-Q8, GGUF-W8A16.
- �� Conduct zero-shot evaluation on five Bangla NLU benchmarks.
- �� Record accuracy changes for each model under different quantization formats.
Experiments
Experiments were conducted on five Bangla NLU benchmarks, including Bangla MMLU, CommonsenseQA-BN, OpenBookQA-BN, PIQA-BN, and BoolQ-BN. Each benchmark tested models in both full precision and quantized formats, using the lm-evaluation-harness tool for evaluation.
Results
Experimental results show that GPT-OSS loses up to 57.35% accuracy on reasoning tasks under GGUF-W8A16, while Qwen and LLaMA remain stable under GPTQ, with some quantized versions outperforming full precision. BoolQ-BN remains stable across all models and formats.
Applications
The study indicates that GPTQ-Int8 and GPTQ-Q8 formats are suitable for on-device deployment of Bangla language models, especially in resource-constrained environments. The stability of BoolQ-BN suggests its applicability in text-based fact retrieval applications.
Limitations & Outlook
The study only evaluated INT8-range quantization formats, excluding INT4. Inference latency and memory usage were not measured, and no Bangla-native models were included. Future research could explore lower bit-width quantization formats and quantization-aware training for Bangla-native models.
Plain Language Accessible to non-experts
Imagine you're cooking in a kitchen. Large language models are like chefs handling various ingredients (data) to create delicious dishes (outputs). Quantization is like using fewer spices to make the same tasty dish. For Bangla, the chef deals with more complex ingredient combinations (language structure), so we need to carefully choose spices (quantization methods) to ensure the dish's flavor remains unchanged.
ELI14 Explained like you're 14
Imagine you're playing a complex game where your character needs to find treasure on a big map. Large language models are like this character, with each point on the map being information. Quantization is like giving your character a smaller backpack. Even though the backpack is smaller, the character can still find the treasure if they choose their path wisely. For Bangla, the map is more complex, so the character needs to be smarter in choosing the path.
Glossary
Quantization
A technique to reduce model computational and storage demands by lowering weight precision.
Used to reduce memory footprint and inference time of large language models.
Post-Training Quantization
Quantization applied after model training, avoiding the complexity of quantization-aware training.
Used in this study to evaluate the impact on Bangla language understanding.
GPTQ
A quantization method using approximate second-order information to minimize layer reconstruction error.
Used for quantizing Qwen and LLaMA models in the study.
GGUF
A serialization format for CPU/GPU inference, supporting various internal bit depths.
Used for quantizing the GPT-OSS model in the study.
Bangla MMLU
A Bangla language multitask understanding benchmark covering multiple subjects.
Used to evaluate model performance on broad encyclopedic reasoning tasks.
Open Questions Unanswered questions from this research
- 1 How can Bangla language models be further compressed without losing accuracy?
- 2 How do Bangla's linguistic complexities affect quantization outcomes?
- 3 Are there quantization methods better suited for Bangla?
Applications
Immediate Applications
On-Device Deployment
Deploy Bangla language models on resource-constrained devices using GPTQ-Int8 or GPTQ-Q8 formats to maintain high accuracy.
Text Retrieval
Use quantized models in text retrieval applications, especially where quick responses are needed.
Long-term Vision
Low-Resource Language Support
Support more low-resource language NLP applications through quantization techniques, overcoming computational resource limitations.
Abstract
Post-training quantization lowers the memory footprint of Large Language Models (LLMs) and speeds up inference, which is why it is now common for on-device deployment. Most of what we know about its effects, however, comes from English benchmarks. It is not clear whether the same holds for morphologically complex, low-resource languages such as Bangla, and this gap is what we address here. We evaluate three model families---Qwen-2.5-7B, LLaMA-3.1-8B, and GPT-OSS-20B---in full precision and in three quantized formats (GPTQ-Int8, GPTQ-Q8, GGUF-W8A16) across five Bangla natural language understanding benchmarks (Bangla MMLU, CommonsenseQA-BN, OpenBookQA-BN, PIQA-BN, and BoolQ-BN), using zero-shot evaluation through lm-evaluation-harness. To our knowledge this is the first controlled comparison of quantization formats on Bangla NLU. The three families do not respond the same way: GPT-OSS loses up to 57.35% accuracy on reasoning-heavy tasks under GGUF-W8A16, while Qwen and LLaMA hold steady under GPTQ, and in a few cases the quantized version edges out the full-precision one. BoolQ-BN, a comprehension task, stays stable across all three families regardless of format. Taken together, these results suggest quantization can work well for Bangla deployment, but the choice of architecture and quantization method matters more than the bit width alone. We discuss what this means for practitioners choosing a model to run on constrained hardware.