Performance Evaluation of Large Language Models in Bangla Consumer Health Query Summarization
Evaluated nine large language models' zero-shot performance on Bengali health query summarization; Mixtral outperformed others in ROUGE-1 and ROUGE-L.
Key Findings
Methodology
Using the BanglaCHQ-Summ dataset with 2350 question-answer pairs, nine state-of-the-art models (e.g., GPT-4, Llama3-70B, Mixtral) were evaluated in zero-shot settings. Prompt engineering ensured concise, focused summaries. ROUGE-1, ROUGE-2, and ROUGE-L metrics quantified performance, revealing Mixtral-8x22B-Instruct's superiority in ROUGE-1 and ROUGE-L, while Bangla T5 excelled in ROUGE-2. The study demonstrated that large pre-trained models can rival fine-tuned models without task-specific training, highlighting their potential in low-resource language applications.
Key Results
- Mixtral-8x22B-Instruct achieved ROUGE-1 score of 51.36 and ROUGE-L of 49.17, surpassing Bangla T5 (50.05 and 48.35), indicating strong zero-shot summarization capability.
- GPT-4 scored 49.74 in ROUGE-1 but lagged in ROUGE-2 (15.26), showing limitations in bi-gram coverage despite high overall fluency.
- Average summary length ranged from 19 to 37 words, with Mixtral producing concise summaries close to reference length (26 words), demonstrating efficiency.
Significance
This research underscores the capability of large-scale pre-trained models to perform effective abstractive summarization in low-resource languages without extensive fine-tuning. It addresses a critical bottleneck in deploying AI for healthcare in underrepresented languages, enabling scalable, cost-effective solutions. The findings suggest that with proper prompt design, zero-shot models can deliver high-quality summaries, facilitating faster medical responses and better health information dissemination. This work paves the way for broader application of multilingual AI in public health, especially where annotated data is scarce.
Technical Contribution
The study provides a comprehensive evaluation framework combining multiple ROUGE metrics to assess zero-shot performance across diverse models. It introduces prompt optimization strategies tailored for Bengali, demonstrating significant performance gains. The comparative analysis between fine-tuned and zero-shot models offers insights into the evolving landscape of NLP, highlighting the diminishing gap and potential for scalable deployment. Additionally, the research analyzes factors like output length and phonetic variations impacting model accuracy, contributing to the understanding of multilingual model robustness.
Novelty
This is the first systematic evaluation of multiple large models on Bengali health query summarization under zero-shot conditions, emphasizing low-resource language challenges. Unlike prior work focused on English, this study validates the generalization ability of pre-trained models without task-specific fine-tuning, revealing promising results that challenge traditional reliance on domain-specific datasets. The integration of prompt engineering and multi-metric evaluation sets a new standard for low-resource NLP research.
Limitations
- Models showed limited bi-gram overlap (ROUGE-2 scores), indicating challenges in capturing local coherence essential for clinical decision-making.
- Some models, like Athene-70B, underperformed, highlighting the importance of model scale and training strategies for low-resource adaptation.
- Evaluation relied solely on ROUGE metrics, which may not fully capture semantic accuracy or medical relevance; human evaluation is needed for comprehensive validation.
Future Work
Future efforts will focus on fine-tuning models with domain-specific data to improve bi-gram and semantic accuracy. Exploring multi-modal inputs, such as combining text with speech or images, could enhance understanding of spoken queries. Developing robust models resilient to phonetic and dialectal variations will be prioritized. Additionally, expanding datasets and evaluation metrics, including human assessments, will be essential to validate practical utility. Long-term, integrating these models into real-world healthcare systems can transform low-resource language medical services, making them more accessible and reliable.
AI Executive Summary
The rapid growth of online health consultation platforms has led to an explosion of consumer health queries (CHQs), especially in low-resource languages like Bengali. These queries often contain extraneous details, making manual processing time-consuming and inefficient. Traditional NLP solutions, heavily reliant on large annotated datasets for fine-tuning, are limited in such contexts due to data scarcity. This study explores the potential of large pre-trained language models (LLMs) in addressing this challenge through zero-shot summarization. Evaluating nine models—including GPT-4, Llama3-70B, Mixtral, and others—on the BanglaCHQ-Summ dataset, the research demonstrates that models like Mixtral significantly outperform the baseline fine-tuned Bangla T5 in key ROUGE metrics. The results reveal that with optimized prompting, zero-shot models can generate concise, relevant summaries that rival or surpass fine-tuned counterparts, even in complex medical domains. This finding underscores the transformative potential of pre-trained LLMs for low-resource language NLP tasks, especially in healthcare, where rapid, accurate information extraction is critical. The study also highlights factors influencing performance, such as output length and phonetic variability, providing insights for future model improvements. Despite these advances, challenges remain in capturing nuanced medical semantics and ensuring robustness across dialects. The work points toward future directions involving domain adaptation, multi-modal integration, and broader evaluation frameworks. Overall, this research paves the way for scalable, accessible AI-driven healthcare solutions in underrepresented languages, promising to enhance global health equity and resource efficiency.
Deep Dive
Abstract
Consumer Health Queries (CHQs) in Bengali (Bangla), a low-resource language, often contain extraneous details, complicating efficient medical responses. This study investigates the zero-shot performance of nine advanced large language models (LLMs): GPT-3.5-Turbo, GPT-4, Claude-3.5-Sonnet, Llama3-70b-Instruct, Mixtral-8x22b-Instruct, Gemini-1.5-Pro, Qwen2-72b-Instruct, Gemma-2-27b, and Athene-70B, in summarizing Bangla CHQs. Using the BanglaCHQ-Summ dataset comprising 2,350 annotated query-summary pairs, we benchmarked these LLMs using ROUGE metrics against Bangla T5, a fine-tuned state-of-the-art model. Mixtral-8x22b-Instruct emerged as the top performing model in ROUGE-1 and ROUGE-L, while Bangla T5 excelled in ROUGE-2. The results demonstrate that zero-shot LLMs can rival fine-tuned models, achieving high-quality summaries even without task-specific training. This work underscores the potential of LLMs in addressing challenges in low-resource languages, providing scalable solutions for healthcare query summarization.