BenHalluEval: A Multi-Task Hallucination Evaluation Framework for Large Language Models on Bengali
BenHalluEval evaluates Bengali LLM hallucinations; BenHalluScore ranges from 7.72% to 55.42%.
Key Findings
Methodology
BenHalluEval framework evaluates Bengali LLM hallucinations using a dual-track protocol across four tasks: QA, Bangla-English Code-Mixed QA, Summarization, and Reasoning. 12,000 hallucinated candidates generated using GPT-5.4 are assessed across seven LLMs.
Key Results
- In QA, GPT-4.1 mini achieves the best BenHalluScore at 15.56%, while Mistral-nemo-12B scores the worst at 53.84%.
- In Bangla-English Code-Mixed QA, Qwen2.5-32B shows significant improvement with a BenHalluScore of 24.00%.
- TigerLLM-9B outperforms larger multilingual models in Bengali-specific tasks with a BenHalluScore of 28.28%.
Significance
BenHalluEval is the first dedicated hallucination benchmark for Bengali, addressing gaps in low-resource language evaluation. It reveals inadequacies of single-track and prompting-only approaches.
Technical Contribution
Introduces BenHalluScore as a dual-track calibration metric, balancing error rates and preventing uniform response bias. Provides a new evaluation standard for hallucination detection in low-resource languages.
Novelty
BenHalluEval is the first hallucination detection benchmark specifically for Bengali, covering both native-script Bengali and Bangla-English code-mixed input.
Limitations
- Certain models exhibit systematic biases in hallucination detection, affecting accuracy.
- Hallucination evaluation in low-resource languages still faces data scarcity issues.
Future Work
Future research could explore hallucination detection in more low-resource languages and improve model performance on Bangla-English code-mixed input.
AI Executive Summary
BenHalluEval is the first framework designed specifically for hallucination detection in Bengali, filling a gap in low-resource language evaluation. The framework evaluates language models' hallucination performance across four tasks: QA, Bangla-English Code-Mixed QA, Summarization, and Reasoning using a dual-track protocol. 12,000 hallucinated candidates are generated using GPT-5.4 and assessed across seven different LLMs. Experimental results show GPT-4.1 mini performs best in QA, while Mistral-nemo-12B performs worst. BenHalluScore, as a dual-track calibration metric, reveals systematic biases in hallucination detection. The study highlights the inadequacies of single-track and prompting-only approaches in low-resource language settings and provides directions for future research.
The technical contribution of the BenHalluEval framework lies in the introduction of BenHalluScore as a dual-track calibration metric, balancing error rates and preventing uniform response bias. The framework offers a new evaluation standard for hallucination detection in low-resource languages, with significant academic and industrial implications. Future research could explore hallucination detection in more low-resource languages and improve model performance on Bangla-English code-mixed input.
While BenHalluEval has made significant progress in hallucination detection, it still faces challenges with certain models exhibiting systematic biases. Additionally, hallucination evaluation in low-resource languages continues to face data scarcity issues. Future research could address these challenges by expanding datasets and improving models.
Deep Analysis
Background
Hallucination evaluation has been explored in various languages, but Bengali, the sixth most spoken language, remains under-evaluated. Existing Bengali models and multilingual benchmarks fail to address hallucination detection.
Core Problem
Bengali language models may generate outputs that contradict known facts, leading to hallucinations. Without a comprehensive benchmark, there's no reliable way to assess LLM trustworthiness.
Innovation
BenHalluEval framework is the first to provide a dedicated hallucination detection benchmark for Bengali, covering native-script Bengali and Bangla-English code-mixed input using a dual-track protocol.
Methodology
- �� Generate hallucinated candidates using GPT-5.4
- �� Evaluate seven LLMs across four tasks
- �� Introduce BenHalluScore as a dual-track calibration metric
- �� Validate annotation consistency among three Bengali annotators
Experiments
Using TyDiQA-GoldP, BanglaCHQ-Summ, and SOMADHAN datasets, nine LLMs are evaluated for hallucination detection across different tasks. Experimental settings include dual-track protocol and chain-of-thought prompting.
Results
GPT-4.1 mini performs best in QA with a BenHalluScore of 15.56%. Qwen2.5-32B shows significant improvement in Bangla-English Code-Mixed QA with a BenHalluScore of 24.00%.
Applications
BenHalluEval provides a new evaluation standard for hallucination detection in low-resource languages, applicable to academic research and industrial applications.
Limitations & Outlook
Certain models exhibit systematic biases in hallucination detection, affecting accuracy. Hallucination evaluation in low-resource languages still faces data scarcity issues.
Plain Language Accessible to non-experts
Imagine you're in a library where some books have incorrect information. BenHalluEval acts like a librarian, checking these books for hallucinated information through four tasks: QA, Code-Mixed QA, Summarization, and Reasoning. Each task has specific methods to ensure information accuracy. Just as a librarian ensures each book's information is correct, BenHalluEval ensures language model outputs are free of hallucinations.
ELI14 Explained like you're 14
Imagine you're playing a game where characters tell you information, but sometimes it's wrong. BenHalluEval is like a game master, checking if this information is correct through four tasks: QA, Code-Mixed QA, Summarization, and Reasoning. Each task has specific methods to ensure accuracy. Just like a game master ensures each character's information is correct, BenHalluEval ensures language model outputs are free of hallucinations.
Glossary
Hallucination
Outputs generated by a model that contradict known facts or conflict with provided context.
Used in BenHalluEval to detect hallucinated information in model outputs.
Dual-track protocol
Independently measures false-positive rate on ground-truth instances and hallucination detection rate on hallucinated candidates.
Used to evaluate model performance across different tasks.
BenHalluScore
A dual-track calibration metric balancing error rates and preventing uniform response bias.
Used to assess model performance in hallucination detection.
Chain-of-thought prompting
A prompting strategy that guides models through multi-step reasoning to improve hallucination detection.
Evaluated in BenHalluEval for model performance.
TyDiQA-GoldP
A Bengali dataset containing question, passage, and answer triples.
Used to evaluate model performance in QA tasks.
Open Questions Unanswered questions from this research
- 1 How to improve model performance on Bangla-English code-mixed input?
- 2 How to expand datasets to enhance hallucination detection accuracy in low-resource languages?
Applications
Immediate Applications
Academic Research
BenHalluEval provides a new evaluation standard for hallucination detection in low-resource languages, applicable to academic research.
Long-term Vision
Industrial Applications
The framework can be used to improve language models for low-resource languages, enhancing information accuracy in industrial applications.
Abstract
Despite Bengali being the sixth most spoken language in the world, no prior work has systematically evaluated hallucination in large language models (LLMs) for Bengali. We introduce BenHalluEval, a fine-grained hallucination evaluation framework for Bengali covering four tasks: Generative Question Answering (GQA), Bangla-English Code-Mixed QA, Summarization, and Reasoning. We construct 12,000 hallucinated candidates using GPT-5.4 across twelve task-specific hallucination types, drawn from three existing Bengali datasets, and evaluate seven LLMs spanning reasoning-oriented, multilingual, and Bengali-centric categories under a dual-track protocol that independently measures false-positive rate on ground-truth instances (Track A) and hallucination detection rate on hallucinated candidates (Track B). To jointly penalise both failure modes and prevent inflated scores from uniform response bias, we propose BenHalluScore, a dual-track calibration metric that ranges from 7.72% to 55.42% across models and tasks, revealing substantial variation in hallucination calibration. Chain-of-thought prompting, applied as a mitigation strategy, shifts response distributions without consistently improving hallucination discrimination. BenHalluEval establishes the first dedicated hallucination benchmark for Bengali and highlights the inadequacy of single-track and prompting-only evaluation approaches for low-resource language settings. The dataset and code are available at https://anonymous.4open.science/r/BanglaHalluEval-EB77.