Do Large Language Models Speak All Languages Equally? A Comparative Study in Low-Resource Settings
This study compares GPT-4, Llama 2, and Gemini in zero-shot tasks across English and low-resource South Asian languages, revealing significant performance gaps.
Key Findings
Methodology
The research evaluates GPT-4, Llama 2, and Gemini on English, Bangla, Hindi, and Urdu using zero-shot prompting. Datasets for sentiment analysis, hate speech, and natural language inference (XNLI, SemEval-2017, Davidson) were translated and extended. The models were prompted with natural language instructions, and performance metrics (accuracy, F1) were collected across tasks and languages. Comparative analysis focused on performance disparities, influenced by training data volume, cultural differences, and translation quality.
Key Results
- GPT-4 achieved an accuracy of 86.73% on English NLI, outperforming Bangla, Hindi, and Urdu by 18-22%. In sentiment and hate speech tasks, English performance was 13-22% higher. Llama 2 and Gemini also performed better in English, but with smaller margins. Low-resource languages suffered from limited training data, cultural nuances, and translation errors, especially in hate speech detection. Urdu showed strong results due to Arabic script support, aligning with its linguistic roots.
- Overall, NLI tasks yielded higher performance due to structured data and clear labels, with GPT-4 leading. Sentiment and hate speech detection were more challenging due to social and cultural context dependence. The performance gaps highlight the need for better low-resource language data and cultural adaptation. Gemini excelled in Urdu, likely due to shared Arabic semantics. The results underscore the importance of data diversity and cultural understanding for equitable multilingual AI.
- GPT-4 consistently outperformed Llama 2 and Gemini across all tasks, especially in reasoning. Llama 2 was better at hate speech detection but weaker in inference and sentiment. The performance disparities reveal the impact of training data bias, model architecture, and cultural factors. These findings emphasize that current models are not yet universally effective for low-resource languages, necessitating further research in data collection, multilingual training, and cultural adaptation.
Significance
This work highlights the persistent performance gap between resource-rich and low-resource languages in large language models, emphasizing the dominance of English in training data. It underscores the challenges faced by AI in achieving fairness and inclusivity across diverse linguistic and cultural contexts. The findings inform future directions for developing more equitable multilingual models, crucial for global AI deployment, especially in regions like South Asia where low-resource languages are prevalent. Improving model performance in these languages can enhance accessibility, social inclusion, and local language technology applications, fostering broader AI adoption.
Technical Contribution
The study introduces a comprehensive benchmarking framework for evaluating GPT-4, Llama 2, and Gemini across multiple low-resource languages and tasks. It innovates by leveraging translation-augmented datasets and natural language instruction prompts for zero-shot evaluation. The comparison across models and languages reveals performance bottlenecks and cultural influences, providing insights into model generalization and robustness. The approach advances understanding of multilingual model capabilities, emphasizing the importance of data diversity and cultural context in model training and evaluation.
Novelty
This is the first systematic comparison of GPT-4, Llama 2, and Gemini on South Asian low-resource languages across multiple NLP tasks using zero-shot prompting. The study uniquely combines translation-based data augmentation with detailed performance analysis, highlighting how cultural and linguistic factors influence model outputs. It also demonstrates the effectiveness of natural language instruction prompts in low-resource settings, offering new avenues for improving multilingual AI systems. These contributions fill a critical gap in multilingual NLP research, especially for underrepresented languages.
Limitations
- The training data for low-resource languages remains limited, leading to suboptimal performance and cultural misalignment. The reliance on automatic translation introduces errors, affecting evaluation accuracy. The study does not explore multi-turn dialogues or multimodal inputs, which are common in real-world applications. Additionally, the models' biases and ethical considerations were not deeply examined, requiring further investigation for deployment in sensitive contexts.
Future Work
Future research should focus on collecting high-quality, culturally diverse datasets for low-resource languages, possibly through community engagement. Incorporating multimodal data (audio, images) and multi-turn dialogue capabilities can enhance understanding and contextual reasoning. Fine-tuning models with low-resource language corpora and domain-specific data may reduce performance gaps. Exploring adaptive prompting techniques and reinforcement learning from human feedback (RLHF) tailored to cultural nuances will further improve model robustness and fairness. These efforts will facilitate more inclusive and effective multilingual AI systems.
AI Executive Summary
Large language models (LLMs) like GPT-4, Llama 2, and Gemini have revolutionized natural language processing, achieving remarkable performance in resource-rich languages such as English. However, their effectiveness diminishes significantly when applied to low-resource languages, especially in regions like South Asia, where languages like Bangla, Hindi, and Urdu lack extensive annotated datasets. This disparity hampers the deployment of AI solutions for critical applications including sentiment analysis, hate speech detection, and natural language inference.
To address this challenge, the study constructed multilingual datasets by translating existing English datasets into Bangla, Hindi, and Urdu, enabling comprehensive zero-shot evaluation. Using natural language prompts, the models were tested across three core NLP tasks. Results showed GPT-4 consistently outperformed Llama 2 and Gemini, with English achieving an accuracy of 86.73% in NLI, far surpassing low-resource languages by 18-22%. Similar trends appeared in sentiment and hate speech tasks, where performance gaps ranged from 13% to 22%. The analysis revealed that training data volume, cultural differences, and translation quality heavily influence low-resource language performance.
The findings underscore the importance of data diversity, cultural adaptation, and model fine-tuning to improve multilingual AI. While GPT-4 demonstrated superior reasoning and inference capabilities, the performance disparities highlight the need for targeted data collection and model adaptation strategies. The research provides a roadmap for future work, including multilingual data augmentation, multi-task training, and cultural-aware prompting, to foster fairer, more inclusive AI systems.
Overall, this work emphasizes that current LLMs are not yet universally effective across all languages. Addressing these gaps is vital for deploying AI technologies that serve diverse linguistic communities, especially in underrepresented regions. Continued efforts in data collection, model training, and evaluation will be essential to realize truly multilingual, culturally sensitive AI solutions that can operate equitably worldwide.
Deep Dive
Abstract
Large language models (LLMs) have garnered significant interest in natural language processing (NLP), particularly their remarkable performance in various downstream tasks in resource-rich languages. Recent studies have highlighted the limitations of LLMs in low-resource languages, primarily focusing on binary classification tasks and giving minimal attention to South Asian languages. These limitations are primarily attributed to constraints such as dataset scarcity, computational costs, and research gaps specific to low-resource languages. To address this gap, we present datasets for sentiment and hate speech tasks by translating from English to Bangla, Hindi, and Urdu, facilitating research in low-resource language processing. Further, we comprehensively examine zero-shot learning using multiple LLMs in English and widely spoken South Asian languages. Our findings indicate that GPT-4 consistently outperforms Llama 2 and Gemini, with English consistently demonstrating superior performance across diverse tasks compared to low-resource languages. Furthermore, our analysis reveals that natural language inference (NLI) exhibits the highest performance among the evaluated tasks, with GPT-4 demonstrating superior capabilities.