Comparative Analysis of Large Language Models in Generating Telugu Responses for Maternal Health Queries

In recent years, large language models (LLMs) have made significant advancements in the field of natural language processing, particularly in generating fluent and contextually appropriate text. However, their performance in low-resource languages remains inconsistent. Existing research primarily focuses on resource-rich languages like English and Chinese, neglecting the potential and challenges of applying LLMs in regional languages. In sensitive domains like maternal health, the accuracy and trustworthiness of models are crucial. Therefore, systematically evaluating LLM performance in low-resource languages is of great importance.

The core problem of this study is to evaluate LLM performance in generating Telugu responses to maternal health queries. As Telugu is a low-resource language, the performance of existing models in this language has not been thoroughly studied. Additionally, the maternal health domain requires high accuracy and completeness of information, necessitating the development of a comprehensive evaluation framework to thoroughly assess model performance in this domain.

The core innovations of this study include:

1) The first systematic evaluation of LLM performance in the low-resource language of Telugu, specifically in the maternal health domain.

2) A comprehensive performance analysis combining BERT Score and expert evaluation, revealing the impact of model selection and prompt language on answer quality.

3) A proposed comprehensive evaluation framework that combines automated semantic analysis and expert evaluation, offering new perspectives and methods for optimizing LLMs in regional languages.

The methodology of this study includes the following steps:

• �� Data Collection: Collect common maternal health-related questions covering topics such as nutrition, symptom management, fetal development, and antenatal care, presented in both English and Telugu.
• �� Model Generation: Use ChatGPT-4o, GeminiAI, and Perplexity AI to generate Telugu responses.
• �� Automated Evaluation: Use BERT Score to assess the semantic similarity between generated answers and expert responses.
• �� Expert Evaluation: Invite ten gynecologists fluent in Telugu to conduct qualitative evaluations of the generated answers, examining accuracy, fluency, relevance, coherence, and completeness.
• �� Comprehensive Analysis: Combine automated and expert evaluation results to analyze the impact of input language on the quality of generated answers.

The experimental design includes the following aspects:

• �� Datasets: Use a bilingual dataset covering common maternal health-related questions.
• �� Baselines: Select ChatGPT-4o, GeminiAI, and Perplexity AI as baseline models.
• �� Evaluation Metrics: Use BERT Score to assess semantic similarity and combine expert evaluation to examine accuracy, fluency, relevance, coherence, and completeness.
• �� Hyperparameters: Conduct experiments based on the default settings of the models to ensure reproducibility of the results.
• �� Ablation Studies: Analyze the impact of input language on the quality of generated answers.

The results analysis shows:

• �� Perplexity achieved the highest F1 score of 0.704 when prompted in English, indicating strong semantic alignment with expert responses.
• �� Gemini demonstrated high semantic similarity in both English and Telugu prompts, with F1 scores closely following Perplexity.
• �� ChatGPT's semantic alignment improved when prompted in Telugu, highlighting its sensitivity to input language.
• �� Expert evaluations show that Gemini excels in producing accurate and coherent Telugu responses related to maternal health, while Perplexity performs well when prompted in Telugu.

The application scenarios of this study include:

• �� Medical Consultation: Use LLMs to provide accurate maternal health information in low-resource language settings, enhancing the accessibility of healthcare services.
• �� Educational Training: Provide training materials in regional languages for medical professionals, promoting the dissemination and sharing of knowledge.
• �� Health Management: Offer personalized health management advice for pregnant women, improving the effectiveness of health management.

The limitations of this study include:

• �� The study is limited to Telugu and does not cover other low-resource languages, restricting the generalizability of the conclusions.
• �� The subjectivity of expert evaluations may affect the objectivity of the results, requiring further validation.
• �� The training data of LLMs was not analyzed in detail, which may affect the comprehensive understanding of model performance. Future research directions include expanding to more low-resource languages, increasing the diversity of maternal health questions, and fine-tuning LLMs specifically for regional languages.