Benchmarking Large Language Models on CMExam -- A Comprehensive Chinese Medical Exam Dataset
CMExam is the first Chinese medical exam dataset with comprehensive annotations; GPT-4 achieved 61.6% accuracy on it.
Key Findings
Methodology
CMExam is sourced from the Chinese National Medical Licensing Examination, containing 60K+ multiple-choice questions with explanations. Medical experts annotated five dimensions: disease groups, clinical departments, medical disciplines, competency areas, and difficulty levels. Initial annotations were automated using GPT-4, followed by expert review.
Key Results
- GPT-4 achieved 61.6% accuracy and a weighted F1 score of 0.617 on CMExam, significantly lower than human accuracy of 71.6%.
- GPT-3.5 and GPT-4 generated reasonable answers for explanation tasks, but with low BLEU and ROUGE scores.
- Lightweight models like ChatGLM, after fine-tuning, approached GPT-3.5's performance, using only 3% of its parameters.
Significance
CMExam fills the gap of standardized datasets in the Chinese medical domain, providing a crucial benchmark for LLM applications in medical QA systems. The study reveals challenges and potential solutions for LLMs in the medical field.
Technical Contribution
Introduced the first Chinese dataset with comprehensive medical annotations, providing a benchmark for LLM performance in the medical domain. The GPT-4-assisted annotation strategy significantly reduced the time and labor costs of large-scale dataset annotation.
Novelty
CMExam is the first Chinese medical exam dataset with comprehensive annotations, covering multiple dimensions to support thorough LLM evaluation in the medical field.
Limitations
- LLMs still underperform in explanation tasks, generating short and insufficiently detailed explanations.
- Existing medical domain LLMs have limited coverage, resulting in poor zero-shot performance.
Future Work
Future research could explore integrating more medical knowledge to enhance LLM performance in medical applications, particularly in explanation tasks.
AI Executive Summary
Recent advancements in large language models (LLMs) have significantly impacted natural language processing. However, evaluating LLMs in the medical field remains challenging due to the lack of standardized and comprehensive datasets. To address this, researchers introduced CMExam, a dataset based on the Chinese National Medical Licensing Examination, containing over 60K multiple-choice questions with explanations. The dataset provides five dimensions of annotations, including disease groups, clinical departments, medical disciplines, competency areas, and difficulty levels.
In experiments, GPT-4 achieved 61.6% accuracy and a weighted F1 score of 0.617 on CMExam, although it still lags behind human accuracy of 71.6%. Additionally, the study found that lightweight models like ChatGLM, after fine-tuning, could approach GPT-3.5's performance, despite using only about 3% of its parameters.
The introduction of CMExam provides a crucial benchmark for LLM evaluation in the Chinese medical domain, revealing challenges and potential solutions for LLMs in the medical field. Future research could further explore integrating more medical knowledge to enhance LLM performance in medical applications, particularly in explanation tasks.
Deep Analysis
Background
Large language models (LLMs) like T5 and GPT-4 have revolutionized natural language processing. However, evaluating LLMs in the medical field poses challenges due to the lack of standardized and comprehensive datasets. Existing medical datasets often come from online forums and consumer feedback, which may contain biases and errors, especially in Chinese, where language resource inequality is more pronounced.
Core Problem
The evaluation of LLMs in the medical field lacks standardized datasets. Existing datasets are insufficient in size and diversity, preventing comprehensive assessment of LLM capabilities. Moreover, most datasets focus on text generation tasks, lacking clear choice evaluations, which impedes objective and quantitative measurement of LLM performance.
Innovation
CMExam is the first Chinese dataset with comprehensive medical annotations, providing multiple dimensions of annotations to support thorough LLM evaluation in the medical field. The GPT-4-assisted annotation strategy significantly reduced the time and labor costs of large-scale dataset annotation.
Methodology
- �� The dataset is sourced from the Chinese National Medical Licensing Examination, containing 60K+ multiple-choice questions.
- �� Provides five dimensions of annotations: disease groups, clinical departments, medical disciplines, competency areas, difficulty levels.
- �� Initial annotations were automated using GPT-4, followed by expert review.
Experiments
Experiments used the CMExam dataset to evaluate LLM performance on multiple-choice prediction and explanation tasks. Baseline models included GPT-3.5, GPT-4, ChatGLM, among others. Evaluation metrics were accuracy and weighted F1 score, with BLEU and ROUGE scores for explanation tasks.
Results
GPT-4 achieved the best performance on the multiple-choice prediction task with 61.6% accuracy. Lightweight models like ChatGLM, after fine-tuning, approached GPT-3.5's performance. In explanation tasks, GPT models generated short explanations, resulting in low BLEU scores.
Applications
CMExam provides a crucial benchmark for LLM evaluation in the Chinese medical domain, suitable for developing and evaluating medical QA systems. The study reveals challenges and potential solutions for LLMs in the medical field.
Limitations & Outlook
LLMs still underperform in explanation tasks, generating short and insufficiently detailed explanations. Existing medical domain LLMs have limited coverage, resulting in poor zero-shot performance. Future research could explore integrating more medical knowledge to enhance LLM performance in medical applications.
Plain Language Accessible to non-experts
Imagine you're taking a medical exam with multiple-choice questions. CMExam is like that exam question bank, helping AI models like GPT-4, a 'brain,' to answer these questions. Just like we study different subjects in school, this dataset has different annotations like disease groups and clinical departments. Researchers used GPT-4 to initially annotate these questions, then medical experts reviewed them to ensure accuracy. This way, AI models can better understand and answer medical questions, just like a smart student excelling in exams.
ELI14 Explained like you're 14
Imagine you're playing a medical quiz game, and CMExam is the question bank for that game. It has lots of multiple-choice questions, helping AI models like GPT-4, a 'super brain,' to answer them. Researchers used GPT-4 to annotate these questions, then doctors checked them to make sure they're right. This way, AI models can perform well in medical exams, just like a smart student! In the future, AI might help doctors find causes of diseases faster, which is super cool!
Glossary
GPT-4
An advanced large language model capable of understanding and generating natural language text.
Evaluated on the CMExam dataset, showing strong performance.
CMExam
A Chinese medical exam dataset containing 60K+ multiple-choice questions for evaluating LLM performance.
Used to assess models like GPT-4 in the medical domain.
BLEU
A metric for evaluating the quality of machine-translated text by comparing it to reference translations.
Used to evaluate LLM performance on explanation tasks.
ROUGE
A metric for evaluating text summarization quality by measuring overlap with reference summaries.
Used to evaluate LLM performance on explanation tasks.
ICD-11
A globally recognized standard classification system for health conditions, used for annotating disease groups in CMExam.
Used for disease group annotations in CMExam.
Open Questions Unanswered questions from this research
- 1 How to improve LLM performance in medical explanation tasks? Current methods generate short and insufficiently detailed explanations, requiring further research.
- 2 Existing medical domain LLMs have limited coverage; how to expand their knowledge base to improve zero-shot performance?
Applications
Immediate Applications
Medical QA Systems
CMExam can be used to develop smarter medical QA systems, helping doctors quickly access information.
Long-term Vision
Medical Education
CMExam can be used in medical education to help students better prepare for exams and improve learning efficiency.
Abstract
Recent advancements in large language models (LLMs) have transformed the field of question answering (QA). However, evaluating LLMs in the medical field is challenging due to the lack of standardized and comprehensive datasets. To address this gap, we introduce CMExam, sourced from the Chinese National Medical Licensing Examination. CMExam consists of 60K+ multiple-choice questions for standardized and objective evaluations, as well as solution explanations for model reasoning evaluation in an open-ended manner. For in-depth analyses of LLMs, we invited medical professionals to label five additional question-wise annotations, including disease groups, clinical departments, medical disciplines, areas of competency, and question difficulty levels. Alongside the dataset, we further conducted thorough experiments with representative LLMs and QA algorithms on CMExam. The results show that GPT-4 had the best accuracy of 61.6% and a weighted F1 score of 0.617. These results highlight a great disparity when compared to human accuracy, which stood at 71.6%. For explanation tasks, while LLMs could generate relevant reasoning and demonstrate improved performance after finetuning, they fall short of a desired standard, indicating ample room for improvement. To the best of our knowledge, CMExam is the first Chinese medical exam dataset to provide comprehensive medical annotations. The experiments and findings of LLM evaluation also provide valuable insights into the challenges and potential solutions in developing Chinese medical QA systems and LLM evaluation pipelines. The dataset and relevant code are available at https://github.com/williamliujl/CMExam.