A Benchmark for Automatic Medical Consultation System: Frameworks, Tasks and Datasets
Introduced IMCS-21 dataset to support five tasks in automatic medical consultation systems.
Key Findings
Methodology
The paper proposes two frameworks: dialogue understanding and task-oriented interaction. The dialogue understanding framework extracts structured information from dialogues, while the task-oriented interaction framework learns dialogue policies to select the next action. Five tasks include named entity recognition, dialogue act classification, symptom label inference, medical report generation, and diagnosis-oriented dialogue policy.
Key Results
- In the named entity recognition task, the ERNIE-Health model achieved the highest F1 score of 91.24%.
- In the dialogue act classification task, the ERNIE-Health model achieved the highest accuracy of 82.37%.
- In the symptom label inference task, the MC-BERT-MTL model achieved an F1 score of 80.42 on POS labels.
Significance
This research provides a comprehensive benchmark for automatic medical consultation systems, addressing existing gaps in dataset and task design. It offers a standardized testing platform for academia and industry, promoting further development of related technologies.
Technical Contribution
The IMCS-21 dataset provides multi-level fine-grained annotations supporting five independent tasks. Neural network models used in the study provide benchmark results for each task, demonstrating the dataset's usability.
Novelty
IMCS-21 is the first large-scale medical dialogue dataset with multi-level fine-grained annotations, covering a complete task chain from named entity recognition to diagnosis-oriented dialogue policy.
Limitations
- The dataset focuses mainly on pediatric diseases, which may limit its generalization to other medical fields.
- Learning dialogue policies may be constrained by the dataset's scale and diversity.
Future Work
Future research could expand the dataset's disease types, increase dialogue samples, and explore more complex dialogue policy learning methods.
AI Executive Summary
Automatic medical consultation systems have gained significant attention recently. Existing solutions fall short in dataset and task design.
This paper introduces IMCS-21, a large-scale medical dialogue dataset supporting five tasks: named entity recognition, dialogue act classification, symptom label inference, medical report generation, and diagnosis-oriented dialogue policy. Each task has detailed benchmark results.
IMCS-21 provides a standardized testing platform for automatic medical consultation systems, promoting the development of related technologies. Future work will expand the dataset's disease types and dialogue samples.
Deep Analysis
Background
Automatic medical consultation systems have great potential in improving healthcare service efficiency and patient experience. Existing research lacks comprehensive datasets and task designs, limiting further technological development.
Core Problem
Automatic medical consultation systems need a comprehensive benchmark to support task design and dataset development. Existing datasets lack fine-grained annotations and task completeness.
Innovation
The IMCS-21 dataset provides multi-level fine-grained annotations supporting five independent tasks, covering a complete task chain from named entity recognition to diagnosis-oriented dialogue policy.
Methodology
- �� Dialogue understanding framework extracts structured information from dialogues.
- �� Task-oriented interaction framework learns dialogue policies.
- �� Five tasks include named entity recognition, dialogue act classification, symptom label inference, medical report generation, and diagnosis-oriented dialogue policy.
Experiments
Experiments use the IMCS-21 dataset with neural network models for benchmark testing. Each task has detailed benchmark results, demonstrating the dataset's usability.
Results
The ERNIE-Health model performed best in named entity recognition and dialogue act classification tasks. In the symptom label inference task, the MC-BERT-MTL model achieved the highest F1 score on POS labels.
Applications
The IMCS-21 dataset provides a standardized testing platform for automatic medical consultation systems, promoting the development of related technologies.
Limitations & Outlook
The dataset focuses mainly on pediatric diseases, which may limit its generalization to other medical fields. Learning dialogue policies may be constrained by the dataset's scale and diversity.
Plain Language Accessible to non-experts
Imagine a doctor and a patient having a conversation. The doctor needs to extract key information from the patient's description, such as symptoms and medical history, and then make a diagnosis. This process is like finding the right puzzle pieces in a puzzle game. The IMCS-21 dataset is like a complete puzzle box, with various shapes and colors of pieces, helping doctors quickly find the information they need.
ELI14 Explained like you're 14
Imagine you're playing a super complex game where the goal is to become the best doctor! You need to find clues from the patient's description, like what symptoms they have, and then make a diagnosis. The IMCS-21 dataset is like a game guide, showing you the best ways to pass each level. Isn't that cool?
Glossary
Named Entity Recognition
Identifies specific entities in text, such as symptoms and drugs.
Used to extract key information from medical dialogues.
Dialogue Act Classification
Identifies the intention of each sentence in a dialogue, such as asking or informing.
Analyzes the communication intent between doctors and patients.
Symptom Label Inference
Determines whether a patient has specific symptoms.
Helps doctors make diagnostic decisions.
Medical Report Generation
Generates a report summarizing the dialogue content.
Records and conveys medical information.
Diagnosis-oriented Dialogue Policy
Learns the optimal dialogue strategy for automatic diagnosis.
Selects the next action to gather more symptom information.
Open Questions Unanswered questions from this research
- 1 How to expand the dataset to cover more disease types?
- 2 How to improve the effectiveness of dialogue policy learning?
Applications
Immediate Applications
Online Medical Consultation
Doctors can use the IMCS-21 dataset to quickly extract patient information, improving diagnostic efficiency.
Long-term Vision
Automatic Diagnosis Systems
The IMCS-21 dataset will drive the development of automatic diagnosis systems, achieving more precise medical services.
Abstract
In recent years, interest has arisen in using machine learning to improve the efficiency of automatic medical consultation and enhance patient experience. In this article, we propose two frameworks to support automatic medical consultation, namely doctor-patient dialogue understanding and task-oriented interaction. We create a new large medical dialogue dataset with multi-level finegrained annotations and establish five independent tasks, including named entity recognition, dialogue act classification, symptom label inference, medical report generation and diagnosis-oriented dialogue policy. We report a set of benchmark results for each task, which shows the usability of the dataset and sets a baseline for future studies. Both code and data is available from https://github.com/lemuria-wchen/imcs21.