MedRoundsQA: A Persona and Difficulty Aware Evaluation for Multi-Turn Medical Consultations
MedRoundsQA evaluates medical diagnosis through multi-turn dialogues, with accuracy dropping 13-39 points.
Key Findings
Methodology
MedRoundsQA extracts 24-slot clinical records from 1,387 exam cases to generate multi-turn doctor-patient dialogues. It uses model uncertainty for difficulty classification and evaluates 15 LLM doctor agents. By comparing single-turn and multi-turn diagnoses, it analyzes the impact of patient personas on diagnostic accuracy.
Key Results
- Multi-turn diagnostic accuracy drops 13-39 points, indicating increased difficulty with multi-turn dialogues.
- More dialogue turns improve question relevance, but diagnostic accuracy plateaus after 6-12 turns.
- Patient persona differences shift diagnostic accuracy by 7-8 points, revealing equity risks missed by single-turn evaluations.
Significance
This study reveals the limitations of existing single-turn diagnostic benchmarks by introducing multi-turn dialogue evaluations, emphasizing the importance of considering patient communication styles in real medical scenarios. It provides a new perspective on diagnostic equity through patient persona analysis, with significant academic and practical implications.
Technical Contribution
MedRoundsQA offers a novel evaluation framework combining standardized clinical records, patient persona variations, and difficulty classification, enabling more comprehensive assessments of diagnostic models. It achieves comparative analysis of the same case under different patient personas, revealing the impact of communication styles on diagnostic performance.
Novelty
This is the first framework to combine standardized clinical records and patient persona variations for multi-turn diagnostic evaluation, allowing analysis of communication style impacts while keeping clinical content constant.
Limitations
- Models collect information inefficiently in multi-turn dialogues, leading to decreased diagnostic accuracy.
- Variations in patient personas may cause model performance instability.
- Increasing dialogue turns does not significantly improve diagnostic accuracy.
Future Work
Future research can explore the impact of more complex patient personas and emotional variations on diagnosis, and how to enhance models' information collection capabilities in multi-turn dialogues.
AI Executive Summary
Existing medical diagnostic benchmarks often rely on single-turn, multiple-choice questions, failing to reflect the complexity of clinical dialogues. MedRoundsQA introduces multi-turn dialogue evaluations, revealing the limitations of single-turn diagnostics. The study shows that increasing dialogue turns improves question relevance, but diagnostic accuracy plateaus after 6-12 turns. Additionally, patient persona differences shift diagnostic accuracy by 7-8 points, highlighting equity risks missed by single-turn evaluations. This research emphasizes the importance of considering patient communication styles in real medical scenarios and provides a new perspective for evaluating diagnostic models. Future research can explore the impact of more complex patient personas and emotional variations on diagnosis, and how to enhance models' information collection capabilities in multi-turn dialogues.
Deep Analysis
Background
Medical diagnosis is inherently interactive, with clinicians gathering information incrementally. However, existing diagnostic benchmarks mostly rely on single-turn, multiple-choice questions, failing to reflect the complexity of clinical dialogues. With the rise of large language models, evaluating diagnostic models' performance in multi-turn dialogues has become an important topic.
Core Problem
Existing single-turn diagnostic benchmarks fail to reflect the complexity of clinical dialogues, ignoring the impact of patient communication styles on diagnosis. This may lead to poor model performance in real applications, especially when information arrives gradually or patients communicate imprecisely.
Innovation
MedRoundsQA converts 1,387 exam cases into 24-slot clinical records to generate multi-turn doctor-patient dialogues. It introduces patient persona variations and uses model uncertainty for difficulty classification, enabling more comprehensive assessments of diagnostic models.
Methodology
- �� Extract 24-slot clinical records from exam cases
- �� Generate multi-turn doctor-patient dialogues
- �� Introduce patient persona variations
- �� Use model uncertainty for difficulty classification
- �� Evaluate 15 LLM doctor agents
Experiments
The experiment uses 1,387 cases to evaluate 15 LLM doctor agents. By comparing single-turn and multi-turn diagnoses, it analyzes the impact of patient personas on diagnostic accuracy. Model uncertainty is used for difficulty classification.
Results
Multi-turn diagnostic accuracy drops 13-39 points. More dialogue turns improve question relevance, but diagnostic accuracy plateaus after 6-12 turns. Patient persona differences shift diagnostic accuracy by 7-8 points.
Applications
This study can be used to evaluate and improve medical diagnostic models, especially in scenarios involving multi-turn dialogues and variations in patient communication styles. It has significant implications for assessing the equity and robustness of medical AI.
Limitations & Outlook
Models collect information inefficiently in multi-turn dialogues, leading to decreased diagnostic accuracy. Variations in patient personas may cause model performance instability. Increasing dialogue turns does not significantly improve diagnostic accuracy.
Plain Language Accessible to non-experts
Imagine you're chatting with a doctor, and the doctor needs to ask questions to understand your condition. This process is like the doctor putting together a puzzle, with each question being a piece of the puzzle. Existing diagnostic methods are like the doctor looking at a complete picture rather than assembling the puzzle step by step. MedRoundsQA allows the doctor to assemble the puzzle through multi-turn dialogues, reflecting the diagnostic process more realistically. This method shows how different patient personas can affect the doctor's diagnosis, just as different puzzle pieces can change the final picture.
ELI14 Explained like you're 14
Imagine you're playing a detective game, where the doctor is the detective and you're the clue provider. The doctor needs to ask questions to gather clues and find the root of the problem. Existing diagnostic methods are like the detective getting all the clues at once, rather than collecting them gradually. MedRoundsQA allows the doctor to gather clues through multi-turn dialogues, reflecting the detective's work process more realistically. This method shows how different roles can affect the detective's judgment, just like different clues can change the final conclusion.
Glossary
Multi-turn Dialogue
A dialogue form where information is collected incrementally over multiple interactions.
Used to simulate real doctor-patient dialogues.
Patient Persona
Different patient communication styles and background settings.
Used to analyze the impact of communication styles on diagnosis.
Diagnostic Accuracy
The model's ability to provide correct answers in diagnostic tasks.
Used to evaluate model performance in multi-turn dialogues.
Model Uncertainty
The degree of uncertainty a model has about its predictions.
Used for difficulty classification and robustness evaluation.
Equity
The model's ability to perform consistently across different patient personas.
Used to assess model performance in diverse patient backgrounds.
Open Questions Unanswered questions from this research
- 1 How to improve models' information collection capabilities in multi-turn dialogues?
- 2 How do variations in patient personas affect diagnostic equity?
- 3 How to enhance diagnostic accuracy in multi-turn dialogues?
Applications
Immediate Applications
Medical AI Evaluation
Used to evaluate and improve the multi-turn dialogue capabilities of medical diagnostic models.
Long-term Vision
Equity Enhancement
Enhance the equity and robustness of medical AI by analyzing patient persona differences.
Abstract
Medical benchmarks are dominated by single-turn, multiple-choice clinical cases that poorly reflect real consultations. Practically, clinicians elicit evidence interactively and patient communication varies widely. We introduce MedRoundsQA, a multi-turn diagnostic benchmark derived from 1,387 board-exam cases across 17 specialties. Each case is converted into a structured 24-slot clinical record, and then instantiated as controlled doctor-patient dual-agent dialogues under varying patient personas, with the underlying clinical content held fixed. We further classify cases by difficulty using model-based uncertainty to enable easy-to-hard analysis. Evaluations of fifteen LLM doctor agents show that (i) moving from a single-turn diagnosis on the standardized records to multi-turn consultations causes large degradations of roughly 13-39 points; (ii) more turns reliably improves question relevance, but diagnostic accuracy exhibits diminishing returns and typically plateaus after 6-12 turns; and (iii) patient persona differences can shift diagnosis accuracy by about 7-8 points (lowest to highest education), highlighting equity risks that single-turn benchmarks miss.