IMCBench: A benchmark for multimodal LLMs in Image-grounded Medical Conversations
IMCBench evaluates multimodal LLMs in image-grounded medical dialogues; Claude Opus 4.6 scores highest at 3.61.
Key Findings
Methodology
IMCBench combines real dermatological images with synthetic patient profiles to evaluate multimodal LLMs in medical dialogues. It uses an LLM-as-Jury scoring system, calibrated against clinical expert annotations, covering safety, accuracy, and uncertainty handling.
Key Results
- Claude Opus 4.6 scores highest in safety and accuracy but ranks second in uncertainty handling.
- Visual input and EHR context significantly contribute to safety, with drops of 0.18 and 0.23 when removed.
- No single model excels in all dimensions, with safety degrading by 0.27 for malignant and rare conditions.
Significance
IMCBench fills a gap in existing medical AI benchmarks by providing a multidimensional evaluation framework, highlighting that accurate clinical descriptions do not equate to safe patient guidance. This study sets new standards for evaluating multimodal dialogue systems in medical AI, promoting safer and more accurate clinical applications.
Technical Contribution
IMCBench is the first to combine image and EHR in multi-turn dialogue evaluation, introducing an LLM-as-Jury scoring system that offers a self-optimizing rubric framework, significantly improving alignment with clinical experts.
Novelty
IMCBench is the first benchmark to combine image and EHR in multi-turn dialogues, providing a comprehensive evaluation framework that addresses the lack of multimodal dialogue assessment in existing benchmarks.
Limitations
- Models show significant safety degradation in handling malignant and rare conditions.
- No existing model shows clear superiority across all evaluation dimensions.
Future Work
Future research could explore more complex multimodal input combinations, optimize models for uncertainty handling, and expand to other medical fields.
AI Executive Summary
IMCBench is a benchmark for evaluating multimodal large language models (LLMs) in image-grounded medical dialogues. Existing medical AI benchmarks often support only single modalities, failing to effectively assess the safety and accuracy of multimodal dialogues. IMCBench addresses this gap by combining real dermatological images with synthetic patient profiles to simulate realistic patient-clinician interactions, providing a comprehensive evaluation framework.
In experiments, IMCBench evaluated eight frontier multimodal models, finding that Claude Opus 4.6 performed best in safety and accuracy but was slightly less effective in handling uncertainty. The study shows that visual input and EHR context significantly contribute to model safety, with notable declines when either is removed.
The significance of IMCBench lies in filling the gap in existing benchmarks, setting new standards for evaluating multimodal dialogue systems in medical AI, and promoting safer and more accurate clinical applications. Future research could explore more complex multimodal input combinations and optimize models for uncertainty handling.
Deep Analysis
Background
Recent advances in large language models and vision-language models have expanded their potential in healthcare. However, existing medical AI benchmarks often support only single modalities, failing to effectively assess the safety and accuracy of multimodal dialogues. IMCBench aims to fill this gap by combining real dermatological images with synthetic patient profiles, providing a comprehensive evaluation framework.
Core Problem
Existing medical AI benchmarks lack effective multimodal dialogue assessment, particularly in terms of safety and accuracy. Single-modality benchmarks fail to simulate realistic patient-clinician interactions, leading to limitations in evaluation results.
Innovation
IMCBench's core innovation is the first-ever combination of image and EHR in multi-turn dialogue evaluation. It introduces an LLM-as-Jury scoring system, offering a self-optimizing rubric framework that significantly improves alignment with clinical experts.
Methodology
- �� IMCBench combines real dermatological images with synthetic patient profiles to simulate realistic patient-clinician interactions.
- �� It uses an LLM-as-Jury scoring system, calibrated against clinical expert annotations, covering safety, accuracy, and uncertainty handling.
- �� Ablation studies assess the contribution of visual input and EHR context to model safety.
Experiments
IMCBench evaluated eight frontier multimodal models, including Claude, GPT, Nova, and Llama series. Each model was scored on a 1-5 scale using the LLM-as-Jury system, calibrated against clinical expert annotations.
Results
Claude Opus 4.6 performed best in safety and accuracy but was slightly less effective in handling uncertainty. Visual input and EHR context significantly contribute to model safety, with notable declines when either is removed.
Applications
IMCBench can be used to evaluate medical AI systems in multimodal dialogues, particularly in terms of safety and accuracy. Its evaluation framework can be applied to other medical fields, promoting safer and more accurate clinical applications.
Limitations & Outlook
Models show significant safety degradation in handling malignant and rare conditions. Future research could explore more complex multimodal input combinations and optimize models for uncertainty handling.
Plain Language Accessible to non-experts
Imagine you're at a hospital, and the doctor needs to understand your health condition through conversation and images. IMCBench is like a simulator that trains AI doctors by combining real dermatological images with fictional patient information, helping AI doctors better understand and handle conditions. This way, AI doctors can provide safer and more accurate advice even in uncertain situations.
ELI14 Explained like you're 14
Hey, buddy! Imagine you're playing a super complex game where doctors are NPCs, and they need to figure out your health issues by looking at pictures and chatting. IMCBench is a tool that trains these NPCs, using real pictures and made-up stories to make AI doctors smarter and safer when giving advice. So, when you have health questions, they can help you better!
Glossary
IMCBench
A benchmark for evaluating multimodal large language models in medical dialogues.
Used to test models in image-grounded multi-turn medical dialogues.
LLM-as-Jury
A scoring system using large language models as judges, calibrated against clinical expert annotations.
Used in IMCBench to score model safety and accuracy.
EHR
Electronic Health Record, containing a patient's health information.
Used in IMCBench to provide patient background information.
Ablation Study
A method to study the impact of removing certain components on the overall system.
Used to assess the contribution of visual input and EHR context to model safety.
Multimodal
Technology that combines multiple data types, such as text and images.
Models evaluated by IMCBench need to handle multimodal inputs.
Open Questions Unanswered questions from this research
- 1 How to improve model safety in handling malignant and rare conditions?
- 2 How to optimize models for uncertainty handling?
Applications
Immediate Applications
Medical AI System Evaluation
IMCBench can be used to evaluate medical AI systems in multimodal dialogues, particularly in terms of safety and accuracy.
Long-term Vision
Cross-Field Applications
IMCBench's evaluation framework can be applied to other medical fields, promoting safer and more accurate clinical applications.
Abstract
Recent advances in large language models and vision-language models have enabled reasoning over multimodal data, offering opportunities for clinical applications such as decision support and triaging. However, existing medical AI benchmarks are fragmented: some support multi-turn dialogues but lack images, while others provide multimodal inputs but focus on single-turn QA tasks. To address this gap, we introduce IMCBench, an image-grounded, multi-turn medical conversation benchmark that pairs real, publicly available clinical images with synthetic patient profiles to simulate realistic patient-clinician interactions. Each conversation is evaluated across three clinical dimensions: safety, accuracy, and appropriate use of uncertainty in diagnosis. We benchmark eight multimodal frontier models across four model families (Claude, GPT, Nova, and Llama), scoring each on a 1-5 scale using LLM-as-Jury scoring calibrated against expert clinician annotations. Our results show that Claude Opus 4.6 achieves the highest overall score (3.61), followed by Claude Sonnet 4.6 (3.30) and GPT-5.2 (3.29), though no model dominates all dimensions and safety degrades for both malignant and rare conditions ($Δ$ = -0.27 each). Ablation studies further reveal that both visual input and EHR context contribute to safe guidance (safety drops of 0.18 and 0.23 on average when each is removed), with stronger models leveraging visual features more effectively. Together, these findings demonstrate that accurate clinical description does not guarantee safe patient guidance, motivating the need for multi-dimensional evaluation frameworks in medical AI.