MedObvious: Exposing the Medical Moravec's Paradox in VLMs via Clinical Triage

With the advancement of artificial intelligence technology, Vision Language Models (VLMs) are increasingly applied in medical imaging. These models can generate radiology-style descriptions, answer clinical questions, and perform multi-step reasoning over images and text. Recently, general-purpose models such as GPT-4o, Flamingo, and LLaVA, as well as medical adaptations like LLaVA-Med and RadFM, have been used for core perception in medical imaging. However, despite their excellent performance in generating coherent diagnostic narratives, they still exhibit significant gaps in basic sanity checks. Moravec's Paradox suggests that perception and spatial reasoning, trivial for humans, can be disproportionately difficult for machines even when higher-level outputs appear plausible. In medical imaging, this gap is consequential because failures occur before diagnosis: when the input is invalid or inconsistent, downstream reports become clinically uninterpretable.

In clinical practice, interpretation begins with pre-diagnostic sanity checks: clinicians first verify body part, view, modality, laterality, orientation, and basic image integrity, and they do not proceed to diagnosis if these checks fail. This requirement is amplified in multi-view ultrasound, multi-slice CT/MRI, and multi-panel viewer agentic workflows. Existing medical VLM benchmarks such as VQA-RAD, PathVQA, PMC-VQA, VQA-Med, and SLAKE primarily assess the correctness of final answers, while ignoring the importance of input validation. This leads to models producing coherent diagnostic narratives while ignoring basic sanity checks, making them brittle and potentially unsafe in multi-image or agentic workflows.

MedObvious is the first benchmark focusing on input validation in medical imaging, emphasizing set-level consistency over small multi-panel image sets. Its innovations include: 1) testing input validation as a distinct capability rather than assuming this step is solved; 2) comprehensively evaluating model robustness across interfaces using five progressive tiers and five evaluation formats; 3) introducing negative controls to directly measure false alarm rates, revealing models' calibration deficiencies on normal inputs.

The design of MedObvious includes the following key steps:

• �� Task Construction: Create 1,880 tasks across five progressive tiers, from basic orientation/modality mismatches to clinically motivated anatomy/viewpoint verification and triage-style cues.

• �� Evaluation Formats: Include five evaluation formats to test model robustness across interfaces.

• �� Dataset Selection: Use various medical imaging datasets, including chest radiographs, CT, MRI, ultrasound, etc., defined by metadata filtering.

• �� Template Generation: Create tasks by inserting images from different categories or via controlled integrity violations (e.g., orientation change or physically inconsistent composite).

• �� Negative Controls: Introduce explicit negative controls to directly measure false alarm rates.

The experimental design includes evaluating 17 different VLMs using five evaluation formats to test their robustness across interfaces. Datasets include various medical imaging datasets such as chest radiographs, CT, MRI, ultrasound, etc. Baseline models include general-purpose models like GPT-4o, Flamingo, and LLaVA, as well as medical adaptations like LLaVA-Med and RadFM. Evaluation metrics include accuracy, false alarm rates, etc., with key hyperparameters including task number, evaluation formats, etc. Ablation studies are used to analyze the impact of different task tiers and evaluation formats on model performance.

Among the 17 evaluated VLMs, many models hallucinate anomalies on normal (negative-control) inputs, performance degrades when scaling to larger image sets, and measured accuracy varies substantially between multiple-choice and open-ended settings. The best mean accuracy reaches 63.2%, yet negative-control accuracy spans a wide range, indicating that false alarms on normal inputs remain common. There are large gaps between multiple-choice and open-ended variants, indicating strong format sensitivity. Ablation studies show significant differences in model performance across different task tiers and evaluation formats, particularly in set-level consistency over multi-image sets.

MedObvious provides a new benchmark for pre-diagnostic visual sanity checking in medical VLMs, emphasizing the importance of sanity checks. Direct application scenarios include input validation in medical imaging, clinical triage, etc. Prerequisites include model calibration on normal inputs, set-level consistency over multi-image sets, etc. Industry impact includes enhancing the safety and reliability of medical VLMs in clinical applications, reducing false alarms, and improving diagnostic accuracy.

One limitation of this study is the use of simplified grids for testing rather than full multi-series volumes and interactive viewers, which may limit the applicability of the results. Additionally, models exhibit high false-alarm rates on normal inputs, indicating that normal-case calibration is a distinct problem from diagnostic fluency. Computational costs are also a factor to consider, especially when evaluating on large-scale datasets and complex models. Future work should extend to full multi-series volumes and interactive viewer-based evaluation to better simulate real clinical environments and explore how to improve model calibration on normal inputs to reduce false alarms.