Auditing Medical Vision-Language Models on Chest Radiographs: Estimating Reference Agreement Across Institutions
Strict institution-held-out testing found Beta-Binomial EB and target-only logistic regression best, with Brier scores of 0.0853 and 0.0855.
Key Findings
Methodology
The study evaluated MedGemma 4B, CheXagent 8B, and LLaVA-Med 7B on MIMIC-CXR, OpenI, and PadChest, covering six findings and narrative versus binary elicitation. More than 345,000 finding-level judgments were produced. Correctness was defined as agreement between the model judgment and the institutional reference label. Ten estimators were compared, including HD-Cal, hierarchical GLMM, HAS, target-only logistic regression, and constrained Beta-Binomial empirical Bayes.
Key Results
- Under strict development that entirely excluded the receiving institution, adaptive selection among seven estimators obtained mean Brier 0.1083, worse than fixed Beta-Binomial empirical Bayes at 0.0853 and target-only logistic regression at 0.0855.
- Reliability varied by institution, finding, prediction direction, and elicitation protocol. After Benjamini–Hochberg correction, 23 false-positive-rate and 24 false-negative-rate contrasts remained significant; the two primary models shared only Jaccard 0.35 of significant contrasts.
- The plug-in empirical-Bayes posterior-predictive interval had nominal 95% coverage but empirical coverage of only 87.0%, with poorer coverage at the hardest institution. Its apparent advantage over pooling was concentrated at one site.
Significance
The paper reframes deployment from average accuracy to the probability that a model agrees with a receiving institution’s own reference standard. This addresses a practical black-box problem: hospitals often receive only categorical outputs, yet can collect 25–200 local labels. The findings caution against treating cross-dataset averages as transportability guarantees and clearly distinguish label agreement from clinical correctness.
Technical Contribution
The work provides a deployment-time audit for categorical interfaces. It estimates finding-by-direction cells using local counts, cross-site additive and interaction logistic models, hierarchical GLMMs, and Beta-Binomial shrinkage. Algorithm 1 adds institution-held-out gating, Brier-based selection, interval reporting, and revalidation after model or prompt changes. It requires neither logits nor retraining, making it applicable to closed APIs and structured VLM endpoints.
Novelty
The novelty is primarily evaluative and operational rather than a new probability distribution. Classical empirical Bayes is combined with strict institution-held-out deployment testing, while adaptive selection is tested against simple fixed policies. Unlike much calibration work, which assumes continuous scores and reports aggregate metrics, this study audits categorical VLM outputs across institution, finding, direction, and interface, and shows that added selection complexity does not guarantee better transfer.
Limitations
- Only three retrospective datasets and six findings were studied. MIMIC-CXR uses CheXpert labels, whereas PadChest requires dictionary mapping, so label-pipeline differences are inseparable from institution effects.
- The estimand is agreement with institutional labels, not disease truth or clinical outcomes. The interval audit applies only to Beta-Binomial estimation and uses heavily overlapping repeated folds.
- LLaVA-Med narrative outputs did not vary with the image, so most inferential comparisons concern MedGemma and CheXagent.
Future Work
Future studies should include more hospitals, countries, devices, readers, and prospective cases, using expert consensus and patient outcomes as additional endpoints. Methodological work should improve hierarchical priors, predictive interval calibration, and monitoring under model, prompt, language, and decoding changes. Patient-level disjoint validation and continuous post-deployment surveillance are also needed.
AI Executive Summary
Medical vision-language models are increasingly used to structure chest-radiograph findings, yet many deployment interfaces return only “yes” or “no.” A hospital therefore cannot tell how much confidence to place in an individual answer. This study asks a narrower but operationally important question: can a receiving institution estimate how often a model agrees with its own reference labels using only a small local labeling budget? The authors stress that reference agreement is not clinical correctness.
MedGemma 4B, CheXagent 8B, and LLaVA-Med 7B were evaluated on MIMIC-CXR, OpenI, and PadChest, across six findings, two elicitation protocols, and more than 345,000 finding-level predictions. The study compared HD-Cal logistic models, hierarchical GLMMs, HAS, target-only logistic regression, and Beta-Binomial empirical Bayes. With 25–200 local labels, strict leave-one-institution-out testing showed that adaptive selection among seven estimators achieved Brier 0.1083, worse than fixed Beta-Binomial empirical Bayes at 0.0853 and target-only logistic regression at 0.0855.
The results expose substantial deployment dependence: institutions, findings, prediction directions, and interfaces all changed agreement patterns. The two primary models shared only Jaccard 0.35 of significant contrasts, while a nominal 95% predictive interval covered held-out counts only 87.0% of the time. Hospitals should therefore re-audit every model and interface locally rather than importing a global reliability number. The study also provides a reproducibility warning: changing scikit-learn versions altered calibration error by as much as 0.23 in some small-budget MIMIC-CXR settings.
Deep Analysis
Background
Medical VLMs such as LLaVA-Med and MedGemma generate radiology text, while CheXagent provides a structured binary chest-radiograph interface. Prior work has emphasized hallucination, aggregate AUC, or domain-shift degradation. However, deployment teams need a local estimate of how often a categorical judgment matches their reference process. Standard post-hoc calibration, selective prediction, and conformal methods generally assume continuous scores, which may be unavailable in a black-box API.
Core Problem
Let ŷ be the model judgment and y the institutional reference label. The target is P(c=1), where c=1[ŷ=y], stratified by institution, finding, and prediction direction. The challenge is to estimate these cells with few labels despite prevalence shift, label-pipeline differences, prompt effects, and limited independence. Selection must also prevent the receiving institution from influencing estimator development.
Innovation
First, the study maps model reliability along the institution–finding–direction axis rather than reporting only global scores. Second, it evaluates adaptive estimator selection under a strict design in which the receiving institution is absent from development, priors, and selection. Third, it compares local target estimation with cross-site pooling under identical label budgets. Fourth, it audits posterior-predictive coverage as an empirical endpoint and reports solver-version sensitivity, an often ignored source of reproducibility variation.
Methodology
- �� Data: MIMIC-CXR supplied 3,066 evaluable studies, OpenI 3,851, and PadChest 4,998; findings were atelectasis, cardiomegaly, consolidation, edema, pleural effusion, and pneumothorax.
- �� Models: MedGemma and LLaVA-Med used parsed narrative responses; CheXagent returned Yes/No judgments. All were tested with narrative and binary elicitation.
- �� Estimators: HD-Cal used σ(β0+βr+βd+βk+βd,k), L-BFGS, and C=1. The GLMM used crossed random effects with HalfNormal(1) hyperpriors. Beta-Binomial empirical Bayes used posterior mean (y+α)/(n+α+β).
- �� Evaluation: each institution was held out in turn; local budgets were 0, 25, 50, 100, and 200 labels, repeated ten times. Brier score and 10-bin ECE were reported.
Experiments
The two primary models contributed 105,362 narrative predictions, with a further 60,070 from LLaVA-Med. Ten fixed estimators were benchmarked, while the adaptive selector had seven candidates because HAS variants require two source institutions. Each target model, institution, budget, and repetition produced a deployment decision, yielding 240 evaluations. Predictive intervals used exact equal-tailed 95% Beta-Binomial integer quantiles and required at least 20 test observations per cell.
Results
Adaptive selection reached mean Brier 0.1083, versus 0.0853 for Beta-Binomial empirical Bayes and 0.0855 for target-only logistic regression. The 0.0003 difference between the two leaders was smaller than solver sensitivity, and each led in roughly half the settings. Pooling advantages were concentrated at one institution and were not confirmatory after institution clustering. Coverage failures were concentrated at the hardest site and in predictions asserting a finding.
Applications
Before deployment, a radiology department can sample 25–200 local historical cases, record model judgments and reference labels, and estimate reliability by finding and direction. The procedure needs no model internals, logits, or retraining. It can support human-review thresholds, API acceptance testing, model-version audits, and comparisons among prompts or decoding protocols across hospital sites.
Limitations & Outlook
The study has few institutions, heterogeneous label pipelines, and incomplete independence at study and patient levels. MIMIC-CXR’s uncertain or unmentioned labels were excluded, so results apply to the definite-label subset. Agreement may reproduce systematic reference bias rather than medical truth. The interval analysis covers only one estimator, and retrospective data cannot establish prospective safety, clinical utility, or effects on radiologist workflow.
Plain Language Accessible to non-experts
Imagine hiring an assistant to inspect X-ray images at a new hospital. The assistant answers only “present” or “absent,” never explaining how sure it is. A national exam score might tell you how the assistant performed elsewhere, but it cannot guarantee performance in your hospital: patients differ, staff use different labeling habits, and questions may be phrased differently.
The researchers therefore give the new hospital a small set of old cases with known local answers. They record what the assistant said, separate different findings, and also separate answers that claim a finding is present from answers that deny it. Several bookkeeping strategies are compared. Some rely mainly on local cases; others borrow experience from previous hospitals; one tries to choose automatically among them.
The surprising result is that automatic choice did not win. Two simpler strategies had Brier scores around 0.085, while adaptive selection scored 0.1083; lower is better. Reliability changed substantially across hospitals and question formats. The practical lesson is simple: test the assistant locally before trusting it, repeat the test after changing the model or interface, and remember that matching an old label is not the same as discovering the medical truth.
ELI14 Explained like you're 14
Imagine an AI teammate in a video game who scans the map and says, “Enemy here!” or “No enemy here!” It never shows a confidence meter. If you move to a new map, should you trust its old win rate? Not automatically—new maps have different layouts, rules, and opponents.
The researchers did something similar with chest X-rays. They gave the AI a small set of cases from a new hospital where the answers were already known. They checked six kinds of findings and separated “the AI said yes” from “the AI said no.” Then they compared several statistical coaches: one mostly trusted the new map, one borrowed experience from other maps, and another tried to pick the best coach automatically.
The twist is that automatic picking did not win! The best simple methods had Brier scores of 0.0853 and 0.0855, while the selector scored 0.1083; lower is better. Different hospitals produced different error patterns, and even the wording of the question mattered.
So what should a hospital do? Run a small local test before launch, repeat it whenever the model, prompt, or interface changes, and use the results to decide when a human should check the answer. One more important warning: matching the hospital’s old labels means “similar to the old answers,” not necessarily “medically correct.”
Glossary
Reference agreement
The probability that a model judgment matches an institution’s reference label. Technically, the paper defines correctness as c=1[ŷ=y], which is distinct from clinical truth.
It is the central estimand for every reliability estimator.
Beta-Binomial empirical Bayes
A binomial cell-rate model with a Beta prior updated by local counts. It shrinks unstable small-sample estimates toward a source-informed prior.
It achieved mean Brier 0.0853.
HD-Cal
A logistic calibration model with direction, finding, institution, and institution-by-finding interaction indicators. The interaction represents site-specific finding heterogeneity.
It was fitted with L-BFGS and L2 regularization, C=1.
Brier score
The mean squared difference between predicted reliability and observed correctness; lower is better. Unlike binned ECE, it is a proper score without calibration bins.
It governed estimator selection and headline comparisons.
Strict institution-held-out validation
The receiving institution is excluded from development, prior fitting, and estimator selection. Other institutions simulate the development environment.
This design tests genuine transportability rather than ordinary target holdout.
Posterior-predictive coverage
The fraction of future observable test counts contained by predictive intervals. It evaluates whether uncertainty intervals protect against future sampling variation.
The nominal 95% interval covered only 87.0% empirically.
Open Questions Unanswered questions from this research
- 1 It remains unknown whether the two simple leaders remain stable across many more hospitals, countries, devices, and labeling regimes. Large prospective multi-reader studies are needed.
- 2 Reference labels may contain systematic bias. Expert consensus, repeated readings, and clinical outcomes are required to separate label agreement from medical correctness.
- 3 Model checkpoints, prompts, languages, and decoding can alter categorical outputs. Low-cost continuous monitoring for these changes remains unresolved.
Applications
Immediate Applications
Pre-deployment local audit
A radiology department can sample 25–200 local cases, record model judgments and reference labels, estimate finding-by-direction reliability with Beta-Binomial or target-only logistic regression, and set human-review thresholds before clinical use.
Version and prompt monitoring
Whenever a checkpoint, system prompt, language, parser, or decoding setting changes, repeat the local sample and compare Brier, ECE, and cell-level agreement instead of inheriting reliability from the previous interface.
Long-term Vision
Multi-site reliability governance
Hospitals could maintain an audit registry indexed by institution, finding, direction, and interface version, linking model agreement with expert consensus and outcomes to support continuous monitoring, risk tiers, and rollback decisions.
Abstract
Vision-language models return structured chest-radiograph findings through interfaces exposing no confidence score, so a receiving institution cannot read off how far to trust an individual judgment. Whether agreement with an institution's reference standard transfers across sites, findings, prediction directions and question formats is largely unmeasured. We evaluated three generative vision-language models on three institutional chest-radiograph corpora and six findings under two elicitation protocols, comprising more than 345,000 finding-level predictions, and estimated finding-by-direction reference agreement at a receiving institution from a small budget of local labels. Estimation strategies were then stress-tested under repeated strict institution-held-out evaluation. Under evaluation excluding the receiving institution from development entirely, adaptive selection among the seven estimators that design admits did not improve on simple fixed alternatives: it achieved a mean Brier score of 0.1083, against 0.0853 for always using a Beta-Binomial empirical-Bayes estimator and 0.0855 for a target-only logistic model. Those two differ by 0.0003, less than this family's own sensitivity to a change of solver version, and each leads in about half the settings, so no default can be recommended. Their advantage over estimators pooling across institutions was concentrated at one site and not confirmatory once clustered by institution, and a plug-in empirical-Bayes posterior-predictive count interval at a nominal 95% level covered 87.0%, less at the hardest institution. Reference agreement therefore has to be re-evaluated per site and per interface; these results concern agreement with institutional labels, not clinical correctness.