CRAFT: Clinical Reward-Aligned Finetuning for Medical Image Synthesis

Medical image synthesis has evolved from traditional generative models to deep learning-based approaches, notably diffusion models like Stable Diffusion, which produce high-fidelity natural images. Researchers have adapted these models to medical domains such as chest X-rays, MRI, and dermatology, aiming to generate realistic data for training and augmentation. Prior works include Ho et al.'s latent diffusion, Ali et al.'s domain-specific tuning, and prompt-based methods like Textual Inversion and LoRA. Despite progress, these methods primarily optimize for visual realism, often neglecting clinical correctness, leading to hallucinations or missing key diagnostic features. The challenge remains to develop evaluation metrics and training strategies that ensure generated images are both visually plausible and clinically valid, especially under limited labels and high-stakes settings.

Current generative models lack reliable metrics for clinical relevance, often producing images that look realistic but do not contain necessary pathological features. This disconnect hampers their utility in clinical training, decision support, and data augmentation. The core problem is how to quantify and optimize the clinical alignment of synthetic images, especially when labeled data is scarce. Existing metrics like FID or Inception Score do not reflect pathology-specific criteria, and manual expert review is impractical at scale. Therefore, a systematic, scalable approach to evaluate and improve clinical fidelity is urgently needed, alongside methods that can incorporate structured clinical knowledge into the training process.

This work introduces a multi-dimensional clinical alignment score (CAS) that combines text-image similarity (VDC), checklist satisfaction (CCS), diagnostic discriminability (DD), and semantic similarity (SFS). It leverages foundation models—MLLMs for prompt generation and VLMs for reward computation—to create a comprehensive, differentiable reward signal. The core innovation is the reward-guided fine-tuning framework CRAFT, which employs DRaFT-K to update only LoRA parameters, efficiently aligning the generative model with clinical standards. Unlike prior approaches limited to visual fidelity or single-criterion alignment, this method systematically integrates structured clinical knowledge and semantic consistency, enabling the generation of diverse, clinically plausible images with minimal manual intervention.

• �� Use MLLMs (e.g., MedGemma) to generate instance-specific visual prompts (tep) and disease-level clinical checklists (tc) from sparse labels.
• �� Construct structured clinical criteria based on disease characteristics, encoded as checklists.
• �� Fine-tune a pretrained diffusion model (e.g., Stable Diffusion v2.1) by incorporating the enriched prompts and clinical checklists.
• �� Decode the generated latent representations into images via VAE.
• �� Evaluate generated images with a frozen VLM (e.g., MedSigLIP), computing rewards for visual description consistency (cosine similarity between image and prompt embeddings), clinical criteria satisfaction (similarity with checklist), diagnostic discriminability (linear probe on embeddings), and semantic feature similarity (image-image cosine similarity with real references).
• �� Optimize the model using DRaFT-K, backpropagating reward gradients through the last K denoising steps, updating only LoRA parameters.
• �� Aggregate multiple stochastic trajectories to stabilize training.
• �� Validate via expert review, checklist pass rates, and tail distribution analysis.

The authors evaluate CRAFT on four datasets: Fitzpatrick17k, CheXpert, BreakHis, and ORIGA, comparing against baselines like TI+LoRA and DPO. Metrics include CAS, downstream classification accuracy, expert preference, and checklist pass rate. Hyperparameters such as λdiff and λcam are tuned for reward balance. Ablation studies assess the impact of prompt enrichment and reward components. The experiments focus on measuring improvements in clinical alignment, diversity, and tail failure reduction, with extensive qualitative and quantitative analyses to validate robustness and generalization.

CRAFT consistently outperforms baselines, with CAS scores reaching 0.515 on CheXpert and 0.417 on Fitzpatrick17k, surpassing previous methods by over 20%. It notably reduces the proportion of low-alignment samples by up to 34%, indicating fewer hallucinations. Expert evaluations favor CRAFT-generated images, and downstream classification tasks show accuracy gains of 1-3%. Tail analysis confirms a significant decrease in clinically implausible samples, demonstrating enhanced reliability. These results establish CRAFT as a robust framework for clinically aligned medical image synthesis.

The method can be applied in data augmentation for training diagnostic models, especially in rare disease scenarios with limited data. It also supports the creation of high-quality, clinically plausible synthetic datasets for privacy-preserving sharing. Additionally, CRAFT can assist in educational tools, training clinicians, and validating AI algorithms by providing reliable synthetic images that meet clinical standards. Future integration with multi-modal data and real-time clinical feedback could further enhance its utility.

Dependence on foundation models introduces biases inherent in pretraining data, potentially affecting clinical accuracy. Computational complexity remains high due to multi-modal inference and reward calculation. The current static checklist approach may not capture complex, dynamic clinical scenarios. Further research is needed to improve efficiency, incorporate real-world feedback, and extend to broader clinical tasks. Addressing these limitations will be crucial for practical deployment.