Frozen Brain-MRI Foundation Models Are Site Fingerprints
This study reveals that frozen brain MRI foundation models encode site information as a linearly decodable fingerprint, surpassing clinical variables, rooted in low-level image statistics.
Key Findings
Methodology
Using ABIDE-I and ABIDE-II datasets, three architectures (Swin, ViT, ResNet), and random initialization, the study probes deep features for site, clinical, and demographic decodability. Linear and nonlinear probes evaluate the extent of site encoding. INLP and ComBat are employed to remove site bias, confirming its intrinsic nature and linear accessibility. Cross-cohort validation ensures robustness.
Key Results
- Across all models and layers, site decodability reaches ~0.9, far exceeding clinical/demographic variables (~0.4-0.5). Random models alone achieve similar high decodability, indicating the fingerprint stems from low-level image statistics rather than pretraining. In ABIDE-II, similar results are replicated, with site decodability at 0.88-0.95. Residualizing population covariates leaves site decodability unchanged, confirming acquisition-driven bias. Post hoc removal via INLP and ComBat significantly reduces site information (from 0.94 to near zero), but in dense segmentation, the entanglement with anatomy complicates bias removal without performance loss.
Significance
The findings highlight a fundamental bias embedded in frozen MRI models, raising concerns about data sharing, federated learning, and model fairness. Recognizing that low-level image statistics encode site information urges the community to develop more robust, bias-aware models. This work emphasizes the importance of auditing pretrained models for hidden confounds, ensuring reliable and equitable neuroimaging analyses across diverse acquisition environments.
Technical Contribution
This research pioneers a systematic quantification of site information in frozen MRI models, demonstrating its linear accessibility and intrinsic origin from low-level image features. It shows that architectures inherently encode site bias, independent of pretraining, across multiple datasets and model families. The combination of INLP and ComBat provides practical tools for bias mitigation, revealing the linear entanglement with anatomical signals, especially in dense segmentation tasks. These insights advance understanding of model biases and open avenues for bias correction in large-scale neuroimaging pipelines.
Novelty
This is the first comprehensive study to demonstrate that frozen brain MRI foundation models inherently encode site information as a linearly decodable fingerprint, rooted in low-level image statistics. It shifts the focus from pretraining-induced biases to architecture-intrinsic properties, challenging assumptions in current model deployment and data harmonization practices. The cross-architecture and cross-cohort validation further establish the universality of this phenomenon.
Limitations
- The study is limited to ABIDE datasets; generalization to other MRI types or acquisition protocols remains to be validated. The linear decodability does not fully address the entanglement with anatomical signals, especially in dense segmentation tasks, where bias removal can impair performance. The root causes of architecture-intrinsic bias are not fully explored, necessitating further investigation into hardware and protocol influences.
Future Work
Future research should explore bias detection and correction across diverse imaging modalities and clinical tasks. Developing architecture designs less sensitive to low-level statistics could mitigate inherent biases. Integrating hardware and protocol metadata may help trace bias origins. Extending bias removal techniques to multi-task and federated learning scenarios will enhance model fairness. Long-term, these efforts aim to establish transparent, equitable neuroimaging pipelines suitable for clinical deployment.
AI Executive Summary
In recent years, multi-center neuroimaging studies have become essential for achieving large sample sizes, but they face the challenge of systematic biases introduced by different scanners, protocols, and populations. Traditional harmonization methods like ComBat have been used to adjust derived features, yet they do not address biases embedded directly within model representations. This study investigates the extent to which frozen brain MRI foundation models encode site-specific information.
Using two independent cohorts, ABIDE-I and ABIDE-II, the authors probe the deep features extracted by three architectures—Swin, ViT, and ResNet—and find that site information is highly linearly decodable at deep layers, with accuracy around 0.9. Interestingly, even randomly initialized models exhibit similar site decodability, indicating that the bias originates from the architecture's sensitivity to low-level image statistics rather than pretraining. This is further confirmed by the ability to decode site directly from raw downsampled images.
The study employs linear and nonlinear probes, along with bias removal techniques like INLP and ComBat, to demonstrate that the site fingerprint is intrinsic and can be effectively mitigated post hoc. However, in dense segmentation tasks, the entanglement of site and anatomy signals complicates bias removal, potentially affecting performance. These findings underscore the importance of auditing and correcting for site effects in pretrained models, especially in federated and multi-center settings.
Overall, this work reveals a fundamental bias embedded in frozen MRI models, driven by low-level image features, and provides practical tools for bias detection and mitigation. It calls for increased vigilance in model deployment and data sharing, ensuring fair and reliable neuroimaging analyses across diverse acquisition environments. The insights gained pave the way for more robust, bias-aware models in neuroimaging research and clinical practice.
Deep Dive
Abstract
Frozen foundation-model (FM) embeddings are increasingly used as off-the-shelf brain-MRI representations, on the assumption that they capture anatomy. We audit what they actually encode and find that acquisition site is a large, intrinsic component of the representation. Across two independent cohorts (ABIDE-I, ABIDE-II), three frozen 3-D encoders (brain-pretrained, CT-pretrained, and randomly initialized), and every network depth, site is linearly decodable at roughly 0.9 balanced accuracy at deep layers, exceeding the decodability of every clinical or demographic variable (sex, age, autism diagnosis) at every layer. The effect is intrinsic rather than learned: a randomly initialized encoder is already a ~0.9 site classifier on both cohorts and across three architecture families (Swin, ViT, ResNet), and site is decodable at ~0.95 directly from the raw downsampled image with no encoder, so the fingerprint reflects low-level image statistics that any encoder preserves rather than a product of pretraining. Residualizing measured population covariates leaves site decodability essentially unchanged, indicating an acquisition- rather than population-driven effect. A nonlinear probe matches the linear one, so the fingerprint is fully linearly accessible. The site subspace is removable post hoc by iterative null-space projection or ComBat (site decodability 0.94 -> 0.07/0.00), and is a site-attribution concern for shared or federated embeddings; but for dense segmentation this removal is not free, because site and anatomy occupy an entangled linear subspace (a matched-rank random-direction projection is Dice-neutral, whereas removing the site subspace is destructive). We recommend site-audited use of frozen brain-MRI FMs and release an open audit toolkit.