Enhancing Cardiovascular Disease Prediction through Multi-Modal Self-Supervised Learning
Multi-modal self-supervised learning enhances CVD prediction, improving balanced accuracy by 7.6% on limited data using ECG, CMR, and clinical info.
Key Findings
Methodology
This study integrates cardiac magnetic resonance (CMR), electrocardiogram (ECG), and clinical data, employing masked autoencoders (MAE) for ECG pretraining, ResNet50 for CMR image encoding, and multi-modal contrastive learning (MMCL) with CLIP loss for feature alignment. The process involves: • Using MAE to reconstruct masked ECG patches, extracting robust features; • Pretraining CMR images with SimCLR contrastive learning; • Applying MMCL to transfer knowledge across modalities; • Fine-tuning on myocardial infarction (MI) prediction. These steps enable effective multi-source feature fusion and knowledge transfer, addressing data scarcity.
Key Results
- On UK Biobank, the model achieved a balanced accuracy of 71.5% and AUC of 80%, outperforming single-modality and supervised models by 7.6%. The multi-modal pretraining significantly enhanced ECG and clinical feature representations, boosting predictive performance.
- Using only CMR images, the model reached 74% AUC, confirming its rich information content. The integration of MMCL facilitated cross-modal knowledge transfer, mitigating annotation limitations.
- Comparative analysis showed that combining MAE, MMCL, and clinical data yielded the best results, demonstrating the effectiveness of multi-modal self-supervised learning in medical prediction tasks.
Significance
This work addresses the challenge of limited annotated data in cardiovascular disease prediction by leveraging cross-modal transfer learning. It demonstrates how integrating diverse data sources—imaging, signals, and clinical info—can produce more comprehensive and accurate models. The approach enhances early diagnosis, supports personalized treatment, and reduces reliance on extensive labeled datasets. The methodology paves the way for scalable, cost-effective AI solutions in clinical settings, advancing both research and healthcare practice by enabling robust, generalizable models that operate effectively with limited supervision.
Technical Contribution
The study introduces a novel framework combining MAE-based ECG pretraining, SimCLR image contrastive learning, and CLIP-based cross-modal alignment. It innovatively merges these techniques to facilitate knowledge transfer across high-cost imaging and low-cost signals, enabling effective multi-source feature fusion. The approach enhances model robustness and generalization, especially under limited labeled data, and provides a scalable pipeline adaptable to various biomedical prediction tasks. The integration of these components represents a significant technical advancement over existing single-modal or shallow fusion methods.
Novelty
This research is the first to systematically apply multi-modal self-supervised learning for cardiovascular disease prediction, especially in low-label regimes. It uniquely combines MAE, SimCLR, and CLIP losses to achieve deep feature alignment across ECG, CMR, and clinical data. Unlike prior work focusing on single modalities or shallow fusion, this framework enables rich cross-modal knowledge transfer, setting a new standard for multi-source biomedical modeling and demonstrating superior predictive performance.
Limitations
- The model’s performance may decline on rare or extreme cases due to limited data diversity. Its reliance on high-quality pretraining models increases computational costs, limiting real-time deployment. Validation is currently confined to MI prediction; generalization to other CVDs remains unverified. Further, the approach requires substantial pretraining resources and may face challenges in clinical integration due to variability in data quality and acquisition protocols.
Future Work
Future directions include end-to-end joint training of all modalities, incorporating additional data sources like electronic health records and genomics, and deploying models in clinical environments for real-world validation. Enhancing computational efficiency and robustness, exploring unsupervised domain adaptation, and extending to other cardiovascular conditions are also promising avenues. These efforts aim to make the framework more scalable, interpretable, and applicable across diverse healthcare settings.
AI Executive Summary
Cardiovascular diseases (CVD) remain the leading cause of death worldwide, demanding early and accurate diagnosis. Traditional models often rely on single data modalities, limiting their ability to capture the complex physiological state of patients. Recent advances in deep learning have enabled automatic feature extraction, but the scarcity of annotated data hampers their broader clinical application. To address this, the present study introduces a multi-modal self-supervised learning framework that integrates cardiac magnetic resonance (CMR) images, electrocardiograms (ECG), and clinical data.
The approach begins with pretraining ECG encoders using masked autoencoders (MAE), which reconstruct masked patches of raw signals, extracting robust features resilient to noise. Simultaneously, CMR images are pretrained with SimCLR contrastive learning, enhancing their discriminative power. The core innovation lies in applying multi-modal contrastive learning (MMCL) with CLIP loss to align features across ECG, CMR, and clinical data, enabling effective knowledge transfer despite limited labels.
This comprehensive training pipeline culminates in fine-tuning the encoders on myocardial infarction (MI) prediction tasks. Results on the UK Biobank dataset show a 7.6% increase in balanced accuracy, reaching 71.5%, and an AUC of 80%, outperforming models trained on individual modalities or traditional supervised methods. The multi-modal framework demonstrates robustness and generalization, effectively leveraging unannotated data to improve predictive performance.
The significance of this work lies in its potential to transform cardiovascular diagnostics, especially in resource-limited settings. By combining diverse data sources and advanced self-supervised techniques, the model offers a scalable, cost-effective solution for early disease detection. Future work will explore broader applications, including other cardiovascular conditions and real-world clinical deployment, aiming to make AI-driven diagnostics more accessible and reliable.
Deep Analysis
Background
心血管疾病（CVD）是全球主要死亡原因，早期诊断和预防成为医学研究的重点。传统方法多依赖单一模态如影像或信号，难以全面反映疾病的复杂性。近年来，深度学习推动了多模态融合的发展，结合影像、信号和临床数据，提升了疾病预测的准确性。然而，标注数据稀缺限制了模型的泛化能力。UK Biobank等大规模数据库为多模态学习提供了丰富资源，Autoencoders（AEs）和对比学习（如SimCLR）已在多模态预训练中展现潜力，但深度融合仍待突破。
Core Problem
现有模型多局限于单一模态，难以充分利用多源信息，导致预测性能不足，特别是在数据标注有限的情况下。心血管疾病的多样性和复杂性要求模型能融合影像、信号和临床信息，捕获多层次特征。如何在有限标注数据下实现跨模态知识迁移、增强特征表达，成为核心难题。传统方法缺乏有效的模态对齐机制，难以实现信息的深度融合，限制了模型的泛化和临床应用。
Innovation
本研究提出结合掩码自编码器（MAE）、SimCLR预训练和CLIP对比学习的多模态自监督框架，创新点包括：• 利用MAE对ECG信号进行鲁棒特征学习，减少冗余；• 采用SimCLR增强CMR图像的判别能力；• 通过MMCL实现ECG、影像和临床特征的跨模态对齐，促进知识迁移；• 引入CLIP损失，优化不同模态特征空间的一致性。该框架突破了传统单模态限制，为心血管疾病的多模态预测提供新思路。
Methodology
- �� 使用MAE对ECG信号进行掩码重建，提取鲁棒特征；• 采用ResNet50在CMR图像上进行SimCLR预训练，增强图像特征表达；• 利用多模态对比学习（MMCL）结合ECG、CMR和临床特征，进行跨模态特征对齐，优化特征空间；• 最后在心肌梗死（MI）任务上微调编码器，提升预测性能。具体流程包括：• 输入ECG信号，划分patches，随机掩码后通过Transformer编码；• 图像通过增强和ResNet50编码，映射到特征空间；• 采用CLIP损失实现ECG、图像和临床特征的空间对齐；• 在有限标注数据上微调模型，增强泛化能力。
Experiments
采用UK Biobank数据集，涵盖ECG、CMR影像和临床信息，进行多模态预训练和微调。模型性能通过AUC和平衡准确率评估，验证不同模态融合效果。对比单模态、浅层融合和本方法，验证多模态自监督学习优势。超参数包括ECG掩码比75%、SimCLR对比损失、CLIP权重调节等。还进行消融实验，验证各组件贡献。模型在心肌梗死预测任务中表现优异，验证其有效性。
Results
在UK Biobank，模型达80%的AUC和71.5%的平衡准确率，比单模态模型提升7.6%。引入多模态对比学习后，ECG和临床特征的表达更丰富，模型泛化能力增强。单用CMR图像的AUC为74%，验证其信息丰富性。不同预训练策略显示MAE和MMCL结合效果最佳，验证多模态自监督的优势。模型在有限标注条件下表现优异，展现实际应用潜力。
Applications
该方法适用于心血管疾病早筛，尤其在临床资源有限地区。通过融合易得的ECG和临床数据，结合高成本影像，实现低成本高效预测。未来可扩展至其他疾病和个性化治疗，推动医疗AI普及。模型还可结合电子健康记录（EHR）和基因组信息，构建更全面的健康评估体系。
Limitations & Outlook
模型在极端病例或少数类别样本上表现仍有限，受限于数据多样性。多模态融合依赖高质量预训练模型，训练成本较高，实际部署存在挑战。验证主要在心肌梗死，泛化到其他心血管疾病仍需验证。未来需优化模型效率和泛化能力，增强临床适应性。
Plain Language Accessible to non-experts
想象你在准备一份复杂的菜肴，需要用到不同的食材：肉、蔬菜和调料。每种食材都代表一种数据类型，比如心电图、影像和临床信息。单独用一种食材做菜，味道可能不够丰富，但将它们合理搭配，就能做出美味佳肴。这个研究就像厨师用不同的食材搭配，利用一种特别的调料（自监督学习），让各种食材的味道融合得更好，最终做出一份既健康又美味的菜肴。这种方法能帮助医生更准确地判断心血管疾病，像厨师一样，把不同的食材融合在一起，做出最棒的菜。
ELI14 Explained like you're 14
想象你在玩拼图游戏，每块拼图代表不同的健康信息，比如心电图、心脏影像和你的身体数据。单独一块拼图可能不太能看出完整的画面，但如果你把所有拼图拼在一起，就能看到一个完整的心脏健康图像。这个研究就像用一种聪明的办法，把这些拼图用特殊的胶水粘在一起，让它们更好地配合，帮医生更快更准地知道你心脏是不是健康。它还用了一些“魔法”技巧，让不同的拼图变得更容易融合在一起，即使没有很多标记的图片，也能学会判断。这样一来，即使没有很多数据，医生也能用这个方法早早发现心脏问题，救命又省钱！
Abstract
Accurate prediction of cardiovascular diseases remains imperative for early diagnosis and intervention, necessitating robust and precise predictive models. Recently, there has been a growing interest in multi-modal learning for uncovering novel insights not available through uni-modal datasets alone. By combining cardiac magnetic resonance images, electrocardiogram signals, and available medical information, our approach enables the capture of holistic status about individuals' cardiovascular health by leveraging shared information across modalities. Integrating information from multiple modalities and benefiting from self-supervised learning techniques, our model provides a comprehensive framework for enhancing cardiovascular disease prediction with limited annotated datasets. We employ a masked autoencoder to pre-train the electrocardiogram ECG encoder, enabling it to extract relevant features from raw electrocardiogram data, and an image encoder to extract relevant features from cardiac magnetic resonance images. Subsequently, we utilize a multi-modal contrastive learning objective to transfer knowledge from expensive and complex modality, cardiac magnetic resonance image, to cheap and simple modalities such as electrocardiograms and medical information. Finally, we fine-tuned the pre-trained encoders on specific predictive tasks, such as myocardial infarction. Our proposed method enhanced the image information by leveraging different available modalities and outperformed the supervised approach by 7.6% in balanced accuracy.