A multimodal and temporal foundation model for virtual patient representations at healthcare system scale

The rapid development of modern medicine has led to the generation of vast amounts of multimodal data, often siloed across different systems, creating so-called data silos. This data isolation limits the integration and utilization of clinical information, affecting the efficiency of medical research and clinical decision-making. Traditional medical data analysis methods typically focus on single-modality data, failing to fully exploit the potential of multimodal data. In recent years, with advancements in machine learning and artificial intelligence, researchers have begun exploring how to integrate multimodal data into a unified framework to improve prediction and decision accuracy. However, existing methods still face limitations in the breadth and temporal depth of data integration, making it difficult to meet practical application needs. The Apollo model was proposed to address this issue by integrating multimodal and temporal sequence data, providing new possibilities for medical research and clinical applications.

The core problem is how to integrate the vast amounts of multimodal and temporal sequence data generated in modern medicine into a unified patient representation. The challenges include the diversity and complexity of the data, such as the heterogeneity between different modalities, the high dimensionality of the data, and the long span of temporal sequences. Additionally, the dispersion and incompleteness of the data increase the difficulty of integration. Existing methods typically only handle single-modality data or experience information loss when integrating multimodal data. This not only limits the predictive capabilities of the models but also affects the accuracy of clinical decision-making. Therefore, developing a model capable of effectively integrating multimodal and temporal sequence data is crucial for improving the efficiency of medical research and clinical applications.

The core innovations of the Apollo model lie in its breakthroughs in integrating multimodal and temporal sequence data. First, Apollo can handle 25 billion records from 7.2 million patients, covering 28 distinct medical modalities and 12 major medical specialties, which is unprecedented in terms of data scale and diversity. Second, Apollo learns a unified representation space that integrates over 100,000 unique medical events, images, and clinical text, forming an 'atlas of medical concepts.' This atlas provides a computational substrate for modeling entire patient care journeys, compressing sequences of structured and unstructured events into virtual patient representations. Compared to existing methods, Apollo not only shows significant improvements in data integration breadth and temporal depth but also demonstrates unique advantages in representation learning and predictive capabilities.

The implementation of the Apollo model includes several key steps:

• �� Data Collection and Preprocessing: Over three decades of longitudinal hospital records from a major US hospital system were collected, including 25 billion records from 7.2 million patients. The data covers 28 distinct medical modalities and 12 major medical specialties.

• �� Representation Learning: A unified representation space is learned, integrating over 100,000 unique medical events, images, and clinical text, forming an 'atlas of medical concepts.'

• �� Model Training: The Apollo model is trained using multimodal and temporal sequence data to generate virtual patient representations.

• �� Task Evaluation: 322 prognosis and retrieval tasks were created from a held-out test set of 1.4 million patients to assess the generalized clinical forecasting potential of Apollo embeddings.

The experimental design includes creating 322 prognosis and retrieval tasks from a held-out test set of 1.4 million patients. Specifically, the experiments include the following aspects:

• �� Dataset: Utilizes 25 billion records from 7.2 million patients, covering 28 distinct medical modalities and 12 major medical specialties.

• �� Baselines: Compared with existing state-of-the-art methods to evaluate the performance improvements of the Apollo model.

• �� Evaluation Metrics: Includes predicting new disease onset risk, disease progression, treatment response, risk of treatment-related adverse events, and hospital operations endpoints.

• �� Hyperparameters: Model hyperparameters are tuned to optimize performance across different tasks.

• �� Ablation Studies: Ablation studies are conducted to evaluate the contribution of each component to the model's performance.

Experimental results show that the Apollo model performed exceptionally well across 322 prognosis and retrieval tasks. Specifically, Apollo was able to predict new disease onset risk with significantly improved accuracy up to five years in advance. In 95 tasks, Apollo predicted new disease onset risk; in 78 tasks, it predicted disease progression; in 59 tasks, it predicted treatment response; in 17 tasks, it predicted the risk of treatment-related adverse events; and in 12 tasks, it predicted hospital operations endpoints. Additionally, Apollo demonstrated semantic similarity search capabilities in 61 retrieval tasks and showed potential as a multimodal medical search engine. Using feature attribution techniques, the study showed that model predictions align with clinically interpretable multimodal biomarkers.

The application scenarios of the Apollo model include:

• �� Clinical Prediction: By predicting new disease onset risk, disease progression, and treatment response, it helps doctors formulate more effective treatment plans.

• �� Medical Research: By integrating multimodal data, it supports more comprehensive medical research, revealing potential mechanisms of diseases.

• �� Healthcare Management: By predicting hospital operations endpoints, it helps hospitals optimize resource allocation and improve operational efficiency.

• �� Personalized Medicine: By generating virtual patient representations, it supports the formulation of personalized medical plans, improving patient treatment outcomes.

Although Apollo excels in integrating multimodal data, it may perform poorly when handling extremely sparse or missing data, as the model relies on complete event sequences for accurate predictions. Additionally, the model's computational complexity is high, especially when processing large-scale datasets, which may limit its application in resource-constrained environments. Future research directions include optimizing the computational efficiency of the Apollo model for application in resource-constrained environments. Additionally, further exploration of integrating more types of biomedical data, such as genomic data, into the model to enhance prediction accuracy and applicability is planned.