Improving Clinical Foundation Models with Multi-modal Learning and Domain Adaptation for Chronic Disease Prediction

Modelling patient trajectories from longitudinal electronic health records (EHRs) is crucial for early chronic disease prediction. Foundation models (FMs), benefiting from the computational power and generalization abilities, offer a promising direction towards understanding patient health progression. However, key challenges of adopting FMs in clinical decisions remain in (1) incorporating multi-modal EHR data into an FM effectively for unified patient representations and (2) ensuring model generalizability across various clinical domains with distribution shifts. To address these challenges, we propose MsHeCare, an FM-based, two-stage learning framework integrating multi-modal representation learning and multi-source domain adaptation (MSDA). In the pretraining stage, MsHeCare performs self-supervised contrastive learning and masked language modelling tasks to mitigate semantic biases across text-based diagnostic and treatment sequences. A cross-attention mechanism further fuses these temporal features with structured static demographic information, enhancing personalized patient representations. In the fine-tuning stage, MsHeCare incorporates an MSDA framework with a novel source importance estimation method, facilitating adaptive knowledge transfer across domains and improving model generalizability. Experiments on two real-world EHR datasets demonstrate that MsHeCare significantly outperforms single-domain baselines and state-of-the-art MSDA methods. Furthermore, we validate the robustness of MsHeCare across varying targetdomain data sizes and demonstrate its alignment with clinical practices through case studies, underscoring its potential for multi-modal, domain-adaptive predictive healthcare systems.