SAMMS: Multi-modality Deep Learning with the Foundation Model for the Prediction of Cancer Patient Survival

Cancer survival prediction is pivotal in tailoring individualized treatment strategies and guiding clinician decision-making. Yet, existing methodologies grapple with efficiently harnessing the intricate distribution of medical data spanning various modalities. In response, we present SAMMS, an advanced multi-omics multimodal deep learning framework tailored for survival prediction. SAMMS leverages the robust image segmentation model, "Segment Anything" to adeptly characterize pathological images. This prowess is further enhanced by integrating multi-omics data and clinical insights, facilitating holistic modeling across a diverse modal spectrum. The framework weaves a modality-specific subnetwork with a cross-modality common subnetwork, meticulously capturing intra-modality nuances and inter-modality correlations. SAMMS eclipsed its contemporaries by delivering remarkable performance on TCGA’s LGG and KIRC tumor datasets. A battery of analyses underscored SAMMS’s unparalleled capability to distill multifaceted insights from multimodal datasets, yielding richer and more integrative multimodal representations. Such strides promise significant advancements in cancer survival analytics, bolstering the precision and efficacy of patient-centric treatments, disease oversight, and clinical decision processes.