Multimodal Spatiotemporal Signature Integrating Pathomics and Longitudinal Magnetic Resonance Imaging Predicts Response to Neoadjuvant Therapy and Prognosis in Rectal Cancer

PURPOSE The purpose of this study was to accurately identify patients with locally advanced rectal cancer (LARC) who are likely to achieve pathologic complete response (pCR) after neoadjuvant therapy (NAT). This study develops a Multimodal Spatiotemporal Attentive Fusion Network (MSTAF-Net) to predict pCR and derives a Multimodal Spatiotemporal Signature (MSTAF-MSS) for disease-free survival (DFS) and explored its association with immune-related transcriptomic features. METHODS This retrospective multicenter study included 642 patients with LARC. Longitudinal multiparametric magnetic resonance imaging (MRI) acquired before and after NAT and pretreatment hematoxylin and eosin–stained whole-slide images were collected. A dual-stream MSTAF-Net was designed to integrate longitudinal MRI features and pathomics features. Model performance was evaluated using receiver operating characteristic curves and survival analysis. Transcriptomic analyses were conducted to investigate the biologic correlates of the MSTAF-MSS and its association with immune-related transcriptomic features. RESULTS The multimodal transformer fusion model achieved AUC values of 0.894 (95% CI, 0.847 to 0.942) in the internal validation cohort and 0.865 (95% CI, 0.786 to 0.943) in the external validation cohort, outperforming single-modality model. The MSTAF-MSS enabled effective risk stratification, with low-risk patients showing significantly longer DFS than high-risk patients (log-rank P < .05). Cox regression analyses identified MSTAF-MSS as an independent predictor of DFS in multivariate models (hazard ratio, 0.31 [95% CI, 0.13 to 0.73], P = .007). Distinct patterns of immune cell infiltration were observed between MSTAF-MSS–defined groups. CONCLUSION The proposed MSTAF-Net integrates pathomics and longitudinal multiparametric MRI to capture spatial heterogeneity and treatment-related temporal dynamics of tumors. It demonstrates robust performance in predicting response to NAT and enables prognostic risk stratification. Furthermore, transcriptomic analysis supports potential biologic relevance of the model in associations with immune-related tumor biology.