Integrating deep learning with multimodal MRI habitat radiomics: toward personalized prediction of risk stratification and androgen deprivation therapy outcomes in prostate cancer

OBJECTIVES: Androgen deprivation therapy (ADT) is essential for treating prostate cancer (PCa) but is limited by tumor heterogeneity. This study develops a non-invasive multiparametric Magnetic Resonance Imaging (mpMRI) radiomics framework to predict ADT response and improve risk stratification. MATERIALS AND METHODS: A cohort of 550 ADT-treated PCa patients from three centers was analyzed. Patients were randomly divided into training (n = 270) and internal validation (n = 115) cohorts. An external test cohort (n = 165) from Centers 2 and 3 was used for generalizability. Radiomics models based on T2-weighted and diffusion-weighted imaging (DWI), habitat radiomics, and a 3D Vision Transformer (ViT) deep learning model were developed. Ensemble integration of these models was performed, with SHapley Additive exPlanations (SHAP) used for interpretability. Predictive performance was evaluated using receiver operating characteristic (ROC) curves and area under the curve (AUC). RESULTS: Habitat radiomics outperformed conventional radiomics in Gleason score stratification. For predicting ADT treatment efficacy, the radiomics model achieved AUCs of 0.969 (training), 0.767 (internal validation), and 0.771 (test). The habitat model showed AUCs of 0.987, 0.849, and 0.820, while the ViT model achieved AUCs of 0.831, 0.805, and 0.796. The ensemble model reached the highest AUC of 0.886. SHAP analysis shows that the ViT model contributes most to the combined model, followed by the habitat model, with the radiomics model contributing the least. CONCLUSION: mpMRI-based habitat radiomics enables precise risk stratification in PCa. Integrated with conventional radiomics and deep learning, it forms a robust framework for predicting ADT response and guiding personalized treatment. CRITICAL RELEVANCE STATEMENT: This study demonstrates that integrating habitat radiomics with deep learning improves the prediction of androgen deprivation therapy response in PCa, advancing personalized radiological decision-making through interpretable multi-model analysis of tumor microenvironment heterogeneity. KEY POINTS: Multi-model integration of habitat radiomics and 3D Vision Transformer achieves superior prediction for ADT response compared to conventional methods. Habitat radiomics outperforms traditional radiomics in Gleason score stratification. SHAP analysis provides clinical interpretability, identifying key model linked to ADT outcomes for actionable insights.