Transformer-based multimodal fusion framework for predicting postoperative cognitive improvement in glioma: integrating radiomics and pathomics

OBJECTIVE: To address the challenge of predicting postoperative cognitive dysfunction in glioma patients, this study aims to develop and validate a Transformer-based multimodal fusion framework that integrates preoperative MRI radiomics and whole-slide histopathological pathomics for personalized cognitive outcome prediction. METHODS: A retrospective multicenter study was conducted involving 189 glioma patients. We proposed a novel Transformer-based model to synergistically fuse 2.5D radiomic features from preoperative MRI and pathomic features from whole-slide images. Model performance was evaluated across training, validation, and independent test cohorts using AUC values, with comparison to unimodal approaches and traditional ensemble methods. Decision curve analysis and Grad-CAM visualizations were employed to assess clinical utility and model interpretability. RESULTS: The proposed model achieved AUC values of 0.973 (training), 0.860 (validation), and 0.829 (independent test), significantly outperforming unimodal approaches (radiomics AUC: 0.596; pathomics AUC: 0.658) and traditional ensemble methods (AUC: 0.675). Pathomics features demonstrated superior predictive performance (AUC difference: 0.177-0.233). The model showed high specificity (0.921 in validation) and maintained reasonable specificity (0.737) in the test cohort. The decision curve analysis indicated a superior net benefit across a range of clinically reasonable threshold probabilities (approximately 20% to 80%), with a potential reduction in unnecessary interventions compared to the "treat-all" or "treat-none" strategies. CONCLUSION: The Transformer-based multimodal fusion framework provides an effective solution for predicting postoperative cognitive improvement in glioma patients, demonstrating superior performance against conventional methods. By bridging computational innovation with clinical practice, this approach presents a promising proof-of-concept for a multimodal framework that could, upon future prospective validation, serve as a tool for optimizing rehabilitation strategies and improving functional independence in glioma survivors.