Deep learning-based fusion of nuclear segmentation features for microsatellite instability and tumor mutational burden prediction in digestive tract cancers: a multicenter validation study

Abstract Microsatellite instability (MSI) and tumor mutational burden (TMB) are crucial biomarkers in gastric (GC) and colorectal cancer (CRC), yet their conventional sequencing-based detection is costly and time-consuming. Since only ~20% of patients are MSI-high or TMB-high and likely to benefit from immunotherapy, expensive genomic testing is often unjustified. This study developed a deep learning framework to predict MSI and TMB status directly from routinely available Hematoxylin and Eosin (H&E)-stained whole-slide images, leveraging fused nuclear segmentation features to improve accuracy. Using samples from TCGA (350 GC and 376 CRC for MSI; 400 GC and 387 CRC for TMB), image features were extracted with CLAM and nuclear features with Hover-Net. These features were combined via Multimodal Compact Bilinear Pooling and utilized in six distinct deep learning models. By fusing the nucleus segmentation features, the model increased area under the receiver operating characteristic curve (AUC) by 1%–3% and recall by 5%–11% in five-fold cross-validation, significantly outperforming models that relied solely on image features. External validation on a CRC dataset from the China-Japan Friendship hospital further validated the model's robustness, achieving an AUC of 0.81 and a recall of 0.80 for MSI prediction. Additionally, notable differences in cellular composition were observed across cancer types and clinical groups, emphasizing the pivotal role of cellular features in cancer development. These findings highlight the advantages of integrating H&E-stained image features with nuclear segmentation data and advanced deep learning techniques to improve predictive accuracy and reduce the cost of MSI/TMB testing, potentially advancing personalized cancer treatment strategies.