Use of a Convolutional Neural Network to Predict the Malignant Potential of Gastrointestinal Stromal Tumors in Transabdominal Ultrasound Images: Visualization of the Focus of the Prediction Model

Objective We established a deep convolutional neural network (CNN) model based on ultrasound images (US-CNN) for predicting the malignant potential of gastrointestinal stromal tumors (GISTs). Methods A total of 980 ultrasound images from 245 pathology-confirmed GIST patients after surgical operation were retrospectively collected and divided into a low (very-low-risk, low-risk) and a high (medium-risk, high-risk) malignant potential group. Eight pre-trained CNN models were used to extract the features. The CNN model with the highest accuracy in the test set was selected. The model's performance was evaluated by calculating accuracy, sensitivity, specificity, positive-predictive value (PPV), negative-predictive value (NPV) and the F1 score. Three radiologists with different experience levels also predicted the malignant potential of GISTs in the same test set. US-CNN and human assessments were compared. Subsequently, gradient-weighted class activation diagrams (Grad-CAMs) were used to visualize the model's final classification decisions. Results Among the eight transfer learning-based CNNs, ResNet18 performed best. The accuracy, sensitivity, specificity, PPV, NPV and F1 score were 0.88, 0.86, 0.89, 0.82, 0.92 and 0.90, respectively, which were significantly better than those achieved by radiologists (resident doctor: 0.66, 0.55, 0.79, 0.74, 0.62 and 0.69; attending doctor: 0.68, 0.59, 0.78, 0.70, 0.69 and 0.73; professor: 0.69, 0.63, 0.72, 0.51, 0.80 and 0.76). Model interpretation with Grad-CAMs revealed that the activated areas mainly focused on cystic necrosis and margins. Conclusion The US-CNN model predicts GIST malignant potential well, which can assist in clinical treatment decision-making. We established a deep convolutional neural network (CNN) model based on ultrasound images (US-CNN) for predicting the malignant potential of gastrointestinal stromal tumors (GISTs). A total of 980 ultrasound images from 245 pathology-confirmed GIST patients after surgical operation were retrospectively collected and divided into a low (very-low-risk, low-risk) and a high (medium-risk, high-risk) malignant potential group. Eight pre-trained CNN models were used to extract the features. The CNN model with the highest accuracy in the test set was selected. The model's performance was evaluated by calculating accuracy, sensitivity, specificity, positive-predictive value (PPV), negative-predictive value (NPV) and the F1 score. Three radiologists with different experience levels also predicted the malignant potential of GISTs in the same test set. US-CNN and human assessments were compared. Subsequently, gradient-weighted class activation diagrams (Grad-CAMs) were used to visualize the model's final classification decisions. Among the eight transfer learning-based CNNs, ResNet18 performed best. The accuracy, sensitivity, specificity, PPV, NPV and F1 score were 0.88, 0.86, 0.89, 0.82, 0.92 and 0.90, respectively, which were significantly better than those achieved by radiologists (resident doctor: 0.66, 0.55, 0.79, 0.74, 0.62 and 0.69; attending doctor: 0.68, 0.59, 0.78, 0.70, 0.69 and 0.73; professor: 0.69, 0.63, 0.72, 0.51, 0.80 and 0.76). Model interpretation with Grad-CAMs revealed that the activated areas mainly focused on cystic necrosis and margins. The US-CNN model predicts GIST malignant potential well, which can assist in clinical treatment decision-making.