Prognosis prediction based on liver histopathological image via graph deep learning and transformer

Liver cancer is one of the leading causes of cancer-related deaths globally. Accurately predicting the prognosis of liver cancer patients is crucial for improving their treatment and developing new anticancer drugs. However, analyzing whole slide images is time-consuming and labor-intensive for pathologists. Although deep learning methods can improve analysis efficiency, cancer prognosis prediction remains challenging due to the need for both histological features and context-aware representations to accurately infer patient survival probabilities. Several context-aware models based on graph neural networks have been proposed for weakly supervised deep learning. However, most of these methods extract WSI features using a classification network pretrained on ImageNet, which does not include cancer cell-level images. Additionally, most GNN-based methods employ a fixed number of graph convolutional layers, limiting their ability to learn multi-scale information. To address these limitations, we propose Multi-Trans-GACN, a context-aware parallel multi-scale GNN based on Transformers. Multi-Trans-GACN hierarchically aggregates instance-level histology features on different scales in the liver cancer microenvironment. A Transformer-based scale attention mechanism is utilized to combine the features extracted from different scales. We also propose a method that utilizes InceptionV3, pretrained on cellular-level liver cancer images, to construct graph structures for liver cancer images. We evaluated Multi-Trans-GACN on two liver cancer datasets. Compared to existing methods, our approach achieved significant improvements in the C-index by 5.3 and 2.50, demonstrating its superior performance in liver cancer prognosis prediction tasks. The code id available in https://github.com/z19991013/MTG.