RPF-Net: A multimodal model for the postoperative UISS risk stratification of non-metastatic ccRCC based on CT and whole-slide images

BACKGROUND AND OBJECTIVES: Postoperative non-metastatic clear cell renal cell carcinoma (nccRCC) patients face the risk of tumor recurrence and metastasis. However, prognosis assessment for nccRCC remains time-consuming and subjective. In the current diagnostic landscape, computed tomography (CT) images provide macro-scale anatomical information, and whole-slide images (WSIs) offer micro-scale details that are inaccessible to CT imaging. To address this gap, the study proposes a multimodal approach that leverages both CT and WSI data to develop an automated model for postoperative risk stratification in nccRCC. METHODS: This study proposes a multimodal model named the Radiology-Pathology Fusion Network (RPF-Net), which employs self-attention, graph-attention, and dynamic attention fusion mechanisms to integrate CT images and WSIs for classifying nccRCC patients into low-risk and intermediate-high-risk groups per the University of California, Los Angeles, Integrated Staging System (UISS) criteria. The proposed model is divided into three steps. First, the ResNet-50 and 3D ResNet-50 are used as feature extractors to respectively extract representative feature maps from WSIs and CT images. Second, a dual-branch module is designed to extract global and local features of the WSIs. Finally, a multilayer dynamic attention fusion (MDAF) module is developed to facilitate cross-modal feature interaction and predict the risk stratification results. RESULTS: The area under the curve (AUC), accuracy, precision, and F1 Score of the RPF-Net on the internal validation set are 0.949±0.013, 0.894±0.019, 0.895±0.020, and 0.894±0.019, respectively. Furthermore, the RPF-Net shows robust generalization, achieving an AUC of 0.901 on the external validation set and 0.924 on the public dataset. CONCLUSIONS: The RPF-Net models the diagnostic process of multimodal data and shows strong generalization and excellent performance. This model may be a potential tool to facilitate clinical risk stratification and management for postoperative nccRCC patients.