A Neighbor-Sensitive Multi-Modal Flexible Learning Framework for Improved Prostate Tumor Segmentation in Anisotropic MR Images

Accurate segmentation of prostate tumors from multi-modal magnetic resonance (MR) images is crucial for the diagnosis and treatment of prostate cancer. However, the robustness of existing segmentation methods is limited, mainly because these methods 1) fail to flexibly assess subject-specific information of each MR modality and integrate modality-specific information for accurate tumor delineation, and 2) lack effective utilization of inter-slice information across thick slices in MR images to segment the tumor as a whole 3D volume. In this work, we propose a neighbor-sensitive multi-modal flexible learning network (NesMFle) for accurate prostate tumor segmentation from multi-modal anisotropic MR images. Specifically, we perform multi-modal fusion for each slice by developing a Modality-informativeness Flexible Learning (MFLe) module for selecting and flexibly fusing informative representations of each modality based on inter-modality correlation in a pre-trained manner. After that, we exploit inter-slice feature correlation to derive volumetric tumor segmentation. In particular, we first use a Unet variant equipped with a Sequence Layer, which can coarsely capture slice relationship using 3D convolution and an attention mechanism. Then, we introduce an Activation Mapping Guidance (AMG) module to refine slice-wise representations using information from adjacent slices, ensuring consistent tumor segmentation across neighboring slices based on slice quality assessment on activation maps. Besides, during the network training, we further apply a random mask strategy to each MR modality for improving feature representation efficiency. Experiments on both in-house and public (PICAI) multi-modal prostate tumor datasets demonstrate that our proposed NesMFLe achieves competitive performance compared to state-of-the-art methods.