E2MISeg: Enhancing edge-aware 3D medical image segmentation via feature progressive co-aggregation

• We propose a novel enhancing edge-aware neural network for multi-modal 3D medical image segmentation. • A feature progressive co-aggregation strategy for improving feature representation and edge voxel classification. • Compared with the most advanced methods, our model achieves better performance and generalization ability. • We construct a challenging clinical diagnostic dataset of PET images for mantle cell lymphoma. 3D segmentation is critically essential in the clinical medical field, which aids physicians in locating lesions and assists in clinical decision-making. The unique properties of organ and tumour images with large-scale variations and low-edge pixel-level contrast make clear segment edges difficult. Facing these problems, we propose an Enhancing Edge-aware Medical Image Seg mentation (E2MISeg) for smooth segmentation in boundary ambiguity. Firstly, we propose the Multi-level Feature Group Aggregation (MFGA) module to enhance the accuracy of edge voxel classification through the boundary clue of lesion tissue and background. Secondly, to minimize the influence of background noise on the model’s sensitivity to the foreground, the Hybrid Feature Representation (HFR) block utilizes an interactive CNN and Transformer to deeply mine the lesion area and edge texture features while providing more clues for the MFGA module. Finally, we introduce the Scale-Sensitive (SS) loss function that dynamically adjusts the weights assigned to targets based on segmentation errors, with these weights guiding the network to focus on regions where segmentation edges are unclear. Furthermore, we retrospectively collated the Mantle Cell Lymphoma PET Imaging Diagnosis (MCLID) dataset of 176 patients from multiple central hospitals, which enhances our algorithm’s robustness against complex clinical data. The extensive experimental results on three public challenge datasets and the MCLID clinical dataset demonstrate our approach, which outperforms the state-of-the-art methods. Further analysis shows that our components work together to achieve smooth edge segmentation, which is of great significance for accurate clinical diagnosis and prognosis analysis. The Code available at: https://github.com/SoloTillDawn/E2MISeg