A multimodal fusion network for brain tumor segmentation based on prototype enhancement and cross-scale attention

Abstract In brain tumor magnetic resonance imaging (MRI) segmentation, conventional U-Net structures are limited by insufficient multimodal fusion, weak multi-scale semantic representation, and deep feature overlapping. This results in suboptimal precision when identifying and segmenting complex lesion areas. To address these limitations, this paper proposes a multimodal brain tumor segmentation method based on multi-branch fusion and pyramid attention mechanisms. Specifically, cross-scale attention modules are integrated into all skip connections, leveraging split pyramid convolution and efficient channel attention to achieve dynamic fusion of multi-scale features. Furthermore, a multi-branch modality feature fusion module is introduced in the first three stages of the skip connections to effectively integrate complementary information across different MRI modalities, thereby enhancing feature representation capabilities. A prototype-enhanced Atrous spatial pyramid pooling (ASPP) module is employed at the bottleneck layer, where a prototype-aware block (PAB) is embedded within the conventional ASPP. By extracting class prototypes using masks and calculating their similarity with feature maps, the PAB performs prototype-guided reweighting of multi-scale Atrous features, thereby directing the network’s attention toward highly discriminative regions. Comparative experiments on the public BraTS2020 and BraTS2021 datasets demonstrate that the proposed method achieves favorable segmentation results. On BraTS2020, the proposed method achieved dice scores of 90.45%, 86.65%, and 79.85% for the whole tumor, tumor core, and enhancing tumor (ET) regions, respectively. On BraTS2021, it achieved dice scores of 92.82%, 87.91%, and 83.71%, with corresponding HD95 values of 4.38 mm, 6.81 mm, and 5.07 mm. In addition, boundary-oriented evaluation in the ET region further demonstrated good delineation capability. These results suggest that the proposed method has certain advantages in complex lesion segmentation, boundary delineation, and small-volume lesion identification.