S 2 Net: Self‐adaptive weighted fusion and self‐adaptive aligned network for multi‐modal MRI segmentation

Abstract Background Accurate segmentation of lesions is beneficial for quantitative analysis and precision medicine in multimodal magnetic resonance imaging (MRI). Purpose Currently, multimodal MRI fusion segmentation networks still face two main issues. On one hand, simple feature concatenation fails to fully capture the complex relationships between different modalities, as it overlooks the importance of dynamically changing feature weights across modalities. On the other hand, the unlearnable nature of upsampling in segmentation networks leads to feature misalignment issues during feature aggregation with the decoder, resulting in spatial misalignments between feature maps of different levels and ultimately pixel‐level classification errors in predictions. Methods This paper introduces the Self‐adaptive weighted fusion and Self‐adaptive aligned Network (S 2 Net), which comprises two key modules: the Self‐Adaptive Weighted Fusion Module (SWFM) and the Self‐Adaptive Aligned Module (SAM). S 2 Net can adaptively assign fusion weights based on the importance of different modalities and adaptively learn feature deformation fields to generate dynamic and flexible variability grids for feature alignment. This approach results in the generation of upsampled late‐stage features with correct spatial locations and precise lesion boundaries. Results This paper conducts experiments on two MRI datasets: ISLES 2022 and BraTS 2020. In the ISLES 2022 dataset, compared to the sub‐optimal network MedNeXt, the proposed S 2 Net showed improvements of 3.52% in Dice Similarity Coefficient (DSC), 1.67% in Intersection over Union (IoU), and 4.7% in sensitivity, with a decrease of 0.33 mm in Hausdorff Distance 95 (HD95). In the BraTS 2020 dataset, compared to the sub‐optimal network MedNeXt, the proposed S 2 Net achieved increases of 1.32% in mean DSC, 2.07% in mean IoU, and 2.17% in mean sensitivity, with a decrease of 0.10 mm in mean HD95. The code is open‐sourced and available at: https://github.com/Cooper‐Gu/S2Net . Conclusions Experimental results demonstrate that S 2 Net exhibits superior segmentation performance in multimodal MRI segmentation compared to MedNeXt, FFNet, and ACMINet.