MCKTNet: Multiscale Cross-Modal Knowledge Transfer Network for Semantic Segmentation of Remote Sensing Images

Multimodal data fusion can provide valuable and diverse information for remote sensing image segmentation. However, different modal data have different feature distributions, which causes some conflicts and redundancies in cross-modal feature fusion. In addition, existing multimodal fusion networks usually adopt a two-branch structure with a large number of parameters and high computational cost. To address these problems, we propose a multiscale cross-modal knowledge transfer network (MCKTNet) for remote sensing image segmentation. First, we use a cross-modal migration learning method that combines channel discrete loss and spatial discrete loss to facilitate cross-modal migration of geometric and semantic features and reduces redundancy by minimizing the differences in feature distribution. Then, we use the polarized cross-self-attention mechanism to establish long-range correlations between different modal features across spatial and channel dimensions, and only a small number of parameters need to be added to achieve complementary cross-modal feature fusion. Finally, to accurately capture object edges, we propose a multiscale edge perception module to optimize the edge details in the prediction results at the pixel level. Extensive experiments demonstrate that the proposed method is effective, robust, and generalizable, and achieves state-of-the-art performance in multiple remote sensing image semantic segmentation tasks with only 13.51 million parameters. The code will be available at https://github.com/NUAALISILab/MCKTNet.