AMIANet: Asymmetric Multimodal Interactive Augmentation Network for Semantic Segmentation of Remote Sensing Imagery

In recent years, the inherent 2-D characteristics of optical images have led to a plateau in semantic segmentation performance. The complementary nature of light detection and ranging (LiDAR) point clouds and camera images can effectively enhance semantic segmentation capabilities, and thus, research into multimodal joint semantic segmentation is garnering increasing attention. However, the domain gaps between different dimensions present challenges for the fusion of multimodal data. In this article, we introduce a novel asymmetric multimodal interaction augmented network (AMIANet), which directly processes heterogeneous data from images and point clouds. The treatment of the disparities in modal data ensures consistency in the features of both modes. Through the newly developed synergistic multimodal interaction module (SMI Module), AMIANet is capable of combining the complementary characteristics of cross-modal data. This is achieved by interactively fusing and extracting precise and rich structural information from point cloud features to enhance image characteristics. The experimental results on the N3C-California, WHU-RRDSD, and ISPRS Vaihingen datasets demonstrate that AMIANet surpasses benchmark methods and current state-of-the-art (SOTA) approaches. The code will be available at https://github.com/2012153946/AMIANet.