A Spatial-Spectral Bilinear Representation Fusion Network for Multimodal Classification

The complementary and heterogeneous properties fusion of multimodal data (such as hyperspectral, lidar, and synthetic aperture radar data) can significantly improve the accuracy of remote sensing (RS) images joint classification. Thus, we propose a spatial-spectral bilinear representation fusion network (S ² BRFNet), which captures long-range dependencies cross-modality and within the same modality to achieve the final joint classification. Firstly, a cross-modal spatial-spectral representation module (S ² RM) is designed, it utilizes spatial-spectral attention and self-attention between heterogeneous data to enhance the characterization capabilities of cross-modal complementary properties and spatial-spectral features of single-source data. Secondly, a semantic space-guided bilinear feature fusion module (S ² BFM) is developed, which uses deep and shallow features to regain fine-grained features. It uses shallow location details to improve the semantic prediction of deep features. Furthermore, it uses the different representation capabilities of different layers for objects with obvious feature differences to enhance the feature advantages. Therefore, rich global context information is obtained. Finally, the semantic space re-weight strategy is used to guide the outer product fusion of heterogeneous features, which enhances the ability of the network to identify similar features. Classification experiments are carried out on four common datasets of different modality combinations (HS-SAR-DSM Augsburg, Berlin, Trento, and Muufl), and this can prove the superiority of the S ² BRFNet.