AMSSE-Net: Adaptive Multiscale Spatial–Spectral Enhancement Network for Classification of Hyperspectral and LiDAR Data

With the abundant emergence of remote sensing data sources, multimodal remote sensing observation has become an active field. Extracting valuable information from multi-modal data has the potential to make a significant contribution to applications such as urban planning and monitoring. However, existing studies are deficient in extracting spectral and spatial features from hyperspectral remote sensing data. Meanwhile, the method of fusing multimodal features has limitations and poses a challenge to the convergence of the model loss function, which increases the complexity of the network model optimisation process. Therefore, this paper proposes an Adaptive Multi-scale Spatial–Spectral Enhancement Network for Classification of Hyperspectral and LiDAR Data called AMSSE-Net. First, we perform deep mining of spectral features in hyperspectral images by the involution operator. The main idea is to take full advantage of the involution operator in characterising spectral features by using the property that the convolution kernel shares the feature channels within the group. Furthermore, the multi-branching approach is used to extract the multi-scale information, and then the spectral-spatial features are formed with the strategy of hierarchical fusion. Meanwhile, we employ three-layer convolution for extracting shallow features from LiDAR data, offering supplementary information. Finally, we propose the "Adaptive Feature Fusion Module," an innovative and comprehensive mechanism designed for the fusion of features from diverse sources in multi-source data fusion. These dynamically assigned weights guide the selection of the optimal model, which is determined by the joint loss across the three methods, ultimately leading to the generation of an accurate prediction map. This approach not only helps to deeply explore the spectral spatial information in the hyperspectral data, but also effectively fuses the hyperspectral information with the elevation information from the LiDAR data. The expression ability of model features is rapidly improved by adaptive weighting, which in turn enhances the performance and generalisation ability of the model. Compared with some existing methods, extensive experiments on three popular HSI and LiDAR datasets show that our proposed AMSSE-Net can achieve better classification performance. The codes will be available at https://github.com/haofeng0003/AMSSE-Net, contributing to the RS community.