Stereo Cross-Attention Network for Unregistered Hyperspectral and Multispectral Image Fusion

The necessary prerequisite for effective data fusion is the strict registration of low-resolution hyperspectral images (LR-HSI) and high-resolution multispectral images (HR-MSI). However, registration requires a complex process that takes into account the effects of light, imaging angle, and geometric distortion of the image during acquisition. Therefore, to avoid complex registration, we focused on developing an unregistered HSI and MSI fusion method for pixel shifting, obtaining fused images with high resolution, high signal-to-noise ratio, and feature identifiability. We identified that the unregistered LR-HSI and HR-MSI in the case of pixel shift are very similar to the disparity maps in stereo vision. Inspired by this, we simulate the structure of stereo cameras to propose a stereo cross-attention network (SCANet) to achieve an accurate fusion of unregistered LR-HSI and HR-MSI. Considering the model complexity and computing efficiency, we design a simple and stackable stereo cross-fusion block (SCFBlock) based on a Transformer to simulate the process of light entering the left and right cameras by extracting the abstract features of the images. Moreover, the purpose of cross-convergence fusion self-attention (CCFSA) is to learn cross-complementary attention and collect contextual information in horizontal and vertical directions to fuse unregistered images using multi-directional cross-view information. We have conducted extensive experiments on Pavia University (PaviaU), Chikusei, and PYLake datasets. The results show that SCANet achieves superior or competitive performance in fusing unregistered LR-HSI and HR-MSI in comparison with the other competitors.