Multispectral-Hyperspectral Image Fusion via Similarity-Guided Graph Attention and VAE-Transformer

Fusion of a high-spatial-resolution multispectral image (MSI) and a low-spatial-resolution hyperspectral image (HSI) aims to generate a high-spatial-resolution HSI (HR-HSI). Most fusion methods use simple upsampling techniques to increase the resolution of HSI without guidance, which can introduce unwanted artifacts and lead to spectral distortion. Additionally, they face challenges in generalization and robustness. To address these challenges, this paper introduces a cross-modal fusion network for MSI and HSI, named CSGAV, which is built on similarity-guided graph attention (SGA) and a variational autoencoder-Transformer (VAET). Specifically, we develop a similarity measure algorithm to compute the similarity degree between the source images and construct an SGA module to mitigate modal differences, producing precise upsampled outputs. Moreover, we present an adaptive weighted Transformer, with the weights guided by a variational autoencoder, thereby enhancing the generalization and robustness of the model. The SGA and VAET are integrated in the cross-modal interactive architecture to achieve the final HR-HSI image. Experimental results conducted on four public datasets show that CSGAV is superior compared to existing state-of-the-art fusion methods both in fusion performance and generalization. The code of this work is available at https://github.com/yotick/CSGAV.