Deep Unfolding Network Enhanced by Transformer Priors for Unregistered Hyperspectral and Multispectral Image Fusion

In satellite remote sensing, the complementary nature of hyperspectral (HSI) and multispectral (MSI) imagery necessitates their fusion to enhance both spatial and spectral resolution. However, the inherent misalignment between these datasets, due to differences in acquisition conditions, poses a significant challenge. This study presents a novel approach, the deep unfolding network enhanced by Transformer priors (DUNET), to address the simultaneous registration and fusion of HSI and MSI. Unlike conventional deep fusion methods, which are often treated as opaque “black boxes,” DUNET incorporates the deep unfolding method, leveraging mutual information and deep priors to facilitate a better degradation model-informed fusion process. The proposed network incorporates hybrid attention Transformers (HATs) and spatial-frequency modules to fully exploit the spatial-spectral information of HSI, resulting in a more accurate and detailed representation of the scene. We conducted extensive quantitative and visual experiments on three standard HSI datasets. The results demonstrate that our proposed DUNET method outperforms the existing mainstream algorithms in the field of remote sensing image fusion, showcasing its effectiveness. Specifically, our proposed method achieves the improvements of 3.4, 5.1, and 8.2 dB in terms of peak signal-to-noise ratio (PSNR) compared with the latest methods on the ICVL, Chikusei, and Houston datasets, respectively.