Multiple Deep Proximal Learning for Hyperspectral-Multispectral Image Fusion

Fusing low resolution (LR) hyperspectral image (HSI) with a high resolution (HR) multispectral image (MSI) could enhance the spatial resolution and quality of HSI. Current deep learning (DL) HSI-MSI fusion networks have achieved encouraging results, but their performance relies on large number of training images with known degradations consistent with the testing data. The trained DL model may fail on data with unseen degradations during inference. In this study, we propose a multiple deep proximal learning network (MDPro-Net) for HSI-MSI fusion, the unknown spatial-spectral degradations and latent HR HSI can be adaptively inferred. We first propose a joint variational fusion model with both the degradations and HR HSI as to-be-solved variables, which are regularized by multiple deep priors. Then we optimize the fusion model using quadratic splitting and alternative optimization strategy. The unknown blurring kernel, spectral degradation, and HR HSI are explicitly solved by three deep proximal operators. Through unrolling the solutions into a DL network, we build MDPro-Net, in which the deep proximal operators for degradations and HR HSI are learned in an end-to-end manner. Furthermore, in the deep proximal operator for latent HR HSI, a multi-scale transformer is designed to exploit the local and non-local dependencies. Experiments demonstrate that the proposed MDPro-Net is competitive with state-of-the-art fusion methods, in particular, it is robust in inferring the unseen degradations.