A self-supervised remote sensing image fusion framework with dual-stage self-learning and spectral super-resolution injection

Pan-sharpening is a very productive technique to enhance the spatial details of multispectral images with the aid of panchromatic images. Nowadays, deep learning-based pan-sharpening has scored tremendous achievements. However, strict requirement for training image pairs and low generalization hamper the development of supervised pan-sharpening with limited samples. Unsupervised image fusion is an effective technique to fuse images without adequate ground truth as training samples. Existing unsupervised pan-sharpening methods are commonly based on the image generator model, suffering from unsatisfactory spatial details. In this study, we proposed a self-supervised pan-sharpening method with the aid of spectral super-resolution named sSRPNet. Coarsening-scale self-learning exploits the internal information in multispectral images at a coarsening scale and trains the initial fusion model without other labels. Spectral super-resolution injection explores the missing spatial details in the initial fused images and recovers it. Degradation self-learning introduces strong spectral constraints with original multispectral images. Both reduced-resolution and full-resolution experiments on three datasets have proved the superiority of sSRPNet. Furthermore, the proposed spectral super-resolution injection can be implemented on any existing pan-sharpening algorithms, improving their performance.