Zero-Sharpen: A universal pansharpening method across satellites for reducing scale-variance gap via zero-shot variation

Pansharpening is a technique that combines a high-resolution panchromatic image (HRPAN) and a low-resolution multi-spectral image (LRMS) to generate a high-resolution multi-spectral image (HRMS). Traditional methods perform sharpening based on given image pairs, but their performance is limited due to the employment of scale-varying linear mapping assumptions. Existing deep-learning-based methods can establish arbitrary non-linear sharpening functions based on large-scale training data. However, supervised methods suffer from scale-variance generalization for training on the simulated reduced resolution data, while unsupervised methods arise distortion for the absence of reference and introduction of inaccurate spectral observation assumptions. Besides, the generic satellite-specific learning (i.e. training and testing on the homologous satellite data) causes low generalization when processing the heterologous satellite data. To this end, we combine the advantages of deep learning and variational optimization, to propose a universal pansharpening method that can be applied across different satellites with reducing the scale variance, termed as Zero-Sharpen. On the one hand, we build image-pair-specific neural networks to extend the spatial mapping and spectral observation assumptions to the nonlinear space. These assumptions are incorporated as deep spatial prior and deep spectral observation prior for regularization, assisting variational models to iteratively adapt to the full-resolution scale. On the other hand, the variational optimization mechanism also promotes the optimization of deep networks so as to achieve the preservation of spectral and spatial information in a zero-shot learning manner. In doing so, our method can be easily available across different satellites. Extensive experiments demonstrate the superiority of our method over the state-of-the-art methods both qualitatively and quantitatively. Code is publicly available at https://github.com/Baixuzx7/ZeroSharpen.