STCAFormer: Spatio-temporal cross attention based transformer for fusion of remote sensing images

Spatio-temporal fusion is an important technique used to address the trade-off between temporal and spatial resolution in remote sensing satellite images. However, existing deep learning-based methods often rely on a single fusion approach, which limits their ability to effectively balance the preservation of spatial details in fine images with the integration of temporal information from coarse images. In this paper, we proposed a hybrid model architecture STCAFormer which combines temporal and spatial fusion attention mechanism, leveraging their complementary strengths to maximize the utilization of temporal and spatial information from coarse and fine images. Specifically, we design a dual-input encoder structure that processes two input images simultaneously. Moreover, inspired by the unified workflow of traditional methods that implement spatio-temporal fusion in pixel space, STCAFormer introduces a fusion stage designed for spatio-temporal fusions in feature space, enabling the effective integration of spatial and temporal features from dual-input encoder. Three novel blocks are designed in this research: a swin-temporal fusion block utilizing cross-attention mechanism to capture and fuse temporal variation features, a multi-scale spatial fusion block that integrates weighted summation for temporal variation and fine spatial features at both local and global scales, and a dual-path up-sampling block designed to achieve high-fidelity image restoration. STCAFormer was compared with six different types of spatio-temporal fusion methods on remote sensing image pairs, including Landsat 8/MODIS and Sentinel-2/3 pairs, from various regions. Experimental results demonstrate that STCAFormer performs better than baseline methods when the regions and satellite sensors remain unchanged. The transfer ability of STCAFormer was also proved to be more competitive than baseline methods. The ablation studies show that the proposed blocks are effective for performance improvement of spatio-temporal fusion The proposed STCAFormer demonstrates robust generalization across Landsat 8/MODIS and Sentinel-2/3, as well as across land-cover types represented in the test datasets. The code of the proposed STCAFormer is available at: https://github.com/Neooolee/STCAFormer .