TMGformer: Text-Modulated Multiscale Guidance Transformer for Pansharpening

Pansharpening, a critical task in remote sensing, aims to fuse panchromatic (PAN) images with low-resolution multispectral (LRMS) images to generate high-resolution multispectral (HRMS) outputs. Existing deep learning methods primarily focus on pixel-level spatial and spectral features, often overlooking the deeper, text-level semantic information that can enhance fusion. To address this limitation, we introduce a novel Text-modulated Multiscale Guidance Transformer (TMGformer). Specifically, we first establish a self-contained workflow to generate text features, eliminating reliance on external Application Programming Interfaces (APIs). Subsequently, we develop a Text-modulated Visual Feature Refinement (TVFR) module that transforms these text features into spectrally compatible and semantically consistent guidance. This guidance is then injected into our Asymmetric Multiscale Fusion (AMF) module to guide the fusion process at multiple scales, effectively addressing the vast variance in the scale of terrestrial objects described in their corresponding texts. Furthermore, we introduce a hybrid text-visual loss function to provide comprehensive supervision. Extensive experiments demonstrate that TMGformer achieves state-of-the-art performance. Notably, on the GaoFen-2 dataset, our method reduces the ERGAS by 5.11% and the D_s by 16.44% compared to the second-best competitor. To facilitate future research, we release a new vision-language dataset containing textual descriptions and corresponding text features. The code and dataset are available at: https://github.com/PeterZhaoXJTU/TMGformer.