A Multitask Network and Two Large-Scale Datasets for Change Detection and Captioning in Remote Sensing Images

Remote sensing change detection (RSCD) recognizes pixel-level change regions between images, while remote sensing change captioning (RSCC) describes the nature and properties of these changes in natural language. The past have studied two tasks individually. Resulting in a single interpretation content produced and limited application scenarios. And the complementarity of change regions and deep semantic information can further improve the performance and robustness of the model. Therefore, we try simultaneously to solve both RSCD and RSCC tasks (i.e., RS-CDC) under a multitask framework, namely a CNN-Transformer-based multitask network (CTMTNet). Specifically, we design multiattention feature enhancement module (MAFEM) and feature fusion block (FFB) to enhance local information and location perception of features from bitemporal images. The MAFEM weights the channel and space separately to capture local information more accurately and enhance location perception. The FFB fuses bitemporal features and uses multilevel residual connections to ensure that change information is not lost during transfer. Finally, we use two decoders to output the change maps (CMs) and change captioning, respectively. During training, we use an improved multitask loss function for CTMTNet to balance the two tasks. For exploring the RS-CDC task, we construct two large-scale datasets named LEVIR-CDC and WHU-CDC dataset. We benchmark the existing state-of-the-art (SOTA) change detection (CD) and change captioning methods on these two datasets and a newly publicized LEVIR-MCI dataset, and the results show that the proposed CTMTNet significantly outperforms comparative methods.