A Decoder-Focused Multitask Network for Semantic Change Detection

Recently, Semantic Change Detection (SCD) has gained growing attention from the Remote Sensing (RS) research community due to its critical role in Earth observation applications. Typical approaches tackle the task using a multi-task network, comprising one Change Detection (CD) sub-task and two Semantic Segmentation (SS) sub-tasks. Although these approaches have achieved good performance, one crucial question persists: What is the effective way to handle the feature interactions across SCD sub-tasks? To address this issue, this paper first offers an overview of existing SCD networks and compares them from a perspective view of Multi-Task Learning (MTL). Following that, we select an architecture combining a two-branch encoder and a three-branch decoder as the baseline due to its compatibility with MTL. Then, one simple yet very effective module, decoder feature interaction across sub-tasks (DFIT), is introduced. DFIT seeks to enhance the CD decoding feature by leveraging the feature differences between two SS decoding branches on a layer-wise basis. Additionally, the feature aggregation module (FAM) is designed further to enhance the network performance in cooperation with DFIT. FAM aims to produce more representative shared information across the SS and CD sub-tasks by merging the outputs from the final three encoder layers. Combining DFIT and FAM, the proposed network exploiting Decoder-Focused MTL (DEFO-MTLSCD) presents more representative information by capitalizing on both CD and SS losses back-propagations across all coding paths and achieves better performance. Experimental results reveal that our method outperforms state-of-the-art performances relative to previous SCD efforts. Our source code is released at https://github.com/byyztgxz/Decoder_Fusion.