ProtoMix: Unified Prototype Learning for Semantic Change Detection in Remote Sensing

As an emerging topic, semantic change detection (SCD) in remote sensing aims to both identify the locations of changes before and after events and determine the land cover (LC) classes of changed regions. While deep learning has demonstrated its great potential for SCD, the following two fundamental challenges remain: large variations among the same LC class as well as close similarity between different LC classes, and the severe underutilization of massive unchanged regions, which dominate remote sensing scenes but lack detailed semantic annotations. To address these issues, we propose a novel prototype learning framework for SCD, namely ProtoMix, by leveraging both labeled and unlabeled regions (i.e., unchanged regions) and exploiting interclass similarity and intraclass variation. ProtoMix functions through a Siamese encoder-decoder architecture with multitask learning for both change detection and semantic segmentation. Specifically, the framework represents each semantic class using a cluster of prototypes, capturing variations within and across classes. To leverage the rich semantics in unchanged regions, a pseudoprototype assignment strategy is devised for deriving semantic classes of unlabeled pixels with high confidence, while treating low-confidence pixels as negative samples in a contrastive learning scheme. Intuitively, by representing each class with a cluster of multiple prototypes, ProtoMix naturally accommodates high intraclass variance. Simultaneously, the contrastive learning scheme explicitly pushes prototypes of different classes apart, effectively mitigating interclass similarity. Comprehensive experiments on two widely used datasets SECOND and LANDSAT-SCD demonstrate the state-of-the-art performance of our ProtoMix proposed.