AIG 2 AN: Ambiguous-Interpretation Generalized GAN for Self-Supervised Raster-Vector Semantic Segmentation for Cross-Modal Remote Sensing Image

Remote sensing image interpretation faces easily overlooked issues: (1) Ambiguity: mutual occlusion between objects, e.g., buildings are occluded by trees and vice versa, which leads to ambiguity (building or tree) in GIS; (2) Expression conflicts: topology conflicts between the occluding and occluded objects. However, there have been being no reports on the Remote Sensing Ambiguous Interpretation (RSAI) or its Dataset (RSAID). To address these issues, a RSAID was built and an ambiguous-interpretation generalized generative adversarial network (AIG²AN) with self-supervised raster-vector semantic segmentation for remote sensing image, is proposed for the first time. With the occluding image generated, AIG²AN not only recognizes occlusion regarded as cheating but also restores the occluded semantic labels, which includes three modules: (1) Random occlusion generator automatically generates the occluding image with the occluding and occluded labels for self-supervised learning with avoiding hallucination issue; (2) Ambiguity dual-interpreters, including both occluding and occluded interpretations, is the first to be proposed; (3) Consistency contrastive learning module is to process the inconsistent interpretations in unambiguous region. Furthermore, we propose a novel multispectral cross-modal learning paradigm via self-supervised pseudo-labeling to automatically enhance the generalization of our AIG²AN in different sensor modalities; and propose a conflict-free and regularized raster-vector expression for interpretation results of ambiguous objects. Experiments on our manually restored RSAID from ISPRS Vaihingen and Potsdam show that AIG²AN improves mIoU and building IoU by 3.28% and 1.80%, with ambiguity elimination index of 85.32% and 76.41% respectively. Moreover, we reveal occlusion restoration mechanism. RSAID and code are available at: https://github.com/wenlailiu/AIG2AN.