SAM Enhanced Semantic Segmentation for Remote Sensing Imagery Without Additional Training

Semantic segmentation is a critical process in remote sensing image analysis, supporting various applications. The recent development of the segment anything model (SAM), a visual foundation model designed to segment-anything, highlights the potential of foundational models in computer vision. However, SAM generates segmentation results without category labels, and predictions from semantic segmentation models for remote sensing often exhibit excessive fragmentation and imprecise boundaries. To address these limitations, we propose a strategy that integrates SAM with semantic segmentation models, replacing the imprecise boundaries of remote sensing segmentation masks with the more boundary-accurate SAM masks while retaining the original semantic information. Subsequently, a framework is designed and realized to enhance the prediction results of semantic segmentation models for remote sensing imagery by leveraging the raw outputs generated by SAM. This approach requires no additional training, modification to the semantic segmentation model, or changes to the visual foundation model, making it efficient and straightforward compared with other methods. Specifically, experimental results on two well-known datasets, LoveDA Urban and ISPRS Potsdam, demonstrate the effectiveness and broad applicability of our approach. In addition, incorporating recent visual foundation models, such as SAM-HQ and semantic SAM, further improves segmentation accuracy. As these models advance, the potential of our framework to enhance the performance of semantic segmentation for remote sensing imagery will grow. The source code for this work will be accessible at https://github.com/qycools/SESSRS.