YOLOSAM: A YOLO-Guided SAM for Accurate Building Segmentation in Remote Sensing Images

Accurate building segmentation in remote sensing images is crucial for applications like disaster assessment, 3D urban modeling, and monitoring urban transformations. However, this task presents significant challenges due to the vast geographical coverage, dense building clusters, and the complexity of building contours, roof geometries, and surrounding environments. While the Segment Anything Model (SAM) offers a promising solution for extracting building masks in remote sensing images, its reliance on interactive input cues, difficulty in capturing fine edge details, and inability to integrate global semantic context with local fine-grained visual features often result in poor boundary detection and fragmented masks, limiting its effectiveness in fully automated, end-to-end building segmentation. To address these limitations, we propose YOLOSAM, a YOLO-guided adaptation of SAM designed for precise and automated building segmentation. Our framework introduces three lightweight yet effective innovations: (i) an Automatic Prompt Generator, based on YOLOv8, that automatically produces bounding box prompts to eliminate manual input; (ii) a High-Quality Token (HQ-Token) that improves edge fidelity and mask coherence by refining SAM's decoder representations; and (iii) a Global-Local Feature Fusion module, which enhances segmentation quality by fusing semantic context from deeper layers with fine edge details from earlier stages of SAM's frozen architecture. Importantly, our method preserves SAM's pre-trained generalization ability by freezing the original encoder and decoder while training only the lightweight modules. Experimental results demonstrate a significant improvement in segmentation accuracy, with mIoU increasing to 76.7% on the WHU building segmentation dataset, 69.1% on the Vaihingen building dataset, and 73.2% on the Inria Aerial Image Labeling dataset, compared to SAM's “segment everything” mode. Our model also significantly outperforms both classical deep learning baselines and other SAM-based frameworks.