Remote Sensing-Oriented Small-Object Detection Based on Rotation-Normalized Prompting Segment Anything Model

Remote sensing object detection is critical for earth observation and analysis. However, accurately detecting small objects remains a significant challenge due to complex and heterogeneous backgrounds as well as the inherently limited information these objects provide. While numerous specialized algorithms have been proposed to enhance small object detection, the weak feature representation of objects within cluttered scenes continues to severely limit performance, making this a persistent and open research problem. Therefore, in this article, to enhance the discriminative representation of small objects in complex background, a novel two-phase rotation normalized prompting oriented small object detection (RNP-OSOD) framework is proposed. In the first mask generation phase, a rotation normalized prompt for segment anything model (RNP-SAM) is proposed to obtain high-quality instance and semantic mask annotations. In this method, image patches containing objects are cropped, enlarged and rotation normalized to acquire RNP before being fed into SAM, which then produces accurate mask supervision for training. In the second mask supervising object detection phase, to amplify the feature responses of small objects and balance instances across different scales, a scale-adaptive instance focusing (SAIF) loss is proposed by employing the generated masks as supervisory signals. This loss comprises an area-guided semantic reconstruction loss, called scale-adaptive semantic (SAS) loss and a detection-decoupled instance segmentation loss. To rigorously evaluate the proposed framework, we conduct extensive experiments on the SODA dataset and create a new dataset, Small-DOTA, constructed by filtering and downsampling DOTA to match SODA’s characteristics. Experimental results demonstrate that our method achieves state-of-the-art performance across both benchmarks. The source code will be available at https://github.com/cbq233333/RNP-OSOD.