An enhanced YOLO11-based optical images ship detection algorithm with discrete Fourier transform and synergistic attention mechanism

Abstract Ship remote sensing detection technology plays a crucial role in modern maritime navigation and ocean management. However, ship targets exhibit multi-scale characteristics, diverse categories, and dense distributions, while image quality is often affected by weather variations and complex sea surface backgrounds, making detection highly challenging. In this paper, we propose RS-YOLO, an accurate ship detection algorithm based on YOLO11. Firstly, an Adaptive Convolutional Fourier Frequency Fusion Module is designed with a spatial-frequency cascading structure to effectively integrate local and global information, thereby enhancing feature extraction under low-resolution and complex background conditions. In addition, a Synergistic Convolution-Attention Mechanism is developed to strengthen the interaction between local feature extraction and global relationship modeling, enabling the model to accurately recognize ships in multi-resolution and densely distributed scenarios. Finally, a Shape IoU loss function is incorporated to better adapt to the complex shapes and scale variations of ships, improving the robustness and accuracy of bounding box regression. Experimental results demonstrate that the proposed method consistently outperforms the YOLO11n baseline across four public datasets, achieving mAP0.5 improvements of 2.1%, 2.2%, 1.6%, and 1.8% on HRSC2016, NWPU VHR-10, LEVIR-Ship, and DIOR, respectively. Moreover, comparative evaluations with several state-of-the-art detection algorithms further confirm the superior performance and robustness of RS-YOLO in complex maritime environments.