Modality Registration and Object Search Framework for UAV-based Unregistered RGB-T Image Salient Object Detection

UAVs are widely used in various industries, and various visual tasks under the perspective of the UAV have been widely studied. In particular, the RGB-T detection method based on UAVs has shown significant advantages. However, existing RGB-T methods are designed based on registration image pairs rather than detecting images directly acquired by UAVs. This detection process is limited by the accuracy of image registration. And image registration wastes a lot of time. To solve the above problems, we construct an unregistered RGB-T image salient object detection (SOD) dataset under the UAV perspective, known as UAV RGB-T 2400. The dataset includes many challenging scenes, and the images are not manually registered. Further, we construct a modality registration and object search (MROS) framework for unregistered RGB-T SOD. Firstly, a modality registration scheme is proposed to solve the unregistration problem of modal features. We successively perform pixel-level registration from a local perspective and semantic-level registration from a global perspective for different modal features. And we carry out the channel and spatial interaction for the different modal features in modality registration. Aiming at the interference problem in the UAV detection environment, we propose an object search scheme. The two high-level features are used to search the object location, and the three low-level features are used to refine the object and produce prediction results. Experimental results on the UAV RGB-T 2400 dataset show that MROS is effective compared with state-of-the-art methods. The code is available at: https://github.com/VDT-2048/UAV-RGB-T-2400.