Adaptive Modality Selection Drone-Based RGBT Detector for Tiny Targets

Drone-based target detection is an indispensable technology in numerous applications, such as surveillance, search, and rescue. The widely used single-modal RGB detector is easily affected by numerous factors, such as inadequate illumination and adverse weather. Introducing the thermal modal on top of the RGB modal can significantly enhance detection performance and robustness in complex environments. Nevertheless, drone-based RGBT target detection faces several challenges, such as the difficulty of detecting tiny targets, misaligned modality space, and redundant modality information. To address these issues, we propose a novel Adaptive Modality Selection Drone-based RGBT Detector (AMSDet), which eliminates redundant information in different modalities and enables accurate detection of tiny targets. Specifically, we first design a policy module to filter modalities before the fusion module and select the relevant modality information for input into the subsequent network. Second, we introduce a fusion module based on the attention mechanism to integrate the complementary information of the two modalities and improve the detection performance of tiny targets. Furthermore, we employ an appropriate training strategy and loss functions to jointly train the policy module and other layers in AMSDet. The superiority of our proposed method is validated through extensive experiments on the RGBTDronePerson and VTUAV-det datasets, achieving excellent detection performance for drone-based tiny targets.