UAV-Based Human Detection With Visible-Thermal Fused YOLOv5 Network

Timely and effective search and rescue (SAR) is highly desired in the disaster rescues. Unmanned aerial vehicles (UAVs) can quickly conduct aerial searches to assist SAR with the equipped sensors. A visible-thermal human detection model based on an improved you only look once version 5 (YOLOv5) network is proposed to compensate the deficiencies of visible data with thermal images. The complementary information between the visible images and thermal images are considered with a partially shared two-stream backbone network, so as to better preserve the information of each branch while reducing the domain distinction and extracting the modality-invariant features. Features of the two modalities are fused via a fusion module with a multidimensional attention mechanism. By taking the pixels outside the region of interest as negative samples, the extra loss function can suppress the uncorrelated feature extraction of the backbone to enhance the effective feature representation. The proposed visible-thermal human detection model has been deployed on the UAV with satisfied human detection performance. Comparative experiments on the multispectral pedestrian dataset KAIST have also been performed to demonstrate that the proposed model outperforms other visible-thermal object detection models with log-average miss rate.