Object detection based on multi-modal adaptive fusion using YOLOv3

Object detection is a technique used to localize and classify objects in an image or a video sequence. It is an emerging topic of research in the field of computer vision. However, detections in a video are affected by sensor-specific challenges. A convolutional neural network-based You Only Look Once, version 3 (YOLOv3) object detection algorithm was used to get optimized computation time and accuracy. In the proposed methodology, the YOLOv3 architecture extracted significant features from both the visible and thermal imaging domains, and an adaptive fusion of both domains was performed to determine the dominant imaging domain and provide robust detections. The resulting YOLOv3 detections included the bounding box coordinates, confidence score, and class from each imaging domain, which were fused implicitly into a single plane. The sensor domain having the maximum number of object detections was chosen as the reference to be compared with the other domain for the adaptive fusion process. After fusion, the algorithm removed redundancy using adaptive intersection over union thresholding. The mean average precision result obtained from the fusion algorithm was 44.25%. A comparative study was also carried out between pre-trained common objects in context weights and custom CAMEL dataset weights; it showed the significance of using adaptive fusion in challenging situations such as nighttime, shadow, varying illumination, moving camera, and crowd.