DFECF-DET: All-Weather Detector Based on Differential Feature Enhancement and Cross-Modal Fusion With Visible and Infrared Sensors

In complex environments such as night, fog, and battlefield camouflage, a single camera sensor is not sufficient to reflect scene information, and a multisource sensor can raise environmental awareness. A visible light camera with an infrared sensor is an efficient combination. However, there are huge differences in the inputs from different sensors, and how to fuse the information from two sensors and apply it to a specific task is a problem that needs to be solved. Hence, a multisource input detection algorithm with the combination of infrared sensors and visible cameras is proposed in this article. Its purpose is to solve the problem of low detection accuracy of a single sensor in complex and changing environments. First, a differential feature enhancement (DFE) module to enhance features that are constantly degraded during network transmission is designed in this article. Second, a cross-modal fusion (CF) module to fuse features from multiple sources is designed. Finally, two modules are embedded in a two-stream network. Experiments on the publicly available FLIR and LLVIP datasets show that the algorithm in this article improves mAP75 by 8.3/4.3 compared to a single-source detector. In some special environments, the module in this article uses 0.1 MB of storage at the cost of a 1.7 mAP boost! Extensive ablation experiments demonstrate that the module proposed in this article is lightweight, efficient, and plug-and-play.