Infrared and visible image fusion with entropy-based adaptive fusion module and mask-guided convolutional neural network

Since infrared and visible images have different descriptions of the same scene and have a lot of complementary information, they are usually fused to obtain a more comprehensive image. To overcome the detail information losing, fusion artifacts, and unnatural fusion in the traditional image fusion, we propose a novel algorithm for infrared and visible image fusion. Firstly, we introduce a guided filter to decompose the input image into a first-level base layer, a second-level base layer and a detail layer. Specifically, for the first-level base layer fusion, we propose an entropy-based image adaptive fusion module, which is guided by the image quality and the information entropy. To reduce the low-frequency information losing, we use the maximum absolute value rule to fuse the second-level base layer. In addition, for the detail layer containing many textures and edges, we construct a mask-guided deep convolutional neural network to fuse it. We label the infrared image pixel-by-pixel in the training set to obtain the foreground target mask, and construct the loss function jointly considering the mask, gradient, and the structural similarity of the image. Masks are only used when training the network. Finally, we reconstruct the image. Experiments show that our algorithm has achieved good performance in both subjective and objective evaluation.