IVFuseNet: Fusion of infrared and visible light images for depth prediction

Abstract Depth prediction is an essential component in the research of unmanned driving. Most existing research works predict depth only based on visible light images or infrared images. However, both visible light images and infrared images have their own advantages and disadvantages, and these two kinds of images contain complementary information when the images are filmed from the same scence. In order to fuse the complementary information and predict depth under various conditions, this paper proposes a convolutional-neural-network-based architecture, called infrared and visible light images fusion network (IVFuseNet), for depth prediction. Specifically, we construct common-feature-fusion subnetwork, full-feature-fusion subnetwork, and high-resolution reconstruction subnetwork, aiming to leverage the complementarity of these two kinds of images. The common-feature-fusion subnetwork adopts a two-stream multilayer convolutional structure whose filters for each layer are partially coupled to fuse the common features extracted from infrared images and visible light images respectively. The full-feature-fusion subnetwork fuses the two-stream features generated from the common-feature-fusion subnetwork by adaptive fusion weights instead of prefixed fusion weights. Additional, residual dense convolution that can accurately map the fused low-resolution features to the corresponding high-resolution features is adopted in the high-resolution reconstruction subnetwork to enhance the reconstruction of the details for depth prediction. All three subnetworks collaborate together to conduct the depth prediction task. Our NUST-SR dataset is composed of the actual road scenes captured while unmanned vehicle driving. The proposed IVFuseNet obtains the best performances on this dataset. IVFuseNet decreases the root mean squared error to 3.4513 and the mean relative error to 0.1651 respectively and outperforms other methods. The model and dataset are available at https://github.com/liyuqi1234/IVFN.