ParallelNet: A Depth-Guided Parallel Convolutional Network for Scene Segmentation

In the past few years, deep convolutional neural networks (CNN) have shown great superiority and also been the first choice in semantic segmentation. However, the pooling layers in the CNN cause the increasing loss (mainly positioning structure details) which is not favourable for segmentation. Moreover, the vast majority of previous studies only utilize the color or textural information of the image, without considering the depth information which is helpful for segmentation. In this paper, we propose a novel and effective end-to-end network for semantic segmentation namely Depth-guided Parallel Convolutional Network (ParallelNet). Compared to previous work, the contribution of our ParallelNet is that we have taken advantages of the mutual benefit and strong correlations between depth information and semantic information, which are combined to guide scene semantic segmentation. Besides, we utilise a new method to obtain the depth information of the image by calculating the correlation distance with \(\mathcal {L}_1\)-norm between left and right feature maps, thus, we just need to input the RGB images instead of RGB images and encoded 3D images in some conventional methods. Furthermore, we apply the concept of our ParallelNet to the current popular networks by exploiting the guidance of the depth information and transfer their learned representations with fine-tuning. The extensive experiments on the popular dataset Cityscape exhibit that our ParallelNet outperforms the original methods.