Semantic Segmentation With Light Field Imaging and Convolutional Neural Networks

Semantic segmentation is of great importance and a challenge in computer vision. One of its main problems is how to efficiently obtain rich information (geometric structure) and identify useful features from higher dimensions. A light field camera, due to its special microlens array structure, can completely record the angular-spatial information of scenes, which is attractive and has great potential to improve the performance of semantic segmentation task. Inspired by this, we propose an end-to-end semantic segmentation network that can process light field macro-pixel image robustly and extract its features. In addition, this network can flexibly and efficiently load the different popular deep learning backbones. Furthermore, we propose an efficient angular model, which to learn the angular features between the different viewpoints of the macro-pixel image, improve the nonlinearity of angular-spatial features and enhance multichannel semantic correlations. To evaluate the network, we construct a new real scene light field dataset comprising 800 high-quality samples. The quantitative and qualitative results show that the highest mean intersection over union (mIoU) based of our algorithm is greater than 57.00%. Our algorithm achieves a 10.30% increase compared with state-of-the-art semantic segmentation algorithms. In combination with different backbones or multi-scale light field macro-pixel images, the network can also achieve comparable results. This preliminary work demonstrates that the combination of light field imaging and deep learning technology has potential applications in the future study of semantic segmentation.