PIF-Net: A Deep Point-Image Fusion Network for Multimodality Semantic Segmentation of Very High-Resolution Imagery and Aerial Point Cloud

Semantic segmentation is of great significance in many applications. However, automating such a task on single-modality data is challenging in the field of remote sensing due to complex scenes, occlusions, and homogeneous data. In this article, we propose a deep point-image fusion network, namely PIF-Net, for multimodality semantic segmentation. The proposed PIF-Net encompass an encoder–decoder structure, where the encoder uses two independent branches with Res-Pooling blocks and point attention (Pt-Atten) blocks to extract deep and condensed multimodal features, and the decoder upsamples these features. A hierarchical fusion module is proposed to adaptively fuse multimodal features at different levels to ensure they are fully mixed. It outputs joint features in both the point and pixel representations, which are further input to a classification module to fulfill the multiple classification tasks and obtain image and point cloud semantic segmentation results. The proposed network was tested on two benchmark datasets: the Urban Semantic 3-D (US3D) dataset and the ISPRS Vaihingen dataset. Evaluation results showed that PIF-Net achieved an overall accuracy (OA) of 91.5% and 97.2% for image and point segmentation on the US3D dataset, and an OA of 90.1% and 89.3% for image and point segmentation on the ISPRS Vaihingen dataset. Comparisons with existing single-modality and multimodality methods have indicated that PIF-Net outperformed most classic methods and could bring significant improvements. It has also demonstrated that deep multimodality learning exhibit great potentials in remote sensing applications.