Automatic part segmentation of facial anatomies using geometric deep learning toward a computer-aided facial rehabilitation

Detection, identification, and segmentation of facial landmarks and anatomies play an essential role in the automatic reconstruction of patient specific model of the human head for facial diagnosis, monitoring, and rehabilitation. The objective of the present study was to apply geometric deep learning to perform part segmentation on the human face to automatically segment facial anatomies from a 3D point set. A database of Computed Tomography images of 333 subjects was reconstructed. Labels of facial anatomies (eyes, nose, and mouth) were manually performed. Two state-of-the-art geometric deep learning models (PointNet++ and PointCNN) were implemented and evaluated. Then, the best model was applied to perform part segmentation on new Kinect-driven face data of healthy subjects and facial palsy patients. Accuracy and Intersection over Union (IoU) were used as evaluation metrics. An accuracy level of 99.19% and an IoU of 89.09% are obtained for the CT database using the PointNet++ model. Regarding the use of the PointCNN model, an accuracy level of 98.43 and an IoU of 78.33 were obtained. An accuracy range of [81.45%–92.09%] and [81.05%–84.08%] was obtained by using PointNet++ model on Kinect data for healthy subjects and facial palsy patients respectively. This study suggested that geometric deep learning can be used for automatic segmentation of facial anatomies from a 3D data set. The obtained outcomes confirmed the accuracy of PointNet++ and PointCNN architectures. As perspectives, the proposed method will be implemented into an available computer vision system for facial monitoring and rehabilitation.