FRNet: Improving Face De-occlusion via Feature Reconstruction

Face de-occlusion is essential to improve the accuracy of face-related tasks. However, most existing methods only focus on single occlusion scenarios, rendering them sub-optimal for multiple occlusions. To alleviate this problem, we propose a novel framework for face de-occlusion called FRNet, which is based on feature reconstruction. The proposed FRNet can automatically detect and remove single or multiple occlusions through the predict-extract-inpaint approach, making it a universal solution to deal with multiple occlusions. In this paper, we propose a two-stage occlusion extractor and a two-stage face generator. The former utilizes the predicted occlusion positions to get coarse occlusion masks which are subsequently fine-tuned by the refinement module to tackle complex occlusion scenarios in the real world. The latter utilizes the predicted face structures to reconstruct global structures, and then uses information from neighboring areas and corresponding features to refine important areas, so as to address the issues of structural deficiencies and feature disharmony in the generated face images. We also introduce a gender-consistency loss and an identity loss to improve the attribute recovery accuracy of images. Furthermore, to address the limitations of existing datasets for face de-occlusion, we introduce a new synthetic face dataset including both single and multiple occlusions, which effectively facilitates the model training. Extensive experimental results demonstrate the superiority of the proposed FRNet compared to state-of-the-art methods.