Under-display face-recognition system with neural network-based feature extraction from lensless encrypted images

In this work, we present a novel under-display lensless facial-recognition system, to the best of our knowledge, which consists of a transparent micro-LED display, a specially designed mask for amplitude modulation, a CMOS sensor, and a deep learning model. By utilizing this kind of lensless optical component, the system can optically encrypt input facial information, ensuring that the light field information at the imaging plane is incomprehensible to humans. Compared to current technologies that encrypt facial images, the advantage of this approach is that the system never captures any clear facial features, fundamentally protecting user privacy. To extract effective and generalizable features from these human-incomprehensible images, a recognition algorithm based on deep learning model is proposed. However, the conventional deep learning models used for recognition systems have a fixed number of classes, necessitating retraining of the model during user registration or removal. To address this issue, we removed the output layer of the well-trained model and transformed the deep learning model into a feature extractor for lensless images. By comparing the distance between each lensless image and the registered facial templates in the latent space, the system performs the recognition task. This allows the system to successfully register and recognize new users without the need to retrain the deep learning model. Our experimental results show that this system can provide stable recognition performance while preserving user privacy, with 93.02% accuracy, 97.51% precision, and 97.74% specificity.