Robust LWIR-based Eye Center Detection through Thermal to Visible Image Synthesis

This paper proposes a novel approach to automatically detect the eye centers in challenging thermal (Long-Wave Infrared; 8 – 14 µm) face images so they can be geometrically normalized, which is an important part of face recognition systems. Developing a new LWIR based eye center detection model would require many thermal images, which is not possible due to the unavailability of publicly available large scale LWIR face datasets. An alternative solution would be to use a pre-trained, visible band based facial landmark detection model and test it on LWIR face images. In the latter case, the challenges are the significant differences between the visible and thermal face images. In this paper, we focus on addressing this research gap by proposing a solution that is based on the following approach. First, we synthesize visible from thermal face images. Then, we exploit the existing robust visible band facial landmark detection models to detect eye centers in the synthesized visible band face images. While we empirically test different image synthesis models, we determine that StarGAN2 (an image-to-image translation generative adversarial network model that learns a mapping between the different visual domains) yields the highest eye center detection accuracy when compared to the other state-of-the-art models. Thus, we train a StarGAN2 model to be able to synthesize good quality visible band images from their thermal band counterparts. Next, we use an efficient visible band based facial landmark detection model to detect the eye centers in the synthesized visible band face images. Finally, we map these coordinates to the original LWIR face images, which are used for geometric normalization and, finally run a set of face recognition experiments. Compared to the baseline model, our approach increases the eye detection accuracy by 14%, 50%, and 30% when the normalized error (e) is set to be ≤ 0.05, ≤ 0.10, and ≤ 0.25 respectively compared to the baseline model. Our approach yields up to a 15 % and 7% increase in face recognition accuracy when using advanced deep-learning based matchers, namely Facenet and VGGFace respectively, and by 30% when using other conventional face matching techniques such as LBP-LTP matchers.