Facial expression translations preserving speaking content

This paper shows a method of translating a facial expression into other facial expressions using a DNN-based style embedding technique. Unlike existing work that translated a facial still image into other facial expressions, our algorithm can take into account the temporal movement of facial expressions. First, facial landmarks are obtained from the facial image sequence. Then, the relationship between the temporal facial landmark sequences, emotions and speaking context are learned by a GAN-based style transformer. Specifically, the transformer is trained to change the facial motions between different emotions while preserving the speaking content. The quality of the translation was evaluated through comprehensive experiments, including an objective evaluation and a subjective user study that evaluated the outputs using existing and GAN-based approaches combined with different subspace representations of facial landmark points. As a result, the proposed algorithm performed the best in facial emotion presentation and naturalness, and considerable good performance was achieved in the lip sync to the input original motion.