Spontaneous Facial Expressions and Micro-expressions Coding: From Brain to Face

Facial expressions are a vital way for humans to show their perceived emotions. It is convenient for detecting and recognizing expressions or micro-expressions by annotating a lot of data in deep learning. However, the study of video-based expressions or micro-expressions requires that coders have professional knowledge and be familiar with action unit (AU) coding, leading to considerable difficulties. This paper aims to alleviate this situation. We deconstruct facial muscle movements from the motor cortex and systematically sort out the relationship among facial muscles, AU, and emotion to make more people understand coding from the basic principles: We derived the relationship between AU and emotion based on a data-driven analysis of 5,000 images from the RAF-AU database, along with the experience of professional coders. We discussed the complex facial motor cortical network system that generates facial movement properties, detailing the facial nucleus and the motor system associated with facial expressions. The supporting physiological theory for AU labeling of emotions is obtained by adding facial muscle movements patterns. We present the detailed process of emotion labeling and the detection and recognition of AU. Based on the above research, the video's coding of spontaneous expressions and micro-expressions is concluded and prospected.