Survey on multimodal approaches to emotion recognition

Emotion is an instinctive state of mind created by the neurophysiological changes occurring in the human body as reactions to various internal or external stimuli. Emotions play a vital role in decision-making. The choices one makes in day-to-day life determine their behaviour and thus their character. Emotion and behaviour recognition are the key processes in ascertaining Emotional Intelligence (EQ) which is the inherent human potential to understand and manage one's own emotions in positive ways. But the process requires high expertise in the field of psychology and is exhaustive and time-consuming. This has opened a new horizon for exploring the computational recognition of EQ. Emotion Recognition (ER) is one of its sub-processes that identifies various human emotional states. Emotions are detected from physiological signals and also through non-invasive, vision-based algorithms by exploiting video and audio modalities. With the emergence of big data and state-of-art deep learning architectures combined with the vast availability of emotion-rich video content from various streaming platforms, Multimodal Emotion Recognition (MER) which detects emotions through multiple and complementary input modalities from video has gathered momentum in recent years. This survey paper elaborately discusses the unimodal ER through visual, auditory, and linguistic modalities and reviews MER with combined features from these modalities. It also discusses the joint representations and fusion mechanisms used to acquire the intermodal correlations. Finally, we put forward the limitations and gaps identified in the literature along with a few suggestions for future work.