WiFi and Vision enabled Multimodal Emotion Recognition

Emotion recognition plays a vital role in current research on human-computer interaction, and human emotion expressions are multi-modal. In this paper, we propose a passive multi-modal emotion recognition system based on facial expression and gesture. To achieve the system design, two major challenges must be addressed, namely, how to capture facial expression and gesture without disturbing the subject, and how to use the correlation between the two modalities to better recognize emotions. For the former, we use WiFi and vision for the passive gesture and facial expression capture, respectively. For the latter, we design a Multi-Source Learning method inspired by Multi-Task Learning to efficiently exploit the correlation between modalities for better emotion recognition. Finally, to evaluate the effectiveness of our system, we use low-cost vision and WiFi devices to prototype the system and build a WiFi-Vision emotion dataset for related research, and we verify the effectiveness of our system in emotion recognition and the superiority of multi-modality over single-modality through conduct extensive experiments.