Bio-Signal Based Multimodal Fusion with Bilinear Model for Emotion Recognition

In this paper, we propose a feature-based multimodal fusion deep-learning approach to combine biosignals, such as electrocardiogram (ECG) and galvanic skin response (GSR) for effective emotion recognition. We make use of a bilinear fusion method to fuse ECG and GSR based features and to jointly capture and complement the complex information from both modalities to enhance the performance of an emotion recognition system. The performance of the proposed fusion approach is evaluated on the MAHNOB-HCI dataset and compared with existing multimodal fusion approaches. Experimental results show that the combination of ECG and GSR features have a high discriminating ability in classifying (or predicting) the emotional states 'Arousal' and 'Valence' better than that of a single modality based emotion recognition. We also provide the model's uncertainty analysis to understand how confident the obtained predictions are.