Multi-View Cognitive Prior Graph Convolutional Network for Multimodal Emotion Recognition

Electroencephalography (EEG) has been established as the primary modality in the field of affective computing. Integrating EEG with electrocardiogram (ECG) signals based on heart-brain coupling can overcome the inherent limitations of unimodal approaches by leveraging complementary neural dynamics, achieving more robust emotion representation. However, the existing methods face two critical challenges. First, poor modeling of psychophysiological spatial correlations limits high-order cognitive feature extraction. Second, the existing methods struggle to capture deep cross-modal interactions due to inherent modality heterogeneity. To overcome these two challenges, this study proposes a multi-view cognitive prior graph convolutional network (MCP-GCN), which adopts domain generalization for emotion recognition. Particularly, two branch graphs are constructed: a functional connectivity branch based on neuropsychological knowledge and a data-driven branch with dynamic feature enhancement. The data-driven branch graph addresses the problem of modality heterogeneity using three constraint mechanisms. The MCP-GCN employs graph convolutional networks with multi-readout functions to capture local and global cognitive state features simultaneously. An attention-based fusion mechanism is developed to combine graph representations from both branches, which allows for enhancing affectiveembed dings. In addition, domain generalization methods are designed that explicitly consider subject-related covariates to extract subject-invariant emotional representations. The cross-subject evaluations achieve the accuracy of 96.00% (DREAMER) and 96.23% (MAHNOB-HCI), with subject-independent tests' accuracy reaching 98.03% (DREAMER) and 98.27% (MAHNOB HCI), demonstrating the state-of-the-art performance across both datasets. Finally, visualization results of multimodal connectivity matrices and graph structures reveal emotion-sensitive heart brain coupling, supporting biological interpretability of the MCP GCN.