A multimodal approach to estimating vigilance in SSVEP-based BCI

Brain-computer interface (BCI) is a communication system that allows a direct connection between the human brain and external devices, which is able to provide assistance and improve the quality of life for people with disabilities. Vigilance is an important cognitive state and plays an important role in human–computer interaction. In BCI tasks, the low-vigilance state of the BCI user would lead to the performance degradation. Therefore, it is desirable to develop an efficient method to estimate the vigilance state of BCI users. In this study, we built a 4-target BCI system based on steady-state visual evoked potential (SSVEP) for cursor control. Electroencephalogram (EEG) and electrooculogram (EOG) were recorded simultaneously from 18 subjects during a 90-min continuous cursor-control BCI task. We proposed a multimodal vigilance estimating network, named MVENet, to estimate the vigilance state of BCI users through the multimodal signals. In this architecture, a spatial-temporal convolution module with an attention mechanism was adopted to explore the temporal-spatial information of the EEG features, and a long short-term memory module was utilized to learn the temporal dependencies of EOG features. Moreover, a fusion mechanism was built to fuse the EEG representations and EOG representations effectively. Experimental results showed that the proposed network achieved a better performance than the compared methods. These results demonstrate the feasibility and effectiveness of our methods for estimating the vigilance state of BCI users.