脑电图
解码方法
脑-机接口
语音识别
计算机科学
卷积神经网络
神经解码
深度学习
刺激(心理学)
特征提取
人工智能
模式识别(心理学)
心理学
认知心理学
神经科学
电信
作者
Ciaran Cooney,Raffaella Folli,Damien Coyle
出处
期刊:IEEE Transactions on Biomedical Engineering
[Institute of Electrical and Electronics Engineers]
日期:2022-06-01
卷期号:69 (6): 1983-1994
被引量:11
标识
DOI:10.1109/tbme.2021.3132861
摘要
Brain-computer interfaces (BCI) studies are increasingly leveraging different attributes of multiple signal modalities simultaneously. Bimodal data acquisition protocols combining the temporal resolution of electroencephalography (EEG) with the spatial resolution of functional near-infrared spectroscopy (fNIRS) require novel approaches to decoding.We present an EEG-fNIRS Hybrid BCI that employs a new bimodal deep neural network architecture consisting of two convolutional sub-networks (subnets) to decode overt and imagined speech. Features from each subnet are fused before further feature extraction and classification. Nineteen participants performed overt and imagined speech in a novel cue-based paradigm enabling investigation of stimulus and linguistic effects on decoding.Using the hybrid approach, classification accuracies (46.31% and 34.29% for overt and imagined speech, respectively (chance: 25%)) indicated a significant improvement on EEG used independently for imagined speech (p = 0.020) while tending towards significance for overt speech (p = 0.098). In comparison with fNIRS, significant improvements for both speech-types were achieved with bimodal decoding (p<0.001). There was a mean difference of ∼12.02% between overt and imagined speech with accuracies as high as 87.18% and 53%. Deeper subnets enhanced performance while stimulus effected overt and imagined speech in significantly different ways.The bimodal approach was a significant improvement on unimodal results for several tasks. Results indicate the potential of multi-modal deep learning for enhancing neural signal decoding.This novel architecture can be used to enhance speech decoding from bimodal neural signals.
科研通智能强力驱动
Strongly Powered by AbleSci AI