Hybrid BCI-based instruction set for dual robotic arm control using EEG and eye movement signals

Abstract A brain–computer interface (BCI) establishes a pathway for information transmission between a human (or animal) and an external device. It can be used to control devices such as prosthetic limbs and robotic arms, which in turn assist, rehabilitate, and enhance human limb function. At present, although most studies focus on brain signal acquisition, feature extraction and recognition, and further explore the use of brain signals to control external devices, the features obtained via noninvasive approaches are fewer and less robust, which makes it difficult to directly control devices with more degrees of freedom such as robotic arms. To address these issues, we propose an extended instruction set based on motor imagery that fuses eye-movement signals and electroencephalogram (EEG) signals for motion control of a dual collaborative robotic arm. The method incorporates spatio-temporal convolution and attention mechanisms for brain-signal classification. Starting from a small base of control commands, the hybrid BCI combining eye-movement signals and EEG expands the command set, enabling motion control of the dual cooperative manipulator. On the Webots simulation platform, we carried out kinematic control and three-dimensional motion simulation of a dual 6-degree-of-freedom collaborative robotic arm (UR3e). The experimental results demonstrate the feasibility of the proposed method. Our algorithm achieves an average accuracy of 83.8% with only 8.8k parameters, and the simulation results are within the expected range. The results demonstrate that the proposed extended instruction set based on motor imagery is effective not only for controlling dual collaborative robotic arms to perform grasping tasks in complex scenarios, but also for operating other multi-degree-of-freedom peripheral devices.