Soft Optoelectronic Sensors with Deep Learning for Gesture Recognition

Abstract With the rapid development of deep learning and computing power, human–computer interactions, and interfaces are attracting attentions in industrial and academic research. Flexible human–computer interaction can greatly improve productivity and enable robots to work in extreme environments that humans cannot tolerate. The research of gesture recognition is emerging and provides a new way of studying the human–computer interactions. However, compared with the entire human body, human hands are dexterous organs with more complex and flexible joints, which makes hand gesture recognition a challenging problem. Here, a robust and cost‐effective gesture recognition system is reported through the soft optoelectronic sensors. An array of polymer‐encapsulated U‐shaped microfiber (UMF) attached to a glove is fabricated for sensitive finger motion detection. The anisotropic strain response of UMF is measured with a sensitivity of 15.98 (2.20) in the x ‐direction ( y ‐direction). A deep learning network (VGGNet) is developed to process the optical signals for analyzing and classifying hand gestures. The experiments show that VGGNet has high recognition accuracy of 99.2% for the test datasets with ten classified gestures. This work provides a potential optical interface in studying gesture recognition and biomechanical signatures, which can also be applied in virtual reality systems and interactive game platforms.