Portable vision-based gait assessment for post-stroke rehabilitation using an attention-based lightweight CNN

Stroke is one of the leading causes of disability due to a lack of blood flow and bleeding in the brain, resulting in hemiplegia gait. Usually, after being treated at the hospital, patients still need ongoing rehabilitation training post-discharge. Thus, a specific and automatic monitoring system that assesses recovery outside the hospital is necessary. Gait analysis based on deep learning has great potential for this target because different degrees of hemiplegia gait characterized by spasms, poor balance, and coordination can reflect the recovery of patients. Therefore, this work presents a novel portable vision-based system to assess recovery from stroke by observing gait based on deep learning, whose operation does not require the presence of skilled technicians and can be performed almost anytime and anywhere. The proposed system offers real-time gait assessment on a mobile device for home-based stroke rehabilitation. It consists of the following main components: data acquisition, pre-processing and gait representation learning to map to severity levels. Specifically, 2D walking sequences are captured by a camera, and then the key points of a skeleton of the walking subject are detected to create a template-based feature (i.e., skeleton energy image (SEI)) as the input modality. Subsequently, discriminative gait representations are learned by our proposed attention-based lightweight CNN, which is composed of a lightweight CNN architecture and a multi-dimensional attention module. Finally, the representation is mapped into one hemiplegia severity level, achieving end-to-end and data-driven classification. The proposed system is evaluated on two datasets: (i) a public pathological dataset GAIT-IST, which includes four different types of pathological gaits and one normal gait, containing 10 subjects. (ii) Our self-constructed dataset, which consists of three severity levels of hemiplegia gaits in different recovery stages after stroke and one normal gait, containing 14 subjects. On the two datasets, the proposed system achieved classification accuracies of 96.91% and 98.10%, which were 0.70% and 1.56% higher than those of the SOTA model, respectively. More importantly, the number of parameters of our method drops by 53 times, and the number of floating point operations per second (Flops) drops by 111 times. The experimental results demonstrate that our system achieves a good hemiplegia classification accuracy and is lightweight and feasible for deployment on a mobile device that is convenient for a daily life setting. The system is a potential tool that can enable more regular evaluations and support the detection of recovery progress in daily post-stroke rehabilitation.