Manual acupuncture manipulation recognition model with a multimodal fusion of tactile and visual features

Most traditional parametric analysis systems for manual acupuncture manipulation (MAM) fail to solve multimodal MAM recognition problems, due to the absence of robust MAM recognition models integrating both hand visual information and tactile feedback from the needle-holding (NH) fingers during MAM operations. This study attempts to fill this gap by constructing an explainable multimodal MAM recognition network using MAM video data captured by a binocular camera and tactile data from a self-developed wearable tactile array finger cot, which is worn on the acupuncturist’s fingers to collect tactile feedback during MAM operations. For this recognition model, we designed feature extraction methods for two tactile (dual-channel PVDF film piezoelectric signals and accelerometer signals) and two visual (joint motion data and RGB image data of the NH hand) submodalities. Next, we constructed a tactile-visual multimodal feature fusion frame, including a gating mechanism-based feature self-interaction memory strategy for each branch submodality of respective tactile and visual modalities, an interaction attention-based fusion method to generate the corresponding respective entire features of tactile and visual modality, and a tactile and visual multimodal feature fusion method using a low-rank fusion technique. Finally, we developed a feature element fine-tuning-based interpretability mechanism for the multimodal MAM recognition model. Thirty acupuncturists from TCM medical institutions in China validated the proposed multimodal MAM acquisition system and recognition model for four typical MAM techniques. Satisfactory results were obtained, proving the proposed method’s feasibility and effectiveness.