Fuzziness based semi-supervised multimodal learning for patient’s activity recognition using RGBDT videos

Automatic recognition of bedridden patients’ physical activity has important applications in the clinical process. Such recognition tasks are usually accomplished on visual data captured by RGB, depth, and/or thermal cameras by utilizing supervised machine learning. However, supervised machine learning requires a large amount of labeled data and the accuracy depends on extracting appropriate features based on the domain knowledge. A plausible solution to these issues is using semi-supervised learning that focuses less on labeled data and domain knowledge. In this paper, a novel fuzziness-based semi-supervised multimodal learning algorithm, called (FSSL-PAR) is proposed for bedridden patient activity recognition. We use a synergistic interaction on RGB, Depth, and Thermal videos to assess the effect of visual multimodality for the first time in this semi-supervised learning setting. Experiments are conducted on a dataset collected by mimicking the patients with Acute Brain Injury (ABI) from a neurorehabilitation center. The results exhibit the superiority of the proposed algorithm over the existing supervised learning algorithms. We also see a positive correlation between the performance and the size of the labeled data in the proposed FSSL-PAR.