FL-FD: Federated learning-based fall detection with multimodal data fusion

Multimodal data fusion is a critical element of fall detection systems, as it provides more comprehensive information than single-modal data. Yet, data heterogeneity between sources has posed a challenge for the effective fusion of such data. This paper proposes a novel multimodal data fusion method under a federated learning (FL) framework that addresses the privacy concerns of users while exploiting the complementarity of such data. Specifically, we fuse time-series data from wearable sensors and visual data from cameras at the input level, where the data is first transformed into images using the Gramian Angular Field (GAF) method. Moreover, each user is treated as a private client in the FL system whereby the fall detection model is trained without requiring the sharing of user data. The proposed method is evaluated using the UP-Fall dataset, where we perform different fall detection tasks: binary classification for fall and non-fall detection yields a remarkable accuracy of 99.927%, while multi-classification for different fall activity recognition attains an accurate result of 89.769%.