Multi-sensor fusion federated learning method of human posture recognition for dual-arm nursing robots

Human posture estimation is a significant pre-imperative for intelligent nursing robot. However, most existing studies suffer from the driven data deficiencies, because of the concerns about poor information safety in the private healthcare environment. Federated Learning is able to solve the problem, as it can improve the learning ability of the models as well as protecting the data privacy. In this paper, we proposed a Federated Learning Human Posture Recognition (FL-HPR) framework according to image and point cloud fusion. FL-HPR benefits the information flow in the global model while ensuring the data privacy of local models. Furthermore, FL-HPR optimizes the local dynamic graph edge convolution network of robot, which improves the recognition accuracy of each body limb and enhance robustness. Experiments on Non-IID datasets illustrate that the presented FL-HPR remarkably outperforms non-federated learning methods. The developed frame improves the accuracy of human joint estimation, which indicates the proposed FL-HPR can be integrated into the intelligent nursing robot in practical. The open-source code and videos are available at https://github.com/Hebut-LEVO/.