Federated Learning Strategies for Integrating Composite Meta-Consistency Loss With Multi-Head Attention

In order to solve the problem of accuracy decline caused by feature redundancy, this paper designs a federated learning strategy that combines composite meta-consistency loss and multi-head attention. Firstly, this paper decorrelates the features based on the dual theory of constraints to eliminate redundant information, and improves the stability of the model through gradient-based regularization. Composite meta-consistency loss is constructed based on these two optimization methods. Experiments show that compared with the latest algorithms, the maximum accuracy of CIFAR-10 and Oxford-Pets in this paper is improved by 0.82% and 2.19%, respectively. After that, this paper introduces multi-head attention into the framework of federated learning. After capturing richer context information in the process of feature extraction, the combination of inner-layer update and outer-layer update of the meta-learning method enables the federated learning framework to effectively cope with the data distribution of different clients and finally accelerate the convergence speed. Compared with other algorithms, the average accuracy of the first 40 rounds in the MINIST, CIFAR-10 and CIFAR-100 data sets is higher. In CIFAR-10, SVHN, Oxford-Pets, taking Robust-HDP as the benchmark, the speedup ratio reaches 1.5, 1.42, and 1.34, respectively, which is faster than other algorithms.