APFed: Attention-based Personalized Federated Learning in Model Heterogeneity Scenario

In recent years, federated learning (FL) has gradually become one of the leading frameworks for deploying edge computing due to its privacy attributes and efficient training performance. Due to the high degrees of system heterogeneity and statistical heterogeneity owned by users’ devices, these devices need to train heterogeneous models for personalized learning requirements. However, existing FL schemes have focused on the global aggregation of a common model architecture, and thus cannot facilitate FL across edge devices with heterogeneous model architectures. To address this problem, this paper proposes a novel Attention-based Personalized Federated Learning framework called APFed that can aggregate potential heterogeneous models by users to tackles both system and statistical heterogeneity. Different from the parameter average aggregation of traditional FL, APFed embeds different locally updated parameters of clients and different local models as tokens, taking into account both system, statistical and model heterogeneity, thus achieving personalized model aggregation.