pFedMoE: Data-Level Personalization With Mixture of Experts in Model-Heterogeneous Personalized Federated Learning

With growing client diversity, model-heterogeneous personalized federated learning (MHPFL) supports collaboration over structure-heterogeneous client models. However, existing MHPFL methods only achieve client-level personalization but ignore inherent discrepancies within each client's different data samples, leading to limited model performance. To this end, we propose a novel model-heterogeneous personalized Federated learning with Mixture of Experts (pFedMoE) to achieve a fine-grained data-level personalization. As the first work that incorporates MoE in MHPFL, it introduces three innovations: (1) Different clients hold heterogeneous local models, we add a small proxy global homogeneous feature extractor shared by clients for knowledge exchange. (2) To achieve a fine-grained data-level personalization, we construct a personalized local MoE for each client: a local expert (local heterogeneous client model's feature extractor), a global expert (global proxy homogeneous feature extractor), and a local personalized gating network, which dynamically balances the generalization and personalization of the local model at the data sample level. (3) We customize a lightweight linear gating network to capture the generalized and personalized data characteristics of each local data sample. We theoretically prove its $\mathcal {O}(1/T)$ convergence rate. Experiments on 3 benchmark image datasets, 1 real-world image dataset and 1 real-world text dataset against 9 baselines demonstrate its state-of-the-art model accuracy with up to 2.79% accuracy improvement while saving up to 43.12% computational overheads and keeping satisfactory communication costs.