FER-FL: Robust federated learning for facial expression recognition in heterogeneous environments

Vision-based facial expression recognition (FER) has shown strong potential across numerous applications, providing a non-invasive and real-time approach for analyzing emotional states and social intentions. However, deploying FER systems in practice often raises concerns about centralized facial data collection, as large-scale data from individual users or healthcare settings may contain sensitive personal information. Federated learning (FL), as a distributed training paradigm, reduces the need to share raw facial images by enabling collaborative model learning across decentralized clients. Despite this advantage, FL in the FER domain faces a major challenge: the non-independent and identically distributed (non-IID) nature of facial data, caused by variations in age, gender, ethnicity, identity, illumination, and individual expression styles. These discrepancies often lead to client drift, slow convergence, and reduced model accuracy. To address these issues, we propose FER-FL, a federated learning framework specifically designed for FER under heterogeneous client distributions. FER-FL introduces a client-side variance-reduced primal–dual optimization strategy to mitigate client drift and improve local–global consistency. In addition, a server-side public-data calibration mechanism is incorporated to refine the global decision boundary through KL-based distillation. Comprehensive experiments on FERPlus, RAF-DB, and FER2013 demonstrate that FER-FL improves Top-1 accuracy by up to 7.0% over FedAvg and accelerates convergence by up to 5.1 × , while maintaining stable performance under different levels of data heterogeneity and client participation. The proposed framework provides an effective privacy-aware federated training solution for FER without requiring the exchange of raw facial images.