When Federated Learning Meets Knowledge Distillation

Federated learning (FL) has garnered significant attention in the Internet-of-things (IoT) domain due to its ability to facilitate collaborative learning among distributed privacy-sensitive devices without compromising their local data. However, FL's development and application are hindered by the heterogeneity, vulnerability, and limited computing and wireless communication resources of IoT systems. To solve the issues, knowledge distillation (KD), a technique that can transfer learned knowledge from one network to another in a quick and light manner, is incorporated into FL to enhance its usability and universality in IoT. KD-based FL can effectively achieve collaborative learning among resource-limited IoT devices that are heterogeneous in data distribution, model architectures, or quantity of resources, and offer enhanced privacy guarantees. Given the increasing adoption and benefits of KD in FL, it is essential to review KD-based FL schemes to identify the common methodologies and potential future directions. In this article, we examine the challenges associated with applying FL in IoT and sort out three key steps for incorporating KD into FL, along with the general workflow for KD-based FL. Furthermore, we conduct a comprehensive retrospective analysis of existing KD-based FL schemes and tease out their approaches to utilize KD to solve challenges, and then we compare them based on various design aspects. Based on our analysis and comparison, we shed light on several unresolved research questions that warrant further investigation.