Federated Bayesian Optimization for Privacy-Preserving Neural Architecture Search

Deep neural networks have achieved remarkable success in various fields, leading to an increasing demand for automatic design of high-quality network architectures instead of manual design. Neural architecture search offers a solution to automatic tuning of network architectures and associated hyperparameters. Early work on NAS assumes that all the training data is stored centrally on a single device, which is unrealistic in many real-world applications. In recent years, there has been a surge in research on federated neural architecture search due to the growing amount of data on edge devices and the increasing concerns over security and privacy protection. It allows several parties to collaboratively search for an optimal model without leaking the privacy of the data. However, existing work on federated NAS transmits the entire weight parameters of the candidate architectures between the clients and the server, leading to high communication costs and potential risks of leaking raw data. To fill this gap, this paper proposes a federated Bayesian optimization NAS approach called FL-BONAS. Instead of uploading the original weight values to the server during each round, the clients cooperatively train an ensemble surrogate model as the neural predictor, and each client searches for its own architectures via Bayesian optimization without revealing either the local raw data or the architectures. To the best of our knowledge, this is the first work to train an ensemble model in Bayesian optimization for NAS under federated learning scenarios. FL-BONAS significantly improves the data efficiency while protecting privacy. The experimental results show that the proposed FL-BONAS can achieve better performance and protect raw data compared with centralized NAS.