MoEE: Mixture of Edge Experts for Collaborative Inference of Heterogeneous Models Based on Out-of-Distribution Detection

As the performance of Internet of Things (IoT) devices at the edge improves, deep learning models are increasingly being deployed on these devices to enhance the reliability of real-time data processing. However, the significant heterogeneity in computing power and storage capacity among devices results in local models that differ in type, size, and accuracy. Moreover, these models are typically trained with specific local datasets, leading to limited generalization when handling data from unseen or diverse environments. In open-world scenarios, inference requests often deviate from the local training distribution, causing local models to misclassify out-of-distribution (OoD) samples. Frequent retraining to address such issues is time-consuming, incurs substantial overhead, and may compromise accuracy on the original distribution. To overcome these challenges, this paper proposes Mixture of Edge Experts (MoEE), a collaborative inference and routing framework tailored for heterogeneous edge computing power networks (CPNs). MoEE enables devices to efficiently identify OoD samples and dynamically route them to suitable peers across the edge CPN, taking into account both inference accuracy and latency constraints. By intelligently orchestrating routing decisions based on device capabilities and sample characteristics, MoEE effectively utilizes distributed computing resources while avoiding unnecessary retraining. Extensive experiments with multiple heterogeneous deep neural network (DNN) models and diverse datasets demonstrate that MoEE significantly improves system efficiency in distributed edge AI scenarios.