SDPMP: Inference Acceleration of CNN Models in Heterogeneous Edge Environment

The deployment of neural network models to edge devices for task inference is becoming increasingly common. However, due to network bandwidth and computing capability limitations, a single edge device is often insufficient to meet the inference delay requirements of computation-intensive tasks and continuous inference tasks with high load demands. To address this issue, we propose an efficient collaborative distributed inference acceleration framework that offloads tasks to multiple edge devices for parallel execution. The framework takes into account the speed of inference performed by heterogeneous edge devices and combines the principles of pipeline parallelism and the partial dependence characteristics of CNN intra-layer partitions. It also performs intelligent inter-layer and intra-layer partitions to maximize the throughput of continuous task inference and meet the requirements of low single task inference delay. We propose self-adaptive dynamic programming for model partition algorithm (SDPMP) to achieve this. Simulation results indicate that our proposed strategy significantly improves throughput compared to other classical algorithms.