Neural Network Inference on Mobile SoCs

The ever-increasing demand from mobile Machine Learning (ML) applications\ncalls for evermore powerful on-chip computing resources. Mobile devices are\nempowered with heterogeneous multi-processor Systems-on-Chips (SoCs) to process\nML workloads such as Convolutional Neural Network (CNN) inference. Mobile SoCs\nhouse several different types of ML capable components on-die, such as CPU,\nGPU, and accelerators. These different components are capable of independently\nperforming inference but with very different power-performance characteristics.\nIn this article, we provide a quantitative evaluation of the inference\ncapabilities of the different components on mobile SoCs. We also present\ninsights behind their respective power-performance behavior. Finally, we\nexplore the performance limit of the mobile SoCs by synergistically engaging\nall the components concurrently. We observe that a mobile SoC provides up to 2x\nimprovement with parallel inference when all its components are engaged, as\nopposed to engaging only one component.\n