High-Throughput CNN Inference on Embedded ARM Big.LITTLE Multicore Processors

IoT Edge intelligence requires Convolutional Neural Network (CNN) inference\nto take place in the edge devices itself. ARM big.LITTLE architecture is at the\nheart of prevalent commercial edge devices. It comprises of single-ISA\nheterogeneous cores grouped into multiple homogeneous clusters that enable\npower and performance trade-offs. All cores are expected to be simultaneously\nemployed in inference to attain maximal throughput. However, high communication\noverhead involved in parallelization of computations from convolution kernels\nacross clusters is detrimental to throughput. We present an alternative\nframework called Pipe-it that employs pipelined design to split convolutional\nlayers across clusters while limiting parallelization of their respective\nkernels to the assigned cluster. We develop a performance-prediction model that\nutilizes only the convolutional layer descriptors to predict the execution time\nof each layer individually on all permitted core configurations (type and\ncount). Pipe-it then exploits the predictions to create a balanced pipeline\nusing an efficient design space exploration algorithm. Pipe-it on average\nresults in a 39% higher throughput than the highest antecedent throughput.\n