Reliable and Efficient LLM Inference on Resource-Constrained Mobile Devices via Dynamic Scheduling

Large Language Models (LLMs) have demonstrated exceptional natural language understanding and generation capabilities in many emerging applications. However, providing end users with real-time LLM inference on their mobile devices poses substantial challenges. In this work, we propose a fully distributed LLM inference framework, where spare computational resources across heterogeneous mobile devices are effectively harnessed. Our approach integrates two key components: initial model preloading and layer-wise execution scheduling. The proposed methods tackle two major challenges, optimal parameter preloading without prior knowledge of device availability and real-time inference scheduling under dynamic conditions. We evaluate the performance of our framework in real-world environments, experimental results show that it significantly reduces end-to-end delay by 44%∼83% compared to existing approaches.