计算机科学
人工智能
推论
钥匙(锁)
光学(聚焦)
特征(语言学)
数据挖掘
机器学习
领域(数学)
鉴定(生物学)
作者
Chuhao Xu,Zijun LI,Quan Chen,Han Zhao,Xueyan Tang,Minyi Guo
出处
期刊:Proceedings
[Institute of Electrical and Electronics Engineers]
日期:2026-01-31
卷期号:: 1-18
被引量:8
标识
DOI:10.1109/hpca68181.2026.11408548
摘要
The rise of LLMs has driven demand for private serverless deployments, characterized by moderate-sized models and infrequent requests. While existing serverless solutions follow exclusive GPU allocation, we take a step back to explore modern platforms and find that: Emerging CPU architectures with built-in accelerators are capable of serving LLMs but remain underutilized, and both CPUs and GPUs can accommodate multiple LLMs simultaneously. We propose SLINFER, a resource-efficient serverless inference scheme tailored for small- to mid-sized LLMs that enables elastic and on-demand sharing across heterogeneous hardware. SLINFER tackles three fundamental challenges: (1) precise, fine-grained compute resource allocation at token-level to handle fluctuating computational demands; (2) a coordinated and forward-looking memory scaling mechanism to detect out-ofmemory hazards and reduce operational overhead; and (3) a dual approach that consolidates fragmented instances through proactive preemption and reactive bin-packing. Experimental results on 4 32-core CPUs and 4 A100 GPUs show that SLINFER improves serving capacity by 47% - 62% through sharing, while further leveraging CPUs boosts this to 86% - 154%.
科研通智能强力驱动
Strongly Powered by AbleSci AI