安全性令牌
解码方法
计算机科学
调度(生产过程)
推论
算法
性能指标
地铁列车时刻表
随机过程
公制(单位)
概率分布
延迟(音频)
过程(计算)
分布(数学)
实时计算
源代码
统计推断
数学优化
简单(哲学)
作者
Haoyu Zheng,Yongqiang Zhang,Fangcheng Fu,Xiaokai Zhou,Hao Luo,Hongchao Zhu,Yuanyuan Zhu,Hao Wang,Xiao Yan,Jiawei Jiang
标识
DOI:10.48550/arxiv.2604.00499
摘要
To schedule LLM inference, the \textit{shortest job first} (SJF) principle is favorable by prioritizing requests with short output lengths to avoid head-of-line (HOL) blocking. Existing methods usually predict a single output length for each request to facilitate scheduling. We argue that such a \textit{point estimate} does not match the \textit{stochastic} decoding process of LLM inference, where output length is \textit{uncertain} by nature and determined by when the end-of-sequence (EOS) token is sampled. Hence, the output length of each request should be fitted with a distribution rather than a single value. With an in-depth analysis of empirical data and the stochastic decoding process, we observe that output length follows a heavy-tailed distribution and can be fitted with the log-t distribution. On this basis, we propose a simple metric called Tail Inflated Expectation (TIE) to replace the output length in SJF scheduling, which adjusts the expectation of a log-t distribution with its tail probabilities to account for the risk that a request generates long outputs. To evaluate our TIE scheduler, we compare it with three strong baselines, and the results show that TIE reduces the per-token latency by $2.31\times$ for online inference and improves throughput by $1.42\times$ for offline data generation.
科研通智能强力驱动
Strongly Powered by AbleSci AI