计算机科学
解码方法
GSM演进的增强数据速率
算法
理论计算机科学
语言模型
编码(内存)
自然语言
人工智能
程序设计语言
钥匙(锁)
口译(哲学)
语义学(计算机科学)
计算
自然语言处理
作者
Yulin Chen,Meng Tian,Chao Qiu,X L Wang,Haipeng Yao,Yuan He,Yunhao Liu
出处
期刊:
[Institute of Electrical and Electronics Engineers]
日期:2026-05-18
卷期号:: 1-10
标识
DOI:10.1109/infocom59046.2026.11571680
摘要
The growing demand for deploying large language models (LLMs) in applications requiring low latency and strong privacy has brought increasing attention to edge inference. While distributing computation across edge devices helps alleviate resource bottlenecks, the sequential and layer-wise nature of autoregressive decoding still limits real-time deployment, leading to device underutilization, increased latency, and workload imbalance. Existing methods rely on heavyweight drafts and static policies, which cannot adapt to device heterogeneity and bandwidth variation, leading to limited parallelism and elevated latency. To bridge this gap, we propose EdgeSpec, a hierarchical speculative decoding framework at edges. Instead of relying on last-layer speculation, EdgeSpec enables intermediate transformer layers to emit draft tokens, which are validated and used to predict the next token in parallel across devices via attention masking. The system combines offline entropy profiling with an online Kalman filter to adaptively select confident trigger layers based on runtime load and input complexity. To further improve efficiency, EdgeSpec jointly optimizes model partitioning, entropy thresholds, and candidate sizes through a two-stage strategy that integrates offline planning with online adaptation. Experiments on LLaMA2 showed that EdgeSpec reduced inference latency by up to 17.1× while preserving output quality comparable to standard decoding.
科研通智能强力驱动
Strongly Powered by AbleSci AI