V-Rex: Real-Time Streaming Video LLM Acceleration via Dynamic KV Cache Retrieval

计算机科学 加速度 隐藏物 实时计算 计算机硬件 管道(软件) 嵌入式系统 计算机图形学(图像) 噪音(视频) 移动设备
作者
Donghyuk Kim,Sejeong Yang,Wonjin Shin,Joo-Young Kim
出处
期刊:Proceedings [Institute of Electrical and Electronics Engineers]
卷期号:: 1-14 被引量:14
标识
DOI:10.1109/hpca68181.2026.11408603
摘要

Streaming video large language models (LLMs) are increasingly used for real-time multimodal tasks such as video captioning, question answering, conversational agents, and augmented reality. However, these models face fundamental memory and computational challenges because their key-value (KV) caches grow substantially with continuous streaming video input. This process requires an iterative prefill stage, which is a unique feature of streaming video LLMs. Prior works reduce excessive cache overhead by utilizing the KV cache retrieval algorithm, which offloads the full KV cache to CPU memory or storage, then selectively fetches the most relevant entries. Nevertheless, due to its iterative prefill stage, they suffer from significant limitations, including extensive computation, substantial data transfer, and degradation in accuracy. Crucially, this issue is exacerbated for edge deployment, which is the primary target for these models. The memory footprint exceeds the memory capacity within minutes of video streams, making low-latency, energy-efficient inference infeasible. In this work, we propose V-Rex, the first software-hardware co-designed accelerator that comprehensively addresses both algorithmic and hardware bottlenecks in streaming video LLM inference. At its core, V-Rex introduces ReSV, a training-free dynamic KV cache retrieval algorithm. ReSV exploits temporal and spatial similarity-based token clustering to reduce excessive KV cache memory across video frames, and dynamically adjusts token selection per transformer layer and attention head to minimize the number of selected tokens. To fully realize these algorithmic benefits, V-Rex offers a compact, low-latency hardware accelerator with a dynamic KV cache retrieval engine (DRE), featuring bit-level and early-exit based computing units, as well as hierarchical KV cache memory management. Evaluated on COIN benchmarks, V-Rex achieves unprecedented real-time of 3.9-8.3 FPS and energy-efficient streaming video LLM inference on edge deployment with negligible accuracy loss. While DRE only accounts for 2.2% power and 2.0% area, the system delivers 1.9-19.7× speedup and 3.1-18.5× energy efficiency improvements over AGX Orin GPU. This work is the first to comprehensively tackle KV cache retrieval across algorithm and hardware, enabling real-time streaming video LLM inference on resource-constrained edge devices, with clear potential for scalable deployment in large-scale server environments.
最长约 10秒,即可获得该文献文件

科研通智能强力驱动
Strongly Powered by AbleSci AI
科研通是完全免费的文献互助平台,具备全网最快的应助速度,最高的求助完成率。 对每一个文献求助,科研通都将尽心尽力,给求助人一个满意的交代。
实时播报
ZC发布了新的文献求助10
1秒前
1秒前
鱼鱼鱼发布了新的文献求助10
2秒前
科研通AI65的应助被zzk采纳,获得10
2秒前
cys发布了新的文献求助10
2秒前
3秒前
可爱的函函的应助被王永顺采纳,获得10
5秒前
小叶子的太阳完成签到,获得积分10
5秒前
科研通AI6.4的应助被Frank采纳,获得10
5秒前
6秒前
小红帽发布了新的文献求助10
7秒前
luoluo发布了新的文献求助10
7秒前
Mortal发布了新的文献求助10
7秒前
内啡呔发布了新的文献求助10
8秒前
丘比特的应助被xiyiding采纳,获得10
9秒前
10秒前
ZC完成签到,获得积分10
10秒前
香蕉觅云的应助被无私的沁采纳,获得10
12秒前
斯文败类的应助被Endeavor采纳,获得10
12秒前
PASSl0N完成签到,获得积分10
14秒前
Ava的应助被咕嘎采纳,获得10
14秒前
14秒前
Mi完成签到,获得积分10
15秒前
Hh发布了新的文献求助10
15秒前
王永顺完成签到,获得积分20
15秒前
Austin111327完成签到 ,获得积分10
16秒前
白羊发布了新的文献求助30
17秒前
乐乐的应助被小红帽采纳,获得10
17秒前
17秒前
19秒前
19秒前
852的应助被Cylara采纳,获得10
20秒前
香蕉觅云的应助被琴酒采纳,获得10
20秒前
Aki完成签到,获得积分10
20秒前
Akim的应助被Mi采纳,获得10
20秒前
铁甲小宝完成签到 ,获得积分10
20秒前
王永顺发布了新的文献求助10
21秒前
阿孟孟完成签到 ,获得积分10
21秒前
yunyueqixun完成签到,获得积分10
22秒前
郝佳音发布了新的文献求助10
23秒前
高分求助中
(应助此贴封号)【重要!!请各用户(尤其是新用户)详细阅读】【科研通的精品贴汇总】 10000
拉伸高分子熔体中缠结行为的模拟和小角中子散射研究 2000
Agyptische Geschichte der 21.30. Dynastie 510
高分子基功能界面水蒸发材料与应用研究 500
外场增强的水汽发电研究 500
基于结构调控的功能化水诱导发电材料与器件研究 500
基于界面调控的高效湿气发电材料与器件研究 500
热门求助领域 (近24小时)
化学 材料科学 医学 生物 纳米技术 计算机科学 工程类 有机化学 化学工程 生物化学 物理 复合材料 内科学 催化作用 光电子学 人工智能 细胞生物学 无机化学 物理化学 数学
热门帖子
关注 科研通微信公众号,转发送积分 8192647
求助须知:如何正确求助?哪些是违规求助? 9579118
关于积分的说明 21863752
捐赠科研通 7792215
什么是DOI,文献DOI怎么找? 3436974
关于科研通互助平台的介绍 2560633
邀请新用户注册赠送积分活动 2483864