TransVOD: End-to-End Video Object Detection With Spatial-Temporal Transformers

计算机科学 人工智能 目标检测 变压器 编码器 计算机视觉 特征提取 模式识别(心理学) 工程类 操作系统 电气工程 电压
作者
Qianyu Zhou,Xiangtai Li,Lu H,Yibo Yang,Guangliang Cheng,Yunhai Tong,Lizhuang Ma,Dacheng Tao
出处
期刊:IEEE Transactions on Pattern Analysis and Machine Intelligence [IEEE Computer Society]
卷期号:45 (6): 7853-7869 被引量:201
标识
DOI:10.1109/tpami.2022.3223955
摘要

Detection Transformer (DETR) and Deformable DETR have been proposed to eliminate the need for many hand-designed components in object detection while demonstrating good performance as previous complex hand-crafted detectors. However, their performance on Video Object Detection (VOD) has not been well explored. In this paper, we present TransVOD, the first end-to-end video object detection system based on simple yet effective spatial-temporal Transformer architectures. The first goal of this paper is to streamline the pipeline of current VOD, effectively removing the need for many hand-crafted components for feature aggregation, e.g., optical flow model, relation networks. Besides, benefited from the object query design in DETR, our method does not need post-processing methods such as Seq-NMS. In particular, we present a temporal Transformer to aggregate both the spatial object queries and the feature memories of each frame. Our temporal transformer consists of two components: Temporal Query Encoder (TQE) to fuse object queries, and Temporal Deformable Transformer Decoder (TDTD) to obtain current frame detection results. These designs boost the strong baseline deformable DETR by a significant margin (3 %-4 % mAP) on the ImageNet VID dataset. TransVOD yields comparable performances on the benchmark of ImageNet VID. Then, we present two improved versions of TransVOD including TransVOD++ and TransVOD Lite. The former fuses object-level information into object query via dynamic convolution while the latter models the entire video clips as the output to speed up the inference time. We give detailed analysis of all three models in the experiment part. In particular, our proposed TransVOD++ sets a new state-of-the-art record in terms of accuracy on ImageNet VID with 90.0 % mAP. Our proposed TransVOD Lite also achieves the best speed and accuracy trade-off with 83.7 % mAP while running at around 30 FPS on a single V100 GPU device. Code and models are available at https://github.com/SJTU-LuHe/TransVOD.
最长约 10秒,即可获得该文献文件

科研通智能强力驱动
Strongly Powered by AbleSci AI
科研通是完全免费的文献互助平台,具备全网最快的应助速度,最高的求助完成率。 对每一个文献求助,科研通都将尽心尽力,给求助人一个满意的交代。
实时播报
流年完成签到 ,获得积分10
刚刚
刚刚
在水一方的应助被Skyler采纳,获得10
刚刚
科研通AI6.2的应助被kinlo97采纳,获得10
刚刚
WY发布了新的文献求助10
1秒前
1秒前
yuiiuy发布了新的文献求助10
1秒前
认真水平完成签到,获得积分10
1秒前
ding的应助被lykkk采纳,获得10
2秒前
Hantheex发布了新的文献求助10
2秒前
2秒前
2秒前
王彬发布了新的文献求助10
2秒前
3秒前
礼拜九完成签到,获得积分20
3秒前
3秒前
风驻云停完成签到,获得积分10
4秒前
科研通AI2S的应助被andjdd采纳,获得10
4秒前
彩色德天发布了新的文献求助10
4秒前
4秒前
4秒前
流水落花发布了新的文献求助10
4秒前
充电宝的应助被秾晓豆采纳,获得10
5秒前
竹墨笙箫完成签到,获得积分10
5秒前
于心发布了新的文献求助30
6秒前
6秒前
别来找喔完成签到 ,获得积分10
6秒前
开心鱼完成签到,获得积分20
6秒前
Hantheex完成签到,获得积分20
6秒前
6秒前
QIAO发布了新的文献求助10
6秒前
.....发布了新的文献求助10
6秒前
6秒前
科研通AI6.2的应助被草本采纳,获得10
6秒前
无极微光的应助被小鹿啊采纳,获得20
7秒前
QingY发布了新的文献求助10
7秒前
杨克学发布了新的文献求助10
7秒前
7秒前
一切顺利完成签到,获得积分10
8秒前
bkagyin的应助被合梦易采纳,获得10
8秒前
高分求助中
(应助此贴封号)【重要!!请各用户(尤其是新用户)详细阅读】【科研通的精品贴汇总】 10000
拉伸高分子熔体中缠结行为的模拟和小角中子散射研究 1500
The World Atlas of Wine 9th edition 1000
奈特基础组织学:配套组织病理学(第4版)(Netter's Essential Histology With Correlated Histopathology) 800
John McDowells Bildungstheorie 800
八旗制度の研究 600
Analyses de reconnaissances de dettes passées devant les échevins d'Ypres (1249-1291) 600
热门求助领域 (近24小时)
化学 材料科学 医学 生物 纳米技术 计算机科学 工程类 有机化学 化学工程 生物化学 物理 复合材料 内科学 催化作用 光电子学 人工智能 细胞生物学 无机化学 物理化学 数学
热门帖子
关注 科研通微信公众号,转发送积分 8162621
求助须知:如何正确求助?哪些是违规求助? 9556889
关于积分的说明 21737583
捐赠科研通 7753002
什么是DOI,文献DOI怎么找? 3427836
关于科研通互助平台的介绍 2552719
邀请新用户注册赠送积分活动 2472507