多光谱图像
计算机科学
人工智能
计算机视觉
像素
目标检测
卷积神经网络
传感器融合
模式识别(心理学)
作者
Yung-Yao Chen,Sin-Ye Jhong,Y. Martin Lo
出处
期刊:IEEE Sensors Journal
[Institute of Electrical and Electronics Engineers]
日期:2023-11-01
卷期号:23 (21): 26873-26886
标识
DOI:10.1109/jsen.2023.3319230
摘要
Multispectral object detection using visible–thermal vision sensors is a crucial method for enabling intelligent vehicles to perceive their environment accurately. However, fusing and complementing multispectral information is challenging because of problems such as miscalibration resulting from disparate camera fields of view (FOV) and pattern discrepancies in images captured at different wavelengths. To address these challenges, we propose a novel multispectral object detection system named the reinforcement- and alignment-based fusion model (RA-FMDet). This system integrates a cross-modality reinforcement (CMR) module with a hierarchical modality alignment (HMA) strategy. The CMR module uses hybrid transformer–convolutional neural network (CNN) architecture for the effective extraction of high-quality semantic representations from multispectral data. The HMA strategy addresses miscalibration at the pixel and anchor levels. To evaluate RA-FMDet, we established a comprehensive dataset covering various scenarios and conditions. This dataset was obtained by optimizing the hardware configuration of visible–thermal vision sensors and mitigating the shift between image modalities through the use of a modified calibration board. Extensive experiments conducted on the challenging KAIST dataset and the established dataset indicated that the proposed RA-FMDet outperformed state-of-the-art models in object detection and achieved mean average precision values of 92.60%, 93.37%, and 93.60% in the detection of cars, pedestrians, and motorcyclists, respectively.
科研通智能强力驱动
Strongly Powered by AbleSci AI