Nighttime Pedestrian Detection Based on a Fusion of Visual Information and Millimeter-Wave Radar

A target detection algorithm based on the fusion of vision sensors and millimeter wave radar data can effectively improve the safety of self-driving vehicles. However, a single sensor cannot obtain comprehensive category status information on the target in the nighttime environment. To improve pedestrian detection in nighttime traffic scenarios, this paper proposes a nighttime pedestrian detection method based on the fusion of infrared vision information and millimeter wave (MMW) radar data. The lateral localization and category features of the target are obtained using the improved YOLOv5 deep learning algorithm, the distance and velocity information of the target is obtained by preprocessing the MMW radar acquisition data, the pedestrian target is tracked by using the extended Kalman filter for MMW radar detection, the projection of the valid radar target point on the IR image is completed according to the spatiotemporal fusion, and then the correlation gate method is used to correlate the data, and the visual information is inherited to the radar points to get the target multimodal information, and the success associated valid target sequences are weighted to obtain the accurate target position. Finally, a decision-level fusion algorithm framework is proposed to complete the output of pedestrian multimodal information in nighttime traffic scenes. Theoretical analysis and experimental results show that the accuracy and robustness of this method for nighttime pedestrian recognition are better than those of a single sensor.