Deep learning based camera and radar fusion for object detection and classification

Autonomous Vehicles (AV) require a very highly accurate perception system to reduce the likelihood of road accidents, which are most commonly caused by unrecognizable targets, human mistakes, and other avoidable reasons. This is achieved through the use of Modern AV such as cameras, radars, and LIDARs. One of the most commonly used sensors in automotive industries and traffic control applications was the Millimeter-wave radar for its high performance. However, these types of sensors are expensive, and suffer from disclosing false alarms. A recent approach is using object detection and classification algorithms along with a car-mounted camera to solve this issue. The fusion of camera and Radar measurements provides a much more efficient detection system. In this paper, we introduce a more robust approach that fuses camera and radar outputs using neural networks and provides more reliable level accuracy for low-quality radar readings. In our approach, we use only the box size (box height and box width) predictions of the YOLO-v4, with simulated noise radar readings to classify car types. The proposed method can learn to improve object detection of radar measurements and furthermore classify car types with 60.0% accuracy when 10% noise is present in the radar readings. Our proposed method shows that it is possible to use cheaper radar sensors, along with a budget camera, and still provide predictions of car types.