C4RFNet: Camera and 4D-Radar Fusion Network for Point Cloud Enhancement

In the landscape of recent technological advancements, the advent of 4D millimeter-wave radar has ushered in a new era of data quality improvements, showcasing the potential to rival, or even surpass, Lidar systems. Despite its innovative prowess, the lower density and accuracy of 4D millimeter-wave radar's point clouds, in comparison to those generated by Lidar, pose significant limitations to the technology's broader application. Addressing these constraints, our research introduces a comprehensive, end-to-end methodology for augmenting point cloud data through a fusion of monocular camera imagery and 4D millimeter-wave radar. Firstly, the monocular image is transformed into a pseudo-point cloud. Subsequently, features from both the radar-generated point clouds and the pseudo-point clouds are independently extracted and merged using two distinct feature extraction modules. To refine this process further, a novel loss function is designed, taking into account the global and local feature consistency between the reconstructed point cloud and the Lidar raw point cloud. The experimental results, particularly within the realms of object detection, illustrate a marked enhancement in point cloud quality over the baseline provided by native 4D millimeter-wave radar outputs. Additionally, the application of this method to Simultaneous Localization and Mapping (SLAM) demonstrates a significant improvement in accuracy, achieving a level of performance that is competitive with Lidar. Notably, the proposed method is low computational demand, enabling real-time inference within a mere 30ms on resource-constrained platforms such as the NVIDIA Jetson Nano 2G.