Real-Time Volumetric Perception for Unmanned Surface Vehicles Through Fusion of Radar and Camera

In recent years, unmanned surface vehicles (USVs) have played an increasingly important role in various applications. Due to the expansion of USV application scenes from common marine areas to inland waters with complex environments, environmental perception has become an essential requirement for autonomous navigation systems of USVs. Traditional perception methods utilize either LiDAR or radar to construct volumetric maps for environmental perception. To improve the accuracy of perception systems and reduce deployment costs, this paper proposes a novel radar and camera fusion volumetric map network named FVMNet for real-time volumetric perception. FVMNet is based on a novel radar and image fusion architecture and comprises four modules. 1) The radar and image encoders can extract different features; 2) Only using in training stage without extra valid time costs, auxiliary segmentation head advances the image encoder; 3) To eliminate the representation difference between image features and radar features, the BEV spatial transformer module transfers image feature representations from the perspective view to BEV space; 4) The fusion segmentation head predicts the volumetric perception results. Compared to other baseline methods that use a single modality, FVMNet achieves state-of-the-art accuracy in public USVInland dataset and our collected wharf dataset. We conducted comprehensive ablation experiments to validate the efficacy of the designed modules. Moreover, the proposed method demonstrates generalization in zero-shot real-world scenarios and robustness under extreme weather conditions.