CAF-RCNN: multimodal 3D object detection with cross-attention

ABSTRACTLiDAR and camera are pivotal sensors of 3D (three-dimensional) object detection. As a result of their different characteristics, increasingly multimodal-based object detection methods have been proposed. Now, popular methods are to hardly associate camera features with LiDAR features, but the features are frequently enhanced and aggregated, so there is a major challenge in how to align two features effectively. Therefore, we propose CAF-RCNN. On the basis of PointRCNN, using Feature Pyramid Network (FPN) to extract advanced semantic features at different scales, then fusing these features with the LiDAR features of the Set Abstraction (SA) module output in PointRCNN and subsequent steps. Regarding the features fusion module, we design a module based on the cross-attention mechanism, CAFM (Cross-Attention Fusion Module). It combines two channel attention streams in a cross-over fashion to utilize rich details about significant objects in the Image Stream and Geometric Stream. We did a lot of experiments on the KITTI dataset, and the result shows that our method is 6.43% higher than PointRCNN in 3D accuracy.KEYWORDS: 3D object detectionmultimodal fusioncross-attention mechanismfeature pyramid network Disclosure statementNo potential conflict of interest was reported by the author(s).Data availability statementThe data that support the findings of this study are available on request from the corresponding author, Deer Liu, upon reasonable request.Additional informationFundingThis work was supported by the National Natural Science Foundation of China under Grant 42271434 and Natural Science Foundation of Jiangxi Province under Grant 20202BAB202025.