Monocular 3D Ray-Aware RPN For Roadside View Object Detection

3D perception is one of the most important tasks of autonomous vehicles. Both methods based on expensive LiDAR and stereo cameras, and methods based on monocular cameras, have achieved great success in 3D object detection from vehicle view. The roadside view, as an important component of the entire intelligent transportation system, has distinct features from the vehicle's forward view. The 3D object detection from the roadside view has enormous research and application value. However, current research on 3D object detection from roadside view lags far behind the research on 3D object detection from vehicle view. Based on the work of M3D-RPN, we analyze the differences in sample space between roadside view and vehicle view. We find that although the post-optimization based on 2D-3D geometric consistency can improve 3D detection performance in the front view of the vehicle, it can reduce the performance of 3D detection in the roadside view. At the same time, to adapt to the characteristics of the roadside view, we propose a novel ray-aware convolution to replace the depth-aware convolution for the vehicle view. Compared to the M3D-RPN, our proposed M3D-RA-RPN improves the performance of monocular 3D object detection and BEV object detection on the Rope3d dataset.