Multi-Modal Sensor Fusion-Based Deep Neural Network for End-to-End Autonomous Driving With Scene Understanding

This study aims to improve the performance and generalization capability of \nend-to-end autonomous driving with scene understanding leveraging deep learning \nand multimodal sensor fusion techniques. The designed end-to-end deep neural \nnetwork takes as input the visual image and associated depth information in an \nearly fusion level and outputs the pixel-wise semantic segmentation as scene \nunderstanding and vehicle control commands concurrently. The end-to-end deep \nlearning-based autonomous driving model is tested in high-fidelity simulated \nurban driving conditions and compared with the benchmark of CoRL2017 and \nNoCrash. The testing results show that the proposed approach is of better \nperformance and generalization ability, achieving a 100% success rate in static \nnavigation tasks in both training and unobserved situations, as well as better \nsuccess rates in other tasks than the prior models. A further ablation study \nshows that the model with the removal of multimodal sensor fusion or scene \nunderstanding pales in the new environment because of the false perception. The \nresults verify that the performance of our model is improved by the synergy of \nmultimodal sensor fusion with scene understanding subtask, demonstrating the \nfeasibility and effectiveness of the developed deep neural network with \nmultimodal sensor fusion.