Maritime Environment Perception Based on Deep Learning

Environment perception is an essential aspect of automated maritime vehicles, especially in high-traffic areas. In recent years, deep-learning-based object detection using LiDAR has been well developed in the automotive sector but has not yet seen a similar level of sophisticated development in maritime applications. In these applications, LiDAR detection should be fused with other maritime navigation systems such as the automatic identification system (AIS) to expand the detection range. To address this, we propose a novel deep-learning-based concept for maritime environment perception by using LiDAR as a primary sensor and AIS as an assisting information source. This approach consists of three functional modules: object detection, multi-object tracking, and static environment mapping. For object detection, we apply a convolutional neural network (CNN) to detect floating objects represented as oriented bounding boxes. To train the CNN, we propose a method that generates simulated labeled datasets. The detected objects from CNN are tracked with Kalman Filter banks. The remaining LiDAR data points are treated as static environments and represented by polygons. We evaluated the approach by using simulative and real-world datasets. In the simulation, the average precision of the CNN object detector reaches 60.8%, with a data processing rate of 40 Hz in GPU. Our real-world evaluations show that this approach can track 83% of the vessels in a crowded harbor, with the overall intersection over union reaching 64%. Our proposed approach represents the first application of CNN for LiDAR-based maritime environment perception, demonstrating its high potential for future online and real-world applications.