Improving Autonomous Vehicle Visual Perception by Fusing Human Gaze and Machine Vision

With high-definition sensors and sophisticated machine vision algorithms, the visual perception capability of autonomous vehicle (AV) has largely advanced. However, the visual perception performance of AVs may still be unstable in complex traffic environment. To improve the robustness and capability of risk detection of AV visual perception system, this work proposes a framework to fuse human gaze and the object detection results from vehicle vision based on the Laplacian Pyramid algorithm. We evaluate the proposed method on a level-2 AV to perceive the interactive vehicles at unsignalized intersections. Using Extended Kalman Filter, the trajectory of the human drivers' gaze and the anchor boxes from AV object detection are fused. Results reveal that with human-vehicle visual fusion, the actual trajectory of interactive vehicles can be predicted more accurately than separately using human gaze or object detection algorithm. The findings show that human-vehicle visual fusion improves the perception accuracy and robustness of interactive objects in complex traffic environment. The method has the potential to enhance the attention mechanism of AV vision.