Pixel‐wise anomaly detection on road by encoder–decoder semantic segmentation framework with driving vigilance

Abstract Semantic segmentation struggles with detecting undefined road obstacles, critical for autonomous driving in urban environments. This study addresses the need for accurate unknown obstacle detection, inspired by drivers’ instinctual vigilance toward unexpected objects. It explores the impact of unexpected object position patterns on anomaly detection using human fixation scan‐paths and gaze density heat maps. Data augmentation based on these patterns enhances the outlier dataset for anomaly detection networks. The proposed driving vigilance enhancement framework (DVEF) improves classification accuracy with multi‐scale detailed features and a vigilance enhancement model, generating vigilance score maps to prioritize unknown regions. An improved energy model joint loss function expands vigilance scores, enhancing anomaly detection accuracy. Compared with recent methods on Fishyscapes (FS) LostAndFound, FS Static, and average datasets, average precision improvements of 2.16%, 2.22%, and 2.89% are achieved on these datasets, respectively. In addition, the false positive rate at a true positive rate of 95% are decreased to 5.79%, 5.62%, and 17.89%, respectively. It is indicated that the performance of the encoder–decoder semantic segmentation network is improved by DVEF, with enhanced consistency and robustness.