Pavement Friction Evaluation Based on Vehicle Dynamics and Vision Data Using a Multi-Feature Fusion Network

The tire–road friction coefficient is a critical evaluation index of the service performance of roads: it governs the stopping distance, traction control, and stability of vehicles. Moreover, friction information is also needed in many function units of modern vehicles. This paper proposes a novel data-driven approach for inference of the maximum tire–road friction coefficient using a combination of vehicle dynamics signal and machine vision data. The approach is aimed at robust road condition perception that can provide frequent measurements over large areas across all weather conditions. Two different neural network architectures were adopted to extract in-depth features behind vehicle dynamics signals and road surface images. Features from these two types of data were then fused in two different levels, namely feature level and decision level, forming a multi-feature fusion neural network. The proposed network performs better than models based only on dynamic signals or vision data. The method proposed was applied to real data obtained from an electric car in a highway driving scenario. For classification of the maximum tire–road friction coefficient, the proposed network can yield F1-score increments of 0.09 and 0.18 from dynamics-based and vision-based sub-models, respectively. For the maximum tire–road friction coefficient value regression, the proposed model also achieves the highest R-square score of 0.71. Of these two types of data collected under highway driving scenarios, the vision data contribute more to the overall performance of the proposed model. Nevertheless, the dynamics data possess excellent potential in poor lighting conditions. With these fused features, the proposed multi-feature fusion network can not only improve the accuracy of maximum tire–road friction coefficient estimation but also is deemed workable for a broader range of scenarios.