A Multi-Modal Approach to Rail Surface Condition Analysis: The MFDF-Net

Surface defects on the track may lead to accidents and pose a serious threat to the safety of passengers and staff. Regarding current detection methods, there are still shortcomings: manual detection is time-consuming and labor-intensive; vibration acceleration and machine vision methods rely solely on a single sensor data type, making them vulnerable to external interference, including coupled vibrations and oil contamination, leading to errors. The article proposes a new model for identifying the surface condition of steel rails using multimodal data of vibration and images: the multi-scale feature dynamic fusion network (MFDF-Net). Features are categorized into three scales: shallow, intermediate, and deep. An attention mechanism is incorporated to emphasize features of different scales, enabling dynamic weighting during the fusion of vibration and multi-scale features. Additionally, attention is given to fundamental features such as frequency and edge characteristics, as well as abstract features like the correlation between time series and image structure. MFDF-Net utilizes data collected by service vehicles to demonstrate that, in evaluating the model’s stability, the average accuracy of the test set across multiple training sessions reaches 99.333%. This represents a 2.67% improvement over the test accuracy of a single vibration mode and a 7.8% enhancement over that of a single image mode. These results indicate that the fusion network effectively discerns rail surface states, with multi-modal fusion offering higher precision and improved robustness compared to single-mode rail surface state analysis. This network holds promise for advancing intelligent diagnosis of rail surface defects.