Self-Supervised Learning for Multimodal Fault Diagnosis With Shapley-Value Weighted Transformers

In recent years, there has been extensive exploration of intelligent fault diagnosis methodologies. Nevertheless, there has been a paucity of prior research addressing the integration of multi-sensor and multi-modal information of fault features. To address these challenges, a novel multi-modal cross-sensor Transformer (MCT) incorporating self-supervised learning is proposed for fault diagnosis. Specifically, during the training phase, first, a channel-based embedding method leveraging self-attention mechanisms is proposed to transform data collected from different sensors, each treated as a distinct channel, into variable tokens, significantly boosting the Transformer’s ability to generalize across diverse sensor fault diagnosis tasks. Subsequently, a novel self-supervised multi-modal learning approach is devised to investigate the correlation and divergences between time-domain and frequency-domain features, thereby augmenting feature recognition capabilities. Finally, an uncertainty-based multi-modal loss weighting mechanism is proposed to dynamically adjust the weights of losses according to their uncertainty, thus concentrating the learning process on the most informative features, which enhances diagnostic accuracy and model robustness. In the model deployment phase, Shapley values are utilized to allocate influence among different modalities, ensuring that key modalities dominate the model predictions. Experimental evaluations are carried out on a publicly available bearing dataset and a dataset collected from a chemical factory, demonstrating that the MCT enhances the F1 score by 6.43% to 10.05% and the accuracy by 5.23% to 10% in comparison with state-of-the-art methods.