Hybrid multimodal fusion with deep learning for rolling bearing fault diagnosis

For vibration signal of rolling bearing with long time series obtained from multiple sampling points, hybrid multimodal fusion with deep learning is proposed for fault diagnosis. Feature-level multimodal fusion method is used to extract time domain features from vibration signal samples of the whole life cycle. Moreover, those features are transformed into multimodal samples, which are composed of grayscale images and time series. Convolutional neural network (CNN), which is commonly applied in image processing, is used to deal with grayscale images, while deep belief network (DBN) is utilized to train time series samples. Subsequently, decision-level multimodal fusion can be achieved by combining several different deep learning models, so as to obtain comprehensive fault prediction result. The effectiveness of the proposed method is verified by rolling bearing datasets with multiple typical faults. Compared with individual deep learning models and other traditional models, the proposed method can achieve higher fault diagnosis accuracy.