A Multimodal Spatiotemporal Fusion Network With Physical-Data-Driven Learning for Additive Manufacturing Fault Diagnosis

In additive manufacturing process, the fusion of multi-source sensing signals and cross-modal interaction are crucial to improve the accuracy of fault diagnosis. Most of the existing methods focus mainly on signal time-series features, but the spatial and time-frequency features were always ignored. In this study, a multi-modal spatio-temporal fusion network (MSTNet) with a three-branch synergetic architecture is proposed. The network integrates acoustic emission (AE) and thermal modalities to achieve fault diagnosis in additive manufacturing. Firstly, a channel-enhanced Mamba (CEMamba) architecture is designed to achieve feature extraction of multi-channel frequency-space joint signals generated by continuous wavelet transform (CWT), gramian angular difference fields (GADF), and recurrence plot. The CEMamba architecture can effectively capture the multi-channel features of AE signals including spatial, time-series, and time-frequency features. Secondly, the bi-directional gated recurrent unit is introduced to capture the dynamic evolution laws of AE signals. Specially, a novel physics-based loss function is designed to constrain the physics informed neural network (PINN) for generating high-resolution thermal modalities. Finally, an adaptive and dynamic weighting mechanism during the feature fusion enables the information complementarity of cross-modal features. When tested on two manufacturing datasets of selective laser melting and laser directed energy deposition, the fault diagnosis accuracy of MSTNet achieves 99.71% and 99.55%, respectively. The scalability of MSTNet is further verified on two public datasets of CWRU and Jiangnan University bearing.