A multimodal domain generalization bearing fault diagnosis method based on acoustic–vibration fusion

Abstract Extensive research on cross-domain bearing fault diagnosis has predominantly focused on single-modal scenarios, particularly under unknown operating conditions. However, with the increasing heterogeneity of data acquired from mechanical systems, multimodal domain generalization (DG) has garnered growing attention. To address this challenge, this paper proposes a multimodal DG framework based on acoustic–vibration fusion. In the proposed method, the spectral representations of vibration and acoustic signals are first utilized as model inputs. Two parallel one-dimensional transformer-based subnetworks are then designed to extract discriminative and domain-invariant features from each modality. To effectively model inter-modal relationships, a dual-branch cross-attention mechanism is introduced for feature fusion. Furthermore, a hyperbolic prototype triplet loss is proposed to regulate the geometric relationships between sample embeddings and class prototypes in hyperbolic space, thereby enhancing feature discriminability and facilitating class-level domain alignment. In addition, a conditional domain adversarial strategy is incorporated to further encourage domain-invariant feature learning and improve generalization to unseen domains. Finally, extensive DG experiments are conducted on a multi-source heterogeneous bearing dataset. Experimental results demonstrate that the proposed framework achieves more effective multimodal feature integration and delivers superior diagnostic accuracy compared with existing methods.