Causality-Inspired Graph Neural Networks for Cross-Modal Retrieval

Multimedia data have rich semantic knowledge, and cross-modal retrieval (CMR) methods are able to explore their correlations. Graph neural networks (GNN) can represent complex connection information, so some CMR methods apply GNNs as semantic comprehender to improve matching accuracy. However, fine-grained classifiers can accurately obtain object-centric semantics, but these semantics may be conflicting, potentially leading to inexplicability responses that are difficult to ground, for example. Meanwhile, it may be concerned that the credibility of GNN, mainly includes sensitivity to out-of-distribution changes and lack of interpretability. Therefore, we attempt to integrate causal learning into GNNs and capture potential causal relationships rather than surface object-centric classification. Firstly, we analyze semantic causality and build cross-modal structure causal model, then achieve cross-modal interventional-causal learning by causality-inspired graph neural network (CIGNN). Secondly, we propose modality contrastive learning to characterize the intra-modal and inter-modal correlations, and project into the common representation space. Thirdly, a new soft rank loss method is designed beyond binary similarity to achieve fine-grained similarity sorting. Comprehensive experiments on three widely used benchmark datasets prove the superiority of our proposed method, while ablation experiments demonstrated the effectiveness of each component.