Residual Fuzzy Alignment on Hypergraph for Open-Set 3D Cross-Modal Retrieval

Existing 3D cross-modal retrieval (3CMR) methods heavily rely on prior knowledge of training categories, which leads to the problem of modality shift and unseen center deviation when encountering unseen categories under the open-set environment. Aiming at the open-set 3CMR, this paper introduces the Hypergraph-Based Residual Fuzzy Alignment (ReFA) framework, which revisits the open-set retrieval task and navigates uncertainty of it through the lens of Fuzzy Theory. Facing the challenges of boundaryless space caused by uncertain unseen categories, we explore the representation and measurement in the fuzzy membership space as an alternative to fixed close-set category space. Specifically, to address the problem of modality shift caused by unseen categories, we utilize the Residual Sampling Generation (RSG) module to generate modality sampling embeddings that are independent of seen categories under the guidance of fuzzy representation, which residually decouples the entangled interactions of seen categories and modalities. To overcome the problem of unseen center deviation, we propose the Center Fuzzy Alignment (CFA) module to leverage the high-order fuzzy correlations for generalized metric, by constructing a fuzzy hypergraph based on the inherent and fuzzy correlations among both modalities and categories. The comprehensive evaluations of comparison and ablation studies on the four benchmarks demonstrate the superiority of our proposed framework compared to state-of-the-art methods.