CLIP4VI-ReID: Learning Modality-Shared Representations via CLIP Semantic Bridge for Visible-Infrared Person Re-Identification

The core of visible-infrared person re-identification (VI-ReID) lies in learning modality-shared representations across different modalities. Given the significant modality gap between visible and infrared images, VI-ReID still remains a challenge. Recent research tends to use pre-trained CLIP to extract modality-shared representations, but they fail to consider differences in physical characteristics of natural images and infrared images when extracting visual features of infrared images and generating their text semantics. This causes the learned infrared text semantics to struggle to delicately describe the images, thereby limiting the performance of VI-ReID. In addition, after obtaining the text semantics, they do not make any adaptive adjustments, resulting in an increased risk of mismatches. To address these issues, we propose a novel CLIP-driven modality-shared representation learning network named CLIP4VI-ReID for VI-ReID task, which consists of Text Semantic Generation (TSG), Infrared Feature Embedding (IFE), and High-level Semantic Alignment (HSA). Specifically, considering the huge gap in the physical characteristics between natural images and infrared images, the TSG is designed to generate text semantics only for visible images, thereby enabling preliminary visible-text modality alignment. Then, the IFE is proposed to rectify the feature embeddings of infrared images using the generated text semantics. This process injects id-related semantics into the shared image encoder, enhancing its adaptability to the infrared modality. Besides, with text serving as a bridge, it enables indirect visible-infrared modality alignment. Finally, the HSA is established to refine the high-level semantic alignment. This process ensures that the fine-tuned text semantics only contain id-related information, thereby achieving more accurate cross-modal alignment and enhancing the discriminability of the learned modal-shared representations. Extensive experimental results demonstrate that the proposed CLIP4VI-ReID achieves superior performance than other state-of-the-art methods on some widely used VI-ReID datasets. The code will be available at https://github.com/y0406/CLIP4VI-ReID.