CLIP-Guided Cross-Modal Feature Fusion based Few-Shot Learning for Nighttime Pavement Defect Detection

Pavement defect detection has drawn significant attention in computer vision research due to its indispensable role in infrastructure safety. Although deep learning has achieved huge success in daytime scenarios with abundant labeled data, its performance severely degrades under nighttime conditions due to insufficient defect samples and complex low-light noise. While few-shot learning alleviates data scarcity problem, existing approaches rely solely on visual features, ignoring the potential of cross-modal semantics to disambiguate defects from nighttime noise. Inspired by the success of vision-language models (VLMs) like Contrastive Language-Image Pre-Training (CLIP), this paper attempts to leverage text semantics to enhance robustness and proposes: (1) a Dual-Modal Hybrid Feature Extraction module which fully utilize ResNet local features, CLIP’s noise-robust visual features and text features from class-specific prompts, and (2) a Semantic-Guided Proposal Filter module that filter RPN proposals with low CLIP similarity to target-class text. Experiment results demonstrate that the proposed method outperforms baseline method, achieving a mean Average Precision (mAP) of 45.1% which surpasses vision-only baseline by 29.9% under 10-shot setting.