DefectSAM: Prototype Prompt Guided SAM for Few-Shot Defect Segmentation

Industrial defect segmentation plays a crucial role in ensuring quality and safety within manufacturing processes. Previous research in this field has focused on training models with limited available data to enable the segmentation of new classes. However, the limited and singular training data lead to limited feature extraction capabilities of the models, hindering their generalization to other industrial scenarios. Recently, the segment anything model (SAM), an interactive visual segmentation foundation model, demonstrates remarkable zero-shot generalization capabilities. However, its performance in the specific domain of industrial defect detection is relatively poor. Therefore, we propose an approach that leverages visual foundation model to achieve few-shot defect segmentation. This method extracts prior knowledge from example images, derives multitype robust prompts from the prior knowledge, and fine-tunes SAM with few-shot learning paradigm and low-rank adaptation (LoRA) strategy. Specifically, given a few support image-mask pairs, foreground and background prototypes are extracted, and their similarities with the query image are computed to obtain prototype prior probability maps. Subsequently, a prototype prompt generator specifically for defect data extracts salient information from these prior probability maps, producing robust prototype prompts. These prototype prompts are then used to fine-tune the LoRA layers in SAM’s image encoder and the lightweight mask decoder of SAM, enabling SAM to adapt to the few-shot defect segmentation task. Experimental results demonstrate that our proposed method outperforms state-of-the-art few-shot defect segmentation techniques and can be generalized to various industrial scenarios, exhibiting significant application value.