SAM based Region-Word Clustering and Inference Score Adjusting for Open-Vocabulary Object Detection

Open-vocabulary object detection aims to detect and recognize novel categories that are not seen in the training set.Most existing methods rely on the Region Proposal Network (RPN) to extract regions of interest and align them with textual descriptions through one-to-one region-word alignment, such as bipartite matching.However, these methods encounter three major challenges: 1) Insufficient novel proposals: RPN tends to generate high-confidence proposals for base categories, but low-confidence ones for novel categories. 2) Missing matches for duplicate instances: Bipartite matching struggles to handle multiple instances of the same category in an image. 3) Inference bias: During inference, classifiers are often biased toward seen categories with lower prediction scores for novel categories. To address these challenges, we propose a SAM based region-word clustering and inference score adjusting model for open-vocabulary object detection (coined CADet). Specifically, our method consists of three components: 1) Enhanced proposal generation: To ensure sufficient proposals for novel categories, we incorporate an unsupervised localization SAM, which generates more comprehensive proposals covering both base and novel categories. 2) Region-word clustering: To mine more matching samples, we cluster similar proposals derived from bipartite matching and assign them the same pseudo-labels. 3) Score adjusting: We introduce a similarity-guided score adjustment strategy to effectively mitigate classifier bias against novel categories during inference. Extensive experiments on two datasets demonstrate the superior performance of our approach, achieving 36.4% mAP on COCO and 29.6% mask mAP on LVIS, outperforming existing methods on novel categories.