LGF$$^2$$: Local and Global Feature Fusion for Text-Guided Object Detection

A baby can successfully learn to identify objects in an image with the corresponding text description provided by their parents. This learning process leverages the multimodal information of both the image and text. However, classical object detection approaches only utilize the image modality to distinguish objects, neglecting the text modality. While many Vision-Language Models have been explored in the object detection task, they often require large amounts of pre-training data and can only specify a particular model structure. In this paper, we propose a lightweight and generic Local and Global Feature Fusion (LGF $$^2$$ ) framework for text-guided object detection to enhance the performance of image modality detection models. Our adaptive text-image fusion module is designed to learn optimal fusion rules between image and text features. Additionally, we introduce a word-level contrastive loss to guide the local-focused fusion of multimodal features and a sentence-level alignment loss to drive the global consistent fusion of multimodal features. Our paradigm is highly adaptable and can be easily embedded into existing image-based object detection models without any extra modification. We conduct extensive experiments on two multimodal detection datasets, and the results demonstrate that our LGF $$^2$$ significantly improves their performance.