DIDPT: Dense Interaction Deep Prompt RGBT Tracking

Existing RGB-infrared object tracking methods struggle with effectively fusing data from both modalities, further hindered by the magnitude disparity between them. While large-parameter models in RGB tracking demonstrate robustness on extensive datasets, their performance remains underutilized when incorporating infrared data. To address these challenges, this paper proposes a deep prompt learning method based on dense interaction to enhance RGB-infrared fusion and leverage the strengths of large models in RGBT object tracking. We treat infrared information as a prompt for the tracker and freeze the pre-trained parameters of the RGB backbone model. During the initial feature extraction phase of backbone model, dense infrared prompt interaction encoder is employed to integrate infrared information. Subsequently, we introduce learnable prompts in the Transformer module while freezing the parameters of the Transformer encoder layers, updating only the parameters of the learnable prompts and the fully connected operation layer to enhance the model’s capacity to learn information after expanding to an additional modality. This approach requires updating only 2.8% of the parameters in the model during training, thereby saving computational resources. Extensive experiments conducted on widely tested datasets RGBT234 and LasHeR demonstrate the effectiveness of proposed method. Overall, our approach better integrates RGB and infrared images and introduces prompt learning to address the issue of magnitude imbalance in the data, providing a promising solution to the challenges in RGBT object tracking.