RWKV-Inspired Multi-modal Relation Modeling for Vision-Language Tracking

Vision-language object tracking can provide more state representations for targets by introducing the language modality, achieving more robust tracking and localization. Therefore, designing multi-modal interactions to achieve feature alignment between vision and language has been one of the research hotspots. However, existing multi-modal interaction methods face two key issues: on the one hand, they lack effective exploration of modeling the relationship between the contextual information of language sequences and visual features; on the other hand, the introduction of modalities leads to increased computational time costs in multi-modal interactions, which severely affects the real-time performance of vision-language tracking algorithms. To address these challenges, we propose a vision-language tracking framework called RWKV-Inspired Multi-modal Relation Modeling for Vision-Language Tracking (RrmTrack). We introduce a novel modality interaction method specific to vision-language object tracking based on RWKV, providing customized interaction for different modalities in vision-language tracking and effectively reducing the computational time cost of cross-modal interaction. Specifically, this method uses a time mixing module to model the relationship between language information and image features, and a channel mixing module to facilitate information interaction between images. By combining parallelized training with a linear attention mechanism and efficient RNN inference, it enables accurate and fast target localization in vision-language tracking. Additionally, we propose a novel feature extraction structure that integrates Siamese and One-stream architectures. An information restoration module is designed to reduce the information interference introduced by the search image to the template image during interaction. RrmTrack achieves state-of-the-art results and speed on multiple vision-language object tracking benchmarks, including TNL2k, LaSOT, OTB-Lang, LaSOText, and MGIT.