MSDF-Mamba: Mutual-Spectrum Perception Deformable Fusion Mamba for Drone-Based Visible–Infrared Cross-Modality Vehicle Detection

To ensure all-day detection performance, unmanned aerial vehicles (UAVs) usually need both visible and infrared images for dual-modality fusion object detection. However, misalignment between the RGB-IR image pairs and complexity of fusion models constrain the fusion detection performance. Specifically, typical alignment methods choose only one modality as a reference modality, leading to excessive dependence on the chosen modality quality. Furthermore, current multimodal fusion detection methods still struggle to strike a balance between high accuracy and low computational complexity, thus making the deployment of these models on resource-constrained UAV platforms a challenge. In order to solve the above problems, this paper proposes a dual-modality UAV image target detection method named Mutual-Spectrum Perception Deformable Fusion Mamba (MSDF-Mamba). First, we designed a Mutual Spectral Deformable Alignment (MSDA) module. This module employs a bidirectional cross-attention mechanism to enable one modality to actively extract the semantic information of the other, generating fusion features rich in cross-modal context as shared references. These fusion features are then used to predict spatial offsets, with deformable convolutions achieving feature alignment. Based on the MSDA module, a Selective Scan Fusion (SSF) module is carefully designed to project the aligned features onto a unified hidden state space. With this method, we achieve full interaction and enhanced fusion of intermodal features with low computational complexity. Experiment results demonstrate that our method outperforms existing state-of-the-art cross-modality detection methods on the mAP metric, achieving a relative improvement of 3.1% compared to baseline models such as DMM, while still maintaining high computational efficiency.