Illumination-Aware Multimodal Hierarchical Fusion Network for RGB-Infrared Object Detection

RGB-infrared (RGB-IR) object detection has attracted significant attention in drone-based applications due to its robustness under all-weather conditions. How to effectively fuse the complementary information in both modalities is one key for accurate object detection. However, the performance is limited by the inherent differences between modalities and the varying illumination conditions across different weather scenarios. Focused on this issue, we propose an illumination-aware multimodal hierarchical fusion network (IMHFNet) for RGB-IR object detection. First, an illumination aware module (IAM) is designed to extract local illumination features from RGB image, which is used to guide the subsequent multimodal feature fusion process. Then, considering the differences in semantic expression and detail representation of different feature layers of multimodal data, we separately design shallow and deep feature fusion strategies. In specific, the shallow feature fusion module is constructed based on convolutional operators and illumination-guided adaptive weight fusion, focusing on capturing and enhancing local detail information. For the deep feature fusion, illumination feature is incorporated as an auxiliary information, to guide the global semantic information integration across different modalities via adopting a transformer structure. In this work, we also construct a new drone-based RGB-IR dataset, named by DroneShip. It contains 4,306 images annotated with 17,054 oriented ship object instances, which covers a wide range of natural illumination conditions from daytime to nighttime. Finally, to validate the effectiveness of the proposed method, we evaluate the IMHFNet on the constructed DroneShip and two publicly available RGB-IR datasets (KAIST and DroneVehicle), which respectively focus on ship, pedestrian and vehicle targets. Experimental results on all three datasets consistently demonstrate the effectiveness and robustness of IMHFNet across diverse scenarios and illumination conditions. The source code of the proposed method will be made publicly available at https://github.com/luting-hnu/DroneShip.