FNI-DETR: real-time DETR with far and near feature interaction for small object detection

Abstract In recent years, real-time object detectors have gained significant traction in domains such as autonomous driving, industrial inspection, and remote sensing. The Detection Transformer has emerged as a research focal point due to its end-to-end architecture that eliminates the need for post-processing. However, due to the Transformer’s tendency to focus on global information, small objects are often overlooked. To address this limitation, we propose FNI-DETR, a real-time Detection Transformer tailored for small object detection by incorporating Far and Near Feature Interaction. Specifically, FNI-DETR integrates state space models with the Transformer to form a Mamba-Encoder block, enabling the interaction of feature information across different spatial scales. This enhances the representation and learning of near-end information while improving the extraction of semantic information. Additionally, we introduce a Lightweight Spatial Attention block in the backbone stage to capture detailed information in regions of interest. Furthermore, the ADOWN block is employed for downsampling, reducing the likelihood of discarding small objects from the feature map and increasing the model’s focus on small objects. Experimental results show that FNI-DETR achieves an average precision(mAP50:95) of 49.5% on the COCO val2017 dataset, which is 4.2% higher than the Real-Time Detection Transformer (RT-DETR) and 1.7% higher than the YOLOv10-L network. The detection results for small targets also reach 31.7% APs. Moreover, our network achieves a real-time detection speed of 116 FPS on the COCO dataset. On the VisDrone 2019 test dataset, FNI-DETR’s mAP50 and mAP50:95 achieved 37.4% and 21.7%, reaching the SOTA detection level. Our code is made available at https://github.com/hzx-123-wq/FNI-DETR/tree/master/FNI-DETR .