RMTP-YOLO: an improved dense pedestrian detection algorithm based on YOLOv8

We address the challenges in dense pedestrian detection, including insufficient accuracy, high miss rates, and poor detection of occluded and small objects, by proposing an improved algorithm, RMTP-YOLO, based on YOLOv8. First, to solve the problem of multi-scale object detection, receptive field attention is integrated into the convolution module (Conv module) to enhance the network’s feature extraction capabilities. Second, the C2f module in the backbone network is partially replaced with the MobileViTv3 module, which leverages both convolutional neural networks and vision Transformers to improve object detection by fusing local and global features. In addition, to improve the detection accuracy of distant small pedestrians, a TinyHead for very small object detection is added to the original detection head structure. The bounding box regression loss function, Powerful-IoUv2 (PIoUv2), is adopted, along with an object-size-adaptive penalty factor and a gradient adjustment function based on anchor box quality to further reduce false and missed detections. Extensive experiments on several challenging dense pedestrian datasets, including CrowdHuman, WiderPerson, Caltech Pedestrian, CityPersons, and TJU-Ped Traffic, demonstrate that RMTP-YOLO significantly outperforms YOLOv8n, with improvements of 3.9%, 2.6%, 3%, 6.7%, and 4.6% in mAP@0.5 and 3.8%, 2.2%, 1.8%, 4.5%, and 4% in mAP@0.5:0.95, respectively. Furthermore, cross-dataset testing reveals RMTP-YOLO’s robust generalization across different datasets, consistently achieving higher precision, recall, and mAP metrics compared with YOLOv8n, particularly in dense pedestrian scenes. Thus, the algorithm effectively enhances pedestrian detection performance in dense scenes.