BCE YOLOv8: A novel YOLO model for brain tumor instance segmentation

Abstract Precision instance segmentation of brain tumor is a crucial for realizing intelligent healthcare and alleviating the strain of physicians. Aim at the challenges of low segmentation precision and missed detections that occurred during the segmentation process, this paper proposes a brain tumor intelligent segmentation method that is founded on Bidirectional Feature Pyramid Network (Bi-FPN)-Coordinate Attention (CA)- Efficient-IoU (EIoU) YOLOv8 (BCE YOLOv8). In the feature extraction stage, the CA is incorporated into the C2f module. The CA possesses the characteristic of automatically learning the weights among different channels, enabling the feature extraction process to prioritize significant tumor characteristics, thereby enhancing efficacy. To improve the precision of architecture, adopt weighted Bi-FPN to complete more sophisticated feature fusion by bidirectional feature paths. Furthermore, the EIoU loss function, which effectively measures the difference between the ground truth box and the anchor box, is incorporated into the architecture to accelerate convergence. Finally, the Bi-FPN-CA-EIoU YOLOv8 is proposed. The experimental outcomes indicate that the precision of BCE YOLOv8 is 67.8%, and the values of mAP@0.5 and mAP@0.5:0.95 are 64.6% and 50.6%. In comparison to the YOLOv8, the precision of BCE YOLOv8 has improved by 21.07%, and the mAP@0.5 has increased by 2.54%. There is also a significant improvement compared to the lasted YOLOv11, with an increase of 16.49%. BCE YOLOv8 enhances the precision of brain tumor instance segmentation and mitigates the situation of missed detections, delivering optimal overall performance and offering technical assistance for intelligent brain tumor detection.