Chili-Seg: a high-precision detection and instance segmentation model for chili flowers in complex environments

Accurate recognition and instance segmentation of chili flowers are fundamental to the development of intelligent breeding robots and the improvement of operational efficiency in protected agriculture. However, in unstructured field environments, chili flowers are typically small-scale targets and are frequently affected by occlusion from leaves and branches as well as illumination variations, leading to degraded feature representation and suboptimal segmentation performance in existing models. To address these challenges, this study proposes a high-precision detection and segmentation framework, termed Chili-Seg, built upon the YOLO11n-seg baseline architecture (YOLO: You Only Look Once). To enhance detection and segmentation performance, an Efficient Head is introduced to replace the original decoupled head. By optimizing the parameter-sharing mechanism between classification and regression branches, the proposed head improves both localization accuracy and segmentation quality while maintaining computational efficiency. To further strengthen feature extraction under complex field conditions, a C2f with Dynamic Position Bias (C2DPB) module is designed to replace the original C2PSA module based on Position-Sensitive Attention (PSA) in the backbone network. The dynamic position-bias mechanism enhances the representation of fine-grained morphological characteristics of chili flowers while effectively suppressing background interference, thereby reducing mis-segmentation caused by environmental noise. Moreover, a Generalized Dynamic Feature Pyramid Network (GDFPN) is introduced to reconstruct the neck architecture. By enabling dynamic cross-scale feature interaction and adaptive weighted fusion, the proposed design mitigates feature loss caused by large variations in object scale and significantly improves multiscale feature representation capability. Experimental results on a self-constructed chili flower dataset demonstrate that Chili-Seg achieves improved overall instance-segmentation performance compared with representative YOLO-based baselines, including YOLOv5n-seg and YOLO11nseg; semantic-segmentation models are evaluated separately using their corresponding pixel-level metrics. For detection tasks, the proposed method achieves a precision of 96.8%, mean Average Precision at an Intersection over Union (IoU) threshold of 0.50 ( mAP 50 ) of 90.6%, and mAP 50–95 , averaged over IoU thresholds from 0.50 to 0.95, of 77.9%, improving the baseline by 1.7, 1.2, and 1.8 percentage points, respectively. For segmentation tasks, it achieves a precision of 96.6%, mAP 50 of 89.8%, and mAP 50–95 of 64.7%, with improvements of 2.1, 1.1, and 1.3 percentage points over the original model. Furthermore, the proposed model is deployed on the NVIDIA Jetson AGX Orin platform for real-time evaluation, achieving a processing speed of 102.4 frames per second (FPS). The results indicate that Chili-Seg achieves a favorable trade-off between accuracy, segmentation quality, and real-time performance, providing a robust foundation for subsequent extraction of chili flower phenotypic parameters.