Multi-network fusion algorithm with transfer learning for green cucumber segmentation and recognition under complex natural environment

• A multi-network model fusion strategy was proposed. • The segmentation and recognition of cucumber in multi-scene environment were explored. • Feature extraction process was analyzed by using model visualization. • The transfer learning can improve model recognition performance. Accurate segmentation and detection of cucumber in non-structural environment is of great significance for automatic robotic harvesting. However, the high similarity of cucumber color to that of branches and leaves and the complexity of the growing environment make it difficult for cucumbers to be correctly detected by harvesting robots. This paper presented an improved automatic method for segmentation and recognition of matured cucumbers that combines data processing, single-stage target recognition network (YOLO-v3 and SSD), U-Net semantic segmentation network and Transfer Learning. The algorithm adopted a multi-network model fusion strategy; an improved U-Net model was implemented for fine pixel-wise cucumber segmentation, and then YOLO-v3 or SSD model was applied for cucumber recognition based on the segmented images. In order to deeply analyze the segmentation process and recognition principle on cucumber images, several main feature layers in YOLO-v3 and U-Net network were respectively visualized to demonstrate the integrated feature maps learned and model interpretability. In parallel, to accelerate convergence and improve model detection robustness, Transfer Learning method was introduced to optimize network structure. All three network models combined with Transfer Learning showed good performance in cucumber image processing. For the two target recognition models, i.e., YOLO-v3 and SSD, the F1 scores were above 0.95, and the average accuracy (AP) was as high as 99%, but overall, the recognition performance of YOLO-v3 was still higher than that of the SSD model. Meanwhile, the mIOU value and average pixel accuracy of U-Net semantic segmentation model also reached 94.24% and 97.46%, respectively. Finally, YOLO-v3 was combined with the U-Net segmented image to further implement cucumber detection. Compared to the individual YOLO-v3 model, the hybrid U-Net + YOLO-v3 model obtained better prediction results with higher AP, F1 score and precision values, especially the AP value, which improved by 6%. The detection model combined with U-Net and YOLO- V3 network can enhance the shape, texture and other feature information of green cucumbers in complex agricultural environment, and improve the accurate positioning and grasping of the harvesting robot.