A mango picking vision algorithm on instance segmentation and key point detection from RGB images in an open orchard

The vision system of a fruit picking robot must perform two difficult tasks, the accurate pixel-wise instance segmentation of fruit and the correct detection of picking points. Due to the unsatisfactory performance of the vision system on these tasks, the picking performance of the robot is insufficient and its application is not prevalent. In this study, performing fruit instance segmentation and the localization of picking points, a vision algorithm using RGB images was designed in a deep learning framework for a potential visual system of mango picking robots. The algorithm not only performed these two tasks successfully, but also integrated them into an end-to-end network, in which parallel branches performed the two tasks simultaneously. The average precision (at IoU of 0.75) and the average recall of instance segmentation reached 0.947 and 0.929 respectively, and the best precision and recall of picking-point detection reached 0.984 and 0.908 respectively. In addition, the tasks face various illumination and background interference in outdoor orchards, along with complex problems in terms of occlusion, overlap and object scale. In this study, the performance of the vision system was analysed in detail on several datasets and subsets of the various complex conditions, and the major factors that affect the performance were discussed. The results demonstrated that this system was robust against various illuminations and complex backgrounds, and yielded satisfactory segmentation and picking-point detection performances for minor and medium occlusion or overlap, and for medium and large mangoes. The model visualization and the influence analysis of model training demonstrated the training process and modelling effect of the deep learning network.