Multi-vision-based Localization and Pose Estimation of Occluded Apple Fruits for Harvesting Robots

Locating and grasping occluded fruits are crucial and challenging tasks in robotic harvest, suffering from insufficient filling rate and noise of stereo cameras. In this paper, we propose a multi-vision-based method to locate and estimate a grasping pose for an occluded fruit, based on a deep learning multi-task network and a new frustum-based method of point-cloud-processing. The multi-task deep learning network is presented to detect the complete bounding box and segment the visible part of occluded targets. Once the detection and segmentation are finished, we introduce the concept of 3D frustum for ease of estimating the centroid of the visible part and then reconstruct the shape of the occluded fruits. Accordingly, the estimation of the grasping approach is derived. Combining the measurements from multiple stereo visions, we obtained a new centroid of the fruit and an grasping pose of the occluded fruit. To demonstrate the effectiveness of the proposed method, the experiments in orchards were performed. All shown results supported the research claims.