Three-dimensional pose detection method based on keypoints detection network for tomato bunch

Non-destructive picking of fresh tomatoes is a delicate agronomical operation, based on comprehensive information about the plant organ, such as the location of stem, peduncle, and fruits. The matching between visual information supply and information demand from the agronomical technic is the key power to promote the picking robot from the laboratory to the field. The three-dimensional pose information, containing the location of each organ of the plant, can meet the demand of agronomical technic. It is the premise of precisely handling the cluster of fruits. In order to realize the fine tomato bunch harvesting operation in a bunch, this paper proposed a three-dimensional pose detection method for tomato bunch. The method, named Tomato Pose Method (TPM), is composed of a priori geometric model, a cascaded multi-task network, and a three-dimensional reconstruction process. Based on prior knowledge and agronomic technology, this prior geometric model comprehensively and flexibly describes the spatial location information of tomato bunch. The cascaded multi-task network is designed based on hourglass structure and transfer learning, which is suitable for bounding box and key point prediction of tomato bunches in complex environments. Finally, combining the prior geometric model and the spatial position information of each key point, the tomato bunch is reconstructed. Only a medium training dataset, containing 1800 RGBD images covering changing lighting, occlusion, and various poses, is needed for training. Its success rate of TPM on two-dimensional keypoint detection is 94.02%, the accuracy of 85.77% predicted points are at medium level. And 70.05% tomato bunch with multi-pose can be constructed. More importantly, this method only needs one RGBD image taken by a commercial camera to realize the three-dimensional reconstruction of a single-bunch scenario in 1.0 s, and a multi-bunch scenario in 2.0 s. It provides comprehensive information, and provides data basis for target positioning and path planning of picking robot, which makes the non-destructive harvesting possible.