Object localization methodology in occluded agricultural environments through deep learning and active sensing

The predominance of branch and leaf shade in agricultural environments presents a barrier for accurate target recognition. Particularly for picking robots, precise localization of the picking object is essential. For this purpose, this paper proposes detection and localization methods based on deep learning and active sensing for harvesting robots in real-world environments with occlusion and varying lighting conditions. Using a deep learning network, the detection method firstly extracts the peduncle and fruit regions; the fruit region is then used to calculate the occlusion rate and the offset distance of the peduncle relative to the fruit. With such information, the robot arm adjusts the camera's field of view to perform multiple recognitions until the confidence is satisfied. Furthermore, to solve the picking problem caused by the peduncle's random tilting, this paper proposes a method to calculate the peduncle's tilt angle for controlling the end-effector to make the corresponding angle rotation. The robot arm and its end-effector are directed to complete the harvesting with the picking point location and tilt angle. In this study, data collection, detection and picking tests were implemented in the field, the results indicated that the method obtained an average successful picking rate of 90% after 300 trials, the error between the estimated occlusion ratio and the genuine value is 16% in average, and the active sensing method has improved the confidence score in occluded situations by over 50%. The proposed active methods have a 33% increase in precision and a 43% increase in efficiency compared to constant methods.