Fruit Quantity and Ripeness Estimation Using a Robotic Vision System

Accurate localization of crop remains highly challenging in unstructured environments, such as farms. Many developed systems still rely on the use of hand selected features for crop identification and often neglect the estimation of crop quantity and ripeness, which is a key to assigning labor during farming processes. To alleviate these limitations, we present a robotic vision system that can accurately estimate the quantity and ripeness of sweet pepper (Capsicum annuum L), a key horticultural crop. This system consists of three parts: detection, ripeness estimation, and tracking. Efficient detection is achieved using the FasterRCNN (FRCNN) framework. Ripeness is then estimated in the same framework by learning a parallel layer which we experimentally show results in superior performance than treating ripeness as extra classes in the traditional FRCNN framework. Evaluation of these two techniques outlines the improved performance of the parallel layer, where we achieve an F ₁ score of 77.3 for the parallel technique yet only 72.5 for the best scoring (red) of the multiclass implementation. To track the crop, we present a vision only tracking via detection approach, which uses the FRCNN with parallel layers as input. Being a vision only solution, this approach is cheap to implement as it only requires a camera and in experiments, using field data, we show that our proposed system can accurately estimate the number of sweet pepper present, to within 4.1% of the visual ground truth.