Deep Learning Predicts Rapid Over-softening and Shelf Life in Persimmon Fruits

In contrast to the progress in the research on physiological disorders relating to shelf life in fruit crops, it has been difficult to non-destructively predict their occurrence. Recent high-tech instruments have gradually enabled non-destructive predictions for various disorders in some crops, while there are still issues in terms of efficiency and costs. Here, we propose application of a deep neural network (or simply deep learning) to simple RGB images to predict a severe fruit disorder in persimmon, rapid over-softening. With 1,080 RGB images of 'Soshu' persimmon fruits, three convolutional neural networks (CNN) were examined to predict rapid over-softened fruits with a binary classification and the date to fruit softening. All of the examined CNN models worked successfully for binary classification of the rapid over-softened fruits and the controls with > 80% accuracy using multiple criteria. Furthermore, the prediction values (or confidence) in the binary classification were correlated to the date to fruit softening. Although the features for classification by deep learning have been thought to be in a black box by conventional standards, recent feature visualization methods (or "explainable" deep learning) has allowed identification of the relevant regions in the original images. We applied Grad-CAM, Guided backpropagation, and layer-wise relevance propagation (LRP), to find early symptoms for CNNs classification of rapid over-softened fruits. The focus on the relevant regions tended to be on color unevenness on the surface of the fruit, especially in the peripheral regions. These results suggest that deep learning frameworks could potentially provide new insights into early physiological symptoms of which researchers are unaware.