Crop yield prediction from multi-spectral, multi-temporal remotely sensed imagery using recurrent 3D convolutional neural networks

Crop yield prediction has played a vital role in maintaining food security and has been extensively investigated in recent decades. Most research has focused on excavating fixed spectral information from remote sensing images. However, the growth of crops is a highly complex trait determined by diverse features. To maximally explore these heterogeneous features, we aim to simultaneously exploit spatial, spectral, and temporal information from multi-spectral and multi-temporal remotely sensed imagery. Therefore, in this paper, we propose a novel deep learning architecture for crop yield prediction, namely, SSTNN (Spatial-Spectral-Temporal Neural Network), which combines 3D convolutional and recurrent neural networks to exploit their complementarity. Specifically, the SSTNN incorporates a spatial-spectral learning module and a temporal dependency capturing module into a unified convolutional network to recognize the joint spatial-spectral-temporal representation. The novel spatial-spectral feature learning module first exploits sufficient spatial-spectral features from the multi-spectral images. Then, the temporal dependency capturing module is concatenated on top of the spatial-spectral feature learning module to mine the temporal relationship from the long time-series images. Furthermore, we introduce a new loss function that eliminates the influence of an imbalanced distribution of crop yield labels. Finally, the proposed SSTNN is validated on winter wheat and corn yield predictions from China. The results are compared with widely used machine learning methods as well as state-of-art deep learning methods. The experimental results demonstrate that the proposed method can provide better prediction performance than the competitive methods.