Wheat biomass, yield, and straw-grain ratio estimation from multi-temporal UAV-based RGB and multispectral images

Timely and accurate crop monitoring and harvest property prescription map making before harvesting is valuable for the precision operation of combine harvesters. The development of Unmanned Aerial Vehicles (UAV) and sensor technology makes it feasible to acquire crop remote sensing images with high temporal and spatial resolution. The aim of this study is to evaluate the efficacy of UAV-based time series remote sensing data and multimodal data fusion using RGB and multispectral sensors in estimating wheat yield, biomass, and straw-grain ratio in the ripening stage. Wheat RGB and multispectral images of nine major growth stages were collected using a UAV (DJI P4M, Phantom4 Multispectral). To extract data's temporal and spatial features, a new deep neural network, CNN-LSTM, was proposed. Multimodal information, including wheat canopy colour, spectral, and space structure features, was extracted and integrated to predict wheat harvest properties in the model, and the three types of feature predictive power were compared. The highest precision rates (0.553, 0.527) were obtained by CNN-LSTM on yield and biomass, when the three types of features of nine stage were inputs. The prediction accuracy of the classical convolutional neural network (CNN) (0.367, 0.387) and long short term memory (LSTM) (0.467, 0.46) used for comparison was worse than that of the proposed model. Furthermore, the increasing rate in yield prediction at each period indicates that heading is the most important stage for wheat yield estimation. The study proves that the proposed approach can provide a relatively precise estimations of crop harvest properties.