Weed resistance assessment through airborne multimodal data fusion and deep learning: A novel approach towards sustainable agriculture

Herbicide-resistant weeds represent a significant challenge to modern agriculture. The need for innovative and sustainable weed management strategies has become increasingly pressing as the threat of herbicide-resistant weeds continues to escalate. Unmanned aerial vehicles (UAVs) and various sensors have become indispensable tools in plant phenotyping studies. In this study, a comprehensive resistance score (CRS) was proposed to effectively quantify weed resistance in the field. Multimodal data fusion and deep learning were utilized to perform regression of CRS, three different fusion methods for 3D-CNN and 2D-CNN to extract and fuse multimodal information collected by UAVs including spectral, structural, and texture information for weed resistance. Our findings demonstrate that (1) discernible differences in spectral response exist between susceptible and resistant weeds, with the optimal band for the Successive Projections Algorithm (SPA) selection coinciding with the optimal band for resistance expression band; (2) resistance assessment accuracy is enhanced through multimodal data fusion, with the late deep fusion network exhibiting the best accuracy, R2 of 0.777 and RMSE of 0.547; (3) the multimodal fusion network model displays robust adaptability in resistance assessment across varying densities and effectively generates weed resistance map. Overall, this research demonstrates the effectiveness of using multimodal data fusion and CRS, combined with deep learning for achieving accurate and reliable weed resistance assessment in agricultural fields.