Adaptive fusion of multi-modal remote sensing data for optimal sub-field crop yield prediction

Accurate crop yield prediction is of utmost importance for informed decision-making in agriculture, aiding farmers, industry stakeholders, and policymakers in optimizing agricultural practices. However, this task is complex and depends on multiple factors, such as environmental conditions, soil properties, and management practices. Leveraging Remote Sensing (RS) technologies, multi-modal data from diverse global data sources can be collected to enhance predictive model accuracy. However, combining heterogeneous RS data poses a fusion challenge, like identifying the specific contribution of each modality in the predictive task. In this paper, we present a novel multi-modal learning approach to predict crop yield for different crops (soybean, wheat, rapeseed) and regions (Argentina, Uruguay, and Germany). Our multi-modal input data includes multi-spectral optical images from Sentinel-2 satellites and weather data as dynamic features during the crop growing season, complemented by static features like soil properties and topographic information. To effectively fuse the multi-modal data, we introduce a Multi-modal Gated Fusion (MMGF) model, comprising dedicated modality-encoders and a Gated Unit (GU) module. The modality-encoders handle the heterogeneity of data sources with varying temporal resolutions by learning a modality-specific representation. These representations are adaptively fused via a weighted sum. The fusion weights are computed for each sample by the GU using a concatenation of the multi-modal representations. The MMGF model is trained at sub-field level with 10 m resolution pixels. Our evaluations show that the MMGF outperforms conventional models on the same task, achieving the best results by incorporating all the data sources, unlike the usual fusion results in the literature. For Argentina, the MMGF model achieves an R 2 value of 0.68 at sub-field yield prediction, while at the field level evaluation (comparing field averages), it reaches around 0.80 across different countries. The GU module learned different weights based on the country and crop-type, aligning with the variable significance of each data source to the prediction task. This novel method has proven its effectiveness in enhancing the accuracy of the challenging sub-field crop yield prediction. Our investigation indicates that the gated fusion approach promises a significant advancement in the field of agriculture and precision farming. • Multi-modal model that adaptively fuses four data sources to predict crop yield. • Prediction at 10 m spatial resolution over multiple crops, regions, and years. • Gated fusion weights allow a simple interpretation of modalities contribution. • Optimal predictions are consistently obtained when all modalities are used.