Differentiable modeling for soil moisture retrieval by unifying deep neural networks and water cloud model

Machine learning has been widely used in high-spatial-resolution surface soil moisture (SSM) retrieval studies, but in recent years, this purely data-driven retrieval method has been controversial due to its lack of physical interpretability and generalization ability. Physical retrieval models based on the theory of radiative transfer equations respect physical laws, but their retrieval accuracy is usually affected by many insufficient accurate inputs, the complex model structure, and parameter adjustment method. In order to explore the retrieval method of unifying these two types of models, in this study, a differentiable model (DM) was constructed to realize the soil moisture retrieval at 10 m resolution based on Sentinel data. The differentiable soil moisture retrieval model takes the water cloud model (WCM) as the skeleton, and united the WCM and neural networks by implementing differentiable programming of the WCM in a machine learning platform. The differentiability makes the retrieval model trained by the gradient descent method the same as the neural network, which allows the retrieval model to be physical while obtaining more accurate retrieval results. Luan River Basin, Shandian River Basin, Maqu, and Lake Tahoe study areas with various land cover types and climate types were selected for model evaluation, and the performances of DM were close to that of the random forest model with Pearson correlation coefficient (R) of 0.747, 0.853, 0.838 and 0.792 in four study areas, respectively. While in the assessment of extrapolation capability of retrieval models, the DM showed its strong generalization ability and retrieval performance that exceeded that of the other retrieval models, with R of 0.786, unbiased root mean square error (ubRMSE) of 5.523 vol% and bias of 0.054 vol%. The DM synthesizes the advantages of both physical and machine learning models while providing high-resolution SSM estimates with acceptable accuracy for the study areas. This study creates favorable conditions for the realization of large-scale soil moisture retrieval with high resolution and high accuracy, and provides new ideas for the combination of machine learning and physical knowledge in other retrieval studies. • High resolution (10 m) soil moisture was generated by differentiable model (DM). • DM integrates physical model with deep learning models within a unified network. • DM uses gradient descent method for end-to-end retrieval model training. • DM shows excellent retrieval performance in a variety of environments. • DM has strong physical interpretability and generalization ability.