Deep multimodal fusion model for moisture content measurement of sand gravel using images, NIR spectra, and dielectric data

A fast and accurate moisture content (MC) measurement of sand gravel is essential for hydraulic engineering project sites. Most existing measurement methods are unimodal, facing non-robust against external interference. To address this issue, a deep multimodal fusion (DMF) model for measuring the MC of sand gravel using images, near-infrared (NIR) spectra, and dielectric data, is proposed. A modified bottleneck transformer network (BoTNet) added with an extremely efficient spatial pyramid (EESP) block is first proposed to extract image features from different receptive fields. The improved convolutional neural network with attention blocks added (A-CNN) and gated recurrent unit with attention blocks added (A-GRU) networks are then adopted to extract local and sequential features from NIR spectra, respectively. The square root of dielectric data and above multimodal features are effectively fused according to their contribution to the target indicator in the Fusion module. Among other comparative models, the DMF model yielded the best performance (R2 = 0.962, RMSE = 0.645, RPD = 5.124) on the original sand gravel dataset, and still maintained the best accuracy (the average R2 and RPD mostly exceeded 0.85 and 2.5, respectively) when against general external noise.