MMF-RNN: A Multimodal Fusion Model for Precipitation Nowcasting Using Radar and Ground Station Data

Precipitation nowcasting is crucial for economic development and social life. Numerous deep learning models have recently been developed and have achieved better results than traditional extrapolation models. However, they mainly focus on improving model architectures, ignoring the impact of error accumulation and data inconsistency. This article proposes a multimodal fusion model named multimodal fusion recurrent neural network (MMF-RNN) for precipitation prediction. Specifically, we use a dual-branch encoder to extract features from radar and ground station data and then fuse them effectively through attention mechanisms and multimodal loss (ML). To address the error accumulation problem, we propose a block-based dynamic weighted loss (BDWLoss) that enables the model to focus more on hard-to-predict areas during training to reduce error accumulation. Based on BDWLoss, we propose an ML that encourages the model to maintain consistency between single-modal and fused multimodal features. In addition, MMF-RNN is compatible with various RNN models such as ConvLSTM, PredRNN, PredRNN++, and MIM. The experimental results on the RAIN-F dataset demonstrate that MMF-RNN outperforms both the single-modal model MS-RNN and the multimodal model MM-RNN. In particular, MMF-RNN achieves significant improvement in predicting heavy precipitation. Compared to MM-PredRNN++, MMF-PredRNN++ shows marked improvements across various performance metrics, with critical success index (CSI) ( $R\geq 5 $ ) and Heidke skill score (HSS) ( $R\geq 5$ ) increasing by 58.08% and 48.55%, respectively, and CSI ( $R\geq 10$ ) and HSS ( $R\geq 10$ ) showing more pronounced gains. These advancements are facilitated not only by the proposed architectural innovations but also by sample weighting, which collectively contribute to superior performance on imbalanced precipitation datasets.