MFJLN: Multi-Frequency Feature Joint Learning Network for Rain Removal

Most deep learning-based rain removal methods consider rain streaks as a part of the high-frequency information in the image to extract and remove rain streak features. This usually leads to the problem of excessive or insufficient removal of rain streaks, leading to blurred edges or residual rain streaks in the rain removal results. To resolve this problem, a multi-frequency feature joint learning network (MFJLN)is proposed, which is constructed as a U-shaped structure including an encoder and a decoder. At each scale layer, a full-frequency feature fusion module (3FM) consisting of a spatial domain branch and a frequency domain branch is constructed to achieve accurate feature extraction at different scales. In the spatial domain branch, considering that rain streaks not only exist in high-frequency components, a multifrequency feature extraction block (MFEB) is constructed to extract rich rain streak features from multiple frequency layers, namely low-frequency, mid-frequency, and high-frequency layers. In addition, in the low-frequency layer, a masked self-attention block (MSAB) is designed by defining an adaptive position weight mask to learn accurate long-distance features. To achieve the reuse of encoding features, a feature fusion block (FFB) is constructed to fuse the features from all encoding layers with the features of each decoding layer through dense links for feature reconstruction. Numerous experimental results on both synthetic and real-world datasets have shown that our MFJLN is superior to some state-ofthe-art (SOTA) rain removal methods in terms of visual effects and quantitative metrics. In addition, MFJLN can not only remove rain streaks from rainy images, but also effectively remove raindrops and snow marks.