Synthetic aperture radar and optical image registration using local and global feature learning by modality-shared attention network

The registration of synthetic aperture radar (SAR) and optical images is a meaningful but challenging multimodal task. Due to the large radiometric differences between SAR and optical images, it is difficult to obtain discriminative features only by mining local features in the traditional Siamese convolutional networks. We propose a modality-shared attention network (MSA-Net) that introduces nonlocal attention (NLA) to the partially shared two-stream network to jointly exploit local and global features. First, a modality-specific feature learning module is designed to efficiently extract shallow modality-specific features from SAR and optical images. Subsequently, a modality-shared feature learning (MShFL) module is designed to extract deep modality-shared features. The local feature extraction module and the NLA module in MShFL extract deep local and global features to enrich feature representations. Furthermore, a triplet loss function with a cross-modality similarity constraint is constructed to learn modality-shared feature representations, thereby reducing nonlinear radiometric differences between the two modalities. The MSA-Net is trained on a public SAR and optical dataset and tested on five pairs of SAR and optical images. In the registration results of five pairs of test SAR and optical images, the matching rate of the MSA-Net is 5% to 15% higher than that of other compared methods, and the matching errors of the matched inliers are on average reduced by about 0.28. Several ablation experiments verify the effectiveness of the partially shared network structure, the MShFL module, and the cross-modality similarity constraint.