SAR-to-Optical Image Translation Using Vision Transformer-Based CGAN

In this paper, we propose a modified conditional generative adversarial network (CGAN) for synthetic aperture radar (SAR)-to-optical image translation. SAR imaging is widely employed in remote sensing due to its all-weather capability, yet its inability to capture color information can limit the extraction of crucial details. Consequently, several attempts aim to convert SAR images into optical images to overcome these shortcomings. Our research adopts a multi-level approach that jointly incorporates high-level and low-level terrain extracted from SAR images and color features. Our proposed method is composed of multi-scale vision Transformer (viT) blocks, enhanced loss function based on perceptual differences, and a two-phase transfer learning technique to guide the network properly. First, we obtain the multi-aspect features of SAR images by using the intermediate viT processor and viT bottleneck. Second, we add a perceptual loss function which is driven from the pre-trained VGG-19 network to overcome the blurring nature of traditional L1 loss. Finally, we employ a twophase transfer learning using patched grayscale optical images and noise-removed images pre-processed by the Lucy- Richardson filter. Compared to previous studies that use only a small portion of the SEN 1-2 dataset under 20,000 image pairs, our model is trained on 75,724 image pairs from this dataset. It ensures a significantly broader and more representative coverage of conditions and scenes compared to previous studies. The output image of our proposed method stands out with the highest peak signal-to-noise ratio (PSNR) of 15.96 dB and structural similarity index measure (SSIM) of 0.2805, along with the lowest mean squared error (MSE) 0.0363 and Fréchet inception distance (FID) score of 142.1333, proving that it generates superior images compared to conventional models in all respects.