Self-Attention Driven Image-to-image Translation with Conditional Adversarial Networks

In this paper, we introduce a novel model that combines the strengths of Pix2pix with a Self-Attention mechanism. Our self-attention module is uniquely designed to not only compute relationships and dependencies as represented in the attention map but also to determine attention values. These values facilitate the bifurcation of input image features between two distinct generators. To enhance the efficacy of this separation, we devised a method to remap attention values. Additionally, our model integrates a paired down-scaling and up-scaling process, which significantly conserves GPU memory. This efficiency makes our model particularly suitable for lightweight devices. Experimental results indicate that our proposed model outperforms the original Pix2pix in image quality, as evidenced by both visual assessments and quantitative scores from semantic segmentation models.