Visual Correspondence Learning and Spatially Attentive Synthesis via Transformer for Exemplar-Based Anime Line Art Colorization

Exemplar-based anime line art colorization of the same character has been a challenging problem in digital art production because of the sparse representation of line images and the significantly different anime appearance between line and color images. Therefore, it is a fundamental problem to find semantic correspondence between two kinds of images. In this paper, we propose a correspondence learning Transformer network for exemplar-based line art colorization, called ArtFormer, which utilizes a Transformer-based architecture to learn both spatial and visual relationships between line art and color images. ArtFormer mainly consists of two parts: correspondence learning and high-quality image generation. In particular, the correspondence learning module is composed of several Transformer blocks, each of which formulates the deep line image features and color images features as queries and keys, and learns the dense correspondence between two image domains. Then, the network synthesizes high-quality images with a newly proposed Spatial Attention Adaptive Normalization (SAAN) that uses warped deep exemplar features to modulate the shallow features for better adaptive normalization parameters generation. Both qualitative and quantitative experiments show that our method achieves the best performance on exemplar-based line art colorization compared with state-of-the-art methods and other baselines.