嵌入
可解释性
计算机科学
面子(社会学概念)
人工智能
投影(关系代数)
线性子空间
图像(数学)
可控性
计算机视觉
模式识别(心理学)
自然语言处理
算法
数学
社会科学
几何学
应用数学
社会学
作者
Chenliang Zhou,Fangcheng Zhong,A. Cengiz Öztireli
标识
DOI:10.1145/3588432.3591532
摘要
Recently introduced Contrastive Language-Image Pre-Training (CLIP) [Radford et al. 2021] bridges images and text by embedding them into a joint latent space. This opens the door to ample literature that aims to manipulate an input image by providing a textual explanation. However, due to the discrepancy between image and text embeddings in the joint space, using text embeddings as the optimization target often introduces undesired artifacts in the resulting images. Disentanglement, interpretability, and controllability are also hard to guarantee for manipulation. To alleviate these problems, we propose to define corpus subspaces spanned by relevant prompts to capture specific image characteristics. We introduce CLIP projection-augmentation embedding (PAE) as an optimization target to improve the performance of text-guided image manipulation. Our method is a simple and general paradigm that can be easily computed and adapted, and smoothly incorporated into any CLIP-based image manipulation algorithm. To demonstrate the effectiveness of our method, we conduct several theoretical and empirical studies. As a case study, we utilize the method for text-guided semantic face editing. We quantitatively and qualitatively demonstrate that PAE facilitates a more disentangled, interpretable, and controllable face image manipulation with state-of-the-art quality and accuracy.
科研通智能强力驱动
Strongly Powered by AbleSci AI