Sketch-Based 3D Shape Retrieval With Multi-View Fusion Transformer

Sketch-based 3D shape retrieval aims to retrieve similar 3D shapes given a 2D sketch query. Although this task has been studied for years, the inherent cross-modal gap and data imbalance between 2D sketches and 3D shapes remain challenging. To address the problems, we propose a simple and effective framework based on Multi-view Fusion Transformer. To be specific, we project 3D shapes into twelve distinct views, and their CNN features are combined with position embeddings, passing together into a transformer encoder to learn view weights. Then we process them through average pooling and MLPs to obtain the final 3D shape representation. Furthermore, to narrow the data imbalance between 2D sketches and 3D shapes, affine transformation and elastic deformation are fully utilized for sketch augmentation, so as to extract more comprehensive sketch features for feature matching with the multi-view 3D shape representation. Extensive experiments on SHREC13, SHREC14 and PART-SHREC14 datasets demonstrate our method achieves superior performance than previous competitive methods.