Sketch and Vector-Guided 3D Shape Generation via Cross-Modal Diffusion

In 3D modeling and generation tasks, it remains challenging to drive 3D shape synthesis from sparse lines and high-level semantics provided by sketch drawings. To address this, we propose a 3D shape generation framework based on cross-modal feature fusion and diffusion models. Specifically, we first employ a VQ-VAE to learn discrete representations of 3D shapes, constructing a compact and quantizable latent space. Then, within the diffusion model, sketch images and explicit vectors (SVG parameters) are used as dual-modal conditions to progressively denoise and refine the latent features. We design a Dual Cross-Attention mechanism, enabling semantic information from sketches and geometric constraints from vector sequences to fully interact during the denoising process. This allows the model to preserve global contour consistency while enhancing local structure and detail generation. Experimental results show that introducing explicit vector conditions effectively compensates for the limited expressiveness of sketches, improving the fidelity and detail quality of generated shapes, and outperforming existing single-modal conditional methods.