HTF: Hierarchical Transformer Fusion for Fine-Grained Sketch-based Retrieval

Fine-grained sketch-based image retrieval (FGSBIR) aims to identify target photographs through minimal sketch input, yet current approaches face critical limitations in balancing local feature precision with global contextual awareness. Although existing methods predominantly focus on stroke-level variations through convolutional architectures, they often neglect the crucial interplay between detailed sketch patterns and holistic structural semantics. This oversight leads to suboptimal performance during the early retrieval phases when the sketch input remains incomplete and abstract. To bridge this critical gap, we present a novel Hierarchical Transformer Fusion (HTF) framework that synergizes dual-branch feature extraction with reinforcement learning optimization. Our architecture introduces three key innovations: First, a dual-branch encoder combines Swin Transformer for local stroke-level feature extraction with ResNet-50 and Transformer encoder for global semantic understanding, providing rich multi-level feature representations that facilitate effective cross-modal matching. Second, an adaptive fusion mechanism employs learnable weights to dynamically integrate multi-scale features from both branches, prioritizing semantically salient regions through content-aware attention. Third, we devise a reinforcement learning-driven cross-modal alignment strategy that optimizes retrieval policy through iterative feedback, allowing the model to accommodate diverse sketch patterns and user preferences. Experimental results confirm our method’s superior early retrieval accuracy compared to existing FG-SBIR systems and baseline models.