Bridging Low-Level and High-Level Visions: A Hybrid Transformer-CNN Multitask Framework for Hyperspectral Image Classification

Hyperspectral images (HSIs) are often subject to blurring during the imaging process, particularly around class boundaries. Such blurring typically results in the loss of spatial details, posing a great challenge to HSI classification task, especially when the number of training samples is limited. Super-resolution (SR) as a powerful tool in low-level vision tasks, can recover fine-grained spatial details from degraded images. However, most existing methods for HSI classification and super-resolution are implemented independently. On the one hand, this paradigm neglects the connection between low-level and high-level vision tasks; on the other hand, the multi-tasking framework enables a more comprehensive exploration of the feature space with limited training samples. In this paper, we propose a novel multi-task framework that integrates super-resolution as an auxiliary task to enhance classification performance. Specifically, we exploit the strengths of convolution neural network (CNN) and Transformer in extracting high-frequency and low-frequency information to construct a dual-branch network and design a frequency-domain interaction module (FIM) to facilitate information exchange between the two branches for a powerful representation capability. In addition, we propose a cross-task interaction module (CTIM) that effectively integrates spatial details captured by the SR task into the classification task. Furthermore, we propose a frequency-domain contrastive learning (FCL) to enforce the model to learn the consistency between both tasks and boost the instance discriminability. Extensive experiments on four hyperspectral datasets demonstrates that the proposed method achieve competitive results compared with the state-of-the-art HSI classification methods.