Multiple vision architectures-based hybrid network for hyperspectral image classification

More recently, vision transformer (ViT) has shown competitive performance with convolutional neural network (CNN) on computer vision tasks, which provided more possibilities for accurate classification of hyperspectral image (HSI). However, whether CNN or ViT, they generally only focus on single type of feature, resulting in insufficient information utilization. For instance, CNN has powerful local feature extraction ability, while ViT pays more attention to long-range dependencies and global features. To consider multiple types of feature information, we propose a multiple vision architectures-based hybrid network (MVAHN) for HSI classification, which consists of joint CNN and transformer (JCT) structure and graph convolutional module (GCM). Firstly, JCT successfully embeds convolution operations into ViT to capture local and global features, which mainly include: 1) A spectral spatial convolution block (SSCB) is proposed to unearth local spectral spatial features. 2) A convolution embedding is aggregated into self-attention to design a local–global attention (LGA) mechanism, which can realize the seamless integration of CNN and ViT, thereby capturing local–global combined features. Secondly, a plug-and-play GCM is developed in parallel with transformer encoders to further improve the model classification ability by mining the similarity relationship between pixels in HSI. Overall, an elegant integration of these seemingly distinct paradigms is realized by MVAHN to capture multiple types of feature information. The overall accuracies (OAs) of MVAHN on Pavia University, Houston 2013, Salinas Valley, Kennedy Space Center, Indian Pines and Botswana datasets are 96.37%, 88.33%, 97.57%, 98.96%, 96.25% and 99.26%, respectively. Compared with the state-of-the-art hybrid models, MVAHN achieves competitive classification results. The source code will be available at https://github.com/ZJier/MVAHN.