GTFN: GCN and Transformer Fusion Network With Spatial-Spectral Features for Hyperspectral Image Classification

Transformer has been widely used in classification tasks for hyperspectral images (HSI) in recent years. Because it can mine spectral sequence information to establish long-range dependence, its classification performance can be comparable with the convolutional neural network (CNN). However, both CNN and Transformer focus excessively on spatial or spectral domain features, resulting in an insufficient combination of spatial-spectral domain information from HSI for modeling. To solve this problem, we propose a new end-to-end graph convolutional network (GCN) and Transformer fusion network with the spatial-spectral feature extraction (GTFN) in this paper, which combines the strengths of GCN and Transformer in both spatial and spectral domain feature extraction, taking full advantage of the contextual information of classified pixels while establishing remote dependencies in the spectral domain compared with previous approaches. In addition, GTFN uses Follow Patch as an input to the GCN and effectively solves the problem of high model complexity while mining the relationship between pixels. It is worth noting that the spectral attention module is introduced in the process of GCN feature extraction, focusing on the contribution of different spectral bands to the classification. More importantly, to overcome the problem that Transformer is too scattered in the frequency domain feature extraction, a neighborhood convolution module is designed to fuse the local spectral domain features. On Indian Pines, Salinas, and Pavia University datasets, the overall accuracies (OAs) of our GTFN are 94.00%, 96.81%, and 95.14%, respectively. The core code of GTFN is released at https://github.com/1useryang/GTFN.