SpectralFormer: Rethinking Hyperspectral Image Classification with\n Transformers

Hyperspectral (HS) images are characterized by approximately contiguous\nspectral information, enabling the fine identification of materials by\ncapturing subtle spectral discrepancies. Owing to their excellent locally\ncontextual modeling ability, convolutional neural networks (CNNs) have been\nproven to be a powerful feature extractor in HS image classification. However,\nCNNs fail to mine and represent the sequence attributes of spectral signatures\nwell due to the limitations of their inherent network backbone. To solve this\nissue, we rethink HS image classification from a sequential perspective with\ntransformers, and propose a novel backbone network called \\ul{SpectralFormer}.\nBeyond band-wise representations in classic transformers, SpectralFormer is\ncapable of learning spectrally local sequence information from neighboring\nbands of HS images, yielding group-wise spectral embeddings. More\nsignificantly, to reduce the possibility of losing valuable information in the\nlayer-wise propagation process, we devise a cross-layer skip connection to\nconvey memory-like components from shallow to deep layers by adaptively\nlearning to fuse "soft" residuals across layers. It is worth noting that the\nproposed SpectralFormer is a highly flexible backbone network, which can be\napplicable to both pixel- and patch-wise inputs. We evaluate the classification\nperformance of the proposed SpectralFormer on three HS datasets by conducting\nextensive experiments, showing the superiority over classic transformers and\nachieving a significant improvement in comparison with state-of-the-art\nbackbone networks. The codes of this work will be available at\nhttps://github.com/danfenghong/IEEE_TGRS_SpectralFormer for the sake of\nreproducibility.\n