MHCFormer: Multiscale Hierarchical Conv-Aided Fourierformer for Hyperspectral Image Classification

Convolutional neural networks (CNNs) have dominated the hyperspectral image (HSI) classification due to their tremendous feature learning capability. However, the formidable local sensitivity is both a strength and a weakness. Recently, the vision transformers have exhibited impressive performances on various vision problems. Compared with CNNs, they can model long-range dependencies to learn more abundant interactions between spatial locations. Nevertheless, the existing transformer-based HSI classification methods also concentrate too much on the advantages of the transformer architecture and disregard the importance of local dependencies. In addition, token generation and token mixers in transformer-like architectures have not been adequately explored, leading to difficulties in obtaining the best classification performance. To deal with these problems, a novel multiscale hierarchical conv-aided Fourierformer (MHCFormer) is proposed for HSI classification. To the best of our knowledge, this is the first time that CNN, transformer, and Fourier transform are skillfully combined for hyperspectral image classification. The proposed MHCFormer involves three stages, i.e., multiscale spectral-spatial token generation, hierarchical token learning and a classification head. The multiscale spectral-spatial token generation is constructed to transform HSI into tokens with multiscale enhanced spectral-spatial information. The hierarchical token learning is designed to explore multiscale tokens globally and locally by integrating the design philosophy of transformers and CNNs along with Fourier transforms into a block and stacking the blocks hierarchically. Extensive experimental results on the new WHU-Hi-HanChuan dataset and the widely used Indian Pines and Houston 2013 datasets have demonstrated the superiority of MHCFormer over other state-of-the-art methods. The code of our work will be available publicly at https://github.com/Tikiten/MHCFormer.