MMTFN: Multi‐modal multi‐scale transformer fusion network for Alzheimer's disease diagnosis

Abstract Alzheimer's disease (AD) is a severe neurodegenerative disease that can cause dementia symptoms. Currently, most research methods for diagnosing AD rely on fusing neuroimaging data of different modalities to exploit their heterogeneity and complementarity. However, effectively using such multi‐modal information to construct fusion methods remains a challenging problem. To address this issue, we propose a multi‐modal multi‐scale transformer fusion network (MMTFN) for computer‐aided diagnosis of AD. Our network comprises 3D multi‐scale residual block (3DMRB) layers and the Transformer network that jointly learns potential representations of multi‐modal data. The 3DMRB with multi‐scale aggregation efficiently extracts local abnormal information related to AD in the brain. We conducted five experiments to validate our model using MRI and PET images of 720 subjects from the Alzheimer's Disease Neuroimaging Initiative (ADNI). The experimental results show that our proposed network outperformed existing models, achieving a final classification accuracy of 94.61% for AD and Normal Control.