Multi-modal global- and local- feature interaction with attention-based mechanism for diagnosis of Alzheimer’s disease

Alzheimer's disease is a complex neurodegenerative disease. Subjects with Mild Cognitive Impairment will progress to Alzheimer's disease, thus how to effectively diagnose Alzheimer's disease or Mild Cognitive Impairment using the clinical tabular data and Magnetic Resonance Images of the brain together has been a major concern of researches. Deep multi-modal learning-based methods can improve Alzheimer's disease diagnostic accuracy compared to the single modality-based methods. However, most existing multi-modal fusion methods only focus on learning global features fusion from image and clinical tabular data by concatenation, lacking the ability to jointly analyze and integrate global–local information of image with clinical tabular data. To address these limitations, this paper explored a novel Multi-Modal Global–Local Fusion method to perform multi-modal Alzheimer's disease classification through 3D Magnetic Resonance Images and clinical tabular data. Specifically, we adopt a global module that uses concatenation to fuse features to learn the global information. Moreover, we design an attention-based local module which encourages clinical tabular features to guide the learning of local 3D Magnetic Resonance Images information, thus, enhancing the power of features fusion from each modality. Our method considers both global and local information of the two modalities for multi-modal fusion. Experiment results show that our method in this paper is highly effective in combining 3D Magnetic Resonance Images and clinical tabular data for Alzheimer's disease classification with accuracy of 86.34% and 86.77% in ADNI and OASIS-1 datasets respectively, which outperforms the current state-of-the-art methods. Detailed ablation experiments are conducted to highlight the contribution of various components. code is available at: https://github.com/nananana0701/MMGLF.