Transformer-Based Multimodal Fusion for Early Diagnosis of Alzheimer's Disease Using Structural MRI And PET

Magnetic Resonance Imaging (MRI) and Positron Emission Tomography (PET) are both the widely used imaging modalities for early diagnosis of Alzheimer's disease (AD). Combining these two modalities allow using both anatomical and metabolic information for evaluating brain status. However, the commonly-used multimodal fusion strategy, i.e., through channel concatenation, cannot effectively exploit complementary information among these two modalities. To encourage effective information exchange between structural MRI (sMRI) and FDG-PET as used in our study for early AD diagnosis, we propose a novel transformer-based multimodal fusion framework. Specially, our proposed model composes of three parts: 1) Feature extraction based on adversarial training; 2) Feature fusion based on multimodal transformer through cross-attention mechanism; 3) Classification head based on full connection. By resorting to adversarial learning, the feature gap between two modalities becomes smaller, thus easing the cross-attention operation to achieve more effective fusion. In the experiment, we show that our model outperforms other representative models by a large margin.