PTsFusion: Multimodal Medical Image Fusion Based on Parallel Transformers with Global-Local Feature Interaction

Multimodal medical image fusion aim is to integrate complementary information from different modal images. In response to the existing multimodal medical image fusion methods, transformers struggles to extract local features, and loss of detailed information in the feature extraction process. Our proposed solution is parallel transformers with global-local feature interaction (PTsFusion). The PTsFusion mostly composed of three modules — Encoder, Fusion, and Decoder. Within the Encoder module, we use parallel transformers with global-local feature interaction, which consist of global transformer blocks, local transformer blocks, and global-local feature interaction blocks. The global transformer block focuses on extracting global deep features using the global attention mechanism. The local transformer block handles local deep features through the local attention mechanism. The global-local feature interaction block capture correlations between the global deep features and local deep features. In the Fusion module, the global deep features and local deep features from various modal images are fused by the -norm-based image-sequence matrix fusion rule. Ultimately, the Decoder module remodels the fused image by deconvolution. Experimental findings illustrate that the generated fused image exhibits clear textures edges and local information, surpassing other comparative methods in qualitative and quantitative analyses.