An end-to-end medical image fusion network based on Swin-transformer

Medical image fusion is a method that generates an individual picture with different modality information by exploiting images obtained from different medical examinations. The Transformer mechanism can model long-term dependencies through self-attention for better utilization of various pieces of information available in an image. Inspired by this, we propose an end-to-end image fusion network built based on Swin-Transformer mechanism, which can fuse well the local and long-range (or global context) information about images. The proposed network is trained with a two-phase training scheme. In the first phase, we train a multi-layer Transformer-based autoencoder network for extracting shallow and multi-scale deep features. In the second phase, multiple features are fused Transformer and Convolutional Neural Networks (CNN), which have enough ability to capture both local and long-range features. The conducted experiments demonstrate that our method outperforms multitudinous competing fusion algorithms on both subjective and objective evaluations. Furthermore, in other applications, the network can be combined more easily with the Transformer mechanism based networks for reducing the complexity and running time of the overall system, and improving the comprehensive image processing performance of embedded devices.