Cross-modality synthesis aiding lung tumor segmentation on multi-modal MRI images

The multi-modal images provide complementary information for the quantitative analysis of the cancer treatment. However, there are some challenges for automatic multi-modal tumor segmentation algorithms, including segmenting the tumors adherent to the normal tissues, the content shift caused from the distinct imaging mechanisms, and high cost of acquiring the paired functional modality images. To alleviate these problems, this paper proposed a multi-modal tumor segmentation model based on the cross-modality synthesis network. The proposed model consists of the cross-modality synthesis network and the multi-modal segmentation network: the cycle-consistent image conditional variational autoencoders (CICVAE) and the residual U-net (Res-Unet), respectively. Trained in a novel semantic cycle-consistency loss, CICVAE model synthesizes the paired auxiliary images solely from the anatomical images, in place of scanning functional images for the multi-modal tumor segmentation. Consequently, these synthesized images display high signal in the tumor region similar to the scans of functional modality but with no content shift. Then the anatomical modality images are concatenated with the synthesized images to Res-Unet for the segmentation of lung tumors. The effectiveness of the proposed generative segmentation model is demonstrated on a T2W-DWI MRI dataset of 57 patients with 355 slices. Compared with other multi-modal segmentation methods, Dice Similarity Coefficient (DSC) and 95% Hausdorff Distance (95HD) of the proposed model on testing sets are improved by 3.14% and 4.89%, respectively. The experimental results show that the proposed model outperforms the single modal segmentation model and achieves competitive results with low model complexity.