Penta-Encoder with Medical Transformer for Incomplete Multimodal Learning of Brain Tumor Segmentation

Accurate brain tumor segmentation of magnetic resonance imaging (MRI) images requires the integration of multimodal data in effective collaborative learning. However, clinical scenarios often face missing modalities, significantly impacting segmentation performance. In this paper, we propose the penta-encoder with medical transformer (Penta-MedTrans) for incomplete multimodal learning, which consists of four main components: a modality-specific encoder (MSE), a modality-dynamic encoder (MDE), a multimodal fusion module (MMFM) and a convolutional decoder. The MSE extracts unique features for each modality, while the MDE captures inter-modal semantics to address missing modalities. The MMFM establishes and aligns long-range correlations between modal-invariant features and global semantics of tumor regions. The convolutional decoder generates robust segmentation results by progressively upsampling and fusing modality-invariant features. We extensively validated our method on the BraTS 2020 and BraTS 2021 datasets for brain tumor segmentation. Results showed that our method outperforms SOTA for incomplete multimodal brain tumor segmentation across most subsets.