Multimodal CT and MR Segmentation of Head and Neck Organs-at-Risk

Radiotherapy (RT) is a standard treatment modality for head and neck (HaN) cancer that requires accurate segmentation of target volumes and nearby healthy organs-at-risk (OARs) to optimize radiation dose distribution. However, computed tomography (CT) imaging has low image contrast for soft tissues, making accurate segmentation of soft tissue OARs challenging. Therefore, magnetic resonance (MR) imaging has been recommended to enhance the segmentation of soft tissue OARs in the HaN region. Based on our two empirical observations that deformable registration of CT and MR images of the same patient is inherently imperfect and that concatenating such images at the input layer of a deep learning network cannot optimally exploit the information provided by the MR modality, we propose a novel modality fusion module (MFM) that learns to spatially align MR-based feature maps before fusing them with CT-based feature maps. The proposed MFM can be easily implemented into any existing multimodal backbone network. Our implementation within the nnU-Net framework shows promising results on a dataset of CT and MR image pairs from the same patients. Furthermore, the evaluation on a clinically realistic scenario with the missing MR modality shows that MFM outperforms other state-of-the-art multimodal approaches.