Medical Language Mixture of Experts for Improving Medical Image Segmentation

Traditional medical image segmentation methods are mostly uni-modal approaches solely based on the image modality. Recently, the emergence of text-guided image segmentation methods, by utilizing text annotations to compensate for the quality deficiency in image data, has shown promise for improving medical image segmentation. Despite their success, these methods often experience inadequate utilization of beneficial text information, and have applicability issues in the missing text modality scenario. To address these limitations, in this paper, we propose a Medical Language Mixture of Experts (MLMoE), which introduces multiple sub-experts for extracting more diverse information from medical text. These different experts are then combined by a gating module, thus aggregating beneficial text information to assist the image segmentation. Furthermore, to guarantee its performance in the text-absent scenario, a virtual prompt based distillation module is proposed, which distills the valuable knowledge of MLMoE learned from available text information to the virtual prompt, as an alternative text input. Experimental results on two multi-modal medical segmentation datasets demonstrate the effectiveness of our opposed method, achieving state-of-the-art performance. Code will be available at: https://github.com/Rango-bit/MLMoE.git.