计算机科学
人工智能
像素
蒸馏
分割
推论
模式识别(心理学)
模态(人机交互)
图像分割
模式
机器学习
社会科学
社会学
有机化学
化学
作者
Cheng Chen,Qi Dou,Yueming Jin,Quande Liu,Pheng‐Ann Heng
出处
期刊:IEEE Transactions on Medical Imaging
[Institute of Electrical and Electronics Engineers]
日期:2022-03-01
卷期号:41 (3): 621-632
被引量:22
标识
DOI:10.1109/tmi.2021.3119385
摘要
Multimodal learning usually requires a complete set of modalities during inference to maintain performance. Although training data can be well-prepared with high-quality multiple modalities, in many cases of clinical practice, only one modality can be acquired and important clinical evaluations have to be made based on the limited single modality information. In this work, we propose a privileged knowledge learning framework with the ‘Teacher-Student’ architecture, in which the complete multimodal knowledge that is only available in the training data (called privileged information) is transferred from a multimodal teacher network to a unimodal student network, via both a pixel-level and an image-level distillation scheme. Specifically, for the pixel-level distillation, we introduce a regularized knowledge distillation loss which encourages the student to mimic the teacher’s softened outputs in a pixel-wise manner and incorporates a regularization factor to reduce the effect of incorrect predictions from the teacher. For the image-level distillation, we propose a contrastive knowledge distillation loss which encodes image-level structured information to enrich the knowledge encoding in combination with the pixel-level distillation. We extensively evaluate our method on two different multi-class segmentation tasks, i.e., cardiac substructure segmentation and brain tumor segmentation. Experimental results on both tasks demonstrate that our privileged knowledge learning is effective in improving unimodal segmentation and outperforms previous methods.
科研通智能强力驱动
Strongly Powered by AbleSci AI