A modality‐collaborative convolution and transformer hybrid network for unpaired multi‐modal medical image segmentation with limited annotations

计算机科学 人工智能 情态动词 模态(人机交互) 分割 卷积(计算机科学) 医学影像学 图像分割 计算机视觉 人工神经网络 材料科学 高分子化学
作者
Hong Liu,Yuzhou Zhuang,Enmin Song,Xiangyang Xu,Guangzhi Ma,Coskun Cetinkaya,Chih‐Cheng Hung
出处
期刊:Medical Physics [Wiley]
卷期号:50 (9): 5460-5478 被引量:12
标识
DOI:10.1002/mp.16338
摘要

BACKGROUND: Multi-modal learning is widely adopted to learn the latent complementary information between different modalities in multi-modal medical image segmentation tasks. Nevertheless, the traditional multi-modal learning methods require spatially well-aligned and paired multi-modal images for supervised training, which cannot leverage unpaired multi-modal images with spatial misalignment and modality discrepancy. For training accurate multi-modal segmentation networks using easily accessible and low-cost unpaired multi-modal images in clinical practice, unpaired multi-modal learning has received comprehensive attention recently. PURPOSE: Existing unpaired multi-modal learning methods usually focus on the intensity distribution gap but ignore the scale variation problem between different modalities. Besides, within existing methods, shared convolutional kernels are frequently employed to capture common patterns in all modalities, but they are typically inefficient at learning global contextual information. On the other hand, existing methods highly rely on a large number of labeled unpaired multi-modal scans for training, which ignores the practical scenario when labeled data is limited. To solve the above problems, we propose a modality-collaborative convolution and transformer hybrid network (MCTHNet) using semi-supervised learning for unpaired multi-modal segmentation with limited annotations, which not only collaboratively learns modality-specific and modality-invariant representations, but also could automatically leverage extensive unlabeled scans for improving performance. METHODS: We make three main contributions to the proposed method. First, to alleviate the intensity distribution gap and scale variation problems across modalities, we develop a modality-specific scale-aware convolution (MSSC) module that can adaptively adjust the receptive field sizes and feature normalization parameters according to the input. Secondly, we propose a modality-invariant vision transformer (MIViT) module as the shared bottleneck layer for all modalities, which implicitly incorporates convolution-like local operations with the global processing of transformers for learning generalizable modality-invariant representations. Third, we design a multi-modal cross pseudo supervision (MCPS) method for semi-supervised learning, which enforces the consistency between the pseudo segmentation maps generated by two perturbed networks to acquire abundant annotation information from unlabeled unpaired multi-modal scans. RESULTS: Extensive experiments are performed on two unpaired CT and MR segmentation datasets, including a cardiac substructure dataset derived from the MMWHS-2017 dataset and an abdominal multi-organ dataset consisting of the BTCV and CHAOS datasets. Experiment results show that our proposed method significantly outperforms other existing state-of-the-art methods under various labeling ratios, and achieves a comparable segmentation performance close to single-modal methods with fully labeled data by only leveraging a small portion of labeled data. Specifically, when the labeling ratio is 25%, our proposed method achieves overall mean DSC values of 78.56% and 76.18% in cardiac and abdominal segmentation, respectively, which significantly improves the average DSC value of two tasks by 12.84% compared to single-modal U-Net models. CONCLUSIONS: Our proposed method is beneficial for reducing the annotation burden of unpaired multi-modal medical images in clinical applications.
最长约 10秒,即可获得该文献文件

科研通智能强力驱动
Strongly Powered by AbleSci AI
科研通是完全免费的文献互助平台,具备全网最快的应助速度,最高的求助完成率。 对每一个文献求助,科研通都将尽心尽力,给求助人一个满意的交代。
实时播报
Hello的应助被guo采纳,获得10
刚刚
刚刚
不开花花发布了新的文献求助10
刚刚
顾矜的应助被桃子汽水采纳,获得10
1秒前
123发布了新的文献求助10
2秒前
情怀的应助被hwl采纳,获得10
2秒前
3秒前
能干晓啸完成签到,获得积分10
3秒前
无辜蜗牛完成签到,获得积分10
3秒前
黑米粥发布了新的文献求助200
4秒前
IGNIS完成签到 ,获得积分10
4秒前
JIA发布了新的文献求助200
4秒前
阳光幻枫完成签到,获得积分10
4秒前
Owen的应助被干就完采纳,获得10
6秒前
6秒前
大模型的应助被无辜蜗牛采纳,获得10
7秒前
Maestro_S的应助被寸烛驱夜采纳,获得30
8秒前
蝈蝈发布了新的文献求助10
8秒前
9秒前
9秒前
123完成签到,获得积分10
9秒前
JamesPei的应助被小豆糕采纳,获得10
10秒前
邱邱科研发布了新的文献求助10
10秒前
丘比特的应助被elysia采纳,获得10
11秒前
思源的应助被风吹麦浪采纳,获得10
11秒前
香蕉觅云的应助被风吹麦浪采纳,获得10
11秒前
今后的应助被风吹麦浪采纳,获得30
11秒前
Lin关闭了Lin的文献求助
12秒前
12秒前
qingqingdandan完成签到 ,获得积分10
12秒前
12秒前
ccc关闭了ccc的文献求助
13秒前
13秒前
15秒前
zz完成签到,获得积分10
15秒前
duck发布了新的文献求助10
15秒前
15秒前
黑米粥发布了新的文献求助200
16秒前
ding的应助被汤汤采纳,获得10
16秒前
王明磊完成签到 ,获得积分10
16秒前
高分求助中
(应助此贴封号)【重要!!请各用户(尤其是新用户)详细阅读】【科研通的精品贴汇总】 10000
拉伸高分子熔体中缠结行为的模拟和小角中子散射研究 1700
The World Atlas of Wine 9th edition 1000
奈特基础组织学:配套组织病理学(第4版)(Netter's Essential Histology With Correlated Histopathology) 800
八旗制度の研究 600
Improving the wear resistance of vehicle gear oil by oil soluble LaF3 nanoparticles and typical additives of gear oil 500
Boundaries and Bodhisattvas 500
热门求助领域 (近24小时)
化学 材料科学 医学 生物 纳米技术 计算机科学 工程类 有机化学 化学工程 生物化学 物理 复合材料 内科学 催化作用 光电子学 人工智能 细胞生物学 无机化学 物理化学 数学
热门帖子
关注 科研通微信公众号,转发送积分 8183792
求助须知:如何正确求助?哪些是违规求助? 9573020
关于积分的说明 21828776
捐赠科研通 7780959
什么是DOI,文献DOI怎么找? 3434116
关于科研通互助平台的介绍 2558301
邀请新用户注册赠送积分活动 2480611