M2ANet: Multi-branch and multi-scale attention network for medical image segmentation

Abstract Convolutional neural networks (CNNs)-based medical image segmentation technologies have been widely used in medical image segmentation because of their strong representation and generalization abilities. However, due to the inability to effectively capture global information from images, CNNs can easily lead to loss of contours and textures in segmentation results. Notice that the transformer model can effectively capture the properties of long-range dependencies in the image, and furthermore, combining the CNN and the transformer can effectively extract local details and global contextual features of the image. Motivated by this, we propose a multi-branch and multi-scale attention network (M2ANet) for medical image segmentation, whose architecture consists of three components. Specifically, in the first component, we construct an adaptive multi-branch patch module for parallel extraction of image features to reduce information loss caused by downsampling. In the second component, we apply residual block to the well-known convolutional block attention module to enhance the network’s ability to recognize important features of images and alleviate the phenomenon of gradient vanishing. In the third component, we design a multi-scale feature fusion module, in which we adopt adaptive average pooling and position encoding to enhance contextual features, and then multi-head attention is introduced to further enrich feature representation. Finally, we validate the effectiveness and feasibility of the proposed M2ANet method through comparative experiments on four benchmark medical image segmentation datasets, particularly in the context of preserving contours and textures. The source code of M2ANet will be released at https://github.com/AHUT-MILAGroup/M2ANet.