Integrating Prior Knowledge into Bi-Branch Pyramid Network for Medical Image Segmentation

Medical image segmentation is crucial for accurate diagnoses, and while convolutional neural network (CNN)-based methods have made strides in recent years, they struggle with modeling long-range dependencies. Transformer-based methods improve this aspect but require more computational resources. The Segment Anything Model (SAM) can generate pixel-level segmentation in natural images using sparse manual prompts, but it performs poorly on low-contrast, noisy ultrasound images. To address this issue, we propose a new medical image segmentation network architecture that integrates the transformer components, CNN modules, and SAM encoder into a unified framework. This allows us to capture both long-range dependencies and local features simultaneously. Additionally, we incorporate the extracted image features from the SAM model as prior knowledge to further improve segmentation accuracy with limited training data. To reduce computational stress, we employ the axial attention mechanism to approximate the transformer's effects by expanding the receptive field. Instead of replacing transformer components with lightweight attention modules, our model is divided into a global branch and a local branch. The global branch extracts context features with the transformer components, while the local branch processes patch tokens with the axial attention mechanism. We also construct an image pyramid to excavate internal statistics and multiscale representations to obtain more accurate segmentation regions. This bi-branch pyramid transformer (Bi-BPT) architecture is effective and robust for medical image segmentation, surpassing other related segmentation network architectures. The experimental results on various medical image datasets demonstrate its effectiveness.