Self-attention feature fusion network for semantic segmentation

Capturing rich contextual information is very helpful for semantic segmentation. In addition, the effective use of low-level details and high-level semantics is crucial for semantic segmentation. In this paper, we start from these two aspects, and we propose a self-attention feature fusion network for semantic segmentation (SA-FFNet) to improve semantic segmentation performance. Specifically, we introduced the vertical and horizontal compression attention module (VH-CAM) and the unequal channel pyramid pooling module (UC-PPM). The previous position attention module, such as the position attention module (PAM) in DANet, calculates the similarity between each pixel and all other pixels. However, the amount of information contained in a single isolated pixel is too small, so the resulting position attention weight is not perfect. Our approach is to compress a feature map from both vertical and horizontal directions so that the information contained in each pixel will be richer, and the spatial feature map obtained from this will be better. Experiments show that the position attention weight generated by our vertical and horizontal compression attention module is better than PAM in DANet. Additionally, our UC-PPM on each decoder can provide high-level rich semantic information to guide the selection of low-level feature maps. The proposed model achieves a 76.42% mean IoU on PASCAL VOC2012 and a 73.13% mean IoU on Cityscapes.