Balancing Performance and Efficiency: Toward Superior Image Segmentation With Adaptive Sparse Attention

Recently, most image segmentation methods exhibit an extreme trade-off between performance and efficiency, resulting in approaches with high performance typically having low computational efficiency, while efficient methods compromise on segmentation accuracy. To address this dual challenge, this study introduces a simple yet efficient segmentation framework based on Multi-scale Prototype matching and visual Sparse Attention mechanisms (MPSA), which is a transformer-based architecture designed to optimize the balance between performance and efficiency. The proposed MPSA integrates a novel lightweight cross-attention mechanism and prototype selection and filtering strategy to accurately correlate category queries with corresponding visual objects with a multi-scale Feature Pyramid Network (FPN). Within the pixel decoder, our Axial Convolution Enhanced (ACE) module mitigates lost global context by combining depth-wise separable convolutions with deformable convolutions, thereby recovering global semantics while preserving fine-grained spatial details. Through this innovative design, MPSA demonstrates outstanding performance in both semantic and panoptic segmentation tasks across multiple datasets. Remarkably, MPSA achieves surprising 83.9% mIoU with only 114M parameters on the Cityscapes dataset while compared to some state-of-the-art architectures, highlighting its ability to deliver exceptional results with significantly reduced resource consumption. Our code is released at https://github.com/zxqing01/MPSA.