Separate first, then segment: An integrity segmentation network for salient object detection

Current methods aggregate multi-level features or introduce auxiliary information to get more refined saliency maps. However, little attention is paid to how to obtain complete salient objects in cluttered background. To address this problem, we propose an integrity segmentation network (ISNet) with a novel detection paradigm that first separates the targets completely and then segment them finely. Specifically, the ISNet architecture consists of a target separation (TS) branch and an object segmentation (OS) branch, trained using a hierarchical difference-aware (HDA) loss. The TS branch equipped with a fractal structure is utilized to produce saliency features with expanded boundary (SF w/ EB), which can enlarge the difference of border details to separate the target from background completely. Compared with the edge and skeleton information, the SF w/ EB contains a more complete structure, which can supplement the defect of salient objects. The OS branch is leveraged to generate complementary features, which gradually integrates the SF w/ EB and aggregated features to segment complete saliency maps. Moreover, we propose the HDA loss to further improve the structural integrity of prediction, which hierarchically assigns weight to pixels according to their differences. Hard pixels will be given more attention to discriminate the similar parts between foreground and background. Comprehensive experimental results on five datasets show that the proposed ISNet outperforms the state-of-the-art methods both quantitatively and qualitatively. Concretely, compared with three typical models, the average gain percentage reaches 2.6% in terms of Fβ, Sm and MAE on two large complex datasets. The improvements demonstrate that the proposed ISNet are beneficial for improving the integrity of prediction. Besides, the ISNet is efficient and runs at a real-time speed of 39.5 FPS when processing an image with size of 320 × 320. Furthermore, the proposed model has better generalization, which can also be applied to other vision tasks to handle complex scenes. Codes are available at https://github.com/lesonly/ISNet.