Exploring viewport features for semi-supervised saliency prediction in omnidirectional images

Compared with the annotated data for the 2D image saliency prediction task, the annotated data for training omnidirectional image (or 360° image) saliency prediction models are not sufficient. Most existing fully-supervised saliency prediction methods for omnidirectional images (ODIs) adopt a scheme, first training the methods on a labeled large 2D image saliency prediction dataset and then fine-tuning the methods on the labeled tiny ODI saliency prediction dataset. However, this strategy is time-consuming and may not inadequately mine the visual features built in ODIs. To explore the visual attributes targeted at ODIs and address the shortage of labels on these ODIs, in this paper, we propose an end-to-end semi-supervised network, namely VFNet, which relies on viewport features and only utilizes ODIs as training data, for ODI saliency prediction. Concretely, we adopt consistency regularization as our semi-supervised learning framework. The predictions between main and auxiliary saliency inference networks in the VFNet enforce consistency. Aiming at ODIs, we introduce a new form of perturbation, i.e., DropView, to improve the effectiveness of consistency regularization. By randomly dropping out different 360° cubemap viewport features before the auxiliary saliency inference network, the proposed DropView enhances the robustness of the final ODI saliency prediction. To adaptively interact with the equirectangular and different cubemap viewport features according to their contributions, we introduce a Viewport Feature Adaptive Integration (VFAI) module and deploy the VFAI module at different levels in the VFNet to raise the capacity of feature encoding of our VFNet. Compared with state-of-the-art fully-supervised methods, our VFNet with fewer labeled training data achieves competitive performance demonstrated by extensive experiments on two publicly available datasets.