Quality Enhancement of Compressed 360-Degree Videos Using Viewport-based Deep Neural Networks

360-degree video provides omnidirectional views by a bounding sphere, thus also called omnidirectional video. For omnidirectional video, people can only see specific content in the viewport through head movement, i.e., only a small portion of the 360-degree content is exposed at a given time. Therefore, the viewport quality is of particular importance for 360-degree videos. In this article, we propose a quality enhancement of compressed 360-degree videos using viewport-based deep neural networks, named V-DNN. V-DNN is mainly composed of two modules: viewport prediction network (VPN) and viewport quality enhancement network (VQEN). VPN based on spherical convolution and 2D convolution generates potential viewports for omnidirectional video. VQEN takes the current viewport and its reference viewports as the input and enhances residual for the current viewport based on bidirectional offset prediction and Spatio-temporal deformable convolutions. Compared with HM16.16 baseline at QP = 37 under the Low Delay P (LDP) configuration, experimental results show that V-DNN achieves an average 0.605 dB and 0.0139 gains in viewport-based ΔPSNR and ΔMS-SSIM, respectively, and is 0.379 dB (59.63%) and 0.0073 (110.61%) higher than the multi-frame quality enhancement (MFQE-2.0) scheme at QP = 37, respectively. Moreover, V-DNN consistently outperforms MFQE-1.0, MFQE-2.0, and HM16.16 baseline at the other QPs in terms of ΔPSNR, ΔWS-PSNR, and ΔMS-SSIM.