Coarse-to-Fine Depth Super-Resolution With Adaptive RGB-D Feature Attention

Depth maps suffer from multiple kinds of degradation such as noise and low resolution, due to the limitations of sensors. To improve the spatial resolution and quality of depth maps, RGB-D-based depth super-resolution (SR) methods utilize the corresponding color image to provide extra structure information. However, the inconsistency between the color texture and depth structure can lead to texture-copying artifacts if the two kinds of features are fused without selection. In this article, we propose a novel coarse-to-fine framework for RGB-D-based depth SR, which consists of two sub-networks, i.e., CONet for coarse SR and RFNet for refinement. Through the proposed coarse supervision strategy, CONet can alleviate multiple degradations in depth maps and assist with further SR in the refinement stage. Moreover, the branch attention module (BAM) is incorporated in the RFNet to adaptively select important information from RGB-D features and suppress the texture-copying artifact. Additionally, we propose an edge-aware spatial attention module (ESAM) to further locate and restore the depth discontinuity in the fused RGB-D features. Extensive experiments on multiple benchmarks demonstrate that compared to the state-of-the-art methods, the proposed method achieves improved results both quantitatively and qualitatively.