Deep plug-and-play and deep unfolding methods for image restoration

Model-based methods and learning-based methods have been the two dominant strategies for solving various image restoration problems in low-level vision. Typically, those two kinds of methods have their respective merits and drawbacks, e.g., model-based methods are flexible for handling different image restoration problems but are usually time-consuming with sophisticated priors for the purpose of good performance; meanwhile, learning-based methods are continuously showing superior effectiveness and efficiency over traditional model-based methods, largely due to the end-to-end training, but generally lack the flexibility to handle different image restoration tasks. In this chapter, we attempt to provide a gentle introduction to deep plug-and-play methods and deep unfolding methods, which have shown great promise by leveraging both learning-based methods and model-based methods. The main idea of deep plug-and-play methods is that, with the aid of variable splitting techniques, a learning-based denoiser can implicitly serve as the image prior for model-based image restoration methods, while the main idea of deep unfolding methods is that, by unfolding the model-based methods via variable splitting algorithms, an end-to-end trainable, iterative network can be obtained by replacing the corresponding subproblems with neural modules. As a result, the deep plug-and-play methods and deep unfolding methods can inherit the flexibility of model-based methods, while maintaining the advantages of learning-based methods. Experimental results on two representative image restoration tasks, including deblurring and superresolution, demonstrate the flexibility and effectiveness of deep plug-and-play methods and deep unfolding methods.