An Accelerated Variance Reduction Stochastic ADMM for Nonconvex Nonsmooth Optimization

The variance reduction technique-based Alternating Direction Method of Multipliers (ADMM) has attracted a lot of attention due to its simplicity and practicability to provide an acceleration characteristic of various machine learning models. Most stochastic ADMM-type methods focus on convex models, however, it is not clear whether accelerated SVRG-ADMM (ASVRG-ADMM) for solving widely used nonconvex models has a similar acceleration feature or convergence rate as in the convex setting. To fill this gap, we consider a general nonconvex nonsmooth optimization problem and study the convergence of ASVRG-ADMM with momentum acceleration. By using a well-defined potential energy function, we establish its sublinear convergence rate. A number of experiments validate that the proposed ASVRG-ADMM performs better than some state-of-the-art algorithms.