A manifold inexact augmented Lagrangian method for nonsmooth optimization on Riemannian submanifolds in Euclidean space

Abstract We develop a manifold inexact augmented Lagrangian framework to solve a family of nonsmooth optimization problem on Riemannian submanifold embedding in Euclidean space, whose objective function is the sum of a smooth function (but possibly nonconvex) and a nonsmooth convex function in Euclidean space. By utilizing the Moreau envelope, we get a smoothing Riemannian minimization subproblem at each iteration of the proposed method. Consequentially, each iteration subproblem is solved by a Riemannian Barzilai–Borwein gradient method. Theoretically, the convergence to critical point of the proposed method is established under some mild assumptions. Numerical experiments on compressed modes problems in physic and sparse principal component analysis demonstrate that the proposed method is a competitive method compared with some state-of-the-art methods.