A nonmonotone proximal point algorithm for nonconvex regularized optimization with box constraints

In this paper, we design a nonmonotone proximal point algorithm for a class of nonconvex sparsity-promoting penalties with box constraints, which include smoothly clipped absolute deviation (SCAD), capped ℓ1 (CAP) and minimax concavity penalty (MCP). The new algorithm iteratively solves a proximal operator problem, which in turn utilizes a closed form solution of SCAD, CAP and MCP penalties with box constraints. To accelerate the convergence of the algorithm, a nonmonotone line search strategy is used. We verify that any accumulation point of the sequence generated by the algorithm is a critical point. Furthermore, we prove that the worst-case iteration complexity for finding an ϵ scaled first-order stationary point is O(ϵ−2). The numerical experiments on various synthetic and real data demonstrate the efficiency of the proposed algorithm.