Large-scale machine learning with fast and stable stochastic conjugate gradient

In deterministic optimization, conjugate gradient (CG) type approaches are preferred with a superior convergence rate than the ordinary gradient approaches. The requirement of solving large-scale data, growing exponentially, makes recent works study the effectiveness of the CG-type approaches with stochastic approximation, especially for large-scale machine learning problems. However, it is challenging that how to incorporate the noisy gradients into CG-type approaches. In this paper, we develop a class of fast and robust stochastic conjugate gradient (SCG) type approach via using the stochastic recursive gradient algorithm (SARAH) and the hyper-gradient descent (HD) technique in the mini-batching setting. That the use of the SARAH gradient estimator makes the proposed approaches enjoy the low variance accelerates the convergence rate and saves the gradient complexity of the conventional SCG-type approach. In addition, using HD to determine the learning rate for the SCG-type approach greatly saves the computational burden, comparing with the existing literature that usually works with the line search technique in practice. We rigorously prove that the proposed approach attains a linear convergence rate for strongly convex loss functions and show that its complexity matches modern stochastic optimization approaches. Various experimental results on machine learning problems are provided to demonstrate the property and the effectiveness of the proposed approaches respectively.