Least squares auto-tuning

Least squares auto-tuning automatically finds hyper-parameters in least squares problems that minimize another (true) objective. The least squares tuning optimization problem is non-convex, so it cannot be solved efficiently. This article presents a powerful proximal gradient method for least squares auto-tuning that can be used to find good, if not the best, hyper-parameters for least squares problems. The application of least squares auto-tuning to data fitting is discussed. Numerical experiments on a classification problem using the MNIST dataset demonstrate the effectiveness of the method; it is able to cut the test error of standard least squares in half.