Automatic and Simultaneous Adjustment of Learning Rate and Momentum for\n Stochastic Gradient Descent

Stochastic Gradient Descent (SGD) methods are prominent for training machine\nlearning and deep learning models. The performance of these techniques depends\non their hyperparameter tuning over time and varies for different models and\nproblems. Manual adjustment of hyperparameters is very costly and\ntime-consuming, and even if done correctly, it lacks theoretical justification\nwhich inevitably leads to "rule of thumb" settings. In this paper, we propose a\ngeneric approach that utilizes the statistics of an unbiased gradient estimator\nto automatically and simultaneously adjust two paramount hyperparameters: the\nlearning rate and momentum. We deploy the proposed general technique for\nvarious SGD methods to train Convolutional Neural Networks (CNN's). The results\nmatch the performance of the best settings obtained through an exhaustive\nsearch and therefore, removes the need for a tedious manual tuning.\n