Energy Efficient Hyperparameters Tuning through Augmented Gaussian Processes and Multi-information Source optimization

Searching for the optimal values of hyperparameters of a Machine Learning algorithm can be an extremely computationally expensive and awfully energivorous process. This paper proposes multi information source optimization as a strategy to drastically reduce computational time and energy consumed by this search process, where the information sources, with different computational costs, are smaller portions of a large dataset. The algorithm fits into the scheme of Gaussian Process based Bayesian optimization: an Augmented Gaussian Process is proposed, which is trained using only “reliable” information among available sources. A novel acquisition function is also defined according to the Augmented Gaussian Process. Results on the hyperparameter optimization of a Support Vector Machine for a classification task on a large dataset are presented, with a comparison in terms of the misclassification error, on 10 fold-cross validation, and computational cost between the proposed approach and traditional hyperparameter optimization using the entire dataset.