On Automating Hyperparameter Optimization for Deep Learning Applications

Given a large amount of data and appropriate hyperparameters, deep learning techniques can deliver impressive performance if several challenging issues with training, such as vanishing gradients, can be overcome. Often, deep learning training techniques produce suboptimal results because the parameter search space is large and populated with many less-than-ideal solutions. Automatic hyperparameter tuning algorithms, known as autotuners, offer an attractive alternative for automating the training process, though they can be computationally expensive. Additionally, autotuners democratize state-of-the-art machine learning approaches and increase the accessibility of deep learning technology to different scientific communities and novice users. In this paper, we investigate the efficacy of autotuning using Keras Tuner on both synthetic and real-world datasets. We show that autotuning performed well on synthetic datasets but was inadequate on real data. As we increase model complexity, autotuning produces errors that are tedious to resolve for those with limited experience in machine learning. Avoiding overfitting, for example, requires extensive knowledge of an algorithm's unique characteristics (e.g., adding dropout layers). Autotuning tools are excellent for creating baseline models on new datasets, but they need more attention to formulate optimal solutions for end-users with less background in deep learning. Because of this, manual tuning based on domain knowledge and experience is still preferred in machine learning because it produces better performance, even though it requires extensive machine learning expertise.