A Comprehensive Review of Metaheuristics for Hyperparameter Optimization in Machine Learning

Hyperparameter optimization is a critical step in the development and fine-tuning of machine learning (ML) models. Metaheuristic optimization techniques have gained significant popularity for addressing this challenge due to their ability to search the hyperparameter space efficiently. In this review, we present a detailed analysis of various metaheuristic techniques for hyperparameter optimization in ML, encompassing population-based, single solution-based, and hybrid approaches. We explore the application of metaheuristics in Bayesian optimization and neural architecture search, two prominent areas within the field. Moreover, we provide a comparative analysis of these techniques based on established criteria and evaluate their performance in diverse ML applications. Finally, we discuss future directions and open challenges with special emphasis on the opportunities for improvement in metaheuristics. Other crucial issues like adaptability to new ML paradigms, computational complexity, and scalability issues are also discussed critically. This review aims to provide researchers and practitioners with a comprehensive understanding of the state-of-the-art metaheuristic optimization techniques for hyperparameter tuning, thereby facilitating informed decisions and advancements in the field.