OPT-VAE: A Variational Autoencoder-based Estimation of Distribution Algorithm for Continuous Optimization

Estimation of distribution algorithms (EDAs) face substantial difficulty in navigating complex landscapes efficiently due to predefined prior assumptions. Deep generative model-based EDAs solve this issue by utilizing the deep generative model to construct a flexible distribution for guiding optimization. However, constructing accurate distribution suffers from the relative scarcity of high-quality solutions, leading to inferior performance. To deal with the challenge, this study proposes an optimizer with Variational Autoencoder-based EDA (OPT-VAE), which utilizes the VAE’s advantage of low data requirement to construct distribution for guiding search effectively. OPT-VAE integrates the proposed spatial-temporal tournament selection mechanism along with the novel dual-space exploration mechanism to aid VAE navigating the complex landscapes. Experimental results on the well-known benchmark suites, including BBOB and CEC2013 benchmark, highlight OPT-VAE’s superior adaptability and optimization performance on various complex problems compared to eleven state-of-the-art optimization algorithms.