DGMM: A Deep Learning-Genetic Algorithm Framework for Efficient Lead Optimization in Drug Discovery

Lead optimization in drug discovery faces the dual challenge of maintaining structural diversity while preserving core molecular features and optimizing the balance between biological activity and drug-like properties. To address these challenges, we introduce the Deep Genetic Molecule Modification (DGMM) algorithm, a novel computational framework that synergistically integrates deep learning architectures with genetic algorithms for efficient molecular optimization. DGMM leverages a variational autoencoder (VAE) with an enhanced representation learning strategy that incorporates scaffold constraints during training, significantly improving the latent space organization to balance structural variation with scaffold retention. A multiobjective optimization strategy, combining Monte Carlo search and Markov processes, enables systematic exploration of the trade-offs between drug likeness and target activity. Evaluation results indicate that DGMM achieves state-of-the-art performance in activity optimization, generating structurally diverse, yet pharmacologically relevant compounds. To rigorously establish its utility, we first demonstrated its generalizability through extensive retrospective validation on three diverse targets (CHK1, CDK2, and HDAC8), reproducing their known optimization pathways. Building on this validated generalizability, we deployed DGMM in a prospective campaign, which culminated in the wet-lab discovery of novel ROCK2 inhibitors with a notable 100-fold increase in biological activity. This success establishes DGMM as an effective tool for structural optimization of drug molecules.