Simulation-based optimization of radiotherapy: Agent-based modeling and reinforcement learning

Abstract Along with surgery and chemotherapy, radiotherapy is an effective way to treat cancer. Many cancer patients take delivery of radiation. The goal of radiotherapy is to destroy the tumor without damaging healthy tissue. Due to the complexity of the procedure, modeling and simulation can be useful for radiotherapy. In this research we propose a new approach to optimize dose calculation in radiotherapy. We consider fix schedule of irradiation and varying the fraction size during the treatment. The proposed approach contains two steps. At the first step, we develop an agent-based simulation of vascular tumor growth based on biological evidences. We consider a multi-scale model in which cellular and subcellular scales are observed. We consider heterogeneity of tumor oxygen diffusion and also the effects of cancer cells hypoxia on radiotherapy. Besides, different radiosensitivity of cells related to their cell-cycle phase is modeled. The agent-based model was implemented in NetLogo package. Based on this model, we simulate different scenarios of radiotherapy. At the second step, we propose an algorithm for the optimization of radiotherapy. Radiation dose and fractionation scheme are considered as two key elements of radiation therapy. To optimize the therapy we apply Q-learning algorithm. Finally, we combine the simulation and optimization compartments together using R-NetLogo package. By tuning the parameters of learning algorithm optimal treatment plans are achieved to cure tumor together with minimum side effects. Our research presents the power of agent-based approach combined with reinforcement learning for simulating and optimizing complex biological problems such as radiotherapy. The proposed modeling approach lets us to study different scenarios of tumor growth and radiotherapy. Furthermore, our optimization algorithm works fast and finds the best treatment plan.