Reinforcement learning of robust active flow control strategy with surrogate-based hybrid environment

This article introduces a training framework for active flow control (AFC) based on deep reinforcement learning (DRL) called PyFAC, which employs the Soft Actor–Critic algorithm as the DRL method. To reduce computational resource consumption and improve training efficiency, a Long Short-Term Memory (LSTM) model is incorporated as a surrogate environment for computational fluid dynamics (CFD). Additionally, a free-stream turbulence (FST) model is introduced to validate the robustness of the trained AFC agent under disturbing incoming flow conditions. Under the condition without incoming flow disturbance, the agent gains 3.16% extra lift compared with the baseline, which is not stimulated by AFC, and 28.3% improvement of control performance compared with ordinary control strategies without the involvement of DRL. It is observed that an agent trained in a plain condition without FST struggles to perform effectively in an FST environment due to the disturbances from the incoming flow. Conversely, introducing FST perturbations into the training environment allows for the development of a robust agent. Subsequently, the LSTM model is integrated as the CFD surrogate. The LSTM demonstrates high accuracy, with discrepancies compared to CFD remaining below 10−3. Agents trained in the LSTM surrogate environment producing control strategies comparable to those of the model-free agents and achieving nearly identical control performances. A comparative analysis of training times for LSTM-based and model-free DRL approaches is also conducted, revealing a 60% improvement in training efficiency with the LSTM-based method.