Worst-Case Soft Actor-Critic-Based Safe Reinforcement Learning Method for Nonlinear Constrained Waterflood Reservoir Production Optimization

Summary In the process of waterflood development in oil and gas reservoirs, the injection/production ratio is one of the key indicators affecting both the development effectiveness and economic benefits. A reasonable injection/production ratio directly impacts the development parameters, such as reservoir pressure maintenance levels and water-cut rise rates, and it helps maintain high production capacity in the oil field, ultimately achieving a higher recovery factor. However, existing constrained-waterflood reservoir production optimization methods often face challenges such as time-consuming calculations and the risk of getting trapped in local optima when dealing with such nonlinear constraint problems, making it difficult to effectively achieve a balance between constraints and economic benefits. To address this issue, we propose an injection/production ratio–constrained production optimization method based on the worst-case soft actor-critic (WCSAC) algorithm. This method effectively handles the constraint issue by introducing a safety critic, ensuring the injection/production ratio constraint is met, and maximizing economic benefits throughout the entire life cycle. Specifically, this method models the injection/production ratio–constrained production optimization problem as a constrained Markov decision process (CMDP). On the one hand, it uses conditional value at risk (CVaR) as a safety metric to assess the safety risks of the policy and achieves a balance between reward and safety by adjusting adaptive safety weights. On the other hand, it uses the maximum entropy mechanism to enhance policy exploration and improve the efficiency of global optimization. This results in the training of a stable and reliable safe reinforcement learning (RL) agent to achieve injection/production ratio–constrained production optimization. The agent can adjust and determine the production optimization plan that satisfies the injection/production ratio constraint in real time based on the current reservoir development status, without the need for retraining, thus reducing computational time. To validate the performance of the proposed method, tests were conducted in two reservoir models. The analysis results indicate that, compared with other optimization methods, the proposed method can achieve a higher net present value (NPV) while satisfying the injection/production ratio constraints, demonstrating good model applicability.