Constrained Reinforcement Learning for Predictive Control in Real-Time Stochastic Dynamic Optimal Power Flow

Deep Reinforcement Learning (DRL) has emerged as a favored approach for resolving control challenges in power systems. Traditional DRL guides the agent through exploration of numerous policies, each embedded within a neural network (NN), aiming to maximize the associated reward function. However, this approach can lead to infeasible solutions that violate physical constraints such as power flow equations, voltage limits, and dynamic constraints. Ensuring these constraints are met is crucial in power systems, as they are a safety critical infrastructure. To address this issue, existing DRL algorithms remedy the problem by projecting the actions onto the feasible set, which can result in sub-optimal solutions. This paper introduces a pioneering primal-dual approach to learn optimal constrained DRL policies specifically for predictive control in real-time stochastic dynamic optimal power flow. The focus is on controlling power generations and battery outputs while ensuring compliance with critical constraints. We also prove the convergence of the critic and actor networks. Our case studies, based on IEEE standard systems, underscore the preeminence of the proposed approach in identifying near-optimal actions for various states while concurrently adhering to safety constraints.