Real-Time Joint Regulations of Frequency and Voltage for TSO-DSO Coordination: A Deep Reinforcement Learning-Based Approach

The increasing scale of distributed energy resources (DERs) in the active distribution network (ADN) offers valuable opportunities for distribution system operators (DSOs) to assist transmission system operators (TSOs) in regulating operational issues and reducing overall operational costs. The paper proposes a multi-agent deep reinforcement learning (MADRL)-based TSO-DSO coordination framework for jointly regulating the frequency and voltage measured at the grid supply point (GSP) of the transmission network (TN) with a sufficient utilization of control flexibility of converter-based DERs. To facilitate MADRL, a simple grid partitioning method is employed for the balanced partitioning of ADNs with the connectivity constraint imposed on each sub-region. On this basis, a MADRL algorithm is designed by combining the QMIX and twin delayed deep deterministic policy (TD3) algorithm for an effective optimization of the decentralized regulation scheme. The proposed QMIX-TD3 is equipped with the graph convolutional network (GCN) for its temporal-spatial learning ability to tackle the challenges of the complicated transmission and distribution (T/D) system dynamics. In addition, a policy smooth regularization (PSR) loss is proposed to damp action oscillations and enhance sample efficiency. Experiments on the integrated T/D system demonstrate that the proposed framework can effectively mitigate the impact of system disturbances and therefore benefit the system operation.