Double Deep $Q$ -Learning-Based Distributed Operation of Battery Energy Storage System Considering Uncertainties

Q-learning-based operation strategies are being recently applied for optimal operation of energy storage systems, where, a Q-table is used to store Q-values for all possible state-action pairs. However, Q-learning faces challenges when it comes to large state space problems, i.e., continuous state space problems or problems with environment uncertainties. In order to address the limitations of Q-learning, this paper proposes a distributed operation strategy using double deep Q-learning method. It is applied to managing the operation of a community battery energy storage system (CBESS) in a microgrid system. In contrast to Q-learning, the proposed operation strategy is capable of dealing with uncertainties in the system in both grid-connected and islanded modes. This is due to the utilization of a deep neural network as a function approximator to estimate the Q-values. Moreover, the proposed method can mitigate the overestimation that is the major drawback of the standard deep Q-learning. The proposed method trains the model faster by decoupling the selection and evaluation processes. Finally, the performance of the proposed double deep Q-learning-based operation method is evaluated by comparing its results with a centralized approach-based operation.