Mean-Field Multi-Agent Reinforcement Learning for Peer-to-Peer Multi-Energy Trading

With increasing numbers of prosumers employed with multi-energy systems (MES) towards higher energy utilization efficiency, an advanced energy management scheme is becoming increasingly important. The incorporation of MES into the existential energy market holds promise for future power systems. The continuous double auction (CDA) market, in a decentralized manner, makes it ideal for enabling peer-to-peer (P2P) energy trading due to its high transparency and efficiency. However, the CDA market is difficult to model when considering the highly stochastic and dynamic behaviors of market participants. For this reason, we formulate this task as a Decentralized Partially Observed Markov Decision Process and propose a novel multi-agent reinforcement learning method that allows each prosumer agent to stabilize the training performance with mean-field approximation and also to maintain the scalability and privacy with market public information. Case studies constructed on a real-world scenario of 100 prosumers show that our method captures the economic benefits of the P2P energy trading paradigm without violating the prosumers' privacy, and outperforms the state-of-the-art methods in terms of policy performance, scalability, and computational performance.