Multi-Market Bidding Behavior Analysis of Energy Storage System Based on Inverse Reinforcement Learning

The bidding behaviors of the energy storage systems (ESS) are complicated due to time coupling and market coupling limited by their capacity states. The existing research is mainly based on optimization models and reinforcement learning (RL) models, which are idealized with analytical objective functions, rational decisions, and virtual historical data. This leads to mismatches between theoretical results and actual bidding behaviors, and the obtained results cannot be generalized to a real-world market. In recent years, the disclosure of market data has enabled data-driven analysis, while the complexity of actual data makes the analysis process challenging and calls for more suitable methods. On the basis of our investigation of ESS bidding behaviors and market data, we propose a novel inverse RL (IRL)-based framework to identify the bidding decision objective function of ESS in coupled multi-market through their historical bidding records and operation status. The identification results can be used as the model parameters to help RL-based simulation models available for a real-world market. Based on the proposed framework, we conduct an empirical analysis on the real-world data of a battery ESS in the Australian electricity market. The results show how the ESS balances its objective among different markets and maintains its state of charge (SOC) to gain more profits. This framework can help us model ESS behaviors by relying on historical data to better understand the ESS bidding decision-making mechanism, which will facilitate market equilibrium analysis and mechanism design.