Observer-Based Multi-Agent Reinforcement Learning for Pursuit-Evasion Game With Multiple Unknown Uncertainties

This paper aims to investigate the challenging problem of a multi-agent game with multiple pursuers and a single evader in an environment with multiple unknown uncertainties. A coupled approach combining decentralized observers and reinforcement learning (RL) controllers is proposed to deal with this scenario. Firstly, decentralized observers driven by auxiliary control laws are introduced to estimate the states of uncertain systems, with their best responses obtained through the adaptive dynamic programming (ADP) method. The estimated states, which reflect the actual states of the pursuers’ systems, are concurrently transmitted to the RL controllers. Subsequently, the controllers are trained with observer-based heterogeneous-agent proximal policy optimization (OHAPPO) algorithm, in which a novel global multi-function cost is designed. The algorithm utilizes the advantage decomposition for policy updates in the way of credit assignment, resulting in more stable and efficient updates compared to traditional value decomposition. Moreover, to further enhance the performance of both observers and controllers, a sequential game is established between them, where observers’ policies are influenced by controllers’ optimal control and vice versa. Finally, the simulation results verify the effectiveness of the designed OHAPPO algorithm in the pursuit-evasion game.