Opponent Exploitation Based on Bayesian Strategy Inference and Policy Tracking

In a multi-agent competitive environment, it is important for an agent to detect the opponent's policy and adopt a suitable policy to exploit the opponent. Conventionally, most methods, e.g., Bayesian Policy Reuse (BPR) variants, assume the opponent adopts a fixed policy or a randomly changing policy. In this paper, we make a more realistic and reasonable assumption that the opponent may select its policy based on the previous observation. Here, we define the term "strategy" as the mapping from the previous observation to the opponent's selected policy, and we propose the Bayesian Strategy Inference (BSI) framework to infer the opponent's strategy. Furthermore, to deal with opponents who may randomly select their policies, the BSI framework is combined with an intra-episode policy tracking mechanism to construct the Bayesian Strategy Inference plus Policy Tracking (BSI-PT) algorithm. In our experiments, we design an extended batter vs. pitcher game (EBvPG) for the evaluation of the proposed BSI-PT framework. The experimental results demonstrate that BSI-PT obtains higher policy prediction accuracy and winning percentage than three other BPR variants against the opponents with a specific policy selection strategy, with a random selection strategy, or with a partially random strategy.