A Contrastive Free Energy-Enhanced Transformer Framework for Efficient Reinforcement Learning

To address the challenges of low training efficiency and unstable performance in multiagent reinforcement learning (MARL), this article proposes a novel efficient MARL algorithm called contrastive free energy-enhanced multiagent Transformer (CFMAT), which integrates upstream perception and downstream decision-making in a joint training framework. The upstream perception module consists of a representation encoder and a prediction model. Specifically, the representation encoder is first learned using contrastive learning to generate a unified and compressed representation of the joint observations. The prediction model is then designed to predict future observation representation sequences, which are subsequently utilized in the downstream decision-making module to enhance the policy network's performance. To ensure the stability of the upstream module's learning process, we propose a novel contrastive free energy loss inspired by active inference (AIF) theory. Additionally, the downstream decision-making module employs a Transformer-based encoder-decoder architecture. The representation encoder is further co-trained with the actor-critic network in the downstream learning process. By integrating the upstream perception and downstream decision-making modules into a joint framework, the training efficiency is improved by enabling efficient learning of the overall observation representation. We evaluate CFMAT on challenging multiagent benchmarks, where experimental results show that it significantly outperforms state-of-the-art baselines in both training efficiency and stable performance across various scenarios.