Curriculum Learning-Based Multi-Agent Reinforcement Learning for Multi-UAV Cooperative Encirclement

The advancements in machine learning and autonomous control have significantly enhanced the intelligence and coordination of unmanned aerial vehicles (UAVs), enabling more efficient and fault-tolerant encirclement for dynamic and moving targets. However, traditional multi-UAV cooperative encirclement approaches heavily rely on preset encirclement points, frequently yielding unrealizable formation states due to their poor adaptability in complex dynamic environments. This paper focuses on the cooperative encirclement problem of UAVs, and tries to propose a curriculum learning-based multi-agent reinforcement learning framework to provide reasonable flight paths for UAVs and achieve an effective encirclement of moving targets with adaptively-generated encirclement points. First, with the models of UAVs and obstacles, we analyze the collision avoidance, motion continuity, and position adjustment constraints of UAVs, and formulate the multi-UAV cooperative target encirclement problem as a multi-constraint combinatorial optimization one. Then, by mimicking the way that humans learn with curriculum, we develop a multi-agent reinforcement learning approach to dynamically adjust reward values in different stages of the training process and efficiently carry out the encirclement mission by quickly providing a good enough flight path for each UAV. Simulation experiments with randomly-generated static and dynamic obstacles are conducted to evaluate the performance of our approach, and the experimental results demonstrate that our approach has a better performance in terms of average reward, encirclement success rate, and task completion time.