Graph-based multi-agent reinforcement learning for large-scale UAVs swarm system control

In this study, a novel graph-embedding technique based on a graph neural network (GNN) is proposed to identify the topology in the motion of a unmanned aerial vehicles (UAV) swarm and quickly obtain local information around each agent. We also propose a model reference reinforcement learning method to learn the potential field function and determine an appropriate strategy for each agent that can satisfy the requirements of collaborative motion and obstacle avoidance for large-scale UAV swarms. First, a new swarm structure is proposed to provide reserved maneuvering space for UAVs during flight. In addition, a method was proposed to encode the obstacle avoidance behavior of multiple UAVs in a continuous space into spatial maps. A graph attention mechanism (GAT) structure based on local information was proposed to obtain dynamic graph information, and each individual output action was obtained according to the current state information. To improve the training effect, this method can restrain the UAV group while maintaining the formation and preventing collisions among the UAV. Second, a new distributed control algorithm based on multi-agent reinforcement learning (MARL) is proposed by learning the potential field function using local information obtained by a GNN. Each individual can repel and cooperate with the target within a short range and attract objects over a long distance. Finally, simulation results demonstrate the effectiveness and superiority of the proposed method, which has great potential for application in online autonomous collaboration.