Joint Trajectory Control, Frequency Allocation, and Routing for UAV Swarm Networks: A Multi-Agent Deep Reinforcement Learning Approach

Collaborative unmanned aerial vehicle (UAV) swarm networks can effectively execute various emerging missions such as surveillance and communication coverage. However, due to high mobility and constrained transmission range, packet routing encounters mutual interferences, link breakages, and unexpected delays. In such networks, routing performance is coupled with trajectory control, frequency allocation, and relay selection. In this study, we propose a joint trajectory control, frequency allocation, and packet routing (JTFR) algorithm, in which link utility is maximized by considering the link stability, signal-to-interference-plus-noise ratio, queuing delay, and residual energy of UAVs. The proposed JTFR employs adaptive distributed multi-agent deep deterministic policy gradient coupled with the swarming behavior to obtain the optimal solution. For each UAV, an actor network is established by utilizing a long short-term memory-based state representation layer containing two-hop neighbor information to adopt the dynamic time-varying topology. Subsequently, a scalable multi-head attentional critic network is set up to adaptively adjust the actor network policy of each UAV by collaborating with neighbors. The extensive simulation results show that JTFR outperforms existing routing protocols by 30-60% less end-to-end delay, 15-32% better packet delivery ratio, and 20-46% less energy consumption.