Dual-Centralized Q-Network-Based Reinforcement Learning for Cooperative Path Planning of Multiple UAVs

Due to the low cost and high maneuverability, unmanned aerial vehicles (UAVs) have been commonly used and played an important role in both the civilian and military fields. Although UAVs can significantly achieve enhanced flexibility and extensibility for large-scale intelligent systems, they result in a serious path planning problem. Especially in complex environments with a large number of irregular obstacles, UAVs have to efficiently find near-optimization flight paths and automatically move to target positions to finish the group task while avoiding collisions and satisfying various constraints. In this work, we focus on the cooperative path planning problem of homogeneous UAVs and present a multi-agent reinforcement learning-based approach to solve the problem. First, with the UAV and obstacle models, we analyse the collision avoidance, motion continuity, and energy consumption constraints in UAV flying, and formulate the cooperative path planning problem as a multi-constraint combinatorial optimization one with a high computational complexity. Then, inspired by the twin delayed deep deterministic policy gradient algorithm where clipped dual Q-networks are used to decrease the overestimation error of critic networks, we propose a multi-agent reinforcement learning-based approach with a dual-centralized Q-network mechanism to automatically produce feasible and collision-free flight path for each UAV. Finally, simulation experiments are conducted in a multi-agent particle environment to evaluate the effectiveness and efficiency of the proposed approach.