Heterogeneous optimal formation control of nonlinear multi-agent systems with unknown dynamics by safe reinforcement learning

This article presents the problem of distributed training with a decentralized execution policy as a safe, optimal formation control for a heterogeneous nonlinear multi-agent system. The control objective is to guarantee safety while achieving optimal performance. This objective is achieved by introducing novel distributed optimization problems with cost and local control barrier functions (CBFs). Designing an optimal formation controller is defined as optimal performance and modeled by a cost function. A local CBF trains a safe controller to ensure multi-agent systems operate within the safe regions. Instead of optimizing constrained optimization problems, this method generates safe, optimal controllers from unconstrained optimization problems by utilizing local CBFs. As a result, the presented approach has a lower computational cost than constrained optimization problems. It is proven that the proposed controller's optimality and stability are not affected by adding the local CBF to the cost function. A safe, optimal policy is iteratively derived using a new off-policy multi-agent reinforcement learning (MARL) algorithm that does not need knowledge of the agents' dynamics. Finally, the effectiveness of the proposed algorithm is evaluated through simulation of the collision-free problem of the multi-quadrotor formation control.