Secure Communication for UAV-BS Assisted by RIS Based on Deep Reinforcement Learning

Unmanned aerial vehicle (UAV) plays an increasingly vital role in communication networks. However, ensuring the security of UAV communications remains a significant challenge. This study investigates a secure communication system that employs a reconfigurable intelligent surface (RIS) to assist a UAV base station (UAV-BS), while accounting for the threats posed by eavesdropper and jammer. To enhance the security of UAV-BS communications under imperfect channel state information, we formulate the system as a Markov decision process and propose a method that jointly optimizes the UAV-BS beamforming matrix, trajectory, and RIS phase shift matrix. The coupling of these optimization variables complicates the solution of the nonconvex optimization problem. To address this, we propose the mean target Q-value hierarchical twin delayed deep deterministic policy gradient (MTQ-HTD3) algorithm. The algorithm decomposes the problem into two subproblems using hierarchical reinforcement learning: one focuses on optimizing the UAV-BS beamforming matrix and RIS phase shift matrix, while the other optimizes the UAV-BS trajectory. Additionally, we employ the mean value of multiple target Q-values in the critic network to minimize estimation bias and enhance network stability. Simulation results demonstrate that the proposed MTQ-HTD3 method significantly improves the secrecy rate of the RIS-assisted UAV-BS secure communication system.