Energy Efficient RIS-Assisted UAV Networks Using Twin Delayed DDPG Technique

Unmanned Aerial Vehicle (UAV) has emerged as a promising technology to provide wireless signals from air to the ground users in specific scenarios such as earthquakes, tsunamis and other disasters. The performance of the UAV is degraded when the signals are blocked by obstacles in dense urban scenarios. To address this issue and enhance the signal quality available to the ground users, Reconfigurable Intelligent Surface (RIS) has emerged as a new technological paradigm. It offers an intelligent configuration for the signal propagation environment by redirecting the signals to the users. In this article, we solve a non-convex optimization problem of RIS-assisted UAV network by jointly optimizing the RIS phase shift and 3D trajectory of UAV to maximize the energy efficiency of a rotatory-wing UAV. The considered optimization problem is solved using Deep Reinforcement Learning (DRL) based techniques in an on-line fashion to reduce the computational complexity. We leverage Twin-delayed Deep Deterministic Policy Gradient (TD3) to solve the problem by considering the UAV trajectory as a set of continuous actions. For comparison, we also use the Soft Actor-Critic (SAC), Deep Deterministic Policy Gradient (DDPG) and Double Deep Q-Network (DDQN) for continuous and discrete optimization of the UAV trajectory, respectively. Extensive simulations show that the TD3 outperforms all the considered DRL techniques with the highest energy efficiency and throughput, and the lowest propulsion energy.