Deep Reinforcement Learning for Secure and Energy-Efficient RIS-Assisted UAV Networks under Imperfect CSI

The increasing complexity of urban environments has amplified the demand for secure, reliable, and energy-efficient wireless communications. Traditional terrestrial base stations frequently face challenges in dense urban areas, mainly due to recurring non-line-of-sight (NLOS) scenarios, intense interference, and high susceptibility to eavesdropping. Unmanned aerial vehicles (UAVs) have stood out as a valuable supplementary option, thanks to their ability to support flexible deployment and reliable line-of-sight (LOS) connections. Meanwhile, millimeter-wave (mmWave) bands provide abundant spectrum to satisfy the growing capacity demands of urban networks, but remain highly susceptible to path loss, blockages, and beam misalignment, limiting their effectiveness in such environments. As a cost-efficient and energy-conscious option, reconfigurable intelligent surfaces (RIS) work by smartly reconfiguring wireless reflections to recover blocked links, expand coverage range, and boost secrecy performance. This paper investigates a RIS-aided UAV mmWave communication framework engineered to jointly address energy efficiency and physical layer security under potential eavesdropping threats. A deep reinforcement learning (DRL) strategy is formulated, where the UAV agent interacts with the dynamic urban environment based on imperfect channel state information (CSI) and local position feedback. The agent jointly optimizes UAV active beamforming, RIS phase shifts, and UAV trajectory in real time. To mitigate estimation bias, a softmax operator is incorporated into the learning process. Simulation findings demonstrate that the method put forward in this study outperforms baseline solutions relying on DRL. Specifically, it alleviates both overestimation and underestimation issues while significantly improving secrecy capacity and secure energy efficiency in RIS-assisted UAV mmWave systems.