Transformer-Enhanced Reinforcement Learning for Autonomous UAV Navigation in Unknown Environments

In unknown environments, autonomous navigation of unmanned aerial vehicle (UAV) based on local observations is prone to getting trapped in local optima or collisions. To address this issue, this paper proposes an end-to-end autonomous navigation method that combines Transformer with Soft Actor-Critic (SAC). This method utilizes the powerful sequence modeling capability of Transformer to integrate historical observation data for extracting global environmental features, thereby enhancing the UAV's understanding of the environment. Simultaneously, it leverages the maximum entropy strategy of SAC to enable efficient exploration and decision-making within continuous action spaces, thus improving the robustness and stability of the navigation strategy. Furthermore, a non-sparse multi-level reward function is designed to alleviate the convergence difficulties commonly encountered in reinforcement learning under sparse reward scenarios. Experiments conducted in complex simulation environments built with AirSim and UE4 validate the effectiveness of the proposed method, demonstrating an approximately 12 % increase in navigation success rate and a roughly 15 % reduction in average flight distance compared to traditional methods.