RAHE-QL: an improved Q-learning algorithm incorporating reverse ant colony optimization for path planning in unknown environments

Abstract This study addresses key challenges in mobile robot path planning within unknown environments, including low exploration efficiency, sparse rewards, and vulnerability to local optima. We propose an improved Q-learning algorithm enhanced with reverse ant colony optimization (RACO), referred to as RAHE-QL. First, RACO releases ant colonies from target points to conduct pre-exploration, and their pheromone distribution is used to initialize the Q-table, thereby reducing ineffective exploration during the early phase. Second, a double-distance heuristic correction coefficient, integrating Euclidean and Chebyshev distances, is introduced to directly influence Q-value updates, which improves search direction, enhances path smoothness, and mitigates abrupt changes. Third, a dual-parameter adaptive exploration rate and a multi-stage exploration strategy are employed to dynamically balance exploration and exploitation, maintaining diversity in early exploration while accelerating convergence in later stages. Experimental results in complex unknown environments demonstrate that RAHE-QL outperforms traditional Q-learning, ant colony optimization (ACO), particle swarm optimization (PSO), and multiple Deep Q-Network (DQN) variants about convergence velocity, path optimality, smoothness, and the effectiveness of computing. The algorithm effectively avoids local optima, demonstrates strong robustness and real-time performance, and offers a reliable solution for efficient path planning in challenging unknown environments.