UAV Path Planning for Data Gathering in Wireless Sensor Networks: Spatial and Temporal Substate-Based Q-Learning

Unmanned aerial vehicles (UAVs) integrated with wireless sensor networks (WSNs) on the ground have proven to be a reliable and robust solution to a variety of applications, such as wide-area environment monitoring, surveillance, and event tracking systems. However, determining the optimal movement paths of asynchronous UAVs to obtain the highest sensing information value while satisfying energy constraints in unknown environments remains a challenge. In this article, we propose a novel approach called spatial and temporal substate-based $Q$ -learning (STSQL) to acquire time-varying sensing data using UAVs in an unknown environment. Our approach utilizes a $Q$ -learning algorithm, a reinforcement learning technique, to perform work in a 3-D topographic area. A spatial substate is defined as a hexagonal area to ensure disjoined coverage of a UAV, while the temporal substate models the evolution of discrete sensing information based on elapsed time from previous data acquisition. We aim to find the best trajectory for UAVs that maximizes the accumulated value of sensing data while minimizing energy consumption. Hence, we design a multiobjective reward function. Comprehensive experiments demonstrate that the proposed STSQL method outperforms other methods in terms of convergence time and the amount of acquired sensing information.