Online Altitude Control and Scheduling Policy for Minimizing AoI in UAV-assisted IoT Wireless Networks

This article considers unmanned aerial vehicle (UAV) assisted Internet of Things (IoT) networks, where low resource IoT devices periodically sample a stochastic process and need to upload more recent information to a Base Station (BS). Among the myriad of applications, there is a need for timely delivery of data (for example, status-updates) before the data becomes outdated and loses its value. Since transmission capabilities of IoT devices are limited, it may not always be feasible to transmit over one hop transmission to the BS. To address this challenge, UAVs with virtual queues are deployed as middle layer between IoT devices and the BS to relay recent information over unreliable channels. In the absence of channel conditions, the optimal online scheduling policy is investigated as well as dynamic UAV altitude control that maintains a fresh status of information at the BS. The objective of this paper is to minimize the Expected Weighted Sum Age of Information (EWSA) for IoT devices. First, the problem is formulated as an optimization problem that is however generally hard to solve. Second, an online model free Deep Reinforcement Learning (DRL) is proposed, where the deployed UAV obtains instantaneous channel state information (CSI) in real time along with any adjustment to its deployment altitude. Third, we formulate the online problem as a Markov Decision Process (MDP) and Proximal Policy Optimization (PPO) algorithm, which is a highly stable state-of-the-art DRL algorithm, is leveraged to solve the formulated problem. Finally, extensive simulations are conducted to verify findings and comprehensive comparisons with other baseline approaches are provided to demonstrate the effectiveness of the proposed design.