A Q-Learning-Based Hierarchical Routing Protocol With Unequal Clustering for Underwater Acoustic Sensor Networks

Underwater acoustic sensor networks (UASNs) have emerged as a viable networking approach due to their numerous aquatic applications in recent years. As a vital component of UASNs, routing protocols are essential for ensuring reliable data transmissions and extending the longevity of UASNs. Recently, several clustering-based routing protocols have been proposed to reduce energy consumption and overcome the resource constraints of deployed sensor nodes. However, they rarely consider the hot-spots' problem and the sink node isolation problem in the multihop underwater sensor networks. In this article, we propose a Q-learning-based hierarchical routing protocol with unequal clustering (QHUC) for determining an effective data forwarding path to extend the lifespan of UASNs. First, a hierarchical network structure is constructed for initialization. Then, a combination of unequal clustering and the Q-learning algorithm is applied to the hierarchical structure to disperse the remaining energy more evenly throughout the network. With the use of the Q-learning algorithm, a global optimal CH and next-hop can be determined better than a greedy one. In addition, the ${Q}$ value that guarantees the optimal routing decisions can be computed without incurring any additional costs by combining the Q-learning algorithm with clustering. The simulation results show that the QHUC can achieve efficient routing and prolong the network lifetime significantly.