Joint Codebook Selection and UE Scheduling for Unlicensed MmWave NR-U/WiGig Coexistence Based on Deep Reinforcement Learning

Unlicensed millimeter-wave (mmWave) communication is a promising technique for the New Radio-based access to Unlicensed spectrum (NR-U) network to guarantee the ever-increasing data rate demand. A critical challenge of NR-U in unlicensed mmWave bands is to maintain equitable and harmonious coexistence with the original Wireless Gigabit (WiGig) network. In this paper, we develop an intelligent joint codebook selection and user equipment (UE) scheduling scheme for mmWave NR-U and WiGig coexistence networks. Specifically, we first formulate the joint problem as a two-time scale system, wherein the codebook selection is performed on the large-time scale whilst the UE scheduling is optimized on the small-time scale. To address the multi-time scale issue, we put forth a new deep reinforcement learning (DRL) algorithm that enables operations on different time scales to benefit each other towards the target system objective, referred to as layered deep Q-network (L-DQN). Thereafter, with the judicious definitions of the state, action, and reward in L-DQN paradigms, we propose the Deep reinforcement learning based CodeBook selection and UE scheduling (DeepCBU) scheme. DeepCBU aims to attain different trade-offs between two conflicting goals, i.e., (i) maximizing the total data rate of NR-U with as little interference to WiGig as possible and (ii) guaranteeing the fairness among UEs, e.g., the quality of service (QoS) requirement of each UE. To fulfill this mission, we modify the conventional deep neural network architecture of DeepCBU by introducing the target branch for each objective. The gist is that different target branches evaluate the contribution of DeepCBU's strategy to different goals, and the decision of DeepCBU is determined by all target branches in a weighted fashion. Simulation results demonstrate that compared with DRL-dirLBT, TS-dirLBT, and TS-DRL schemes, DeepCBU is more Pareto efficient even without any prior network knowledge, e.g., UE mobility, random channel fading, and transmissions of WiGig, in terms of the data rate of NR-U, the data rate of WiGig, and the number of satisfied UEs. Furthermore, DeepCBU is robust to miscellaneous QoS requirement setups.