Intelligent Spectrum Sensing and Access With Partial Observation Based on Hierarchical Multi-Agent Deep Reinforcement Learning

Dynamic spectrum access (DSA) has been regarded as a viable solution to the spectrum shortage problem. To find idle spectrum, partial spectrum sensing could be employed by selecting a suitable sensing window (SW). Since the SW selection determines how many available bands to access, the transmission performance after the access could be used to guide the SW selection. Hence, a sophisticated joint design on spectrum sensing and access is necessary, which, however, is a challenging task when considering the dynamic nature of spectrum environment, and also the mutual impact among different secondary users (SUs). In this paper, we propose a joint partial spectrum sensing and power allocation (PA) scheme to facilitate SUs to make the best decisions on SW and PA to maximize the network throughput with reduced mutual interference. Considering the environmental dynamics and spectrum uncertainty, we develop a viable solution based on hierarchical multi-agent deep reinforcement learning (HMADRL). Our solution enables mutual design with two stages: making each SU learn the best SW and PA strategies autonomously while adapting to the dynamic environment. By using both simulated spectrum data and real spectrum data measured by SAM60-BX, we have demonstrated the effectiveness of our proposed scheme.