Towards Beam Hopping and Power Allocation in Multi-Beam Satellite Systems With Parameterized Reinforcement Learning

The simultaneous optimisation of beam hopping and power allocation is a crucial technique for enhancing the performance of Multi-Beam Satellite (MBS) systems. However, the previous joint optimisation approaches cannot well handle with the issues of high-dimensional state space and discrete-continuous hybrid action space. In this paper, we propose a joint optimization approach based on parameterized reinforcement learning to simultaneously regulate beam hopping and power allocation for MBS systems (called DeepMBS). In DeepMBS, a multi-objective problem is firstly formulated to optimize system throughput and energy efficiency. Then, the optimization problem is modelled as a Markov Decision Process (MDP), and the original deep Q-network is extended with a parameterized action space to simultaneously determine the beam hopping (discrete action) and power allocation (continuous action). In addition, we design an empirical filtering mechanism to enhance the performance of DeepMBS. Finally, the results of extensive experiments demonstrate that the proposed DeepMBS can gain a better performance in terms of throughput and energy efficiency compared to the baseline algorithms. Furthermore, the proposed DeepMBS (EFM) algorithm demonstrates superior accuracy and sensitivity in capturing changes of communication demands.