Dynamic Beam Hopping Method Based on Multi-Objective Deep Reinforcement Learning for Next Generation Satellite Broadband Systems

When regarding the inherent uncertainty of differentiated services requirements as well as the non-uniform spatial distribution of capacity requests, it is essential to flexibility adjust resources of the satellite to satisfy the different conditions. How to match the system capacity demand with efficient utilization of beam is a brand-new challenge. The convention beam hopping methods ignores the intrinsic correlation between decisions, do not consider the long-term reward, and only achieve the optimal solution at the current time. Therefore, system complexity increases significantly as the increase of the demand for differentiated services or beam number. This paper investigates the optimal policy for beam hopping in DVB-S2X satellite with multiple purposes of assuring the fairness of each beam services, minimizing the delay of real-time services transmission, and maximizing the throughput of non-instant services transmission. Since wireless channel conditions, differentiated services arrival rates have stochastic properties, and the multi-beam satellite environment's dynamics are unknown, the model-free multi-objective deep reinforcement learning approach is used to learn the optimal policy through interactions with the situation. To solve the problem with action dimensional disaster, a novel multi-action selection method based on a Double-Loop Learning (DLL) is proposed. Moreover, the multi-dimensional state is reformulated and obtained by the deep neural network. Under realistic conditions achieving evaluation results demonstrate that the proposed method can pursue multiple objectives simultaneously, and it can also allocate resource intelligently adapting to the user requirements and channel conditions.