Hierarchical Reinforcement Learning in Multi-Domain Elastic Optical Networks to Realize Joint RMSA

To improve the network scalability, a large elastic optical network is typically segmented into multiple autonomous domains, where each domain possesses high autonomy and privacy. This architecture is referred to as the multi-domain elastic optical network (MDEON). In the MDEON, the routing, modulation, and spectrum allocation (RMSA) for the inter-domain service requests are challenging. As a result, deep reinforcement learning (DRL) has been introduced recently where the RMSA policies are learned during the interaction of the DRL agents with the MDEON environment. Due to the autonomy of each domain in the MDEON, joint RMSA is essential to improve the overall performance. To realize the joint RMSA, we propose a hierarchical reinforcement learning (HRL) framework which consists of a high-level DRL module and multiple low-level DRL modules (one for each domain), with the collaboration of DRL modules. More specifically, for the inter-domain service requests, the high-level module obtains some abstracted information from the low-level DRL modules and generates the inter-domain RMSA decision for the low-level modules. Then the low-level DRL module gives the intra-domain RMSA decision and feeds back to the high-level module. The proposed HRL framework preserves the autonomy of each single domain while delivering effective overall network performance through the cooperation of the high-level and low-level DRL modules. Simulation results demonstrate that our proposed method outperforms previous approaches.