Graph Deep Reinforcement Learning for Multi-Cycle Queuing and Scheduling in Deterministic Networking

Deterministic networking (DetNet) offers guaranteed transmission services for critical real-time applications, such as industrial automation and intelligent transport systems. It is challenging to fully utilise link resources of different rates to achieve deterministic scheduling in real-world network scenarios with heterogeneous link rates (e.g., hierarchical networks). The cycle specified queuing and forwarding (CSQF) is a typical approach to provide deterministic end-to-end delay in DetNet. However, to achieve deterministic scheduling, the CSQF mechanism sets the same cycle length for both high-speed links and low speed links, resulting in a significant waste of high-speed link resources. To address this issue, we propose a multi-cycle CSQF (MCCSQF) mechanism for multi-link rate networks to reduce queuing delay during high-speed link scheduling and consequently leading to a lower end-to-end flow latency. Furthermore, to fully exploit the exploration and decision-making capabilities of deep reinforcement learning (DRL) in complex environments, we design a DRL framework to achieve deterministic flow routing and low-latency scheduling in MCCSQF. However, DRL algorithms are not capable of fully utilizing network topology information for decision making. We, therefore introduce a graph DRL (GDRL) algorithm- incorporating graph convolution into DRL to extract topological spatial features of network links. Our numerical evaluation results from various network scenarios with different topologies and multiple link rates demonstrate that our proposed GDRL outperforms DRL in flow scheduling while MCCSQF effectively reduces the end-to-end delay compared to CSQF.