Fast Reinforcement Learning for Resource Optimization in Dynamic Vehicular Communications

In recent years, vehicular communications have attached great interests in both academy and industry for its potential of promoting safety and autonomous driving. Unlike classical communication scenarios, in vehicular communications the optimal resource allocation must be accomplished in a real-time manner, in order to maximally reduce the response delay. This presents a substantial challenge for current machine learning based intelligent resource optimization methods which may be sample inefficient, especially when the problem space becomes extremely huge. In this study, we develop a fast reinforcement learning (RL) framework for the real-time resource optimization of vehicular communications, whereby the transmitting power and the accessing frequency channels need to be jointly allocated. The main concept of our new method is that it incorporates a sample efficient structured exploration mechanism in the action space, which firstly ignores the local exploitation but focuses on a randomized global exploration. Thus, our exploration-first method, in contrast to classical exploitation-first RL, can reconstruct the coarse-grained global landscape of a huge Q-table from only the few samples. This learned prior knowledge would remarkably accelerate the convergence of subsequent incremental learning process, by concentrating on the identified attentional subspace of the Q-table. As demonstrated by numerical results, our new method would reduce the time complexity or the response delay by around 10 folds. As such, our fast RL method would have the great potential to such challenging optimization problems whereby the acquisition of massive training samples is time demanding, which hence provides the great promise to the emerging vehicular networks.