A Reinforcement Learning Approach to CAV and Intersection Control for Energy Efficiency

The advent of intelligent Connected Autonomous Vehicles (CAVs), smart traffic control infrastructure, and Vehicle-to-Infrastructure (V2I) communication provide many opportunities to increase energy efficiency while minimizing the waiting time at signalled intersections, even though only some vehicles are CAVs and the others are human-driven. Unlike the traditional approaches using Model Predictive Control (MPC) that rely on manually designing speed control strategies, this paper proposes a reinforcement learning-based method, named Energy Efficiency Intersection Control and CAVs' speed control (E²-ICCAV). More specifically, the method consists of a manager module for the intersection and a worker module for each CAV. The manager module focuses on learning about the phase selections of the intersection to minimize all vehicles' waiting time and energy consumption. To achieve this end, we adopt an options framework for the manager module and empower it to choose reasonable phases and maintain them for a reasonable period. In the worker module, we regard each CAV as a worker and formulate CAVs' speed control problem as a Markov game. Each worker aims to minimize both hard accelerations and braking, while always keeping a safe distance from the leading vehicle. The evaluation results show that the proposed E²-ICCAV method can achieve the shortest waiting time and queue length as well as the fewest hard accelerations and braking. In addition, we have developed an infrastructure-assisted eco-drive automated driving system using a real-world test bed which is a proof-of-concept for large scale intersection control in smart cities.