Multi-Agent Deep Reinforcement Learning for Recharging-Considered Vehicle Scheduling Problem in Container Terminals

The worldwide popularity of electric automated guided vehicles and autonomous vehicles in container terminals requires efficient vehicle scheduling management considering the capacitated battery energy and the charging station limitation. In this paper, the recharging-considered vehicle scheduling problem is formulated as a decentralized partially observable Markov decision process to maximize the cumulative reward, providing a highly adaptive decision-making mechanism for multi-agent-powered terminal transport system by supporting decentralized decisions and accommodating partial observability. Considering the limited number of charging stations and tight schedules, a novel scheduling method based on the actor-critic multi-agent deep reinforcement learning framework is developed to facilitate cooperation among vehicles and charging stations and enhance the stability and efficiency of the learning process. Furthermore, to address the challenge on algorithm convergence due to the vast state space, we employ a heterogeneous graph neural network in the proposed framework for feature extraction and a multi-agent proximal policy optimization algorithm for parameter training. Numerical results indicate that the proposed method outperforms the distributed-agent deep reinforcement learning and several benchmark heuristics, showcasing its superior performance in both solution quality and efficiency. Moreover, the well-trained model can be directly applied to various scenarios, demonstrating its high generalization capability.