An adaptive heuristic algorithm based on reinforcement learning for ship scheduling optimization problem

Due to the development of ship sizes and the traffic increase in port, ships having long turnaround time in port often result in port congestion, which seriously affects the efficiency of the ship navigation and environmental sustainability of port, it has been evident that effective ship scheduling presents a solution of the fundamental and strategic importance to port congestion. In this paper, a mixed-integer linear programming mathematical model is proposed to realize the optimization of the ship scheduling in port to minimize the total time spent by ships in port. Its methodological novelty is gained by an innovative adaptive genetic simulated annealing algorithm based on a reinforcement learning algorithm (GSAA-RL) to support the developed mathematical model, in which the genetic algorithm is considered as the basic optimization algorithm, and Q-learning with a unique property of selecting suitable parameters dynamically is developed to adjust the parameters of crossover and mutation to improve the search ability of the algorithm. Meanwhile, the dynamic parameter turning process is formulated into a Markov decision process (MDP) model with well defining the state, action, and reward function in GSAA-RL. Specifically, the state sets are proposed by analyzing the key factors affecting the scheduling efficiency and a new reward mechanism that can reduce the objective value significantly based on the quality of selected parameters is designed. The annealing operation is performed on some excellent individuals to further expand the search scope. Simulation experiments demonstrate that the proposed GSAA-RL algorithm can significantly shorten the total time spent by ships in port compared to existing approaches. This study hence helps port operators/planners to improve operational efficiency and reduce port congestion, reduce ship fuel consumption, and deliver goods to cargo owners in a timely manner, which has important practical significance for achieving the “dual carbon” goal.