Safe Deep Reinforcement Learning-Based Constrained Optimal Control Scheme for HEV Energy Management

Considering physical constraints in online optimization and training safety is a challenge for the implementation of the deep reinforcement learning (DRL) algorithm. Especially for the nonlinear system, the mapping relationship between the output action of the agent and the control signals is difficult to obtain. This article proposes a novel DRL framework for online optimization in energy management of a power-split hybrid electric vehicle (HEV), which combines a neural network (NN)-based multiconstraints optimal strategy and a rule-based-restraints system (RBRS). The proposed method named reward-directed policy optimization (RDPO) adopts the exterior point method (EPM) and curriculum learning (CL) to direct the agent to recognize and avoid irrational control signals and optimize the fuel economy. The energy management strategy (EMS) considering fuel consumption minimization and irrational control signals' avoidance is optimized by training the agent through the world light vehicle test cycle (WLTC). A competitive fuel economy, 4.495 L/100 km, is achieved with no irrational control signals. Based on the online adaptability evaluation conducted, the fuel consumption of the vehicle under the New European Driving Cycle (NEDC) and the China Typical Urban Driving Cycle (CTUDC) has been reduced to 4.113 L/100 km and 3.221 L/100 km, respectively, with no irrational control signals. The superiority in optimization, calculation efficiency, and safety is verified through comparisons with various DRL agents.