Dynamic Programming and the Hamilton-Jacobi-Bellman Equation

In this chapter, we turn our attention away from the derivation of necessary and sufficient conditions that can be used to find the optimal time paths of the state, costate, and control variables, and focus on the optimal value function more closely. In particular, we will derive the fundamental first-order partial differential equation obeyed by the optimal value function, known as the Hamilton-Jacobi-Bellman equation. This shift in our attention, moreover, will lead us to a different form for the optimal value of the control vector, namely, the feedback or closed-loop form of the control. This form of the optimal control typically gives the optimal value of the control vector as a function of the current date, the current state, and the parameters of the control problem. In contrast, the form of the optimal control vector derived via the necessary conditions of optimal control theory is termed open-loop , and in general gives the optimal value of the control vector as a function of the independent variable time, the parameters, and the initial and/or terminal values of the planning horizon and the state vector. Essentially, the feedback form of the optimal control is a decision rule, for it gives the optimal value of the control for any current period and any admissible state in the current period that may arise. In contrast, the open-loop form of the optimal control is a curve, for it gives the optimal values of the control as the independent variable time varies over the planning horizon.