5. On Automatic Differentiation of Computer Programs

Using the techniques presented in Chapter 3, one can program the algebraic sensitivity analysis in optimal sizing and shape design problems with a reasonable amount of work. In the papers [HM92], [MT94], and [TMH98], algebraic sensitivity analysis was performed by hand for quite complicated shape optimization problems. However, full hand coding of derivatives requires a lot of time. This is not acceptable when solving industrial design problems.Automatic differentiation (AD) is a technique for augmenting computer programs with derivative computations [Gri89]. It exploits the fact that every computer program executes a sequence of elementary arithmetic operations. By applying the chain rule of differential calculus repeatedly to these operations, accurate derivatives of arbitrary order can be computed automatically.AD is now under active study and is used in many applications in engineering [BCG+92], [MMRS96] and mathematical physics [KL91], [IG00].5.1 Introduction to automatic differentiation of programsAD of computer programs is still sometimes confused with finite difference approximations of derivatives or symbolic differentiation of a single expression using packages like Mathematica [Wol99].It is always possible to approximate partial derivatives of a smooth function f : ℝn → ℝ by using simple divided difference approximations such as the forward difference approximation∂ƒ(x0)∂xi≈ƒ(x0+δe(i))−ƒ(x0)δ.5.1Here e(i) is the ith unit vector and δ > 0 is the step length. It is well known that this approach leads to the first order approximation of the partial derivatives. Computation of approximate derivatives in this way has the advantage that one needs f only as a "black box." There are, however, two disadvantages to this approach: first, the accuracy of this approximation is difficult to estimate.