Prev Next Index-> contents reference index search external Up-> CppAD Appendix Theory ReverseTheory CppAD-> Install Introduction AD ADFun preprocessor multi_thread utility ipopt_solve Example speed Appendix Appendix-> Faq directory Theory glossary Bib wish_list whats_new deprecated compare_c numeric_ad addon License Theory-> ForwardTheory ReverseTheory reverse_identity ReverseTheory-> exp_reverse log_reverse sqrt_reverse sin_cos_reverse atan_reverse asin_reverse acos_reverse tan_reverse erf_reverse Headings-> Taylor Notation Binary Operators ---..Addition ---..Subtraction ---..Multiplication ---..Division Standard Math Functions Contents

The Theory of Reverse Mode

Taylor Notation
In Taylor notation, each variable corresponds to a function of a single argument which we denote by t (see Section 10.2 of Evaluating Derivatives ). Here and below $X(t)$, $Y(t)$, and Z(t) are scalar valued functions and the corresponding p-th order Taylor coefficients row vectors are $x$, $y$ and $z$; i.e., $$\begin{array}{lcr} X(t) & = & x^{(0)} + x^{(1)} * t + \cdots + x^{(p)} * t^p + O( t^{p+1} ) \\ Y(t) & = & y^{(0)} + y^{(1)} * t + \cdots + y^{(p)} * t^p + O( t^{p+1} ) \\ Z(t) & = & z^{(0)} + z^{(1)} * t + \cdots + z^{(p)} * t^p + O( t^{p+1} ) \end{array}$$ For the purposes of this discussion, we are given the p-th order Taylor coefficient row vectors $x$, $y$, and $z$. In addition, we are given the partial derivatives of a scalar valued function $$G ( z^{(j)} , \ldots , z^{(0)}, x, y)$$ We need to compute the partial derivatives of the scalar valued function $$H ( z^{(j-1)} , \ldots , z^{(0)}, x, y) = G ( z^{(j)}, z^{(j-1)} , \ldots , z^{(0)}, x , y )$$ where $z^{(j)}$ is expressed as a function of the j-1-th order Taylor coefficient row vector for $Z$ and the vectors $x$, $y$; i.e., $z^{(j)}$ above is a shorthand for $$z^{(j)} ( z^{(j-1)} , \ldots , z^{(0)}, x, y )$$ If we do not provide a formula for a partial derivative of $H$, then that partial derivative has the same value as for the function $G$.

Binary Operators

The forward mode formula for addition is $$z^{(j)} = x^{(j)} + y^{(j)}$$ If follows that for $k = 0 , \ldots , j$ and $l = 0 , \ldots , j-1$ $$\begin{array}{rcl} \D{H}{ x^{(k)} } & = & \D{G}{ x^{(k)} } + \D{G}{ z^{(k)} } \\ \\ \D{H}{ y^{(k)} } & = & \D{G}{ y^{(k)} } + \D{G}{ z^{(k)} } \\ \D{H}{ z^{(l)} } & = & \D{G}{ z^{(l)} } \end{array}$$

Subtraction
The forward mode formula for subtraction is $$z^{(j)} = x^{(j)} - y^{(j)}$$ If follows that for $k = 0 , \ldots , j$ $$\begin{array}{rcl} \D{H}{ x^{(k)} } & = & \D{G}{ x^{(k)} } - \D{G}{ z^{(k)} } \\ \\ \D{H}{ y^{(k)} } & = & \D{G}{ y^{(k)} } - \D{G}{ z^{(k)} } \end{array}$$

Multiplication
The forward mode formula for multiplication is $$z^{(j)} = \sum_{k=0}^j x^{(j-k)} * y^{(k)}$$ If follows that for $k = 0 , \ldots , j$ and $l = 0 , \ldots , j-1$ $$\begin{array}{rcl} \D{H}{ x^{(j-k)} } & = & \D{G}{ x^{(j-k)} } + \sum_{k=0}^j \D{G}{ z^{(j)} } y^{(k)} \\ \D{H}{ y^{(k)} } & = & \D{G}{ y^{(k)} } + \sum_{k=0}^j \D{G}{ z^{(j)} } x^{(j-k)} \end{array}$$

Division
The forward mode formula for division is $$z^{(j)} = \frac{1}{y^{(0)}} \left( x^{(j)} - \sum_{k=1}^j z^{(j-k)} y^{(k)} \right)$$ If follows that for $k = 1 , \ldots , j$ $$\begin{array}{rcl} \D{H}{ x^{(j)} } & = & \D{G}{ x^{(j)} } + \D{G}{ z^{(j)} } \frac{1}{y^{(0)}} \\ \D{H}{ z^{(j-k)} } & = & \D{G}{ z^{(j-k)} } - \D{G}{ z^{(j)} } \frac{1}{y^{(0)}} y^{(k)} \\ \D{H}{ y^{(k)} } & = & \D{G}{ y^{(k)} } - \D{G}{ z^{(j)} } \frac{1}{y^{(0)}} z^{(j-k)} \\ \D{H}{ y^{(0)} } & = & \D{G}{ y^{(0)} } - \D{G}{ z^{(j)} } \frac{1}{y^{(0)}} \frac{1}{y^{(0)}} \left( x^{(j)} - \sum_{k=1}^j z^{(j-k)} y^{(k)} \right) \\ & = & \D{G}{ y^{(0)} } - \D{G}{ z^{(j)} } \frac{1}{y^{(0)}} z^{(j)} \end{array}$$

Standard Math Functions
The standard math functions have only one argument. Hence we are given the partial derivatives of a scalar valued function $$G ( z^{(j)} , \ldots , z^{(0)}, x)$$ We need to compute the partial derivatives of the scalar valued function $$H ( z^{(j-1)} , \ldots , z^{(0)}, x) = G ( z^{(j)}, z^{(j-1)} , \ldots , z^{(0)}, x)$$ where $z^{(j)}$ is expressed as a function of the j-1-th order Taylor coefficient row vector for $Z$ and the vector $x$; i.e., $z^{(j)}$ above is a shorthand for $$z^{(j)} ( z^{(j-1)} , \ldots , z^{(0)}, x )$$

Contents
 exp_reverse Exponential Function Reverse Mode Theory log_reverse Logarithm Function Reverse Mode Theory sqrt_reverse Square Root Function Reverse Mode Theory sin_cos_reverse Trigonometric and Hyperbolic Sine and Cosine Reverse Theory atan_reverse Inverse Tangent and Hyperbolic Tangent Reverse Mode Theory asin_reverse Inverse Sine and Hyperbolic Sine Reverse Mode Theory acos_reverse Inverse Cosine and Hyperbolic Cosine Reverse Mode Theory tan_reverse Tangent and Hyperbolic Tangent Reverse Mode Theory erf_reverse Error Function Reverse Mode Theory

Input File: omh/appendix/theory/reverse_theory.omh