Prev

Next

Index-> contents reference index search external

Up-> CppAD Introduction exp_eps exp_eps_rev2

CppAD-> Install Introduction AD ADFun multi_thread library cppad_ipopt_nlp Example preprocessor Appendix

Introduction-> get_started.cpp exp_2 exp_eps exp_apx_main.cpp

exp_eps-> exp_eps.hpp exp_eps.cpp exp_eps_for0 exp_eps_for1 exp_eps_rev1 exp_eps_for2 exp_eps_rev2 exp_eps_cppad

exp_eps_rev2-> exp_eps_rev2.cpp

Headings-> Purpose Mathematical Form epsilon f_7 Index 7: f_6 Index 6: f_5 Index 5: f_4 Index 4: f_3 Index 3: f_2 Index 2: f_1 Verification Exercises

exp_eps: Second Order Reverse Sweep
Purpose
In general, a second order reverse sweep is given the first order expansion for all of the variables in an operation sequence. Given a choice of a particular variable, it computes the derivative, of that variables first order expansion coefficient, with respect to all of the independent variables.

Mathematical Form
Suppose that we use the algorithm exp_eps.hpp to compute exp_eps(x, epsilon) with x is equal to .5 and epsilon is equal to .2. For this case, the mathematical function for the operation sequence corresponding to exp_eps is $$\mathit{f}(\mathit{x},\mathrm{\epsilon })=1+\mathit{x}+{\mathit{x}}^2/2$$ The corresponding derivative of the partial derivative with respect to $\mathit{x}$ is $$\begin{array}{ccc}\hfill \frac{{\partial }^2}{\partial {\mathit{x}}^2}\mathit{f}(\mathit{x},\mathrm{\epsilon })& \hfill =\hfill & 1\hfill \\ \hfill {\partial }_{\mathrm{\epsilon }}{\partial }_{\mathit{x}}\mathit{f}(\mathit{x},\mathrm{\epsilon })& \hfill =\hfill & 0\hfill \end{array}$$
epsilon
Since $\mathrm{\epsilon }$ is an independent variable, it could included as an argument to all of the ${\mathit{f}}_{\mathit{j}}$ functions below. The result would be that all the partials with respect to $\mathrm{\epsilon }$ would be zero and hence we drop it to simplify the presentation.

f_7
In reverse mode we choose one dependent variable and compute its derivative with respect to all the independent variables. For our example, we chose the value returned by exp_eps.hpp which is ${\mathit{v}}_7$ . We begin with the function ${\mathit{f}}_7$ where ${\mathit{v}}_7$ is both an argument and the value of the function; i.e., $$\begin{array}{ccc}\hfill {\mathit{f}}_7\left({\mathit{v}}_1^{(0)},{\mathit{v}}_1^{(1)},\dots ,{\mathit{v}}_7^{(0)},{\mathit{v}}_7^{(1)}\right)& \hfill =\hfill & {\mathit{v}}_7^{(1)}\hfill \\ \hfill \frac{\partial {\mathit{f}}_7}{\partial {\mathit{v}}_7^{(1)}}& \hfill =\hfill & 1\hfill \end{array}$$ All the other partial derivatives of ${\mathit{f}}_7$ are zero.

Index 7: f_6
The last operation has index 7, $$\begin{array}{ccc}\hfill {\mathit{v}}_7^{(0)}& \hfill =\hfill & {\mathit{v}}_4^{(0)}+{\mathit{v}}_6^{(0)}\hfill \\ \hfill {\mathit{v}}_7^{(1)}& \hfill =\hfill & {\mathit{v}}_4^{(1)}+{\mathit{v}}_6^{(1)}\hfill \end{array}$$ We define the function ${\mathit{f}}_6\left({\mathit{v}}_1^{(0)},\dots ,{\mathit{v}}_6^{(1)}\right)$ as equal to ${\mathit{f}}_7$ except that ${\mathit{v}}_7^{(0)}$ and ${\mathit{v}}_7^{(1)}$ are eliminated using this operation; i.e. $${\mathit{f}}_6={\mathit{f}}_7\left[{\mathit{v}}_1^{(0)},\dots ,{\mathit{v}}_6^{(1)},{\mathit{v}}_7^{(0)}\left({\mathit{v}}_4^{(0)},{\mathit{v}}_6^{(0)}\right),{\mathit{v}}_7^{(1)}\left({\mathit{v}}_4^{(1)},{\mathit{v}}_6^{(1)}\right)\right]$$ It follows that $$\begin{array}{cccc}\hfill \frac{\partial {\mathit{f}}_6}{\partial {\mathit{v}}_4^{(1)}}& \hfill =\hfill & \frac{\partial {\mathit{f}}_7}{\partial {\mathit{v}}_4^{(1)}}+\frac{\partial {\mathit{f}}_7}{\partial {\mathit{v}}_7^{(1)}}*\frac{\partial {\mathit{v}}_7^{(1)}}{\partial {\mathit{v}}_4^{(1)}}\hfill & =1\hfill \\ \hfill \frac{\partial {\mathit{f}}_6}{\partial {\mathit{v}}_6^{(1)}}& \hfill =\hfill & \frac{\partial {\mathit{f}}_7}{\partial {\mathit{v}}_6^{(1)}}+\frac{\partial {\mathit{f}}_7}{\partial {\mathit{v}}_7^{(1)}}*\frac{\partial {\mathit{v}}_7^{(1)}}{\partial {\mathit{v}}_6^{(1)}}\hfill & =1\hfill \end{array}$$ All the other partial derivatives of ${\mathit{f}}_6$ are zero.

Index 6: f_5
The previous operation has index 6, $$\begin{array}{ccc}\hfill {\mathit{v}}_6^{(0)}& \hfill =\hfill & {\mathit{v}}_5^{(0)}/2\hfill \\ \hfill {\mathit{v}}_6^{(1)}& \hfill =\hfill & {\mathit{v}}_5^{(1)}/2\hfill \end{array}$$ We define the function ${\mathit{f}}_5\left({\mathit{v}}_1^{(0)},\dots ,{\mathit{v}}_5^{(1)}\right)$ as equal to ${\mathit{f}}_6$ except that ${\mathit{v}}_6^{(0)}$ and ${\mathit{v}}_6^{(1)}$ are eliminated using this operation; i.e. $${\mathit{f}}_5={\mathit{f}}_6\left[{\mathit{v}}_1^{(0)},\dots ,{\mathit{v}}_5^{(1)},{\mathit{v}}_6^{(0)}\left({\mathit{v}}_5^{(0)}\right),{\mathit{v}}_6^{(1)}\left({\mathit{v}}_5^{(1)}\right)\right]$$ It follows that $$\begin{array}{cccc}\hfill \frac{\partial {\mathit{f}}_5}{\partial {\mathit{v}}_4^{(1)}}& \hfill =\hfill & \frac{\partial {\mathit{f}}_6}{\partial {\mathit{v}}_4^{(1)}}\hfill & =1\hfill \\ \hfill \frac{\partial {\mathit{f}}_5}{\partial {\mathit{v}}_5^{(1)}}& \hfill =\hfill & \frac{\partial {\mathit{f}}_6}{\partial {\mathit{v}}_5}+\frac{\partial {\mathit{f}}_6}{\partial {\mathit{v}}_6^{(1)}}*\frac{\partial {\mathit{v}}_6^{(1)}}{\partial {\mathit{v}}_5^{(1)}}\hfill & =0.5\hfill \end{array}$$ All the other partial derivatives of ${\mathit{f}}_5$ are zero.

Index 5: f_4
The previous operation has index 5, $$\begin{array}{ccc}\hfill {\mathit{v}}_5^{(0)}& \hfill =\hfill & {\mathit{v}}_3^{(0)}*{\mathit{v}}_1^{(0)}\hfill \\ \hfill {\mathit{v}}_5^{(1)}& \hfill =\hfill & {\mathit{v}}_3^{(1)}*{\mathit{v}}_1^{(0)}+{\mathit{v}}_3^{(0)}*{\mathit{v}}_1^{(1)}\hfill \end{array}$$ We define the function ${\mathit{f}}_4\left({\mathit{v}}_1^{(0)},\dots ,{\mathit{v}}_4^{(1)}\right)$ as equal to ${\mathit{f}}_5$ except that ${\mathit{v}}_5^{(0)}$ and ${\mathit{v}}_5^{(1)}$ are eliminated using this operation; i.e. $${\mathit{f}}_4={\mathit{f}}_5\left[{\mathit{v}}_1^{(0)},\dots ,{\mathit{v}}_4^{(1)},{\mathit{v}}_5^{(0)}\left({\mathit{v}}_1^{(0)},{\mathit{v}}_3^{(0)}\right),{\mathit{v}}_5^{(1)}\left({\mathit{v}}_1^{(0)},{\mathit{v}}_1^{(1)},{\mathit{v}}_3^{(0)},{\mathit{v}}_3^{(1)}\right),\right]$$ Given the information from the forward sweep, we have ${\mathit{v}}_1^{(0)}=0.5$ , ${\mathit{v}}_3^{(0)}=0.5$ , ${\mathit{v}}_1^{(1)}=1$ , ${\mathit{v}}_3^{(1)}=1$ , and the fact that the partial of ${\mathit{f}}_5$ with respect to ${\mathit{v}}_5^{(0)}$ is zero, we have $$\begin{array}{cccc}\hfill \frac{\partial {\mathit{f}}_4}{\partial {\mathit{v}}_1^{(0)}}& \hfill =\hfill & \frac{\partial {\mathit{f}}_5}{\partial {\mathit{v}}_1^{(0)}}+\frac{\partial {\mathit{f}}_5}{\partial {\mathit{v}}_5^{(1)}}*\frac{\partial {\mathit{v}}_5^{(1)}}{\partial {\mathit{v}}_1^{(0)}}\hfill & =0.5\hfill \\ \hfill \frac{\partial {\mathit{f}}_4}{\partial {\mathit{v}}_1^{(1)}}& \hfill =\hfill & \frac{\partial {\mathit{f}}_5}{\partial {\mathit{v}}_1^{(1)}}+\frac{\partial {\mathit{f}}_5}{\partial {\mathit{v}}_5^{(1)}}*\frac{\partial {\mathit{v}}_5^{(1)}}{\partial {\mathit{v}}_1^{(1)}}\hfill & =0.25\hfill \\ \hfill \frac{\partial {\mathit{f}}_4}{\partial {\mathit{v}}_3^{(0)}}& \hfill =\hfill & \frac{\partial {\mathit{f}}_5}{\partial {\mathit{v}}_3^{(0)}}+\frac{\partial {\mathit{f}}_5}{\partial {\mathit{v}}_5^{(1)}}*\frac{\partial {\mathit{v}}_5^{(1)}}{\partial {\mathit{v}}_3^{(0)}}\hfill & =0.5\hfill \\ \hfill \frac{\partial {\mathit{f}}_4}{\partial {\mathit{v}}_3^{(1)}}& \hfill =\hfill & \frac{\partial {\mathit{f}}_3}{\partial {\mathit{v}}_1^{(1)}}+\frac{\partial {\mathit{f}}_5}{\partial {\mathit{v}}_5^{(1)}}*\frac{\partial {\mathit{v}}_5^{(1)}}{\partial {\mathit{v}}_3^{(1)}}\hfill & =0.25\hfill \\ \hfill \frac{\partial {\mathit{f}}_4}{\partial {\mathit{v}}_4^{(1)}}& \hfill =\hfill & \frac{\partial {\mathit{f}}_5}{\partial {\mathit{v}}_4^{(1)}}\hfill & =1\hfill \end{array}$$ All the other partial derivatives of ${\mathit{f}}_5$ are zero.

Index 4: f_3
The previous operation has index 4, $$\begin{array}{c}\hfill {\mathit{v}}_4^{(0)}=1+{\mathit{v}}_3^{(0)}\\ \hfill {\mathit{v}}_4^{(1)}={\mathit{v}}_3^{(1)}\end{array}$$ We define the function ${\mathit{f}}_3\left({\mathit{v}}_1^{(0)},\dots ,{\mathit{v}}_3^{(1)}\right)$ as equal to ${\mathit{f}}_4$ except that ${\mathit{v}}_4^{(0)}$ and ${\mathit{v}}_4^{(1)}$ are eliminated using this operation; i.e. $${\mathit{f}}_3={\mathit{f}}_4\left[{\mathit{v}}_1^{(0)},\dots ,{\mathit{v}}_3^{(1)},{\mathit{v}}_4^{(0)}\left({\mathit{v}}_3^{(0)}\right),{\mathit{v}}_4^{(1)}\left({\mathit{v}}_3^{(1)}\right)\right]$$ It follows that $$\begin{array}{cccc}\hfill \frac{\partial {\mathit{f}}_3}{\partial {\mathit{v}}_1^{(0)}}& \hfill =\hfill & \frac{\partial {\mathit{f}}_4}{\partial {\mathit{v}}_1^{(0)}}\hfill & =0.5\hfill \\ \hfill \frac{\partial {\mathit{f}}_3}{\partial {\mathit{v}}_1^{(1)}}& \hfill =\hfill & \frac{\partial {\mathit{f}}_4}{\partial {\mathit{v}}_1^{(1)}}\hfill & =0.25\hfill \\ \hfill \frac{\partial {\mathit{f}}_3}{\partial {\mathit{v}}_2^{(0)}}& \hfill =\hfill & \frac{\partial {\mathit{f}}_4}{\partial {\mathit{v}}_2^{(0)}}\hfill & =0\hfill \\ \hfill \frac{\partial {\mathit{f}}_3}{\partial {\mathit{v}}_2^{(1)}}& \hfill =\hfill & \frac{\partial {\mathit{f}}_4}{\partial {\mathit{v}}_2^{(1)}}\hfill & =0\hfill \\ \hfill \frac{\partial {\mathit{f}}_3}{\partial {\mathit{v}}_3^{(0)}}& \hfill =\hfill & \frac{\partial {\mathit{f}}_4}{\partial {\mathit{v}}_3^{(0)}}+\frac{\partial {\mathit{f}}_4}{\partial {\mathit{v}}_4^{(0)}}*\frac{\partial {\mathit{v}}_4^{(0)}}{\partial {\mathit{v}}_3^{(0)}}\hfill & =0.5\hfill \\ \hfill \frac{\partial {\mathit{f}}_3}{\partial {\mathit{v}}_3^{(1)}}& \hfill =\hfill & \frac{\partial {\mathit{f}}_4}{\partial {\mathit{v}}_3^{(1)}}+\frac{\partial {\mathit{f}}_4}{\partial {\mathit{v}}_4^{(1)}}*\frac{\partial {\mathit{v}}_4^{(1)}}{\partial {\mathit{v}}_3^{(1)}}\hfill & =1.25\hfill \end{array}$$
Index 3: f_2
The previous operation has index 3, $$\begin{array}{ccc}\hfill {\mathit{v}}_3^{(0)}& \hfill =\hfill & {\mathit{v}}_2^{(0)}/1\hfill \\ \hfill {\mathit{v}}_3^{(1)}& \hfill =\hfill & {\mathit{v}}_2^{(1)}/1\hfill \end{array}$$ We define the function ${\mathit{f}}_2\left({\mathit{v}}_1^{(0)},\dots ,{\mathit{v}}_2^{(1)}\right)$ as equal to ${\mathit{f}}_3$ except that ${\mathit{v}}_3^{(0)}$ and ${\mathit{v}}_3^{(1)}$ are eliminated using this operation; i.e. $${\mathit{f}}_2={\mathit{f}}_3\left[{\mathit{v}}_1^{(0)},\dots ,{\mathit{v}}_2^{(1)},{\mathit{v}}_3^{(0)}\left({\mathit{v}}_2^{(0)}\right),{\mathit{v}}_3^{(1)}\left({\mathit{v}}_2^{(1)}\right)\right]$$ It follows that $$\begin{array}{cccc}\hfill \frac{\partial {\mathit{f}}_2}{\partial {\mathit{v}}_1^{(0)}}& \hfill =\hfill & \frac{\partial {\mathit{f}}_3}{\partial {\mathit{v}}_1^{(0)}}\hfill & =0.5\hfill \\ \hfill \frac{\partial {\mathit{f}}_2}{\partial {\mathit{v}}_1^{(1)}}& \hfill =\hfill & \frac{\partial {\mathit{f}}_3}{\partial {\mathit{v}}_1^{(1)}}\hfill & =0.25\hfill \\ \hfill \frac{\partial {\mathit{f}}_2}{\partial {\mathit{v}}_2^{(0)}}& \hfill =\hfill & \frac{\partial {\mathit{f}}_3}{\partial {\mathit{v}}_2^{(0)}}+\frac{\partial {\mathit{f}}_3}{\partial {\mathit{v}}_3^{(0)}}*\frac{\partial {\mathit{v}}_3^{(0)}}{\partial {\mathit{v}}_2^{(0)}}\hfill & =0.5\hfill \\ \hfill \frac{\partial {\mathit{f}}_2}{\partial {\mathit{v}}_2^{(1)}}& \hfill =\hfill & \frac{\partial {\mathit{f}}_3}{\partial {\mathit{v}}_2^{(1)}}+\frac{\partial {\mathit{f}}_3}{\partial {\mathit{v}}_3^{(1)}}*\frac{\partial {\mathit{v}}_3^{(1)}}{\partial {\mathit{v}}_2^{(0)}}\hfill & =1.25\hfill \end{array}$$
Index 2: f_1
The previous operation has index 1, $$\begin{array}{ccc}\hfill {\mathit{v}}_2^{(0)}& \hfill =\hfill & 1*{\mathit{v}}_1^{(0)}\hfill \\ \hfill {\mathit{v}}_2^{(1)}& \hfill =\hfill & 1*{\mathit{v}}_1^{(1)}\hfill \end{array}$$ We define the function ${\mathit{f}}_1\left({\mathit{v}}_1^{(0)},{\mathit{v}}_1^{(1)}\right)$ as equal to ${\mathit{f}}_2$ except that ${\mathit{v}}_2^{(0)}$ and ${\mathit{v}}_2^{(1)}$ are eliminated using this operation; i.e. $${\mathit{f}}_1={\mathit{f}}_2\left[{\mathit{v}}_1^{(0)},{\mathit{v}}_1^{(1)},{\mathit{v}}_2^{(0)}\left({\mathit{v}}_1^{(0)}\right),{\mathit{v}}_2^{(1)}\left({\mathit{v}}_1^{(1)}\right)\right]$$ It follows that $$\begin{array}{cccc}\hfill \frac{\partial {\mathit{f}}_1}{\partial {\mathit{v}}_1^{(0)}}& \hfill =\hfill & \frac{\partial {\mathit{f}}_2}{\partial {\mathit{v}}_1^{(0)}}+\frac{\partial {\mathit{f}}_2}{\partial {\mathit{v}}_2^{(0)}}*\frac{\partial {\mathit{v}}_2^{(0)}}{\partial {\mathit{v}}_1^{(0)}}\hfill & =1\hfill \\ \hfill \frac{\partial {\mathit{f}}_1}{\partial {\mathit{v}}_1^{(1)}}& \hfill =\hfill & \frac{\partial {\mathit{f}}_2}{\partial {\mathit{v}}_1^{(1)}}+\frac{\partial {\mathit{f}}_2}{\partial {\mathit{v}}_2^{(1)}}*\frac{\partial {\mathit{v}}_2^{(1)}}{\partial {\mathit{v}}_1^{(1)}}\hfill & =1.5\hfill \end{array}$$ Note that ${\mathit{v}}_1$ is equal to $\mathit{x}$ , so the second partial derivative of exp_eps(x, epsilon) at x equal to .5 and epsilon equal .2 is $$\frac{{\partial }^2}{\partial {\mathit{x}}^2}{\mathit{v}}_7^{(0)}=\frac{\partial {\mathit{v}}_7^{(1)}}{\partial \mathit{x}}=\frac{\partial {\mathit{f}}_1}{\partial {\mathit{v}}_1^{(0)}}=1$$ There is a theorem about algorithmic differentiation that explains why the other partial of ${\mathit{f}}_1$ is equal to the first partial of exp_eps(x, epsilon) with respect to $\mathit{x}$ .

Verification
The file exp_eps_rev2.cpp contains a routine that verifies the values computed above. It returns true for success and false for failure. It only tests the partial derivatives of ${\mathit{f}}_{\mathit{j}}$ that might not be equal to the corresponding partials of ${\mathit{f}}_{\mathit{j}+1}$ ; i.e., the other partials of ${\mathit{f}}_{\mathit{j}}$ must be equal to the corresponding partials of ${\mathit{f}}_{\mathit{j}+1}$ .

Exercises

1. Consider the case where $\mathit{x}=.1$ and we first preform a zero order forward mode sweep for the operation sequence used above (in reverse order). What are the results of a first order reverse mode sweep; i.e., what are the corresponding values for $\frac{\partial {\mathit{f}}_{\mathit{j}}}{\partial {\mathit{v}}_{\mathit{k}}}$ for all $\mathit{j},\mathit{k}$ such that $\frac{\partial {\mathit{f}}_{\mathit{j}}}{\partial {\mathit{v}}_{\mathit{k}}}\ne 0$ .
2. Create a modified version of exp_eps_rev2.cpp that verifies the values you obtained for the previous exercise. Also create and run a main program that reports the result of calling the modified version of exp_eps_rev2.cpp .

---

Input File: introduction/exp_apx/exp_eps.omh