This lab is derived from material that I produced for a training session on Automatic Differentiation for VLC students and staff in the late Spring of 2018. A lot of inspiration for the content came from this excellent blog post on Reverse Mode AD.
Automatic Differentiation (AD) the key to writing and optimising complex differentiable programs. With AD we no do not have to rely on hand-computing or automatically computing analytic solutions to gradients, nor do we have to rely on, potentially highly inaccurate, numerical gradient estimates.
Surprisingly, Automatic Differentiation is actually not that hard to implement. In this lab we’ll start by hand-coding simple forward and reverse mode AD systems in Python so we can get a feeling for how they work, before moving on to looking at how AD works in PyTorch.
Through this lab you’ll learn how to:
To work through this lab you’ll use the Python 3 language in a Jupyter Notebook environment, with the
pytorch tensor library. We will primarily be using Google Colab to run the notebooks as this gives you access to an environment with all the tools required. If you wish to run the notebooks locally, see the information in the section below.
You will likely need to refer to the “autograd” lecture slides for this lab - you can get those here: http://comp6248.ecs.soton.ac.uk/lectures/autograd.pdf.
The following is a list of the notebooks for this lab, with links to open directly in Google Colab (once opened you should immediately save a copy in your Google Drive otherwise anything you do will be lost once the browser closes), or to download locally. You should work through the notebooks in numeric order as they follow on from each other.
|2.1 Forward Mode Automatic Differentiation||preview||download|
|2.2 Reverse Mode Automatic Differentiation||preview||download|
|2.3 Automatic Differentiation in PyTorch||preview||download|
You’ll need access to a computer with the following installed:
If you want to work on your own machine we recommend using the Anaconda python distribution. Running
conda install pytorch torchvision -c pytorch (see https://pytorch.org/get-started/locally/ for more options) will install both pytorch and torchvision (used in later labs).