01 MNIST example.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To explore this ipython notebook, press `SHIFT+ENTER` to progress to the next cell. Feel free to make changes, enter code, and hack around. You can create new code cells by selecting `INSERT->Insert Cell Below`\n",
    "\n",
    "# MNIST Example\n",
    "\n",
    "MNIST is a computer vision dataset consisting of 70,000 images of handwritten digits. Each image has 28x28 pixels for a total of 784 features, and is associated with a digit between 0-9.\n",
    "\n",
    "<img src=\"http://corpocrat.com/wp-content/uploads/2014/10/figure_1.png\" width=200px>\n",
    "\n",
    "\n",
    "In this tutorial, we will construct a multi-layer perceptron (also called softmax regression) to recognize each image. Note that this tutorial assumes some basic familiarity with python and machine learning.\n",
    "\n",
    "This tutorial is similar to the model specified in `examples/mnist_mlp.py`.\n",
    "\n",
    "## Setup\n",
    "\n",
    "This example works with Python 2.7. The urllib request method needs to be changed in the inference steps for Python 3.x.\n",
    "\n",
    "Your environment needs to have the following packages installed:\n",
    "- neon v1.8.1\n",
    "- matplotlib (for the inference)\n",
    "\n",
    "## Preamble\n",
    "The first step is to set up our compute backend."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from neon.backends import gen_backend\n",
    "\n",
    "be = gen_backend(batch_size=128)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "The MNIST dataset can be found on Yann LeCunn’s website. We have included an easy function that downloads the MNIST dataset into your `~/nervana/data/` directory, loads it into memory, and returns `ArrayIterator` objects for the training and validation set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from neon.data import MNIST\n",
    "\n",
    "mnist = MNIST(path='data/')\n",
    "train_set = mnist.train_iter\n",
    "valid_set = mnist.valid_iter"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "During training, neon iterates over the training examples to compute the gradients. The `train_iter` and `valid_iter` handle sending data to the model for training and validation, respectively.\n",
    "\n",
    "For small datasets like MNIST, this step may seem trivial. However, for large datasets that cannot fit into memory (e.g. ImageNet or Sports-1M), the data has to be efficiently loaded and fed to the optimizer in batches. This requires more advanced iterators described in Loading data.\n",
    "\n",
    "## Model specification\n",
    "Training a deep learning model in Neon requires \n",
    "* Specifying the dataset\n",
    "* Building a model from a list of layers\n",
    "* Cost function\n",
    "* Learning rule. \n",
    "\n",
    "Here we guide you through each item in turn.\n",
    "\n",
    "### Initializing weights\n",
    "Neon supports many ways of initializing weight matrices. In this tutorial, we initialize the weights using a Gaussian distribution with zero mean and 0.01 standard deviation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from neon.initializers import Gaussian\n",
    "\n",
    "init_norm = Gaussian(loc=0.0, scale=0.01)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Model architecture\n",
    "The model is specified as a list of layers. For classifying MNIST images, we use a multi-layer perceptron with fully connected layers.\n",
    "\n",
    "- Affine (i.e. fully-connected) layer with 100 hidden units and a rectified linear activation function, defined as Rectlin().\n",
    "- An output layer with 10 units to match the number of labels in the MNIST dataset. We use the Softmax() activation function to ensure the outputs sum to one and are within the range [0,1].\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from neon.layers import Affine\n",
    "from neon.transforms import Rectlin, Softmax\n",
    "\n",
    "layers = []\n",
    "layers.append(Affine(nout=100, init=init_norm, activation=Rectlin()))\n",
    "layers.append(Affine(nout=10, init=init_norm,\n",
    "                     activation=Softmax()))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We initialize the weights in each layer with the init_norm defined previously. Neon supports many other layer types (convolutional, pooling, recurrent, etc.) that will be described in subsequent examples. We then construct the model via"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# initialize model object\n",
    "from neon.models import Model\n",
    "\n",
    "mlp = Model(layers=layers)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Costs\n",
    "The cost function is wrapped within a GeneralizedCost layer, which handles the comparison of the outputs with the provided labels in the dataset. One common cost function which we use here is the cross entropy loss."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from neon.layers import GeneralizedCost\n",
    "from neon.transforms import CrossEntropyMulti\n",
    "\n",
    "cost = GeneralizedCost(costfunc=CrossEntropyMulti())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Learning rules\n",
    "For learning, we use stochastic gradient descent with a learning rate of 0.1 and momentum coefficient of 0.9.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from neon.optimizers import GradientDescentMomentum\n",
    "\n",
    "optimizer = GradientDescentMomentum(0.1, momentum_coef=0.9)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Callbacks\n",
    "Neon provides an API for calling operations during the model fit (see Callbacks). Here we set up the default callback, which is displaying a progress bar for each epoch.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from neon.callbacks.callbacks import Callbacks\n",
    "\n",
    "callbacks = Callbacks(mlp, eval_set=valid_set)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Putting it all together\n",
    "We are ready to put all the ingredients together and run our model!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "mlp.fit(train_set, optimizer=optimizer, num_epochs=10, cost=cost,\n",
    "        callbacks=callbacks)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "At the beginning of the fitting procedure, neon propagates train_set through the model to set the input and output shapes of each layer. Each layer has a `configure()` method that determines the appropriate layer shapes, and an `allocate()` method to set up the needed buffers for holding the forward propagation information.\n",
    "\n",
    "During the training, neon sends batches of the training data through the model, calling each layers’ `fprop()` and `bprop()` methods to compute the gradients and update the weights."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Using the trained model\n",
    "Now that the model is successfully trained, we can use the trained model to classify a novel image, measure performance, and visualize the weights and training results.\n",
    "\n",
    "#### Get outputs\n",
    "Given a set of images such as those contained in the iterable `test_set`, we can fetch the ouput of the final model layer via\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "results = mlp.get_outputs(valid_set)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The variable results is a numpy array with shape `(num_test_examples, num_outputs) = (10000,10)` with the model probabilities for each label.\n",
    "\n",
    "#### Performance\n",
    "Neon supports convenience functions for evaluating performance using custom metrics. Here we measure the misclassification rate on the held out test set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from neon.transforms import Misclassification\n",
    "\n",
    "# evaluate the model on test_set using the misclassification metric\n",
    "error = mlp.eval(valid_set, metric=Misclassification())*100\n",
    "print('Misclassification error = %.1f%%' % error)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Inference\n",
    "\n",
    "Now let's download a new digit image from the web, and use our trained model to recognize the digit. We first download the image and scale it to the 28x28 pixels that our model expects."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import urllib\n",
    "from PIL import Image\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline \n",
    "\n",
    "# download image\n",
    "url = \"http://datawrangling.s3.amazonaws.com/sample_digit.png\"\n",
    "urllib.urlretrieve(url, filename=\"data/digit.jpg\")\n",
    "\n",
    "# scale to 28x28 pixels\n",
    "img = Image.open(\"data/digit.jpg\")\n",
    "img.thumbnail((28, 28))\n",
    "\n",
    "digit = np.asarray(img, dtype=np.float32)[:, :, 0]\n",
    "\n",
    "# reshape to a single feature vector\n",
    "digit = digit.reshape(784, 1)\n",
    "\n",
    "# store digit into a GPU tensor\n",
    "x_new = be.zeros((28*28, 128), dtype=np.float32)\n",
    "x_new[:, 0] = digit\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# forward pass through the model\n",
    "outputs = mlp.fprop(x_new)\n",
    "outputs = outputs.get()[:, 0]\n",
    "\n",
    "# examine the output of the model for this image\n",
    "print \"Model final layer was: {}\".format(outputs)\n",
    "print \"The most probable guess is digit: {}\".format(np.argmax(outputs))\n",
    "plt.figure(2)\n",
    "plt.imshow(img)\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}