Exercise+4a.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Writing a Custom Iterator and Model\n",
    "This notebook will walk you through implementing a custom iterator for a modified version of the Street View House Number (SVHN) dataset. You will then design a network to train on this dataset. \n",
    "\n",
    "## SVHN dataset\n",
    "\n",
    "This dataset is a collection of 73,257 images of house numbers collected from Google Streetview. The original dataset has bounding boxes for all the digits in the image:\n",
    "\n",
    "<img src=\"http://ufldl.stanford.edu/housenumbers/examples_new.png\" width=500px>\n",
    "\n",
    "We have modified the dataset such that each image is 64x64 pixels (with 3 color channels), and the target is a *single* bounding box over all the digits. Your goal is to build a network that, given an image, returns bounding box coordinates for the location of the digit sequence.\n",
    "\n",
    "This notebook is split into two parts:\n",
    "* Writing a custom dataiterator\n",
    "* Building a prediction network\n",
    "\n",
    "## Custom dataset\n",
    "\n",
    "Because the training set of ~27,000 images can fit into the memory of a single Titan X GPU, we could use the `ArrayIterator` class to provide data to the model. However, when the dataset may have more images or larger image sizes, that is no longer an option. Our high-performance `DataLoader`, which loads image in batches and performs complex augmentation, cannot currently handle bounding box data (stay tuned, an object localization dataloader is coming in a future neon release!).\n",
    "\n",
    "We've saved the dataset as a pickle file `svhn_64.p`. This file has a few variables:\n",
    "- `X_train`: a numpy array of shape `(num_examples, num_features)`, where `num_examples = 26624`, and `num_features = 3*64*64 = 12288`\n",
    "- `y_train`: a numpy array of shape `(num_examples, 4)`, with the target bounding box coordinates in `(x_min, y_min, w, h)` format.\n",
    "- `X_test`: a numpy array of shape `(3328, 12288)`\n",
    "- `y_test`: a numpy array of shape `(3328, 4)`\n",
    "\n",
    "Let's first import our backend:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from neon.backends import gen_backend\n",
    "\n",
    "be = gen_backend(batch_size=128, backend='mkl')\n",
    "\n",
    "# set the debug level to 10 (the minimum)\n",
    "# to see all the output\n",
    "import logging\n",
    "main_logger = logging.getLogger('neon')\n",
    "main_logger.setLevel(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The modified SVHN dataset can be found at: https://s3-us-west-1.amazonaws.com/nervana-course/svhn_64.p. Place this dataset into the data/ folder and we can then load the pickle file with our SVHN dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Loading data/svhn_64.p...\n"
     ]
    }
   ],
   "source": [
    "import _pickle as cPickle\n",
    "\n",
    "fileName = 'data/svhn_64.p'\n",
    "print(\"Loading {}...\".format(fileName))\n",
    "\n",
    "with open(fileName, \"rb\") as f:\n",
    "    svhn = cPickle.load(f, encoding='latin1')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "Below is a skeleton of the SVHN data iterator for you to fill out, with notes to help along the way. The goal is an object that returns, with each call, a tuple of `(X, Y)` for the input and the target bounding boxes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "ename": "SyntaxError",
     "evalue": "invalid syntax (<ipython-input-5-63a1f5ba76fc>, line 61)",
     "output_type": "error",
     "traceback": [
      "\u001b[0;36m  File \u001b[0;32m\"<ipython-input-5-63a1f5ba76fc>\"\u001b[0;36m, line \u001b[0;32m61\u001b[0m\n\u001b[0;31m    self.dev_X ...\u001b[0m\n\u001b[0m                 ^\u001b[0m\n\u001b[0;31mSyntaxError\u001b[0m\u001b[0;31m:\u001b[0m invalid syntax\n"
     ]
    }
   ],
   "source": [
    "# import some useful packages\n",
    "from neon.data import NervanaDataIterator\n",
    "import numpy as np\n",
    "import _pickle as cPickle\n",
    "import os\n",
    "\n",
    "class SVHN(NervanaDataIterator):\n",
    "\n",
    "    def __init__(self, X, Y, lshape):\n",
    "\n",
    "        # Load the numpy data into some variables. We divide the image by 255 to normalize the values\n",
    "        # between 0 and 1.\n",
    "        self.X = X / 255.\n",
    "        self.Y = Y\n",
    "        self.shape = lshape  # shape of the input data (e.g. for images, (C, H, W))\n",
    "\n",
    "        # 1. assign some required and useful attributes\n",
    "        self.start = 0  # start at zero\n",
    "        self.ndata = ...  # number of images in X (hint: use X.shape)\n",
    "        self.nfeatures = ...  # number of features in X (hint: use X.shape)\n",
    "\n",
    "        # number of minibatches per epoch\n",
    "        # to calculate this, use the batchsize, which is stored in self.be.bsz\n",
    "        self.nbatches = ...\n",
    "        \n",
    "        \n",
    "        # 2. allocate memory on the GPU for a minibatch's worth of data.\n",
    "        # (e.g. use `self.be` to access the backend.). See the backend documentation.\n",
    "        # to get the minibatch size, use self.be.bsz\n",
    "        # hint: X should have shape (# features, mini-batch size)\n",
    "        # hint: use some of the attributes previously defined above\n",
    "        self.dev_X = ...\n",
    "        self.dev_Y = ...\n",
    "\n",
    "\n",
    "    def reset(self):\n",
    "        self.start = 0\n",
    "\n",
    "    def __iter__(self):\n",
    "        # 3. loop through minibatches in the dataset\n",
    "        for index in range(self.start, self.ndata, self.be.bsz):\n",
    "            # 3a. grab the right slice from the numpy arrays\n",
    "            inputs = ...\n",
    "            targets = ...\n",
    "            \n",
    "            # The arrays X and Y data are in shape (batch_size, num_features),\n",
    "            # but the iterator needs to return data with shape (num_features, batch_size).\n",
    "            # here we transpose the data, and then store it as a contiguous array. \n",
    "            # numpy arrays need to be contiguous before being loaded onto the GPU.\n",
    "            inputs = np.ascontiguousarray(inputs.T)\n",
    "            targets = np.ascontiguousarray(targets.T)\n",
    "                        \n",
    "            # here we test your implementation\n",
    "            # your slice has to have the same shape as the GPU tensors you allocated\n",
    "            assert inputs.shape == self.dev_X.shape, \\\n",
    "                   \"inputs has shape {}, but dev_X is {}\".format(inputs.shape, self.dev_X.shape)\n",
    "            assert targets.shape == self.dev_Y.shape, \\\n",
    "                   \"targets has shape {}, but dev_Y is {}\".format(targets.shape, self.dev_Y.shape)\n",
    "            \n",
    "            # 3b. transfer from numpy arrays to device\n",
    "            # - use the GPU memory buffers allocated previously,\n",
    "            #    and call the myTensorBuffer.set() function. \n",
    "            self.dev_X ...\n",
    "            self.dev_Y ...\n",
    "            \n",
    "            # 3c. yield a tuple of the device tensors.\n",
    "            # the first should of shape (num_features, batch_size)\n",
    "            # the second should of shape (4, batch_size)\n",
    "            yield (..., ...)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Check your implementation! Below we grab an iteration and print out the output of the dataset. Importantly: make sure that the output tensors are contiguous (e.g. `is_contiguous = True` in the output below). This means that they are allocated on a contiguous set of memory, which is important for the downstream calculations. Contiguity can be broken by operations like transpose."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "ename": "NameError",
     "evalue": "name 'SVHN' is not defined",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mNameError\u001b[0m                                 Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-6-0335354c7b83>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[1;32m      1\u001b[0m \u001b[0;31m# setup datasets\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 2\u001b[0;31m \u001b[0mtrain_set\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mSVHN\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mX\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0msvhn\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m'X_train'\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mY\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0msvhn\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m'y_train'\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mlshape\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;36m3\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;36m64\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;36m64\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m      3\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      4\u001b[0m \u001b[0;31m# grab one iteration from the train_set\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      5\u001b[0m \u001b[0miterator\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mtrain_set\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m__iter__\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mNameError\u001b[0m: name 'SVHN' is not defined"
     ]
    }
   ],
   "source": [
    "# setup datasets\n",
    "train_set = SVHN(X=svhn['X_train'], Y=svhn['y_train'], lshape=(3, 64, 64))\n",
    "\n",
    "# grab one iteration from the train_set\n",
    "iterator = train_set.__iter__()\n",
    "(X, Y) = iterator.next()\n",
    "print(X)  # this should be shape (12288, 128)\n",
    "print(Y)  # this should be shape (4, 128)\n",
    "assert X.is_contiguous\n",
    "assert Y.is_contiguous"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If all goes well, you are ready to try training on this network! First, let's reset the dataset to zero (since you drew one example from above). We also add a test set for evaluation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "train_set.reset()\n",
    "\n",
    "# generate test set\n",
    "test_set = SVHN(X=svhn['X_test'], Y=svhn['y_test'], lshape=(3, 64, 64))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Model architecture\n",
    "We recommend using a VGG-style convolutional neural network to train this model, using the ConvNet Design Philosophy we introduced earlier. We've imported some relevant packages that you may want to use, and have some guiding steps for implementing your network. Experiment with networks of different sizes!\n",
    "\n",
    "Some tips:\n",
    "- Training a model for 10 epochs should take 30s/epoch. If you are taking longer than that, your network is too large.\n",
    "- Compare the training set cost and the validation set loss to make sure you are not overfitting on the data.\n",
    "- Try to get a validation set loss of ~220 after 10 epochs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from neon.callbacks.callbacks import Callbacks\n",
    "from neon.initializers import Gaussian\n",
    "from neon.layers import GeneralizedCost, Affine, Conv, Pooling, Linear, Dropout\n",
    "from neon.models import Model\n",
    "from neon.optimizers import GradientDescentMomentum, RMSProp\n",
    "from neon.transforms import Rectlin, Logistic, CrossEntropyMulti, Misclassification, SumSquared\n",
    "\n",
    "# set up weight initializer\n",
    "...\n",
    "\n",
    "# set up model layers\n",
    "layers = []\n",
    "\n",
    "layers.append(...)\n",
    "\n",
    "# the last layer should be a linear layer with nout=4, for the 4 coordinates of the bounding box.\n",
    "layers.append(Linear(nout=4, init=Gaussian(loc=0.0, scale=0.01)))\n",
    "\n",
    "# use SumSquared cost\n",
    "cost = GeneralizedCost(costfunc=SumSquared())\n",
    "\n",
    "# setup optimizer\n",
    "optimizer = RMSProp()\n",
    "\n",
    "# initialize model object\n",
    "mlp = Model(layers=layers)\n",
    "\n",
    "# configure callbacks\n",
    "callbacks = Callbacks(mlp, eval_set=test_set, eval_freq=1)\n",
    "\n",
    "# run fit\n",
    "mlp.fit(train_set, optimizer=optimizer, num_epochs=10, cost=cost, callbacks=callbacks)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Below we plot the cost data over time to help you visualize the training progress. This is similiar to using the `nvis` command line tool to generate plots."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from neon.visualizations.figure import cost_fig, hist_fig, deconv_summary_page\n",
    "from neon.visualizations.data import h5_cost_data, h5_hist_data, h5_deconv_data\n",
    "from bokeh.plotting import output_notebook, show\n",
    "\n",
    "cost_data = h5_cost_data('data.h5', False)\n",
    "output_notebook()\n",
    "show(cost_fig(cost_data, 300, 600, epoch_axis=False))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "To understand how the network performed, we sample images and plot the network's predicted bounding box against the ground truth bounding box. We evaluate this on the `test_set`, which was not used to train the network."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "import matplotlib\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "# get a minibatch's worth of\n",
    "# inputs (X) and targets (T)\n",
    "iterator = test_set.__iter__()\n",
    "(X, T) = iterator.next()\n",
    "\n",
    "# fprop the input to get the model output\n",
    "y = mlp.fprop(X)\n",
    "\n",
    "# transfer from device to numpy arrays\n",
    "y = y.get()\n",
    "T = T.get()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Our ground truth box `T` and the model prediction `y` are both arrays of size `(4, batch_size)`. We can plot an image below. Feel free to modify `i` to check performance on various test images. Red boxes are the model's guess, and blue boxes are the ground truth boxes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "plt.figure(2)\n",
    "imgs_to_plot = [0, 1, 2, 3]\n",
    "for i in imgs_to_plot:\n",
    "    plt.subplot(2, 2, i+1)\n",
    "\n",
    "    title = \"test {}\".format(i)\n",
    "    plt.imshow(X.get()[:, i].reshape(3, 64, 64).transpose(1, 2, 0))\n",
    "    ax = plt.gca()\n",
    "    ax.add_patch(plt.Rectangle((y[0,i], y[1,i]), y[2,i], y[3,i], fill=False, edgecolor=\"red\")) # model guess\n",
    "    ax.add_patch(plt.Rectangle((T[0,i], T[1,i]), T[2,i], T[3,i], fill=False, edgecolor=\"blue\")) # ground truth\n",
    "    plt.title(title)\n",
    "    plt.axis('off')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "i=0\n",
    "print(\"Target box had coordinates: {}\".format(T[:,i]))\n",
    "print(\"Model prediction has coordinates: {}\".format(y[:, i]))"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}