P04-C03-deep-cnns-alexnet.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Deep convolutional neural networks \n",
    "\n",
    "In the previous chapters, you got a sense \n",
    "for how to classify images with convolutional neural network (CNNs). \n",
    "Specifically, we implemented a CNN with two convolutional layers \n",
    "interleaved with pooling layers, a singly fully-connected hidden layer,\n",
    "and a softmax output layer. \n",
    "That architecture loosely resembles a neural network affectionately named LeNet,\n",
    "in honor [Yann LeCun](http://yann.lecun.com/),\n",
    "an early pionneer of convolutional neural networks and the first \n",
    "to [reduced them to practice in 1989](http://www.mitpressjournals.org/doi/abs/10.1162/neco.1989.1.4.541) \n",
    "by training them with gradient descent (i.e. backpropagation).\n",
    "At the time, this was fairly novel idea.\n",
    "A cadre of reserchers interested in biologically-inspired learning models had taken to investigating artificial simulations of neurons as learning models. \n",
    "However, as remains true to this day, few researchers believed that real brains learn by gradient descent.\n",
    "The community of neural networks researchers had explored many other learning rules.\n",
    "LeCun demonstrated that CNNs trained by gradient descent,\n",
    "could get state-of-the-art results on the task of recognizing hand-written digits. \n",
    "These groundbreaking results put CNNs on the map.\n",
    "\n",
    "However, in the intervening years, neural networks were superceded by a numerous other methods.\n",
    "Neural networks were considered slow to train, \n",
    "and there wasn't wide consensus on whether it was possible \n",
    "to train very deep neural networks from a random initialization of the weights. \n",
    "Moreover, training networks with many channels, layers, \n",
    "and parameters were required excessive computation \n",
    "relative to the resources available decades ago. \n",
    "While it was possible to train a LeNet for MNIST digit classification and get good scores,\n",
    "neural networks fell out of favor on larger, real-world datasets. \n",
    "\n",
    "Instead researchers precomputed features based on a mixture of elbow grease, \n",
    "knowledge of optics, and black magic. \n",
    "A typical pattern was this: \n",
    "1. Grab a cool dataset\n",
    "2. Preprocess it with giant bag of predetermined feature functions\n",
    "3. Dump the representations into a simple linear model to do the *machine learning part*. \n",
    "\n",
    "This was the state of affairs in computer vision up until 2012, \n",
    "just before deep learning began to change the world of applied machine learning. \n",
    "One of us (Zack) entered graduate school in 2013. \n",
    "A friend in graduate school summarized the state of affairs thusly:\n",
    "\n",
    "If you spoke to machine learning researchers, \n",
    "they believed that machine learning was both important and beautiful.\n",
    "Elegant theories proved the properties of various classifiers.\n",
    "The field of machine learning was thriving, rigorous and eminently useful.\n",
    "However, if you spoke to a comuter vision researcher, you'd hear a very different story.\n",
    "The dirty truth of image recognition, they'd tell you,\n",
    "is that the really important aspects of the ML for CV pipeline were data and features.\n",
    "A slightly cleaner dataset, or a slightly better hand-tuned feature mattered a lot to the final accuracy.\n",
    "However, the specific choice of classifier was little more than an afterthought.\n",
    "At the end of the day you could throw your features in a logistic regression model, a support vector machine, or any other classifier of choice, and they would all perform roughly the same. \n",
    "\n",
    "\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Learning the representations\n",
    "\n",
    "Another way to cast the state of affairs is that fixing a dataset, \n",
    "the most important part of the pipeline was the representation.\n",
    "And up until 2012, this part was done mechanically, based on some hard-fought intuition.\n",
    "In fact, engineering a new set of feture functions, improving results, and writing up the method was a prominent genre of paper.\n",
    "\n",
    "Another group of researchers had other plans. They believed that features themselves ought to be learned. \n",
    "Moreover they believed that to be reasonable complex, the fetures ought to be hierarchically composed.\n",
    "These researchers, including LeCun, Geoff Hinton, Yoshua Bengio (in America), and Juergen Schmidhuber (in Switzerland) believed that by jointly training many layers of a neural network, they might come to learn hierarchical representations of data. \n",
    "In the case of an image, the lowest layers might come to detect edges, colors, and textures. \n",
    "\n",
    "![](./img/filters.png)\n",
    "\n",
    "\n",
    "Higher layers might build upon these representations to represent larger structures, like eyes, noses, blades of grass, and features. \n",
    "Yet higher layers might represent whole objects like people, airplanes, dogs, or frisbees. \n",
    "And ultimately, before the classification layer, the final hidden state might represent a compact representation of the image that summarized the contents in a space where data belonging to different categories would be linearly seperable. For years, researchers "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Missing ingredient 1: data \n",
    "\n",
    "Despite the sustained interest of a committed group of researchers in learning deep representations of visual data, for a long time these ambitions were frustrated. The failures to make progress owed to a few factors. First, while this wasn't yet known, supervised deep models with many representation require large amounts of labeled training data in order to out perform classical approaches. However, given the limited storage capacity of computers and the comparatively tighter research budgets in the 1990s and prior, most research relied on tiny datasets. For example, most many credible research papers relied on a small set of corpora hosted by UCI, many of which contained hundreds or a few thousand images.\n",
    "\n",
    "This changed in a big way when Dr. Fei-Fei Li, now a Vice President at Google and then an assistant professor at UIUC, presented the ImageNet database in 2009. The ImageNet dataset dwarved all previous research datasets. It contained one million images: one thousand each from one thousand distinct classes. \n",
    "\n",
    "![](./img/imagenet.jpeg)\n",
    "\n",
    "This dataset pushed both comuputer visions and machine learning research into a new regime where the previous best methods would no longer dominate. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Missing ingredient 2: hardware\n",
    "\n",
    "While neural network researchers couldn't have known precisely \n",
    "how much data they'd need to get deep learning working, \n",
    "they did know that if they wanted to train a gigantic network,\n",
    "they'd need much more computation than a typical PC or workstation provided. \n",
    "\n",
    "A major breakthrough came in 2012 when Geoffrey Hinton's students, Alex Krizhevsky and Ilya Sutskever \n",
    "implemented a deep convolutional neural network that could run on GPU hardware.\n",
    "GPUs don't provide the full breadth of functionality that CPUs do, and they do not run at nearly the clock speeds that world's fastest CPUs do. \n",
    "But while CPUs can run several cores in parallel, GPUs can execute hundreds of operations in parallel (**there's some funny business about GPU cores and CPU cores not being 1-to-1: defer to Mu for a couple lines of clearer exposition here**). \n",
    "Originally, GPUs were developed for rendering graphics. \n",
    "Every time a the model updates in a video game, the screen must be repainted. \n",
    "The frequency with which the a screen can be repainted can be accelerated by having multiple GPU cores draw multiple pixels simultaneously. \n",
    "\n",
    "By the time Khrizhevsky and company rigged a GPU to train neural networks to they had already become fashionable among scientists who need the parallel processing to run various simulations as are sommon in astrophysics and fluid dynamics (**Alex, elaborate here**).\n",
    "Krizhevsky realized that the computational bottlenecks in CNNs (convolutions and matrix multiplications) are all operatiosn that could be parallelized in hardware.  \n",
    "Using a two NIVIDA GTX 580s with 3GB of memory (depicted below), he rigged an implementation so fast, \n",
    "that for many years his [cuda-convnet](https://code.google.com/archive/p/cuda-convnet/) implementation set the industry standard and powered the first couple yers of the deep learning boom.  \n",
    "\n",
    "![](./img/gtx-580-gpu.jpeg)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## AlexNet\n",
    "\n",
    "In 2012, using their cuda-convnet implementation on an eight-layer CNN, \n",
    "Khrizhevsky, Sutskever and Hinton won the ImageNet challenge on image recognition by a wide margin.\n",
    "Their model, [introduced in this paper](https://papers.nips.cc/paper/4824-imagenet-classification-with-deep-convolutional-neural-networks.pdf),\n",
    "\n",
    "In the rest of the chapter we're going to implement a similar model to the one designed  by Khrizhevsky. In their paper, they did some whacky things to make the model fit on the limited GPU memory. For example, they designed a dual-stream architecture in which half of the nodes live on each GPU. The two streams, and thus the two GPUs only communicate at certain layers. This limits the amount of overhead for keeping the two GPUs in sync with each other. Fortunately, distributed deep learning has advanced a long way in the last few years, so we won't be needing those features. In later sections, we'll go into greater depth on how you can speed up your networks by training on many GPUs (in AWS you can get up to 16 on a single machine with 12GB each), and how you can train on many machine simultaneously. Because these incredible speedups can be achieved without the dual-stream architecture. We'll leave that detail out. As usual, we'll start by importing the same dependencies as in the past gluon tutorials:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 103,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from __future__ import print_function\n",
    "import mxnet as mx\n",
    "from mxnet import nd, autograd\n",
    "from mxnet import gluon\n",
    "import numpy as np\n",
    "mx.random.seed(1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 104,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "ctx = mx.gpu()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load up a dataset\n",
    "\n",
    "Now let's load up a dataset. This time we're going to use gluon's new `vision` package, and import the CIFAR dataset. Cifar is a much smaller color dataset, roughly the dimensions of ImageNet. It contains 50,000 training and 10,000 test images. The images belong in equal quantities to 10 categories. While this dataset is considerably smaller than the 1M image, 1k category, 256x256 imagenet dataset, we'll use it here to demonstrate the model because we don't want to assume that you have a license to the dataset or a machine that can store it comfortably. To give you some sense for the proportions of working with ImageNet data, we'll upsample the images to 224x224 (the size used in the original AlexNet.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def transformer(data, label):\n",
    "    data = mx.image.imresize(data, 224, 224)\n",
    "    data = mx.nd.transpose(data, (2,0,1))\n",
    "    data = data.astype(np.float32)\n",
    "    return data, label\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "batch_size = 64\n",
    "train_data = gluon.data.DataLoader(\n",
    "    gluon.data.vision.CIFAR10('./data', train=True, transform=transformer),\n",
    "    batch_size=batch_size, shuffle=True, last_batch='discard')\n",
    "\n",
    "test_data = gluon.data.DataLoader(\n",
    "    gluon.data.vision.CIFAR10('./data', train=False, transform=transformer),\n",
    "    batch_size=batch_size, shuffle=False, last_batch='discard')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for d, l in train_data:\n",
    "    break"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(d.shape, l.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "d.dtype"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The AlexNet architecture\n",
    "\n",
    "This model has some notable features. \n",
    "First, in contrast to the relatively tiny LeNet, \n",
    "AlexNet contains 8 layers of transformations,\n",
    "five convolutional layers followed by two fully connected hidden layers and an output layer.\n",
    "\n",
    "The convolutional kernels in the first convolutional layer are reasonably large at $11 \\times 11$, in the second  they are $5\\times5$ and thereafter they are $3\\times3$. Moreover, the first, second, and fifth convolutional layers are each followed by overlapping pooling operations with pool size $3\\times3$ and stride ($2\\times2$). \n",
    "\n",
    "Following the convolutional layers, the original AlexNet had fully-connected layers with 4096 nodes each. Using `gluon.nn.Sequential()`, we can define the entire AlexNet architecture in just 14 lines of code.  Besides the specific architectural choices and the data preparation, we can recycle all of the code we'd used for LeNet verbatim. \n",
    "\n",
    "[**right now relying on a different data pipeline (the new gluon.vision). Sync this with the other chapter soon and commit to one data pipeline.**]\n",
    "\n",
    "[add dropout once we are 100% final on API]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 136,
   "metadata": {},
   "outputs": [],
   "source": [
    "alex_net = gluon.nn.Sequential()\n",
    "with alex_net.name_scope():\n",
    "    ########################################\n",
    "    #  First convolutional layer\n",
    "    ########################################\n",
    "    alex_net.add(gluon.nn.Conv2D(channels=96, kernel_size=11, strides=(4,4), activation='relu'))\n",
    "    alex_net.add(gluon.nn.MaxPool2D(pool_size=3, strides=2))    \n",
    "    \n",
    "    ########################################\n",
    "    #  Second convolutional layer\n",
    "    ########################################\n",
    "    alex_net.add(gluon.nn.Conv2D(channels=192, kernel_size=5, activation='relu'))\n",
    "    alex_net.add(gluon.nn.MaxPool2D(pool_size=3, strides=(2,2)))            \n",
    "    \n",
    "    ########################################\n",
    "    # Third convolutional layer\n",
    "    #########################################\n",
    "    alex_net.add(gluon.nn.Conv2D(channels=384, kernel_size=3, activation='relu'))\n",
    "    \n",
    "    ########################################\n",
    "    # Fourth convolutional layer\n",
    "    ########################################\n",
    "    alex_net.add(gluon.nn.Conv2D(channels=384, kernel_size=3, activation='relu')) \n",
    "    \n",
    "    ########################################\n",
    "    # Fifth convolutional layer\n",
    "    ########################################    \n",
    "    alex_net.add(gluon.nn.Conv2D(channels=256, kernel_size=3, activation='relu'))\n",
    "    alex_net.add(gluon.nn.MaxPool2D(pool_size=3, strides=2))    \n",
    "\n",
    "    ########################################\n",
    "    # Flatten and apply fullly connected layers\n",
    "    ########################################\n",
    "    alex_net.add(gluon.nn.Flatten())\n",
    "    \n",
    "    alex_net.add(gluon.nn.Dense(4096, activation=\"relu\"))\n",
    "    alex_net.add(gluon.nn.Dense(4096, activation=\"relu\"))\n",
    "\n",
    "    alex_net.add(gluon.nn.Dense(10))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Initialize parameters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 137,
   "metadata": {},
   "outputs": [],
   "source": [
    "alex_net.collect_params().initialize(mx.init.Xavier(magnitude=2.24), ctx=ctx)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Optimizer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 142,
   "metadata": {},
   "outputs": [],
   "source": [
    "trainer = gluon.Trainer(alex_net.collect_params(), 'sgd', {'learning_rate': .001})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Softmax cross-entropy loss"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 143,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "loss = gluon.loss.SoftmaxCrossEntropyLoss()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Evaluation loop"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 144,
   "metadata": {},
   "outputs": [],
   "source": [
    "def evaluate_accuracy(data_iterator, net):\n",
    "    acc = mx.metric.Accuracy()\n",
    "    for d, l in data_iterator:\n",
    "        data = d.as_in_context(ctx)\n",
    "        label = l.as_in_context(ctx)\n",
    "        output = net(data)\n",
    "        predictions = nd.argmax(output, axis=1)\n",
    "        acc.update(preds=predictions, labels=label)\n",
    "    return acc.get()[1]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Training loop"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "epochs = 10\n",
    "\n",
    "for e in range(epochs):\n",
    "    moving_loss = 0.\n",
    "    for d, l in train_data:\n",
    "        data = d.as_in_context(ctx)\n",
    "        label = l.as_in_context(ctx)\n",
    "        with autograd.record():\n",
    "            output = alex_net(data)\n",
    "            cross_entropy = loss(output, label)\n",
    "        cross_entropy.backward()\n",
    "        trainer.step(data.shape[0])\n",
    "        \n",
    "        moving_loss = .99 * moving_loss + .01 * nd.mean(cross_entropy).asscalar()\n",
    "            \n",
    "    test_accuracy = evaluate_accuracy(test_data, alex_net)\n",
    "    train_accuracy = evaluate_accuracy(train_data, alex_net)\n",
    "    print(\"Epoch %s. Loss: %s, Train_acc %s, Test_acc %s\" % (e, moving_loss, train_accuracy, test_accuracy))    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.4.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}