Exercise+3b.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Classifying tiny images with a Convolutional Neural Network\n",
    "======================================\n",
    "\n",
    "Outline\n",
    "------------------------\n",
    "This interactive notebook shows how to do image classification with a Convnet. You can edit code in the code cells, and run it with `Shift+Return`. The notebook is read-only, so feel free to hack the code, and reload the page if something breaks. The exercise covers how to:\n",
    "* Build a small convNet in neon.\n",
    "* Train it on the [Cifar10](https://www.kaggle.com/c/cifar-10) dataset. \n",
    "* Upload a new image, and classify it into one of the 10 categories.\n",
    "\n",
    "\n",
    "<img src=\"https://kaggle2.blob.core.windows.net/competitions/kaggle/3649/media/cifar-10.png\">\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Setting up a model\n",
    "==================\n",
    "The pieces we need to set up a model are described in the [neon user guide](http://neon.nervanasys.com/docs/latest/index.html):\n",
    "* The CIFAR10 dataset.\n",
    "* layer configuration and a  [model](http://neon.nervanasys.com/docs/latest/models.html).\n",
    "* a compute [backend](http://neon.nervanasys.com/docs/latest/backends.html).\n",
    "* an [optimizer](http://neon.nervanasys.com/docs/latest/optimizers.html) to train the model.\n",
    "* [callbacks](http://neon.nervanasys.com/docs/latest/callbacks.html) to keep us updated about the progress of training."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Start by generating the backend:\n",
    "from neon.backends import gen_backend\n",
    "be = gen_backend(backend='mkl', batch_size=128)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Loading a dataset\n",
    "-----------------\n",
    "We use the [aeon](aeon.nervanasys.com) dataloader to present the data to the model. \n",
    "\n",
    "**Note: This assumes the data has already been downloaded and ingested. If that is not the case, follow the instructions in the 02 VGG Fine-tuning notebook to process the CIFAR-10 dataset**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "ename": "RuntimeError",
     "evalue": "Unable to create internal loader object: config element {image: \"data/cifar10/train-index.csv\"} is not understood\nconfig is: {\"batch_size\":128,\"etl\":[{\"flip_enable\":true,\"height\":32,\"scale\":[0.800000011920929,0.800000011920929],\"type\":\"image\",\"width\":32}],\"image\":{\"flip_enable\":true,\"height\":32,\"scale\":[0.800000011920929,0.800000011920929],\"width\":32},\"manifest_filename\":\"data/cifar10/train-index.csv\",\"manifest_root\":\"data/cifar10\",\"minibatch_size\":128,\"type\":\"image,label\"}\n",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mRuntimeError\u001b[0m                              Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-17-9e14a768fbbc>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[1;32m     26\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     27\u001b[0m \u001b[0;31m# build train_set\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 28\u001b[0;31m \u001b[0mtrain_set\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mAeonDataLoader\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mconfig\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mbe\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     29\u001b[0m \u001b[0mtrain_set\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mOneHot\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mtrain_set\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mindex\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;36m1\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mnclasses\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;36m10\u001b[0m\u001b[0;34m)\u001b[0m  \u001b[0;31m# perform onehot on the labels\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     30\u001b[0m \u001b[0mtrain_set\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mTypeCast\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mtrain_set\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mindex\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mdtype\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mnp\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mfloat32\u001b[0m\u001b[0;34m)\u001b[0m  \u001b[0;31m# cast the image to float32\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/opt/conda/lib/python3.6/site-packages/nervananeon-2.6.0-py3.6.egg/neon/data/aeon_shim.py\u001b[0m in \u001b[0;36mAeonDataLoader\u001b[0;34m(config, adapter)\u001b[0m\n\u001b[1;32m     31\u001b[0m \u001b[0;32mdef\u001b[0m \u001b[0mAeonDataLoader\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mconfig\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0madapter\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mTrue\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     32\u001b[0m     \u001b[0;32mif\u001b[0m \u001b[0madapter\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 33\u001b[0;31m         \u001b[0;32mreturn\u001b[0m \u001b[0mDataLoaderAdapter\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mAeonLoader\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mconfig\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     34\u001b[0m     \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     35\u001b[0m         \u001b[0;32mreturn\u001b[0m \u001b[0mAeonLoader\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mconfig\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mRuntimeError\u001b[0m: Unable to create internal loader object: config element {image: \"data/cifar10/train-index.csv\"} is not understood\nconfig is: {\"batch_size\":128,\"etl\":[{\"flip_enable\":true,\"height\":32,\"scale\":[0.800000011920929,0.800000011920929],\"type\":\"image\",\"width\":32}],\"image\":{\"flip_enable\":true,\"height\":32,\"scale\":[0.800000011920929,0.800000011920929],\"width\":32},\"manifest_filename\":\"data/cifar10/train-index.csv\",\"manifest_root\":\"data/cifar10\",\"minibatch_size\":128,\"type\":\"image,label\"}\n"
     ]
    }
   ],
   "source": [
    "from neon.data.aeon_shim import AeonDataLoader\n",
    "from neon.data.dataloader_transformers import OneHot, TypeCast, BGRMeanSubtract\n",
    "import numpy as np\n",
    "\n",
    "image_config = {\n",
    "    'type': 'image',\n",
    "    'height': 32, 'width': 32,\n",
    "    'scale': [0.8, 0.8],\n",
    "    'flip_enable': True\n",
    "}\n",
    "\n",
    "# define configuration file for CIFAR-10 dataset\n",
    "config = {\n",
    "    'manifest_filename': 'data/cifar10/train-index.csv',  # CSV manifest of data\n",
    "    'manifest_root': 'data/cifar10',  # root data directory\n",
    "    'image': {'height': 32, 'width': 32,  # output image size \n",
    "              'scale': [0.8, 0.8],  # random scaling of image before cropping\n",
    "              'flip_enable': True},  # randomly flip image\n",
    "    'type': 'image,label',  # type of data\n",
    "    'minibatch_size': be.bsz,  # batch size\n",
    "    'batch_size': be.bsz,\n",
    "    'etl': [image_config]\n",
    "}\n",
    "\n",
    "from neon.data.aeon_shim import AeonDataLoader\n",
    "\n",
    "# build train_set\n",
    "train_set = AeonDataLoader(config, be)\n",
    "train_set = OneHot(train_set, index=1, nclasses=10)  # perform onehot on the labels\n",
    "train_set = TypeCast(train_set, index=0, dtype=np.float32)  # cast the image to float32\n",
    "train_set = BGRMeanSubtract(train_set, index=0)  # subtract image color means (based on default values)\n",
    "\n",
    "# build test set\n",
    "config['manifest_filename'] = 'data/cifar10/val-index.csv'\n",
    "test_set = AeonDataLoader(config, be)\n",
    "test_set = OneHot(test_set, index=1, nclasses=10)  # perform onehot on the labels\n",
    "test_set = TypeCast(test_set, index=0, dtype=np.float32)  # cast the image to float32\n",
    "test_set = BGRMeanSubtract(test_set, index=0)  # subtract image color means (based on default values)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Generating layers\n",
    "The core of the model is the layers. This can be as simple as a list, but merging and branching makes it easy to specify complex topologies."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "from neon.initializers import Uniform\n",
    "from neon.transforms import Rectlin, Softmax\n",
    "from neon.layers import Activation, Conv, Pooling, Affine, MergeSum\n",
    "\n",
    "# This is a simple convnet with a one conv layer,\n",
    "# max-pooling, and a fully connected layer. \n",
    "#\n",
    "# input - Conv - ReLu - Pooling - Affine - ReLu - Affine - Softmax\n",
    "#\n",
    "layers = [Conv((5, 5, 16), init=Uniform(-0.1, 0.1), activation=Rectlin()),\n",
    "          Pooling((2, 2)),\n",
    "          Affine(nout=500, init=Uniform(-0.1, 0.1), activation=Rectlin()),\n",
    "          Affine(nout=10, init=Uniform(-0.1, 0.1), activation=Softmax())]\n",
    "\n",
    "# We can use a MergeSum layer to combine differnt layers in parallel\n",
    "#\n",
    "#             - Conv3 - ReLu - \n",
    "#           /                  \\\n",
    "# input   -                     Sum - ReLu - ...\n",
    "#           \\                  /\n",
    "#             - Conv5 - ReLu - \n",
    "#\n",
    "conv3 = Conv((3, 3, 16), init=Uniform(-0.1, 0.1), activation=Rectlin())\n",
    "conv5 = Conv((5, 5, 16), padding=1, init=Uniform(-0.1, 0.1), activation=Rectlin())\n",
    "\n",
    "\n",
    "layers = [MergeSum([conv3, conv5]), Activation(Rectlin()),\n",
    "          Pooling((2, 2)),\n",
    "          Affine(nout=500, init=Uniform(-0.1, 0.1), activation=Rectlin()),\n",
    "          Affine(nout=10, init=Uniform(-0.1, 0.1), activation=Softmax())]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The Deep Residual Network\n",
    "A resnet module is a MergeSum layer containing a main path with conv layers, and a side path with a SkipNode() configured as the identity function. This allows earlier layer activations to bypass a series of layers.\n",
    "\n",
    "We use some helper functions to succinclty define the deep network:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from neon.initializers import Kaiming, IdentityInit\n",
    "from neon.layers import SkipNode\n",
    "from neon.models import Model\n",
    "\n",
    "# helper functions simplify init params for conv and identity layers\n",
    "def conv_params(fsize, nfm, stride=1, relu=True, batch_norm=True):\n",
    "    return dict(fshape=(fsize, fsize, nfm), \n",
    "                strides=stride, \n",
    "                padding=(1 if fsize > 1 else 0),\n",
    "                activation=(Rectlin() if relu else None),\n",
    "                init=Kaiming(local=True),\n",
    "                batch_norm=batch_norm)\n",
    "\n",
    "def id_params(nfm):\n",
    "    return dict(fshape=(1, 1, nfm), \n",
    "                strides=2, \n",
    "                padding=0, \n",
    "                activation=None, \n",
    "                init=IdentityInit())\n",
    "\n",
    "# A resnet module\n",
    "#\n",
    "#             - Conv - Conv - \n",
    "#           /                \\\n",
    "# input   -                   Sum - Relu - output\n",
    "#           \\               /\n",
    "#            -  Identity - \n",
    "#\n",
    "def module_factory(nfm, stride=1):\n",
    "    mainpath = [Conv(**conv_params(3, nfm, stride=stride)),\n",
    "                Conv(**conv_params(3, nfm, relu=False))]\n",
    "    sidepath = [SkipNode() if stride == 1 else Conv(**id_params(nfm))]\n",
    "\n",
    "    module = [MergeSum([mainpath, sidepath]),\n",
    "              Activation(Rectlin())]\n",
    "    return module"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The model is a collection of resnet modules between an input conv and output pooling and affine layer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Set depth = 3 for quick results \n",
    "# or depth = 9 to reach 6.7% top1 error in 150 epochs\n",
    "\n",
    "depth = ...\n",
    "nfms = [2**(stage + 4) for stage in sorted(range(3) * depth)]\n",
    "strides = [1] + [1 if cur == prev else 2 for cur, prev in zip(nfms[1:], nfms[:-1])]\n",
    "\n",
    "layers = [Conv(**conv_params(3, 16))]\n",
    "for nfm, stride in zip(nfms, strides):\n",
    "    layers.append(module_factory(nfm, stride))\n",
    "layers.append(Pooling('all', op='avg'))\n",
    "layers.append(Affine(10, init=Kaiming(local=False), \n",
    "                     batch_norm=True, activation=Softmax()))\n",
    "model = Model(layers=layers)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Cost function\n",
    "--------------\n",
    "The cost function compares network outputs with ground truth labels, and produces and error that we can backpropagate through the layers of the network.\n",
    "\n",
    "For our binary classification task, we use a cross entropy cost function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from neon.transforms import CrossEntropyMulti\n",
    "from neon.layers import GeneralizedCost\n",
    "\n",
    "cost = ..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Optimizer\n",
    "---------\n",
    "We now have a cost function to minimize by gradient descent. We do this\n",
    "iteratively over small batches of the data set, making it stochastic gradient \n",
    "decesent (SGD). There are other [optimizers](http://neon.nervanasys.com/docs/latest/optimizers.html) such as RMSProp and AdaDelta that are supported in neon, but often simple gradient descent works well."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from neon.optimizers import GradientDescentMomentum, Schedule\n",
    "\n",
    "opt = GradientDescentMomentum(0.1, 0.9, wdecay=0.0001, \n",
    "                              schedule=Schedule([90, 135], 0.1))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Callbacks\n",
    "---------\n",
    "To provide feedback while the model is training, neon lets the user specify a set of callbacks that get evaluated at the end of every iteration (minibatch) or pass through the dataset (epoch). Callbacks include evaluating the model on a validation set or computing missclassification percentage. There are also callbacks for saving to disk and for generating visualizations. Here we will set up a progress bar to monitor training."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "ename": "NameError",
     "evalue": "name 'model' is not defined",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mNameError\u001b[0m                                 Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-5-73cd53afcb3d>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[1;32m      4\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      5\u001b[0m \u001b[0mvalmetric\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m...\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 6\u001b[0;31m \u001b[0mcallbacks\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mCallbacks\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mmodel\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0meval_set\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mtest_set\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mmetric\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mvalmetric\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[0;31mNameError\u001b[0m: name 'model' is not defined"
     ]
    }
   ],
   "source": [
    "# set up callbacks. By default sets up a progress bar\n",
    "from neon.transforms import Misclassification\n",
    "from neon.callbacks.callbacks import Callbacks\n",
    "\n",
    "valmetric = ...\n",
    "callbacks = Callbacks(model, eval_set=test_set, metric=valmetric)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Training the model\n",
    "------------------\n",
    "Now all the pieces are in place to run the network. We use the fit function and pass it a dataset, cost, optmizer, and the callbacks we set up."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "ename": "NameError",
     "evalue": "name 'model' is not defined",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mNameError\u001b[0m                                 Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-6-672e37fa64ac>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[1;32m      1\u001b[0m \u001b[0;31m# And  run the model for 10 epochs\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      2\u001b[0m \u001b[0mepochs\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m...\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 3\u001b[0;31m \u001b[0mmodel\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mfit\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m...\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[0;31mNameError\u001b[0m: name 'model' is not defined"
     ]
    }
   ],
   "source": [
    "# And  run the model for 10 epochs\n",
    "epochs = ...\n",
    "model.fit(...)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Congrats! If you made it this far you have trained a convolutional network in neon.\n",
    "\n",
    "Evaluating the model\n",
    "--------------------\n",
    "We can now compute the misclassification on the test set to see how well we did."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Check the performance on the supplied test set\n",
    "from neon.transforms import Misclassification\n",
    "\n",
    "error_pct = 100 * model.eval(test_set, metric=Misclassification())\n",
    "print('Misclassification error = %.1f%%' % error_pct)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "By increasing the depth of the network and the number of epochs, we can improve the performance to match state of the art.\n",
    "\n",
    "This was quite a lot of code! Generally, to set up a new model from scratch it is best to follow one of the examples from the neon/examples directory. It's easy to mix and match parts!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Inference\n",
    "=========\n",
    "Now we want to grab a few new images from the internet and classify them through our network."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXQAAAB4CAYAAADi1gmcAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJztfXmUJddZ3+9Wva379Tbds3bPPiPJGm0e2VpHQrIWGy8C\nhWAbB4dAYsISlgQwBkLAHMxxDoSTxDEBjglx8IIxxuAFg2VJli150eaRRtIsmq1n7eltuvt1v379\nlqrKH99373erXk2rF6ln5vn+zpmp11W3qm7dulX1+3YVRREcHBwcHC5/eBe7Aw4ODg4Orw7cC93B\nwcGhReBe6A4ODg4tAvdCd3BwcGgRuBe6g4ODQ4vAvdAdHBwcWgSX7QtdKfXjSqmHlrH/Tyqlnng1\n++TgsBwopa5SSu1VSk0rpX7pYvfH4fLDZftCj6LoU1EUvfli98NhcVBKDSql7rvY/bhE8esAHoui\nqDOKoo9c7M44XH64bF/o80EplbnYfXBwWAK2AHgpbYNSyl/hvjhcAJfy++WSf6ErpX5DKXWUxdD9\nSql/wetjKhOlVKSU+g9KqcMADlvrfkkpdUwpNaaU+iOlVOo1K6X+p1LqlFKqpJR6Vil1p7Xtg0qp\nzyql/or78ZJS6o3W9n6l1N8ppUaVUse/X8RlpdQmpdTn+brHlVIfVUrtUEo9yn+PKaU+pZTq4faf\nALAZwJeUUjNKqV+/uFdw6UAp9SiANwH4KI/Np5VSf6qU+opSqgzgTUqpbp6Do0qpE0qp39bzWSnl\nK6X+mMf8uFLqF3j+X7Ivn4uFxc5b3mdQKfUBpdQ+AOVLdlyjKLqk/wF4J4B+0Mfn3QDKADYA+EkA\nT1jtIgBfA9ALoM1a93VetxnAywDex9uS+78XQB+ADIBfBXAOQIG3fRDAHIC3AfABfBjAd3mbB+BZ\nAL8DIAdgO4BjAN5yscfuNb4vPoDnAfx3AEUABQB3ANgJ4H4AeQBrAHwTwP+w9hsEcN/F7v+l+A/A\nY9b8/DiAKQB7eI4VAPwVgC8A6ASwlefzv+P2PwtgP4CNAFYBeJjnf+ZiX9el9G+Z8/Y5AJv0++VS\n/HfRO7CEG/IcgB++wAv9nkTbCMAPWn//PIBH+Hds/5TzTAC4gX9/EMDD1rZdACr8+xYAJxP7/iaA\n/3uxx+o1vg+3ARh9pRcGgAcB7LX+di/0C49V8oX+V9Y2H0AVwC5r3c+AdO4A8CiAn7G23ede6Klj\nvJx5+28vdv9f6d+lKTZYUEr9BIBfATESAOgAsBpAkNL81CusOwFi+2nn+VUA7+PtEYAuPo/GOev3\nLIACi11bAPQrpSat7T6Ax9OvqGWwCcCJKIoa9kql1FoAHwFwJ4hJeqCPo8PiYc/d1SAJ8IS17gSA\nAf7dn2if9iw4LG/eXvJjeknr0JVSWwB8DMAvAOiLoqgHwIsA1AV2SUsducn6vRnA2ZTz3AngAwDe\nBWAVn2dqnvPYOAXgeBRFPda/ziiK3raAfS9nnAKwOUWX+GHQfbg+iqIukCrLHkeX3nPhsMdqDEAd\nRCA0NgM4w7+HQOoWDXveOwiWOm+By2DuXtIvdJCOKwKJSFBK/RSAaxd5jPcrpVYppTYB+GUAf5PS\nphNAg8+TUUr9DoihLwRPASixwaSNjVPXKqVuWmQ/Lzc8BXqJ/FelVFEpVVBK7QGN5QyASaXUAID3\nJ/YbBtkZHBaBKIoCAJ8F8AdKqU4mO78C4JPc5LMAflkpNcDGvA9cpK5e6ljqvL0scEm/0KMo2g/g\njwF8B/QiuA7AtxZ5mC+AjJbPAfhHAP8npc1XAfwTyMh0AmQAXZB4xQ/aAwBeD+A4iEn9BYDuRfbz\nsoJ13TsBnARwGmS0/j0AN4IknH8E8PnErh8G8NtKqUml1K+tXI9bAr8Icgo4BuAJAJ8G8Je87WMA\nHgKwD8BeAF8BkZQ01eT3LZYxby8LKFb4tySUUhGAK6IoOnKx++LgsJJQSr0VwJ9FUbTlFRs7tAwu\naYbu4OCwMLC6721KqQyrDH4XwN9f7H45rCzcC93BoTWgQGqDCZDK5QAoNsLh+wgtrXJxcHBw+H6C\nY+gODg4OLQL3QndwcHBoEaxopOhHvniQ9TuWvz7/TKp+0iJ6wjAEAHjK2sr7hbxUvM3eX69LnsP+\nu+n8UXq7WH/syzBtaGXG+lT6/DtoUJvAtAlNG8+jRj//4HULCWZqwqaiFwFAw+pqGNLtrUV5AEDk\n5ehckSTuU5HH56fT+tbZC1na1ka7IZNpcJuqaZMB/Q4C2tbgcchZA6DvTb1B15vPyfmzenA4ZiMM\n9bI5Fmm4TOtOl2rNA7AARFG0pLEFgM1XbIoAQM1zCJWyyczHtI2JNlB6PgTWNlpGZn7z3ykxLqnz\nVPGx9Cn4APp+A4DiuadSnzr97CSWMnUR8L06vv/Eksb35LnRpo6bZ9xcv17GHkw+Py0bgXSq0aD5\nWKvXaVmlZbUmc7cyN8fLSmw5y0sAmKtQm1qN5lyDj2f3RY+bn6F5nclmTZtsnh6eQr4AAGjP5802\nvS5XoGVbjra18T4AkON1N1y9Y0Fj6xi6g4ODQ4tgRRm659EXLEZWDPOgb4tmHvYnWzf3deZbix5E\nzC48vQdvs9NHJ1lSGpPR67QUEGPvScmAN9mSQrLfCVkgdiGe/jOWyXfJ5JH6HTBbscJIopDYRFbR\nMseny+fkXDlmFR5PhdDKcBE0aIdqjRjDWJ3atOWEIa/r1OegvzUxzyi5R3qYCkxcPGubz6KQZoyR\np8df+qjZ+nRFX9xFyCQQ6rM1n08lWKSKzQvENqYxdWHfzbMnQoLhqzgrtPdv6qx9TN3cLNPYvL4Q\ne17G+yTPqbXb0gUfAEB3e1tzV5I/otiC/9DSOXhpMXR+ZjVrr9UbsSUgrHuuyssEYweA2UqcvWtW\nDwDVKrH9OrP2gKWC+pxIAQEfu5ah/Wbzwt7zzNDbmKG3F2gcqm0F06a9rR2LgWPoDg4ODi0C90J3\ncHBwaBGsqMpFRUZuvTBMG8vqYsROVqeEsi3kb5K28WgR09490qoWI7bqfeR7ZrQ5/jyVvvT+fH5l\nXUiYFHtDS/fB7TO+Fpe9eH+sa1wqIkViWkOJ+Nrg42d9us0NPVaBPTi0yLLxrKCk336eDWgZWrcu\nUwYAtBdEbNWGU93/EM332EvQBt+zjaLUt4hvWBhpkV52GinRtnJDDFIrDa2Km8+4mYaExuAVGulz\npKgEtcpD/x3XPeheNu0fJVQ0lilUfiWvKeYRkOhHpFWLtlopxHKQzTQ/cwsZ5vlCaEw/jSqV1tct\nnWQ90GoYWldl1Um1KirFOVafpKlcZvn3XDW+tPfXapioQecIqzKHK3xereop5+gchZwYTsMGZe3e\n84brL3yxFhxDd3BwcGgRrChDD+r662a7HTJ7NkbFqKmN/uWlMJcgjLMDwzYi+RJrl8AQKvZ3YJ3D\nT7jt2azSmIWM8NBIbAFUqA2+2qjXzFqEXXEb2MxkeSxnLiTDZU3KICJi9ouQDDEN7mNosXD4xBg8\nj64p9C2GrmhdxmOjaoaOYxs8PZ9+53ipohQWHfdAQ8ZiZOZe8JBqSSewxuZMSbtENh96xWCMcrbB\nkaWJefu1mE6nGFwTBvV5I7tTDJ7N7Hue3edxGhD3xTTr5PJ4oTaIL1b6SSLe73h/teCdycg58hE9\nH0GO7mm9QfN7riBug9pAWayScXKuKgZP/bvGrpCa8VcqwuLPDRPDHhkfAwBMnj9vtpWmpukcNXpm\n9PuprU2kbO2Q8RPvfHCeKxc4hu7g4ODQIljZEnSh1udZX9IEexN+3hz0EzWTd8OXPK0jNAEQ0sa4\nHerPF+uQY4FBAbMt9im0pQHjIqZZt2FLth4xiPU1jrhLY2j2s3XZy2PoXoO+9r2ZabOuwHc3z4y4\nwAE9bZbrVCHHrJvZvGcNbhDRtlpAOj1W+WGkJG0qAbGLzX20cX0X68QDy/+RxR49ppmYzpSlJ9ar\na5fGoZJICsMz9FslJwtegbG+itDM3D6dii6kV7fnrm7cHPDmJWd9CkFN6s7TzmH6Y46TogNP9HHB\nbNg0i7tdxt03lzd3zRmieSSLBWC+fdLmiba7eexr63PgXcYSz7M8Lys8Z7M5eWV2dnQAAGZm6Jl7\nft9zAIDvPPFN0+bQgf18Ljpmf/8Gs62rm2rotLFE0FEsAgC6u6WUQiYn0sJC4Bi6g4ODQ4tgRRl6\nmMbQjWVe/x0PMAKAMIgXXfF96wuqA4hUku3YDFuvijNtL41JaR2uzcRMTEdcf2j3Ua5J61VtL4L0\nwA+blKtF6VqbUcjQ1933hX0rrUPPkh4wytEytMLy68xGGnx+3/7Es+LR0+H83N+uvDQq8vj39NHf\nHR20rV4TS7/iwKYMyIqf92xGF9d16jly4sS4aVHT8yYe18Xr5tP7vopI83JJupyYvqCpjYpPHTqk\n2Y+OrYOsQqTNXR4zHYBl9y1xnBhPXRDTTUjAKdvkLy2JWlLWMsd+IfdusYz9QseMBX0l031oxm55\nuuW4jZdhG5Uvr8ynn3oSAPD5v/ssAODQ/pcAAGvWrjVt7rr7bgDA6173OgDA2jVSd358nPTphw8f\npXNwB6oNed9NlMYucIXpcAzdwcHBoUXgXugODg4OLYKVNYpGWlxrNqKI7SYeEBBvFBfLCWwwi8e3\nJMRAbZSMZ/aLy+5peUIQO5+oaC7sWqmNZ/aX0ks0bxaol++y1d1DIl3eSovR4OPP1jhwQY//nBgs\n6/VZAEAQVLlH0vOeTnbZYsNpVxcZcTZvWGPa9LN42dVBbl0NNpKOjY+aNtNTJFpWa5MAgFxO7n+7\nItWMx2qYM+epP6cmxP1RC8BaFaFi6ixavtamUcmsJ2gyhSZUL4BlpDeqFysPkW6j89ekGEC1mi9M\n7K8sQ2Qoky+2D3VFz3WtmtRtrHN4ySu5sEqsOUBJgpeWijqrGHyv+XlIqtTSn5P5VEaJlmmvFV4a\nFZjVKMvZEY8dobLEn/rEJ8y2xx79GgCgt5f0jT/6rncDAG688UbTRhtO0955g8dPAADODQ0BAFaz\nOsYOqjt/fuKVL8qCY+gODg4OLYKLwtBj33dNrU2QRorrlWmaZgCLG6KU+VpbwS/KS2yL70u/wlgX\n7UyISolzJCCMUaVzbV7aLCmeJVIHEMTYxjINSwVmyp4nzHa2UgIAlMqc6W12BoDkigYkj3nIrpxt\nOcnutmUtuU/t3DYAANi2fQv9vXOnadOzitroHNDaRXRsTIyaQ0NnAQDnzp6kc9XLZlu9Tqy9PSKD\n6dFzxObLdSsjIy81O4xSDFuvtXF0vgydC5GuZF7LuoRHoLj1xuhzslFyaT0qJrNiihuhZryGqTe7\nNsqUt8Y3jDdPOg8kD7UUjE7SvMxn5XWU499ZNlBq9h7L425+qsTfAukmSzpWX7XkHWgnCTbwz1rB\nQ//wmc8AAD7/OTJ8Hjt21Gx781t+EADwIz/6owCA3p5uqzeEgB069HMxU541255++hkAwNTUFABg\n0+ZN1NZKjaH8xb2iHUN3cHBwaBGsbOh/lHQxtEiFrEHTGv5KG54cWFVDEueQiiqyf5jQETbnn05x\nbbSOrL96vkq4WNpfe6MkTWFHCclE2N6rE5ABAPtH6MvfCCx3sirpvINGD3cj7loJQGZAhn5s3SZK\n+LfctR0AsOtaWq7tJx3fqlWrTBtfM3Pt7slVidZskACKLVu3AgAmp3YBAIaHTpttk+MUGj1+mpjP\nyMxxAEC3VdlFs7Mad79sMai0VBGvBSJN7ezEV/rMCX2vrcOOmmijHYzGa/jYOhd8aOeSN3r1uLEg\nbiFiGw+nb/Bic1C7AavY3/H0APqX1s9bfTZScbLX9v7LG/uzw8MAgIJ1z/OcoCrH8yvLjD1judz6\nuhZCWlUoE/ynl81ihN5P50w/cPAAAOBj//ujps3xY4cBAEPnaJ729Yn96N3vJp15X+8qPgfnXq/J\n+6nO0rCWig8cPGS2DbHuPMfBQ+MTpC+fmxOX38WmmncM3cHBwaFF4F7oDg4ODi2ClVW56B+2wZK/\nKeLOpcVH+dY0zpEYrtgAp9j9DQDC/DoAgK+jIbXKxdq/SehNLQMWd8eyRbQwqSpJUblEif3SC+5q\n8VdnJmx2D1sq2upkWJmYtkpWhZwTwieVi5cjFUyYlfwQfo76O7CG+nb3bil/9fobtgEA1m2kMe7u\nI5WLl5Fo1GT9tQwfz29YBi6OUC12Un/6+iSSrjxDhtvBblq3Z5b2myqXTJuRUYqWe/7wET5VmtFQ\nd+e1MY5qo1aa5c0EIOvc4ynG7vnc7YxbrJlXdo4f3t8k/KdFLittMpyvvsKSeiwrKbvj6jJ+ijNo\netajL2UCUhwRjIrHt7sMz8rYGS5TczgySi6ueStvSZ7VL1odkdWqF8tI6OtIZl3kOmWItVpFD9+5\n8zKvvvMyvVeeeIZysBx47J8BAOcnJCNinl121/WRCvGa664z2558+nsAgJkSHfPU6VMAgGpFStjp\nvs2WyRFgzsqnrq3QuoTdi8+/CADo6JGMqT19vc0XNQ8cQ3dwcHBoEaxsxSJdncaq5mNyeCSMP6H1\nJa4+Tl/OYkCGifIbbzfballyj+saoIoeGZ++6DHDlD5/oj9pRE9sqTbLooX5+qW4oCUL7Npujz5X\nKqo29G58jWhmckvF67dtBAAMjothaahM+V2qIEOn8mnZ1SZ9u5Ztl/feQMz+rjuuNNumS2Sk+ecv\nfgUAsH4LuS3uvukNpk3fag6GSPbf+lu7g2mWGloM0mcj18CmrQCA4bNnAACHD4t7Vw50Tz3jziVG\npxRh6TVByHM27s6asLwlCjkDYhRVhqk3H9s3uW10wn1r7Lw4/c3nqB/vf59IYpvWUZ8++UViit/c\na7l8GvbKUqG2iSpxXTWZSnUvrMEMGrQ249F52/L0XJZn5UK6CvFcS4tFid1pczUrCygz2SxLg16W\n3Ret94Iwc83CpU8ZpuTHRuj98OVnDgIAnnjpiGkzep4kfY8lot7N9A6pbJR7POPR+bZvWU/96ZZx\nPzDJxmx0AgCCPnYe6OowbdryNHfzvoTHmWOz8VMz9Bw/C/lOybY4HiyOczuG7uDg4NAiWFmGzstY\nJkXzxYpTl0ZdXHf6H/xxAMBt3aSbCotdZtscf91O1ulLPjitmX4K/dZ68nn0mU37AMgksuiFafpu\no2tkVyiL5uSYoXa107JeJ3ZUsAIpMil1FReDtb10zO4OOe/WCvWlxK6Ene3UZnOvsIybXk9BQzt2\nkK5u444tZtuRg8SSZsZIh31ojBj70YMHTZu77rsHAHDlLnJJ1IEfGStjXcQSis4E6Wekj2MTFHT0\nnccfpnMcPAYAKOREjwivwMek/RVED2lS3Otz4bVBlLDx0Do9n3TgGDO2lOChlFg203lTwYqlgJxV\nMeett5NL3PgYufaNjtEBtvfJ+G5YS2N19w10li1rxMa09wgd8yzvV635iauQZ0Xr2Tf0ybatG6hv\nb76Z7sETT1Lfjp0QVv7TPyR66aXAuLxatoM5rgoW1dk202DXZStwTjFD1/nLp61anl94ntwNH9pH\ny8os1+u0nrledr+dZXfeKrPpHatFb722h941PZwHfcM62bZ5NY17zujyee5nRdLQ7sxz/MzXGnKN\nmwokTWeL9DweLlEfH35BpIgX9r0AAPjz/4QFwTF0BwcHhxbBCjN0+qrbHixBkB667Vl6xJk86aj6\n7rwNANBdlG7rVlfM0pf7se8NAgBePiOsIRM30KcESchvEyBkb+Luah2XMjnPpVExTydZ1Z7lpTCo\n3i76Eq/uIpajrfPKE5YzPSPh8EvBGFvmazVhr1n2LlnPHgOrukmnvn2TsPArryIl+mYO5y9Y2b0G\nuF0PB1P09pIecWpKGODDXyL9eqFA+11xLTH15oRP4s0RTQvLeva7zwIATp0gT4etW8iLYHhixrRp\n76T9crZ3zQpDIvBt2wxfT9KTJWaciUuFMVODCSjSniisy+2UuXP1DvL+Odm9GQBw7x4KwNr7giQ/\nOz1M4/mJL7EEE4kO9823E2u99gq693/4cWozMikdyXKytBt3Uj/e81ZhkVpl/OT3aF499jS1efcd\nMnc3bVjea6Q9T/PUfi/4PF6+0sFDcX05raPl4Gmyu3z6WQnLH52h/m7rIRYerCJmPTorz0c7s/Vb\ndlLI/XYWTezkf2u66d0z0EtMXQchAcBYiZ7Z0QoFunms3+/2xY4V8jUVumggO4siHR+eof0eOUT9\n37ePApvqQ6fkGkOr8tcC4Bi6g4ODQ4vAvdAdHBwcWgQrqnLJsHHQNholJfMoxeBY4+rEp86SUa5v\np1ht6iwCFdhocfu1JD553pBpc3KExHetBtBl1nzLEKmNklodkrF0Lj5nJGxjA+4AF0LOZ6SvpRkK\n7JksUx87V0kuk30vUuDCc0fI7XLzABkiVVXE5v/yod+n/U8cbrr+hSDHLn25gojbAbuqeT67nLGI\nee1VV5g2/Rupn9rgCCu4pL2DRMdNW0n1Ume3sg1topYJOJ96uURj7LPbqGeNrTb66TVznhXUkiFR\nds2aHQDEXTWIROWy+w3kJrl+I7lI/r+/+2uzrR4YX9AY5is1tjRcOIOhMYinuCbKuZtdGpXsSEse\n+6kZCUxZ10di+XveeS0A4J8epjw4n/naOtPmzltITfXjD+4DAHzrSVGHqJBULWuK1P+7bySVQ7FD\nDK/XXUF3pr+PHoz2gqgMRsdJZTMyQeO8cTUd5+SY3MOzZ1llg6Whq5PUGvY9M4ZmE0/FrolWjcSj\nR0jF8r8OkNF+KCfl3TJrqP0Uj+2tG8iA+d4d8lzuWEPrzkxQkeeXT9HzmbXu34lRep6fPk7vk/Gq\njG2F1SldrG5cz2qVKuT+bWY1a5ei+3hmSIKWDjxHBs+2kxTgdFed2jzVLobXkmcVOFgAHEN3cHBw\naBGssFGUGaNFmAyP05WKUowfet3pUTJC7L5KMp4FnHmvERGTKBbpi3jvG7eaNjrjGZihKs5+GJam\n5TgTZEQNq2ywswpT+wU6ZqGH2O8Tz78MAJj1xcBx8GUqEPuxz/w9AOCBN91ttukKP5l2Tl3g0/FW\ndQibvnPPvVgOShwElM3aYfm0aMtQP9ewYad/QFhKW5EYgK/dwaxMlmjQ2OrsiuOcq7xnlUhI23e9\nHQCwdisZTOE3u4Yqvsva1pQriNvpTXe+CQDQt54MQWMjxLZqVkbF9jZqr1MXBMHyAlmWgjSWL96w\nzMxTMiHK/nqftLQT7AjAf0/PyrX/+WfIhe2qq4iRDwywq131nGnz8Lcpg99v/QzNr4+8RcLTZzmF\nQlQniaf9aXqGPv+QBG59m9JyY+t66sGeG2R8r95Bz8x73k6s/V/9EC0LeZm7n/0yGcnf0XzZC0J7\ngZ8jW3JP1Aww86kq7PeZCs2rY5tI4swGYkBcx9Lpv7mC5uUuZsqHhiVP/6eeoXEbn6ax0MbQijW/\nqtyPgFMQhNYrcz2/z/o9knquLtD4ZctyjnPHyN201E7P2fGjYrjtZueC0RF6dp/fejMAYOsOCe5b\n6y1urjuG7uDg4NAiWFGGbkL+7cALo4fUIdKB/Sc3oi/xufPEMmaqos/s4i9flZPflM9QVZx6RViO\nz/mKZ/7xywCA0pPfBgCE7IYHAJve+xMAgI4t/bRPh+iuPO1myH/fyky3XJZz3Lqb9Mxvv/tWAOLi\nCABdXaQjbAQ6gIOrl0wL29h15VVYDkqc5KpusYs2tivoCkFZn1wTc+2SgCvgwCzNrC3PMXisVy/w\nGA9sIftEzzrR3xa5nmKWdeZRoEPMRUeLhHtfIxL9erZITG89HzvHbpM2I57j4JKRYWLvSlnTNoq7\ndb1WFYuawvvtcyWYuc3Ck/ms4/3T4xLGTuH7stMLx4j9fegj3wIA/NYvUiDX7/+GsLiREQrGauNn\n50/+Ythsu3sPsfYtrF7+2R8jN8gfuEnc9548QL+feYnm86lhmQQ338AJpEJalqd5fkUy968csBLC\nLQE653ncJTHOzHVO/EMVud/fYWbex8/neitL2K0BPVuqRuz7vx0k1nx8XFxusxxs5FXo+rWtzH52\nMxzk1RNS25vb5Rxb8nTDplnSf/6lQQBAwbIxZbnOQHWObBmjZRn3s90055/uJevDxm30DvEjkZIn\nrffYQuAYuoODg0OLYGUZ+jzkKTJ1/TiQAPIlDgPSj8/yqv0PPWa2rTn+PADg9EuUerI2RnpeVZSq\nOup+qv0XzdJxovX0Jcz0Wqkpz5Bzf3iKdLlB3UogViHJIGRdvGIWnC1YLJ5ZZ3ED1/bMyNCWRogV\n6CvSof+1ulxjJmcz2sWjrToCACjYn2j+uLflyZrfU6A+2QxQX5OxGWTlADqpVr1GDL/QSWNqM3zF\nDDlSrGPUoc2Wq4AulFSrUdt6IBPB5+CnTJ629W4gCQkFOccI6+67u/n8GRmrWS1hvMbZuXRyLTtt\nhaHfyVqc80x0ldJRE6Ck/7ZovXYWenwvMc0/+hNSeH/oAw+aNnfvIonpU39NgSkf/bzox0+cJsnt\n936apMoZHvtt2ztNmyuvpLG+/Tqa58cHhUUeOUhj/7kn6CK3baH5/Y4fEE8YlZu64PUuBIWCDiyy\nGDqPpfY+Oz1NffrdLz1u2nScIq+eW1eRlGdXPDrBz+N+fsWdzdA56nmRJkqcUqKuvWw4rfQ1gbDi\n3RVKTfHg7msAAHlJAo5DnAKjby1J+qvWk22qKpZBHJ0iZv/QOfak2SDS7RzbhnQm4sEZGv+Jukju\nD24Re+FC4Bi6g4ODQ4vAvdAdHBwcWgQrrHLRxp94rrc4SC1SmRTDTrlMxjBw8Mzn2LUNACZHyAg6\nOUtiZrGXRMLbbtlj2mxit7v1V5MRZU0f/V0sFk2bNhb7zj1FuUVmRiTop52zr2mbbmOaA5WsQtTV\nOTr/qvUkUq2+aqvZFiTsafrypy2j6PGTg1gOdq3Xxisx2ujzFAs6MIpVTnYbFTfU2um3/SyJsFuu\nJoNtLk/jlYnlVGFjKourNc5j7RfkHtc4YGJmcpb7aJ2DrbAdbHgNiuz61S33JuI87m0HycCXy4o6\nZra+vBwGiZYwAAAXvklEQVQ4C4Z2TUxZZ6pk8WpbLdOUg9/e3ayM7IW5FwCgk1bq3CaPfo/cFf/z\nH/+DafOhXyQD6R03kgj/r98keYxu5vwsOqtpjZPyV8bFtW7oLLUfr9E9XLVaVBenhuk+dHbSPX/H\nPaRW8DwxLm7fJRWoloJ2NiKqFGeJPBtMj7xAAXfVr3zStPEq5O53kvO1lMsyF8IgXpxZP7xFi8O2\n66pImyiP+eQD7wEAHFy/1bQpDNIz43G+lYo196YzFCBYGiK13zjndhmalKC4Ue3ueAXl4kGPuOxm\n+ZnxOL9MbZTuiVcXo+g3HiNjOH7qh7EQOIbu4ODg0CJYUYbeYOYRNwzpajacE5n/zvZsNi16e7dz\nG/o72iSf8nWazTToq9Zo0Bd12GJuo2c4T/IJcuqfY8f/9jar7iUzgbDMBsyzp822DjYCrmbjx9qN\nVB2ou12CK9qLxMzPc33BxlkJ/NAh+IdfpoCkvVw78PFvP2LaDJ4kg9b3vi25xheD3m7qS80yOOpi\njzqsP6wSQ67NitGs0KaZMFcTsrLJRRn63d5GTEa7jtn5GiYmiF1895knuS0ZmG69/QY5ToHOMcUu\nZ488+m2zbaZE433nbeTuubqfjUZ28BkbxvIcgFLIiGFL4Tw3f61rFjUj6SKp57Xtqtjk7dhcCCtW\nHaBpjQ460rVEOd3E43tlfv3HD9N8/sm3EVN+7w+LBJVli9ux08Qa165hKacgY7huK92f1Rx8VLIY\n5s419Fzd/gZ6BmtTFKY+fF5C2G++9xYsB1l2r42H/tMywzU9O9lNsjdjVfzRFax4qaxqRlrKCfhA\nDR5Su85CUOG0FeM0d7vOkrTf+LGfM22e3ELG0GfYZbZ2xqpnPEUSAqbYKDyjJSMrGGgtufUqPlfU\nKwx9YBWN+/lz5NBQnSTDaWnfd2X/l77JP/4QC4Fj6A4ODg4tgpXVoXvNX2JNWYImviIsJQjrKfvF\nW5mq4KwXKyj5EubYhVDXGdS6/MBio1rXFnSzy9bqa822c+P0BR2qUT+qL5I+zbeZ6hi5O54epERc\nhZzo2m58w10AgENHid3MVtgNsGubtLltd9O1LQbdnaTnr8yJK2SVc6PnWFcXMDuJrKrkUYMDudj9\n0LOCdgITJMR/s/QTeXKO85zb/B8eptQHBZZatm7qN20GrqaoljlFutKvPivssjFLTO+666mvvXxO\nW9LQ/fDZXbGrU1xSR6cpaVIQLS5v9KuL+dwUm38JwsSWCx/HPBX8DOQsM8YLg3Ttv/6nNC9v2CT3\n8MfuoPvx0FM09m+7nZa3X28Fbg0T6zx+ihjnnOW2t20HzdE617Y8fpTu3fHT4tp46PReAMDP/sEF\nuz8vcnwxNrvUz3qWk3F1c7qJTVdfb9oMD5JevaOda+daFYuqNZ6rifoGyoqc81gbEPG6Wa4p0Ps3\nf2baRPdTwOFElaWBMbHfgSVdxc9ZVNPjZjH0Bj9rnFJDnZd0I4PatbnCNUVP0vXckRE30P577sZi\n4Bi6g4ODQ4vAvdAdHBwcWgQrqnLJaeNHmmVIq0N0Sj5L+mwk3B0zlvFDl62S3A/N2cmUNpCEuhit\nVjNIGx2l5ufIWKRd9ACg2Lma+83H43MalygAuS5y5zp8mDK4jYyKaHbrm8ntb8/ON/H+ieLCsIsQ\nLw2ZHInWqibqlCy7Fxa4+G3I0a+z0+LW1rmGjJARq7UiS40UKr4+7qY20IWW1W/tGjKCvv/X/j0A\noF6m83e2WcZVPkwn5774zZ+TKEePb3QbF46uch8DS+XC9m5UWWVUq4u4H66QqiW1rjjiZRPN2sg2\n7iUjTJsNf2apMwym52uM/2Wd09MZLnn/Z60Czie5FOMt22nMDh6mv7/zrIzb4AgN8BTvdvv1UqD7\njbeTofX0KS7z9nXOf2IV+n7pn8nFeKkql/ZMPFeSfS3abXGaXRIfPSZGyXfccR8td9PzdfKUlG47\nzZHfI8PUtwk24k6VZO6X+HfG16UlCd2rxQ2zOkQqvQmdw6YqDgXa6cDcWx2B7Vn6MFYBe5y5NDp7\n1myKOI+Rjoq9p5PuyY0brjZtvvLVr2IxcAzdwcHBoUWwogzdNy5cNjXW6/jr5jcbTjPMrENj2bBz\nPjAT5GP6yfzJgPmShogXng2SqfAgLB5WoVjN/vWXWHGQQiYrOUVmuGKRz1WIVq+SHAz5jtWx/cOU\nfN5pBt/FQHkcJJGzGGCGmFdHUbsd0vpqWVhKjQOitKSTsZLVK3a3jHyWcJgV1qyKOlFE+1+xnYpD\n57vI3XR2TALD/Awx61V9NA4bt1r5NDhT3eAgMZeGzi5nZWScm6N142NknC7NSFDMSkElgohonfnF\n2/TY2dJW/L7aU7+pylLaejPlm0oyWfuxYwHP2bwV1JXj1PdfOERjWN1L2wpF2b+LbcyzUzRPSt8V\no/NbbiFJ89NfoftznGOOdluG15+60cqJtARkdPRUSmCRrny1nt3/gokzpk3pJEmH63/wTgDA+Qlx\npeztIffM7k5yjujgjKf9/VIL4JkXyFX4y0/tp3OyNDDkyfycmmBGrnOtW5KRYsnXZ+nWn6V3QH5W\n+tHFYk+Rpd3TA+JsUWYj/0auZhScpdz3Q/5G06a9fXGZLB1Dd3BwcGgRrGxgEetFrbKA8BP6K8Nu\nLCajdPizDrEOheEm9Y2GQdlUSOdUNhnzCHbub/Ph9ZuZcsTfPaV1+fp4SpjYmjUUdNTwSP+4YeB1\nZlueszKaSkkm97vtsLa8wBjNsAttVha8UEsinIec3aqGjx8xbTLs8tW3nlhB4MvYah26DtgI2c1L\nZ00EgLkZ0ml6Z+gauzkPetXK4zyzf5DabN3Ox5WBL3E6gAYfc5Z18JVZOcf4GLnTDZ87AQCoNywJ\nITkQrzFs6VLYerIXzW65Jlujl96KjsfrrZQSMkUS57Djx/iW5Tj55833if1n4w5yn50cIylpusTB\nZTUrQIfV0iefJRZ7zy3icvviyzTmL1eoI/fuJsZ78xZhjj1dIk0tBcYGYM0L47bIDP2FgzRnOzrk\n2g4/Q2HxH8/StXR1d5ttjUAH1em+8d8bxZ12dScd69QRCvgD28+y3VKRK7OGnos62206Z0U6bJ8h\niTGYIimmXiFpc/ducUHu7qQ+Pc/1SucKkuVSsXFoZ5HuTSenvzhx4qRp88ADD2AxcAzdwcHBoUWw\nogxd14IMLZqiP8qaLaswRb/Mq8KwmY8lfQe0njy1KpKs4J3t3NasH+eOhPY2w8ri57eTKK1ZTQz9\nXe/7HQCAb+VD1x4kJsGT1g9a/QrnSxa/AGj9rV1TNMOBXA0OfGhw9ZbxMydMm2I36T/bOTw/7BBm\nnC128rUQ64/MoAoj09415QnSH9YblF6hWpV7PDxEDDuviKVkuJI6IDnhy8zMq5weoGIx/FPHKeDi\n7EliaVEgXi7CkedLg7V8KLYtRBZ7FrYen0+xmkT6vur5bXszGb28bquZqtUkOcP5z6AhZ2nroN97\n3kpscN0mydM/yakZ5rhW7hxXrc/U5B7e00+M/Jo7yMaxbpUkh/rq16ldeZzmx+Bx2n+qIcnrsu10\n/ruwNHg6dD8+crF1Q9Pk5bLxZkm6N3OMmPQ3XiCG3WFJzFWuYRAxCy5w8NJjjwiL189o/vwg9UNL\nBUVh2DOTpA/vK9Ez01mW666wbafOkm+eE/yVrKpE9Szdi/GtFBCVXS22NZ/zn/cF9Fxu2UkVxdb3\nD5g2nd2Ls084hu7g4ODQInAvdAcHB4cWwYqqXDQCS1XBqUTgh1om5Q22aKpFUq2ySJGmm0RvT0TK\n1PJhSAQ4aYk21GJws/ueLo+n1SMqlnucs+KxysbOOW4diE+lj22pN5arImBx3XY71PagDAc3aJVX\nrSbqjNETFAg1ywad/iskV0Y3cnxMEiV1QFRk5UNXPh1buzJOcF73Wk24Qphjdzh2O6xb457jnPQB\nj60OLDp9SrJdHtj3BG0rkytkzgp+8lkFMsvXX08J4Hk11C9aJRhaxzJTpKkStPzUY2ZUeinBZMnA\npNjck4lJC10p0PJmu+leUnutWkf36+xZUQvUWYVVmmQX2xIFkr17j9zD1d10vu5euk9P7Zf9Jz0y\n2F1/E8+ZDs6102apC8UOvzSk3TJGlVVyv/leCkZ756AYDMc5B8qREzRXHvuGlKcbmiLX3Dl2xJiY\nJDXfEZ6DAFAbpzb5/q0AgM6NWwAAwz1bTJscl4PLd5IxdbouheWrbFgO+V2j1SuHipbhk3PQZNfR\nfu2WKra9TNc2dJjcJ09x3YDbbrvdtFkfLc7g7Bi6g4ODQ4tgRRm6NiJ51nfEsHVNu42FyN6Rl8xS\n5g3C0cw8xYBqiJ1hS/Y5vNg2myXp31FCNLANmSq5n2pmWWL0au6bzdyWAu3+6Vs+oYpdtXRllIzH\nBjGL4TYq5HI1wSHiVjYD9AwQO+nr3wQAyHdQwIlnpUVocPmhgLNLZruJOtpX6CligyaTgC18sU9q\nmXNBH+eMlIdffNK0OX36BQDAXJWMSJYQgna+lA4+ow4MF8fG1KmwaCiVcu+ixA+lkhvmZZ8K8fYy\nnZqlC22MDfn8O2+QNj2cQn58glzq8pZL4jWdFMb+ucfp/hQ5oOipI7Zjgg7nJ8NfOZB86NU8B830\n8PnZxTCyottVbnluixKIleIyzIOiK4rtvnaX2aaNmPffRLn3b90pLol/+fFPAAC++E8UOt/gcHyV\nF4NxppNcjL0iuWJ6G8gY6YeWBLqK2pRCCkjKBvKAFPje5Pl5KjD7bs9LwGGRnz2/QYbS6owYTCcP\nPgUA2LyNjr3zKnJ1npwQp4HRYQmkWggcQ3dwcHBoEaysDt1ULEpx3eJvS2RYuNVE0mJxG1u/HV+n\nTH1HW5HJp4/i+kjPYgRJ3fl8yZjS/m6uXmO7JJpfunWs77Rlecm5DHO0klX5Oo1CIuVCJmvlhGYX\n0jl2aRw5+bzZduwIhUQPXE0Vaa66gZbBrLAMP0dsPcrQUidSa1hRW1FI7Cpk4lIvS4Kjaa7yMsnM\n5RTr4Pfv+5ZpE5i6kDq9g1x3LsGKu/keF+zx914Fip6GhIdscg5Qr6JYm1haAKODT7o7ek1tdLKy\njTvoSOu2SZNymfTcQYXcDicn5UiDnFSqcy1X4gIx7lPn5dFvY1fVRpbzetupAwrsYsv3TptPrPx4\nxqVzqdBCuuc1P9emjU47YdXb1NB2qw2bZVC8DrIrVFkvXWwnKaI9K+NfLlHirWiSWPD6gJjxtX3r\nTJvKDI1faYbcJgPLtqdr6+oUINmMTlsifYvYblVlY+H5IbENlcfp/CMdtEP3Wjrvo19/1LS5544f\naLre+eAYuoODg0OLwL3QHRwcHFoEK6py0cKKl2I0SuaLThNfLRlV1iRULCrg0lO2kVHpfDEJ1U3s\n0PPl40j0ImpWmSR1NDGjqta5GLWI3iXN8rs0eGxpjELLXVAboVmU9T09DpbYyL6NWc5vPTstuSrm\npkjMPEZeVeju6+WlRLKFbFCLOP1HnWXzkSkpteUpUrHMclmwyXEpsZXNUXtdLProAVK11Ock77XP\nOpZ8hvpfqYtaSVxJOcc9D27BGs7sMtVZgNze2D1P2u9TbKLqAvObjhVXE2qVkuVxC8XRvn0bOXPm\nWmqz95vSqLuHizyvIpXLTEVy8Y+EZODMFdlAnqPjhGLXxhzbCX3O1Jm1MnZm8noO0d/awO7HEiEt\nb+6GRhVrHdOLP2Np7wO9bZKjSD/2yU+ZbY88+nUAwJXXUHbD++9/MwBg7RqJ1Dx8jKKav/stmnP5\nAvlf3vfA202bHs7SODpKrpwjo+L2qHOsl9hFcqZMYz1blgL1c5zNVJfHU1YeIo8dGA4eosLw23Zc\nAQB414/8S9PmjJXjfSFwDN3BwcGhRbCybos6OMJel/ghX2Rp0/x1bg4I0vlFDNmJsXjtLsnbjBui\nHCaj4pWP5jO8pvcr6dZm76/P58U22blg0lwZFwcdpGIZajnjnE4/orfYX3E9Fsbt0cqHkc/R79k6\nBfQ88cjfAgDWDlxn2qzZeA0AoKyIuUxzQNFzLx2Qk7Br4uQssZOsEnbZnqFznD9JRaanR47ydYjh\ntaCIOdWZoSvfyiVT00Wl6W8/kfMEeAU314UiJethlNyYMLrHd0zrg3ZJjEtQhTaJGurfQds2X0UG\nuAPPkZHv5EGRUtrzZMw700aM0bOeap+NmPlOOv9aNqq2WQxdGzj1fnZMnJZq9ZAHATXKZjpMm2rV\nKpy8BMxxMFnWlxPrsdCVxDyveRx1/vdDx8jd8vCRo2bbwEbKknj1LnJzHD9PkufouPS1kw2n7UUa\n71Mn6Dhf+9pDpk1HJzH0GjPsilVgvVIh9j3HRd/nKuyaWJXAvTrvZ4rQNyRXVcQDXud8Mw9/jVws\n77vvftNmwwYJZFoIHEN3cHBwaBGscOg/uybCyrnN67wmXaMFrXP2mjmmSX9ujs1fdEtvql0hw+T3\ny2LYDW6v++HFdJ06ZD+hr7eDh5R2u2w+tgn1N0FHzWxjuTVFGyb7nj22rJtkJqDHz3b7S0pEGUs3\nWuwgdhJwbHflPLHwg/u+YdrsffZpAEA5IN3qGBPrM+clOEK7iwYBZ76z7k2Rg1I6ud5s1tN6ZNuJ\njw7q6/qjOeljG7M6n++tz+599mj63qvA0E0OezuYjMduAdJVsy4daGbtNAb9O2X9NbcQCyxP0BXV\nudhUh8SumHlZr5gVTSeusEmiwiaSAYnPQf/VCRZuUfQw4JWcxTMKiX2WSiIhtLVZnVkCZrlWbCYQ\nySvrxyXmtHtYnqP9hsdIv10simSzYYDsPHNzNHfOc7COzbB1zd2ZGQmkAoDhc0Pm9zgz+jq7SzYs\n+03AzFpv0+61gVWRLGzoGrm0zc4Ya1IG8H6VSbItfeazfyt9ZBvThz70+03XnwbH0B0cHBxaBCtb\nsShR8cdGYCq70N9eip5cf9z8mKeB1ssHdtN44i3+ndF65jS9pukb62nTzmGYmGYSdqIlfUjN5q1t\nCXeI0LBoSwpYppeLznedsbx7clrnzHpmj1lwvGoOdSpgfbvnS9hzewfljq7OkMeK9jLxLf12UCf9\n4WyZAzCmmcHU5HrquqIOs+/2rGzLMUsJA13TlNYXCsL6NCvU7MaeP3n2yGjU+Fx6jsglxr2qloiE\ngxLsv8RukaYvn+/cceloFXuyXLNHes/ZGjB4gNbNTWp7kLDAgI0k6YJIXIqYYweMo89Kv4I5Uqhf\ndyd5gFRCqUw/V6GBDUNa+jpvg2VrmZ1dXuh/jeduw6qWVUvo0DVsu9N0iZj1VInEloyV+CrHzFaz\nX1Mz165TwPemgyVR/c5RXszKRG1TJGhlJHWWQLm6UmSxcLGzKet/PjIfUzN7/V6y03fYdRUWAsfQ\nHRwcHFoE7oXu4ODg0CJY4cCiZpdEE7BhjJnx9bQfrzPuhpa7XzJbozFcpsifktSEF7bKw7d3h21W\nE0NW0hjafA6JLYkldOBtiQAnuyiut7xva8DGmtA6js9WrrxiVQuLeHXLdUobSOuBHltRdWS46nC9\nwnnM2bBTZyMWAGR4THJsYCoWOKudZdSsswhaZH1KMWNlNjHZ66hvOd5PZ4YEJFdIndVCDcvtMccX\nMGupIIDlq7CSMKKz5RYqtlAVWzH/uS1DOE+xDq5JvGuPdhOVXDfHnqL2w4fpHurh0ioqwAq3S5QU\noG1xV0rTfUuDcGwfGQrnymQxvfI2S4XCP6sVNj7zeFuVDtGwMhAuBVUu5eZZ7qieUbnowLEw1hYA\nxtgVsTRdirUFYF4yYRBXxdpttPEyx/n6/c4Udatuy2qRhmXw1PsHIatMwub89uIj0ax608Zno0bi\n03opxbIXCsfQHRwcHFoEKj3E3sHBwcHhcoNj6A4ODg4tAvdCd3BwcGgRuBe6g4ODQ4vAvdAdHBwc\nWgTuhe7g4ODQInAvdAcHB4cWgXuhOzg4OLQI3AvdwcHBoUXgXugODg4OLQL3QndwcHBoEbgXuoOD\ng0OLwL3QHRwcHFoE7oXu4ODg0CJwL3QHBweHFoF7oTs4ODi0CNwL3cHBwaFF4F7oDg4ODi0C90J3\ncHBwaBG4F7qDg4NDi8C90B0cHBxaBO6F7uDg4NAicC90BwcHhxaBe6E7ODg4tAj+PxN3afJUvmi5\nAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f171ea33588>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "%matplotlib inline\n",
    "import matplotlib.pyplot as plt\n",
    "from PIL import Image\n",
    "import numpy as np\n",
    "\n",
    "imgs = {'frog', 'airplane', 'cat', 'car'}\n",
    "\n",
    "# empty buffer to use for inference dataset\n",
    "# dims [minibatch, imgsize]\n",
    "x_new = np.zeros((128, 32*32*3), dtype=np.float32)\n",
    "\n",
    "# crop/resize images and assign them to slots in x_new\n",
    "# also display with true labels\n",
    "plt.figure(1)\n",
    "for i, name in enumerate(imgs):\n",
    "    plt.subplot(100 + (10 * len(imgs)) + 1 + i)\n",
    "\n",
    "    img = Image.open(\"data/{}.jpg\".format(name))\n",
    "    crop = img.crop((0,0,min(img.size),min(img.size)))\n",
    "    crop.thumbnail((32, 32))\n",
    "\n",
    "    plt.imshow(crop, interpolation=\"nearest\")\n",
    "    plt.title(name)\n",
    "    plt.axis('off')\n",
    "\n",
    "    x_new[i,:] = np.asarray(crop, dtype=np.float32)[:,:,(2,0,1)].transpose(2,0,1).reshape(1,3072) -127"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create a dataset with this image for inference"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from neon.data import ArrayIterator\n",
    "\n",
    "# create a minibatch with the new image \n",
    "inference_set = ArrayIterator(x_new, None, nclass=10, \n",
    "                             lshape=(3, 32, 32))\n",
    "\n",
    "classes =[\"airplane\", \"auto\", \"bird\", \"cat\", \"deer\", \n",
    "          \"dog\", \"frog\", \"horse\", \"ship\", \"truck\"]\n",
    "out = model.get_outputs(inference_set)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Get model outputs on the inference data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "plt.figure(2)\n",
    "for i, name in enumerate(imgs):\n",
    "    plt.subplot(100 + (10 * len(imgs)) + 1 + i)\n",
    "\n",
    "    img = Image.open(\"data/{}.jpg\".format(name))\n",
    "    crop = img.crop((0,0,min(img.size),min(img.size)))\n",
    "    crop.thumbnail((32, 32))\n",
    "\n",
    "    title = \"{} ({:.2})\".format(classes[out[i].argmax()], out[i].max())\n",
    "        \n",
    "    plt.imshow(crop, interpolation=\"nearest\")\n",
    "    plt.title(title)\n",
    "    plt.axis('off')"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}