2. Neural Network Distillation (MNIST).ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Knowledge Distillation of a Neural Network\n",
    "Demonstration of the Knowledge Distillation method on MNIST data\n",
    "- Train an **\"expensive\"** and **large** model on MNIST data to achieve a good generalization performance. This is the **teacher** model.\n",
    "- Create a much **cheaper and smaller** model but instead of using the actual labels, it uses the predictions of the teacher model on training data. These are the **soft labels** and this model is called **student**\n",
    "- Compare its performance with the same model trained on actual labels. It turns out that the student model performs much better on generalizing on test data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "%load_ext autoreload\n",
    "%autoreload 2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "import tensorflow as tf\n",
    "from tensorflow.keras.layers import Dense, Flatten, Conv2D\n",
    "from tensorflow.keras import Model\n",
    "\n",
    "from sklearn.preprocessing import MinMaxScaler\n",
    "from sklearn.metrics import accuracy_score, confusion_matrix\n",
    "from sklearn.model_selection import cross_val_score, KFold\n",
    "\n",
    "import utils"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## MNIST data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "mnist = tf.keras.datasets.mnist\n",
    "\n",
    "(X_train, y_train), (X_test, y_test) = mnist.load_data()\n",
    "\n",
    "X_train = X_train / 255.0\n",
    "X_test = X_test / 255.0\n",
    "\n",
    "# Add a channels dimension\n",
    "X_train = X_train[..., tf.newaxis].astype(\"float32\")\n",
    "X_test = X_test[..., tf.newaxis].astype(\"float32\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.image.AxesImage at 0x7f59f185d730>"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAPsAAAD4CAYAAAAq5pAIAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8vihELAAAACXBIWXMAAAsTAAALEwEAmpwYAAAOZ0lEQVR4nO3dbYxc5XnG8euKbezamMQbB9chLjjgFAg0Jl0ZEBZQoVCCKgGqArGiyKG0ThOchNaVoLQqtKKVWyVElFIkU1xMxUsgAeEPNAm1ECRqcFlcY2wIb8Y0NmaNWYENIX5Z3/2w42iBnWeXmTMv3vv/k1Yzc+45c24NXD5nznNmHkeEAIx/H+p0AwDag7ADSRB2IAnCDiRB2IEkJrZzY4d5ckzRtHZuEkjlV3pbe2OPR6o1FXbb50m6QdIESf8WEctLz5+iaTrV5zSzSQAFa2NN3VrDh/G2J0i6SdLnJZ0oaZHtExt9PQCt1cxn9gWSXoiIzRGxV9Ldki6opi0AVWsm7EdJ+sWwx1try97F9hLbfbb79mlPE5sD0IyWn42PiBUR0RsRvZM0udWbA1BHM2HfJmnOsMefqC0D0IWaCfvjkubZnmv7MElflLS6mrYAVK3hobeI2G97qaQfaWjobWVEbKqsMwCVamqcPSIelPRgRb0AaCEulwWSIOxAEoQdSIKwA0kQdiAJwg4kQdiBJAg7kARhB5Ig7EAShB1IgrADSRB2IAnCDiRB2IEkCDuQBGEHkiDsQBKEHUiCsANJEHYgCcIOJEHYgSQIO5AEYQeSIOxAEoQdSIKwA0kQdiCJpmZxRffzxPJ/4gkfm9nS7T/7F8fUrQ1OPVBc9+hjdxTrU7/uYv3V6w+rW1vX+73iujsH3y7WT713WbF+3J8/Vqx3QlNht71F0m5Jg5L2R0RvFU0BqF4Ve/bfi4idFbwOgBbiMzuQRLNhD0k/tv2E7SUjPcH2Ett9tvv2aU+TmwPQqGYP4xdGxDbbR0p6yPbPI+LR4U+IiBWSVkjSEe6JJrcHoEFN7dkjYlvtdoek+yUtqKIpANVrOOy2p9mefvC+pHMlbayqMQDVauYwfpak+20ffJ07I+KHlXQ1zkw4YV6xHpMnFeuvnPWRYv2d0+qPCfd8uDxe/JPPlMebO+k/fzm9WP/HfzmvWF978p11ay/te6e47vL+zxXrH//JofeJtOGwR8RmSZ+psBcALcTQG5AEYQeSIOxAEoQdSIKwA0nwFdcKDJ792WL9+ttuKtY/Nan+VzHHs30xWKz/zY1fKdYnvl0e/jr93qV1a9O37S+uO3lneWhuat/aYr0bsWcHkiDsQBKEHUiCsANJEHYgCcIOJEHYgSQYZ6/A5GdfKdaf+NWcYv1Tk/qrbKdSy7afVqxvfqv8U9S3Hfv9urU3D5THyWf9838X66106H2BdXTs2YEkCDuQBGEHkiDsQBKEHUiCsANJEHYgCUe0b0TxCPfEqT6nbdvrFgOXnl6s7zqv/HPPEzYcXqw/+fUbP3BPB12383eK9cfPKo+jD77xZrEep9f/AeIt3yyuqrmLniw/Ae+zNtZoVwyMOJc1e3YgCcIOJEHYgSQIO5AEYQeSIOxAEoQdSIJx9i4wYeZHi/XB1weK9ZfurD9WvunMlcV1F/zDN4r1I2/q3HfK8cE1Nc5ue6XtHbY3DlvWY/sh28/XbmdU2TCA6o3lMP42Se+d9f4qSWsiYp6kNbXHALrYqGGPiEclvfc48gJJq2r3V0m6sNq2AFSt0d+gmxUR22v3X5U0q94TbS+RtESSpmhqg5sD0Kymz8bH0Bm+umf5ImJFRPRGRO8kTW52cwAa1GjY+23PlqTa7Y7qWgLQCo2GfbWkxbX7iyU9UE07AFpl1M/stu+SdLakmba3SrpG0nJJ99i+TNLLki5uZZPj3eDO15taf9+uxud3//SXni7WX7t5QvkFDpTnWEf3GDXsEbGoTomrY4BDCJfLAkkQdiAJwg4kQdiBJAg7kARTNo8DJ1z5XN3apSeXB03+/eg1xfpZX7i8WJ/+vceKdXQP9uxAEoQdSIKwA0kQdiAJwg4kQdiBJAg7kATj7ONAadrk1792QnHd/1v9TrF+1XW3F+t/efFFxXr874fr1ub8/c+K66qNP3OeAXt2IAnCDiRB2IEkCDuQBGEHkiDsQBKEHUiCKZuTG/ij04v1O675drE+d+KUhrf96duXFuvzbtlerO/fvKXhbY9XTU3ZDGB8IOxAEoQdSIKwA0kQdiAJwg4kQdiBJBhnR1GcMb9YP2L51mL9rk/+qOFtH//wHxfrv/239b/HL0mDz29ueNuHqqbG2W2vtL3D9sZhy661vc32+trf+VU2DKB6YzmMv03SeSMs/25EzK/9PVhtWwCqNmrYI+JRSQNt6AVACzVzgm6p7Q21w/wZ9Z5ke4ntPtt9+7Snic0BaEajYb9Z0rGS5kvaLuk79Z4YESsiojcieidpcoObA9CshsIeEf0RMRgRByTdImlBtW0BqFpDYbc9e9jDiyRtrPdcAN1h1HF223dJOlvSTEn9kq6pPZ4vKSRtkfTViCh/+ViMs49HE2YdWay/cslxdWtrr7yhuO6HRtkXfemlc4v1Nxe+XqyPR6Vx9lEniYiIRSMsvrXprgC0FZfLAkkQdiAJwg4kQdiBJAg7kARfcUXH3LO1PGXzVB9WrP8y9hbrf/CNK+q/9v1ri+seqvgpaQCEHciCsANJEHYgCcIOJEHYgSQIO5DEqN96Q24HFs4v1l/8QnnK5pPmb6lbG20cfTQ3DpxSrE99oK+p1x9v2LMDSRB2IAnCDiRB2IEkCDuQBGEHkiDsQBKMs49z7j2pWH/um+Wx7lvOWFWsnzml/J3yZuyJfcX6YwNzyy9wYNRfN0+FPTuQBGEHkiDsQBKEHUiCsANJEHYgCcIOJME4+yFg4tyji/UXL/143dq1l9xdXPcPD9/ZUE9VuLq/t1h/5IbTivUZq8q/O493G3XPbnuO7YdtP217k+1v1Zb32H7I9vO12xmtbxdAo8ZyGL9f0rKIOFHSaZIut32ipKskrYmIeZLW1B4D6FKjhj0itkfEutr93ZKekXSUpAskHbyWcpWkC1vUI4AKfKDP7LaPkXSKpLWSZkXEwYuPX5U0q846SyQtkaQpmtpwowCaM+az8bYPl/QDSVdExK7htRiaHXLEGSIjYkVE9EZE7yRNbqpZAI0bU9htT9JQ0O+IiPtqi/ttz67VZ0va0ZoWAVRh1MN425Z0q6RnIuL6YaXVkhZLWl67faAlHY4DE4/5rWL9zd+dXaxf8nc/LNb/9CP3FeuttGx7eXjsZ/9af3it57b/Ka474wBDa1Uay2f2MyR9WdJTttfXll2toZDfY/sySS9LurglHQKoxKhhj4ifShpxcndJ51TbDoBW4XJZIAnCDiRB2IEkCDuQBGEHkuArrmM0cfZv1q0NrJxWXPdrcx8p1hdN72+opyos3bawWF938/xifeb3NxbrPbsZK+8W7NmBJAg7kARhB5Ig7EAShB1IgrADSRB2IIk04+x7f7/8s8V7/2ygWL/6uAfr1s79jbcb6qkq/YPv1K2duXpZcd3j//rnxXrPG+Vx8gPFKroJe3YgCcIOJEHYgSQIO5AEYQeSIOxAEoQdSCLNOPuWC8v/rj138r0t2/ZNbxxbrN/wyLnFugfr/bjvkOOve6lubV7/2uK6g8UqxhP27EAShB1IgrADSRB2IAnCDiRB2IEkCDuQhCOi/AR7jqTbJc2SFJJWRMQNtq+V9CeSXqs99eqIqP+lb0lHuCdONRO/Aq2yNtZoVwyMeGHGWC6q2S9pWUSssz1d0hO2H6rVvhsR366qUQCtM5b52bdL2l67v9v2M5KOanVjAKr1gT6z2z5G0imSDl6DudT2Btsrbc+os84S2322+/ZpT3PdAmjYmMNu+3BJP5B0RUTsknSzpGMlzdfQnv87I60XESsiojcieidpcvMdA2jImMJue5KGgn5HRNwnSRHRHxGDEXFA0i2SFrSuTQDNGjXsti3pVknPRMT1w5bPHva0iySVp/ME0FFjORt/hqQvS3rK9vrasqslLbI9X0PDcVskfbUF/QGoyFjOxv9U0kjjdsUxdQDdhSvogCQIO5AEYQeSIOxAEoQdSIKwA0kQdiAJwg4kQdiBJAg7kARhB5Ig7EAShB1IgrADSYz6U9KVbsx+TdLLwxbNlLSzbQ18MN3aW7f2JdFbo6rs7eiI+NhIhbaG/X0bt/siordjDRR0a2/d2pdEb41qV28cxgNJEHYgiU6HfUWHt1/Srb11a18SvTWqLb119DM7gPbp9J4dQJsQdiCJjoTd9nm2n7X9gu2rOtFDPba32H7K9nrbfR3uZaXtHbY3DlvWY/sh28/XbkecY69DvV1re1vtvVtv+/wO9TbH9sO2n7a9yfa3ass7+t4V+mrL+9b2z+y2J0h6TtLnJG2V9LikRRHxdFsbqcP2Fkm9EdHxCzBsnynpLUm3R8RJtWX/JGkgIpbX/qGcERFXdklv10p6q9PTeNdmK5o9fJpxSRdK+oo6+N4V+rpYbXjfOrFnXyDphYjYHBF7Jd0t6YIO9NH1IuJRSQPvWXyBpFW1+6s09D9L29XprStExPaIWFe7v1vSwWnGO/reFfpqi06E/ShJvxj2eKu6a773kPRj20/YXtLpZkYwKyK21+6/KmlWJ5sZwajTeLfTe6YZ75r3rpHpz5vFCbr3WxgRn5X0eUmX1w5Xu1IMfQbrprHTMU3j3S4jTDP+a5187xqd/rxZnQj7Nklzhj3+RG1ZV4iIbbXbHZLuV/dNRd1/cAbd2u2ODvfza900jfdI04yrC967Tk5/3omwPy5pnu25tg+T9EVJqzvQx/vYnlY7cSLb0ySdq+6binq1pMW1+4slPdDBXt6lW6bxrjfNuDr83nV8+vOIaPufpPM1dEb+RUl/1Yke6vT1SUlP1v42dbo3SXdp6LBun4bObVwm6aOS1kh6XtJ/Serpot7+Q9JTkjZoKFizO9TbQg0dom+QtL72d36n37tCX21537hcFkiCE3RAEoQdSIKwA0kQdiAJwg4kQdiBJAg7kMT/A65XcTMQuIbWAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.imshow(X_train[0, :, :, 0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(60000, 10000)"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "num_train = X_train.shape[0]\n",
    "num_test = X_test.shape[0]\n",
    "num_train, num_test"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "tf.random.set_seed(0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Teacher Model\n",
    "Train a large and compute-intensive model that uses Dropout and generalizes well on test data.  \n",
    "Here by \"large\" we mean a wide neural network with convolution filters and 2 fully-connected hidden layers.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "class TeacherModel(Model):\n",
    "    def __init__(self, T: float):\n",
    "        super(TeacherModel, self).__init__()\n",
    "        \n",
    "        self.T = T\n",
    "        \n",
    "        self.conv1 = Conv2D(32, 3, activation=\"relu\")\n",
    "        self.flatten = Flatten()\n",
    "        \n",
    "        self.d1 = Dense(1200, activation=\"relu\")\n",
    "        self.d2 = Dense(1200, activation=\"relu\")\n",
    "        self.d3 = Dense(10)\n",
    "        \n",
    "        self.dropout_layer_hidden = tf.keras.layers.Dropout(rate=0.5)\n",
    "        \n",
    "        self.output_layer = tf.keras.layers.Softmax()\n",
    "\n",
    "    def call(self, x):\n",
    "        x = self.conv1(x)\n",
    "        x = self.flatten(x)\n",
    "        \n",
    "        x = self.d1(x)\n",
    "        x = self.dropout_layer_hidden(x)\n",
    "        \n",
    "        x = self.d2(x)\n",
    "        x = self.dropout_layer_hidden(x)\n",
    "        \n",
    "        x = self.d3(x)\n",
    "        x = self.output_layer(x / self.T)\n",
    "        return x"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "T = 3.5  # Softmax temperature\n",
    "teacher = TeacherModel(T=T)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Train teacher model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "optimizer = tf.keras.optimizers.Adam()\n",
    "loss = tf.keras.losses.SparseCategoricalCrossentropy(from_logits=False)\n",
    "\n",
    "teacher.compile(\n",
    "    loss=loss,\n",
    "    optimizer=optimizer,\n",
    "    metrics=[\"accuracy\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1500/1500 [==============================] - 238s 159ms/step - loss: 0.1935 - accuracy: 0.9381 - val_loss: 0.0783 - val_accuracy: 0.9748\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<tensorflow.python.keras.callbacks.History at 0x7f79903ce2e0>"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "num_epochs = 1\n",
    "batch_size = 32\n",
    "\n",
    "teacher.fit(X_train, y_train, batch_size=batch_size, epochs=num_epochs, verbose=1, validation_split=0.2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Evaluate generalization of teacher model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_pred_teacher = np.argmax(teacher(X_test), axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.9752"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "acc = accuracy_score(y_test, y_pred_teacher)\n",
    "acc"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "248"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "int((1 - acc) * num_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The teacher model achieved **248 test errors**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAV0AAAFgCAYAAAAYbKrEAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8vihELAAAACXBIWXMAAAsTAAALEwEAmpwYAABRoUlEQVR4nO3de1xUZf4H8M+TpCQpImE1g5cZQCEuKhctzUvudpV0FRQr3VtlW1mW1abdTLPtYr+1i7VtW+26aql4Q9BNu2m7bQlomuANCExmXFFEBBnun98fM44MAzhkcwad7/v1Oq8YzvOcz3nOHL5OM8+co0hCCCGENi7x9A4IIYQ3kaIrhBAakqIrhBAakqIrhBAakqIrhBAa8vH0DjSlfC6j6tzNI9mDI/p4JFd4F0/OFVIezPY2hw4V4fjx4y0e8o5VdDt3Q5cBkz2S/fX2xR7JFd7Fk1M0lZKyq5XhQ+NbXSdvLwghhIak6AohhIak6AohhIak6AohhIak6AohhIak6AohhIak6AohhIak6AohhIak6AohhIak6AohhIY6fNG9cVgEdq97Fjlpc/H47250Wt/n6gBsevchZK6cg81/mwl9rx72dQseHo/s1KeQnfoUkm+KbXf2ls2fICZyACLDQ7Hw1Zed1tfU1GDqnSmIDA/FiGFDcaioyL5u4SsvITI8FDGRA/Dpls2S3cGzPT3mgZHhiIoIw2utZE+7cwqiIsIwcvi1TtlREWEYGBkux/sCyAZg/S64uxYAtwA4ACAfwOxztVeXBdF30IP2pWvsDBb8WMLwsc+xW/zD3H3gMAdNfMGhzZotO3j3s/+k76AHefO9b3B5+nb6DnqQv5rxDj/7Zh/94h5iz2sfZXZOEYOGP+bQt+liqaPDUlldT4PRyL0HClh+uobR0THcuTvXoc3rb77Ne+69j5Y6csmyj5k0aTItdeTO3bmMjo7hycpq7jv4Aw1GIyur650yWlskW9tsLXOrahsdlgpLHQ1GI3P35/NkZTWjo2O4Y1eOQ5tFby7m3fdOZ1VtI5cs/YhJyZNZVdvIHbtyGB0dw7IKC/ceKKDBaGSFpc4p48zijcfbU9mxsXFsrc657ZWuUqoTgLcB3ArgGgB3KKWuac82EqL6oeDwcRSZSlFX34DUzTuRODrGoU248WpsyzwAANiWdRCJo6MBABHGq/CfnfloaGhEVXUt9uSZcNOwCJezszIzERISCoPRiM6dO2NSyhRkpKc5tMlIT8Nd034DAJiYlIytX3wOkshIT8OklCno0qUL+hkMCAkJRVZmpmR30GxPjjk7yzE7eXKKU/bG9A2YasuekJSMrV+ezU6enOKQnZ0lx7ujZp/hzrcXhgDIJ/kDyVoAKwCMb88GdL38UXy0zP7YdLQM+iB/hzZ7DpowfswgAMD4MQPR/fLL0NPfD98ftBbZy3wvRWAPP4yK74/gqwJczjabTQgO7m1/rNcHw2QyObfpbW3j4+OD7v7+KC0thcnk3Ndsduwr2R0n26NjNpmgDw5us7+1jXN28/3W6fUwm+R4d9TsM9x5aUc9gMNNHhcDGNq8kVJqOoDpAIBLL293yJxF67DoyUmYOm4ovt6ZD9PRMjQ0NOLzb/cjLrIvvvzHYzheVont3xeioaHxp41ECCF+Jh7/II3keyTjScYrn8sc1plLyhF85dlXp/orA2A6Vu7Q5sixckx5/H1cd8crmLs4HQBQXmkBALz6wWZcO+VlJN6/GEop5P1Y4vJ+6XR6FBef/TfDZCqGXq93bnPY2qa+vh6nyssRGBgIvd65r07n2FeyO062R8es18NUXNxmf2sb5+zm+202maDTy/HuqNlnuLPomgD0bvI42PY7l2XnHkJonyD01QXiUp9OmHRzLDZu/d6hTWAPP/vFmZ/4/c1YkvYtAOCSSxR6+vsBAKLCdIgK0+Gzb/a7nB2fkID8/DwUFRaitrYWqStXYGziOIc2YxPHYfnSJQCAtWtWY9QNY6CUwtjEcUhduQI1NTUoKixEfn4eEoYMkewOmu3JMcfFO2avXrXSKfu2xNuxzJa9bs1qjBp9Nnv1qpUO2fEJcrw7aradG2cu+AD4AYABQGcAuwFEtmf2gu+gBzl+xts8WHSUBT+W8Lm3NtB30IN88a+bmDTzXfoOepB3PP435h06yoNFR/nh2q/ZPWEmfQc9SP8hM7m3wMy9BWZu3/0Dh0z+U6szF1qavWCpI9dt2MjQsDAajEY+P38BLXXknKefZeraNFrqyLIKCyckJdMYEsK4+ATuPVBg7/v8/AU0GI0M69+f69M3ufzpqmR7Jlur3JZmFaxNy2BoqDV77rwXWFXbyNlPPcNVa9azqraRJ05VccLEs9m5+/PtfefOe8GaHdaf6zZsbHXmQkuzF7zheHsqu63ZC8qdtw9RSt0G4HUAnQB8SPLFttpf0rUXPXW7nrIsuV2PcD+5XY93GD40Hjt2ZGt/jzSSmwBscmeGEEJcSDz+QZoQQngTKbpCCKEhKbpCCKEhKbpCCKEhKbpCCKEhKbpCCKEhKbpCCKEhKbpCCKEhKbpCCKEht34jrb0GR/TB19s983XcgKEzPZILAGXb3/BYthDuJl99diSvdIUQQkNSdIUQQkNSdIUQQkNSdIUQQkNSdIUQQkNSdIUQQkNSdIUQQkNSdIUQQkNSdIUQQkNSdIUQQkMXRNHdsvkTxEQOQGR4KBa++rLT+pqaGky9MwWR4aEYMWwoDhUV2dctfOUlRIaHIiZyAD7dsrlduTdeF47da55Czvpn8Phvf+m0vs9VAdj0lweRueJJbP7rDOh7+dvXvfjwOOxYNRvfrZ6D/3tiYrtyAc+N2VuzPT3mgZHhiIoIw2utZE+7cwqiIsIwcvi1TtlREWEYGBl+wR3vnzLm0tJS3HLjGAQFdMOjM2e0K7NptqeeawBo8b7sP8cC4EMAJQByXO0TGxvndP/4yup6GoxG7j1QwPLTNYyOjuHO3bkObV5/823ec+99tNSRS5Z9zKRJk2mpI3fuzmV0dAxPVlZz38EfaDAaWVld3+J96n1jH3ZYusbPZMHhYwy/fR67DXmUuw8Uc1DSiw5t1ny6k3c/t5S+sQ/z5vve4vKMTPrGPszRv/0z//tdAbvGz2TX+Jn8dvcPvPHeN50yziyeGnNLizdma5lbVdvosFRY6mgwGpm7P58nK6sZHR3DHbtyHNosenMx7753OqtqG7lk6UdMSp7MqtpG7tiVw+joGJZVWLj3QAENRiMrLHVOGWcWT4375xzzsbIKfvblV3xj8Tu87/4HWh2rp8ccGxvH1uqcO1/p/gPALee7kazMTISEhMJgNKJz586YlDIFGelpDm0y0tNw17TfAAAmJiVj6xefgyQy0tMwKWUKunTpgn4GA0JCQpGVmelSbkJkXxQcPoYiUynq6huQumUnEkdHO7QJN1yFbVl5AIBtWXlIHGVdTwJdulyKzpf6oEtnH/j4dEJJaUWHH7O3ZntyzNlZjtnJk1Ocsjemb8BUW/aEpGRs/fJsdvLkFIfs7KyOf7zPZ8x+fn4YNvx6+Pr6ujzOjjDmptxWdEl+BeDE+W7HbDYhOLi3/bFeHwyTyeTcpre1jY+PD7r7+6O0tBQmk3Nfs9mxb2t0vfxRfPSk/bHp6Enog/wd2uzJM2P8mIEAgPE3xKD75b7o6d8V2/cU4avsPBRuno/CzS/gs2/240DR0Q4/Zm/N9uiYTSbog4Pb7G9t45zdfL91ej3MpgvgeJ/HmM+XJ5/rMy6I93Q7qjmL1mNEbAi+Wf4ERsSFwnT0JBoaCGPwFRhguBKht85FyC3PYXRCGIYPMnp6d4UQHYDHi65SarpSKlsplX3s+DGn9TqdHsXFh+2PTaZi6PV65zaHrW3q6+txqrwcgYGB0Oud++p0jn1bYy4pR/CVPeyP9Vf2gOlYuUObI8dPYcoTH+K6uxZi7tsZAIDySgvG3xCDzD1FOG2pxWlLLTb/dx+GxvRzKdeTY/bWbI+OWa+Hqbi4zf7WNs7ZzffbbDJBp78Ajvd5jPl8efK5PsPjRZfkeyTjScYHXRHktD4+IQH5+XkoKixEbW0tUleuwNjEcQ5txiaOw/KlSwAAa9esxqgbxkAphbGJ45C6cgVqampQVFiI/Pw8JAwZ4tJ+Ze/9EaG9g9BX1xOX+nTCpJtisXFbjkObwB5+9oskP/G7G7Fkw7cAgMP/K8OI2FB06nQJfHwuwYjYUOwvdP3tBU+N2VuzPTnmuHjH7NWrVjpl35Z4O5bZstetWY1Ro89mr1610iE7PqHjH+/zGfP58uRzbeeu2Qu2GQz9cJ6zFyx15LoNGxkaFkaD0cjn5y+gpY6c8/SzTF2bRksdWVZh4YSkZBpDQhgXn8C9BwrsfZ+fv4AGo5Fh/ftzffqmVj9VbmlWwfiH3uXBoqMsOHyMzy1Op2/sw3zxvX8x6ZH36Bv7MO944gPmHSrhwaKj/HDdf9l96KP2mQ9/W/0f7vvhCPcWHOEbS79odeZCS7MXtBpza4s3ZmuV29In7GvTMhgaas2eO+8FVtU2cvZTz3DVmvWsqm3kiVNVnDDxbHbu/nx737nzXrBmh/Xnug0b2/VJvlbj/rnH3KdvXwYEBNDPz486vd5p5kNHGHNbsxeUu26loZT6GMBoAFcAOApgLskP2uoTFxfPr7dnu2V/zkVu1yO04I23rvHGMQ8fGo8dO7JbDHfbPdJI3uGubQshxIXK4+/pCiGEN5GiK4QQGpKiK4QQGpKiK4QQGpKiK4QQGpKiK4QQGpKiK4QQGpKiK4QQGpKiK4QQGpKiK4QQGnLb14B/CsJz39P25PUPAoY95pHcE1+/5pFcwHPfifc0bxx3o+cuvYBOHfBwyytdIYTQkBRdIYTQkBRdIYTQkBRdIYTQkBRdIYTQkBRdIYTQkBRdIYTQkBRdIYTQkBRdIYTQkBRdIYTQ0AVRdLds/gQDI8MRFRGG11592Wl9TU0Npt05BVERYRg5/FocKiqyr1v4ykuIigjDwMhwfLplc7tzYyIHIDI8FAtbyZ16Zwoiw0MxYthQp9zI8FDERA5ody4A3HjtAOxOfRI5a+bg8V+PcVrf56oAbHr7D8hc/hg2/+V+6Hv5AwBGxoXg22Wz7EvZv1/G7aOi2pX9U493aWkpbrlxDIICuuHRmTPaPeYz2Z445p58rr0x+9PNn2BwVDhiIsLwfwtbzv31XVMQExGG0dc7nmO33jQGV/bshlkX2DlmR9ItC4DeAL4EsBdALoCZ5+ozODaOVbWNDkuFpY4Go5G5+/N5srKa0dEx3LErx6HNojcX8+57p7OqtpFLln7EpOTJrKpt5I5dOYyOjmFZhYV7DxTQYDSywlLnlFFV20hLHR2Wyup6GoxG7j1QwPLTNYyOjuHO3bkObV5/823ec+99tNSRS5Z9zKRJk2mpI3fuzmV0dAxPVlZz38EfaDAaWVld75RxZvFNmOWwdB36GAsOH2P4+AXsdt0T3H3QxEGTX3Fos+azXbz7+Y/omzCLN9//DpdvzHLaztW/eJqlJ08z4Ponndb5Jsxq8Ticz/E+VlbBz778im8sfof33f9Ai9tv7Xhrfcw7Qq63ZFfWNDos5VV1NBiM3LMvnycqqhkVHcOsXTkObf78xmL+/p7prKxp5N+XfsSJyZNZWdPIoycquOWLr/j6W+9w+h8ecNp288VTY46NjWNrdc6dr3TrATxG8hoA1wJ4UCl1TXs3kp2ViZCQUBiMRnTu3BnJk1OQkZ7m0GZj+gZMnfYbAMCEpGRs/fJzkERGehqSJ6egS5cu6GcwICQkFNlZmS7lZmU65k5KmeKUm5GehrtsuROTkrH1i7O5k1KmOORmZbqWCwAJkX1QUFyKIvMJ1NU3IHXLd0gcGenQJtxwJbZl5QMAtmXnI3Gk86vZCWMGYss3+2GpqXM5+3yOt5+fH4YNvx6+vr4u5zXlqWPuyefaG7OzszJhbHaObWzhHDuTO2HihX+ONeW2okvyCMmdtp8rAOwDoG/vdswmE/TBwfbHen0wzGZTC216AwB8fHzQ3d8fpaWlMJtNCLb9HgB0ej3MJse+reY266vXB8PUrK/ZbEJwb+dck8m5b/N9bosuyB/FR0/aH5tKyqEP8ndosyfPjPE3RAMAxo+ORvfLfdHTv6tDm0k3DcKqLTtdzgXO73ifL08dc08+196Ybd1ms3Ospdwm55h/9wv7HGtKk/d0lVL9AAwGsL2FddOVUtlKqezjx49psTsXhTlvpGNErBHfLJ2FEbFGmI6eRENDo339VYHdEBlyNT795oAH91II0Zzbi65S6nIAawA8QvJU8/Uk3yMZTzL+iiuCnPrr9HqYiovtj02mYuh0+hbaHAYA1NfX41R5OQIDA6HT6VFs+z1gfYWm07v2Yrt5X5OpGPpmfXU6PYoPO+fq9c59m+9zW8zHyhF8ZQ/7Y30vf5iOlTu0OXL8FKY8uQTXTfsz5v7lXwCA8spq+/qkXw7Chq17UN+kELvifI73+fLUMffkc+2N2dZtNjvHWsptco6Vn7qwz7Gm3Fp0lVKXwlpwl5Nc+1O2ERefgPz8PBQVFqK2tharV63E2MRxDm1uS7wdy5YuAQCsW7Mao0aPgVIKYxPHYfWqlaipqUFRYSHy8/MQnzDEpdz4BMfc1JUrnHLHJo7Dclvu2jWrMeqGs7mpK1c45CYMcS0XALL3HkZo7yvQV9cTl/p0wqSbBmPjv3Md2gT6+9kviP3Eb3+BJemO7y1NvmkwVm35zuXMM87neJ8vTx1zTz7X3pgdF5+Agmbn2G0tnGNnctetvfDPMQdunL2gAPwTwOuu9mlp9kJVbSPXpmUwNDSMBqORc+e9wKraRs5+6hmuWrOeVbWNPHGqihMmJtMYEsK4+ATm7s+395077wUajEaGhfXnug0b2/Vp+roNGxkaZs19fv4CWurIOU8/y9S1abTUkWUVFk5IOpu790CBve/z8xdYc/v35/r0TW1+itzSzILxM9/jwUMlLDh8jM+9s5G+CbP44t82M2nWB/RNmMU7nvwH8w6V8OChEn64/lt2H/aEvW//cS/QdPQkLxvyWIvbbmv2wvke7z59+zIgIIB+fn7U6fVOMx/aOt5aHvOOkusN2S3NKliz3naOGYx8bt4LrKxp5JNPPcOVq9ezsqaRx8ur+KuJyTQarbl79uXb+zY/x5rPfGhr9oJWY25r9oJy1+1xlFLXA/g3gD0Azvw/7lMkN7XWJzYunl9/m+WW/TkXT95GRW7XIy5mDR68X0+nSzxzng0fGo8dO7JbDHfbPdJI/gfWV7tCCCFsLohvpAkhxMVCiq4QQmhIiq4QQmhIiq4QQmhIiq4QQmhIiq4QQmhIiq4QQmhIiq4QQmhIiq4QQmjIbd9I+ykUvPProWX//T+P5AYMecgjuQBQlvmWx7IbPfi11Es89LVUT/LUV3E7KnmlK4QQGpKiK4QQGpKiK4QQGpKiK4QQGpKiK4QQGpKiK4QQGpKiK4QQGpKiK4QQGpKiK4QQGpKiK4QQGpKiK4QQGrogiu6WzZ8gJnIAIsNDsfDVl53W19TUYOqdKYgMD8WIYUNxqKjIvm7hKy8hMjwUMZED8OmWzRdErqezbxwWgd1rn0FO2nN4/Lc3Oq3vc3UANr07A5krZ2Pzew9D36uHfd2Ch8che9UcZK+ag+SbYtud7cnnelBUOKIjwvDawpZzf33XFERHhGHU9dfac0tLS3HrTWPQq2c3zJo5o12ZTbO97TzzxjHbkXTLAsAXQCaA3QByAcw7V5/Y2Dha6uiwVFbX02A0cu+BApafrmF0dAx37s51aPP6m2/znnvvo6WOXLLsYyZNmkxLHblzdy6jo2N4srKa+w7+QIPRyMrqeqeMlhZP5WqZ7Tt4htPSNe4hFvxYwvDEueyWMJO7DxRz0MQFDm3WbNnJu5/9J30Hz+DN09/k8ozt9B08g7966C/87Jt99It/mD2vm8XsnCIGXf94izmeHPfpmkaH5VRVHQ0GI3P25bOsoppR0THM3pXj0GbRG4t59z3Tebqmkf9Y+hGTkifzdE0jS05U8NMvvuIbb73D+/7wgNO2my/eeJ51lFwts2Nj49hanXPnK90aAGNIDgQwCMAtSqlr27uRrMxMhISEwmA0onPnzpiUMgUZ6WkObTLS03DXtN8AACYmJWPrF5+DJDLS0zApZQq6dOmCfgYDQkJCkZWZ2aFzPZ2dENUXBcXHUWQqRV19A1I370Di6GiHNuHGq7At6yAAYFvWQSSOsq6PMF6F/+zMR0NDI6qqa7Enz4ybhkV0+HFnZ2XC2CQ3eXJKC7kb7LkTJiZj65fWXD8/Pwwbfj26+Pq6PM6OMGZPZnvjmJtyW9GlVaXt4aW2pd3X1DObTQgO7m1/rNcHw2QyObfpbW3j4+OD7v7+KC0thcnk3Ndsduzb0XI9na0L6oHi/5XZH5tKTjq8fQAAew6aMH7MQADA+DED0f3yy9DTvyu+P2jCTcOuwWW+lyKwhx9GxYch+MqADj9u6zaDHfoeaSk3uElud2vu+fLG88wbx9yUW6+nq5TqBGAHgFAAb5Pc3kKb6QCmA0DvPn3cuTviZzJn0Tosmj0ZU28fiq93FsB0tAwNDcTn3+5HXGQffPn3WTheVont3xeiobHR07srRIfi1g/SSDaQHAQgGMAQpVRUC23eIxlPMj7oiiCnbeh0ehQXH7Y/NpmKodfrndsctrapr6/HqfJyBAYGQq937qvTOfZtjadyPZ1tPnYSwVedfXWq79UDppKTDm2OHD+FKY+/j+vufBVz304HAJRXWgAAr36wBdfe8QoSH3gbSinkHSpxOdujz/XhYoe+V7eUW9wk95Q193x543nmjWNuSpPZCyRPAvgSwC3t7RufkID8/DwUFRaitrYWqStXYGziOIc2YxPHYfnSJQCAtWtWY9QNY6CUwtjEcUhduQI1NTUoKixEfn4eEoYM6dC5ns7Ozv0Rob2D0FcXiEt9OmHSzXHYuG2PQ5vAHn72O3w88fubsCTtWwDWuyL09O8KAIgK0yEqTIfPvt3f4ccdF5+Agia5q1etbCH3dnvuurWrMWr0mJ/lLifeeJ5545gduHH2QhCAHrafLwPwbwCJ7Z29YKkj123YyNCwMBqMRj4/fwEtdeScp59l6to0WurIsgoLJyQl0xgSwrj4BO49UGDv+/z8BTQYjQzr35/r0ze5/CmnJ3O1ym5pVoHv4BkcP+MdHiw6yoIfS/jc4g30HTyDL/51E5NmvkvfwTN4x+PvM+/QUR4sOsoP137N7kMeoe/gGfQf+gj3Fpi5t8DM7d//wCEpL7Wa4clxtzSrYM36DIaGhtFgMHLuvBd4uqaRs596hqtWr+fpmkaWlldxwsRkGo3W3Jx9+fa+ffr2ZUBAAP38/KjT651mPrQ1e8EbzrOOlKtVdluzF5StKP7slFIxAJYA6ATrK+pVJOe31ScuLp5fb892y/4IZ3KPNO154z3SvNHwofHYsSO7xSfbbR+kkfwewGB3bV8IIS5EF8Q30oQQ4mIhRVcIITQkRVcIITQkRVcIITQkRVcIITQkRVcIITQkRVcIITQkRVcIITQkRVcIITTk1ks7io7Nk1/FDbj+jx7LLvvPqx7LdtfX7l3xc1yg50JTW++ZS4u2lSqvdIUQQkNSdIUQQkNSdIUQQkNSdIUQQkNSdIUQQkNSdIUQQkNSdIUQQkNSdIUQQkOtfjlCKfUWgFZncpN82C17JIQQF7G2vpEmd4gUQoifWatFl+SSpo+VUl1JVrl/l4QQ4uJ1zvd0lVLXKaX2AthvezxQKfWO2/esiS2bP0FM5ABEhodi4asvO62vqanB1DtTEBkeihHDhuJQUZF93cJXXkJkeChiIgfg0y2bL4hcb82+8dr+2L3yCeSk/hGPTxvttL7PVT2w6a17kbnsUWx+5z7og/zt63pf2QPpb9yD71Y8hp0fP4Y+Vwe0K9vTx3tgZDiiIsLwWivZ0+6cgqiIMIwcfq09u7S0FLfcOAZBAd3w6MwZ7c49k+1tf1ufbfkEcTERGBTZH39e+EqL2b+dOgWDIvtjzIjrcOiQNXtHViauHxqL64fGYviQwUhPW9fubADWC3C0tQDYDqA3gO+a/C7nXP2atO0E4DsAGedqGxsbR0sdHZbK6noajEbuPVDA8tM1jI6O4c7duQ5tXn/zbd5z73201JFLln3MpEmTaakjd+7OZXR0DE9WVnPfwR9oMBpZWV3vlNHS4qlcb8n2HfqEw9L1uj+y4PBxhk94id2Gz+bugyYOSlno0GbNZ7t597wV9B36BG9+4F0u35RtX7dtRz5ve+g9+g59goGjn2bAyKecMs4snjzeVbWNDkuFpY4Go5G5+/N5srKa0dEx3LErx6HNojcX8+57p7OqtpFLln7EpOTJrKpt5LGyCn725Vd8Y/E7vO/+B5y23XzpKOeZlrnllgaH5URlLfsZjNy1N4/Hyi2Mio7h9p17HNq89vpi/u6e6Sy3NPCDJcs5IWkSyy0NPFJawdKKGpZbGnjgh2JeERRkf9x8GRQbx9bqnEuzF0gebvarhnbU9ZkA9rWjvYOszEyEhITCYDSic+fOmJQyBRnpaQ5tMtLTcNe03wAAJiYlY+sXn4MkMtLTMCllCrp06YJ+BgNCQkKRlZnZoXO9NTvhmt4oKD6OIvMJ1NU3IPXT3UgcGenQJtzQC9uy8wEA23YU2NeH9+sFn06X4IvMPADAaUstLDV1HX7MAJCd5ZidPDnFKXtj+gZMtWVPSErG1i+t2X5+fhg2/Hr4+vq6nNcRxu3J470jKxPGkBAYDNbsiZNSsDFjg0ObTRlpuPOuXwMAfjUxGdu2fgGS6Nq1K3x8rO/IVtdU/+SrtrlSdA8rpYYBoFLqUqXU43CxiCqlggGMBfD+T9o7AGazCcHBve2P9fpgmEwm5za9rW18fHzQ3d8fpaWlMJmc+5rNjn07Wq63ZuuC/FFcUm5/bCophz6ou0ObPXlHMH50FABg/OgodPfzRc/uXRHWJwgnK6qx4uVp+GbJTPxpxlhcconrfxAePd4mE/TBwW32t7Zxzj5f3vq3pXfor8eRZtlHzGbH493dHydsxzs7czuGxkZjWPxALHrzHXsRbg9Xiu4fADwIQA/ADGCQ7bErXgfwR7RxeUml1HSlVLZSKvvY8WMublZ4ozlvbcSIWCO+WTITIwYbYSo5iYbGRvh0ugTDB/XD7Dc34vrfvwWDviemjY339O6Ki1D8kKHYvnMPvvzPdvx54Suorq5u9zbOWXRJHid5F8krSQaRnErynP/MKqUSAZSQ3HGO7b9HMp5kfNAVQU7rdTo9iovPvrthMhVDr9c7tzlsbVNfX49T5eUIDAyEXu/cV6dz7NsaT+V6a7b5WDmCe539YEzfyx+mY6cc2hw5fgpTZi/Fdb95A3Pf/QQAUF5ZDVNJOb4/eARF5hNoaGjEhm25GDSg448ZAHR6PUzFxW32t7Zxzj5f3vq3ZXLob8LVzbKv1ukcj/epcvRsdrwHhEfA7/LLsTc3x+XsM1yZvWBUSqUrpY4ppUqUUmlKKaML2x4OYJxSqgjACgBjlFLL2ruD8QkJyM/PQ1FhIWpra5G6cgXGJo5zaDM2cRyWL7XOcFu7ZjVG3TAGSimMTRyH1JUrUFNTg6LCQuTn5yFhyJAOneut2dn7ihHa+wr0vToAl/p0wqQbB2Ljv/c6tAn072p/H+2J39yAJenZtr6H4d/NF1f08AMAjI4Pwf7Cox1+zAAQF++YvXrVSqfs2xJvxzJb9ro1qzFq9Jif5S4Q3vi3FRufgIL8fBQVWbPXpq7EbWNvd2hz29hx+Gj5PwEA69euxshRN0AphaKiQtTX1wMAfjx0CHkH9qNv334uZ9u5MPvgWwDTYJ3T6wNgKoDtrs5esG1jNH7i7AVLHbluw0aGhoXRYDTy+fkLaKkj5zz9LFPXptFSR5ZVWDghKZnGkBDGxSdw74ECe9/n5y+gwWhkWP/+XJ++yeVP8T2Z6w3ZLc0qGP/IBzx4qIQFh4/zub/8i75Dn+CL73/KpMf/Tt+hT/CO2f9k3o/HePBQCT9M287u18+2973toff4fZ6Ze/LM/GdGFrsNn+3y7AUtj3dLswrWpmUwNNSaPXfeC6yqbeTsp57hqjXrWVXbyBOnqjhh4tns3P359r59+vZlQEAA/fz8qNPrnWY+tDV7wRv+tlqaWZC6Lp0hoWHsZzDymedfYLmlgX+c8ww/Tl3HcksDj5ad5vgJSTQYQxgbl8Bde/NYbmngXz/4B8MjrmF0zEDGDBrM5SvXtLj9c81eUOe6Z5NS6nuSMc1+t5vkQFcLu1JqNIDHSSa21S4uLp5fb5cvwnkDuUea9uQeadoZNXwIvtuR3eIBb+vaCz1tP/5LKTUb1rcICCAFwKb27ADJrQC2tqePEEJcjNqa77AD1iJ7plrf12QdAcxx104JIcTFqq1rLxi03BEhhPAGLs3sVUpFAbgGgP2rLyT/6a6dEkKIi9U5i65Sai6ssw+ugfW93FsB/AeAFF0hhGgnV76RlgzgFwD+R/J3AAYC8G+7ixBCiJa4UnQtJBsB1CulugMogfWqY0IIIdrJlfd0s5VSPQD8DdYZDZUAvnHnTgkhxMXqnEWX5AO2H99VSn0CoDvJ7927W0IIcXFq68sRsW2tI7nTPbskhBAXr7Ze6f5fG+sIYMzPvC/Ci3jyq7gBCT/t1jY/hxOZb3ks2xt19nHpPg0/u7ZS2/pyxA1u2BchhPBqnvlnQAghvJQUXSGE0JAUXSGE0JArd45QSqmpSqnnbI/7KKVcv1S7EEIIO1de6b4D4DoAd9geVwB42217JIQQFzFXvpE2lGSsUuo7ACBZppTq7Ob9EkKIi5Irr3TrlFKdYJ2bC6VUENq4pboQQojWuVJ03wSwDkAvpdSLsF7W8U9u3SshhLhIuXLtheVKqR2wXt5RAfgVyX1u3zMhhLgIuTJ7oQ+AKgDpADYAOG37nWa2bP4EMZEDEBkeioWvvuy0vqamBlPvTEFkeChGDBuKQ0VF9nULX3kJkeGhiIkcgE+3bL4gciVb++wbh0Vg97pnkZM2F4//7kan9X2uDsCmdx9C5so52Py3mdD36mFft+Dh8chOfQrZqU8h+aZWL1nSqi2bP8HAyHBERYThtVbGPO3OKYiKCMPI4dc6jTkqIgwDI8MvqOPtjeeYXWv3Zj+zANgD4Hvbf/MA1APIPVc/W98iW79dALLP1T42Ns7pvvWV1fU0GI3ce6CA5adrGB0dw527cx3avP7m27zn3vtoqSOXLPuYSZMm01JH7tydy+joGJ6srOa+gz/QYDSysrreKaOlxVO5ku3+bN9BDzosXWNnsODHEoaPfY7d4h/m7gOHOWjiCw5t1mzZwbuf/Sd9Bz3Im+99g8vTt9N30IP81Yx3+Nk3++gX9xB7Xvsos3OKGDT8MaeMM0tVbaPDUmGpo8FoZO7+fJ6srGZ0dAx37MpxaLPozcW8+97prKpt5JKlHzEpeTKrahu5Y1cOo6NjWFZh4d4DBTQYjayw1DllnFk6ynPtDedYbGwcW6tz53ylSzKaZIztv2EAhqB919O9geQgkvHt6GOXlZmJkJBQGIxGdO7cGZNSpiAjPc2hTUZ6Gu6a9hsAwMSkZGz94nOQREZ6GialTEGXLl3Qz2BASEgosjIzO3SuZGufnRDVDwWHj6PIVIq6+gakbt6JxNExDm3CjVdjW+YBAMC2rINIHB0NAIgwXoX/7MxHQ0MjqqprsSfPhJuGRbg85uwsxzEnT05xGvPG9A2YahvzhKRkbP3y7JiTJ6c4jDk7q+Mfb288x5pq9zfSbJd0HNrupJ/IbDYhOPjsjSr0+mCYTCbnNr2tbXx8fNDd3x+lpaUwmZz7ms2OfTtarmRrn63r5Y/io2X2x6ajZdAHOd6Ras9BE8aPGQQAGD9mILpffhl6+vvh+4PWInuZ76UI7OGHUfH9EXxVgOtjNpmgDw5uc7+tbZzH3Px46fR6mE0d/3h74znWlCs3ppzV5OElAGIBmF3cPgFsUUoRwF9JvtfC9qcDmA4Avfto+laxEC6bs2gdFj05CVPHDcXXO/NhOlqGhoZGfP7tfsRF9sWX/3gMx8sqsf37QjQ0yIxK0TpXXul2a7J0AbARwHgXt389yVhY7yD8oFJqZPMGJN8jGU8yPuiKIKcN6HR6FBcftj82mYqh1+ud2xy2tqmvr8ep8nIEBgZCr3fuq9M59m2Np3IlW/tsc0k5gq88++pUf2UATMfKHdocOVaOKY+/j+vueAVzF6cDAMorLQCAVz/YjGunvIzE+xdDKYW8H0tcH7NeD1NxcZv7bW3jPObmx8tsMkGn7/jH2xvPsabaLLq2L0V0IznPtrxIcjnJalc2TtJk+28JrHN9233NhviEBOTn56GosBC1tbVIXbkCYxPHObQZmzgOy5cuAQCsXbMao24YA6UUxiaOQ+rKFaipqUFRYSHy8/OQMMS1XfBUrmRrn52dewihfYLQVxeIS306YdLNsdi41fGOVIE9/KCUAgA88fubsSTtWwDAJZco9PT3AwBEhekQFabDZ9/sd3nMcfGOY169aqXTmG9LvB3LbGNet2Y1Ro0+O+bVq1Y6jDk+oeMfb288xxy0MfPAx/bfb1yZqdBCfz9YC/aZn/8L4Jb2zl6w1JHrNmxkaFgYDUYjn5+/gJY6cs7TzzJ1bRotdWRZhYUTkpJpDAlhXHwC9x4osPd9fv4CGoxGhvXvz/Xpm1z+lNOTuZLt3uyWZhWMn/E2DxYdZcGPJXzurQ30HfQgX/zrJibNfJe+gx7kHY//jXmHjvJg0VF+uPZrdk+YSd9BD9J/yEzuLTBzb4GZ23f/wCGT/9TqzIWWZi9U1TZybVoGQ0OtY5477wVW1TZy9lPPcNWa9ayqbeSJU1WcMPHsmHP359v7zp33gnXMYf25bsPGVmcutDR7Qf623Jfd1uwFZSuKTpRSO23XXPgLAD2AVACnmxTrtW0Vc6WUEdZXt4D1veOPSL7YVp+4uHh+vT37XP9OCHFevPV2PWdeqQv3Gz40Hjt2ZLd4wF254I0vgFJY74lGWL+VRgBtFl2SPwAY2L5dFUKIi1tbRbeXbeZCDs4W2zNafnkshBCiTW0V3U4ALodjsT1Diq4QQvwEbRXdIyTna7YnQgjhBdqaMibvugshxM+sraL7C832QgghvESrRZfkCS13RAghvIHcgl0IITQkRVcIITQkRVcIITQkRVcIITTkyteANUMArV0Lwt3ke+new5PXP+g59GGPZZd5cNziLHmlK4QQGpKiK4QQGpKiK4QQGpKiK4QQGpKiK4QQGpKiK4QQGpKiK4QQGpKiK4QQGpKiK4QQGpKiK4QQGrogiu6WzZ9gYGQ4oiLC8NqrLzutr6mpwbQ7pyAqIgwjh1+LQ0VF9nULX3kJURFhGBgZjk+3bG53bkzkAESGh2JhK7lT70xBZHgoRgwb6pQbGR6KmMgB7c6VbO2zPXWOAcCNwyKwe+0zyEl7Do//9kan9X2uDsCmd2cgc+VsbH7vYeh79bCvW/DwOGSvmoPsVXOQfFNsu7M9eby97RyzI+m2BUAPAKsB7AewD8B1bbUfHBvHqtpGh6XCUkeD0cjc/fk8WVnN6OgY7tiV49Bm0ZuLefe901lV28glSz9iUvJkVtU2cseuHEZHx7CswsK9BwpoMBpZYalzyqiqbaSljg5LZXU9DUYj9x4oYPnpGkZHx3Dn7lyHNq+/+Tbvufc+WurIJcs+ZtKkybTUkTt35zI6OoYnK6u57+APNBiNrKyud8pobZFs92Z76hyrqm2k7+AZDkvXuIdY8GMJwxPnslvCTO4+UMxBExc4tFmzZSfvfvaf9B08gzdPf5PLM7bTd/AM/uqhv/Czb/bRL/5h9rxuFrNzihh0/eNOGWeWjvJce8M5Fhsbx9bqnLtf6b4B4BOS4QAG2gpvu2RnZSIkJBQGoxGdO3dG8uQUZKSnObTZmL4BU6f9BgAwISkZW7/8HCSRkZ6G5Mkp6NKlC/oZDAgJCUV2VqZLuVmZjrmTUqY45Wakp+EuW+7EpGRs/eJs7qSUKQ65WZmu5Uq29tmeOscAICGqLwqKj6PIVIq6+gakbt6BxNHRDm3CjVdhW9ZBAMC2rINIHGVdH2G8Cv/ZmY+GhkZUVddiT54ZNw2LcDnbU8fbG8+xptxWdJVS/gBGAvgAAEjWkjzZ3u2YTSbog4Ptj/X6YJjNphba9AYA+Pj4oLu/P0pLS2E2mxBs+z0A6PR6mE2OfVvNbdZXrw+GqVlfs9mE4N7OuSaTc9/m+yzZHSfbU+cYAOiCeqD4f2X2x6aSkw5vHwDAnoMmjB8zEAAwfsxAdL/8MvT074rvD5pw07BrcJnvpQjs4YdR8WEIvjLA5WyPHW8vPMeacuelHQ0AjgH4u1JqIIAdAGaSPN20kVJqOoDpANC7Tx837o4QF6Y5i9Zh0ezJmHr7UHy9swCmo2VoaCA+/3Y/4iL74Mu/z8Lxskps/74QDY2Nnt5dcQ7ufHvBB0AsgL+QHAzgNIDZzRuRfI9kPMn4K64IctqITq+HqbjY/thkKoZOp2+hzWEAQH19PU6VlyMwMBA6nR7Ftt8D1lcrOr1j39Y072syFUPfrK9Op0fxYedcvd65b/N9luyOk+2pcwwAzMdOIviqs69O9b16wFRy0qHNkeOnMOXx93Hdna9i7tvpAIDySgsA4NUPtuDaO15B4gNvQymFvEMlLmd77Hh74TnWlDuLbjGAYpLbbY9Xw1qE2yUuPgH5+XkoKixEbW0tVq9aibGJ4xza3JZ4O5YtXQIAWLdmNUaNHgOlFMYmjsPqVStRU1ODosJC5OfnIT5hiEu58QmOuakrVzjljk0ch+W23LVrVmPUDWdzU1eucMhNGOJarmRrn+2pcwwAsnN/RGjvIPTVBeJSn06YdHMcNm7b49AmsIef/SL7T/z+JixJ+xYAcMklCj39uwIAosJ0iArT4bNv97uc7anj7Y3nmAM3z174N4ABtp+fB7CwvbMXqmobuTYtg6GhYTQYjZw77wVW1TZy9lPPcNWa9ayqbeSJU1WcMDGZxpAQxsUnMHd/vr3v3Hkv0GA0MiysP9dt2Njqp8otfQK5bsNGhoZZc5+fv4CWOnLO088ydW0aLXVkWYWFE5LO5u49UGDv+/z8Bdbc/v25Pn2Ty5+uSrb7sz11jrU0e8F38AyOn/EODxYdZcGPJXxu8Qb6Dp7BF/+6iUkz36Xv4Bm84/H3mXfoKA8WHeWHa79m9yGP0HfwDPoPfYR7C8zcW2Dm9u9/4JCUl1qdudDS7AVPPtcX+znW1uwF5c7b4yilBgF4H0BnAD8A+B3Jstbax8bF8+tvs9y2P22R2/V4D0/dEgqQ2/V4i+FD47FjR3aLRcWt90gjuQtAvDszhBDiQnJBfCNNCCEuFlJ0hRBCQ1J0hRBCQ1J0hRBCQ1J0hRBCQ1J0hRBCQ1J0hRBCQ1J0hRBCQ1J0hRBCQ1J0hRBCQ279GnB7KXjnNRAaGz1zLYBLLvG+Yw149hzz5PUPAkY/45HcE1++4JFcAKip98z1hdv6k5ZXukIIoSEpukIIoSEpukIIoSEpukIIoSEpukIIoSEpukIIoSEpukIIoSEpukIIoSEpukIIoSEpukIIoaELouhu2fwJYiIHIDI8FAtffdlpfU1NDabemYLI8FCMGDYUh4qK7OsWvvISIsNDERM5AJ9u2XxB5J7JHhQVjuiIMLy2sOXsX981BdERYRh1/bX27NLSUtx60xj06tkNs2bOaHfumWxPjtsbn2tPZd84NAy7P5qJnBWP4vGpI53W97myBza9/jtk/mMGNr91N/RB3e3rKrfNx7d/fxDf/v1BpL58V7tyt2z+BAMjwxEVEYbXWhnztDunICoiDCOHX+s05qiIMAyMDP9JY/5syydIGHgNYqMGYNFrr7SY/ftpdyA2agB+OfI6/HioyGH94cM/IjjIH2+9/n/tzgYAkHTLAmAAgF1NllMAHmmrT2xsHC11dFgqq+tpMBq590ABy0/XMDo6hjt35zq0ef3Nt3nPvffRUkcuWfYxkyZNpqWO3Lk7l9HRMTxZWc19B3+gwWhkZXW9U0ZLi5a5p2saHZZTVXU0GIzM2ZfPsopqRkXHMHtXjkObRW8s5t33TOfpmkb+Y+lHTEqezNM1jSw5UcFPv/iKb7z1Du/7wwNO2266eHrcHSXbW8bsO/xph6XriGdYUFzK8Emvsduo57g7z8xBd73u0GbNF3t49wur6Tv8ad780Adc/q/v7OsqTlc7bbOlpaq20WGpsNTRYDQyd38+T1ZWMzo6hjt25Ti0WfTmYt5973RW1TZyie38rqpt5I5dOYyOjmFZhYV7DxTQYDSywlLnlHFmKauqd1iOV9Swn8HI73IP8ujJKkZGx/CbHd87tFm46C3+9u7pLKuq5/tLlnNC0iSH9eN+NZHjJyRx/p9ecdr+mWXQ4Di2Vufc9kqX5AGSg0gOAhAHoArAuvZuJyszEyEhoTAYjejcuTMmpUxBRnqaQ5uM9DTcNe03AICJScnY+sXnIImM9DRMSpmCLl26oJ/BgJCQUGRlZnboXADIzsqEsUl28uSUFrI32LMnTEzG1i+t2X5+fhg2/Hp08fV1Oa+jjNsbn2tPZidEBKOguBRF5jLU1Tcg9bM9SLw+wqFNeL8gbNv5AwBg284fkDgi3OXttyY7y3HMLZ3fG9M3YOqZ8zvp7PmdkZ6G5MkpDmPOznJ9zDuyM2EMCUE/gzV7YvJkbMrY4NDmXxs34I6p0wAA4yckYdvWL868kMTGDWno068fwiOu+cnj1+rthV8AKCB5qL0dzWYTgoN72x/r9cEwmUzObXpb2/j4+KC7vz9KS0thMjn3NZsd+3a03LPbDXbof6Sl7OAm2d2t2efL4+P2xufaQ9m6oO4oLim3PzYdO+Xw9gEA7Mn/H8aPshaY8SOvQXc/X/TsfhkAwLezD/7z/v3Y9tf7cPsIx2LdFrPJBH2w4/ndfL+tbZzH3Px46fR6mE2uj/mI2Qy9vmn/YBwxmx2zm7Q587d1orQUlZWVeOPPr+LJp55zOa8lWl3acQqAj1taoZSaDmA6APTu00ej3RFCuGLO4k+waFYipt46GF/vLoKppBwNtusWDkh+DebjFeinC8Anb/weOQVHUWg+4eE9dp9XXpyH+x96BJdffvl5bcftr3SVUp0BjAOQ2tJ6ku+RjCcZH3RFkNN6nU6P4uLD9scmUzH0er1zm8PWNvX19ThVXo7AwEDo9c59dTrHvq3xVO7Z7RY79L+6peziJtmnrNnny+Pj9sbn2kPZ5mOnENzL3/5YH9QdpmOnHNocKa3AlKc/xnW/fwdz3/sMAFBeWW3tf7wCAFBkLsNX3xViUP+rXcrV6fUwFTue383329rGeczNj5fZZIJO7/qYr9bpYDI17V+Mq3U6x+wmbc78bfUMDER2VibmPj0bMeEh+Mvbb+LPC1/Ge3952+XsM7R4e+FWADtJHv0pneMTEpCfn4eiwkLU1tYideUKjE0c59BmbOI4LF+6BACwds1qjLphDJRSGJs4DqkrV6CmpgZFhYXIz89DwpAhHToXAOLiE1DQJHv1qpUtZN9uz163djVGjR7zs1yc25Pj9sbn2pPZ2ftNCO0diL5XB+BSn06Y9MtobPx6v0ObQP+u9vPqiWkjsWTjTgBAj26+6HxpJ3ub66L7YF9RiUu5cfGOY27p/L4t8XYsO3N+rzl7fo9NHIfVq1Y6jDk+wfUxx8YloCA/H4eKrNlrV6/CrWNvd2hzy2234+NlSwEAaevWYOSoG6CUwr8+24bv9xfg+/0FuP/BhzHridmYfv+DLmfbuWv2QpNZDCsA/M6Vti3NXrDUkes2bGRoWBgNRiOfn7+AljpyztPPMnVtGi11ZFmFhROSkmkMCWFcfAL3Hiiw931+/gIajEaG9e/P9embXP5UWcvclmYWrFmfwdDQMBoMRs6d9wJP1zRy9lPPcNXq9Txd08jS8ipOmJhMo9GanbMv3963T9++DAgIoJ+fH3V6vdPMh7ZmL3jyeHvDc+3J7JZmFox/bAkP/niMBcWlfO6vW+g7/Gm++OEXTPrjUvoOf5p3PP0R8348xoM/HuOHG7LYffRz9B3+NEff9y735B/h7jwz9+Qf4X1/Wuvy7IWq2kauTbOd30br+V1Vazu/16xnVW0jT5yynd+2Mefuz7f3nTvvBeuYw/pz3YaNrc5caGn2QllVPVeu3cCQ0DD2Mxj59Nz5LKuq5xOzn+byVetYVlXPIycqOX5CEg3GEMbGxfO73INO23jyqWd/8uwFdeZTOXdQSvkB+BGAkWT5udrHxcXz6+3Zbtufjkpu1yO0ILfr0c4Nw4fiu53ZLf6BufWDNJKnAZz/G41CCHGRuCC+kSaEEBcLKbpCCKEhKbpCCKEhKbpCCKEhKbpCCKEhKbpCCKEhKbpCCKEhKbpCCKEhKbpCCKEhrS7t6BICcOfXktvyc1ws5qeSr+Nqy1PnGODZ86xs6wKP5AZc+4hHcgGg9L+LPJLb1tMsr3SFEEJDUnSFEEJDUnSFEEJDUnSFEEJDUnSFEEJDUnSFEEJDUnSFEEJDUnSFEEJDUnSFEEJDUnSFEEJDF0TR3bL5EwyMDEdURBhee/Vlp/U1NTWYducUREWEYeTwa3GoqAgAUFpailtuHIOggG54dOaMn5QbEzkAkeGhWNhK7tQ7UxAZHooRw4bacwFg4SsvITI8FDGRA/Dpls2S3cGzf+o5diY3KiIMAyPDL6gxezL7xuvCsXvNU8hZ9zQe/80vnNb3uSoAm955AJkf/xGb/zoD+l7+9nUvPnw7dqx8Et+lzsH/PT6xXbmAdcyDosIRHRGG1xa2POZf3zUF0RFhGHW9Yz259aYx6NWzG2b9hHpi19q92X+OBcCjAHIB5AD4GIBvW+0Hx8Y53be+wlJHg9HI3P35PFlZzejoGO7YlePQZtGbi3n3vdNZVdvIJUs/YlLyZFbVNvJYWQU/+/IrvrH4Hd53/wNO2266WOrosFRW19NgNHLvgQKWn65hdHQMd+7OdWjz+ptv855776Oljlyy7GMmTZpMSx25c3cuo6NjeLKymvsO/kCD0cjK6nqnjNYWyXZv9s95ju3YlcPo6BiWVVi490ABDUYjKyx1cp41WXzjZjosXRMeYcHhYwwfN5/dhs7i7gPFHJT8J4c2az79jnc/t4y+cTN5832LuXxjJn3jZnL07xbxv7sK2DXhEXZNeITf7i7kjdPfcso4s5yuaXRYTlXV0WAwMmdfPssqqhkVHcPsXTkObRa9sZh33zOdp2sa+Q/bc326ppElJyr46Rdf8Y233uF9f3jAadtNl8GxcWytzrntla5SSg/gYQDxJKMAdAIwpb3byc7KREhIKAxGIzp37ozkySnISE9zaLMxfQOmTvsNAGBCUjK2fvk5SMLPzw/Dhl8PX1/fdu9/VqZj7qSUKU65GelpuMuWOzEpGVu/sOZmpKdhUsoUdOnSBf0MBoSEhCIrM1OyO2j2+ZxjGelpSJ6c4pCbndXxx+zJ7ITIvig4fBxFplLU1Tcgdct3SBwV7dAm3HAltmXnAQC2ZechcaR1PQl06XwpOl/qgy6X+sDH5xKUlFa4PObsrEwYz/FcZ6RvsI95wkTnetLlJ9STptz99oIPgMuUUj4AugIwt3cDZpMJ+uBg+2O9Phhms6mFNr2tgT4+6O7vj9LS0vPZb5jNJgTbtnkm12QyObfp7ZxrMjn3bb7Pkt1xss/nHGu+zzq9HmZTxx+zJ7N1vfxRfLTM/thUctLh7QMA2JNnxvgbYgAA42+IQffLfdHTvyu27ynCV9l5KPxkPgo3z8dn3+7HgaKj7Rtzb8fn+khLY276XHc//3rSlNuKLkkTgNcA/AjgCIByklvclSeEuHjMeT0NI2JD8M3yxzEiNgSmoyfR0EAYg6/AAMOVCL1tLkJunYvR8f0xfJDR07vbLu58eyEAwHgABgA6AH5KqakttJuulMpWSmUfP37MaTs6vR6m4mL7Y5OpGDqdvoU2hwEA9fX1OFVejsDAwPPaf51Oj2LbNs/k6vV65zaHnXP1eue+zfdZsjtO9vmcY8332WwyQafv+GP2ZLa5pBzBVwbYH+t79YCppNyhzZHjpzDlj3/HdXe9hrnvbAQAlFdaMP6GaGTuOYTTllqcttRi83/3YWhMv/aN+bDjc311S2Nu+lyfOv960pQ73174JYBCksdI1gFYC2BY80Yk3yMZTzL+iiuCnDYSF5+A/Pw8FBUWora2FqtXrcTYxHEObW5LvB3Lli4BAKxbsxqjRo8574tFxyc45qauXOGUOzZxHJbbcteuWY1RN1hzxyaOQ+rKFaipqUFRYSHy8/OQMGSIZHfQ7PM5x8YmjsPqVSsdcuMTOv6YPZmdvfdHhPa+An11PXGpTydMumkwNn6V49Am0N/P/jf8xO9+iSUbtgMADv/vJEbEhqBTp0vg0+kSjIgNwf5C199eiItPQME5nuuxibfbx7xu7c9TTxy4cebCUFhnLnQFoAAsAfBQe2cvVNU2cm1aBkNDw2gwGjl33gusqm3k7Kee4ao161lV28gTp6o4YWIyjSEhjItPYO7+fHvfPn37MiAggH5+ftTp9U6fSrf2qbKljly3YSNDw6y5z89fQEsdOefpZ5m6No2WOrKswsIJSWdz9x4osPd9fv4CGoxGhvXvz/Xpm1z+RFmy3Z/9c59jc+e9YM0N6891Gza2a5aMNxzvlmYVjH/4XR4sOsqCw8f43NsZ9I2byRff+4RJj75H37iZvOOPHzLvUAkPFh3lh+v+y+7XzrLPfPjbmq+574f/cW/BEb6x7MtWZy60NHvhdE0j16y3PdcG63N9usb2XK9ez9M1jSwttz3XRuuYc/bl2/s2ryfNZz64MntBufPWJUqpeQBSANQD+A7APSRrWmsfGxfPr7/Nctv+tMWTt1ER2vLW2/V4ijferuf66xKwc0d2i0+2W++RRnIugLnuzBBCiAvJBfGNNCGEuFhI0RVCCA1J0RVCCA1J0RVCCA1J0RVCCA1J0RVCCA1J0RVCCA1J0RVCCA1J0RVCCA1J0RVCCA259WvAomOTaxAILZz4xjPXPwCAnsMf90huzf7iVtfJK10hhNCQFF0hhNCQFF0hhNCQFF0hhNCQFF0hhNCQFF0hhNCQFF0hhNCQFF0hhNCQFF0hhNCQFF0hhNDQBVF0t2z+BAMjwxEVEYbXXn3ZaX1NTQ2m3TkFURFhGDn8WhwqKrKvW/jKS4iKCMPAyHB8umVzu3NjIgcgMjwUC1vJnXpnCiLDQzFi2FCn3MjwUMREDmh3bkfI/inHu7S0FLfcOAZBAd3w6MwZ7c49k+2JcXvjmD2Z7am/aQC48doB2J36JHLWzMHjvx7jtL7PVQHY9PYfkLn8MWz+y/3Q9/IHAIyMC8G3y2bZl7J/v4zbR0W1Ox8k3bYAmAkgB0AugEfO1X5wbByrahsdlgpLHQ1GI3P35/NkZTWjo2O4Y1eOQ5tFby7m3fdOZ1VtI5cs/YhJyZNZVdvIHbtyGB0dw7IKC/ceKKDBaGSFpc4po6q2kZY6OiyV1fU0GI3ce6CA5adrGB0dw527cx3avP7m27zn3vtoqSOXLPuYSZMm01JH7tydy+joGJ6srOa+gz/QYDSysrreKaO1Ravslo7D+RzvY2UV/OzLr/jG4nd43/0PtLj91o63J8et1ZjlPNP2b7qqtpG+CbMclq5DH2PB4WMMH7+A3a57grsPmjho8isObdZ8tot3P/8RfRNm8eb73+HyjVlO27n6F0+z9ORpBlz/pNM634RZVF2vZGt1zm2vdJVSUQDuBTAEwEAAiUqp0PZuJzsrEyEhoTAYjejcuTOSJ6cgIz3Noc3G9A2YOu03AIAJScnY+uXnIImM9DQkT05Bly5d0M9gQEhIKLKzMl3Kzcp0zJ2UMsUpNyM9DXfZcicmJWPrF2dzJ6VMccjNynQt19PZ53O8/fz8MGz49fD19XU5ryOM2xvH7MlsT/1NA0BCZB8UFJeiyHwCdfUNSN3yHRJHRjq0CTdciW1Z+QCAbdn5SBzp/Gp2wpiB2PLNflhq6lzOPsOdby9EANhOsopkPYBtACa2dyNmkwn64GD7Y70+GGazqYU2vQEAPj4+6O7vj9LSUpjNJgTbfg8AOr0eZpNj31Zzm/XV64NhatbXbDYhuLdzrsnk3Lf5PnfY7PM43ufLU+P2xjF7MttTf9MAoAvyR/HRk/bHppJy6IP8HdrsyTNj/A3RAIDxo6PR/XJf9PTv6tBm0k2DsGrLTpdzm3Jn0c0BMEIpFaiU6grgNgC9mzdSSk1XSmUrpbKPHz/mxt0RQohzm/NGOkbEGvHN0lkYEWuE6ehJNDQ02tdfFdgNkSFX49NvDvyk7but6JLcB+AVAFsAfAJgF4CGFtq9RzKeZPwVVwQ5bUen18NUfPbalCZTMXQ6fQttDgMA6uvrcaq8HIGBgdDp9Ci2/R6w/uup0zv2bU3zviZTMfTN+up0ehQfds7V6537Nt/nDpt9Hsf7fHlq3N44Zk9me+pvGgDMx8oRfGUP+2N9L3+YjpU7tDly/BSmPLkE1037M+b+5V8AgPLKavv6pF8Owoate1DfpBC3h1tnL5D8gGQcyZEAygAcbO824uITkJ+fh6LCQtTW1mL1qpUYmzjOoc1tibdj2dIlAIB1a1Zj1OgxUEphbOI4rF61EjU1NSgqLER+fh7iE4a4lBuf4JibunKFU+7YxHFYbstdu2Y1Rt1wNjd15QqH3IQhruV6Ovt8jvf58tS4vXHMnsz21N80AGTvPYzQ3legr64nLvXphEk3DcbGf+c6tAn097M/t0/89hdYku74nvHkmwZj1ZbvXM504ubZC71s/+0DYD+AHu2dvVBV28i1aRkMDQ2jwWjk3HkvsKq2kbOfeoar1qxnVW0jT5yq4oSJyTSGhDAuPoG5+/PtfefOe4EGo5FhYf25bsPGdn2avm7DRoaGWXOfn7+AljpyztPPMnVtGi11ZFmFhROSzubuPVBg7/v8/AXW3P79uT59k8ufKGuZ3dqxOJ/j3advXwYEBNDPz486vd7pU+m2jrcnx63FmOU80/ZvuqXZC74Jszh+5ns8eKiEBYeP8bl3NtI3YRZf/NtmJs36gL4Js3jHk/9g3qESHjxUwg/Xf8vuw56w9+0/7gWajp7kZUMea3HbrsxeUO68ZYtS6t8AAgHUAZhF8vO22sfGxfPrb7Pctj9t8cbbx3jr7Xq8ddye4snj7bHb9eQuR+Pp/7X4ZLv1HmkkR7hz+0IIcaG5IL6RJoQQFwspukIIoSEpukIIoSEpukIIoSEpukIIoSEpukIIoSEpukIIoSEpukIIoSEpukIIoSEpukIIoSG3XnuhvZRSxwAc+ondrwBw/GfcnQsh2xvH7K3Z3jjmCzm7L0nna9WigxXd86GUyiYZ703Z3jhmb832xjFfrNny9oIQQmhIiq4QQmjoYiq673lhtjeO2VuzvXHMF2X2RfOerhBCXAguple6QgjR4UnRFUIIDV2QRVcpdYtS6oBSKl8pNVvD3A+VUiVKqRytMptk91ZKfamU2quUylVKzdQw21cplamU2m3LnqdVti2/k1LqO6VUhsa5RUqpPUqpXUqpbI2zeyilViul9iul9imlrtMod4BtvGeWU0qpRzTKftR2fuUopT5WSvlqkWvLnmnLzXX7eN15N2A33WG4E4ACAEYAnQHsBnCNRtkjAcQCyPHAuK8GEGv7uRust7PXatwKwOW2ny8FsB3AtRqOfRaAjwBkaHzMiwBcofVzbcteAuAe28+dcY47abtpHzoB+B+sE/3dnaUHUAjgMtvjVQB+q9E4owDkAOgK630jPwMQ6q68C/GV7hAA+SR/IFkLYAWA8VoEk/wKwAktslrIPkJyp+3nCgD7YD1RtcgmyUrbw0ttiyafwCqlggGMBfC+FnkdgVLKH9Z/4D8AAJK1JE96YFd+AaCA5E/9lmh7+QC4TCnlA2sBNGuUGwFgO8kqkvUAtgGY6K6wC7Ho6gEcbvK4GBoVn45CKdUPwGBYX3FqldlJKbULQAmAT0lqlf06gD8CaNQorykC2KKU2qGUmq5hrgHAMQB/t72t8r5Syk/D/DOmAPhYiyCSJgCvAfgRwBEA5SS3aJEN66vcEUqpQKVUVwC3AejtrrALseh6NaXU5QDWAHiE5Cmtckk2kBwEIBjAEKVUlLszlVKJAEpI7nB3ViuuJxkL4FYADyqlRmqU6wPr21h/ITkYwGkAmn12AQBKqc4AxgFI1SgvANb/YzUA0AHwU0pN1SKb5D4ArwDYAuATALsANLgr70IsuiY4/isUbPvdRU8pdSmsBXc5ybWe2Afb/+Z+CeAWDeKGAxinlCqC9W2kMUqpZRrkArC/+gLJEgDrYH1rSwvFAIqb/N/EaliLsJZuBbCT5FGN8n4JoJDkMZJ1ANYCGKZRNkh+QDKO5EgAZbB+ZuIWF2LRzQIQppQy2P41ngJgg4f3ye2UUgrW9/j2kfyzxtlBSqketp8vA3AjgP3uziU5h2QwyX6wPs9fkNTk1Y9Syk8p1e3MzwBugvV/Q92O5P8AHFZKDbD96hcA9mqR3cQd0OitBZsfAVyrlOpqO9d/AevnFppQSvWy/bcPrO/nfuSuLB93bdhdSNYrpWYA2Azrp6sfkszVIlsp9TGA0QCuUEoVA5hL8gMtsmF91TcNwB7be6sA8BTJTRpkXw1giVKqE6z/UK8iqen0LQ+4EsA6698/fAB8RPITDfMfArDc9sLiBwC/0yrY9o/MjQDu0yqT5Hal1GoAOwHUA/gO2n4FeI1SKhBAHYAH3fnBpXwNWAghNHQhvr0ghBAXLCm6QgihISm6QgihISm6QgihISm6QgihISm6wq2UUg22q1XlKKVSbV+z/Knb+odSKtn28/tKqWvaaDtaKdXuyfW2K4td4ervm7WpbGt9C+2fV0o93t59FBc2KbrC3SwkB5GMAlAL4A9NV9oubtJuJO8h2dYXBkZDw280CeEqKbpCS/8GEGp7FfpvpdQGAHttF9NZqJTKUkp9r5S6D7B+C08ptdh27eTPAPQ6syGl1FalVLzt51uUUjtt1/v93HZBoD8AeNT2KnuE7Vt1a2wZWUqp4ba+gUqpLbbrqL4P62Us26SUWm+7CE5u8wvhKKUW2X7/uVIqyPa7EKXUJ7Y+/1ZKhf8sR1NckC64b6SJC5PtFe2tsF5QBLBeSyCKZKGtcJWTTFBKdQHwtVJqC6xXUhsA4BpYvyG2F8CHzbYbBOBvAEbattWT5Aml1LsAKkm+Zmv3EYBFJP9j+6rnZlgv6TcXwH9IzldKjQVwtwvD+b0t4zIAWUqpNSRLAfgByCb5qFLqOdu2Z8D6zao/kMxTSg0F8A6AMT/hMIqLgBRd4W6XNfna8r9hvX7EMACZJAttv78JQMyZ92sB+AMIg/Wash+TbABgVkp90cL2rwXw1ZltkWztese/BHCN7Wu9ANDddsW2kbBdO5XkRqVUmQtjelgpNcH2c2/bvpbCevnJlbbfLwOw1pYxDEBqk+wuLmSIi5QUXeFuFtslIe1sxed0018BeIjk5mbtbvsZ9+MSWO92Ud3CvrhMKTUa1gJ+HckqpdRWAK3dVoa23JPNj4HwXvKerugINgO433bpSiil+tsuuvIVgBTbe75XA7ihhb7fAhiplDLY+va0/b4C1tsanbEF1ovIwNZukO3HrwDcafvdrQACzrGv/gDKbAU3HNZX2mdcAuDMq/U7YX3b4hSAQqXUJFuGUkoNPEeGuIhJ0RUdwfuwvl+7U1lv+vlXWP8vbB2APNu6fwL4pnlHkscATIf1f+V34+z/3qcDmHDmgzQADwOIt31QtxdnZ1HMg7Vo58L6NsOP59jXTwD4KKX2AXgZ1qJ/xmlYL/CeA+t7tvNtv78LwN22/cuFRreXEh2TXGVMCCE0JK90hRBCQ1J0hRBCQ1J0hRBCQ1J0hRBCQ1J0hRBCQ1J0hRBCQ1J0hRBCQ/8PfPLCdqSvTeoAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 360x360 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "utils.plot_confusion_matrix(y_test, y_pred_teacher, classes=list(range(10)), normalize=True)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Calculate teacher predictions on train set as well\n",
    "These will be later used for training the student model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_train_pred_teacher = teacher(X_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Student model\n",
    "A much smaller and shallow neural network is used as the student model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "class StudentModel(Model):\n",
    "    def __init__(self, T):\n",
    "        super(StudentModel, self).__init__()\n",
    "        \n",
    "        self.T = T\n",
    "        \n",
    "        self.input_layer = tf.keras.layers.Flatten(input_shape=(28, 28))\n",
    "        self.d1 = Dense(10, activation=\"relu\")\n",
    "        self.d2 = Dense(10)\n",
    "        self.output_layer = tf.keras.layers.Softmax()\n",
    "\n",
    "    def call(self, x):\n",
    "        x = self.input_layer(x)\n",
    "        x = self.d1(x)\n",
    "        x = self.d2(x)\n",
    "        x = self.output_layer(x / self.T)\n",
    "        return x"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Train small model on original labels"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First we will train the small model on original labels and evaluate its performance on generalizing on test data.  \n",
    "We expect that its performance will be worse than the teacher model since it has far less parameters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "small_model = StudentModel(T=1.0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "optimizer = tf.keras.optimizers.Adam()\n",
    "loss = tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True)\n",
    "\n",
    "small_model.compile(\n",
    "    loss=loss,\n",
    "    optimizer=optimizer,\n",
    "    metrics=[\"accuracy\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fold: 1, accuracy: 0.84905\n",
      "Fold: 2, accuracy: 0.85485\n",
      "Fold: 3, accuracy: 0.86305\n"
     ]
    }
   ],
   "source": [
    "num_epochs = 3\n",
    "batch_size = 32\n",
    "\n",
    "cv = 3\n",
    "scores = np.zeros(cv)\n",
    "kf = KFold(n_splits=cv)\n",
    "for i, (train_index, test_index) in enumerate(kf.split(X_train)):\n",
    "    X_train_kf, X_test_kf = X_train[train_index], X_train[test_index]\n",
    "    y_train_kf, y_test_kf = y_train[train_index], y_train[test_index]\n",
    "\n",
    "    small_model.fit(X_train_kf, y_train_kf, batch_size=batch_size, epochs=num_epochs, verbose=0)\n",
    "    y_pred = small_model.predict(X_test_kf)\n",
    "    y_pred = np.argmax(y_pred, axis=1)\n",
    "\n",
    "    score = accuracy_score(y_test_kf, y_pred)\n",
    "    scores[i] = score\n",
    "\n",
    "    print(f\"Fold: {i + 1}, accuracy: {score}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.8556499999999999"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "scores.mean()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "small_model.fit(X_train, y_train, batch_size=batch_size, epochs=num_epochs, verbose=0)\n",
    "y_pred_small = np.argmax(small_model.predict(X_test), axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.8634"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "acc = accuracy_score(y_test, y_pred_small)\n",
    "acc"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1366"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "int((1 - acc)*num_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The small model trained on original labels achieved **1366 test errors**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Distill knowledge of the big teacher model to the small student model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [],
   "source": [
    "student_model = StudentModel(T=3.5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [],
   "source": [
    "optimizer = tf.keras.optimizers.Adam()\n",
    "loss = tf.keras.losses.CategoricalCrossentropy()\n",
    "\n",
    "student_model.compile(\n",
    "    loss=loss,\n",
    "    optimizer=optimizer,\n",
    "    metrics=[\"accuracy\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fold: 1, accuracy: 0.9148\n",
      "Fold: 2, accuracy: 0.9235\n",
      "Fold: 3, accuracy: 0.9332\n"
     ]
    }
   ],
   "source": [
    "num_epochs = 3\n",
    "batch_size = 32\n",
    "\n",
    "cv = 3\n",
    "scores = np.zeros(cv)\n",
    "kf = KFold(n_splits=cv)\n",
    "for i, (train_index, test_index) in enumerate(kf.split(X_train)):\n",
    "    X_train_kf, X_test_kf = X_train[train_index], X_train[test_index]\n",
    "    y_train_kf, y_test_kf = y_train_pred_teacher.numpy()[train_index], y_train_pred_teacher.numpy()[test_index]\n",
    "\n",
    "    student_model.fit(X_train_kf, y_train_kf, batch_size=batch_size, epochs=num_epochs, verbose=0)\n",
    "    y_pred = student_model.predict(X_test_kf)\n",
    "    y_pred = np.argmax(y_pred, axis=1)\n",
    "\n",
    "    score = accuracy_score(np.argmax(y_test_kf, axis=1), y_pred)\n",
    "    scores[i] = score\n",
    "\n",
    "    print(f\"Fold: {i + 1}, accuracy: {score}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.9238333333333332"
      ]
     },
     "execution_count": 61,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "scores.mean()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [],
   "source": [
    "student_model.fit(X_train, y_train_pred_teacher, batch_size=batch_size, epochs=num_epochs, verbose=0)\n",
    "y_pred_student = np.argmax(student_model(X_test), axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.9303"
      ]
     },
     "execution_count": 63,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "acc = accuracy_score(y_test, y_pred_student)\n",
    "acc"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "696"
      ]
     },
     "execution_count": 64,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "int((1 - acc)*num_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The distilled model trained on soft labels achieved **696 test errors**, **50% fewer errors than the same model trained on raw labels**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}