gradient_descent.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": [],
      "authorship_tag": "ABX9TyNuLKRVax4m/2yttgIPF38Q",
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/zrcjessica/ml_concepts/blob/main/gradient_descent.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 215,
      "metadata": {
        "id": "bG_RtrEqaEHM"
      },
      "outputs": [],
      "source": [
        "import numpy as np\n",
        "import matplotlib.pyplot as plt\n",
        "import pandas as pd\n",
        "import time"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "np.random.seed(1)"
      ],
      "metadata": {
        "id": "qRntnI6Pj0eX"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "# The goal of this tutorial\n",
        "\n",
        "The goal of this tutorial is to implement batch, mini-batch, and stochastic gradient descent on this data to compare how well they manage to recover the coefficients used to generate a set of simulated data points. Because gradient descent is an optimization algorithm, by training on subsets of a dataset simulated with a certain set of parameters, we should be able to recover the coefficients used to simulate the data."
      ],
      "metadata": {
        "id": "cWMOb5QfoRB1"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "\n",
        "# Simulating the data\n",
        "\n",
        "We will generate a toy dataset containing $n=10000$ data points using a linear function defined as:\n",
        "\n",
        "$$\n",
        "f(x_1, x_2, x_3) = a x_1 + b x_2 + c x_3 + ε\n",
        "$$\n",
        "\n",
        "Where ${x_1,x_2,x_3}$ represent an input vector of size 3 and $ε$ represents noise.\n",
        "\n",
        "To do this, we will first simulate the coefficients of the model, $a,b,$ and $c$ by picking 3 random integers between 1 and 10. "
      ],
      "metadata": {
        "id": "B2WWguzTnH0E"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "coeffs = np.random.randint(1,10, size= 3)\n",
        "\n",
        "print(\"Coefficients: a = %d, b = %d, c = %d\" % (coeffs[0], coeffs[1], coeffs[2]))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "MqZFI_9cjw4P",
        "outputId": "4cb095db-2939-4d5d-e2ac-73789ab34fbd"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Coefficients: a = 1, b = 1, c = 2\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Next, we will generate the inputs and the noise for each sample by randomly sampling from the standard normal distribution, and then calculate the outputs based on the equation defined above."
      ],
      "metadata": {
        "id": "iSVCDCykn7OW"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# define number of data points\n",
        "n = 10000\n",
        "\n",
        "# define inputs in dataset\n",
        "X = np.random.randn(n,3)\n",
        "\n",
        "# define noise\n",
        "noise = np.random.randn(n)\n",
        "\n",
        "# get outputs for dataset\n",
        "y = np.sum(X*coeffs, axis = 1) + noise"
      ],
      "metadata": {
        "id": "sglcqdThkLZI"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Just to help us get a better idea of what the data looks like, let's coerce the dataset we've generated into a data frame and preview it."
      ],
      "metadata": {
        "id": "GU8_Rmu_opqB"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "pd.DataFrame(np.hstack((X, np.reshape(y, (-1,1)))), columns = [\"x1\",\"x2\",\"x3\",\"y\"])"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 424
        },
        "id": "rbgS4GCSpFE4",
        "outputId": "62d95514-7a7a-4007-ee86-1336b1b95816"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "            x1        x2        x3         y\n",
              "0    -0.528172 -1.072969  0.865408  1.416162\n",
              "1    -2.301539  1.744812 -0.761207 -2.431354\n",
              "2     0.319039 -0.249370  1.462108  5.236786\n",
              "3    -2.060141 -0.322417 -0.384054 -0.450817\n",
              "4     1.133769 -1.099891 -0.172428  0.620125\n",
              "...        ...       ...       ...       ...\n",
              "9995 -1.923271  0.199603  0.629266  0.765994\n",
              "9996  0.843702 -2.027953 -0.035839 -0.592611\n",
              "9997 -1.259567  1.096904  0.689824  1.259755\n",
              "9998  0.552820 -0.083838 -1.283535 -1.447908\n",
              "9999  0.342312  0.386055 -0.499152 -2.034884\n",
              "\n",
              "[10000 rows x 4 columns]"
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-2c001d50-73ee-4f14-8a1a-051a09f21647\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>x1</th>\n",
              "      <th>x2</th>\n",
              "      <th>x3</th>\n",
              "      <th>y</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>-0.528172</td>\n",
              "      <td>-1.072969</td>\n",
              "      <td>0.865408</td>\n",
              "      <td>1.416162</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>-2.301539</td>\n",
              "      <td>1.744812</td>\n",
              "      <td>-0.761207</td>\n",
              "      <td>-2.431354</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>0.319039</td>\n",
              "      <td>-0.249370</td>\n",
              "      <td>1.462108</td>\n",
              "      <td>5.236786</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>-2.060141</td>\n",
              "      <td>-0.322417</td>\n",
              "      <td>-0.384054</td>\n",
              "      <td>-0.450817</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>1.133769</td>\n",
              "      <td>-1.099891</td>\n",
              "      <td>-0.172428</td>\n",
              "      <td>0.620125</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>...</th>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9995</th>\n",
              "      <td>-1.923271</td>\n",
              "      <td>0.199603</td>\n",
              "      <td>0.629266</td>\n",
              "      <td>0.765994</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9996</th>\n",
              "      <td>0.843702</td>\n",
              "      <td>-2.027953</td>\n",
              "      <td>-0.035839</td>\n",
              "      <td>-0.592611</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9997</th>\n",
              "      <td>-1.259567</td>\n",
              "      <td>1.096904</td>\n",
              "      <td>0.689824</td>\n",
              "      <td>1.259755</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9998</th>\n",
              "      <td>0.552820</td>\n",
              "      <td>-0.083838</td>\n",
              "      <td>-1.283535</td>\n",
              "      <td>-1.447908</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9999</th>\n",
              "      <td>0.342312</td>\n",
              "      <td>0.386055</td>\n",
              "      <td>-0.499152</td>\n",
              "      <td>-2.034884</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "<p>10000 rows × 4 columns</p>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-2c001d50-73ee-4f14-8a1a-051a09f21647')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "        \n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "      \n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-2c001d50-73ee-4f14-8a1a-051a09f21647 button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-2c001d50-73ee-4f14-8a1a-051a09f21647');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n",
              "  "
            ]
          },
          "metadata": {},
          "execution_count": 6
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Now let's cover the concepts that will be important for this tutorial."
      ],
      "metadata": {
        "id": "Du2CAvfYqanJ"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Gradient descent \n",
        "\n",
        "Gradient descent is an optimization method that is popularly used in machine learning to find the best parameters (more often referred to as _weights_ in the case of neural networks) for a given model. It works quite like how it sounds - by following gradients to descend towards the minimum of a **cost function**. Cost functions are a function of the difference between the true and predicted outputs of a given model. There are a number of different cost functions out there. For example, cross entropy cost functions are popularly used for classification problems while squared error cost functions are popular used for regression problems. Conceptually, gradient descent looks something like this:\n",
        "\n",
        "![gradient-descent.jpeg](data:image/jpeg;base64,/9j/4RuERXhpZgAATU0AKgAAAAgABwESAAMAAAABAAEAAAEaAAUAAAABAAAAYgEbAAUAAAABAAAAagEoAAMAAAABAAIAAAExAAIAAAAeAAAAcgEyAAIAAAAUAAAAkIdpAAQAAAABAAAApAAAANAALca2AAAnEAAtxrYAACcQQWRvYmUgUGhvdG9zaG9wIENTNiAoV2luZG93cykAMjAxODowNjoyMiAxNjo0NjozMQAAA6ABAAMAAAABAAEAAKACAAQAAAABAAACWKADAAQAAAABAAABdgAAAAAAAAAGAQMAAwAAAAEABgAAARoABQAAAAEAAAEeARsABQAAAAEAAAEmASgAAwAAAAEAAgAAAgEABAAAAAEAAAEuAgIABAAAAAEAABpOAAAAAAAAAEgAAAABAAAASAAAAAH/2P/iDFhJQ0NfUFJPRklMRQABAQAADEhMaW5vAhAAAG1udHJSR0IgWFlaIAfOAAIACQAGADEAAGFjc3BNU0ZUAAAAAElFQyBzUkdCAAAAAAAAAAAAAAAAAAD21gABAAAAANMtSFAgIAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAEWNwcnQAAAFQAAAAM2Rlc2MAAAGEAAAAbHd0cHQAAAHwAAAAFGJrcHQAAAIEAAAAFHJYWVoAAAIYAAAAFGdYWVoAAAIsAAAAFGJYWVoAAAJAAAAAFGRtbmQAAAJUAAAAcGRtZGQAAALEAAAAiHZ1ZWQAAANMAAAAhnZpZXcAAAPUAAAAJGx1bWkAAAP4AAAAFG1lYXMAAAQMAAAAJHRlY2gAAAQwAAAADHJUUkMAAAQ8AAAIDGdUUkMAAAQ8AAAIDGJUUkMAAAQ8AAAIDHRleHQAAAAAQ29weXJpZ2h0IChjKSAxOTk4IEhld2xldHQtUGFja2FyZCBDb21wYW55AABkZXNjAAAAAAAAABJzUkdCIElFQzYxOTY2LTIuMQAAAAAAAAAAAAAAEnNSR0IgSUVDNjE5NjYtMi4xAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAABYWVogAAAAAAAA81EAAQAAAAEWzFhZWiAAAAAAAAAAAAAAAAAAAAAAWFlaIAAAAAAAAG+iAAA49QAAA5BYWVogAAAAAAAAYpkAALeFAAAY2lhZWiAAAAAAAAAkoAAAD4QAALbPZGVzYwAAAAAAAAAWSUVDIGh0dHA6Ly93d3cuaWVjLmNoAAAAAAAAAAAAAAAWSUVDIGh0dHA6Ly93d3cuaWVjLmNoAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAGRlc2MAAAAAAAAALklFQyA2MTk2Ni0yLjEgRGVmYXVsdCBSR0IgY29sb3VyIHNwYWNlIC0gc1JHQgAAAAAAAAAAAAAALklFQyA2MTk2Ni0yLjEgRGVmYXVsdCBSR0IgY29sb3VyIHNwYWNlIC0gc1JHQgAAAAAAAAAAAAAAAAAAAAAAAAAAAABkZXNjAAAAAAAAACxSZWZlcmVuY2UgVmlld2luZyBDb25kaXRpb24gaW4gSUVDNjE5NjYtMi4xAAAAAAAAAAAAAAAsUmVmZXJlbmNlIFZpZXdpbmcgQ29uZGl0aW9uIGluIElFQzYxOTY2LTIuMQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAdmlldwAAAAAAE6T+ABRfLgAQzxQAA+3MAAQTCwADXJ4AAAABWFlaIAAAAAAATAlWAFAAAABXH+dtZWFzAAAAAAAAAAEAAAAAAAAAAAAAAAAAAAAAAAACjwAAAAJzaWcgAAAAAENSVCBjdXJ2AAAAAAAABAAAAAAFAAoADwAUABkAHgAjACgALQAyADcAOwBAAEUASgBPAFQAWQBeAGMAaABtAHIAdwB8AIEAhgCLAJAAlQCaAJ8ApACpAK4AsgC3ALwAwQDGAMsA0ADVANsA4ADlAOsA8AD2APsBAQEHAQ0BEwEZAR8BJQErATIBOAE+AUUBTAFSAVkBYAFnAW4BdQF8AYMBiwGSAZoBoQGpAbEBuQHBAckB0QHZAeEB6QHyAfoCAwIMAhQCHQImAi8COAJBAksCVAJdAmcCcQJ6AoQCjgKYAqICrAK2AsECywLVAuAC6wL1AwADCwMWAyEDLQM4A0MDTwNaA2YDcgN+A4oDlgOiA64DugPHA9MD4APsA/kEBgQTBCAELQQ7BEgEVQRjBHEEfgSMBJoEqAS2BMQE0wThBPAE/gUNBRwFKwU6BUkFWAVnBXcFhgWWBaYFtQXFBdUF5QX2BgYGFgYnBjcGSAZZBmoGewaMBp0GrwbABtEG4wb1BwcHGQcrBz0HTwdhB3QHhgeZB6wHvwfSB+UH+AgLCB8IMghGCFoIbgiCCJYIqgi+CNII5wj7CRAJJQk6CU8JZAl5CY8JpAm6Cc8J5Qn7ChEKJwo9ClQKagqBCpgKrgrFCtwK8wsLCyILOQtRC2kLgAuYC7ALyAvhC/kMEgwqDEMMXAx1DI4MpwzADNkM8w0NDSYNQA1aDXQNjg2pDcMN3g34DhMOLg5JDmQOfw6bDrYO0g7uDwkPJQ9BD14Peg+WD7MPzw/sEAkQJhBDEGEQfhCbELkQ1xD1ERMRMRFPEW0RjBGqEckR6BIHEiYSRRJkEoQSoxLDEuMTAxMjE0MTYxODE6QTxRPlFAYUJxRJFGoUixStFM4U8BUSFTQVVhV4FZsVvRXgFgMWJhZJFmwWjxayFtYW+hcdF0EXZReJF64X0hf3GBsYQBhlGIoYrxjVGPoZIBlFGWsZkRm3Gd0aBBoqGlEadxqeGsUa7BsUGzsbYxuKG7Ib2hwCHCocUhx7HKMczBz1HR4dRx1wHZkdwx3sHhYeQB5qHpQevh7pHxMfPh9pH5Qfvx/qIBUgQSBsIJggxCDwIRwhSCF1IaEhziH7IiciVSKCIq8i3SMKIzgjZiOUI8Ij8CQfJE0kfCSrJNolCSU4JWgllyXHJfcmJyZXJocmtyboJxgnSSd6J6sn3CgNKD8ocSiiKNQpBik4KWspnSnQKgIqNSpoKpsqzysCKzYraSudK9EsBSw5LG4soizXLQwtQS12Last4S4WLkwugi63Lu4vJC9aL5Evxy/+MDUwbDCkMNsxEjFKMYIxujHyMioyYzKbMtQzDTNGM38zuDPxNCs0ZTSeNNg1EzVNNYc1wjX9Njc2cjauNuk3JDdgN5w31zgUOFA4jDjIOQU5Qjl/Obw5+To2OnQ6sjrvOy07azuqO+g8JzxlPKQ84z0iPWE9oT3gPiA+YD6gPuA/IT9hP6I/4kAjQGRApkDnQSlBakGsQe5CMEJyQrVC90M6Q31DwEQDREdEikTORRJFVUWaRd5GIkZnRqtG8Ec1R3tHwEgFSEtIkUjXSR1JY0mpSfBKN0p9SsRLDEtTS5pL4kwqTHJMuk0CTUpNk03cTiVObk63TwBPSU+TT91QJ1BxULtRBlFQUZtR5lIxUnxSx1MTU19TqlP2VEJUj1TbVShVdVXCVg9WXFapVvdXRFeSV+BYL1h9WMtZGllpWbhaB1pWWqZa9VtFW5Vb5Vw1XIZc1l0nXXhdyV4aXmxevV8PX2Ffs2AFYFdgqmD8YU9homH1YklinGLwY0Njl2PrZEBklGTpZT1lkmXnZj1mkmboZz1nk2fpaD9olmjsaUNpmmnxakhqn2r3a09rp2v/bFdsr20IbWBtuW4SbmtuxG8eb3hv0XArcIZw4HE6cZVx8HJLcqZzAXNdc7h0FHRwdMx1KHWFdeF2Pnabdvh3VnezeBF4bnjMeSp5iXnnekZ6pXsEe2N7wnwhfIF84X1BfaF+AX5ifsJ/I3+Ef+WAR4CogQqBa4HNgjCCkoL0g1eDuoQdhICE44VHhauGDoZyhteHO4efiASIaYjOiTOJmYn+imSKyoswi5aL/IxjjMqNMY2Yjf+OZo7OjzaPnpAGkG6Q1pE/kaiSEZJ6kuOTTZO2lCCUipT0lV+VyZY0lp+XCpd1l+CYTJi4mSSZkJn8mmia1ZtCm6+cHJyJnPedZJ3SnkCerp8dn4uf+qBpoNihR6G2oiailqMGo3aj5qRWpMelOKWpphqmi6b9p26n4KhSqMSpN6mpqhyqj6sCq3Wr6axcrNCtRK24ri2uoa8Wr4uwALB1sOqxYLHWskuywrM4s660JbSctRO1irYBtnm28Ldot+C4WbjRuUq5wro7urW7LrunvCG8m70VvY++Cr6Evv+/er/1wHDA7MFnwePCX8Lbw1jD1MRRxM7FS8XIxkbGw8dBx7/IPci8yTrJuco4yrfLNsu2zDXMtc01zbXONs62zzfPuNA50LrRPNG+0j/SwdNE08bUSdTL1U7V0dZV1tjXXNfg2GTY6Nls2fHadtr724DcBdyK3RDdlt4c3qLfKd+v4DbgveFE4cziU+Lb42Pj6+Rz5PzlhOYN5pbnH+ep6DLovOlG6dDqW+rl63Dr++yG7RHtnO4o7rTvQO/M8Fjw5fFy8f/yjPMZ86f0NPTC9VD13vZt9vv3ivgZ+Kj5OPnH+lf65/t3/Af8mP0p/br+S/7c/23////tAAxBZG9iZV9DTQAB/+4ADkFkb2JlAGSAAAAAAf/bAIQADAgICAkIDAkJDBELCgsRFQ8MDA8VGBMTFRMTGBEMDAwMDAwRDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAENCwsNDg0QDg4QFA4ODhQUDg4ODhQRDAwMDAwREQwMDAwMDBEMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwM/8AAEQgAZACgAwEiAAIRAQMRAf/dAAQACv/EAT8AAAEFAQEBAQEBAAAAAAAAAAMAAQIEBQYHCAkKCwEAAQUBAQEBAQEAAAAAAAAAAQACAwQFBgcICQoLEAABBAEDAgQCBQcGCAUDDDMBAAIRAwQhEjEFQVFhEyJxgTIGFJGhsUIjJBVSwWIzNHKC0UMHJZJT8OHxY3M1FqKygyZEk1RkRcKjdDYX0lXiZfKzhMPTdePzRieUpIW0lcTU5PSltcXV5fVWZnaGlqa2xtbm9jdHV2d3h5ent8fX5/cRAAICAQIEBAMEBQYHBwYFNQEAAhEDITESBEFRYXEiEwUygZEUobFCI8FS0fAzJGLhcoKSQ1MVY3M08SUGFqKygwcmNcLSRJNUoxdkRVU2dGXi8rOEw9N14/NGlKSFtJXE1OT0pbXF1eX1VmZ2hpamtsbW5vYnN0dXZ3eHl6e3x//aAAwDAQACEQMRAD8A6zD+qvV8VmNXT1CvGFFu+9+PU9j8iBS05ecTc5mVnZDKLa7/AFq/R/XPX/SZONj2q/8AV/o3Uumue/PzTnPfRVTucXlwNT8mzc422Wb3PryamPf+i/mUvrflZ2L0DJfgCwZDgGNtr5qadbch37vpVNf/AG1qYd1t+JTfdU6i2ytrrKXfSY4iX1u/qO9qSkySblOkpSSSSSlJJJJKUkkkkpSSSSSlJJJJKUkkkkpSSSSSlJJJJKf/0PQfrP8A+J7qP/hd/wCRaiy/rP8A+J7qP/hd/wCRaiSnK+robVhW4IG04OTfRs/dZvdfiN/9gr8Zaq45nTOo4317tuOWa682p19LiHFryxvofY7K5ZU702+nb7X+r6ON6n6P9JYzpv2gyvGF17C17bWUWVt90WPezHbt+j+jc+1ljH/6D9InTjGNcMhO4iRr9GR3h6v3VNtJUH9d6Qyyyp2UzfVY2myJIa95LWNe4Da33sfXv/m2W/of51DH1i6QWhzbi4Hb+Y/cd43VbK3NFlzn/Q2Usss3vrTVOmks/wDb/SPSuuGS01Y4YbLAHFsWuNVJreG7bt9zH0/ofU/TMsp/nGKH/OPo+57DeRawuDqix/qe2w47ttIZ6j/0g/Nb/M+nf/M2b0lOmkoseyxjXscHseA5rmmQQdQ5pCkkpSSSSSlJJJJKUkkkkpSz+l5N+Vf1Cxzy7HZkmnGEAANqZVXfqBud+u/ame/9xH6jm19PwMjNsBc3Hrc/YOXED21M/wCEtd+jr/lofR8OzB6Zj49xD8gN35Dxw66wm7Ks/wCuZFlr0lN1JJJJT//R9B+s/wD4nuo/+F3/AJFqLL+s/wD4nuo/+F3/AJFqJKcbq+Pdk33fY7C7qGFXRl4lLtGCxjslu31J2t/aFPr4GR/o6FHAzukfWA5NdB9fGtrottb9EtsJs9rtrt9eRV9mq+j/ADVlS0mv3dQupNwc0U1u+z7fo7nXt9b1PzvW2bNn/Afy1zmP0Dp/1azw9puOD1AtoGQXw7FuLv0Ld9Yr/Vs1xZVvs/7V+l6vrfbf0ToiHDKyRLTgAHpP73Ep23fV/o7vp4zXe4PJcXElw/OO535/85b/AKa79Nb+lRz0zp5ex5x2F1RDq3Rq0tMtLf8ANT4dz3epj3PD8jHIa8iAXNcN9VpYPo+oz/wVlvpqymqc2n6vdMqbk1hjnVZbWMfUXEMaysvfXTSyvZ6VfqXWv/64k/6vdIcCG0ekCQT6TnMna71Y/Rubt937n+C/QfzP6NaSSSmLGMYxrGNDWNADWgQABw1oUkkklKSSSSUpJJJJSkklS6r1L9n47XMrORlXvFOJjNMG21wLms3Q706mNa+7Iu2/oMeu278xJTVzT+0er0dObrj4BZmZp7GwGenYx/68z7e//R/ZcX/B5S11S6V084GLsss9fJucbsvIIg2XPj1LNuu2trWspx6v8BjVU0f4JXUlKSSSSU//0vQfrP8A+J7qP/hd/wCRaiy/rP8A+J7qP/hd/wCRaiSmswu/aNo31lvo1EMH86Duvl9n/Av/AMF/LZcp5uM3Lw78RxAbkVvqJIDhD2lmrHe1/wBL6KgwO/aFp21hvo1Q8fzpO6/22f8AAt/wP8v11ZSU8d0Lp3Uvq3Y/Gz8rbXmOaMfKA9Sj1Z2NpzfU9O6u+/cxtF3qV15H8xZb9r9H7V0uN1LHtcabLK68qt/p2U7wTujf+j+j6jXsdvb/ANP9L6jEXPxGZuDkYTztZk1PpcQAYFjTWfa72/nLkq+k5X1Q6RlPeWdS6dYAb6GzVZW6W1UPxXWOuZY3cWep6r67aP5yq670fRUp/WmUjL9bKURGAj8/F/d9EFPaJLl+j/Wym/pVfUcnIaRXY3FzWODWubYYZ69TP0b2s/w1tNrPU9L1vT/mP0vQ1ZmPbY6oO22sG51TwWPDePU2P27q/wDhG/o0yUJQkYyFGJMSP60d1J0lW6lj35XTsrGx7DTdfS+uq5rixzHPaWMtZY33MfW472vWS3p/1oFllv2ytrrXutLQ4uYCKq6qaK2XUvbXjPuo9TI9P9L+lvfU/wBSxNU76Swq8T60Uhrxlsu2bJx3Fu0ta6z1GMuOP6+99LMVjbLbX++zK/4FGwMX6w15FdmdlsurIi6lobtB/TSai2iq3/uF9Oz/ALl/8Akp10klkW9cOS92P0OsZ94Ja/InbiVOHPrZbQ71rG/9xsP17t/9I+y/zySm71DqOP0+gWW7nvedlGPWN1tthEtpx65bvs0/4uqv9PfZVRXZaq3TMHL9Z3U+qFpz7WljKmHdXjVEh/2Wh0N3ue5jH5eT/wBqrWM/wFGNTTPp/SW41783KtOZ1G1uyzJeNoayd32fEplzcXF3N/m2/pLfZ9qvybf0q0ElKSSSSUpJJJJT/9P0H6z/APie6j/4Xf8AkWosz6ytc/6v9Qaxpe449kNaC4n29mtlzk3/ADm6F/3KH+a//wAgkptsrI6hbb6IAdTU0ZG7Vxa693o+n+b6O/fv/wCHVlYTPrD0IdQuu9aC6mpnrQ/a7a693pBnp/Sq373f8crP/OfoX/cof5j/APyCSnUUXsa9pY8BzHAhzSJBB5BCzf8AnP0L/uUP8x//AJBL/nP0L/uUP8x//kElNbr31Q6b1mvHb/Q7MWW1W0NDSK3fSpEbfZ+4su36s/WOjq2M7EymZXTMVjGVU5Ty39GwBrsSxuPX6NnqOb/PPxrP8HZ/O49S3f8AnP0L/uUP8x//AJBL/nP0L/uUP8x//kFJHNMR4buNSiIyHEI+58xjxfpKeSr659ZcDAzrc3p9+Mypw+zurhlVNtry138+79Yxqv8AB7WWY/8A29VZRtdK+sWZk4tfULGWCrZ+mxntAe4udFV2CK62vtqe1r20M9/2p7/Srs9Sn9NpH6zdCIIOUCDyNj//ACCrX9c+rtmbjXuuLn0h+x4Y7a3cB/OE172/8FtTOZkckKxCODJxxkZxByR9qPz4fanL9P8AziQR1Ftodatef0PS8634srq/9u78dL7R9YsjSrEx8FpP85kWG54HniYrWVO/9yKX/OfoX/cof5j/APyCX/OfoX/cof5j/wDyCCGP7BGVr1fKt6iDzjuirF5nb9jp/pFf8jPszVqMYytja62hjGANYxogADRrWtCzf+c/Qv8AuUP8x/8A5BL/AJz9C/7lD/Mf/wCQSU6iSy/+c/Qv+5Q/zH/+QS/5z9C/7lD/ADH/APkElOoksv8A5z9C/wC5Q/zH/wDkEv8AnP0L/uUP8x//AJBJTqJLByvrv9XsW+uq69wZa0u9cMcWNIIbsfp6u58+39FsW3VbXdUy2s7q7GhzHeIIlpSU/wD/1PVVzT8H61UXZF2Ne2x19lpra+xxaxvrWW44sryDdV/R/Qr/AFSvD9Gn7R6n2u77NYtrqmHdm4bsem841jiC25oktg7vb7mf9Usqz6t5oufdTnu3EOawv9QuDNRXV6vr7nP9J2z7Y79Z/m/+EqSUrIx/rX6jdlwc0O9QbSxoEtd61Vs1brGMc7/J/wDwv/KHr1I+JT9Y/Xqtyb2+iH++ghk+nta1vqPrqb+s+577vSf6PrfzP6D9Go19Bz68SqlnUXi9hc664NcBaXMbrYxlzXb/ALTVTc+z1d/o/aMb/tT6qDX9Xer11vbX1RzH3Ni23a4vBG/0ix7rfe9rLGUvttZ/NUV/zdiSnoUlX6fj3Y2HVRfab7WNh1jiSSf6zve7b9De/wB6sJKUkkkkpo9XzM7DxW3YWIc6wvbWaWu2kepNdVn0X/omZDqftL/8Bievk/4H0nz6Xl3ZuDVl3478R925wx7dLGs3O9H1mEN9O59Ox9tX+Cf+jVtJJSkkkklKSSSSUpJJJJSkkkklIbMTGttbe+tpvY1zGXQPUa130212fTZuj81YX1fxfrRXmtd1iwux2UumHh266a8Rntb+Z9nwv2h/4Y6ndX/gV0aSSn//1fVUl8qpJKfqpJfKqSSn6qSXyqkkp+qkl8qpJKfqpJfKqSSn6qSXyqkkp+qkl8qpJKfqpJfKqSSn6qSXyqkkp+qkl8qpJKf/2f/tI45QaG90b3Nob3AgMy4wADhCSU0EJQAAAAAAEAAAAAAAAAAAAAAAAAAAAAA4QklNBDoAAAAAAOUAAAAQAAAAAQAAAAAAC3ByaW50T3V0cHV0AAAABQAAAABQc3RTYm9vbAEAAAAASW50ZWVudW0AAAAASW50ZQAAAABDbHJtAAAAD3ByaW50U2l4dGVlbkJpdGJvb2wAAAAAC3ByaW50ZXJOYW1lVEVYVAAAAAEAAAAAAA9wcmludFByb29mU2V0dXBPYmpjAAAADABQAHIAbwBvAGYAIABTAGUAdAB1AHAAAAAAAApwcm9vZlNldHVwAAAAAQAAAABCbHRuZW51bQAAAAxidWlsdGluUHJvb2YAAAAJcHJvb2ZDTVlLADhCSU0EOwAAAAACLQAAABAAAAABAAAAAAAScHJpbnRPdXRwdXRPcHRpb25zAAAAFwAAAABDcHRuYm9vbAAAAAAAQ2xicmJvb2wAAAAAAFJnc01ib29sAAAAAABDcm5DYm9vbAAAAAAAQ250Q2Jvb2wAAAAAAExibHNib29sAAAAAABOZ3R2Ym9vbAAAAAAARW1sRGJvb2wAAAAAAEludHJib29sAAAAAABCY2tnT2JqYwAAAAEAAAAAAABSR0JDAAAAAwAAAABSZCAgZG91YkBv4AAAAAAAAAAAAEdybiBkb3ViQG/gAAAAAAAAAAAAQmwgIGRvdWJAb+AAAAAAAAAAAABCcmRUVW50RiNSbHQAAAAAAAAAAAAAAABCbGQgVW50RiNSbHQAAAAAAAAAAAAAAABSc2x0VW50RiNQeGxAcr/74AAAAAAAAAp2ZWN0b3JEYXRhYm9vbAEAAAAAUGdQc2VudW0AAAAAUGdQcwAAAABQZ1BDAAAAAExlZnRVbnRGI1JsdAAAAAAAAAAAAAAAAFRvcCBVbnRGI1JsdAAAAAAAAAAAAAAAAFNjbCBVbnRGI1ByY0BZAAAAAAAAAAAAEGNyb3BXaGVuUHJpbnRpbmdib29sAAAAAA5jcm9wUmVjdEJvdHRvbWxvbmcAAAAAAAAADGNyb3BSZWN0TGVmdGxvbmcAAAAAAAAADWNyb3BSZWN0UmlnaHRsb25nAAAAAAAAAAtjcm9wUmVjdFRvcGxvbmcAAAAAADhCSU0D7QAAAAAAEAEr/74AAQABASv/vgABAAE4QklNBCYAAAAAAA4AAAAAAAAAAAAAP4AAADhCSU0EDQAAAAAABAAAAB44QklNBBkAAAAAAAQAAAAeOEJJTQPzAAAAAAAJAAAAAAAAAAABADhCSU0nEAAAAAAACgABAAAAAAAAAAE4QklNA/UAAAAAAEgAL2ZmAAEAbGZmAAYAAAAAAAEAL2ZmAAEAoZmaAAYAAAAAAAEAMgAAAAEAWgAAAAYAAAAAAAEANQAAAAEALQAAAAYAAAAAAAE4QklNA/gAAAAAAHAAAP////////////////////////////8D6AAAAAD/////////////////////////////A+gAAAAA/////////////////////////////wPoAAAAAP////////////////////////////8D6AAAOEJJTQQAAAAAAAACAAI4QklNBAIAAAAAAAYAAAAAAAA4QklNBDAAAAAAAAMBAQEAOEJJTQQtAAAAAAAGAAEAAAAEOEJJTQQIAAAAAAAQAAAAAQAAAkAAAAJAAAAAADhCSU0EHgAAAAAABAAAAAA4QklNBBoAAAAAA1UAAAAGAAAAAAAAAAAAAAF2AAACWAAAABAAZwByAGEAZABpAGUAbgB0AC0AZABlAHMAYwBlAG4AdAAAAAEAAAAAAAAAAAAAAAAAAAAAAAAAAQAAAAAAAAAAAAACWAAAAXYAAAAAAAAAAAAAAAAAAAAAAQAAAAAAAAAAAAAAAAAAAAAAAAAQAAAAAQAAAAAAAG51bGwAAAACAAAABmJvdW5kc09iamMAAAABAAAAAAAAUmN0MQAAAAQAAAAAVG9wIGxvbmcAAAAAAAAAAExlZnRsb25nAAAAAAAAAABCdG9tbG9uZwAAAXYAAAAAUmdodGxvbmcAAAJYAAAABnNsaWNlc1ZsTHMAAAABT2JqYwAAAAEAAAAAAAVzbGljZQAAABIAAAAHc2xpY2VJRGxvbmcAAAAAAAAAB2dyb3VwSURsb25nAAAAAAAAAAZvcmlnaW5lbnVtAAAADEVTbGljZU9yaWdpbgAAAA1hdXRvR2VuZXJhdGVkAAAAAFR5cGVlbnVtAAAACkVTbGljZVR5cGUAAAAASW1nIAAAAAZib3VuZHNPYmpjAAAAAQAAAAAAAFJjdDEAAAAEAAAAAFRvcCBsb25nAAAAAAAAAABMZWZ0bG9uZwAAAAAAAAAAQnRvbWxvbmcAAAF2AAAAAFJnaHRsb25nAAACWAAAAAN1cmxURVhUAAAAAQAAAAAAAG51bGxURVhUAAAAAQAAAAAAAE1zZ2VURVhUAAAAAQAAAAAABmFsdFRhZ1RFWFQAAAABAAAAAAAOY2VsbFRleHRJc0hUTUxib29sAQAAAAhjZWxsVGV4dFRFWFQAAAABAAAAAAAJaG9yekFsaWduZW51bQAAAA9FU2xpY2VIb3J6QWxpZ24AAAAHZGVmYXVsdAAAAAl2ZXJ0QWxpZ25lbnVtAAAAD0VTbGljZVZlcnRBbGlnbgAAAAdkZWZhdWx0AAAAC2JnQ29sb3JUeXBlZW51bQAAABFFU2xpY2VCR0NvbG9yVHlwZQAAAABOb25lAAAACXRvcE91dHNldGxvbmcAAAAAAAAACmxlZnRPdXRzZXRsb25nAAAAAAAAAAxib3R0b21PdXRzZXRsb25nAAAAAAAAAAtyaWdodE91dHNldGxvbmcAAAAAADhCSU0EKAAAAAAADAAAAAI/8AAAAAAAADhCSU0EFAAAAAAABAAAAAQ4QklNBAwAAAAAGmoAAAABAAAAoAAAAGQAAAHgAAC7gAAAGk4AGAAB/9j/4gxYSUNDX1BST0ZJTEUAAQEAAAxITGlubwIQAABtbnRyUkdCIFhZWiAHzgACAAkABgAxAABhY3NwTVNGVAAAAABJRUMgc1JHQgAAAAAAAAAAAAAAAAAA9tYAAQAAAADTLUhQICAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAABFjcHJ0AAABUAAAADNkZXNjAAABhAAAAGx3dHB0AAAB8AAAABRia3B0AAACBAAAABRyWFlaAAACGAAAABRnWFlaAAACLAAAABRiWFlaAAACQAAAABRkbW5kAAACVAAAAHBkbWRkAAACxAAAAIh2dWVkAAADTAAAAIZ2aWV3AAAD1AAAACRsdW1pAAAD+AAAABRtZWFzAAAEDAAAACR0ZWNoAAAEMAAAAAxyVFJDAAAEPAAACAxnVFJDAAAEPAAACAxiVFJDAAAEPAAACAx0ZXh0AAAAAENvcHlyaWdodCAoYykgMTk5OCBIZXdsZXR0LVBhY2thcmQgQ29tcGFueQAAZGVzYwAAAAAAAAASc1JHQiBJRUM2MTk2Ni0yLjEAAAAAAAAAAAAAABJzUkdCIElFQzYxOTY2LTIuMQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAWFlaIAAAAAAAAPNRAAEAAAABFsxYWVogAAAAAAAAAAAAAAAAAAAAAFhZWiAAAAAAAABvogAAOPUAAAOQWFlaIAAAAAAAAGKZAAC3hQAAGNpYWVogAAAAAAAAJKAAAA+EAAC2z2Rlc2MAAAAAAAAAFklFQyBodHRwOi8vd3d3LmllYy5jaAAAAAAAAAAAAAAAFklFQyBodHRwOi8vd3d3LmllYy5jaAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAABkZXNjAAAAAAAAAC5JRUMgNjE5NjYtMi4xIERlZmF1bHQgUkdCIGNvbG91ciBzcGFjZSAtIHNSR0IAAAAAAAAAAAAAAC5JRUMgNjE5NjYtMi4xIERlZmF1bHQgUkdCIGNvbG91ciBzcGFjZSAtIHNSR0IAAAAAAAAAAAAAAAAAAAAAAAAAAAAAZGVzYwAAAAAAAAAsUmVmZXJlbmNlIFZpZXdpbmcgQ29uZGl0aW9uIGluIElFQzYxOTY2LTIuMQAAAAAAAAAAAAAALFJlZmVyZW5jZSBWaWV3aW5nIENvbmRpdGlvbiBpbiBJRUM2MTk2Ni0yLjEAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAHZpZXcAAAAAABOk/gAUXy4AEM8UAAPtzAAEEwsAA1yeAAAAAVhZWiAAAAAAAEwJVgBQAAAAVx/nbWVhcwAAAAAAAAABAAAAAAAAAAAAAAAAAAAAAAAAAo8AAAACc2lnIAAAAABDUlQgY3VydgAAAAAAAAQAAAAABQAKAA8AFAAZAB4AIwAoAC0AMgA3ADsAQABFAEoATwBUAFkAXgBjAGgAbQByAHcAfACBAIYAiwCQAJUAmgCfAKQAqQCuALIAtwC8AMEAxgDLANAA1QDbAOAA5QDrAPAA9gD7AQEBBwENARMBGQEfASUBKwEyATgBPgFFAUwBUgFZAWABZwFuAXUBfAGDAYsBkgGaAaEBqQGxAbkBwQHJAdEB2QHhAekB8gH6AgMCDAIUAh0CJgIvAjgCQQJLAlQCXQJnAnECegKEAo4CmAKiAqwCtgLBAssC1QLgAusC9QMAAwsDFgMhAy0DOANDA08DWgNmA3IDfgOKA5YDogOuA7oDxwPTA+AD7AP5BAYEEwQgBC0EOwRIBFUEYwRxBH4EjASaBKgEtgTEBNME4QTwBP4FDQUcBSsFOgVJBVgFZwV3BYYFlgWmBbUFxQXVBeUF9gYGBhYGJwY3BkgGWQZqBnsGjAadBq8GwAbRBuMG9QcHBxkHKwc9B08HYQd0B4YHmQesB78H0gflB/gICwgfCDIIRghaCG4IggiWCKoIvgjSCOcI+wkQCSUJOglPCWQJeQmPCaQJugnPCeUJ+woRCicKPQpUCmoKgQqYCq4KxQrcCvMLCwsiCzkLUQtpC4ALmAuwC8gL4Qv5DBIMKgxDDFwMdQyODKcMwAzZDPMNDQ0mDUANWg10DY4NqQ3DDd4N+A4TDi4OSQ5kDn8Omw62DtIO7g8JDyUPQQ9eD3oPlg+zD88P7BAJECYQQxBhEH4QmxC5ENcQ9RETETERTxFtEYwRqhHJEegSBxImEkUSZBKEEqMSwxLjEwMTIxNDE2MTgxOkE8UT5RQGFCcUSRRqFIsUrRTOFPAVEhU0FVYVeBWbFb0V4BYDFiYWSRZsFo8WshbWFvoXHRdBF2UXiReuF9IX9xgbGEAYZRiKGK8Y1Rj6GSAZRRlrGZEZtxndGgQaKhpRGncanhrFGuwbFBs7G2MbihuyG9ocAhwqHFIcexyjHMwc9R0eHUcdcB2ZHcMd7B4WHkAeah6UHr4e6R8THz4faR+UH78f6iAVIEEgbCCYIMQg8CEcIUghdSGhIc4h+yInIlUigiKvIt0jCiM4I2YjlCPCI/AkHyRNJHwkqyTaJQklOCVoJZclxyX3JicmVyaHJrcm6CcYJ0kneierJ9woDSg/KHEooijUKQYpOClrKZ0p0CoCKjUqaCqbKs8rAis2K2krnSvRLAUsOSxuLKIs1y0MLUEtdi2rLeEuFi5MLoIuty7uLyQvWi+RL8cv/jA1MGwwpDDbMRIxSjGCMbox8jIqMmMymzLUMw0zRjN/M7gz8TQrNGU0njTYNRM1TTWHNcI1/TY3NnI2rjbpNyQ3YDecN9c4FDhQOIw4yDkFOUI5fzm8Ofk6Njp0OrI67zstO2s7qjvoPCc8ZTykPOM9Ij1hPaE94D4gPmA+oD7gPyE/YT+iP+JAI0BkQKZA50EpQWpBrEHuQjBCckK1QvdDOkN9Q8BEA0RHRIpEzkUSRVVFmkXeRiJGZ0arRvBHNUd7R8BIBUhLSJFI10kdSWNJqUnwSjdKfUrESwxLU0uaS+JMKkxyTLpNAk1KTZNN3E4lTm5Ot08AT0lPk0/dUCdQcVC7UQZRUFGbUeZSMVJ8UsdTE1NfU6pT9lRCVI9U21UoVXVVwlYPVlxWqVb3V0RXklfgWC9YfVjLWRpZaVm4WgdaVlqmWvVbRVuVW+VcNVyGXNZdJ114XcleGl5sXr1fD19hX7NgBWBXYKpg/GFPYaJh9WJJYpxi8GNDY5dj62RAZJRk6WU9ZZJl52Y9ZpJm6Gc9Z5Nn6Wg/aJZo7GlDaZpp8WpIap9q92tPa6dr/2xXbK9tCG1gbbluEm5rbsRvHm94b9FwK3CGcOBxOnGVcfByS3KmcwFzXXO4dBR0cHTMdSh1hXXhdj52m3b4d1Z3s3gReG54zHkqeYl553pGeqV7BHtje8J8IXyBfOF9QX2hfgF+Yn7CfyN/hH/lgEeAqIEKgWuBzYIwgpKC9INXg7qEHYSAhOOFR4Wrhg6GcobXhzuHn4gEiGmIzokziZmJ/opkisqLMIuWi/yMY4zKjTGNmI3/jmaOzo82j56QBpBukNaRP5GokhGSepLjk02TtpQglIqU9JVflcmWNJaflwqXdZfgmEyYuJkkmZCZ/JpomtWbQpuvnByciZz3nWSd0p5Anq6fHZ+Ln/qgaaDYoUehtqImopajBqN2o+akVqTHpTilqaYapoum/adup+CoUqjEqTepqaocqo+rAqt1q+msXKzQrUStuK4trqGvFq+LsACwdbDqsWCx1rJLssKzOLOutCW0nLUTtYq2AbZ5tvC3aLfguFm40blKucK6O7q1uy67p7whvJu9Fb2Pvgq+hL7/v3q/9cBwwOzBZ8Hjwl/C28NYw9TEUcTOxUvFyMZGxsPHQce/yD3IvMk6ybnKOMq3yzbLtsw1zLXNNc21zjbOts83z7jQOdC60TzRvtI/0sHTRNPG1EnUy9VO1dHWVdbY11zX4Nhk2OjZbNnx2nba+9uA3AXcit0Q3ZbeHN6i3ynfr+A24L3hROHM4lPi2+Nj4+vkc+T85YTmDeaW5x/nqegy6LzpRunQ6lvq5etw6/vshu0R7ZzuKO6070DvzPBY8OXxcvH/8ozzGfOn9DT0wvVQ9d72bfb794r4Gfio+Tj5x/pX+uf7d/wH/Jj9Kf26/kv+3P9t////7QAMQWRvYmVfQ00AAf/uAA5BZG9iZQBkgAAAAAH/2wCEAAwICAgJCAwJCQwRCwoLERUPDAwPFRgTExUTExgRDAwMDAwMEQwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwBDQsLDQ4NEA4OEBQODg4UFA4ODg4UEQwMDAwMEREMDAwMDAwRDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDP/AABEIAGQAoAMBIgACEQEDEQH/3QAEAAr/xAE/AAABBQEBAQEBAQAAAAAAAAADAAECBAUGBwgJCgsBAAEFAQEBAQEBAAAAAAAAAAEAAgMEBQYHCAkKCxAAAQQBAwIEAgUHBggFAwwzAQACEQMEIRIxBUFRYRMicYEyBhSRobFCIyQVUsFiMzRygtFDByWSU/Dh8WNzNRaisoMmRJNUZEXCo3Q2F9JV4mXys4TD03Xj80YnlKSFtJXE1OT0pbXF1eX1VmZ2hpamtsbW5vY3R1dnd4eXp7fH1+f3EQACAgECBAQDBAUGBwcGBTUBAAIRAyExEgRBUWFxIhMFMoGRFKGxQiPBUtHwMyRi4XKCkkNTFWNzNPElBhaisoMHJjXC0kSTVKMXZEVVNnRl4vKzhMPTdePzRpSkhbSVxNTk9KW1xdXl9VZmdoaWprbG1ub2JzdHV2d3h5ent8f/2gAMAwEAAhEDEQA/AOsw/qr1fFZjV09QrxhRbvvfj1PY/IgUtOXnE3OZlZ2Qyi2u/wBav0f1z1/0mTjY9qv/AFf6N1Lprnvz805z30VU7nF5cDU/Js3ONtlm9z68mpj3/ov5lL635Wdi9AyX4AsGQ4Bjba+amnW3Id+76VTX/wBtamHdbfiU33VOotsra6yl30mOIl9bv6jvakpMkm5TpKUkkkkpSSSSSlJJJJKUkkkkpSSSSSlJJJJKUkkkkpSSSSSn/9D0H6z/APie6j/4Xf8AkWosv6z/APie6j/4Xf8AkWokpyvq6G1YVuCBtODk30bP3Wb3X4jf/YK/GWquOZ0zqON9e7bjlmuvNqdfS4hxa8sb6H2OyuWVO9Nvp2+1/q+jjep+j/SWM6b9oMrxhdewte21lFlbfdFj3sx27fo/o3PtZYx/+g/SJ04xjXDITuIka/Rkd4er91TbSVB/XekMssqdlM31WNpsiSGveS1jXuA2t97H17/5tlv6H+dQx9YukFoc24uB2/mP3HeN1WytzRZc5/0NlLLLN7601TppLP8A2/0j0rrhktNWOGGywBxbFrjVSa3hu27fcx9P6H1P0zLKf5xih/zj6Puew3kWsLg6osf6ntsOO7bSGeo/9IPzW/zPp3/zNm9JTppKLHssY17HB7HgOa5pkEHUOaQpJKUkkkkpSSSSSlJJJJKUs/peTflX9Qsc8ux2ZJpxhAADamVV36gbnfrv2pnv/cR+o5tfT8DIzbAXNx63P2DlxA9tTP8AhLXfo6/5aH0fDswemY+PcQ/IDd+Q8cOusJuyrP8ArmRZa9JTdSSSSU//0fQfrP8A+J7qP/hd/wCRaiy/rP8A+J7qP/hd/wCRaiSnG6vj3ZN932Owu6hhV0ZeJS7RgsY7Jbt9Sdrf2hT6+Bkf6OhRwM7pH1gOTXQfXxra6LbW/RLbCbPa7a7fXkVfZqvo/wA1ZUtJr93ULqTcHNFNbvs+36O517fW9T871tmzZ/wH8tc5j9A6f9Ws8Pabjg9QLaBkF8Oxbi79C3fWK/1bNcWVb7P+1fper63239E6IhwyskS04AB6T+9xKdt31f6O76eM13uDyXFxJcPzjud+f/OW/wCmu/TW/pUc9M6eXsecdhdUQ6t0atLTLS3/ADU+Hc93qY9zw/IxyGvIgFzXDfVaWD6PqM/8FZb6aspqnNp+r3TKm5NYY51WW1jH1FxDGsrL3100sr2elX6l1r/+uJP+r3SHAhtHpAkE+k5zJ2u9WP0bm7fd+5/gv0H8z+jWkkkpixjGMaxjQ1jQA1oEAAcNaFJJJJSkkkklKSSSSUpJJUuq9S/Z+O1zKzkZV7xTiYzTBttcC5rN0O9OpjWvuyLtv6DHrtu/MSU1c0/tHq9HTm64+AWZmaexsBnp2Mf+vM+3v/0f2XF/weUtdUuldPOBi7LLPXybnG7LyCINlz49Szbrtra1rKcer/AY1VNH+CV1JSkkkklP/9L0H6z/APie6j/4Xf8AkWosv6z/APie6j/4Xf8AkWokprMLv2jaN9Zb6NRDB/Og7r5fZ/wL/wDBfy2XKebjNy8O/EcQG5Fb6iSA4Q9pZqx3tf8AS+ioMDv2hadtYb6NUPH86Tuv9tn/AALf8D/L9dWUlPHdC6d1L6t2Pxs/K215jmjHygPUo9Wdjac31PTurvv3MbRd6ldeR/MWW/a/R+1dLjdSx7XGmyyuvKrf6dlO8E7o3/o/o+o17Hb2/wDT/S+oxFz8Rmbg5GE87WZNT6XEAGBY01n2u9v5y5KvpOV9UOkZT3lnUunWAG+hs1WVultVD8V1jrmWN3Fnqeq+u2j+cquu9H0VKf1plIy/WylERgI/Pxf3fRBT2iS5fo/1spv6VX1HJyGkV2Nxc1jg1rm2GGevUz9G9rP8NbTaz1PS9b0/5j9L0NWZj22OqDttrBudU8Fjw3j1Nj9u6v8A4Rv6NMlCUJGMhRiTEj+tHdSdJVupY9+V07Kxsew03X0vrqua4scxz2ljLWWN9zH1uO9r1kt6f9aBZZb9sra617rS0OLmAiquqmitl1L214z7qPUyPT/S/pb31P8AUsTVO+ksKvE+tFIa8ZbLtmycdxbtLWus9RjLjj+vvfSzFY2y21/vsyv+BRsDF+sNeRXZnZbLqyIupaG7Qf00motoqt/7hfTs/wC5f/AJKddJJZFvXDkvdj9DrGfeCWvyJ24lThz62W0O9axv/cbD9e7f/SPsv88kpu9Q6jj9PoFlu573nZRj1jdbbYRLaceuW77NP+Lqr/T32VUV2Wqt0zBy/Wd1Pqhac+1pYyph3V41RIf9lodDd7nuYx+Xk/8Aaq1jP8BRjU0z6f0luNe/NyrTmdRtbssyXjaGsnd9nxKZc3Fxdzf5tv6S32far8m39KtBJSkkkklKSSSSU//T9B+s/wD4nuo/+F3/AJFqLM+srXP+r/UGsaXuOPZDWguJ9vZrZc5N/wA5uhf9yh/mv/8AIJKbbKyOoW2+iAHU1NGRu1cWuvd6Pp/m+jv37/8Ah1ZWEz6w9CHULrvWgupqZ60P2u2uvd6QZ6f0qt+93/HKz/zn6F/3KH+Y/wD8gkp1FF7GvaWPAcxwIc0iQQeQQs3/AJz9C/7lD/Mf/wCQS/5z9C/7lD/Mf/5BJTW699UOm9Zrx2/0OzFltVtDQ0it30qRG32fuLLt+rP1jo6tjOxMpmV0zFYxlVOU8t/RsAa7Esbj1+jZ6jm/zz8az/B2fzuPUt3/AJz9C/7lD/Mf/wCQS/5z9C/7lD/Mf/5BSRzTEeG7jUoiMhxCPufMY8X6Snkq+ufWXAwM63N6ffjMqcPs7q4ZVTba8td/Pu/WMar/AAe1lmP/ANvVWUbXSvrFmZOLX1Cxlgq2fpsZ7QHuLnRVdgiutr7anta9tDPf9qe/0q7PUp/TaR+s3QiCDlAg8jY//wAgq1/XPq7Zm417ri59IfseGO2t3AfzhNe9v/BbUzmZHJCsQjgyccZGcQckfaj8+H2py/T/AM4kEdRbaHWrXn9D0vOt+LK6v/bu/HS+0fWLI0qxMfBaT/OZFhueB54mK1lTv/cil/zn6F/3KH+Y/wD8gl/zn6F/3KH+Y/8A8gghj+wRla9Xyreog847oqxeZ2/Y6f6RX/Iz7M1ajGMrY2utoYxgDWMaIAA0a1rQs3/nP0L/ALlD/Mf/AOQS/wCc/Qv+5Q/zH/8AkElOoksv/nP0L/uUP8x//kEv+c/Qv+5Q/wAx/wD5BJTqJLL/AOc/Qv8AuUP8x/8A5BL/AJz9C/7lD/Mf/wCQSU6iSwcr67/V7FvrquvcGWtLvXDHFjSCG7H6erufPt/RbFt1W13VMtrO6uxocx3iCJaUlP8A/9T1Vc0/B+tVF2RdjXtsdfZaa2vscWsb61luOLK8g3Vf0f0K/wBUrw/Rp+0ep9ru+zWLa6ph3ZuG7HpvONY4gtuaJLYO72+5n/VLKs+reaLn3U57txDmsL/ULgzUV1er6+5z/Sds+2O/Wf5v/hKklKyMf61+o3ZcHNDvUG0saBLXetVbNW6xjHO/yf8A8L/yh69SPiU/WP16rcm9voh/voIZPp7Wtb6j66m/rPue+70n+j638z+g/RqNfQc+vEqpZ1F4vYXOuuDXAWlzG62MZc12/wC01U3Ps9Xf6P2jG/7U+qg1/V3q9db219Ucx9zYtt2uLwRv9Ise633vayxlL7bWfzVFf83Ykp6FJV+n492Nh1UX2m+1jYdY4kkn+s73u2/Q3v8AerCSlJJJJKaPV8zOw8Vt2FiHOsL21mlrtpHqTXVZ9F/6JmQ6n7S//AYnr5P+B9J8+l5d2bg1Zd+O/EfducMe3SxrNzvR9ZhDfTufTsfbV/gn/o1bSSUpJJJJSkkkklKSSSSUpJJJJSGzExrbW3vrab2Ncxl0D1Gtd9Ntdn02bo/NWF9X8X60V5rXdYsLsdlLph4duumvEZ7W/mfZ8L9of+GOp3V/4FdGkkp//9X1VJfKqSSn6qSXyqkkp+qkl8qpJKfqpJfKqSSn6qSXyqkkp+qkl8qpJKfqpJfKqSSn6qSXyqkkp+qkl8qpJKfqpJfKqSSn/9k4QklNBCEAAAAAAFUAAAABAQAAAA8AQQBkAG8AYgBlACAAUABoAG8AdABvAHMAaABvAHAAAAATAEEAZABvAGIAZQAgAFAAaABvAHQAbwBzAGgAbwBwACAAQwBTADYAAAABADhCSU0EBgAAAAAABwACAQEAAQEA/+EO8Wh0dHA6Ly9ucy5hZG9iZS5jb20veGFwLzEuMC8APD94cGFja2V0IGJlZ2luPSLvu78iIGlkPSJXNU0wTXBDZWhpSHpyZVN6TlRjemtjOWQiPz4gPHg6eG1wbWV0YSB4bWxuczp4PSJhZG9iZTpuczptZXRhLyIgeDp4bXB0az0iQWRvYmUgWE1QIENvcmUgNS4zLWMwMTEgNjYuMTQ1NjYxLCAyMDEyLzAyLzA2LTE0OjU2OjI3ICAgICAgICAiPiA8cmRmOlJERiB4bWxuczpyZGY9Imh0dHA6Ly93d3cudzMub3JnLzE5OTkvMDIvMjItcmRmLXN5bnRheC1ucyMiPiA8cmRmOkRlc2NyaXB0aW9uIHJkZjphYm91dD0iIiB4bWxuczp4bXA9Imh0dHA6Ly9ucy5hZG9iZS5jb20veGFwLzEuMC8iIHhtbG5zOmRjPSJodHRwOi8vcHVybC5vcmcvZGMvZWxlbWVudHMvMS4xLyIgeG1sbnM6cGhvdG9zaG9wPSJodHRwOi8vbnMuYWRvYmUuY29tL3Bob3Rvc2hvcC8xLjAvIiB4bWxuczp4bXBNTT0iaHR0cDovL25zLmFkb2JlLmNvbS94YXAvMS4wL21tLyIgeG1sbnM6c3RFdnQ9Imh0dHA6Ly9ucy5hZG9iZS5jb20veGFwLzEuMC9zVHlwZS9SZXNvdXJjZUV2ZW50IyIgeG1wOkNyZWF0b3JUb29sPSJBZG9iZSBQaG90b3Nob3AgQ1M2IChXaW5kb3dzKSIgeG1wOkNyZWF0ZURhdGU9IjIwMTgtMDYtMjFUMTY6MzQ6NDUrMDU6NDUiIHhtcDpNb2RpZnlEYXRlPSIyMDE4LTA2LTIyVDE2OjQ2OjMxKzA1OjQ1IiB4bXA6TWV0YWRhdGFEYXRlPSIyMDE4LTA2LTIyVDE2OjQ2OjMxKzA1OjQ1IiBkYzpmb3JtYXQ9ImltYWdlL2pwZWciIHBob3Rvc2hvcDpDb2xvck1vZGU9IjMiIHBob3Rvc2hvcDpJQ0NQcm9maWxlPSJzUkdCIElFQzYxOTY2LTIuMSIgeG1wTU06SW5zdGFuY2VJRD0ieG1wLmlpZDpDNThFMjkyQjA4NzZFODExOEExQjk0QzA3REY3Q0VFNSIgeG1wTU06RG9jdW1lbnRJRD0ieG1wLmRpZDpDMjhFMjkyQjA4NzZFODExOEExQjk0QzA3REY3Q0VFNSIgeG1wTU06T3JpZ2luYWxEb2N1bWVudElEPSJ4bXAuZGlkOkMyOEUyOTJCMDg3NkU4MTE4QTFCOTRDMDdERjdDRUU1Ij4gPHhtcE1NOkhpc3Rvcnk+IDxyZGY6U2VxPiA8cmRmOmxpIHN0RXZ0OmFjdGlvbj0iY3JlYXRlZCIgc3RFdnQ6aW5zdGFuY2VJRD0ieG1wLmlpZDpDMjhFMjkyQjA4NzZFODExOEExQjk0QzA3REY3Q0VFNSIgc3RFdnQ6d2hlbj0iMjAxOC0wNi0yMVQxNjozNDo0NSswNTo0NSIgc3RFdnQ6c29mdHdhcmVBZ2VudD0iQWRvYmUgUGhvdG9zaG9wIENTNiAoV2luZG93cykiLz4gPHJkZjpsaSBzdEV2dDphY3Rpb249ImNvbnZlcnRlZCIgc3RFdnQ6cGFyYW1ldGVycz0iZnJvbSBpbWFnZS9wbmcgdG8gaW1hZ2UvanBlZyIvPiA8cmRmOmxpIHN0RXZ0OmFjdGlvbj0ic2F2ZWQiIHN0RXZ0Omluc3RhbmNlSUQ9InhtcC5paWQ6QzM4RTI5MkIwODc2RTgxMThBMUI5NEMwN0RGN0NFRTUiIHN0RXZ0OndoZW49IjIwMTgtMDYtMjJUMTY6NDE6NTYrMDU6NDUiIHN0RXZ0OnNvZnR3YXJlQWdlbnQ9IkFkb2JlIFBob3Rvc2hvcCBDUzYgKFdpbmRvd3MpIiBzdEV2dDpjaGFuZ2VkPSIvIi8+IDxyZGY6bGkgc3RFdnQ6YWN0aW9uPSJzYXZlZCIgc3RFdnQ6aW5zdGFuY2VJRD0ieG1wLmlpZDpDNThFMjkyQjA4NzZFODExOEExQjk0QzA3REY3Q0VFNSIgc3RFdnQ6d2hlbj0iMjAxOC0wNi0yMlQxNjo0NjozMSswNTo0NSIgc3RFdnQ6c29mdHdhcmVBZ2VudD0iQWRvYmUgUGhvdG9zaG9wIENTNiAoV2luZG93cykiIHN0RXZ0OmNoYW5nZWQ9Ii8iLz4gPC9yZGY6U2VxPiA8L3htcE1NOkhpc3Rvcnk+IDwvcmRmOkRlc2NyaXB0aW9uPiA8L3JkZjpSREY+IDwveDp4bXBtZXRhPiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIDw/eHBhY2tldCBlbmQ9InciPz7/4gxYSUNDX1BST0ZJTEUAAQEAAAxITGlubwIQAABtbnRyUkdCIFhZWiAHzgACAAkABgAxAABhY3NwTVNGVAAAAABJRUMgc1JHQgAAAAAAAAAAAAAAAQAA9tYAAQAAAADTLUhQICAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAABFjcHJ0AAABUAAAADNkZXNjAAABhAAAAGx3dHB0AAAB8AAAABRia3B0AAACBAAAABRyWFlaAAACGAAAABRnWFlaAAACLAAAABRiWFlaAAACQAAAABRkbW5kAAACVAAAAHBkbWRkAAACxAAAAIh2dWVkAAADTAAAAIZ2aWV3AAAD1AAAACRsdW1pAAAD+AAAABRtZWFzAAAEDAAAACR0ZWNoAAAEMAAAAAxyVFJDAAAEPAAACAxnVFJDAAAEPAAACAxiVFJDAAAEPAAACAx0ZXh0AAAAAENvcHlyaWdodCAoYykgMTk5OCBIZXdsZXR0LVBhY2thcmQgQ29tcGFueQAAZGVzYwAAAAAAAAASc1JHQiBJRUM2MTk2Ni0yLjEAAAAAAAAAAAAAABJzUkdCIElFQzYxOTY2LTIuMQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAWFlaIAAAAAAAAPNRAAEAAAABFsxYWVogAAAAAAAAAAAAAAAAAAAAAFhZWiAAAAAAAABvogAAOPUAAAOQWFlaIAAAAAAAAGKZAAC3hQAAGNpYWVogAAAAAAAAJKAAAA+EAAC2z2Rlc2MAAAAAAAAAFklFQyBodHRwOi8vd3d3LmllYy5jaAAAAAAAAAAAAAAAFklFQyBodHRwOi8vd3d3LmllYy5jaAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAABkZXNjAAAAAAAAAC5JRUMgNjE5NjYtMi4xIERlZmF1bHQgUkdCIGNvbG91ciBzcGFjZSAtIHNSR0IAAAAAAAAAAAAAAC5JRUMgNjE5NjYtMi4xIERlZmF1bHQgUkdCIGNvbG91ciBzcGFjZSAtIHNSR0IAAAAAAAAAAAAAAAAAAAAAAAAAAAAAZGVzYwAAAAAAAAAsUmVmZXJlbmNlIFZpZXdpbmcgQ29uZGl0aW9uIGluIElFQzYxOTY2LTIuMQAAAAAAAAAAAAAALFJlZmVyZW5jZSBWaWV3aW5nIENvbmRpdGlvbiBpbiBJRUM2MTk2Ni0yLjEAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAHZpZXcAAAAAABOk/gAUXy4AEM8UAAPtzAAEEwsAA1yeAAAAAVhZWiAAAAAAAEwJVgBQAAAAVx/nbWVhcwAAAAAAAAABAAAAAAAAAAAAAAAAAAAAAAAAAo8AAAACc2lnIAAAAABDUlQgY3VydgAAAAAAAAQAAAAABQAKAA8AFAAZAB4AIwAoAC0AMgA3ADsAQABFAEoATwBUAFkAXgBjAGgAbQByAHcAfACBAIYAiwCQAJUAmgCfAKQAqQCuALIAtwC8AMEAxgDLANAA1QDbAOAA5QDrAPAA9gD7AQEBBwENARMBGQEfASUBKwEyATgBPgFFAUwBUgFZAWABZwFuAXUBfAGDAYsBkgGaAaEBqQGxAbkBwQHJAdEB2QHhAekB8gH6AgMCDAIUAh0CJgIvAjgCQQJLAlQCXQJnAnECegKEAo4CmAKiAqwCtgLBAssC1QLgAusC9QMAAwsDFgMhAy0DOANDA08DWgNmA3IDfgOKA5YDogOuA7oDxwPTA+AD7AP5BAYEEwQgBC0EOwRIBFUEYwRxBH4EjASaBKgEtgTEBNME4QTwBP4FDQUcBSsFOgVJBVgFZwV3BYYFlgWmBbUFxQXVBeUF9gYGBhYGJwY3BkgGWQZqBnsGjAadBq8GwAbRBuMG9QcHBxkHKwc9B08HYQd0B4YHmQesB78H0gflB/gICwgfCDIIRghaCG4IggiWCKoIvgjSCOcI+wkQCSUJOglPCWQJeQmPCaQJugnPCeUJ+woRCicKPQpUCmoKgQqYCq4KxQrcCvMLCwsiCzkLUQtpC4ALmAuwC8gL4Qv5DBIMKgxDDFwMdQyODKcMwAzZDPMNDQ0mDUANWg10DY4NqQ3DDd4N+A4TDi4OSQ5kDn8Omw62DtIO7g8JDyUPQQ9eD3oPlg+zD88P7BAJECYQQxBhEH4QmxC5ENcQ9RETETERTxFtEYwRqhHJEegSBxImEkUSZBKEEqMSwxLjEwMTIxNDE2MTgxOkE8UT5RQGFCcUSRRqFIsUrRTOFPAVEhU0FVYVeBWbFb0V4BYDFiYWSRZsFo8WshbWFvoXHRdBF2UXiReuF9IX9xgbGEAYZRiKGK8Y1Rj6GSAZRRlrGZEZtxndGgQaKhpRGncanhrFGuwbFBs7G2MbihuyG9ocAhwqHFIcexyjHMwc9R0eHUcdcB2ZHcMd7B4WHkAeah6UHr4e6R8THz4faR+UH78f6iAVIEEgbCCYIMQg8CEcIUghdSGhIc4h+yInIlUigiKvIt0jCiM4I2YjlCPCI/AkHyRNJHwkqyTaJQklOCVoJZclxyX3JicmVyaHJrcm6CcYJ0kneierJ9woDSg/KHEooijUKQYpOClrKZ0p0CoCKjUqaCqbKs8rAis2K2krnSvRLAUsOSxuLKIs1y0MLUEtdi2rLeEuFi5MLoIuty7uLyQvWi+RL8cv/jA1MGwwpDDbMRIxSjGCMbox8jIqMmMymzLUMw0zRjN/M7gz8TQrNGU0njTYNRM1TTWHNcI1/TY3NnI2rjbpNyQ3YDecN9c4FDhQOIw4yDkFOUI5fzm8Ofk6Njp0OrI67zstO2s7qjvoPCc8ZTykPOM9Ij1hPaE94D4gPmA+oD7gPyE/YT+iP+JAI0BkQKZA50EpQWpBrEHuQjBCckK1QvdDOkN9Q8BEA0RHRIpEzkUSRVVFmkXeRiJGZ0arRvBHNUd7R8BIBUhLSJFI10kdSWNJqUnwSjdKfUrESwxLU0uaS+JMKkxyTLpNAk1KTZNN3E4lTm5Ot08AT0lPk0/dUCdQcVC7UQZRUFGbUeZSMVJ8UsdTE1NfU6pT9lRCVI9U21UoVXVVwlYPVlxWqVb3V0RXklfgWC9YfVjLWRpZaVm4WgdaVlqmWvVbRVuVW+VcNVyGXNZdJ114XcleGl5sXr1fD19hX7NgBWBXYKpg/GFPYaJh9WJJYpxi8GNDY5dj62RAZJRk6WU9ZZJl52Y9ZpJm6Gc9Z5Nn6Wg/aJZo7GlDaZpp8WpIap9q92tPa6dr/2xXbK9tCG1gbbluEm5rbsRvHm94b9FwK3CGcOBxOnGVcfByS3KmcwFzXXO4dBR0cHTMdSh1hXXhdj52m3b4d1Z3s3gReG54zHkqeYl553pGeqV7BHtje8J8IXyBfOF9QX2hfgF+Yn7CfyN/hH/lgEeAqIEKgWuBzYIwgpKC9INXg7qEHYSAhOOFR4Wrhg6GcobXhzuHn4gEiGmIzokziZmJ/opkisqLMIuWi/yMY4zKjTGNmI3/jmaOzo82j56QBpBukNaRP5GokhGSepLjk02TtpQglIqU9JVflcmWNJaflwqXdZfgmEyYuJkkmZCZ/JpomtWbQpuvnByciZz3nWSd0p5Anq6fHZ+Ln/qgaaDYoUehtqImopajBqN2o+akVqTHpTilqaYapoum/adup+CoUqjEqTepqaocqo+rAqt1q+msXKzQrUStuK4trqGvFq+LsACwdbDqsWCx1rJLssKzOLOutCW0nLUTtYq2AbZ5tvC3aLfguFm40blKucK6O7q1uy67p7whvJu9Fb2Pvgq+hL7/v3q/9cBwwOzBZ8Hjwl/C28NYw9TEUcTOxUvFyMZGxsPHQce/yD3IvMk6ybnKOMq3yzbLtsw1zLXNNc21zjbOts83z7jQOdC60TzRvtI/0sHTRNPG1EnUy9VO1dHWVdbY11zX4Nhk2OjZbNnx2nba+9uA3AXcit0Q3ZbeHN6i3ynfr+A24L3hROHM4lPi2+Nj4+vkc+T85YTmDeaW5x/nqegy6LzpRunQ6lvq5etw6/vshu0R7ZzuKO6070DvzPBY8OXxcvH/8ozzGfOn9DT0wvVQ9d72bfb794r4Gfio+Tj5x/pX+uf7d/wH/Jj9Kf26/kv+3P9t////7gAhQWRvYmUAZIAAAAABAwAQAwIDBgAAAAAAAAAAAAAAAP/bAIQACAYGBgYGCAYGCAwIBwgMDgoICAoOEA0NDg0NEBEMDAwMDAwRDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAEJCAgJCgkLCQkLDgsNCw4RDg4ODhERDAwMDAwREQwMDAwMDBEMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwM/8IAEQgBdgJYAwEiAAIRAQMRAf/EAOcAAQADAAMBAQAAAAAAAAAAAAAFBgcDBAgCAQEBAAIDAQAAAAAAAAAAAAAAAAEEAwUGAhAAAQMEAQMDAwQCAgMAAAAABgMEBQABAgcWIFAXMEA3EGAVERITFHCAITEjNSYRAAICAAQCBAgJBQ0GAwkBAAIDAQQAERIFEwYhIjKTMUJy0iMUNDUgUEFRUmKCM7MwQGFDFRBgcYGSorJTY3ODJAeRobGjdBbCRCVwgPDB4vLjVGQmEgACAQMCAwQIBwEAAAAAAAABAhEAEgMhMVBBE2EiQmJggFFxgZEyUnCQscFyguIE/9oADAMBAQIRAxEAAADs3Si2U+7JlPZNEqUJKG2gAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA+Pr9H5+h8/v6AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAGcWnBT0w4+QAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAznNPSOXFptACtlkdHvAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAADLtRy41ECvWH4KlcMw08AAAAAAAAAAAAAAAAAAAAAAAAAAAjY+HsRMAAAAAAZdqOXGogAyzU8e7dhqzr9iuAAAAAAAAAAAAAAAAAAAAAAAAAAy3Uss1MAAAAAAZdqOXGogA87xe25D0c6Ff+at89Fqfn75EZJgAAjiRAAdbnPoAAAAAAAAAAAAAAAACEm8vJa9cPMAAAAAAMu1HLjUQAMR26rk32s20wrtirtiM+k4bnKtOyXQIOW4LgQcHd6ycvHZagWSD70kUXvc0idDpXChmqy0fIAAAAAAAAAAAAAAAHzlk5NE+AAAAAABl2o5caiABVrTCFH0zrUMtVirEmSgAAAFesI+PsAPzrdoAAAAAAAAAAAAAAACjFe1quWMAAAAAAAZdqOXGogAVa0wpNYDv2H31a3Lz1tWwm/KvO6CO2AAAAAAAAAAAAAAAAAAAAADhy3m0M74AAAAAAAGXajlxqIAFWtMKTXBzikcF+zU0qq92aPtXbEAAAAAAAAAAAAAAAAAAAAKtNZqT17AAAAAAAABl2o5caiABVbVDEyADI5OTwvbzrWoef9eqRYnT7lMAAAAAAAAAAAAAAAAAA4vvIz41zq98AAAAAAAAAZdqOXGogAVS1w5MAA6OTbOsMI239yfE0X87kN4WpXZM74AAAAAAAAAAAAAAAHxxZOfekfsoAAAAAAAAAAMu1HLjUQAKpa4glwAAOPkHn2HvOf8AUzsFpy7Z+biHms+n8SxKzZgAABXLHhprXzncmaXG57wGtswhDamf/hoLErGaUAAABDVnokZqva/QAAAAAAAAAABl2o5caiABU7ZEEuAAADjh5wRGYbNSsjItCzHs9LPpOk9rO+djb/ugyOFbUFMQ5Xx9iqWsZrLXQVqt6SK3E3oUH50AUOYsgAAIzPC+51M6GV2yAAAAAAAAAAAAAy7UcuNRAAqVtiSWAAAAABWJrupYnRPSed7ua/oDJsCw3zDdEzrXaMV6FaNl73LkVBs3JUIrHOjoaZgEwAdSrF0/Mt/C01O12sznQeYAAAAAAAAAAAAAAMu1HLjUQAKlbYolQAAAAAAAIKdSzqzzz28xcHpHj2843VfRmV+0Bqv1wcLn79bl7brPWeNQdfVy781IZxaJ8fn6AAAAAAAAAAAAAAAADLtRy41EACoW+KJUAAAAAAAAAAADp9z5KrbODnAAAAAAAAAAAAAAAB8n0AAABjuxRhAJD6I1KCLrF7EWkvoi0oItKCLSgi0oItKCLSgi0oItKCLSgi0oItKCL684KZPygi0oItKCLSgi0oItKCLSgi0oItKCLSgi0oItKCLSgi0oItKCLSgi8n2zIyWvVVvZMgAAAxrp2vhJ3p1iIL12+r0CPsdJnCZjpKLND7cBPgAAAAAAABnFXNvZzowAAAAAAAAAAAAAAAAAy7UcTIy316wmoAAIDpFsUyBNRU74Lops6SqJrJfFLlSfVv6LEAAAAAAACrU3WxRL2AAAAAAAAAAAAAAAAADg5xx/nKAAKXGaH+GQc+rcRkHT3DplX71o5TLKlvkeUWvbRxGK2LTOI7oAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAADwAPf7wAPf7wAPf7wAPf7wAPf7wAPf7wAPf7wAPf7wAPf7wAPf7wAPf7wAPf7wAPf7wAPf7wAPf7wAPf7wAPf7wAPf7wAPf7wAPf7wAPf7wAPf7wAPf7wAPf7wAPf7wAPf7wAPf7wAPf7wAPf7wAPf7wAPf7wAPf7wAPf7wAPf7wAPf7wAPf7wAPf7wAP/2gAIAQIAAQUA/wAhJsnKifZWcw2Sa55fuz+8YvFDJ3OpN8LfZkVGpusZBrZq47I3drt7qq5q59miUWmTNeydluyo2wurJtGeDJBquvWWN8b9jve97RUmg3RT/ikJOYYN0E8Yd3kgmiqrfK/7axyxyt2C1743VWVVvjOqYoQz5sgkff2HuAa3epJf6O//2gAIAQMAAQUA/wBXL/5vta96va9ux/r9G98f1c3x/T/R3//aAAgBAQABBQAfYSO6JUDGjAPkF94Rv75nZA9Ej8JsnGSltYbCmiIx+xRGeT0xJjpGW7JxHDhsJADoYRH9UgErKImuspdvC7D+xVEklsbWta18cb5Vgmmlb9uP7v8ACYuXXkT/AOyTOY/BDUc7zhnKC6TlD7I2fHkM1mhDzL6R1+nLthv7JD/lD6nJBIDMDDPvykT9ih/yh9Tpnd+Jawe2eB3d5+S/DwgTNyJDAeoH/KH1WSwcI6hVzaZd320//piQow/GDfqB/wAodDe/Htud32Jf80YWta1vUD/lDo2zLM2E8w2pK4KMXrWRad1E/wD6PZHqh/yh0FiCziShF/548FfkEcNxkowmWfcy6X/BjuqIm8eL+qH/ACh0FYKgQOBaBauihm0bMGswlcakMcscsaaT8Y+meqHm2E6j0SEgxiWWGeKmHY9pu1ZR+0apMmvqh/yh0kYrLD0fESSExGKpJrpCWebVChj5V2OVrCI60OUoSWLJN0lNYnhPnrSczOhvCLcEo4cPDIqwDVFS0TJ4jMtO1tZS81NDewSJ/HEhleakNTEpQ9gnAUaXXIYJTY82EDE1iRj3v1FMEkwXDIpNPWD/AJQ6WEg1O4fVkkuxWp7a0eX08H9gRhnIDZmWwbZtsiVkERktgCXHXhRkJFI+9mZOQH3zo4U1/OZjROPvpiZiry0Y500ksmEkTMySl8Nayy4m5HjFVzCMzJV6DNdhSGuYOIbQEP7/AGnO3ix4MguOjvrB/wAodIRCSMI22G0cjREyeN5BmZ/+GK9KVBRCbfJJJIJfW9rZWjo5jEs/f3va1ou1z/YPrh/yh0jZGmRozkQ3nYnWMw5YODp3hiKD0krLxHb9nEmURDhA5iMwHrh/yh0g8LJQratoP7rkjPNJVuDTNpkd7c7dN2LURbODkt9gH/KHSOESRGjRxFKRJC0yziH4mCM7QmaRIOXjZNjLsu2bCmXZBKwsQ0gov2Af8odIPCyUK2p2zaP0CLW8G9iNblikk2qWS4vIWva9u1HRakKxOtxRWIZ+xD/lDpHCJEiS+uxBp0zcihM0KYpZFJwiIq5oMe0zEuxgo4SinxzP+yD/AJQ6QaGkoZt9b2te07FPtcEDjbSGeYQs1lzLtDlygzbqqP8AaxE2bN2Tf2Qf8odI6RIEaXRtJRXEbvZaBdDztwlOXJnrSo2VjZdv2VRTBLCbmJLZcxDQ7CBjvZh/yh0gsPJQ7bomIlnORshrUnTyjouS1bNx0iylWcoMM3ziDm3Lhx2NVVNBOcnpXY0kPD8eNR3tA/5Q6R4hbkSXVIR7OUZrtZ/VUlCTsaQsTe90Y1iYxC0ZHTcRL27A+fM41q+k57aMjAwEcOR/tQ/5Q6QSIkYht1roIOkTJnmJzcY9bOa1dhFLRcwNx8tYelXT3D0sziATLiYmihKIuUR/61KzTWHU9ciJ4kZaNI8j2i+jIxhDsvbB/wAodI+QtiFL0Njxa0bPyUWje2uylqMrpqJq4F7uVgylNyaPbZEUlC52va9uszbqYGuz3rclwTlpFinDNiwwfMTQgkhdoiUlZKzdm8sAJEEm9MX0zKpFzh6cMNfr4k4oTegUbLaR6o9rt7Ju8MME8Pbh/wAodIHEyMS29BdBB0i2DRZm4nxmIJWz7At1VZOdWnFQEtkLSSiaayY9NIw8zksjhl1ICavLYXWklGiCga/zzEhx0PKxuuZVkPDo66h5uNCJBlrqRDCBG7AJJLyb8FkXesyMddTMn1T5TCjaKkmabJUFgmGFkvch/wAodMAQtCJL09mxrl5Buodo8sOyUhBS1tkC6kVLlD6XlxNqEErR9BPh5uLEqRS0Xct2uCSqSyfsZkgh4BB2fkpStA6saJqppppYe6D/AJQ6QKKkYpr6jvXwk8cMoiMjmWzxKMj3v881G210Rx6JIYHl4jOKMiCGUFY2HImJHEOxlGIVz/C2M49ezAkhpTJd6za3sqldP0b3tjab2ILQlZlp8YXidTtP5mrRqxQ94H/KHTAkDMhS9YggWJHHZ6vIMFWuqxrGOJRl7AT35KWZ1rc1gohrOFj0neJO3bJFOWjE4aUn4zMkFmYLMtDaAj4hhCEkTLYdL2UjI7GR2mIsbZbEL52/BDclyhdcisNe1rY298H/ACh0gMXIRbX2JIJxhMm21QjZV4Fi71q+F8Gj3CHlMVyhA9hoj8lMJ0KGDmDmoBt/YSlopCUbp7jbpJX246Vqx4fO7f29xyVuEbElKaadhcc44JFYu9rWxt2EP+UOmCIGRAn7SXHYSdtEi0BBqE48gSxawOYN1goKWhM0lsxOlZvKVxjIxrFsO0B/yh0gEbIRjX3Mv/6rWV8cgTtIf8odMEQMSBP3Kn7P2M/6f9bsGCianoxk7GjexfKQVXlIKrykFV5SCq8pBVBJsNwbbykFV5SCq8pBVeUgqvKQVXlIKrykFV5SCq8pBVeUgqvKQVXlIKrykFV5SCq8pBVeUgqvKQVXlIKrykFV5SCq8pBVeUgqvKQVXlIKrykFVIbMDXLAJOxaCFfKQVXlIKrykFV5SCq8pBVeUgqvKQVXlIKrykFV5SCq8pBVeUgqvKQVXlIKrykFV5SCq8pBVeUgqvKQVXlIKrykFV5SCq8pBVeUgqvKQVXlIKrykFV5SCq8pBVeUgqvKQVXlIKqRmVeSxW1J9ljDbEFplTqkR6Dl1sxAMTvgHBytuFCVcKEq4UJVwkSrEND864UJVwoSrhQlXChKuFCVcKEq4UJVwoSrhQlXChKuFCVcKEq4UJVwoSrhQlXChKuFCVcKEq4UJVwoSrhQlXChKuFCVcKEq4SJVwkSrhQlXChKuFCVcKEq4UJVwoSrhQlXChKuFCVcKEq4UJVwoSrhQlXChKuFCVcKEq4UJVwoSrhQlXChKuFCVcKEq4UJVwoSrhQlXChKuFCVcKEq4UJVwoSrhQlXChKn4rMPSaL1GsrUQKD0Hj1miIivtly3CGpJKFRBnNJbH/UQtsKXaupAzmV54MLWUEPIm89HP3Z3NLEGG0Zh/Dxci2l43vAk7eRG1xAtlXG0Yt28nt09LqL/ublnY5xr4hnHiTosZRT5LW581duB9nKJ6/J0Ywglxkin0dh3EWrpE/F2T1LWYMkqgGex2fMy4zJEhuQtdjj0zLT20+ybrXcis+Qiy4/p3SaF3cT2HYwyqXB0drOWz1dqsTlhaD7I7Ysn6aqCK6Tds3aI/U2dN2QrKnsAPuczODIRWRPl4kgjp+GjVWezIVw+iJB8tseSIY6JknhHFsZpTbQ4nhLSju2wYwviJYZenMDHwI8ZRhC777tv/jXLS+GW2Zy+P8AfWfM4+feTP4JoZPFf54S9vKuyIdeVFh2URmsZ+YcSGuU88MzEaJ4KG1M9ScoJQKLVxsLvq6CDpKzVtZfJiyzu8iIqQQuxZXxajY8yQxbt8F/+6bt27RJAbHWq2DFknleJirvFWLFdpGxEVDpf6Of/9oACAECAgY/APxCOXHiZkE6js3tG7f14MiZAwfGoW0Cb7diDtr5qZ4i4kwNhOsemSLngrBtDfSz+EGsRRVXISZCiJT2sB5vp9DXyZWaFa0KpjWJJNHErXLAZSd4bk3bwUnC9t24gEH4NTZMjFmbcng4JRHYz1SwBI12adltrIMRlA7BD5Z7vBsYyGELKGPsUnvH5UzLjTGVA6ZUAEn7Z8d3+qPRxl7d42HxNFWBBBgg6EEcEAJJA27PdTYc0rDFgwF108jHiolhYj627M1ixGnNrbmrHlwiwlrSskhhE3d77a64C6rcEk32/K2fLdRGJGyECTaC36Ub+7G86R75q5SGB5gyOAhlJBGoI0IPYaByuzkCBcZgUMfTByBbQ86fyK+2nxZmGNi11xH1CAIkfbFdT/hVmxXjqKoNzqFtDld7buVZznV0xMV6auCve8bKp5eo9//aAAgBAwIGPwD8kuBrUERwVp3MRSjn+3qPf//aAAgBAQEGPwDeN43feLlHl+hZmrtu3UWcLPo1QZzkQ6tHDIikCIzPxADG50d23Udy5WEde3OtMIrK5jIpz1RpBWmTFnpNOoOIsQ62LNvbdh3LcdhpM4Vrekrjgj09JDBeL5Zrxt3MKZbuCt3IVbXWqjqc9h/q4EstJCXUPV2T6nbwWxbnsO47PukpKxWr2VZi4QgikVsHo1zAFp1dTxNevG+VN0q32Ksu01IJcCjb1JhxCmwMZQpzOqsi7TGj+8beuWubathe22rE29r3NK5Ytg5aIicvpLFfZ+7ZrE8c0impFHla1TdU2Z1hcreT2L4QlBxOk19tje1w9QLAu1i7yDvG0XA5kWNuomhCCKLE2ZPSeqPCMcXSX0wD0WvVjY9t5y2i9ZGLLLNixt5DD9s4xTImQzq1ejLrhI8Pi9Q9J6MbdtHJnMN7mXlowI90i8hgrqhlPRDHdInnp08Ph9fqdbHN/L25g9O4btuLGUxlZSBCubDZMj8AwSyEgLx/3jaHALB8Ok4goz/gnEREZRHRERiCmIko8BZdMZ/uaVhAD4chiIjP+LGvKNWWWrLpy+bP/wBim9UiZqpW40bfHhjVS6hyP98MsZ9j95V++E5P4fBrfPxW+jDLyZLX9nFHdEZy/b2rc/pnrCXTp/jXqEvLwqyktSnALFlHyiUahn/ZP7ydv27aaDbNNOqzYYEjAyzsLDrEPZHWX2sXNtrUDbuKYL1tEFGoIziJmcygOgtOWU4Rt+81WVbNIiQuGZTrVE6lFEjJeAS4f2P3lczeS38VfwD3XbkKsOFq1kLpLTAnmOrqdPa04o7jOUTaQtxQPggjGCKI/gn943M3kt/FX8DdkCOo4RLQj9Kphv8A4MUhzzmqTK8z5BzI/wAwh+OL+6RlJVUGwILpiTiOpE/b04Vum5qUpzDYIwjVASAFog+vJT1pgvyvM3kt/FX8BiGRmDRIDj54KMpxvmwOnI6diDEf5SD/AN6R+OGVxnI7rlp0/LIjPGP8PG10ZjIlVl64+uUaz/nkX5XmbyW/ir+C5PZr70uSiZ+Umjrj/npP+V8ccvcsjOYaoc+I+SGH1s/8FLMZR4PyvM3kt/FX8Hab1M5Lc9syN4jGYjElDa4mX0ikT6n9WWAbuVFFimeRFNOTFojPTmINkhbl9HUGE36TIdWsBDEsHwEJeD423jmCevXoQSq5fJn7Ov8A5a3H9v8ALczeS38VfweYEviZsetuPp+cZglZfo4OjTgBz6yZ4c/wdoP92Bs19vjcdnh75BaDmLghDJ4hKUccJ4a9ehYsWzy8Bf210PrnnGqM4kSHoIDAsiWwJ6prONY/Gm4biM5NBUhX/S1no15fbLVgbjI9NuTCfJT4ZWPo1Z/xDr+3+W5m8lv4q/gxudJ/qW5QMAwiHWpwj2IcESJag8RoTq09XrYLYrds6ybDWpZKsusaSL0SzL7viwJ6WdrCaVNQprICFpUPRAiMZRGC5ppxlQfIBzBWjs6OwG4BEeBtf/zP9bV+sgMQQzEjMZxMdMTE/ubjsNdhFuG1Ck7i5AoEYsDrVpOY0nmP0fh2H7fJyFWy2m3iAS54qJ0s0wWWoc/AXwXbjuTwrU6463PZOQiPg6f48CwJzEoghn54npj4k2fk6kWbrbRc6I+TVMqVn5PpW/4eEU0RpTXWKlj8whECP/D8tzN5LfxV/C/7kIBXerbu54ksteaGnDKxlMRGnS2NBfVbipulafRWlC0Y+aZjrDP6QLq4NLghimDIMAozghKMiGY+aYxb5eecm3ZHerqIpzIqpxxaZz/glwfKSX7nPP8AcbV+AWIvVSWu3bsKo1nviZUk3TOb2iPWIErAz042kqfOs81Jv2l090oOQIGHGnTFyrKlhwgSeWtEkY6CxaVvvOaeV9vXARtdGiSjtuzjM32oatjY6/VWpQ6SHFjeKl0LG60d4HbEXzTCosqF6wBjkFHouMDPSDpEx8vGx7evmEb+5b5u3BKy+qsVJUdc9awUE5mC2RxldceyC8UuWt03o98obvTfaBlhS1OQ6vI6tJJgRJTBLseL/S3Dc6lwJ3NPMp7bVNiw0cCLMKBJiMR1Mp0kf3n18cvp3Dfy3ulv72VLdZ1daYS2Fy0G1ZV1gXGnTwjI+r9Iuzue+0uYGbJQqXHU9mooStgH6sWgn3eLGt0OP9QJL0fSw23zA4X7iF62hpAMCEcJshoCBiOoGWQYo7XuO7WeWuWnVZb+2KiYYTbfEkfVjcQOiuALyZ2OvjeXv5jr7vUruEq9+kKym3V1rEE3IgdCXgwtZkj6AY2Xk67zR+zuNUO/uXMbkL9YJWuQTWrqWJIW0+tqZo6q16u327nK6d9/7ko+pFe2/dSXAOUQFobUsyILBpdMOWzR2cK50nmThX112WKe3DWTNdy0SecW+jXLbMB2lEsU9TTjbd9FfC9frreSs84AijrhE/LpPUPxATWFAgESRlPREREZzM43bnB0TNavMqpavnONCv5FYdXlP/L8zeS38Vfwt4o2KpVki9+3HBFBzMr6IcOnLT1uuI43Hkzcp02qDDYgZ+bVpeI/o1yLh+q79zbrsdCt1rt298+COKn/ADNaZ+to9aD9zfOYuVx2p1beAqBIbib4MfVlcPohEZdYiPxvo4fT5nPbqG507KL2xWdt4zFg9GZQVgbHaHOdHU8UsbeO5xR2Hb6bBdfZt7JsOuyHgSENWMV6zJ+81am/Wxvm5bNQ23dR3yxFpW5Xmkp9TONPAYIrabq6svRAo143TYbdmq+9b3wd2G5EkAMAmqe2SXpLhHqBmlfX8vHLVyoSxXtG4RctQyZiZXwjXkvKJzLM/lxsvMayXFLb6luu8CmeJJv0aNEZadPV63Wxa2iG1/WXcwzvIFqLR6vNiH6ZnTnxdEdnL7WOWdwqksU7PdO1ahkzBSBKJccPKJzLUXy439PI+77UezFfsMujunGW7bLUz/moEMgFyNUcRWsgD+07eFtawnxYuXHLskOiXATyiHafk4unXgdz2L1bddsbX9Xt7DePghBwWobKHQtvW8RgMj/6ObKLPU9v3HmZi3q2+nBRSrcGRkFjOUTJu0enaK9OvshjaecK6KKuZKVdm37htZNMqtioTNYCuxok1OAo4g9Qh8Xy728cyuRSqlXlFLYaRcVQeMVh9ggAmPnsDw9K+Hihs+0nt8bXuKGqVujiZFiok2MW1fq4jIvYPX9XbxA+v2cUdlpzMooICusi7RQA5ay+sc9YviCduQU+t7rMoER7XCjKXTHlRIq/xcVNvOIizI8a3MfK5nWOPsfdx5H5fmbyW/ir+Fui9yAAK1uFi0mAODzUyYkJnLsl9XG3c87cGcaxVdCPBJRGmNX9+jUrywVhF6ocMr2Vi1Rx8onGqMI3Hwfs67UtZ/VhwrZ/y2H+T/aW7bNWtXOiCcYdJxHg4mWUNy/tNWASgBUpYwC1hECIjEZCIjHQIx8CRKM4nomP0YVt221wq00RMJQqNIDEzJTpjyimfiDOeiI8M4buxRr2fZcor/RLQU8DvWwdjyFr/MOZvJb+Kv4V1q651vUrbaRQZQWqVTlrjT4pYtbVa6F2QkYL5RKOkGR+kDgSxc5I3fqXKRmVSJ8ExE5uUH1YmeOr+yb9TG6EARZVwSXY4ZRJLE40w7L5eEekzH6GrFbcmJlA2Rg1LOcz4fgAj+sfb+MI2qkU/tHdc0hAdsU9lhR9Y9UJX9dn1cIpmMRcd6e6Uf1pRHUj6qhgVj5P5hzN5LfxV/C3Ve5LFRWtxsWUQJweamTEgU6ezM/R/ctXdtXCpo6KlpwzMGwxjrFOWWQjr9X8nHrFIirccCWcqKRnIo0MWceA46dJCeKzCWKX1c6b1L6AE05DGiPkAg0GPxe67bOFV64E1rC8AiMainFjnHcQkdvonooJL5DHpSH+AJcZv/8AQz6n5jzN5LfxV/CutVXOvFK22mUHMFqlU5ScafFL9y7NoP8AJboZWKzi7BycRxkSXghgF1tPjAWDpN1FVd1llETOX0T6O7PC373WNW6WSJ8mtjEuSJfdK1pIJ1AEDqH6eIdXa3mDah+9rNgf2goY8dDY0Bc/uW6XfQafYwrcNudD6zozAxzjpjoISGciAwLqmB9cC7XxbX5D2GdbGMGdwOOkYKMjhZ5fq0j6d/8AhrxW2qlGSa46dU+EynpNhfWYXWL8x5m8lv4q/hbqvclCorW42LKIE4PNTJiQKdPZz+j+4VW8hdlB9pTRgxn7JZ4sI2WmqluEekrtDOIIh/VHM5+iZ2fq9vBcvbvJBvG3RIZN6DaoJ0dbP9cifRu+wzx/3C5mpjp22yUDzBWHsxnkIbkA/IxPZt6fva/pO2jETE5xPTEx8Vya5gtytal0lz05TEdZxx/Vpzz+uehfjYZvm6xJbvuUay4nSa1FOvSX9q4vSv8A5HifmXM3kt/FX8K61KDrxStNplDJGdRKnKTHT4pfAXzvy9mq/TmGXRXGeoRjKLGjxtI+jsB+sR5GAvIyCwGS7lbPOVNyzmP0rLtKPxwwyu8IYpoyDFlGcEJRpIZj5pjD9kssk3bPYKiBnPWNERDahz8+dZix+x8VP3PcGaEIjOcukiKegFhHjGZdURwznPfwyoIPTQrT0gRLnqAP0k1y6xl+us+R+Z8zeS38Vfwt1XuSYSVncbFlEQYnqUyYkC6vZz+j8CYmM4nomJwvmTYVyzZbZaLdSOwGqc5SX0VzPXqs/Us9F2Dxq2nazs1onLjuaKZKY7UCuIbMaZ+njcNzUbUp9XF6tsecTK3smVtNemZFqlhGSzH7vj8Pq/FLLVpgpQkZNrTnIREYzkpmcQlUMrctbcWZH2SnPxv+pePYH/yyOt94eFVKqxTXSMLUoIyERGMoiPzPmbyW/ir+FcahDERStNpnDdOZEqcpMdMz1S+CpA58CzbSq1ETlmvrHon6psAMZxEsoPnwfLH/AOQP+YONovbUMWbBP0IXBaIaDBKGqk5idEaY1FqHqaMZ7vsV2qHhl1eAuLiPlkvViJo9zj1rbLS7Sc8iJc5yM/RMe0B/UONXxMTWlALCJIzKYgRGIzkimfBEYjlzl+ZVsiCg7VyYnScDPQ4/7Po/yqP1x+lPqYTtm3L4aEx4Z6SMp7bGF4zDnpIvzTmbyW/ir+Fuq9yTwCs7jYsoiDE9SmTEgfUmdOr6PwbG13hmUWBymRnIhKJ1AYF4pgUQQ4ZVWqvudU+iHQ2ElI/JLFnHUP8AuyLFLcN4qRdpughhyik4UTIyOFZwA+sgEeMOl69fD6+Fbht7hsVXxqW0PBP6J+iUeMM4/aVIy2zeRj0e5VoiDnLxbC/u7SvpLd9ggw3Zd6WNbe6ocRgLz4NhMzpG3VkumVEXVYsvSIb6M/EM/iM3OMVqWMmxhTECIxGckUz4IjE8s8sRK9nGYm3cKJgWDE/eN+ZHR6FHbs+P6PAbdQHojrPcXbaye0xk/PP8zs/mvM3kt/FX8K42uliYp2m0zhmWZEqcpIdMz1Z+G2huCRsVXjpYo4ziY/8AkUeKXi4K5S13+WrJxLQKfBM9GTJ8CrEeJY+6f2W9fAbhtjuIougwnoYs/lW0PCBjgd6oBLdy2Vg2VkqY4gLzgbIGPhNDE6ocv/F7ahwjddxsp29NySZSU84Bpo1SK2SuZ1Znp19XBfsy8m3IdJioxIhj6wxOof4/iFt6+4K9ZIyTGsnKIiP/AI7OC2nZxKly6ko9ZcyO1EdME+PHOf1VT/EfgNv21ekI6zWl0sac9pjS8Y5/NuZvJb+Kv4W7L3JHAKzuViwiNQlqUyYkD6kzlq+j+QZWsrFqWjIMUcQQkM9EiQz4cX6fKr3rrktc3UCZRpE4k+BEjOtqhD5fvFCejHFqEVa0MTqhZyLMpjIpE4n0gT/9+HDFRcbnSZwbFoo1sYExqQUGWZAHD6nCD0YaOriHjE09yT01NzrZBYUXz6o+8X9NLdamD2hxZ2/dRFe8bYcJvCvOFsgo1JtJz/U2A62n9WziK/V/kw5JYxg7wxfFWMhkso0S3TDM+1wxKcN3veDIKiiAJ4Y6zIjnSIiPRnjZNC7DB3+NVIwUUiMcOHxNif1OYF437lBdlTmTuNoKSZQuWQLGRMibdP3ao09Zn5h6zuTeuecV6wdLWlHyLD+kZdQMBuO6kW38uqPOukPAUR/U5/es+nbMdAfqAwrb9tQNesqMgWHz/KRTPSRl4xF1vzfmbyW/ir+FcbVUxMU7TabIbpiSNU5EQ6ZLqT8n5E93MZijuQriH+KD1jw5WyfE4gQJLz7XXxN2uyKrh60lnoAp+fPxD/gxZoczAypZv8FvrTAkQEYHJfEDLUAlq1cf7vyMC1RQaziCAxmJGYnwTEx4Yxt24oz9S3Jcbe1qlyxwyJ8fgrAILitZpMauoeqTHY4yadHbVF2E3DY9+X9pFeQUBfVFjcCPNVNaKZlAxu9MiZVCSnIYsgcQ6rEz0cUuIj6bAxExOcT0xMfkOad/qBrv8uJ2fd68x0TIIIotLz+gdU26x+rg6CZ4u3bXsVzmFxR0gTXJKvtkT+mNb36fq4/0wpVLBKrbioVXVDEZNAKImAlnEz1S+jjmcS5ntbbU2rdrVTbl1AVB6ggTHjmQFLK64MBWkdH6zWfZxyBujLPBt7pvS6G5koRgXqA7CjiRmOrxeCJlo045s2yOZLO17btNtYUxpgqHa2JE9JtMCn1deWrhR94Rl18WeeLPMzq17aVWZrVaqlDXb6iRgw7gGJy5tglH2SWpfV0rxycHHJVTddnfdt0wn0ZNkFGMzn9DX1cc2U122BWpbCu3UVGUit8w3NoRMdvqjjbv9RS5lc/cICoc7YS1RRNLmLRANARhhOLicRruJ2tfD0dTHLbLXMD92Tv1k6e5U7ALFAnKiaLKYrESQIEHY1H1e1+RnauXQ/ae6mXCggiTUB+DTGjrPb/Zq+2Y4/7g56aVq23IhomUTGXhGLGnqaR8Wsr0QePrwK1jAgMQIiMZRER0RERH5xzN5LfxV/C3YNyRKCsblZsJiSEtSjmJA+rM9r8ideyoXJZGlimDBCUfMQl0TiLVfaawOGcwLRBQM/OAlmIfZjHq+5o1EMTwbAdVy5n5Vs/8PYLBHVtBe2R0kCAbEyMMmJkY4MTqSzo1ehngH4+jEWbe4WH3c9cQ1pCQF9JAhIrDLxeFhewbtYO2myJepWHTm0GLjVKTP9YBhEksi6/V04NLgFi2DIMWcQQkJRkQkM9EjMYfyNk1hIdntgMidS6ZBLiEjnPUmt92g9XWWaldpZ4gDYIlPgGZiJ/2fD3/AHu4am7dvVKvS9WjVriFCQN15xp0nB/JjmHY7W4Lt7nu9eaFa4UFALqJT6vRSXRqySJHq0j4/jY5IKLCf/8AMRlc7Xpf8tFb0PR9KNXXxv7LTltjd91fuSYXq6i3CAiB6ojrxo6dOOUtnZbrk7l7dv2nZYMHpYviPZoVnGevJ8drHMm5vatit6tKs1wDVqAVqhUwzOMtWceLjcOTWWUlctrvLCwOvhRNs2mEznGvq8XrdXHLO7cv3Ko71y/T9QYm2LPVbCjWC29ZfplzqXqWWnHMG87zuVazb33bPUYWhZrVXZGsRBeqSI64CQ9cvSsPX1BxV5ICygbyE1FFZLXwZms1bSmMo19YV9Xq45cvIatYbNdm3YE9WZjKjVpXlE9bMvG+HxNzsQLSjNVVfXczyFx/SLqYKvtS52nYZmRY+ZmIMflg2jkT5/sa/ov6xuImqHHvTGll5sRxJj6K4jqpX9QPtavzrmbyW/ir+FcbUUxUU7LKbIbERMmqciIdMl1J+T8om7VAmFtr4sOWEZlKZEltKIjpnhwWvycQ+uUJYXWFi+kC+tlH9IMV9y3JJ3Km2NgjNXTOZgQCMu7A9BatDdJYZuaLBMYvKP2fp02SMuwArLw5/TguH9fCt3moukYKKqxaHsBra5FB8FloIgl5GPbQOrxcM4WzKTdrTA261j0zB1dlguOSJq2ZdVuG7jyixsyiOIexuYTazxjpME8SSOq6R+7JRcPX21YsblVUSqgO4CYZGTJkAAm6x8XSwyD7GOJZaCV+DWwoCP8AaUxgWpMWLLpEwmCGY/RMfmXH3W2CInsLmc2HPzLWOZn/ALMHt3JG3MUvPSd5kRJjE/LMlmiv9smM+pj9pc0vndb7J1moiKU6vD6Qi9JY+36P+ywKlBALCIEAGIgYiPBERHg/O+ZvJb+Kv4W7huNea5WNys2EwUiWpRzGg+pJdr8qVk9vhRnOpkIM1AUz4ZkFkIf7sfs6lUUmn06kCMaSz7UnE9uS8bXiie1Jigu7DZZK8+HxV6SERD9VqEiL0ePThFuvHj9rKPLjrj9scLtWHxSQaGV3m+YEJIpElBr7PaiZ62F7fsYqs3WBDWWDmSQlZ9iepPpWM8IBq7PXPDTrlXemw47NiqSoUJMZ94QGvrLk/tjgOY9zgd13KzJQ+bQ6wrHE5FUroPUtIJ7OrTxHfemfXwzmTk1IpbXgj3HagiYr2k6ZzKEj1Qen7wTVAGY6w62KT7bdbJrLa95zERJEEGZTPgiOnEt2+ne3GpHhu1KxmicvDKznQToj6SBbisO32hsTahsrEM9Q8CYFsNAsjUQEQiQGOrViBs2FJmfBDDEJn+VMY40GMry1cTONOXz6vB+SkinKI6ZmfBgwO3FuwPhRUyZMT9Y8+EH2jxweWNunbqR5xNwvDl/1DIhY/wCAtp49f5nus3S2U6mLgihcz8xsKeM2PtAP1MBWppCuhcZApQwIxH6BH895m8lv4q/hW20wYA07LKbIbEDMmqcikcpLqfR/Llt16CGM4YlwTkxTB7LFz9KP5w4kE7hUYnPoeYMA8vnJQ6h1fwMxNW+J2rhlJt3AZ4TNU/IsQ6gL+oWv62LW3bfaIQrwBI4nVlimRqg+rHDLr619nxMZXanEGP1gxl/ODUOLlfdbBVJtPh6hISMBiQEJ1EuC0kRD82HGVwl0RMhrbelkhEAMzAm+AmCY0+0WrqB2MGlZtZQORK5thMLg2FgUGaZGZnRxBjT6PyOxhe8k0K+2cAXw0shEFSMEPR8nR1dOG77y+i3SG4uUbk5cLU1wTIzxaosz4Dz0QJtZGrR1vvcG/bKCXOCdNsby+LaE56fTlY4rC1+ET1kB+Lhzdg3ANkt2QNZbYJzCLozHWAaw6tDvoPSv+8wqvXsj67CxJtRkStwzp63omQJzp+kPwpO/cTWGOmZcwQ/pTGJhNk7px4IrhMjn/eHoX/OxK+VtgKFz0RYbBMy/Tq9Ege8PEM5r3qa6JnOaip15f4a+HXH/AJuBYNT1ywHTDreTMp+ovKFB/EGIEYyiOiIjwRH5/wAzeS38Vfwt3DcK5VyfuVl6YKYnUo5jQcaZnoL8yX63rTbRn6vcTMQwILtBOcSLFl4yzx/n92a6tn1kpWKCKPmJkEZR9jTgKrdrrjCl8JBgECYREZDkY5F1frFixt9viV9wrHIuGcpz6eq4M/Cto9cCHCa1G4bG2Dhaa46tZlM+AR60eV9HFLZN1lTNvExiilcgcTKI1CJlEAZQsesMM8bycemoZ/PMQUf8NWP2lNBhDCTVYQEzEsCesPaH9Uca/wCVj/uC7EM3Pcxh7Gz1uGo+smsqZ7CkrkR0j2z1M8bGkvRWkzxKVsOhiHR0gxZeGMi7Q+OPUPAosbU9txUcOxKyCAlgdVkh2p0SUdXGVXluyyZ8GZlP9FRYzocrFAz4JMHT/wAYVGM1VkUAL6QqCY7wnF/Nx/6zzJwVl2lqNk9HkriuGIZud6zdPwzlIrif4Z67P+ZiCqbWjiR+saPFP+U3XOIEYiIjoiI6Ij4i5m8lv4q/hW20gaA07LKbeMMDMsVORSOUlmH0fzUP2tSXZJfQthZiYxPyCwJE4j7WJdtlFaXlGmXzmbMvm4jJI4j9EYOgw5S4ShtWzEZyto9ksvGGc9DA8YMcCNuGx05C9DwhRfW9LIMD7Q4Zum7mDdxcvgrSuZJaFTMSYwRRHEYyY9IeX1BxO1bsBhtyimNt3OBI1SmZzBNghieC5Mej6/o2AOsSxNHlf/O3G9T1sYn1avE9Euc7LQWjtClckxhfysVqCBiQrLFUHMRqLTGUmX1jnrF8U8zeS38Vfwt3DcaxVifudl6YPLrKOY0MjTM9Uvzq9n/+u3+gWNkkZiR4JZTHTH3h/FXM3kt/FX8K22iLRGnZZTbxh0zLFTkUj0lmH0S/Oi4uXDynXqy06cunPP5ML/Z/C9Vy9FwNPDyz8TR1fD83xCWgoLTMiWmYnIo8MTl8v5HmO9vDCroaTVKLhmWopMC6IGJnTkPb7OPby7lvmY9vLuW+Zj28u5b5mPby7lvmY9vLuW+ZjdF7jblZ2txsWkRCzLNTJiVn1BLLV9HHt5dy3zMe3l3LfMx7eXct8zHt5dy3zMe3l3LfMx7eXct8zHt5dy3zMe3l3LfMx7eXct8zHt5dy3zMe3l3LfMx7eXct8zHt5dy3zMe3l3LfMx7eXct8zHt5dy3zMe3l3LfMx7eXct8zHt5dy3zMe3l3LfMx7eXct8zHt5dy3zMe3l3LfMx7eXct8zHt5dy3zMWq67xSbksWESlsZyQyMR0h+nG2bTuFuU26qpByhUwoGZMiyggGRntfJj28u5b5mPby7lvmY9vLuW+Zj28u5b5mPby7lvmY9vLuW+Zj28u5b5mPby7lvmY9vLuW+Zj28u5b5mPby7lvmY9vLuW+Zj28u5b5mPby7lvmY9vLuW+Zj28u5b5mPby7lvmY9vLuW+Zj28u5b5mPby7lvmY9vLuW+Zj28u5b5mPby7lvmY9vLuW+Zj28u5b5mPby7lvmY9vLuW+Zj28u5b5mPby7lvmY9vLuW+Zj28u5b5mPby7lvmY3Tedjvuqqs2Jap6pMIMSiIzNfzdHjhiP2tWVudUZ0zaTMKZnPgzyzVn9leF1wt+qXGzAhVtxwjKZ+QJnNZ/YP4Y2dz29FtwBwwY0IIoDPVpiZ+TVONLNqpBM9ORAETl/HiZXtNM4jwyKxn/hj3NU7oce5qndDj3NU7oce5qndDgtG0Uy0zIlksZymPDE/px7mqd0OPc1Tuhx7mqd0OPc1Tuhx7mqd0OPc1Tuhx7mqd0OPc1Tuhx7mqd0OPc1Tuhx7mqd0OPc1Tuhx7mqd0OPc1Tuhx7mqd0OPc1Tuhx7mqd0OPc1Tuhx7mqd0OPc1Tuhx7mqd0OPc1Tuhx7mqd0OPc1Tuhx7mqd0OPc1Tuhx7mqd0OPc1Tuhx7mqd0OPc1Tuhx7mqd0OPc1Tuhx7mqd0OPc1Tuhx7mqd0OPc1Tuhx7mqd0OPc1Tuhx7mqd0OPc1Tuhx7mqd0OPc1Tuhx7mqd0OPc1Tuhx7mqd0OPc1Tuhx7mqd0OPc1Tuhx7mqd0OPc1Tuhx7mqd0OPc1Tuhx7mqd0OPc1Tuhx7mqd0OPc1Tuhx7mqd0OPc1Tuhxu1TYtpkqa7JLURDw6wRER2TPSPR9XXgG8wblOUf+UpxpGI+binH9BeIjbaCls+V5RxGzMfLLT1H/AL/yCV87TXjav2FEr9cYS18eLRadMiQ+k4fExy5P+lBiW9leXF8NuYxladv6fWpuZkahGB7Hj/zMWtj5W2L9oNoAs7t246alYSbGsFJOVsKwzT29HUXjeeY7O1mi/sDjqbntZNGdL1EEGK3wOkw0t1CejG2WN45bbQ2DeHqq0b5PA3gb/Z5tVAj0It/vT4fj9bq4u7DylssbuzaYCd1svsRVSBsHWFZJyLOI8g7X6tfj45s5ju1rAescw2oTtsjHrJWH8KAqaImR4uvql1sbejnHYI2mnuzRrU7qLQ2RVYZEyqvbiAXwzZ2RYGtevG8cs8u8vFuV7aJTLXMsgivw2qho6mEJELiKdClaS1dZmsNOG7/s3Kz37dtwnO8m+wCTWxPtCqoaT9a4AxrNnox8Tt69FTdaczNa6ldhMlGU6GjBjnHz5F8c828tXHtZVvqVuu2i1hGAjM5uFUFM8MeJYMdA/wBRiN8s2WlsHMlvcNr24CYUpiKopJDFhM8P0k8JfV8Y2Y3cgez9mcuUF1pTDChZWXdbVK4nRJRDbAai/q/hA63R9YoRy/IcVqdaYb63npgzGV8TT9vTivzXy5UMtg3Fgo5l2qoqSgJLoXuKUriZggz9Nwx63+Jqxu1fni3vFbbR4X/btHa4sjWtIIMzKSpDJuskfiMMOH/R/wBQdtVs9qgx18nUtsYJsdCGRXlMas2E49A9chNnWxy4uuhjmL3bamNBYERCIHEmRwMTpEPGKccyBv1ez6hvlodx2u/XQ2wDDNcA2qXAEyW4DH0Yl95jdt2TtFhN6nzd+3w2hwaHsSvhlKgguqbOGWrqdsxIQxtHL3LdS2wh3Ctc3SzZrNrrpprHxDFhOEYmwUxoWsNWOfbDkMWiw3bJruMCEGQFUhPhnMaT0F0FpxzJVbWcFhp7zKkEsoYUMJvD0BMai159TLtY5fS9ZKavb6osUwZExKFDEiQl0iUfmXKu/VLjE7VG4DT3itEzwmKdlImfzcMAd9rRg71S2weVdlv0No3JEFkkztg0nOLo051yzE/rCn6WOZFLuM/YOw100xqQXoytMykmTH0gkLC/5HxLsnO9CJhp1Lu1OKPpMSz1XVHRqgWOYz/Cxy7eqqkdy5bbU3ow8JcRh8SwBTGf3Zv1Fl/U43vm5ozD+Y9zsWRIu1wVmQrEvIadj4i3LZK0RNxoiynqmBjjKMWBGougdenhl9U8b3sG7wB8x7y924vLWMxNqDE0ekHqRxOCGrxfSni5PMEwe9bpdbdvGJCecnkI9YOrOeRN8pnxLCr1ZVpYzqEHALBgsstUQcT09OCQ9YtScaTUYwQzHzSM9Exga9RIIQGehShgAjOdU5CMQMZlOfwNztW71jbULVEsvUozsKjWMalR9KezivtFsrVncX1F2qlavXNzrIkUrgVCuOu7qEZh4o9bG+3K1q5tk7clobjEqJN+mUBJauCXgZp6y8cm7TVncLu226RvuOimTXW4KuHqzBIY1EYEXGucL7rx8c17hd3mwytttuPXYuRkqpPDHJFXKMzWWcaRHrEzFGlbobltYbmcK265uFQkV3sLsADJmciZ+r1iOrHMm2tsMOlWpUGV6xFmsCZDOIQD8knp62Nr2u6Rg/d2MTTOAmVyxY6+GbPABnH3efbxU2GwwovW0NtDEDmC0IjrtefZUHiiReNj131Pcy2WGcKd+GmyaGerRr43blWrq8SF45Up1bRxt92puDXJAvRN0AokmUeNp1ZhhvNlSHfs1S7DTgwybpqyYtyXnPT6ItPTjb+YHk6U7qKi22otcstPJwwYLVXDMiZpnreKP0sWNtBFrbt0qgLXbbuKZrvhRTpFwjMkJqkurrAvj7mD/px/FDG2xOUkPK0SPzxnaiM4x/rBETGf7MpZxH/Rt8OP9Krl+wurUja7QTYccLVBHSQIDLCyDUZdnH+o14aSL8/tpCoXcDiVlywFCL7AZTmtJdfysctr3LnRe/2mbvQdG30lV11lLhkannKOIzTGqBVxXdbVjmuM+n1Dbf8Ag3Fh9GP/AFPaDXu22FHTMWKc8WIj+8XxFfbxzZ/qkFQ7lY6k0tnpsHrnVpq4thcD1urZuSY5f2eG7he5wqKVep/5XlfZ69cBGSiJiqWfFtaK/gsfdcPSWP8ATggKCEtnuSMxOcTE168xMY3PYdwuLVvaQ3Ooe0zP+b4zmP4YDX+9KJ1j6SB4YeP2cf6Y7qe6nslANqmpG8itTV17Dq64CGceCSsbARw+KWKRHzVZ5l3Lb6b5M0V6sVUpdkPCs2Kunrmelil9fseJ8fHXsqByWRkamDBCUfMQlnE4izCQh4hwhdAxrheefDg8tWjPxcPk6yim0MBZmQGeKMRpgW9HpBgejr4XVv0a9qumYlKXKBgBI9AyAHEiOn6uHjNZUja6bUaBybnGn0vR6Tq9Xr4OtT2mmhDCFjFLrqESIJ1ARCI5EQF0h9DB2QUAvbAix0DEGQj2RI8tRQOfVxlPgwNeqoEJDPQpQwARnOc5COUR0zh1mttNNL7IkFhq66hJgn0MEygcyE/Hie3hBLrKEqwSutIgMSoJiIkFZR6MMojqjidxmjXm8QyBW5UHGkZjTIy3Tr06ejTqxNB9ZTacjC5qmsSVIR4A4cxo0/V04JG00UUVHOo11lAoZn5yhcDnP/uO/wD/2Q==)\n",
        "\n",
        "[Image source](https://saugatbhattarai.com.np/what-is-gradient-descent-in-machine-learning/)\n",
        "\n",
        "The y-axis represents the possible values of the cost function, or error, evaluated for different model weights $\\mathbf{w} = \\{w_1, w_2,...,w_j\\}$ where $j$ is the total number of weights in the model. In the beginning of the model training process, we have random weight values which will yield different errors. At each iteration of training, we will calculate the gradient at the point in the cost function given the current weights, multiply the gradient by a pre-determined **learning rate**, and then descend along the cost function curve accordingly. "
      ],
      "metadata": {
        "id": "I_asSI1is1mp"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Types of gradient descent\n",
        "\n",
        "There are a couple of different types of gradient descent out there. In order to understand the differences between these, it is first helpful to define some of the following terms:\n",
        "\n",
        "- **sample**: a sample is a single data point that can be passed to the model\n",
        "- **batch**: a hyperparameter which defines the number of samples that the model must evaluate before updating the weights in the model\n",
        "- **epoch**: a hyperparameter that defines the number of time that the entire training dataset will be passed through the model \n",
        "\n",
        "Now, we can start talking about the different types of gradient descent!\n",
        "\n",
        "## Batch gradient descent \n",
        "In batch gradient descent, every data point in our dataset is evaluated in a given training iteration and the gradients are summed for each data point and then used to make a single update to the weights in the model. In other words, **the size of a batch is equivalent to the size of the entire training data set**, and the number of batches in an epoch is 1. \n",
        "\n",
        "##### Pros\n",
        "- Because we only update the model after evaluating all data points, this results in fewer updates and a more computationally efficient training process\n",
        "- Fewer udpates results in a more stable gradient and more stable convergence\n",
        "\n",
        "##### Cons\n",
        "- The more stable gradient may result in the model converging earlier to a less optimal set of parameters, e.g. a local minimum instead of a global minimum\n",
        "- Prediction errors must be accumulated across all training samples because the model weights are updated after evaluation of all samples\n",
        "- Usually the entire dataset needs to be loaded in memory for the model to work with it\n",
        "- Altogether these cons make this approach slower \n",
        "\n",
        "## Stochastic gradient descent (SGD)\n",
        "In stochastic gradient descent, a random subset of the training dataset is evaluated in each training iteration to provide a single update to the weights in the model. Typically, SGD refers to a random subset size of 1; that is, **each batch consists of a single sample**. The number of batches in a single epoch, then, is equivalent to the number of samples in the entire training data set. \n",
        "\n",
        "##### Pros\n",
        "- Because the model weights are updated more frequently, we can have a higher resolution of the how the model performs and how quickly it's learning\n",
        "- The higher frequency of model updates may help the model learn faster for some problems\n",
        "- The \"noisier\" model updates may help the model avoid local minima\n",
        "\n",
        "##### Cons\n",
        "- Updating the model more frequently is more computational expensive\n",
        "- The more frequent model updates result in noisier gradients, resulting in more variance in the error landscape across training epochs\n",
        "- The noisier updates can also make it harder for the algorithm to optimize \n",
        "\n",
        "## (Stochastic) mini-batch gradient descent\n",
        "As with SGD, we are picking random subsets of the data to pass through the model in order to inform the updating of the weights; however, here **the batch size is somewhere between a single sample and the entirety of the training data set**. Therefore, in a single epoch we see a number of samples roughly equivalent to the total number of samples in the training dataset divided by the batch size. This is very popular for training neural networks.\n",
        "\n",
        "##### Pros\n",
        "- More frequency model updates than batch gradient descent allows for a more robust convergence and a better likelihood of avoiding local minima\n",
        "- Less frequent model updates than SGD results in greater computational efficiency \n",
        "- Smaller batches means that we don't have to have the entire training dataset in memory (as with batch gradient descent)\n",
        "\n",
        "##### Cons\n",
        "- We have to define an additional batch size hyperparameter \n",
        "\n",
        "\n",
        "\n"
      ],
      "metadata": {
        "id": "8YVTmMwtGAxw"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Cost function\n",
        "I will be using two closely related terms in this section: **loss function** and **cost function**. A **loss function** is calculated for a single data point while a **cost function** is the sum of the **losses** across all the points in a batch. For our tutorial, we will use the **Least Squared Error** loss function, which is commonly used for linear regression. For a given sample $j$, it is defined as:\n",
        "\n",
        "$$\n",
        "LSE = \\frac{1}{2}(\\hat{y}_j - y_j)^2\n",
        "$$\n",
        "\n",
        "where $\\hat{y}_i$ is the predicted output for a given input vector $\\mathbf{x_i}$. Its associated cost function is known as **Mean Squared Error (MSE)**:\n",
        "\n",
        "$$\n",
        "MSE = \\frac{1}{2m} \\sum _{j=1} ^{m} (\\hat{y}_j - y_j)^2\n",
        "$$\n",
        "\n",
        "where $m$ is the total number of training samples. \n",
        "\n",
        "In our example, $\\hat{y}_j$ is calculated as a linear function that is essentially the dot product between the weights in the model, $\\mathbf{w}$ and the inputs for a given sample, $\\mathbf{x}_j$:\n",
        "\n",
        "$$\n",
        "\\hat{y}_j = w_1 x_{i_1} + w_2 x_{i_2} + w_3 x_{i_3}\n",
        "$$\n",
        "\n",
        "Thus, the cost function can be expression as a function of $\\mathbf{w}$:\n",
        "\n",
        "$$\n",
        "J(\\mathbf{w}) = \\frac{1}{2m} \\sum _{j=1} ^{m} ((w_1 x_{j_1} + w_2 x_{j_2} + w_3 x_{j_3}) - y_j)^2\n",
        "$$\n",
        "By training on the data we simulated using the coefficients $a, b, c$, we are trying to optimize the weights $w_1, w_2, w_3$ to try to recover the values $a, b, c$ used to generate the data. \n"
      ],
      "metadata": {
        "id": "fWjUVsmduSRj"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "def pred(x, weights):\n",
        "    \"\"\"\n",
        "    predict y given inputs and weights\n",
        "    \"\"\"\n",
        "    y_pred = np.dot(x, weights)\n",
        "    return y_pred\n",
        "\n",
        "\n",
        "def mse(y_pred, y_true, m):\n",
        "    \"\"\"\n",
        "    calculate MSE \n",
        "    \"\"\"\n",
        "    mse = (1/(2*m))*np.sum(np.square(y_pred - y_true))\n",
        "    return mse\n"
      ],
      "metadata": {
        "id": "vyVxoaABcZ85"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Learning rate\n",
        "The learning rate is a small value that determines how far along the curve we move to update the weights. It's important to pick the right learning rate - a large learning rate can result in overshotting the optimum value, while a small learning rate will make it take much longer for the model to converge on the optimum. \n",
        "\n",
        "![learning_rate.jpeg](data:image/jpeg;base64,/9j/4RvvRXhpZgAATU0AKgAAAAgADAEAAAMAAAABA7YAAAEBAAMAAAABAhEAAAECAAMAAAADAAAAngEGAAMAAAABAAIAAAESAAMAAAABAAEAAAEVAAMAAAABAAMAAAEaAAUAAAABAAAApAEbAAUAAAABAAAArAEoAAMAAAABAAIAAAExAAIAAAAeAAAAtAEyAAIAAAAUAAAA0odpAAQAAAABAAAA6AAAASAACAAIAAgACvyAAAAnEAAK/IAAACcQQWRvYmUgUGhvdG9zaG9wIENTNiAoV2luZG93cykAMjAxODowNjoyMiAxNjo0NjoxMwAAAAAEkAAABwAAAAQwMjIxoAEAAwAAAAH//wAAoAIABAAAAAEAAAJYoAMABAAAAAEAAAEaAAAAAAAAAAYBAwADAAAAAQAGAAABGgAFAAAAAQAAAW4BGwAFAAAAAQAAAXYBKAADAAAAAQACAAACAQAEAAAAAQAAAX4CAgAEAAAAAQAAGmkAAAAAAAAASAAAAAEAAABIAAAAAf/Y/+IMWElDQ19QUk9GSUxFAAEBAAAMSExpbm8CEAAAbW50clJHQiBYWVogB84AAgAJAAYAMQAAYWNzcE1TRlQAAAAASUVDIHNSR0IAAAAAAAAAAAAAAAEAAPbWAAEAAAAA0y1IUCAgAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAARY3BydAAAAVAAAAAzZGVzYwAAAYQAAABsd3RwdAAAAfAAAAAUYmtwdAAAAgQAAAAUclhZWgAAAhgAAAAUZ1hZWgAAAiwAAAAUYlhZWgAAAkAAAAAUZG1uZAAAAlQAAABwZG1kZAAAAsQAAACIdnVlZAAAA0wAAACGdmlldwAAA9QAAAAkbHVtaQAAA/gAAAAUbWVhcwAABAwAAAAkdGVjaAAABDAAAAAMclRSQwAABDwAAAgMZ1RSQwAABDwAAAgMYlRSQwAABDwAAAgMdGV4dAAAAABDb3B5cmlnaHQgKGMpIDE5OTggSGV3bGV0dC1QYWNrYXJkIENvbXBhbnkAAGRlc2MAAAAAAAAAEnNSR0IgSUVDNjE5NjYtMi4xAAAAAAAAAAAAAAASc1JHQiBJRUM2MTk2Ni0yLjEAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAFhZWiAAAAAAAADzUQABAAAAARbMWFlaIAAAAAAAAAAAAAAAAAAAAABYWVogAAAAAAAAb6IAADj1AAADkFhZWiAAAAAAAABimQAAt4UAABjaWFlaIAAAAAAAACSgAAAPhAAAts9kZXNjAAAAAAAAABZJRUMgaHR0cDovL3d3dy5pZWMuY2gAAAAAAAAAAAAAABZJRUMgaHR0cDovL3d3dy5pZWMuY2gAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAZGVzYwAAAAAAAAAuSUVDIDYxOTY2LTIuMSBEZWZhdWx0IFJHQiBjb2xvdXIgc3BhY2UgLSBzUkdCAAAAAAAAAAAAAAAuSUVDIDYxOTY2LTIuMSBEZWZhdWx0IFJHQiBjb2xvdXIgc3BhY2UgLSBzUkdCAAAAAAAAAAAAAAAAAAAAAAAAAAAAAGRlc2MAAAAAAAAALFJlZmVyZW5jZSBWaWV3aW5nIENvbmRpdGlvbiBpbiBJRUM2MTk2Ni0yLjEAAAAAAAAAAAAAACxSZWZlcmVuY2UgVmlld2luZyBDb25kaXRpb24gaW4gSUVDNjE5NjYtMi4xAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAB2aWV3AAAAAAATpP4AFF8uABDPFAAD7cwABBMLAANcngAAAAFYWVogAAAAAABMCVYAUAAAAFcf521lYXMAAAAAAAAAAQAAAAAAAAAAAAAAAAAAAAAAAAKPAAAAAnNpZyAAAAAAQ1JUIGN1cnYAAAAAAAAEAAAAAAUACgAPABQAGQAeACMAKAAtADIANwA7AEAARQBKAE8AVABZAF4AYwBoAG0AcgB3AHwAgQCGAIsAkACVAJoAnwCkAKkArgCyALcAvADBAMYAywDQANUA2wDgAOUA6wDwAPYA+wEBAQcBDQETARkBHwElASsBMgE4AT4BRQFMAVIBWQFgAWcBbgF1AXwBgwGLAZIBmgGhAakBsQG5AcEByQHRAdkB4QHpAfIB+gIDAgwCFAIdAiYCLwI4AkECSwJUAl0CZwJxAnoChAKOApgCogKsArYCwQLLAtUC4ALrAvUDAAMLAxYDIQMtAzgDQwNPA1oDZgNyA34DigOWA6IDrgO6A8cD0wPgA+wD+QQGBBMEIAQtBDsESARVBGMEcQR+BIwEmgSoBLYExATTBOEE8AT+BQ0FHAUrBToFSQVYBWcFdwWGBZYFpgW1BcUF1QXlBfYGBgYWBicGNwZIBlkGagZ7BowGnQavBsAG0QbjBvUHBwcZBysHPQdPB2EHdAeGB5kHrAe/B9IH5Qf4CAsIHwgyCEYIWghuCIIIlgiqCL4I0gjnCPsJEAklCToJTwlkCXkJjwmkCboJzwnlCfsKEQonCj0KVApqCoEKmAquCsUK3ArzCwsLIgs5C1ELaQuAC5gLsAvIC+EL+QwSDCoMQwxcDHUMjgynDMAM2QzzDQ0NJg1ADVoNdA2ODakNww3eDfgOEw4uDkkOZA5/DpsOtg7SDu4PCQ8lD0EPXg96D5YPsw/PD+wQCRAmEEMQYRB+EJsQuRDXEPURExExEU8RbRGMEaoRyRHoEgcSJhJFEmQShBKjEsMS4xMDEyMTQxNjE4MTpBPFE+UUBhQnFEkUahSLFK0UzhTwFRIVNBVWFXgVmxW9FeAWAxYmFkkWbBaPFrIW1hb6Fx0XQRdlF4kXrhfSF/cYGxhAGGUYihivGNUY+hkgGUUZaxmRGbcZ3RoEGioaURp3Gp4axRrsGxQbOxtjG4obshvaHAIcKhxSHHscoxzMHPUdHh1HHXAdmR3DHeweFh5AHmoelB6+HukfEx8+H2kflB+/H+ogFSBBIGwgmCDEIPAhHCFIIXUhoSHOIfsiJyJVIoIiryLdIwojOCNmI5QjwiPwJB8kTSR8JKsk2iUJJTglaCWXJccl9yYnJlcmhya3JugnGCdJJ3onqyfcKA0oPyhxKKIo1CkGKTgpaymdKdAqAio1KmgqmyrPKwIrNitpK50r0SwFLDksbiyiLNctDC1BLXYtqy3hLhYuTC6CLrcu7i8kL1ovkS/HL/4wNTBsMKQw2zESMUoxgjG6MfIyKjJjMpsy1DMNM0YzfzO4M/E0KzRlNJ402DUTNU01hzXCNf02NzZyNq426TckN2A3nDfXOBQ4UDiMOMg5BTlCOX85vDn5OjY6dDqyOu87LTtrO6o76DwnPGU8pDzjPSI9YT2hPeA+ID5gPqA+4D8hP2E/oj/iQCNAZECmQOdBKUFqQaxB7kIwQnJCtUL3QzpDfUPARANER0SKRM5FEkVVRZpF3kYiRmdGq0bwRzVHe0fASAVIS0iRSNdJHUljSalJ8Eo3Sn1KxEsMS1NLmkviTCpMcky6TQJNSk2TTdxOJU5uTrdPAE9JT5NP3VAnUHFQu1EGUVBRm1HmUjFSfFLHUxNTX1OqU/ZUQlSPVNtVKFV1VcJWD1ZcVqlW91dEV5JX4FgvWH1Yy1kaWWlZuFoHWlZaplr1W0VblVvlXDVchlzWXSddeF3JXhpebF69Xw9fYV+zYAVgV2CqYPxhT2GiYfViSWKcYvBjQ2OXY+tkQGSUZOllPWWSZedmPWaSZuhnPWeTZ+loP2iWaOxpQ2maafFqSGqfavdrT2una/9sV2yvbQhtYG25bhJua27Ebx5veG/RcCtwhnDgcTpxlXHwcktypnMBc11zuHQUdHB0zHUodYV14XY+dpt2+HdWd7N4EXhueMx5KnmJeed6RnqlewR7Y3vCfCF8gXzhfUF9oX4BfmJ+wn8jf4R/5YBHgKiBCoFrgc2CMIKSgvSDV4O6hB2EgITjhUeFq4YOhnKG14c7h5+IBIhpiM6JM4mZif6KZIrKizCLlov8jGOMyo0xjZiN/45mjs6PNo+ekAaQbpDWkT+RqJIRknqS45NNk7aUIJSKlPSVX5XJljSWn5cKl3WX4JhMmLiZJJmQmfyaaJrVm0Kbr5wcnImc951kndKeQJ6unx2fi5/6oGmg2KFHobaiJqKWowajdqPmpFakx6U4pammGqaLpv2nbqfgqFKoxKk3qamqHKqPqwKrdavprFys0K1ErbiuLa6hrxavi7AAsHWw6rFgsdayS7LCszizrrQltJy1E7WKtgG2ebbwt2i34LhZuNG5SrnCuju6tbsuu6e8IbybvRW9j74KvoS+/796v/XAcMDswWfB48JfwtvDWMPUxFHEzsVLxcjGRsbDx0HHv8g9yLzJOsm5yjjKt8s2y7bMNcy1zTXNtc42zrbPN8+40DnQutE80b7SP9LB00TTxtRJ1MvVTtXR1lXW2Ndc1+DYZNjo2WzZ8dp22vvbgNwF3IrdEN2W3hzeot8p36/gNuC94UThzOJT4tvjY+Pr5HPk/OWE5g3mlucf56noMui86Ubp0Opb6uXrcOv77IbtEe2c7ijutO9A78zwWPDl8XLx//KM8xnzp/Q09ML1UPXe9m32+/eK+Bn4qPk4+cf6V/rn+3f8B/yY/Sn9uv5L/tz/bf///+0ADEFkb2JlX0NNAAL/7gAOQWRvYmUAZIAAAAAB/9sAhAAMCAgICQgMCQkMEQsKCxEVDwwMDxUYExMVExMYEQwMDAwMDBEMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMAQ0LCw0ODRAODhAUDg4OFBQODg4OFBEMDAwMDBERDAwMDAwMEQwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAz/wAARCABLAKADASIAAhEBAxEB/90ABAAK/8QBPwAAAQUBAQEBAQEAAAAAAAAAAwABAgQFBgcICQoLAQABBQEBAQEBAQAAAAAAAAABAAIDBAUGBwgJCgsQAAEEAQMCBAIFBwYIBQMMMwEAAhEDBCESMQVBUWETInGBMgYUkaGxQiMkFVLBYjM0coLRQwclklPw4fFjczUWorKDJkSTVGRFwqN0NhfSVeJl8rOEw9N14/NGJ5SkhbSVxNTk9KW1xdXl9VZmdoaWprbG1ub2N0dXZ3eHl6e3x9fn9xEAAgIBAgQEAwQFBgcHBgU1AQACEQMhMRIEQVFhcSITBTKBkRShsUIjwVLR8DMkYuFygpJDUxVjczTxJQYWorKDByY1wtJEk1SjF2RFVTZ0ZeLys4TD03Xj80aUpIW0lcTU5PSltcXV5fVWZnaGlqa2xtbm9ic3R1dnd4eXp7fH/9oADAMBAAIRAxEAPwD1VUqc3If1jJwXVNbRRRTcy4Olzja69jmuZ+a39X9qq5Nn1jFt/wBnrpNJMVFxhzWbX/pGD3NtyPV/wVvpU+l6P6T+e9MuLvPXcveAH/YsPcBxPqdQmElOkksofWHAorDupX14rrcq/Fp3GA41PsaJJ+j+iq32Pd+jRR9YegO+j1PEPwvr/wDJpKdBJVq+o9Pt/msqmz+rY0/9S5WAQRIMg9wkpdJJJJSklWyss49+HUG7vtdxpJmNsU35O7j3f0fYrKSlKlfn3V9VxcBmOX1ZNV1tmRvaBX6Rpa1vpO99nqev+Z/N/wDnsWbk9aqyXDDxGZFLawWhzxWXP3D1P0v6TY1lbvZ+i/SfpUOp2Q/quA7JaG2/ZswOAEaC3DDHbZdt3s2vSU6ySpZ2RbVl9OrrdtbfkOZaIB3MFGTbt1+j+mrqd7VdSUpJJJJSkkkklKSVTqmW/C6dkZNTQ+6th9Fjph1p9lFen+kucxitCYE6nukp/9D1VZ1H/igzf/CeJ/586gtFZ1H/AIoM3/wnif8AnzqCSlulgNyupUH/AAeVvYDzFtNF27/t597VoOYx4h7Q4eBErPefs3Xq3nSvqNHoz29XHL76mD+Xdj5GW/8AqYa0klNWzpXTLf53DofPO6th/K1AP1d+r556ZiH/AKxX/wCQWikkpzP+bfQvzMOuv/i5r/8APRYq3UMLpHT2M/pb77zsxsWjKyBZa/nZWwZNbPb9Ky2zZRQz9LdZXWr+f1FuM9mNSz7Rn3gnHxgdsgfTuvsh/wBnxKt36bI2f8FTXkZVtGNdHp3S/s1j8zKs+1dRvG27II2gN+l9nxapf9nxWf6Lfvf/ADuRbdd+lSU5vT8HOxc/DxczJOQ4uyc5jXOdZ6LdtGK3CZk3/p8qup2Zdsyb/Tst/wBFV/NroVmYJ+09Wz8wa107MKo9ia91+TYw/wDHZH2az/hMNaaSlLPv/wCXsL/wtlf9XhLQWff/AMvYX/hbK/6vCSUrqH/KPS/+Ps/88XrQWf1LTO6SfHKe35HGy3f98WgkpSSSSSlJJJJKcXqhuyetYGBA+ztIyLC153E1H1mb6dm302XVY/6T1f8ADfzS2lh9HnL6vn57mWV7IoFdrpLHHabdle6xtVd+LT03J2s/0u/+d3rcSU//0fVVnUf+KDN/8J4n/nzqC0VnUf8Aigzf/CeJ/wCfOoJKS9Vw7MzELaHBmVS5t2LY6drbazvr37ff6Nn8xkbP+09tzFPp2dXn4deVW0s3y19T/pV2MJrvosj/AAlFzH1WKysrMxczDzHdT6az1xbH27BBDTbtAYzJxnPLa2Z1dbW1fpHMryqGV022V+jRYkp1VQz+oXMs+w9PY2/qD2h0On06WGWjKzC33en7XehQ39PmWM/RfoqsrKxc/qn1itdhb+iU3Zljniu2yqoudjlzm1uFuLb6Nn2uvfv+zX+j6DP02X6X6Ou9+n9DzvQNeRc7DoscbLqsd5ORc90b7s/qkV2+o9rWezBrxfs+z7PTk2YtddaSmdFuB0myyhjrep9Yv2vyhUA+95j9H6+tePg4zWud9lZfbjYv+h/S/Th1C7rLKq78ixtN1lja8HpuMS71Lj76m5uY5rLbMev0/tOSzEpx/Sxasn1X5lKu2XdK6HjMx6am1mwu+z4eO0G21/L/AEaW+61/599zv5v+fyra6/0qj0/Cyn5J6p1IAZbmmujHadzMepxDnVB/+FybnMY7Ku+h7K6av0dPq3JTa6fhNwcKrFa42Gse+130rLHE2X32f8Jfc591n8t6spJJKUs+/wD5ewv/AAtlf9XhLQWff/y9hf8AhbK/6vCSUx64fSx6MuY+yZNNrj2DHO+zZD3fya8bIusWkh5FFOTRZj3tFlNzHV2MPDmuGx7T/WaVl43U39MaMLrb/T9P2UdSfpVe0fQORd/N4ub/AKau70q8i334W/3046U7CSYEOAc0yDqCOCELMy8fCxbcvJdsppaXPdyYH5rWj3Pe76LGN+m9JTJ2RQ25lDrGNusBLKi4Bzg36RYz6TtqrdZyBj9LybCHn9G4fo7G1PAIhz6rrXVsrfWz9J9JVcPomPkYb7Oq47LM3NIuyi6C6t3+AopuZ7q/2ez9Dj3Y7q/0jPtf9IussQb/AKtW5d+KM7Ndl4OE71K8e2tpe9wLH1/acidlvo+nsrsbj1ZHoPtqvuv9a71Epu9CxXY3Tag9pZbbN1jXfSaXncyp8/Sdj1enj/8AWloJJJKf/9L1VY1vUcDB6/lHNyKsYWYmL6fqvDN22zP37N8b9m9m9bKSSnN/5x/V/wD8ssX/ALeZ/wCSS/5x/V//AMssX/t5n/klpJJKcLLzfqhmWi+3Nxm5DRtGTVkCm4N59P7Tj2VX+l/wXqemgfaegH2u+sT3V/ufa6xp/wAaxrMj/wAGXSKrZntZ1CvB9C9zrWl4vbWTS2N3tsv+ix/s+gkpzcLP+qGC578bMxW22QLb3XtstfH0fWybrLL7dv5vqWK1/wA4/q//AOWWL/28z/yS0kklOb/zj+r/AP5ZYv8A28z/AMkpU/WDomRl1YVGdRbk3h7qq2PDi4Vwbdu327mtdu2fuLQTbRIdA3AEA94PP5ElLrJ6lm4eF1nBtzL68as4+UwPue2tpcXYbtgdYWt37Wu9q1kxAPIlJTn/APOL6v8A/lnh/wDb9f8A5NI/WH6vEEHqeGQdCDfV/wCTV/Yz90fclsZ+6PuSU8/P1HDi6nLw8Unk4mUMaf632O6jcnrt+p1drLXdSpvdU4PrGRnuyGtePo2sqy8m6ttrPzLdnqMW/sZ+6PuVXNzsXCfQy2t7jkvFVfp1l43GA31HMG2rn6dnsSUi/wCcX1f/APLPD/7fr/8AJpf84vq//wCWeH/2/X/5NX9jP3R9yWxn7o+5JTQ/5xfV/wD8s8P/ALfr/wDJpD6xdAdZVU3qWK6y94qqY26txc907K27XfTft9iv7Gfuj7kzqKXlhfW1xrdvYSAdroLd7P3X7Xuakp//0/VUkkySnEyvrd0nDsfVlOLLGZD8bY0tc+WMZcbTXu9T0/0tTf8Artf+DTt+teF9srwn42TXfZezG2uY07bHMbe5rzXbZ/MV2VPu/kW+tX6lHqWs1B/Pv/m+W/1/ou+n/wB8/kLI6Rt/anUtv7On1Wbvs27159TK/pe/27/3PS/7W/tJJSz/AK59JqO20PDw3Ic/btLWDHsdjbbbXPZXXZfaz062f6Syqu70vVq3ufrh080NvZTcabGufXc4NawhrWPBd73XNa+21uN/M/z3/A/pVqfnP/mPpP8Aj9EfT/l/6X/g1ZSU4+J9aMHLzKMOuq5tuQXlhc1gaGMrryfXcW2O/RW1X07P8Kz1a/Xrp3qlhfXvp2Tj477KLK8jJaHjHDmbmsc+mmuy313YzqmP+0eoz1q6vUrot9H1a/sr8m3n7f8AnJ06fsG6DHrbvtf0Mifsm39H/wAX6n+B/aHprWf9F30Ponnjhv0v5H7ySmt0jq+L1fD+2YrbG0lzmNNrSwnb+e0H8x3/AKjs9O31K1dUKvou+j9J30OOT9L+X/pP5amkpSSSSSnEzvrTTgdRvwrsd7/SY17PScx1jxssyLrG0ksYyqiql271L/tL/wDB42x9Nlobvrr09tgoprdZeHMbdU5zWmsv9Vr63uYbm+vRZT+kr/4Wu2my2tbrf56z6HDePp/nfzn/AHxT/OPHA+PflJTjP+tvS6rhTdua/wBXJqeWlrm1jE/nrsp279AxzHV2e/8Awdnq/wAx6lqFV9dOm3UtuppvdW7b7y1ga0ubjuPqfpHP/RPz8Smz02WfpLf0Xq112rY/w5/mvpj+v/Nu5/4T/wBEIzeBxx24+SSnGx/rZg5GRj4zab/VyrDXVozaQ02NfeHep/MtfjZP5v2j9B6voej71UZ9e+nCtxyKX127rhXSHN3OZTUMr1/07sf06n1+31n/AKr7P6R6H6dWuvbf2j0qfsG71fb9t3ep/O439C2ez1v3PV/7XfYFsv4/N+ifpf6/QSU1emdWxuptyHY7XhuNe/Gc57doc6s7bHVfv17varigz6Vn0fpfm8/Rb/Ofy/8Avimkp//Z/+0j9lBob3Rvc2hvcCAzLjAAOEJJTQQEAAAAAAAPHAFaAAMbJUccAgAAAgAAADhCSU0EJQAAAAAAEM3P+n2ox74JBXB2rq8Fw044QklNBDoAAAAAAOUAAAAQAAAAAQAAAAAAC3ByaW50T3V0cHV0AAAABQAAAABQc3RTYm9vbAEAAAAASW50ZWVudW0AAAAASW50ZQAAAABDbHJtAAAAD3ByaW50U2l4dGVlbkJpdGJvb2wAAAAAC3ByaW50ZXJOYW1lVEVYVAAAAAEAAAAAAA9wcmludFByb29mU2V0dXBPYmpjAAAADABQAHIAbwBvAGYAIABTAGUAdAB1AHAAAAAAAApwcm9vZlNldHVwAAAAAQAAAABCbHRuZW51bQAAAAxidWlsdGluUHJvb2YAAAAJcHJvb2ZDTVlLADhCSU0EOwAAAAACLQAAABAAAAABAAAAAAAScHJpbnRPdXRwdXRPcHRpb25zAAAAFwAAAABDcHRuYm9vbAAAAAAAQ2xicmJvb2wAAAAAAFJnc01ib29sAAAAAABDcm5DYm9vbAAAAAAAQ250Q2Jvb2wAAAAAAExibHNib29sAAAAAABOZ3R2Ym9vbAAAAAAARW1sRGJvb2wAAAAAAEludHJib29sAAAAAABCY2tnT2JqYwAAAAEAAAAAAABSR0JDAAAAAwAAAABSZCAgZG91YkBv4AAAAAAAAAAAAEdybiBkb3ViQG/gAAAAAAAAAAAAQmwgIGRvdWJAb+AAAAAAAAAAAABCcmRUVW50RiNSbHQAAAAAAAAAAAAAAABCbGQgVW50RiNSbHQAAAAAAAAAAAAAAABSc2x0VW50RiNQeGxAUgAAAAAAAAAAAAp2ZWN0b3JEYXRhYm9vbAEAAAAAUGdQc2VudW0AAAAAUGdQcwAAAABQZ1BDAAAAAExlZnRVbnRGI1JsdAAAAAAAAAAAAAAAAFRvcCBVbnRGI1JsdAAAAAAAAAAAAAAAAFNjbCBVbnRGI1ByY0BZAAAAAAAAAAAAEGNyb3BXaGVuUHJpbnRpbmdib29sAAAAAA5jcm9wUmVjdEJvdHRvbWxvbmcAAAAAAAAADGNyb3BSZWN0TGVmdGxvbmcAAAAAAAAADWNyb3BSZWN0UmlnaHRsb25nAAAAAAAAAAtjcm9wUmVjdFRvcGxvbmcAAAAAADhCSU0D7QAAAAAAEABIAAAAAQABAEgAAAABAAE4QklNBCYAAAAAAA4AAAAAAAAAAAAAP4AAADhCSU0EDQAAAAAABAAAAB44QklNBBkAAAAAAAQAAAAeOEJJTQPzAAAAAAAJAAAAAAAAAAABADhCSU0nEAAAAAAACgABAAAAAAAAAAE4QklNA/UAAAAAAEgAL2ZmAAEAbGZmAAYAAAAAAAEAL2ZmAAEAoZmaAAYAAAAAAAEAMgAAAAEAWgAAAAYAAAAAAAEANQAAAAEALQAAAAYAAAAAAAE4QklNA/gAAAAAAHAAAP////////////////////////////8D6AAAAAD/////////////////////////////A+gAAAAA/////////////////////////////wPoAAAAAP////////////////////////////8D6AAAOEJJTQQAAAAAAAACAAE4QklNBAIAAAAAAAYAAAAAAAA4QklNBDAAAAAAAAMBAQEAOEJJTQQtAAAAAAAGAAEAAAAEOEJJTQQIAAAAAAAVAAAAAQAAAkAAAAJAAAAAAQAAIAgBADhCSU0EHgAAAAAABAAAAAA4QklNBBoAAAAAA3EAAAAGAAAAAAAAAAAAAAEaAAACWAAAAB4AbABlAGEAcgBuAGkAbgBnAC0AcgBhAHQAZQAtAGcAcgBhAGQAaQBlAG4AdAAtAGQAZQBzAGMAZQBuAHQAAAABAAAAAAAAAAAAAAAAAAAAAAAAAAEAAAAAAAAAAAAAAlgAAAEaAAAAAAAAAAAAAAAAAAAAAAEAAAAAAAAAAAAAAAAAAAAAAAAAEAAAAAEAAAAAAABudWxsAAAAAgAAAAZib3VuZHNPYmpjAAAAAQAAAAAAAFJjdDEAAAAEAAAAAFRvcCBsb25nAAAAAAAAAABMZWZ0bG9uZwAAAAAAAAAAQnRvbWxvbmcAAAEaAAAAAFJnaHRsb25nAAACWAAAAAZzbGljZXNWbExzAAAAAU9iamMAAAABAAAAAAAFc2xpY2UAAAASAAAAB3NsaWNlSURsb25nAAAAAAAAAAdncm91cElEbG9uZwAAAAAAAAAGb3JpZ2luZW51bQAAAAxFU2xpY2VPcmlnaW4AAAANYXV0b0dlbmVyYXRlZAAAAABUeXBlZW51bQAAAApFU2xpY2VUeXBlAAAAAEltZyAAAAAGYm91bmRzT2JqYwAAAAEAAAAAAABSY3QxAAAABAAAAABUb3AgbG9uZwAAAAAAAAAATGVmdGxvbmcAAAAAAAAAAEJ0b21sb25nAAABGgAAAABSZ2h0bG9uZwAAAlgAAAADdXJsVEVYVAAAAAEAAAAAAABudWxsVEVYVAAAAAEAAAAAAABNc2dlVEVYVAAAAAEAAAAAAAZhbHRUYWdURVhUAAAAAQAAAAAADmNlbGxUZXh0SXNIVE1MYm9vbAEAAAAIY2VsbFRleHRURVhUAAAAAQAAAAAACWhvcnpBbGlnbmVudW0AAAAPRVNsaWNlSG9yekFsaWduAAAAB2RlZmF1bHQAAAAJdmVydEFsaWduZW51bQAAAA9FU2xpY2VWZXJ0QWxpZ24AAAAHZGVmYXVsdAAAAAtiZ0NvbG9yVHlwZWVudW0AAAARRVNsaWNlQkdDb2xvclR5cGUAAAAATm9uZQAAAAl0b3BPdXRzZXRsb25nAAAAAAAAAApsZWZ0T3V0c2V0bG9uZwAAAAAAAAAMYm90dG9tT3V0c2V0bG9uZwAAAAAAAAALcmlnaHRPdXRzZXRsb25nAAAAAAA4QklNBCgAAAAAAAwAAAACP/AAAAAAAAA4QklNBBEAAAAAAAEBADhCSU0EFAAAAAAABAAAAAY4QklNBAwAAAAAGoUAAAABAAAAoAAAAEsAAAHgAACMoAAAGmkAGAAB/9j/4gxYSUNDX1BST0ZJTEUAAQEAAAxITGlubwIQAABtbnRyUkdCIFhZWiAHzgACAAkABgAxAABhY3NwTVNGVAAAAABJRUMgc1JHQgAAAAAAAAAAAAAAAQAA9tYAAQAAAADTLUhQICAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAABFjcHJ0AAABUAAAADNkZXNjAAABhAAAAGx3dHB0AAAB8AAAABRia3B0AAACBAAAABRyWFlaAAACGAAAABRnWFlaAAACLAAAABRiWFlaAAACQAAAABRkbW5kAAACVAAAAHBkbWRkAAACxAAAAIh2dWVkAAADTAAAAIZ2aWV3AAAD1AAAACRsdW1pAAAD+AAAABRtZWFzAAAEDAAAACR0ZWNoAAAEMAAAAAxyVFJDAAAEPAAACAxnVFJDAAAEPAAACAxiVFJDAAAEPAAACAx0ZXh0AAAAAENvcHlyaWdodCAoYykgMTk5OCBIZXdsZXR0LVBhY2thcmQgQ29tcGFueQAAZGVzYwAAAAAAAAASc1JHQiBJRUM2MTk2Ni0yLjEAAAAAAAAAAAAAABJzUkdCIElFQzYxOTY2LTIuMQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAWFlaIAAAAAAAAPNRAAEAAAABFsxYWVogAAAAAAAAAAAAAAAAAAAAAFhZWiAAAAAAAABvogAAOPUAAAOQWFlaIAAAAAAAAGKZAAC3hQAAGNpYWVogAAAAAAAAJKAAAA+EAAC2z2Rlc2MAAAAAAAAAFklFQyBodHRwOi8vd3d3LmllYy5jaAAAAAAAAAAAAAAAFklFQyBodHRwOi8vd3d3LmllYy5jaAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAABkZXNjAAAAAAAAAC5JRUMgNjE5NjYtMi4xIERlZmF1bHQgUkdCIGNvbG91ciBzcGFjZSAtIHNSR0IAAAAAAAAAAAAAAC5JRUMgNjE5NjYtMi4xIERlZmF1bHQgUkdCIGNvbG91ciBzcGFjZSAtIHNSR0IAAAAAAAAAAAAAAAAAAAAAAAAAAAAAZGVzYwAAAAAAAAAsUmVmZXJlbmNlIFZpZXdpbmcgQ29uZGl0aW9uIGluIElFQzYxOTY2LTIuMQAAAAAAAAAAAAAALFJlZmVyZW5jZSBWaWV3aW5nIENvbmRpdGlvbiBpbiBJRUM2MTk2Ni0yLjEAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAHZpZXcAAAAAABOk/gAUXy4AEM8UAAPtzAAEEwsAA1yeAAAAAVhZWiAAAAAAAEwJVgBQAAAAVx/nbWVhcwAAAAAAAAABAAAAAAAAAAAAAAAAAAAAAAAAAo8AAAACc2lnIAAAAABDUlQgY3VydgAAAAAAAAQAAAAABQAKAA8AFAAZAB4AIwAoAC0AMgA3ADsAQABFAEoATwBUAFkAXgBjAGgAbQByAHcAfACBAIYAiwCQAJUAmgCfAKQAqQCuALIAtwC8AMEAxgDLANAA1QDbAOAA5QDrAPAA9gD7AQEBBwENARMBGQEfASUBKwEyATgBPgFFAUwBUgFZAWABZwFuAXUBfAGDAYsBkgGaAaEBqQGxAbkBwQHJAdEB2QHhAekB8gH6AgMCDAIUAh0CJgIvAjgCQQJLAlQCXQJnAnECegKEAo4CmAKiAqwCtgLBAssC1QLgAusC9QMAAwsDFgMhAy0DOANDA08DWgNmA3IDfgOKA5YDogOuA7oDxwPTA+AD7AP5BAYEEwQgBC0EOwRIBFUEYwRxBH4EjASaBKgEtgTEBNME4QTwBP4FDQUcBSsFOgVJBVgFZwV3BYYFlgWmBbUFxQXVBeUF9gYGBhYGJwY3BkgGWQZqBnsGjAadBq8GwAbRBuMG9QcHBxkHKwc9B08HYQd0B4YHmQesB78H0gflB/gICwgfCDIIRghaCG4IggiWCKoIvgjSCOcI+wkQCSUJOglPCWQJeQmPCaQJugnPCeUJ+woRCicKPQpUCmoKgQqYCq4KxQrcCvMLCwsiCzkLUQtpC4ALmAuwC8gL4Qv5DBIMKgxDDFwMdQyODKcMwAzZDPMNDQ0mDUANWg10DY4NqQ3DDd4N+A4TDi4OSQ5kDn8Omw62DtIO7g8JDyUPQQ9eD3oPlg+zD88P7BAJECYQQxBhEH4QmxC5ENcQ9RETETERTxFtEYwRqhHJEegSBxImEkUSZBKEEqMSwxLjEwMTIxNDE2MTgxOkE8UT5RQGFCcUSRRqFIsUrRTOFPAVEhU0FVYVeBWbFb0V4BYDFiYWSRZsFo8WshbWFvoXHRdBF2UXiReuF9IX9xgbGEAYZRiKGK8Y1Rj6GSAZRRlrGZEZtxndGgQaKhpRGncanhrFGuwbFBs7G2MbihuyG9ocAhwqHFIcexyjHMwc9R0eHUcdcB2ZHcMd7B4WHkAeah6UHr4e6R8THz4faR+UH78f6iAVIEEgbCCYIMQg8CEcIUghdSGhIc4h+yInIlUigiKvIt0jCiM4I2YjlCPCI/AkHyRNJHwkqyTaJQklOCVoJZclxyX3JicmVyaHJrcm6CcYJ0kneierJ9woDSg/KHEooijUKQYpOClrKZ0p0CoCKjUqaCqbKs8rAis2K2krnSvRLAUsOSxuLKIs1y0MLUEtdi2rLeEuFi5MLoIuty7uLyQvWi+RL8cv/jA1MGwwpDDbMRIxSjGCMbox8jIqMmMymzLUMw0zRjN/M7gz8TQrNGU0njTYNRM1TTWHNcI1/TY3NnI2rjbpNyQ3YDecN9c4FDhQOIw4yDkFOUI5fzm8Ofk6Njp0OrI67zstO2s7qjvoPCc8ZTykPOM9Ij1hPaE94D4gPmA+oD7gPyE/YT+iP+JAI0BkQKZA50EpQWpBrEHuQjBCckK1QvdDOkN9Q8BEA0RHRIpEzkUSRVVFmkXeRiJGZ0arRvBHNUd7R8BIBUhLSJFI10kdSWNJqUnwSjdKfUrESwxLU0uaS+JMKkxyTLpNAk1KTZNN3E4lTm5Ot08AT0lPk0/dUCdQcVC7UQZRUFGbUeZSMVJ8UsdTE1NfU6pT9lRCVI9U21UoVXVVwlYPVlxWqVb3V0RXklfgWC9YfVjLWRpZaVm4WgdaVlqmWvVbRVuVW+VcNVyGXNZdJ114XcleGl5sXr1fD19hX7NgBWBXYKpg/GFPYaJh9WJJYpxi8GNDY5dj62RAZJRk6WU9ZZJl52Y9ZpJm6Gc9Z5Nn6Wg/aJZo7GlDaZpp8WpIap9q92tPa6dr/2xXbK9tCG1gbbluEm5rbsRvHm94b9FwK3CGcOBxOnGVcfByS3KmcwFzXXO4dBR0cHTMdSh1hXXhdj52m3b4d1Z3s3gReG54zHkqeYl553pGeqV7BHtje8J8IXyBfOF9QX2hfgF+Yn7CfyN/hH/lgEeAqIEKgWuBzYIwgpKC9INXg7qEHYSAhOOFR4Wrhg6GcobXhzuHn4gEiGmIzokziZmJ/opkisqLMIuWi/yMY4zKjTGNmI3/jmaOzo82j56QBpBukNaRP5GokhGSepLjk02TtpQglIqU9JVflcmWNJaflwqXdZfgmEyYuJkkmZCZ/JpomtWbQpuvnByciZz3nWSd0p5Anq6fHZ+Ln/qgaaDYoUehtqImopajBqN2o+akVqTHpTilqaYapoum/adup+CoUqjEqTepqaocqo+rAqt1q+msXKzQrUStuK4trqGvFq+LsACwdbDqsWCx1rJLssKzOLOutCW0nLUTtYq2AbZ5tvC3aLfguFm40blKucK6O7q1uy67p7whvJu9Fb2Pvgq+hL7/v3q/9cBwwOzBZ8Hjwl/C28NYw9TEUcTOxUvFyMZGxsPHQce/yD3IvMk6ybnKOMq3yzbLtsw1zLXNNc21zjbOts83z7jQOdC60TzRvtI/0sHTRNPG1EnUy9VO1dHWVdbY11zX4Nhk2OjZbNnx2nba+9uA3AXcit0Q3ZbeHN6i3ynfr+A24L3hROHM4lPi2+Nj4+vkc+T85YTmDeaW5x/nqegy6LzpRunQ6lvq5etw6/vshu0R7ZzuKO6070DvzPBY8OXxcvH/8ozzGfOn9DT0wvVQ9d72bfb794r4Gfio+Tj5x/pX+uf7d/wH/Jj9Kf26/kv+3P9t////7QAMQWRvYmVfQ00AAv/uAA5BZG9iZQBkgAAAAAH/2wCEAAwICAgJCAwJCQwRCwoLERUPDAwPFRgTExUTExgRDAwMDAwMEQwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwBDQsLDQ4NEA4OEBQODg4UFA4ODg4UEQwMDAwMEREMDAwMDAwRDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDP/AABEIAEsAoAMBIgACEQEDEQH/3QAEAAr/xAE/AAABBQEBAQEBAQAAAAAAAAADAAECBAUGBwgJCgsBAAEFAQEBAQEBAAAAAAAAAAEAAgMEBQYHCAkKCxAAAQQBAwIEAgUHBggFAwwzAQACEQMEIRIxBUFRYRMicYEyBhSRobFCIyQVUsFiMzRygtFDByWSU/Dh8WNzNRaisoMmRJNUZEXCo3Q2F9JV4mXys4TD03Xj80YnlKSFtJXE1OT0pbXF1eX1VmZ2hpamtsbW5vY3R1dnd4eXp7fH1+f3EQACAgECBAQDBAUGBwcGBTUBAAIRAyExEgRBUWFxIhMFMoGRFKGxQiPBUtHwMyRi4XKCkkNTFWNzNPElBhaisoMHJjXC0kSTVKMXZEVVNnRl4vKzhMPTdePzRpSkhbSVxNTk9KW1xdXl9VZmdoaWprbG1ub2JzdHV2d3h5ent8f/2gAMAwEAAhEDEQA/APVVSpzch/WMnBdU1tFFFNzLg6XONrr2Oa5n5rf1f2qrk2fWMW3/AGeuk0kxUXGHNZtf+kYPc23I9X/BW+lT6Xo/pP570y4u89dy94Af9iw9wHE+p1CYSU6SSyh9YcCisO6lfXiutyr8WncYDjU+xokn6P6KrfY936NFH1h6A76PU8Q/C+v/AMmkp0ElWr6j0+3+ayqbP6tjT/1LlYBBEgyD3CSl0kkklKSVbKyzj34dQbu+13GkmY2xTfk7uPd/R9ispKUqV+fdX1XFwGY5fVk1XW2ZG9oFfpGlrW+k732ep6/5n83/AOexZuT1qrJcMPEZkUtrBaHPFZc/cPU/S/pNjWVu9n6L9J+lQ6nZD+q4Dslobb9mzA4ARoLcMMdtl23eza9JTrJKlnZFtWX06ut21t+Q5logHcwUZNu3X6P6aup3tV1JSkkkklKSSSSUpJVOqZb8Lp2Rk1ND7q2H0WOmHWn2UV6f6S5zGK0JgTqe6Sn/0PVVnUf+KDN/8J4n/nzqC0VnUf8Aigzf/CeJ/wCfOoJKW6WA3K6lQf8AB5W9gPMW00Xbv+3n3tWg5jHiHtDh4ESs95+zderedK+o0ejPb1ccvvqYP5d2PkZb/wCphrSSU1bOldMt/ncOh887q2H8rUA/V36vnnpmIf8ArFf/AJBaKSSnM/5t9C/Mw66/+Lmv/wA9FirdQwukdPYz+lvvvOzGxaMrIFlr+dlbBk1s9v0rLbNlFDP0t1ldav5/UW4z2Y1LPtGfeCcfGB2yB9O6+yH/AGfEq3fpsjZ/wVNeRlW0Y10endL+zWPzMqz7V1G8bbsgjaA36X2fFql/2fFZ/ot+9/8AO5Ft136VJTm9Pwc7Fz8PFzMk5Di7JzmNc51not20YrcJmTf+nyq6nZl2zJv9Oy3/AEVX82uhWZgn7T1bPzBrXTswqj2Jr3X5NjD/AMdkfZrP+Ew1ppKUs+//AJewv/C2V/1eEtBZ9/8Ay9hf+Fsr/q8JJSuof8o9L/4+z/zxetBZ/UtM7pJ8cp7fkcbLd/3xaCSlJJJJKUkkkkpxeqG7J61gYED7O0jIsLXncTUfWZvp2bfTZdVj/pPV/wAN/NLaWH0ecvq+fnuZZXsigV2ukscdpt2V7rG1V34tPTcnaz/S7/53etxJT//R9VWdR/4oM3/wnif+fOoLRWdR/wCKDN/8J4n/AJ86gkpL1XDszMQtocGZVLm3Ytjp2ttrO+vft9/o2fzGRs/7T23MU+nZ1efh15VbSzfLX1P+lXYwmu+iyP8ACUXMfVYrKyszFzMPMd1PprPXFsfbsEENNu0BjMnGc8trZnV1tbV+kcyvKoZXTbZX6NFiSnVVDP6hcyz7D09jb+oPaHQ6fTpYZaMrMLfd6ftd6FDf0+ZYz9F+iqysrFz+qfWK12Fv6JTdmWOeK7bKqi52OXObW4W4tvo2fa69+/7Nf6PoM/TZfpfo6736f0PO9A15FzsOixxsuqx3k5Fz3Rvuz+qRXb6j2tZ7MGvF+z7Ps9OTZi111pKZ0W4HSbLKGOt6n1i/a/KFQD73mP0fr614+DjNa532Vl9uNi/6H9L9OHULussqrvyLG03WWNrwem4xLvUuPvqbm5jmstsx6/T+05LMSnH9LFqyfVfmUq7Zd0roeMzHpqbWbC77Ph47QbbX8v8ARpb7rX/n33O/m/5/Ktrr/SqPT8LKfknqnUgBluaa6Mdp3Mx6nEOdUH/4XJucxjsq76Hsrpq/R0+rclNrp+E3BwqsVrjYax77XfSsscTZffZ/wl9zn3Wfy3qykkkpSz7/APl7C/8AC2V/1eEtBZ9//L2F/wCFsr/q8JJTHrh9LHoy5j7Jk02uPYMc77NkPd/Jrxsi6xaSHkUU5NFmPe0WU3MdXYw8Oa4bHtP9ZpWXjdTf0xowutv9P0/ZR1J+lV7R9A5F383i5v8Apq7vSryLffhb/fTjpTsJJgQ4BzTIOoI4IQszLx8LFty8l2ymlpc93JgfmtaPc97vosY36b0lMnZFDbmUOsY26wEsqLgHODfpFjPpO2qt1nIGP0vJsIef0bh+jsbU8AiHPqutdWyt9bP0n0lVw+iY+Rhvs6rjsszc0i7KLoLq3f4Cim5nur/Z7P0OPdjur/SM+1/0i6yxBv8Aq1bl34ozs12Xg4TvUrx7a2l73AsfX9pyJ2W+j6eyuxuPVkeg+2q+6/1rvUSm70LFdjdNqD2llts3WNd9JpedzKnz9J2PV6eP/wBaWgkkkp//0vVVjW9RwMHr+Uc3IqxhZiYvp+q8M3bbM/fs3xv2b2b1spJKc3/nH9X/APyyxf8At5n/AJJL/nH9X/8Ayyxf+3mf+SWkkkpwsvN+qGZaL7c3GbkNG0ZNWQKbg3n0/tOPZVf6X/Bep6aB9p6Afa76xPdX+59rrGn/ABrGsyP/AAZdIqtme1nUK8H0L3OtaXi9tZNLY3e2y/6LH+z6CSnNws/6oYLnvxszFbbZAtvde2y18fR9bJussvt2/m+pYrX/ADj+r/8A5ZYv/bzP/JLSSSU5v/OP6v8A/lli/wDbzP8AySlT9YOiZGXVhUZ1FuTeHuqrY8OLhXBt27fbua127Z+4tBNtEh0DcAQD3g8/kSUusnqWbh4XWcG3Mvrxqzj5TA+57a2lxdhu2B1ha3fta72rWTEA8iUlOf8A84vq/wD+WeH/ANv1/wDk0j9Yfq8QQep4ZB0IN9X/AJNX9jP3R9yWxn7o+5JTz8/UcOLqcvDxSeTiZQxp/rfY7qNyeu36nV2std1Km91Tg+sZGe7Ia14+jayrLybq22s/Mt2eoxb+xn7o+5Vc3OxcJ9DLa3uOS8VV+nWXjcYDfUcwbaufp2exJSL/AJxfV/8A8s8P/t+v/wAml/zi+r//AJZ4f/b9f/k1f2M/dH3JbGfuj7klND/nF9X/APyzw/8At+v/AMmkPrF0B1lVTepYrrL3iqpjbq3Fz3Tsrbtd9N+32K/sZ+6PuTOopeWF9bXGt29hIB2ugt3s/dfte5qSn//T9VSSTJKcTK+t3ScOx9WU4ssZkPxtjS1z5YxlxtNe71PT/S1N/wCu1/4NO3614X2yvCfjZNd9l7Mba5jTtscxt7mvNdtn8xXZU+7+Rb61fqUepazUH8+/+b5b/X+i76f/AHz+QsjpG39qdS2/s6fVZu+zbvXn1Mr+l7/bv/c9L/tb+0klLP8Arn0mo7bQ8PDchz9u0tYMex2Ntttc9lddl9rPTrZ/pLKq7vS9Wre5+uHTzQ29lNxpsa59dzg1rCGtY8F3vdc1r7bW438z/Pf8D+lWp+c/+Y+k/wCP0R9P+X/pf+DVlJTj4n1owcvMow66rm25BeWFzWBoYyuvJ9dxbY79FbVfTs/wrPVr9euneqWF9e+nZOPjvsosryMloeMcOZuaxz6aa7LfXdjOqY/7R6jPWrq9Sui30fVr+yvybeft/wCcnTp+wboMetu+1/QyJ+ybf0f/ABfqf4H9oemtZ/0XfQ+ieeOG/S/kfvJKa3SOr4vV8P7ZitsbSXOY02tLCdv57QfzHf8AqOz07fUrV1Qq+i76P0nfQ45P0v5f+k/lqaSlJJJJKcTO+tNOB1G/Cux3v9JjXs9JzHWPGyzIusbSSxjKqKqXbvUv+0v/AMHjbH02Whu+uvT22Cimt1l4cxt1TnNaay/1Wvre5hub69FlP6Sv/ha7abLa1ut/nrPocN4+n+d/Of8AfFP848cD49+UlOM/629LquFN25r/AFcmp5aWubWMT+euynbv0DHMdXZ7/wDB2er/ADHqWoVX106bdS26mm91btvvLWBrS5uO4+p+kc/9E/PxKbPTZZ+kt/RerXXatj/Dn+a+mP6/827n/hP/AEQjN4HHHbj5JKcbH+tmDkZGPjNpv9XKsNdWjNpDTY194d6n8y1+Nk/m/aP0Hq+h6PvVRn176cK3HIpfXbuuFdIc3c5lNQyvX/Tux/TqfX7fWf8Aqvs/pHofp1a69t/aPSp+wbvV9v23d6n87jf0LZ7PW/c9X/td9gWy/j836J+l/r9BJTV6Z1bG6m3IdjteG4178Zznt2hzqztsdV+/Xu9quKDPpWfR+l+bz9Fv85/L/wC+KaSn/9kAOEJJTQQhAAAAAABVAAAAAQEAAAAPAEEAZABvAGIAZQAgAFAAaABvAHQAbwBzAGgAbwBwAAAAEwBBAGQAbwBiAGUAIABQAGgAbwB0AG8AcwBoAG8AcAAgAEMAUwA2AAAAAQA4QklNBAYAAAAAAAcAAgEBAAEBAP/hDLVodHRwOi8vbnMuYWRvYmUuY29tL3hhcC8xLjAvADw/eHBhY2tldCBiZWdpbj0i77u/IiBpZD0iVzVNME1wQ2VoaUh6cmVTek5UY3prYzlkIj8+IDx4OnhtcG1ldGEgeG1sbnM6eD0iYWRvYmU6bnM6bWV0YS8iIHg6eG1wdGs9IkFkb2JlIFhNUCBDb3JlIDUuMy1jMDExIDY2LjE0NTY2MSwgMjAxMi8wMi8wNi0xNDo1NjoyNyAgICAgICAgIj4gPHJkZjpSREYgeG1sbnM6cmRmPSJodHRwOi8vd3d3LnczLm9yZy8xOTk5LzAyLzIyLXJkZi1zeW50YXgtbnMjIj4gPHJkZjpEZXNjcmlwdGlvbiByZGY6YWJvdXQ9IiIgeG1sbnM6eG1wTU09Imh0dHA6Ly9ucy5hZG9iZS5jb20veGFwLzEuMC9tbS8iIHhtbG5zOnN0RXZ0PSJodHRwOi8vbnMuYWRvYmUuY29tL3hhcC8xLjAvc1R5cGUvUmVzb3VyY2VFdmVudCMiIHhtbG5zOmRjPSJodHRwOi8vcHVybC5vcmcvZGMvZWxlbWVudHMvMS4xLyIgeG1sbnM6cGhvdG9zaG9wPSJodHRwOi8vbnMuYWRvYmUuY29tL3Bob3Rvc2hvcC8xLjAvIiB4bWxuczp4bXA9Imh0dHA6Ly9ucy5hZG9iZS5jb20veGFwLzEuMC8iIHhtcE1NOkRvY3VtZW50SUQ9IjE3NERDQjYzRUUzRTIwMTlCMEY2QTY1MEQ0QkIxREIyIiB4bXBNTTpJbnN0YW5jZUlEPSJ4bXAuaWlkOkM0OEUyOTJCMDg3NkU4MTE4QTFCOTRDMDdERjdDRUU1IiB4bXBNTTpPcmlnaW5hbERvY3VtZW50SUQ9IjE3NERDQjYzRUUzRTIwMTlCMEY2QTY1MEQ0QkIxREIyIiBkYzpmb3JtYXQ9ImltYWdlL2pwZWciIHBob3Rvc2hvcDpDb2xvck1vZGU9IjMiIHhtcDpDcmVhdGVEYXRlPSIyMDE4LTA2LTIxVDE2OjM2OjU2KzA1OjQ1IiB4bXA6TW9kaWZ5RGF0ZT0iMjAxOC0wNi0yMlQxNjo0NjoxMyswNTo0NSIgeG1wOk1ldGFkYXRhRGF0ZT0iMjAxOC0wNi0yMlQxNjo0NjoxMyswNTo0NSI+IDx4bXBNTTpIaXN0b3J5PiA8cmRmOlNlcT4gPHJkZjpsaSBzdEV2dDphY3Rpb249InNhdmVkIiBzdEV2dDppbnN0YW5jZUlEPSJ4bXAuaWlkOkM0OEUyOTJCMDg3NkU4MTE4QTFCOTRDMDdERjdDRUU1IiBzdEV2dDp3aGVuPSIyMDE4LTA2LTIyVDE2OjQ2OjEzKzA1OjQ1IiBzdEV2dDpzb2Z0d2FyZUFnZW50PSJBZG9iZSBQaG90b3Nob3AgQ1M2IChXaW5kb3dzKSIgc3RFdnQ6Y2hhbmdlZD0iLyIvPiA8L3JkZjpTZXE+IDwveG1wTU06SGlzdG9yeT4gPC9yZGY6RGVzY3JpcHRpb24+IDwvcmRmOlJERj4gPC94OnhtcG1ldGE+ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgPD94cGFja2V0IGVuZD0idyI/Pv/uACFBZG9iZQBkgAAAAAEDABADAgMGAAAAAAAAAAAAAAAA/9sAhAAIBgYGBgYIBgYIDAgHCAwOCggICg4QDQ0ODQ0QEQwMDAwMDBEMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMAQkICAkKCQsJCQsOCw0LDhEODg4OEREMDAwMDBERDAwMDAwMEQwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAz/wgARCAEaAlgDASIAAhEBAxEB/8QAxgABAAIDAQEBAAAAAAAAAAAAAAYHAgQFAQMIAQEAAAAAAAAAAAAAAAAAAAAAEAABAgUBCAEEAwEBAAAAAAAGBAUAAQIDBxcQIDBAERMWN1AxEhQVYJAhJiMRAAEDAQQEBA4NCAcFCQAAAAIBAwQFABESBiEiMhMxQlIjIDBBUZFictIzk7MUNJQQQGHRorJD0xU1dTYHUHGxgpJTYyRggaHCc0QW8MGDoyXh4vLDVGR0VSYSAQAAAAAAAAAAAAAAAAAAAJD/2gAMAwEBAhEDEQAAAL/AAIySYAAAAAAAAAAAAAAAAAA5p0gAAAAAAAAAAAAAAAAAadGX/XhYXoAAAAAAAAAAAAAAAAAfL87fo6vSwfQAAAAAAAAAAAAAAA819CEFn69aZloV5YdeFhnh6iMuAAAAAAAAAAADgd8AAY/Gsi1fIFqlkV7YNfFhAcnrV4WGAAAAAAAAAAA1o6SsAHJ0ZII2kgV5YdeFhgr6cQXqm/36X7xZSthZKtMiyVaZlkK39LHV34WKr30sFCtwlLjbpuMMwCvbCrywwADmRybCFYTgK9sKvSwgK8sOvSwgAAAAAAAAADXIZMILYh6AAABXlh14WGCvbCrywwAAADzzIfPz6jX+e4OVpSIRDSngrjyyOWVFJPnMiOWFXdiAAAACvbCr0sICv7AgBPwAAAAAAAAAIrKqjOxYkFnQAAAAryw68LDBXtgxv6kgAAAAAAAAIybMS2p+NXbgplONTbAAAAFe2FXpYQEAn8AJ+AAAAAAAAADGopdxiwtgAAAAFeWHXhYYPK8sSuSxnnoAAAAAAeQg2tXYmB56CurFrosUAAAACvbCr0sICvLDrwsMAAAAAAAAA0yF/HTlRKAAAAAK8sOvCwwIXNPDmdStrJAAAAAHitDa7u12wAw4ppbkUsg9AAAAAr2wq9LCAryw68LDAAAAAAAAA4/Yrc5FvV3YoAAAAAryw68LDABAJns1sWYAAABh86xMrB++2PPIITuAac9IFJZPWR3Zj56AAAAAK9sKvSwgIBP+UdD6xCXgAAAAAAAHkE7tTlgzbHIAAAAAV5YdeFhgARaU+EWlVdTI6YAGlrwQ+VoZQgnEE5s8ILO+kHnsfOP3uBOQAAAAABXthV6WEACuLHrqwD6gAAAARbZ5Roz7410Wgjmma3PiFwnWAAAAAAryw68LDAAB5WNn4nP6VbSElHAxqgmG/XNzEInXWAAhp1Ix9p8PQAAAAAAV7YVelhAA+Fa2jyjqe1ZYJ0gAACPEWsmKSweeiHcGzxEJeAAAAAACvLDrwsMAAAHkSlwrCXd+JEq+taZlkK3FjxmNSUi8rkWQAAAAAAAAr2wq9LCAABhBZ6Kz27A1jQ7EQ4xZHtZCdwToToy9AAAAAAAAABXlh1QWui4lCLiUIuJQi4lCLiUIuJQi4lCLiUIuJQi4lCLiUIuJQi4lCLiUIvwixUfkAAAr2wq9LCYemTEZMRkxGTEZMRkxGTEZMRkxGTEZMRkxGTEZMeMdtoDfAA4/YEeSER5IRHkhEeSER5IRHkhEeSER5IRHkhEeSER5IRHkhEeSER5IRHuFPhwu6AACHTEVsskVsskVsskVsskVsskVsskVsskVsskVsskVsskVsskVsskVsskVsskVtx7hEDTwAAAAAAAAAAcDoQnrEvAAAAAAAAAAAAAAAAABq5V5sFhgAAAAAAAAAAAAAAAAA8egAAAAAAAAAAAAAAAAADx6AAAAAAAAAAAPAQGV1r4THlxqRml3KYl5ZfGz4ZO+FFcSUfWopoSHY1PDn70AnJ0OzUloEH6/E+JNOZEZMdOZ0TdB2wAAAAAAUhOYbtnT72jCCcQ3l9osLid2uifYwDaJD7V9jm9rfeMnd6ES7RrWR+bP0mUNZ1Z9cmXM4uuSOx/zz+hgAAD0AAHgAGvsCuZ9sAAAAACD9zuAAAAAAAAAAAAAAAAAB8fsK6nmyAAAAAAP/9oACAECAAEFAP6rP//aAAgBAwABBQD+qz//2gAIAQEAAQUA/iDMcMr4SfC2n1rvPfwbraW320KAFCA7lLp8IpquUWGcdNEuRpdPhA/2V8J0lOGz3Hz9VUqZXF6OzeipcloWQH+yvgJxOrps+6XWGz3HtuETfaIubK/9FhmVawiRZEebimt1Xr7MB/sqJ/QUf3F4e+XVk1tKW7q+qqhCNLlj5fsZHqqsvr9fXtsvq2e49qmf3Zq5ZwW2m5Bjda8OYbtfh5ARorw7XYJ1AGMKbS4cuKneA/2VE/oBT+0xWLkjemGzBoLJ8q5z+7Mm6/N69zbvF62J2VY3bVKx1Fb1DHDZ7j20/wDpmvlsurbyYIa0VpsbuAH+yon9BSmaXJx2Btp02iuHK7SaeJabcSxwR2omF5AsxIey1YiSPMdqX5WarE5PmXrUvMMm2opyCY24nk10tSll1vpiWYxSUU5qx51tZbxzdi1kkBvTtForfi27NV6KLtq5t6xTTNTmre6S3Gz3HtRS+/M3Kr1yRsRFtdL3kCXBD/ZWxLV+Hmfg/bTOO3bidizOJpk84mhSTitqQVxWOMl2LwMKX4u4wBrsXMPgVyK8KhUaOsVuDQYWBqLDpg9FqAbqksytwmz3Hta/9zBLlTOf51YnL9zkjgh/srYTzqQZT5R+f2oca2xleciOAkLpRNsxhOpav4TZ7j2tPt7lWQ3UuJtiJDeShXBD/ZWzLMpo0MpynyZUVtQi3MIs7lDnTTKmlzW0NrbiJFWjAeE2e49rR7e5S5XRaoK3C4pxuhSWkCPgh/srZkFpm9hYO7SfRHkTAzbxRMMBzirc5SlTKMvLriUFa0VttbeE2e49rV/mYJcpkB/QIWl0T2HLI8uEH+ytlVMq6cV1/rJcgYG9serEAi4gVSl0lsMJyfcgS+nCbPce1NVO1mjlDQRYjIsxzTJ2IeEH+ytrnPxzKsvpxZz6QXGylGsDQhOO0bgDOT+U8Ns9x7Vcu1mjk3VyRs7dYK27xzHbLMfDOEH+ytuUGRQ7ig69piJi4nWUFBi5ODkIBrcJotzIhBWOCQYP2xcZ4bZ7j2ktckuVpcmUJETkwko22NCeUpS4Yf7K21SlVSBVeLkvDnPpD2UOxi5DIw1Czbtu3rVi2xF7ASqXSfmOS5S6cRs9x7ckT/EfOTdC9lej1h/6LKHDD/ZW5k1vWJbLU4pHhv4Ny5bt0Lnl2yYrZGNtH2+JzlKVdy3aocsnpb6myBPxXde17Tj8WxyPKmNh4jZ7j25is1+Ep79ClPyM5x/y7YoxW0X2wP4Yf7K3Llum7QIXawcol9N9YsTN6W7edssqG5uRtaWK65W5OuT0M1NsGIi6tsZWxoTdIvVSyCdy+nEbPce0maqXsdxg6VOoJyJqvc2sVp/euwlbopt0cMP9lbp2LeUM4MWeUNe86uzcyILCB2yorTp7KWzOummTxk9stq6QsoMamlia2RLE59IyASrUdAqOIhVj4rZ7j2zgEn+hMOIUHjGL3k+Wmi1fTqU6uzsejZukYBlMigzl9OGH+yt2cusGbI5sjsPvzaTNW4QkTULtjYwPB6500UWaHrJ7SlVyDy40m0MDSxpdpcVoBFpBRZekucZs9x7mSbV5iXJlFlXY4RmTWRIfBA2bKlVtqFbYvi5EAqRY0ZS6xkQ1kDj7xftu94ZYkg2x8QP9lb1VMqqXpndgF4Yn1qI2yJz6QVFjUJoB4TdiB0eslMrco8VMzapmG2dgS7Zz6QVFbUJtoqMO7w7Sl0lxmz3HuK0tlanHHK5jl2lPgzhTKRrkmUpSlE5SnImx40P9+6OZUlSJgqcfvcUP9lb9dFNcnkReRhxFDloKpF5LUMNRXZfRppAkrgcoWIXZB1L0ludYKj9GxqRkJW3XGVMpS47Z7j3X8fbCNtsuZVjWpkf2YiR9d80JrQmPY9Gro2O8mH+yuBOXWRSANBJVSRm4fUxFwqWWrVi1Znt6wQZAFBqdS/IJzULBTIKJuRbPce9ct0XKXjF7UpWydcpjM0+YRe3U3Fou7UyqlOXXY9PzQPIWZIvyERSl05QP9lcKu3RcpIMbixDXSDGzLH5uZEU/3+W5x38yOE6sdk7zA+BDI1TTRTRLkmz3HwJ0ynK+3I1FLjjEKcpzwyO2ZzxMppnpQurm04oG0Cyi3Rbp5QP9lfCtnuP4RnI2AfyTqQAxqQAxqQAxqQAxqQAxqQAxqQAxqQAxqQAxqQAxqQAxqQAxqQAxqQAxqQAxqQAxqQAxqQAxqQAxqQAxqQAxqQAxqQAxqQAxqQAxqQAxqQAxqQAxqQAwTZcHmhOLGY6YJd5tqppzH3bcd23Hdtx3bcd23Hdtx3bcd23Hdtx3bcd23Hdtx3bcd23Hdtx3bcd23Hdtx3bcd23Hdtx3bcd23Hdtx3bcd23Hdtx3bcd23BKSJRluZ3tpf0W6vFmFzU+DiseDiseDiseDiseDiseDiseDiseDiseDiseDiseDiseDiseDiseDiseDiseDiseDiseDiseDiseDiseDiseDiseDiseDiseDiseDiseDiseDiseDisFGMWh6Si4kyCKLeJsajJU5aIhMaIhMaIhMaIhMaIhMaIhMaIhMaIhMaIhMaIhMaIhMaIhMaIhMaIhMaIhMaIhMaIhMaIhMaIhMaIhMaIhMaIhMaIhMaIhMaIhMaIhMaIhMaIhMaIhMEmF2q03Y7xo2Adv+PqjIfRI7LuhvunwlTiitxJalne+DIAVY9FgCxvDSh+ENB5xVEmPmx3k4fB/bKOkvheko6S/kPWOsJXdzrzE/OyhmQjJ3Q/PIxldIXXlGZ7Npsecgza1giYIS9G6uzlZyqUk7aIMjLkOS58IchrR6siyYxjlC/J1pnbh3Idt5IGc5aB4Qt5XTSdHfI8kbsHk/l7KqUOr3lHGJY9uAwgyNXW/sWQiFyNXHKllOraHZE+NnWOsdY6x1jrHWOsdY6x1jrHWOsI760rNH42sAlVjL0r64aPEhMwIcj11P9xUp/Kx/cruA4W7ua46LDZKMKBg5TEc6cm1pnJ3ykjbSF0yTbbXqg7bSoLv5DaRJjHch2Xl+UZms2m1HfvKUiRe+qgxsLXjwhnyJJzuY7yG6EDEny6krl1jrHWOsdY6709r46qhbLhqRkBA24/bFNrKGG0FbPjlqVtNlqNGems3xu3saVIbr1TDkk4vLMnBwk3iy8jVpid3YpM666vyAhWKyZ2Qq68uqWR6qxVktvWqkpQt/FyFjpUOKBK8BoyrKllUeNWOG1uvKzBtXLRjKFkcSjzwIN6JrGuMfO7WVt8mYhtESBvv0ZaoSGLYOp2xQtKLjU6TVAdi+lCkBApDD88QqHx1bENlMyVtq5zmNtiq1lxC3qJZboa3P8C61OU1q9GppzM3LGm22olaZejaMXUvIUUv5O+tePWlZbOQdK5qsesrCwXWzhz3l6FM5oacYOdTOhRJm1FwpgDghcRQWbxFr+DWJLC9JTjFy/UN6FK1oeQn/AAb/2gAIAQICBj8AKz//2gAIAQMCBj8AKz//2gAIAQEBBj8A/ohV8sQnL5lHwb0r0wu36Hd11f5dzmne3/Iz+Xm5IlVIzASn4qX4hacXCJX8HWvHtw5f5Els015I05xowiyCHGjbhCqA4oaMWAta1RCmTiZn5bfgm/KNFPzpuQySzmzG/wCXvUgxbH5FdNkEcdECVtslwoRImqKlpwoS9Wz1VF4H80MNsVOosEWFlxqU4rT8TF1AaYUN2X8PV4ltH5Ez73VO8gX5Gr32NE8r+QFIluROFVtGYeeAHZhKEVtSTE6QirhI2nGwtjj9gKeTwJLdbJ5thSRDJsFETMR6ogRghd17Gfu6p3kC/IX+3V9i72K99jRPK9Axlgt59ISIpT27h5vdAe7W879vFxcPtyt3f/XyvInbL0fEgrT8oR3oSkmLA9LIGnHhFeG4I4BbLQSgaFkd5HzcdyJuHyeOnxVFVXmROYybl37owtknNU5ECTNrTrEQAHAo0+cD4NNHy72Wo7xlyvYz93VO8gXs5rgTDFWqRPGNDQRQVRsm0PWXjrf7Yp+VVjkblQivS0k4kQQRlUTAoXXlj5WLopJgqiQtOKJJwoqCqoqW/D+FOfcfVqlvVuWbhKquvgjcWOTpLpPAUh1zu8FqD5zA3Umpz36dUmRNSSGUZ7zMzIsKYhKS5GDWweGs1mhlncDQcwhFiuieLziMr40yWZJciYHVeeQQ/hAdltXvsaJ5XoIKcjLzq9mV7XlVB5bmojLj7i9q2KmX9g2ptUrj6yJs4Tkqaog3NuOETIJhRNAtYOg+j6ib4xlK8xjPuR1MVRRNpwmSBTacErjbLVtSK1S1aYjxIb1LmxlRUUopYHYyMXJoVh5rZ/dnauMPQ7wzEYO1VEM03htLibIbl5vCevqce2XVaUGqNQUceGPeqmb6NebRURLrt2w0bp4sW3u/Yz93VO8gXs/iA31UqEY/22LOTJ77caM0mJx940ABROURKiWmHRUfeiwzFvz42ibYeJb7/N3Du3uDDr6vtahinydFlGv6zuHonIdOqJUt9xbllA0D64FRUMN27q6yLtcW2UpFGaJ+JSo71HmXkm8SM8AE1IK+7FgkRw3l3762Ypnnj7ZV9oQFsbsMRxFBw34/buSGWZB/xG7ZdyhTgJ6CxMjOVOYaoi7mIXnjjh9d2XJAE1eM4fsV77GieV6A/wCDl8U/bkqvteZCjelVd1ilxk65SXEEk8UjlolOZS5qIy2wCJoS5sUD/d0nP3dU7yBeznmKvA+FOlj+syQL+iwQ6gpNvR1I4khslQmyJNOrfgMVu0iY/s2kRa688y7HNBiS4Et9pHmrtp1jEosmK8jatfEzFWmFTgwVBz/eluYzrWhROBDkC4nwxt/LZ3m+5vmY7vxhtzWcAd9x6nR1/tBUtorVMe/xYRj5MraBoMlE5YSm/wBBWvdolGf9xmQ8HlEK1zmTI73utVEB+O2tv5rIkr3VYmMOJ+gbXyclVgE6uBGXP742ukZcrzK//Bx/2g4tv5iLVI3X30B5PiodsLtRdYX+LEkj/wCUtkw5gYS/li6HxwGyIGY4CKvLfAPjqNuYrlPc7mWyv6DtzM6O53DoF+grc2Yn3Kov6OgMuLBoCIv53pP/AGdLr32NE8r0FUL93Q449l9V9rP1Cc4jMWMBOvureqCApeq6L1tk7L4IhMRd9XZYryWh3cZVFf4ql0rP3dU7yBezPZXQlTobLw9ZSYf3a/BXpWlEtpFF/qtpbHsJbwY9i2lpLXEyKp1l02XHCZLumwX9I256kxT7phtf7ttahw716qMAi/BRLaaOwl/JQx+KaWvaiuMdbcyH2/0OW/lZlRjr1Cbnvp+lVsM6JVcxymbjJ5xmqiyywgXXb5x9C8JfzYgOvhtLGcxu4NLRqIw+84b0p525ScOQ8eESJA3fE49s4yk0jCi0+Ei+6QK8adnpde+xonlegr68mjw07LhL7Wo2Wh0rVpoFJHrxIf8ANSL/AHCIGWv+LbN2YVuJmnoxRIZ8N26TeyUv/wAZelZ+7qneQL2cm1C7m57E6nOF1lQEfb+F7Veq9YkJHiMcJLpIy4rbQcLjh8UBszmPNrJQ6JHNHaLl0/c2JVQT5R4uIyuq3+1jcp0Zw3t9IelPPu3bw3HixKpYURNXQFs51tRuSZW3mG167cUUaBb/ANbpde+xonlegzH7lKhfHL2tmObVKYTTGV4cmIkts/5cBadJ10z3iCW+lI0wIgGLALdok6Vd55WHXqpKK668pJqYr4rB0rP3dU7yBezQMxCuFaLWIj7i/wAJwty4ir1B1ktemlPaa1CqGq413cWI0mJ+Q6uyyw3wmSr+zZnN2ehRHmVx0ihjrMQhXShGi6Hpa8cy2P2BbQR4EtMqDiogRGHHyVeC5sFP/daluure9P3k51VTSqyXCdT4Kj0uvfY0TyvQZj+yoPxi9qk44SCAIpGS8CIiXqq2Q48dqHUM6zgjMCyOFSCY8qg4fLdchNjvT7a0aCylzUZoGW0TktigD/YnSs/d1TvIF7NcpojicdiOG0PXcaTfNfDbG1FquLEciI1vlRb+dAd29/zQP2kyBtlNq01d3S6UxpekOdTQl+BoflHS2bJnHOjgS68Y3RY4aY8FpfkIorfr6ece2vjkiIlyJwJ7E+LH9Kqps02MN92IpLggQ+K3lodOaREbiMNsAidZsEBPi9Lr32NE8r0GYE5VIhkn9ThJ7VeoXnrUarVkRhw2nDQCUJDgx3XkUrkwstmZkXFw2yrluKl8DLkJyqOAmkUVUGLERepq4cQ9Lz93VO8gXsqJJeJJcqe4tsxZNduFyg1J1WA60WWqyI6onW8J7RZpFLY+k8yzk/kKY2vAn7+SSeCjhw4uP8MHcyZlfSp5mmp/MTCTUaDqRoor4JgODV8Ja72cpZYG42oKu12cK6UQWUVqKqp/jKfTK99jRPK9BOBeCRQWST3cEhU9q0GLUnFN2mMvTXoKAqi6wRtgG9c4ADfB4P5bnOSVs3ZsQeZkzRplPXRh82gDuubu4hn0vP3dU7yBdBS6npGFmmIdMkLxUlR+cjEvbGHND7QTK2U2RqGaZA34C8BCbX/MzCTZuvvBraL9nePT5jxVGuz13lSqj+l1014o/u2R+TaHoc1Z0XWYdkDSKYXUWPDS5ww7V17W6ZXvsaJ5XoKca6EkUB5tPdUJGP8AR7UkVSoHuokQFdfcRFLCA7RYRvJbvctnL8Q4kpqQbmOPCwFerbMcVYgAYLrtE++6cjAWtz1qPTXBwvpHF6T1VV5/nnb16uueHpefu6p3kC6B+RA0VOkGFUpxIl5I9FXeXD7pBjG1PrcW7dTmQeQU04SVNcPztuYgXpx5NyLherGzU6sqYo9OBdC3rwOy+QymyW3ZWY+J+W+W9mzn1xvvurpJ101vIlv4vF6GoTYyqs94Uh04B2ikyOaawp1SG/efqWptDC7FFZRHiTRieLXeL9Zwi6ZXvsaJ5XoMlPqlySo9SiqXui2jgp2fak6lz5IxI9QbWGT5KiYVf5oMOLQpqZiIDy7ZS/DOjtCDNUqAyp6ol5uMQkR543iTaJwsOsXJ1baEuTqJ0vP3dU7yBdBculF4UtXPw/f1Y4mtXoN+ysWQXPsh/gPcXu+maLP5UyI8rMZkt3WsyhpBnlRoJfKySTacHwXw7N06mMo2I6zji6xuGu044a6TcLldATzxi20CKRuGqCIonCpEuhEtOjUOUkz6PUBffbEtyqnetzT127dw4dbdraFSQ16VlAUnzl0KB1B5P5Vv3SYb5z9vpte+xonlegyLVVTVZrIxSLrJLbVv+77UpmTHTNh6lyjlSGXQXC++00hQgZIMeIb3SkFvMGHcWr1bXWiZeYbosMr9G/NVelqnbAvNl0zP3dU7yBdDAztRm1Oq5bdWQTY8L0M0wzI6/nb17RapAc3kSY0LzB9cDTEl/bJwEnSjccJAAEUjMluQURL1JVXgRLO0XK7zkPKjRK1U66Gq5MVNBxoKrwM8V2Rxu48KzTaXHCPGZHCAAn9qqukiLjEWsXsXrYnHDQGwRSMyVEFEThVVXgs5SskwnMy1MFwuOsLggsr13phahXclrFi5dgmfiHUlmNIqG3RIakxAbXQqIQoqOycKptOlaVPjxwaYhN4YsRoUFDeLUYZAAu23FFNXi2R+q69aqrp1GrOrwq++uPB/wh1Om177GieV6B2osje7SZUWeF3Cm5eHEqfqlZqS0t7bwC4C9qSISfp9p5o/ENlo1mRN7HlyHkuRThtiCtxv4RmgtkXyjw2iyJqf9QqxnVJxKlxE7KXea3/DwdMz93VO8gXQkBohCaKJCSXoqLouVOtaVkGWSjSaiTk7K7p7Ioq4pUBF67a840Pf209IdmzXgjxWBVx590kEAAeEiJdCWVhje07IjZ6ziXtyKphXg03EzCxfrOeSahQWRYjsigNtNogiIpwCKJ7CmaoICiqRKtyIiaVVVs5SsnxHMy1QVwl5quGGyvXfmrzWjkt47DK/ECpq7GvQgoUBSYgjpv525d7KVO3sESnxm47DaXA22KCKfmFPYBltd5lnKLuJ1U0tyqpdoHtghp/zO1Pp1e+xonlegqlIVL/PYrzA9dCMFQFT8xXWorzirv47CQ5CLtI5FVY5IXbc3i9pVapUYkGoRI5Px8QI4l7dxkmBdrECENsr5FrLIR5+ZJ5S5jbSEh/RzR+eSHZSKuq888X93asLYJhEUQRFOBETQidMz93VO8gXRI3Fd81qsJwZdKmjoJmS3rNlfyS2DsSTW/Na5Tj80rMFdCtPhoxon7p67G0X6nE6N+p1WQEWHHHE684tyJ1kTlES6BAdYrM1CttOU/JrBo7T6Od4uTVRbwlTrvkuM1H/APGYMMAgNgiCIiiIiImhERE4ESykq3ImlVXgRPdsdJyrGczLVxvQmoS/yzS6UvkTNLQJenEx2SRnypqkElxDQKcpMxBTkvuIu9lL3WrYIdNitx2G9ltsUEU/qT43sXraNlbLZX5mrl7UZU/yrHA9Oc5Itji3X8T/AA7RaNBTUYHnHC2nHC0uOn27ha3Tq99jRPK9BotmvJri4WjfSt0wVXhZlXI+IpyWnsI9NbgvI9PqzyYmKTAb30kh6hqCXC232zhWBjMdLqOXhcVEblT2P5ZVXgQnmlNAXu7NyIroPMOohNPNkhgQrwEJDeJJ7NLyaxOGJMKQh1EHhuRxpWsTEdonEwGUp11rwZY9Q7V3O660GJ/0SiLxVaYXFKeDtXXl1S6bn7uqd5AujD8QMqNbyfGDd1inJoGdDS5SHR/mGUTEyXxsOC0es0l5HokkbxXgICTaacHiuNrqmPQu1asPozHb0CiJiNw12WmQ4XHT4o2ZzNnRnzalxi3lFy4q4hb5MmanA5JJNlvZbthFEERT9Fjo+XGHMx1odCxIFystlp9Il+BaG9OpjK29z1UtxTiW9KBSyJqPdp1ZD/hpPbDiwcmwQ6ZEbjMBsttigj+fRwl2xdAVQmIr0hwkZp8FvS7JkF4NltPz7ZcQbSs05mJH8zVe4pRJpCO18lDY5LbQ7fLPp9e+xonleho34hQQUiojis1ZsOFynSFwvaOMrBc4FmpUZwXWHwF1l0dIkBpiAkXrEK9Lk1cw3z6XMwY2m96S7qsNIiadYtYv4YnZ2qVtUl5kqq+cVaYWlVMtO4BeKyxsCA6tjjyGAcacRRNshRRJF6hCqYSsdTyGvnVJJScm5YfNd0V+kjguFf5u9/D8GfwLEdOdVqWzomUyQm7lRy4FF1pdN1/HHUstSbaGTUH3BYgRCvucNVxHiw62EGkMrRMywI6DnfNrPmNHiJivp0RFJp6a9vBAhfBnEG+w4f8A0+piK0KiQk5iG0jaFdcpFwm4vbOGpGXTc/d1TvIF0aiSXotn82ZUjlLpMtd5X6A3ox3bUyGPAMkE0uN/K/FYq1HkDJhvpeJjwivGbcHhBwOOBeykyoKTr7xbqDAZTE/JeXZaZbTSWldYtkLBm/PiCU0L1pdHRcUeAC6UuTZdl/vHuV+rgWj0BhzMNaHV+j6drAC/+4k6WWR5W0Q8m29zpUVg0s9KUClmrbajyZUpOdf7YQ1OTYIlKhtRmQ4jYoKX8peUXbH0K1GpERES7uHDa1n5Ly7LLDfCREv7Fv8AW+dhT6UUVGlUu/EzT2S4g8qSSeGe/wBhRE6nT699jRPK9C7FkgLrDwE262aXiQkmEhJOsqWHJNecL6AluF/pmrOreIYlxLTpDi7JAq8wRbXxOlBDux0XJoo891QdqbyXtj1i81bTF2jmKyInsXKl6L1FsFRZVynVdnSxVIRqzJBeom8Hwg9q5YYiZjiSmQvRqVKp7ZyAReqiou7I+2s/VZshyp12Z6XVJS4nST92HFaZHitB07P3dU7yBdIwkl6LZ3M/4fqLbzpY6nQ3FuizeuSJwMSeS6Pf43Y7OOFV4urOo8pMElkk4dVbt43/ABQ+BZJbUF6pTJDoxYMKOKqrj7l+7Fw7rmW9XXcKzma6wD1VzzVkKPAOO2bkelMknOebCKKLZNgeBt3bcd1/3hk1ludU5VNpVMZTHQ4rbzCuApaxy57iIbxOuERky3qciwxKTDbjNJwi2N169cl2jLtjLokolJZWs5kfRfN6THW/B/EluJqx2h4dbX+PZM150fSpZgJOYRL/ADaGC/Iw212buM9tn8KyIiXInAntCvfY0TyvRO0qqxxkRXkuIC6i8KEK8ImK7Jjs28zrDb+YMqN3IxUGkxzoQcCDID/MsNp8omuI+LsM+iTWpsYrtZor1FV4rgbbZ9qY9Il1ck3khERmBG4VelO6rDQomsV56xYfk8dmmZy7yrTTKdVnl2jlPrjdvVOHB4P9X2pn7uqd5Auk3Wbma8OqxtaJUohbqS0ScGBwdpO0PVt5vmynlmKkt6BrFOBElgKXJfKh6Bcu4xs/Csn0RUWZRomtFVd2+K9Y47mFxLu5tzYoN/W6Hc1GeBzV0N0+Lz8kyXgFGWsRDf8AxMA23MBkspUNxNaQ5cdTdBbtkfBw8XjLE1TmOddXHIkuLvHnTVb1N50tYyUv1faVe+xonlejwmiKllrFDfeodX4fPace6Il4edb8E8K8bENsFQhRc0wguRJEdfMpl3XNskWOf6lkZrzE6gPquFRqEVxAv9x1lHQVO2tiplYhytN2Ft9tSRV644sVrx0p109lyo1qW3CitpepulcqryQBLzcNeKADitGzfV47kOg0xVXLtNeS43DLhqEkOASL5AOJ8Nz9HtTP3dU7yBdLwmKKnu238uEAS00jLZvZeRU4F3rWE/2rJ/pzN8xGhvujVMAnBdw4UI8Lop+a2EmaLUQ5apIjl2LyG2H6CpCLyllO3di62G+jUsV0Ym235J/n11QLf/qc2z5barecWJhgsEnJIGNYktdTIDTTipcT6DidLunjxOr+1bCKIiJ1E9p177GieV6TcqX/AJ7KDrIkK8IqiKn7K6LKUikRca6FMWhAuy3gtip0ibBXrxpj4di8it/L5qrrKdYagap8Ibc/m6vOD1lnkifBGwVGYLtTmt3YJFQdOUQqnAo71cCL+pZAFLkTg9q5+7qneQL8jV77GieV/IueFrlSj0/flA3HnLiN48LC4sGLaw3pfb7x0/x4e/b7x0/x4e/b7x0/x4e/b7x0/wAeHv2+8dP8eHv2+8dP8eHv2+8dP8eHv2+8dP8AHh79vvHT/Hh79vvHT/Hh79vvHT/Hh79vvHT/AB4e/b7x0/x4e/b7x0/x4e/b7x0/x4e/b7x0/wAeHv2+8dP8eHv2+8dP8eHv2+8dP8eHv2+8dP8AHh79vvHT/Hh79vvHT/Hh79vvHT/Hh79vvHT/AB4e/b7x0/x4e/b7x0/x4e/b7x0/x4e/b7x0/wAeHv2+8dP8eHv2Yl0WZBrII5dNjNSwCQjSptxwLVeNF+TxWWTQpgvqCIr8YtV9q/gR1pdYeDa2C4pdHXlJURPoeJpXR8pbbHspbbHspbbHspbbHspbbHspbbHspbbHspbbHspbbHspbbHspbbHspbbHspbbHspbbHspbbHspbbHspbbHspbbHspbbHspbbHspbbHspbbHspbbHspbbHspbbHspbbHspbbHspbbHspbbHspb6UlMPyYouCD6wwR02gK+99wL0LdBdr4cVgqNGmNTYh8DrRX3LySTaA+0NMXRLMnwGJD6ogq460BrcnAl5itvqqJ6u13lvqqJ6u13lvqqJ6u13lvqqJ6u13lvqqJ6u13lvqqJ6u13lvqqJ6u13lvqqJ6u13lvqqJ6u13lvqqJ6u13lvqqJ6u13lvqqJ6u13lvqqJ6u13lvqqJ6u13lvqqJ6u13lvqqJ6u13lvqqJ6u13lvqqJ6u13lvqqJ6u13lvqqJ6u13lvqqJ6u13lvqqJ6u13lvqqJ6u13lvqqJ6u13lvqqJ6u13lvqqJ6u13lvqqJ6u13lvqqJ6u13lvqqJ6u13lmIUAWqWyrl80o0ZlHXmrrt0LqjiZ06cQWSFRowMCV29NEvM1TquOLrmvddGlVqkdTlICNK4BuNqojsoW6IMWG3oznrEj523oznrEj523oznrEj523oznrEj523oznrEj523oznrEj523oznrEj523oznrEj523oznrEj523oznrEj523oznrEj523oznrEj523oznrEj523oznrEj523oznrEj523oznrEj523oznrEj523oznrEj523oznrEj523oznrEj523oznrEj523oznrEj523oznrEj523oznrEj523oznrEj523oznrEj523oznrEj523oznrEj523oznrEj52yDliG39IOuCBPS35Sg00V+8dABcXeOjxALUs48y65JnyQQJMhwlRFRFvuFpFwAN/dOdv/AEgqE+VLRqPSpQwJ5kBIoPkraCOG7EQlvm9cdTDaTRmXMc2G00/JbQVuAHlJGsR7GI92a4NrD+RZO8ksj5miFMQnBTcio40J69ebRQ19fi2ajo+2rzwK6y0hjjNsbsTgDfeQJjDX7b8iOmQtllyqNA/VBVU3iTYzL0WMQhxhwvMud3GC0uXmPdrWqg6HnBNFjHcxmgixRx8bm2t6fbun+RRYhRXHaZmqOxTq48zeiMhEf32/dJODexHH4zdpM6uxXI71HitZdp5up4ZmMZm7LBV2glXxtcf3P9C7/wCkc2iFKcWmBQhlBDVebR7zhoN6g8vARDbzuNTJVWdVwWhiQkBXNfjlvCbEWx458W0zLk+lyaNWoTKSjhyVA0NgiQN4240RCVxEGLu9XHaGFHos9xh1TSoTSBEjw1HHcLr2wbhALbmBviPBxsQ2/wBRM5cnO5cJ5Y7FVU2GxccQ1bv3JnvAbUk8Ifc2gUWNRZdQzFOjpLOkRyaUo7fAvnD+NWR1rx1FMfgYpMiKw9DkwXzhz4MlEF1l9vaAsKkK91ag0hqS4FPkU6S8/FRebNwFLAZDyhtIrtWUvNo9yI22mJxwyXCDbaKqJiJeuvbWjZerdFmUGfPbN6nJLwED4gmIxQ2yXA6IaxNknwrTH38rVN2k081CVUx3IgopdzrLZOY3WtO3qdvhtQX32n5UbMLLr8B2MGMiwNtusgjW2TkkpDTTXblzmAbUmfWqHUIbtXfejx4CghykJtOZRWUuLHJVQBttOOdnMsVKjzKJVNx55HZmIC71m+7Eigq4S7XtT1tW2YsxANRmxqdWX40huY8Dr29M2RUY5aohGDfDu2y1tu1MizqFUoFNrLyRqZVZLaA264a4W1Vu/eNg7eJAp62DXwYMVqnSKPRJdacojYO1h6OTTYMCYq4gjvTEnnMA7DY2brjcF2DFfJfNkeNo1cAdVXE3Jng10NvA5hc1Nm2Z6C/mmbRKdT48V6IEd5tsENyOwppc8JJhxGThYbVuZVTdrqUeXIZgy4zY76ayyKEO6BMAOuFxNOvjEdq0DL9coE2iP1ZHPo12SrRg4rQ4jEt0Zbsv+7auURygyjgRH4UdtB3InDF5C3kiWu81wd8MAtbzA2HLtVG6TQ51YgUMiCr1GLu0aaJtL3RBDJFeVq7Xw/FtEq9OPeQ5rYvMEqXLhJOAhXZIdkun5gpVfzlPy9U4M449IpEJ9IrZxx1mHMJakonAwmQeEw9odqFQqizNrU6ey420/HbE3nno4Ams0Kjzklwxw4NUO5tKoX+mKomYoyiv0QIARK2SYt8TqFu2mxRW8RH++a3e8tPrMOnyxkUw3WJlJwispHmRxk02OIRMyvwhrDr6lqfQK7QJtEdqyH9GPSlaMXSbRCMCRoi3ZaU+Db8Z03zlzEZhWExLzarGkre3p1NKcW2XXHCUzKnRiIyVVVVVob1VVtnynzJTj0SnPQRgsGt4so428TiNp1MaiNqfTW4Uiq1iqkYwKbEQcZo2mJxwzNRFtseV/dBzDU4h0+TT6vR1FJ9KfQFeTGKm2rSiWBwXLrh2P2cFqZDr2XKjRo1YeGLAmyt0qK8a3ADzbZkbOJVHa+IJFao5Vi0idU6xBVjdxoYIe9F9pHycvv5pphDbBwz4zlky21RJ06snFZltw4oiSqjt+MTNVEWkYu5xxzVtmaU5HnUt6kNSI9UigQty2TbAlPcO6QRzVMQIk8IGsFsntORahUArsESg4cMiWStsMm2DyIo76TIJ9tvm9Xe9raTlqpUmXRKowx54DMzAqOMIqCpiQKunW2e71tSx5hj5dnP5c3yxmavjYAXDQ92i7kzRxtoiTwjmHklrWYkSI5RXnQE3IxkJk2SperZE2ptkQ9oWG1bzW5nebCqVOkS0iwjeaJk0jle01ujHeETvgx1v1bUSvSaPKqlTqDTKPRYIAhIRgpb80cMBaZNAxdrvAtWafIokyHXKI0L8ikmrRuOAaYg3DgmjZX3jtYNuy1XMNOciw225Up6tLuxibtlwubERLe4mm9BETfyR2hVCZQqhBy9UX0jQ62+je6UiVRA3GhJXG2jw6HOnyswO0WqVKA9RghCdMilIudV4HdKqrbdyC0WLXxbNqBUKRT67TaEc5xrMEeOwTNT3YbtWsIMmTosOIr15ifJ7W06qsUep06jSKMQw3qsrzjri+cMXq4b5vEyZkDpBHNzebkd9ux3lobdTjFBdQ5TssJIK0SJvT13RcQVRNyIaS4ljp2YGcwLlGPUHZTMODFbepKRweU2cU9xVkvMEus5hXA5xdbXtFzq9DqlXy1Vqc2yD9CN3fsuXiYGQsEDnmxta/DhxnyxG1Sl0Sm1OnhKkJvirKOC8+QAnPAjxG5u9fBiLjBag5h+iahU4UenSGXfo2MUgkN0iQUXSAJ/WdjWj0OpxZFGnx5qwKiz5o9JAQdBwIyibmlAdIsSa3I1rUiXHoOZWalHVxwJNW84WPFubLFjcfMgXe3boBDaxWr0LMVLzDUs3uuv7sRJ9qlssgiKpNILgxnQwI5umhbc357ttsdbHb8Ht7BeJunwXfPlJo7o7gwY273+jmT3zervMPOBbIr0aM6+1GqZOSHGwIxbHCNxuEKKgDfxitl+oNxnCit0yU29KECVsSVTwAbqJgElv1RIrZ2hJTpSy5OYnH48ZGHN440rsRUdbbw4zbuEtcUw6tslhCiuv+bVyA6+LLZHumgEsRmgIu7bDjEWqNsyLSlrAxJsaPHqZ5aYamk4+IkDiSVeXDDdbbXdiLXO/K6uO0FrKwut02JjjIzIHC+DoEpPC+KfLE4e8c7u2cUzBTXTp70OMkCcQGIC75vHbI2HdDZuN63K2bVnKVPgyI9fy6YRosyKyqJKg7xF84hkg4Te3V+83eN/5Tw+xk+o0uk15IkN8wqdTrivkZPGxq4W3nHN22OEkceFptrHgbxWzT57R6g/EzCdN8wnRI5PMJum9yW9cHVbESdLGq7G6PtLV+nZro1fktzJr8yFNoiyDivR3rsLbgRzAN9tY8fcHq4MdNhU6LIhRBaxtQ5qqshpHSV5WnsSkqGBOYcGLU2On1GkS8i1h6vt7yLTZnmOqLt5Cy8EwVQ/NcYg4QkOBwC2cOtb8LfpVh6RJp8WWFUkohOi04scRBH30xDjvTDjI+cO1XqRRTFg6PHZbmK2qAR7682xduwqVwhiDFybfiS9l6LKj1J+vPvQybaIXjjE9zjsRFHnL2sWE2uL4LXw2ybUKXSa6sWHMbGqVKtLIIzeIEXEDLzjm7abwHvXxaBrHgbx2/GEkgyLqhGZGAu6O6QqRpIqjGjnlQiEebxWy/Hktmy+1T4wOtOCoGBC0KEJCVxCSdZbZ2lTcv1mfHqr8RYj1Ohq8Cow24hqpmTQ/LDhwY+NbLefvoerv0dIzsWdTomNipx8RGrbm5aPEmIi5zC5sbZWzZVMhUqtQ64cVptuVVgd3sg0xaIwySNxxxhsettG0IWyhIh0jMT8yFUYDtdqNXWQQI5jTGjLDzh3gJbxw3m2eZbwY3OctnSqPxHAYejQG4sw2yQDRGGd6LTqphPXbHeYC4lqrUzjOIwtGjsNS1AkbUt9icbFy7ApaoEQ2/FoPMn8U431hDujvfvYcFNxo528tXm7fg6XmT6jT4ZDPXdHdHLzOIKI/o5ksYEPOYdYbRKq5HcSmt5eNt6aQFuBNJDxqBvXbtCwa2FS2bSabW2MwFk9qouyW4dOjNvUrzYHlNpSnmqSXGMXOOYC1uJr61o86GYuxZLYPMODpEmzFCAk/OK2qVQGnLEzZHqUiVT3pLZCTqMuIbbJtO3AbL44hHEGDFrWyjUBg1mn0h51wczw6W063MbcbwIAXDgfSMdzhCQ/J/wAbdjbME1uk1CmUudTWkp5VVXXHjTEgKrrjxvGDhGJluCPeNt4NS1S/DGXS59OrAR57aSpDBBDJTcIgwydkkMncOrthjMNW0LLuZstZoWpNE1FkMAsg4SuIqBvwMXPNwjj4S/ZAPB4/aUmmzA3kWW0cd8OU24KgafslYcsv5vnnl4QRjzAGIoOKwPAz50je+w4dUu1tGp0JtGosRoGI7aacLbYoADp6wp0upTMtZkk0WNVXymS4TceM+G/cREdcbOQBm3vMOKy0yAbryuvHKlSpBITrz7t28ecJEFMRYRT8iSIMocceU2bLwcoHBUDH+sVsmWXM3Tyy6gbjzAWYouKwmjcrKRve4cOr3No1Ngt7qJEaBiO2iquFtsUAEvXSuqnV/ov/AP/Z)\n",
        "\n",
        "[Image source](https://saugatbhattarai.com.np/what-is-gradient-descent-in-machine-learning/)\n",
        "\n",
        "The learning rate will be denoted by $\\eta$. In our tutorial, let's assign a learning rate of $\\eta = 0.01$.\n"
      ],
      "metadata": {
        "id": "jBj25P0ve7Q9"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# learning rate\n",
        "lr = 0.01"
      ],
      "metadata": {
        "id": "G2fxQEYMcXTy"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Gradient descent algorithm\n",
        "\n",
        "In each iteration of the training algorithm, we will update each weight in our model $w_i \\rightarrow w_i'$ with the following formula:\n",
        "\n",
        "$$\n",
        "w_i' = w_i - \\eta \\frac{\\partial{J}}{\\partial{w_i}}\n",
        "$$\n",
        "\n",
        "where\n",
        "\n",
        "$$\n",
        "\\frac{\\partial{J}}{\\partial{w_i}} = \\frac{1}{m} \\sum _{j=1} ^{m} ((w_1 x_{j_1} + w_2 x_{j_2} + w_3 x_{j_3}) - y_i)(x_{j_i})\n",
        "$$"
      ],
      "metadata": {
        "id": "EePtLwjXV1g1"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "def grad(m, y_pred, y_true, x_i):\n",
        "    \"\"\"\n",
        "    calculate the partial derivative of cost function wrt w_i\n",
        "    \"\"\"\n",
        "    grad = (1/m) * np.sum((y_pred - y_true)*x_i)\n",
        "    return grad"
      ],
      "metadata": {
        "id": "1PFx0UMEpd6N"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Now let's begin implementing our code! We'll start with an example of **batch gradient descent**.\n",
        "\n",
        "# Example - batch gradient descent\n",
        "In batch gradient descent, each epoch contains only one batch, and the batch size is equivalent to the entire size of the training dataset. We will define 1000 epochs."
      ],
      "metadata": {
        "id": "kxZ8NIilgecP"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "epochs = 1000\n",
        "m = n"
      ],
      "metadata": {
        "id": "prDkE7nTgdtf"
      },
      "execution_count": 245,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# initialize random weights to start\n",
        "weights = np.random.randn(3)\n",
        "print(\"initial weights:\")\n",
        "print(weights)\n",
        "\n",
        "# collect value of cost function at each iter\n",
        "cost_list = []\n",
        "\n",
        "# collect the weights at the end of each epoch\n",
        "weights_updates = np.zeros((epochs+1, 3))\n",
        "weights_updates[0,:] = weights\n",
        "\n",
        "start = time.time()\n",
        "for iter in range(epochs):\n",
        "    print('epoch = %d' % iter)\n",
        "          \n",
        "    # predict on training set \n",
        "    y_pred = np.apply_along_axis(pred, 1, arr = X, weights = weights)\n",
        "\n",
        "    # calculate cost\n",
        "    cost = mse(y_pred, y, m)\n",
        "    cost_list.append(cost)\n",
        "\n",
        "    # update the weights\n",
        "    for i in range(weights.shape[0]):\n",
        "        weights[i] = weights[i] - lr*grad(m, y_pred, y, X[:,i])\n",
        "\n",
        "    # print(weights)\n",
        "    weights_updates[iter+1, :] = weights\n",
        "\n",
        "end = time.time()\n",
        "\n",
        "# calculate time\n",
        "batch_gd_time = end-start"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "T2Q0fTaRgFTc",
        "outputId": "d62f947e-68cf-408c-b098-098e3a90e93e"
      },
      "execution_count": 246,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "initial weights:\n",
            "[ 0.89622489 -0.65381768 -0.26837397]\n",
            "epoch = 0\n",
            "epoch = 1\n",
            "epoch = 2\n",
            "epoch = 3\n",
            "epoch = 4\n",
            "epoch = 5\n",
            "epoch = 6\n",
            "epoch = 7\n",
            "epoch = 8\n",
            "epoch = 9\n",
            "epoch = 10\n",
            "epoch = 11\n",
            "epoch = 12\n",
            "epoch = 13\n",
            "epoch = 14\n",
            "epoch = 15\n",
            "epoch = 16\n",
            "epoch = 17\n",
            "epoch = 18\n",
            "epoch = 19\n",
            "epoch = 20\n",
            "epoch = 21\n",
            "epoch = 22\n",
            "epoch = 23\n",
            "epoch = 24\n",
            "epoch = 25\n",
            "epoch = 26\n",
            "epoch = 27\n",
            "epoch = 28\n",
            "epoch = 29\n",
            "epoch = 30\n",
            "epoch = 31\n",
            "epoch = 32\n",
            "epoch = 33\n",
            "epoch = 34\n",
            "epoch = 35\n",
            "epoch = 36\n",
            "epoch = 37\n",
            "epoch = 38\n",
            "epoch = 39\n",
            "epoch = 40\n",
            "epoch = 41\n",
            "epoch = 42\n",
            "epoch = 43\n",
            "epoch = 44\n",
            "epoch = 45\n",
            "epoch = 46\n",
            "epoch = 47\n",
            "epoch = 48\n",
            "epoch = 49\n",
            "epoch = 50\n",
            "epoch = 51\n",
            "epoch = 52\n",
            "epoch = 53\n",
            "epoch = 54\n",
            "epoch = 55\n",
            "epoch = 56\n",
            "epoch = 57\n",
            "epoch = 58\n",
            "epoch = 59\n",
            "epoch = 60\n",
            "epoch = 61\n",
            "epoch = 62\n",
            "epoch = 63\n",
            "epoch = 64\n",
            "epoch = 65\n",
            "epoch = 66\n",
            "epoch = 67\n",
            "epoch = 68\n",
            "epoch = 69\n",
            "epoch = 70\n",
            "epoch = 71\n",
            "epoch = 72\n",
            "epoch = 73\n",
            "epoch = 74\n",
            "epoch = 75\n",
            "epoch = 76\n",
            "epoch = 77\n",
            "epoch = 78\n",
            "epoch = 79\n",
            "epoch = 80\n",
            "epoch = 81\n",
            "epoch = 82\n",
            "epoch = 83\n",
            "epoch = 84\n",
            "epoch = 85\n",
            "epoch = 86\n",
            "epoch = 87\n",
            "epoch = 88\n",
            "epoch = 89\n",
            "epoch = 90\n",
            "epoch = 91\n",
            "epoch = 92\n",
            "epoch = 93\n",
            "epoch = 94\n",
            "epoch = 95\n",
            "epoch = 96\n",
            "epoch = 97\n",
            "epoch = 98\n",
            "epoch = 99\n",
            "epoch = 100\n",
            "epoch = 101\n",
            "epoch = 102\n",
            "epoch = 103\n",
            "epoch = 104\n",
            "epoch = 105\n",
            "epoch = 106\n",
            "epoch = 107\n",
            "epoch = 108\n",
            "epoch = 109\n",
            "epoch = 110\n",
            "epoch = 111\n",
            "epoch = 112\n",
            "epoch = 113\n",
            "epoch = 114\n",
            "epoch = 115\n",
            "epoch = 116\n",
            "epoch = 117\n",
            "epoch = 118\n",
            "epoch = 119\n",
            "epoch = 120\n",
            "epoch = 121\n",
            "epoch = 122\n",
            "epoch = 123\n",
            "epoch = 124\n",
            "epoch = 125\n",
            "epoch = 126\n",
            "epoch = 127\n",
            "epoch = 128\n",
            "epoch = 129\n",
            "epoch = 130\n",
            "epoch = 131\n",
            "epoch = 132\n",
            "epoch = 133\n",
            "epoch = 134\n",
            "epoch = 135\n",
            "epoch = 136\n",
            "epoch = 137\n",
            "epoch = 138\n",
            "epoch = 139\n",
            "epoch = 140\n",
            "epoch = 141\n",
            "epoch = 142\n",
            "epoch = 143\n",
            "epoch = 144\n",
            "epoch = 145\n",
            "epoch = 146\n",
            "epoch = 147\n",
            "epoch = 148\n",
            "epoch = 149\n",
            "epoch = 150\n",
            "epoch = 151\n",
            "epoch = 152\n",
            "epoch = 153\n",
            "epoch = 154\n",
            "epoch = 155\n",
            "epoch = 156\n",
            "epoch = 157\n",
            "epoch = 158\n",
            "epoch = 159\n",
            "epoch = 160\n",
            "epoch = 161\n",
            "epoch = 162\n",
            "epoch = 163\n",
            "epoch = 164\n",
            "epoch = 165\n",
            "epoch = 166\n",
            "epoch = 167\n",
            "epoch = 168\n",
            "epoch = 169\n",
            "epoch = 170\n",
            "epoch = 171\n",
            "epoch = 172\n",
            "epoch = 173\n",
            "epoch = 174\n",
            "epoch = 175\n",
            "epoch = 176\n",
            "epoch = 177\n",
            "epoch = 178\n",
            "epoch = 179\n",
            "epoch = 180\n",
            "epoch = 181\n",
            "epoch = 182\n",
            "epoch = 183\n",
            "epoch = 184\n",
            "epoch = 185\n",
            "epoch = 186\n",
            "epoch = 187\n",
            "epoch = 188\n",
            "epoch = 189\n",
            "epoch = 190\n",
            "epoch = 191\n",
            "epoch = 192\n",
            "epoch = 193\n",
            "epoch = 194\n",
            "epoch = 195\n",
            "epoch = 196\n",
            "epoch = 197\n",
            "epoch = 198\n",
            "epoch = 199\n",
            "epoch = 200\n",
            "epoch = 201\n",
            "epoch = 202\n",
            "epoch = 203\n",
            "epoch = 204\n",
            "epoch = 205\n",
            "epoch = 206\n",
            "epoch = 207\n",
            "epoch = 208\n",
            "epoch = 209\n",
            "epoch = 210\n",
            "epoch = 211\n",
            "epoch = 212\n",
            "epoch = 213\n",
            "epoch = 214\n",
            "epoch = 215\n",
            "epoch = 216\n",
            "epoch = 217\n",
            "epoch = 218\n",
            "epoch = 219\n",
            "epoch = 220\n",
            "epoch = 221\n",
            "epoch = 222\n",
            "epoch = 223\n",
            "epoch = 224\n",
            "epoch = 225\n",
            "epoch = 226\n",
            "epoch = 227\n",
            "epoch = 228\n",
            "epoch = 229\n",
            "epoch = 230\n",
            "epoch = 231\n",
            "epoch = 232\n",
            "epoch = 233\n",
            "epoch = 234\n",
            "epoch = 235\n",
            "epoch = 236\n",
            "epoch = 237\n",
            "epoch = 238\n",
            "epoch = 239\n",
            "epoch = 240\n",
            "epoch = 241\n",
            "epoch = 242\n",
            "epoch = 243\n",
            "epoch = 244\n",
            "epoch = 245\n",
            "epoch = 246\n",
            "epoch = 247\n",
            "epoch = 248\n",
            "epoch = 249\n",
            "epoch = 250\n",
            "epoch = 251\n",
            "epoch = 252\n",
            "epoch = 253\n",
            "epoch = 254\n",
            "epoch = 255\n",
            "epoch = 256\n",
            "epoch = 257\n",
            "epoch = 258\n",
            "epoch = 259\n",
            "epoch = 260\n",
            "epoch = 261\n",
            "epoch = 262\n",
            "epoch = 263\n",
            "epoch = 264\n",
            "epoch = 265\n",
            "epoch = 266\n",
            "epoch = 267\n",
            "epoch = 268\n",
            "epoch = 269\n",
            "epoch = 270\n",
            "epoch = 271\n",
            "epoch = 272\n",
            "epoch = 273\n",
            "epoch = 274\n",
            "epoch = 275\n",
            "epoch = 276\n",
            "epoch = 277\n",
            "epoch = 278\n",
            "epoch = 279\n",
            "epoch = 280\n",
            "epoch = 281\n",
            "epoch = 282\n",
            "epoch = 283\n",
            "epoch = 284\n",
            "epoch = 285\n",
            "epoch = 286\n",
            "epoch = 287\n",
            "epoch = 288\n",
            "epoch = 289\n",
            "epoch = 290\n",
            "epoch = 291\n",
            "epoch = 292\n",
            "epoch = 293\n",
            "epoch = 294\n",
            "epoch = 295\n",
            "epoch = 296\n",
            "epoch = 297\n",
            "epoch = 298\n",
            "epoch = 299\n",
            "epoch = 300\n",
            "epoch = 301\n",
            "epoch = 302\n",
            "epoch = 303\n",
            "epoch = 304\n",
            "epoch = 305\n",
            "epoch = 306\n",
            "epoch = 307\n",
            "epoch = 308\n",
            "epoch = 309\n",
            "epoch = 310\n",
            "epoch = 311\n",
            "epoch = 312\n",
            "epoch = 313\n",
            "epoch = 314\n",
            "epoch = 315\n",
            "epoch = 316\n",
            "epoch = 317\n",
            "epoch = 318\n",
            "epoch = 319\n",
            "epoch = 320\n",
            "epoch = 321\n",
            "epoch = 322\n",
            "epoch = 323\n",
            "epoch = 324\n",
            "epoch = 325\n",
            "epoch = 326\n",
            "epoch = 327\n",
            "epoch = 328\n",
            "epoch = 329\n",
            "epoch = 330\n",
            "epoch = 331\n",
            "epoch = 332\n",
            "epoch = 333\n",
            "epoch = 334\n",
            "epoch = 335\n",
            "epoch = 336\n",
            "epoch = 337\n",
            "epoch = 338\n",
            "epoch = 339\n",
            "epoch = 340\n",
            "epoch = 341\n",
            "epoch = 342\n",
            "epoch = 343\n",
            "epoch = 344\n",
            "epoch = 345\n",
            "epoch = 346\n",
            "epoch = 347\n",
            "epoch = 348\n",
            "epoch = 349\n",
            "epoch = 350\n",
            "epoch = 351\n",
            "epoch = 352\n",
            "epoch = 353\n",
            "epoch = 354\n",
            "epoch = 355\n",
            "epoch = 356\n",
            "epoch = 357\n",
            "epoch = 358\n",
            "epoch = 359\n",
            "epoch = 360\n",
            "epoch = 361\n",
            "epoch = 362\n",
            "epoch = 363\n",
            "epoch = 364\n",
            "epoch = 365\n",
            "epoch = 366\n",
            "epoch = 367\n",
            "epoch = 368\n",
            "epoch = 369\n",
            "epoch = 370\n",
            "epoch = 371\n",
            "epoch = 372\n",
            "epoch = 373\n",
            "epoch = 374\n",
            "epoch = 375\n",
            "epoch = 376\n",
            "epoch = 377\n",
            "epoch = 378\n",
            "epoch = 379\n",
            "epoch = 380\n",
            "epoch = 381\n",
            "epoch = 382\n",
            "epoch = 383\n",
            "epoch = 384\n",
            "epoch = 385\n",
            "epoch = 386\n",
            "epoch = 387\n",
            "epoch = 388\n",
            "epoch = 389\n",
            "epoch = 390\n",
            "epoch = 391\n",
            "epoch = 392\n",
            "epoch = 393\n",
            "epoch = 394\n",
            "epoch = 395\n",
            "epoch = 396\n",
            "epoch = 397\n",
            "epoch = 398\n",
            "epoch = 399\n",
            "epoch = 400\n",
            "epoch = 401\n",
            "epoch = 402\n",
            "epoch = 403\n",
            "epoch = 404\n",
            "epoch = 405\n",
            "epoch = 406\n",
            "epoch = 407\n",
            "epoch = 408\n",
            "epoch = 409\n",
            "epoch = 410\n",
            "epoch = 411\n",
            "epoch = 412\n",
            "epoch = 413\n",
            "epoch = 414\n",
            "epoch = 415\n",
            "epoch = 416\n",
            "epoch = 417\n",
            "epoch = 418\n",
            "epoch = 419\n",
            "epoch = 420\n",
            "epoch = 421\n",
            "epoch = 422\n",
            "epoch = 423\n",
            "epoch = 424\n",
            "epoch = 425\n",
            "epoch = 426\n",
            "epoch = 427\n",
            "epoch = 428\n",
            "epoch = 429\n",
            "epoch = 430\n",
            "epoch = 431\n",
            "epoch = 432\n",
            "epoch = 433\n",
            "epoch = 434\n",
            "epoch = 435\n",
            "epoch = 436\n",
            "epoch = 437\n",
            "epoch = 438\n",
            "epoch = 439\n",
            "epoch = 440\n",
            "epoch = 441\n",
            "epoch = 442\n",
            "epoch = 443\n",
            "epoch = 444\n",
            "epoch = 445\n",
            "epoch = 446\n",
            "epoch = 447\n",
            "epoch = 448\n",
            "epoch = 449\n",
            "epoch = 450\n",
            "epoch = 451\n",
            "epoch = 452\n",
            "epoch = 453\n",
            "epoch = 454\n",
            "epoch = 455\n",
            "epoch = 456\n",
            "epoch = 457\n",
            "epoch = 458\n",
            "epoch = 459\n",
            "epoch = 460\n",
            "epoch = 461\n",
            "epoch = 462\n",
            "epoch = 463\n",
            "epoch = 464\n",
            "epoch = 465\n",
            "epoch = 466\n",
            "epoch = 467\n",
            "epoch = 468\n",
            "epoch = 469\n",
            "epoch = 470\n",
            "epoch = 471\n",
            "epoch = 472\n",
            "epoch = 473\n",
            "epoch = 474\n",
            "epoch = 475\n",
            "epoch = 476\n",
            "epoch = 477\n",
            "epoch = 478\n",
            "epoch = 479\n",
            "epoch = 480\n",
            "epoch = 481\n",
            "epoch = 482\n",
            "epoch = 483\n",
            "epoch = 484\n",
            "epoch = 485\n",
            "epoch = 486\n",
            "epoch = 487\n",
            "epoch = 488\n",
            "epoch = 489\n",
            "epoch = 490\n",
            "epoch = 491\n",
            "epoch = 492\n",
            "epoch = 493\n",
            "epoch = 494\n",
            "epoch = 495\n",
            "epoch = 496\n",
            "epoch = 497\n",
            "epoch = 498\n",
            "epoch = 499\n",
            "epoch = 500\n",
            "epoch = 501\n",
            "epoch = 502\n",
            "epoch = 503\n",
            "epoch = 504\n",
            "epoch = 505\n",
            "epoch = 506\n",
            "epoch = 507\n",
            "epoch = 508\n",
            "epoch = 509\n",
            "epoch = 510\n",
            "epoch = 511\n",
            "epoch = 512\n",
            "epoch = 513\n",
            "epoch = 514\n",
            "epoch = 515\n",
            "epoch = 516\n",
            "epoch = 517\n",
            "epoch = 518\n",
            "epoch = 519\n",
            "epoch = 520\n",
            "epoch = 521\n",
            "epoch = 522\n",
            "epoch = 523\n",
            "epoch = 524\n",
            "epoch = 525\n",
            "epoch = 526\n",
            "epoch = 527\n",
            "epoch = 528\n",
            "epoch = 529\n",
            "epoch = 530\n",
            "epoch = 531\n",
            "epoch = 532\n",
            "epoch = 533\n",
            "epoch = 534\n",
            "epoch = 535\n",
            "epoch = 536\n",
            "epoch = 537\n",
            "epoch = 538\n",
            "epoch = 539\n",
            "epoch = 540\n",
            "epoch = 541\n",
            "epoch = 542\n",
            "epoch = 543\n",
            "epoch = 544\n",
            "epoch = 545\n",
            "epoch = 546\n",
            "epoch = 547\n",
            "epoch = 548\n",
            "epoch = 549\n",
            "epoch = 550\n",
            "epoch = 551\n",
            "epoch = 552\n",
            "epoch = 553\n",
            "epoch = 554\n",
            "epoch = 555\n",
            "epoch = 556\n",
            "epoch = 557\n",
            "epoch = 558\n",
            "epoch = 559\n",
            "epoch = 560\n",
            "epoch = 561\n",
            "epoch = 562\n",
            "epoch = 563\n",
            "epoch = 564\n",
            "epoch = 565\n",
            "epoch = 566\n",
            "epoch = 567\n",
            "epoch = 568\n",
            "epoch = 569\n",
            "epoch = 570\n",
            "epoch = 571\n",
            "epoch = 572\n",
            "epoch = 573\n",
            "epoch = 574\n",
            "epoch = 575\n",
            "epoch = 576\n",
            "epoch = 577\n",
            "epoch = 578\n",
            "epoch = 579\n",
            "epoch = 580\n",
            "epoch = 581\n",
            "epoch = 582\n",
            "epoch = 583\n",
            "epoch = 584\n",
            "epoch = 585\n",
            "epoch = 586\n",
            "epoch = 587\n",
            "epoch = 588\n",
            "epoch = 589\n",
            "epoch = 590\n",
            "epoch = 591\n",
            "epoch = 592\n",
            "epoch = 593\n",
            "epoch = 594\n",
            "epoch = 595\n",
            "epoch = 596\n",
            "epoch = 597\n",
            "epoch = 598\n",
            "epoch = 599\n",
            "epoch = 600\n",
            "epoch = 601\n",
            "epoch = 602\n",
            "epoch = 603\n",
            "epoch = 604\n",
            "epoch = 605\n",
            "epoch = 606\n",
            "epoch = 607\n",
            "epoch = 608\n",
            "epoch = 609\n",
            "epoch = 610\n",
            "epoch = 611\n",
            "epoch = 612\n",
            "epoch = 613\n",
            "epoch = 614\n",
            "epoch = 615\n",
            "epoch = 616\n",
            "epoch = 617\n",
            "epoch = 618\n",
            "epoch = 619\n",
            "epoch = 620\n",
            "epoch = 621\n",
            "epoch = 622\n",
            "epoch = 623\n",
            "epoch = 624\n",
            "epoch = 625\n",
            "epoch = 626\n",
            "epoch = 627\n",
            "epoch = 628\n",
            "epoch = 629\n",
            "epoch = 630\n",
            "epoch = 631\n",
            "epoch = 632\n",
            "epoch = 633\n",
            "epoch = 634\n",
            "epoch = 635\n",
            "epoch = 636\n",
            "epoch = 637\n",
            "epoch = 638\n",
            "epoch = 639\n",
            "epoch = 640\n",
            "epoch = 641\n",
            "epoch = 642\n",
            "epoch = 643\n",
            "epoch = 644\n",
            "epoch = 645\n",
            "epoch = 646\n",
            "epoch = 647\n",
            "epoch = 648\n",
            "epoch = 649\n",
            "epoch = 650\n",
            "epoch = 651\n",
            "epoch = 652\n",
            "epoch = 653\n",
            "epoch = 654\n",
            "epoch = 655\n",
            "epoch = 656\n",
            "epoch = 657\n",
            "epoch = 658\n",
            "epoch = 659\n",
            "epoch = 660\n",
            "epoch = 661\n",
            "epoch = 662\n",
            "epoch = 663\n",
            "epoch = 664\n",
            "epoch = 665\n",
            "epoch = 666\n",
            "epoch = 667\n",
            "epoch = 668\n",
            "epoch = 669\n",
            "epoch = 670\n",
            "epoch = 671\n",
            "epoch = 672\n",
            "epoch = 673\n",
            "epoch = 674\n",
            "epoch = 675\n",
            "epoch = 676\n",
            "epoch = 677\n",
            "epoch = 678\n",
            "epoch = 679\n",
            "epoch = 680\n",
            "epoch = 681\n",
            "epoch = 682\n",
            "epoch = 683\n",
            "epoch = 684\n",
            "epoch = 685\n",
            "epoch = 686\n",
            "epoch = 687\n",
            "epoch = 688\n",
            "epoch = 689\n",
            "epoch = 690\n",
            "epoch = 691\n",
            "epoch = 692\n",
            "epoch = 693\n",
            "epoch = 694\n",
            "epoch = 695\n",
            "epoch = 696\n",
            "epoch = 697\n",
            "epoch = 698\n",
            "epoch = 699\n",
            "epoch = 700\n",
            "epoch = 701\n",
            "epoch = 702\n",
            "epoch = 703\n",
            "epoch = 704\n",
            "epoch = 705\n",
            "epoch = 706\n",
            "epoch = 707\n",
            "epoch = 708\n",
            "epoch = 709\n",
            "epoch = 710\n",
            "epoch = 711\n",
            "epoch = 712\n",
            "epoch = 713\n",
            "epoch = 714\n",
            "epoch = 715\n",
            "epoch = 716\n",
            "epoch = 717\n",
            "epoch = 718\n",
            "epoch = 719\n",
            "epoch = 720\n",
            "epoch = 721\n",
            "epoch = 722\n",
            "epoch = 723\n",
            "epoch = 724\n",
            "epoch = 725\n",
            "epoch = 726\n",
            "epoch = 727\n",
            "epoch = 728\n",
            "epoch = 729\n",
            "epoch = 730\n",
            "epoch = 731\n",
            "epoch = 732\n",
            "epoch = 733\n",
            "epoch = 734\n",
            "epoch = 735\n",
            "epoch = 736\n",
            "epoch = 737\n",
            "epoch = 738\n",
            "epoch = 739\n",
            "epoch = 740\n",
            "epoch = 741\n",
            "epoch = 742\n",
            "epoch = 743\n",
            "epoch = 744\n",
            "epoch = 745\n",
            "epoch = 746\n",
            "epoch = 747\n",
            "epoch = 748\n",
            "epoch = 749\n",
            "epoch = 750\n",
            "epoch = 751\n",
            "epoch = 752\n",
            "epoch = 753\n",
            "epoch = 754\n",
            "epoch = 755\n",
            "epoch = 756\n",
            "epoch = 757\n",
            "epoch = 758\n",
            "epoch = 759\n",
            "epoch = 760\n",
            "epoch = 761\n",
            "epoch = 762\n",
            "epoch = 763\n",
            "epoch = 764\n",
            "epoch = 765\n",
            "epoch = 766\n",
            "epoch = 767\n",
            "epoch = 768\n",
            "epoch = 769\n",
            "epoch = 770\n",
            "epoch = 771\n",
            "epoch = 772\n",
            "epoch = 773\n",
            "epoch = 774\n",
            "epoch = 775\n",
            "epoch = 776\n",
            "epoch = 777\n",
            "epoch = 778\n",
            "epoch = 779\n",
            "epoch = 780\n",
            "epoch = 781\n",
            "epoch = 782\n",
            "epoch = 783\n",
            "epoch = 784\n",
            "epoch = 785\n",
            "epoch = 786\n",
            "epoch = 787\n",
            "epoch = 788\n",
            "epoch = 789\n",
            "epoch = 790\n",
            "epoch = 791\n",
            "epoch = 792\n",
            "epoch = 793\n",
            "epoch = 794\n",
            "epoch = 795\n",
            "epoch = 796\n",
            "epoch = 797\n",
            "epoch = 798\n",
            "epoch = 799\n",
            "epoch = 800\n",
            "epoch = 801\n",
            "epoch = 802\n",
            "epoch = 803\n",
            "epoch = 804\n",
            "epoch = 805\n",
            "epoch = 806\n",
            "epoch = 807\n",
            "epoch = 808\n",
            "epoch = 809\n",
            "epoch = 810\n",
            "epoch = 811\n",
            "epoch = 812\n",
            "epoch = 813\n",
            "epoch = 814\n",
            "epoch = 815\n",
            "epoch = 816\n",
            "epoch = 817\n",
            "epoch = 818\n",
            "epoch = 819\n",
            "epoch = 820\n",
            "epoch = 821\n",
            "epoch = 822\n",
            "epoch = 823\n",
            "epoch = 824\n",
            "epoch = 825\n",
            "epoch = 826\n",
            "epoch = 827\n",
            "epoch = 828\n",
            "epoch = 829\n",
            "epoch = 830\n",
            "epoch = 831\n",
            "epoch = 832\n",
            "epoch = 833\n",
            "epoch = 834\n",
            "epoch = 835\n",
            "epoch = 836\n",
            "epoch = 837\n",
            "epoch = 838\n",
            "epoch = 839\n",
            "epoch = 840\n",
            "epoch = 841\n",
            "epoch = 842\n",
            "epoch = 843\n",
            "epoch = 844\n",
            "epoch = 845\n",
            "epoch = 846\n",
            "epoch = 847\n",
            "epoch = 848\n",
            "epoch = 849\n",
            "epoch = 850\n",
            "epoch = 851\n",
            "epoch = 852\n",
            "epoch = 853\n",
            "epoch = 854\n",
            "epoch = 855\n",
            "epoch = 856\n",
            "epoch = 857\n",
            "epoch = 858\n",
            "epoch = 859\n",
            "epoch = 860\n",
            "epoch = 861\n",
            "epoch = 862\n",
            "epoch = 863\n",
            "epoch = 864\n",
            "epoch = 865\n",
            "epoch = 866\n",
            "epoch = 867\n",
            "epoch = 868\n",
            "epoch = 869\n",
            "epoch = 870\n",
            "epoch = 871\n",
            "epoch = 872\n",
            "epoch = 873\n",
            "epoch = 874\n",
            "epoch = 875\n",
            "epoch = 876\n",
            "epoch = 877\n",
            "epoch = 878\n",
            "epoch = 879\n",
            "epoch = 880\n",
            "epoch = 881\n",
            "epoch = 882\n",
            "epoch = 883\n",
            "epoch = 884\n",
            "epoch = 885\n",
            "epoch = 886\n",
            "epoch = 887\n",
            "epoch = 888\n",
            "epoch = 889\n",
            "epoch = 890\n",
            "epoch = 891\n",
            "epoch = 892\n",
            "epoch = 893\n",
            "epoch = 894\n",
            "epoch = 895\n",
            "epoch = 896\n",
            "epoch = 897\n",
            "epoch = 898\n",
            "epoch = 899\n",
            "epoch = 900\n",
            "epoch = 901\n",
            "epoch = 902\n",
            "epoch = 903\n",
            "epoch = 904\n",
            "epoch = 905\n",
            "epoch = 906\n",
            "epoch = 907\n",
            "epoch = 908\n",
            "epoch = 909\n",
            "epoch = 910\n",
            "epoch = 911\n",
            "epoch = 912\n",
            "epoch = 913\n",
            "epoch = 914\n",
            "epoch = 915\n",
            "epoch = 916\n",
            "epoch = 917\n",
            "epoch = 918\n",
            "epoch = 919\n",
            "epoch = 920\n",
            "epoch = 921\n",
            "epoch = 922\n",
            "epoch = 923\n",
            "epoch = 924\n",
            "epoch = 925\n",
            "epoch = 926\n",
            "epoch = 927\n",
            "epoch = 928\n",
            "epoch = 929\n",
            "epoch = 930\n",
            "epoch = 931\n",
            "epoch = 932\n",
            "epoch = 933\n",
            "epoch = 934\n",
            "epoch = 935\n",
            "epoch = 936\n",
            "epoch = 937\n",
            "epoch = 938\n",
            "epoch = 939\n",
            "epoch = 940\n",
            "epoch = 941\n",
            "epoch = 942\n",
            "epoch = 943\n",
            "epoch = 944\n",
            "epoch = 945\n",
            "epoch = 946\n",
            "epoch = 947\n",
            "epoch = 948\n",
            "epoch = 949\n",
            "epoch = 950\n",
            "epoch = 951\n",
            "epoch = 952\n",
            "epoch = 953\n",
            "epoch = 954\n",
            "epoch = 955\n",
            "epoch = 956\n",
            "epoch = 957\n",
            "epoch = 958\n",
            "epoch = 959\n",
            "epoch = 960\n",
            "epoch = 961\n",
            "epoch = 962\n",
            "epoch = 963\n",
            "epoch = 964\n",
            "epoch = 965\n",
            "epoch = 966\n",
            "epoch = 967\n",
            "epoch = 968\n",
            "epoch = 969\n",
            "epoch = 970\n",
            "epoch = 971\n",
            "epoch = 972\n",
            "epoch = 973\n",
            "epoch = 974\n",
            "epoch = 975\n",
            "epoch = 976\n",
            "epoch = 977\n",
            "epoch = 978\n",
            "epoch = 979\n",
            "epoch = 980\n",
            "epoch = 981\n",
            "epoch = 982\n",
            "epoch = 983\n",
            "epoch = 984\n",
            "epoch = 985\n",
            "epoch = 986\n",
            "epoch = 987\n",
            "epoch = 988\n",
            "epoch = 989\n",
            "epoch = 990\n",
            "epoch = 991\n",
            "epoch = 992\n",
            "epoch = 993\n",
            "epoch = 994\n",
            "epoch = 995\n",
            "epoch = 996\n",
            "epoch = 997\n",
            "epoch = 998\n",
            "epoch = 999\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Let's visualize the MSE over each training iteration:"
      ],
      "metadata": {
        "id": "ijpQB68ikoIi"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "plt.plot(cost_list)\n",
        "plt.ylabel(\"Cost\")\n",
        "plt.xlabel(\"epoch\")\n",
        "plt.title(\"MSE over training epochs - batch gradient descent\")\n",
        "plt.show()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 295
        },
        "id": "O5xlkWaEjauA",
        "outputId": "4848752c-05b9-4af6-8d35-d2bd48da1995"
      },
      "execution_count": 247,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ],
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEWCAYAAABrDZDcAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3deZxcZZ3v8c+3l3R3ls7aZA8JgkgIECCADuNyUbyIXnAcFHADRBln9IqOszHOVXSc6/jyuuDoiIgOiwzioIzI6CibCwqBDgQEAhJIMAmBdPZ96e7f/eM8nVQq3Z3uTp+u7j7f9+tVr6o656lzfmep+tXzPGdRRGBmZsVVVekAzMysspwIzMwKzonAzKzgnAjMzArOicDMrOCcCMzMCs6JwA5K0hOSXtffZQc7SddJ+uwAzOdKSd8dgPnMlhSSavKeV09Iep2klSXvK7bvDNS2HqwKlwgkLZe0W9KksuGPpC/J7PR+hqQfSForaZOkxyVdnMZ1fKG2lj3OH/AF6kZ/ffEj4tiI+EV/l7VDJ+liSfdVOo7+0F/7zkAl1oGSvsNH5jmPQfHPoAKWARcC/wIg6ThgZFmZG4FHgcOBXcBxwJSyMuMiojXfUHtGUk1fYunr58xKeT8a2gpXI0huBN5b8v4i4IayMqcA10XEtohojYhHIuKnfZmZpGmSbpe0XtJSSR8oGb5D0oSSsiemWkhtev8+SUskbZD0M0mHl5QNSR+S9AzwTCez/lV63phqLK9K/yB/I+nLktYBV0p6maR7JK1L875J0riS+SyX9Ib0+kpJ35d0g6QtqTq/oI9lT0o1sS2S/kPSLd1Vz3uwLj4i6bm0DF+QVJXGVUn6B0nPS1qT4hlb8tk/lvRbSRslreio+SXjJf1XinGhpJelzyitwzWSNkv6naR5XcXeA/Vp+bdIeljSCSXx/Z2kZ9O4JyX9SRp+DHA18Kq0fTem4Q2SvpiWd5Ok+yQ1lMzrXZL+kNbTJ7pZ3xMl/Tgt30OSPlta++hs/5N0VVqHmyUtkvTqkvINyppgNkh6kuw7Vjq/0n2nqmS516X9aEIa11HTvah8OSSdBfw9cH5aJ492sWwnpvW8RdItQH3Z+LdIWpz2id9KOr5k3N9KWpU++7Sk16fh1ZL+vmRbLZI0M417haQ7lf0GPC3pHSXTu07S17vYzzq+w48qz1aHiCjUA1gOvAF4GjgGqAZWkv3zD2B2KncX8BvgAmBW2TRmp7I1PZznr4B/JdvZ5gMtwBlp3D3AB0rKfgG4Or0+F1ia4qwB/gH4bUnZAO4EJgANncz3gDiBi4FW4H+naTYARwJnAnVAU4r3K+XrLL2+EtgJnJ3W3eeAB3pbFhgBPA9cDtQCbwN2A5/tYh32ZF3cm9bFLOD3wPvTuPelzx4BjAZ+CNyYxh0ObCGrIdYCE4H5adx1wDrg1DTPm4DvpXH/E1gEjAOU4prax33ySmAPcF6K4a/Iaq21afzbgWlkf9zOB7Z1zCttz/vKpvd14BfA9LTe/yht24794Vtpu59AVts9pou4vpceI4G5wIrSedHJ/ge8O63DGuDjwItAfRr3z8CvU/mZwOPAyi72ncuBB4AZKfZvAjeX7dedLkdan9/tZn137HsfS+v7vLT+P5vGnwisAU5L6++iFFsdcHRaD9NKYnlZev3XwO9SGaW4JgKj0mcuSevlRGAtMPdg+1nJej4y19/FPCc+GB/sSwT/QPbDdFbamWvYPxGMTzvuE0AbsBg4pWxH3Fj2OOALlXb4NmBMybDPkdU2AN4P3JNeK+0wr0nvfwpcWvK5KmA7cHjJDnJGN8vaEWd5IvjDQdbRW4FHytdZyZfsrpJxc4EdvS0LvAZYBahk/H10nQh6si7OKhn/F8Dd6fXdwF+UjDua7ItfA1wB3NbFPK8Dri15fzbwVHp9BlmyeSVQdYj75JXsn0yrgNXAq7sovxg4t2R73lf22R3ACd3sDzNKhj0IXNBJ2eq0jo4uGfZZDkwEXe5/qcyGjliA58q20WV0nQiWAK8vGTe1ZJt1uxwcPBG8BnihbN/7LfsSwTeAfyz7zNPAa8n+NK0h+w2p7aTMuZ3M73zg12XDvgl86mD7Wcl6zjURFLVpCLLmoXeSfZHKm4WIiA0R8XcRcSwwmezL95+SVFJsUkSMK3ks6WQ+04D1EbGlZNjzZP/WAH5AVrWfSraDtpP9a4Ls3+pVqXq6EVhPliyml0xrRa+WupPPSJos6XupursZ+C4wqfOPAtm/vA7byZo1uupv6qrsNGBVpD29s7jK9HZdPJ/mQXp+vmxcDdl2nQk82818y+MfDRAR9wBfI/v3vUbSNZIayz8s6dXadzDBE93MZ2/sEdFOVkudlqbx3pJmio3APLrePpPIap69XqYyTWTrqHSddrZ9yvelv1LWfLcpxTq2JNZpHLiNunI4cFvJMi8h+0M1uZfL0ZnO9r3SWA4HPt4x7zT/mWS1gKXAR8mSzZr0venYz7ralw4HTiub3rvYv8+xr8vSLwqbCCLiebLq99lkTQXdlV0L/D+yHWhCd2U78QIwQdKYkmGzyP4NExEbgJ+T/Wt4J1mVsGMHXQH8WVmyaYiI35aG113oPRz+f9Ow4yKikax6rwM+1b9WA9PLEuvMbsr3ZF2Ufn4W2bonPR9eNq4VeClN92V9WYCI+GpEnExW03k5WdNAeZlfR8To9Di2m8ntjV1Z38YM4AVl/SDfAj4MTIyIcWRNKh3rrXxbriVrjuvTMpVoIVtHMzqLscTe+af+gL8B3gGMT7FuKol1NQduo66sAN5Utr3rI2JVD2Lv7jvREUf5vlcaywrgn8rmPTIibgaIiH+PiD9mX3Py50s+19l6XwH8smx6oyPiz3uwLAOisIkguZSsarutfISkz0uaJ6km/Yj/ObA0Itb1ZgYRsYKs2vk5SfWp0+lSsn/dHf6drPP6vPS6w9XAFZKOTTGNlfT2Xsy+hayGccRByo0BtgKbJE2nkx+0HNxP9g/vw2kdn0vWRtqVnqyLv5Y0PnXQXQ7ckobfDHxM0hxJo8kS3y2RHeVyE/AGSe9IcUyUNP9gwUs6RdJpyjr1t5H9+Lb3eOkPdLKkt6Xa0kfJ2rwfIGtfDrJtiaRLyGoEHV4CZkgaAXtrE98BvqTsYIRqZQcJ1PUmmIhoI/uDdKWkkZJewf4HWHRmDFnyaAFqJH0SKK0lfZ9sG46XNIOsn6orVwP/lBIhkprSPtITLwGzU0LtzP0pzo9IqpX0Nvbf974FfDBtX0kaJenNksZIOlrSGWl97iRrhuvY7tcC/yjpqPS54yVNBO4AXi7pPWl+tWn/OaYXy3Ow7/AhKXQiiIhnI6K5i9EjgdvI2v6fI8v+55SV6Tgap+Pxl11M60Kyds0X0jQ/FRF3lYy/HTgKeDEi9h7lEBG3kf3b+F5qsnkceFMvlm878E/Ab1KV9JVdFP00cBLZv7f/4iA1pP4QEbvJOogvJVvH7yb7wuzqonxP1sWPyDpwF5Mtx7fT8O+QNQX+iqwWuJP0IxQRfyCrFX6crLlpMVkn38E0kv1gbCBrVlhH1tHfVz8iqxVuAN4DvC0i9kTEk8AXyX68XiI7jPk3JZ+7h6wf60VJa9OwvyLrtHwoLdPn6dt3/cNkTTsvkq2/m+li+yQ/A/6brO/kebL1XNoU9Ok0fBlZLfjGbqZ1Fdn34ueStpAlxdN6GPd/pOd1kh4uH1my711Mtn7Op2SfT78JHyBr+ttAdqDBxWl0HVnf4Vqy9XIYWT8TwJfIkt3Pgc1k+19DahZ+I9mBJy+kz30+TasnrgSuT9/hdxyscF9o/2Yys8qRtJDsiKl/68NnAzgqteFaDiR9HpgSERdVOhbrX4WuEVhlSXqtpCmpSeYi4Hiyf5Q2CCg79v341MxxKlnt7bZKx2X9r6hnFtvgcDRZVXoUWfPbeRGxurIhWYkxZM1B08iapb5I1oRlw4ybhszMCs5NQ2ZmBTfkmoYmTZoUs2fPrnQYZmZDyqJFi9ZGRFNn44ZcIpg9ezbNzV0d8WlmZp2R1OWZ3G4aMjMruNwTQTqz8RFJd3Qy7mJJLek6KoslvT/veMzMbH8D0TR0OdkFow64IFdyS0R8eADiMDOzTuRaI0jXE3kz2TU4zMxsEMq7aegrZFcj7O5iXH8q6TFJt6aLhR1A0mWSmiU1t7S05BKomVlR5ZYIJL0FWBMRi7op9mOyG8EcT3ZzmOs7KxQR10TEgohY0NTU6dFPZmbWR3nWCE4HzpG0nOx2d2dIKr30MhGxLiI6rmZ4LXByjvGYmVkncksEEXFFRMyIiNlkl1+9JyLeXVpG2V25OpxD1qmci6df3MIXfvYUG7btzmsWZmZD0oCfRyDpM5I6ruv/EUlPSHoU+Aj7rvnd75at3cbX732WVRt35DULM7MhaUDOLI6IXwC/SK8/WTL8Cvbd1CFXE0ePAGCdawRmZvspzJnFE0dliWD9tu5usGRmVjwFSgTZXeHWbXWNwMysVGESQWNDDTVVYr2bhszM9lOYRCCJ8aNGuEZgZlamMIkAsn4Cdxabme2vWIlg9Ah3FpuZlSlUIpgwqs59BGZmZQqVCNw0ZGZ2oMIlgi07W9nV2lbpUMzMBo1CJYIJ6eziDdv2VDgSM7PBo1CJoOPs4nXuMDYz26tQiWBCOrvYHcZmZvsUKhF0XHjOicDMbJ9iJYLUNLTWZxebme1VqETQWF9LdZV8UpmZWYlCJYKqKjF+5Ag3DZmZlcg9EUiqlvSIpDs6GVcn6RZJSyUtlDQ773gmjfaF58zMSg1EjeByur4X8aXAhog4Evgy8Pm8g5kwyjUCM7NSuSYCSTOANwPXdlHkXOD69PpW4PWSlGdME3yZCTOz/eRdI/gK8DdAexfjpwMrACKiFdgETCwvJOkySc2SmltaWg4poImjRrBuqzuLzcw65JYIJL0FWBMRiw51WhFxTUQsiIgFTU1NhzStiaPr2Lyzld2tXeUmM7NiybNGcDpwjqTlwPeAMyR9t6zMKmAmgKQaYCywLseYaBqT7l3sQ0jNzIAcE0FEXBERMyJiNnABcE9EvLus2O3ARen1ealM5BUTQNPoLBG0bHEiMDMDqBnoGUr6DNAcEbcD3wZulLQUWE+WMHLVUSNwIjAzywxIIoiIXwC/SK8/WTJ8J/D2gYihgxOBmdn+CnVmMey78JwTgZlZpnCJoK6mmnEja2nxIaRmZkABEwFkHcauEZiZZQqZCCY5EZiZ7VXIRNA0ps5NQ2ZmSXETgWsEZmZAgRPB9t1tbNvVWulQzMwqrpiJwGcXm5ntVcxEkE4qW+t+AjOzYicC1wjMzIqeCFwjMDMrZiIYP3IE1VVyjcDMjIImguoqMXHUCCcCMzMKmgjAZxebmXUobCJoGlPHGicCM7PiJoLJjXW8tHlnpcMwM6u4PG9eXy/pQUmPSnpC0qc7KXOxpBZJi9Pj/XnFU25yYz1rt+6itc03sTezYsvzDmW7gDMiYqukWuA+ST+NiAfKyt0SER/OMY5OTW6spz1g7dbdTBlbP9CzNzMbNPK8eX1ExNb0tjY9cr0xfW9Macx+/F9085CZFVyufQSSqiUtBtYAd0bEwk6K/amkxyTdKmlmnvGU6qgFvLjJicDMii3XRBARbRExH5gBnCppXlmRHwOzI+J44E7g+s6mI+kySc2SmltaWvoltsmpRuAOYzMrugE5aigiNgL3AmeVDV8XER3HcF4LnNzF56+JiAURsaCpqalfYpo4agS11XLTkJkVXp5HDTVJGpdeNwBnAk+VlZla8vYcYEle8ZSrqhKHjannJTcNmVnB5XnU0FTgeknVZAnn+xFxh6TPAM0RcTvwEUnnAK3AeuDiHOM5wOTGOtcIzKzwcksEEfEYcGInwz9Z8voK4Iq8YjiYKWPreerFLZWavZnZoFDYM4sh6zB205CZFV3hE8G23W1s2bmn0qGYmVVMoRPBFB9CamZW7ESw71wCX4XUzIqr0InAZxebmRU9Efh6Q2ZmxU4EDSOqaayvcR+BmRVaoRMBwNSxDbyw0YnAzIqr8Ilg2rh6Vm/aUekwzMwqxolgXAMvbHQiMLPiciIY18CG7XvYvru10qGYmVVE4RPBjPENAK4VmFlhFT4RTBuXJYJV7jA2s4JyIhjnGoGZFVvhE8HkMXVUV4lVG5wIzKyYCp8IaqqrmNJY7xqBmRVW4RMBZOcSrHIiMLOCyvOexfWSHpT0qKQnJH26kzJ1km6RtFTSQkmz84qnO9PGNfCCTyozs4LKs0awCzgjIk4A5gNnSXplWZlLgQ0RcSTwZeDzOcbTpenjGli9cSdt7VGJ2ZuZVVRuiSAyW9Pb2vQo/6U9F7g+vb4VeL0k5RVTV6aNa6C1PWjZ4vsSmFnx5NpHIKla0mJgDXBnRCwsKzIdWAEQEa3AJmBiJ9O5TFKzpOaWlpZ+j3P63nMJ3DxkZsWTayKIiLaImA/MAE6VNK+P07kmIhZExIKmpqb+DRKfS2BmxTYgRw1FxEbgXuCsslGrgJkAkmqAscC6gYip1PTxrhGYWXHledRQk6Rx6XUDcCbwVFmx24GL0uvzgHsiYsB7bEfX1TB+ZC0r1m8f6FmbmVVcTY7TngpcL6maLOF8PyLukPQZoDkibge+DdwoaSmwHrggx3i6NWvCSP7gRGBmBZRbIoiIx4ATOxn+yZLXO4G35xVDb8ycMJLHV22qdBhmZgPOZxYnsyaMZOWGHbS2tVc6FDOzAeVEkMyaMJLW9mD1Jl+O2syKxYkgmTVhJIA7jM2scJwIkpkpEbjD2MyKxokgmTaugZoqORGYWeE4ESTVVWLG+AYnAjMrHCeCEjMnjHQfgZkVjhNBCZ9UZmZF5ERQYtaEkWzYvofNO/dUOhQzswHjRFDCh5CaWRE5EZToOIT0+XVOBGZWHE4EJeZMGgXAsrXbKhyJmdnAcSIoMaquhimN9TzX4kRgZsXRo0Qg6caeDBsO5kwaxbK1Ww9e0MxsmOhpjeDY0jfpHgMn9384lTenaRTPuWnIzAqk20Qg6QpJW4DjJW1Ojy1kN6P/0YBEOMCOmDSKjdv3sGHb7kqHYmY2ILpNBBHxuYgYA3whIhrTY0xETIyIK7r7rKSZku6V9KSkJyRd3kmZ10naJGlxenyys2kNpCOasg5j1wrMrCh6eoeyOySNiohtkt4NnARcFRHPd/OZVuDjEfGwpDHAIkl3RsSTZeV+HRFv6UPsuZgzaTQAz7Vs5eTDx1c4GjOz/PW0j+AbwHZJJwAfB54FbujuAxGxOiIeTq+3AEuA6YcQ64CYOT67CqkPITWzouhpImiNiADOBb4WEV8HxvR0JpJmk92/eGEno18l6VFJP5V0bCfjkXSZpGZJzS0tLT2dbZ/UVFcxa+JIJwIzK4yeJoItkq4A3gP8l6QqoLYnH5Q0GvgB8NGI2Fw2+mHg8Ig4AfgX4D87m0ZEXBMRCyJiQVNTUw9D7rsjJo3yuQRmVhg9TQTnA7uA90XEi8AM4AsH+5CkWrIkcFNE/LB8fERsjoit6fVPgFpJk3oafF7mTBrFsnXbaG+PSodiZpa7HiWC9ON/EzBW0luAnRHRbR+BJAHfBpZExJe6KDMllUPSqSmedb2IPxdHNI1md2s7qzbuqHQoZma56+mZxe8AHgTeDrwDWCjpvIN87HSypqQzSg4PPVvSByV9MJU5D3hc0qPAV4ELUl9ERR11WHbk0DNrtlQ4EjOz/PX08NFPAKdExBoASU3AXcCtXX0gIu4D1N1EI+JrwNd6GMOAOWpy1g/++5e2csYrJlc4GjOzfPW0j6CqIwkk63rx2SFnbEMtkxvr+P1LrhGY2fDX0xrBf0v6GXBzen8+8JN8QhocXj55DM+85IvPmdnwd7BrDR0p6fSI+Gvgm8Dx6XE/cM0AxFcxRx02hmfWbPGRQ2Y27B2seecrwGaAiPhhRPxlRPwlcFsaN2wdPWU0O/e0s2KD71ZmZsPbwRLB5Ij4XfnANGx2LhENEqUdxmZmw9nBEsG4bsY19Gcgg03HIaTuMDaz4e5giaBZ0gfKB0p6P7Aon5AGhzH1tUwbW88zTgRmNswd7KihjwK3SXoX+374FwAjgD/JM7DB4KjJY3jaTUNmNsx1mwgi4iXgjyT9D2BeGvxfEXFP7pENAkdPGcP9z61jT1s7tdXD9rQJMyu4Hp1HEBH3AvfmHMugM3dqI7tb23muZRtHT+nxVbfNzIYU/83txtxpjQA88cKmCkdiZpYfJ4JuHDFpFHU1VTz5QvltFMzMhg8ngm7UVFfxiqmNPOFEYGbDmBPBQcyd2siTqzczCK6ObWaWCyeCgzh2WiObduzxTWrMbNhyIjiIjg5j9xOY2XCVWyKQNFPSvZKelPSEpMs7KSNJX5W0VNJjkk7KK56+OmZKI1XC/QRmNmz19H4EfdEKfDwiHpY0Blgk6c6IeLKkzJuAo9LjNOAb6XnQaBhRzRFNo50IzGzYyq1GEBGrI+Lh9HoLsASYXlbsXOCGyDwAjJM0Na+Y+mretEZ+t2pjpcMwM8vFgPQRSJoNnAgsLBs1HVhR8n4lByYLJF0mqVlSc0tLS15hdumEmeN4afMuXty0c8DnbWaWt9wTgaTRwA+Aj0ZEn9pXIuKaiFgQEQuampr6N8AemD8zuxr34hUbBnzeZmZ5yzURSKolSwI3RcQPOymyCphZ8n5GGjaozJ3WSG21eGSFm4fMbPjJ86ghAd8GlkTEl7oodjvw3nT00CuBTRGxOq+Y+qquppq5Uxt51InAzIahPI8aOh14D/A7SYvTsL8HZgFExNXAT4CzgaXAduCSHOM5JPNnjuPWRStpaw+qq1TpcMzM+k1uiSAi7gO6/cWM7LoNH8orhv40f9Y4rr//eZ5Zs4VXTGmsdDhmZv3GZxb30Akzsg5jNw+Z2XDjRNBDcyaNYmxDLY/8wYnAzIYXJ4IeksTJh4/noeXrKx2KmVm/ciLohVPnTODZlm2s3bqr0qGYmfUbJ4JeOGX2BACaXSsws2HEiaAXjps+lvraKhYucyIws+HDiaAXRtRUceLM8TzoRGBmw4gTQS+dOmcCS1ZvZvPOPZUOxcysXzgR9NKpcybQHrDoeV+AzsyGByeCXjpx1jhqq8UDz66rdChmZv3CiaCXRo6o4aRZ4/n1M2srHYqZWb9wIuiDVx81iSdXb/b5BGY2LDgR9MGrj8pujvObpa4VmNnQ50TQB/Omj2VsQy33uXnIzIYBJ4I+qK4Spx85kfuWriW7kraZ2dDlRNBHf3xkE6s37eTZlq2VDsXM7JDkeavK70haI+nxLsa/TtImSYvT45N5xZKH17x8EgD3PtVS4UjMzA5NnjWC64CzDlLm1xExPz0+k2Ms/W7G+JEcM7WRO5e8VOlQzMwOSW6JICJ+BQzri/KcecxhNC9fz4ZtuysdiplZn1W6j+BVkh6V9FNJx1Y4ll57w9zJtAfc+/SaSodiZtZnlUwEDwOHR8QJwL8A/9lVQUmXSWqW1NzSMnja5OdNG8vkxjrucvOQmQ1hFUsEEbE5Iram1z8BaiVN6qLsNRGxICIWNDU1DWic3amqEq8/ZjK/fLqFXa1tlQ7HzKxPKpYIJE2RpPT61BTLkLuS2xvnTmbb7jZ+9XufXGZmQ1NNXhOWdDPwOmCSpJXAp4BagIi4GjgP+HNJrcAO4IIYgmdnnX7kJMaPrOWOx17gzLmTKx2OmVmv5ZYIIuLCg4z/GvC1vOY/UGqrqzhr3lR+tHgVO3a30TCiutIhmZn1SqWPGhoW/tcJU9m+u417nvLRQ2Y29DgR9IPT5kykaUwdP370hUqHYmbWa04E/aC6Srz5uKnc8/QaNm73yWVmNrQ4EfST806ewe7Wdm53rcDMhhgngn4yb/pYjp3WyC0Prah0KGZmveJE0I/OP2UmT7ywmcdXbap0KGZmPeZE0I/OPWE6I2qq+H6zawVmNnQ4EfSjsSNrOXveFH748Cq27NxT6XDMzHrEiaCfXXz6HLbuauXWRSsrHYqZWY84EfSz+TPHceKscVz/2+W0tw+5K2aYWQE5EeTgktPnsHzddt+nwMyGBCeCHLxp3hSmjq3n6l8+yxC8jp6ZFYwTQQ5qq6v44GtfxkPLN/DAc8P6bp1mNgw4EeTk/FNmctiYOr569zOVDsXMrFtOBDmpr63mz177Mu5/bh0PLnOtwMwGLyeCHL3z1FlMGl3HF3/+tPsKzGzQciLIUcOIai5//ZEsXLaenz/pG9yb2eCUWyKQ9B1JayQ93sV4SfqqpKWSHpN0Ul6xVNKFp87iqMNG87mfLGF3a3ulwzEzO0CeNYLrgLO6Gf8m4Kj0uAz4Ro6xVExNdRWfePMxLF+3nRvuX17pcMzMDpBbIoiIXwHd9ZKeC9wQmQeAcZKm5hVPJb3u6MN47cub+Mpdz7B6045Kh2Nmtp9K9hFMB0ov07kyDTuApMskNUtqbmlpGZDg+ttnzj2W1vZ2PnHb4+44NrNBZUh0FkfENRGxICIWNDU1VTqcPjl84ij+6o1Hc89Ta3wXMzMbVCqZCFYBM0vez0jDhq1LTp/DCTPHceXtT/Dipp2VDsfMDKhsIrgdeG86euiVwKaIWF3BeHJXXSW++PYT2NXazke+9witbT6KyMwqL8/DR28G7geOlrRS0qWSPijpg6nIT4DngKXAt4C/yCuWweTIw0bz2bfO48Fl67nKl58ws0GgJq8JR8SFBxkfwIfymv9g9raTZnD/s+v42r1LOXbaWM6aN6XSIZlZgQ2JzuLh6B/fOo/jZ4zjo7c8wmMrN1Y6HDMrMCeCCqmvreba9y5g4qg6Lr2+mRXrt1c6JDMrKCeCCmoaU8e/XXIKu1vbufBbD7Bqo082M7OB50RQYS+fPIbvXnoam3bs4cJrnAzMbOA5EQwCx80Yy42XnsaG7bt527/+hide2FTpkMysQJwIBon5M8dx6wf/iCqJd1x9P7/wje/NbIA4EQwiR08Zw21/cTqzJo7ikuse4st3/p62dl+XyMzy5UQwyEwZW88P/vxV/MmJ07nq7md4z7cX8oL7DcwsR04Eg9DIETV86R3z+cJ5x/PIHzZy5pd+yQ33L6fdtQMzy4ETwSD29gUz+fnHXsNJh4/nkz96gpC+86UAAAm8SURBVLf+62/47bNrKx2WmQ0zTgSD3MwJI7nhfafylfPns3bLLt75rYVc/G8PsniFz0Y2s/6hoXaTlAULFkRzc3Olw6iInXvauOH+5Xz93mfZtGMPp86ZwAdefQRnvOIwqqtU6fDMbBCTtCgiFnQ6zolg6Nm6q5VbHlrBd+5bxqqNO5jcWMefnDiD806ezpGHjal0eGY2CDkRDFOtbe3ctWQNty5awb1Pt9DWHhx12GjeMHcyZ86dzPwZ46hyTcHMcCIohDVbdnLHo6u5a8lLLFy2nrb2oLG+hlNmT+C0IyZwyuwJHDO1kfra6kqHamYV4ERQMJu27+EXv1/D/c+u48Fl63lu7TYgu0PakU2jOXZaI3OnNfKyw0Yze+IoZoxvoLbaxw2YDWcVSwSSzgKuAqqBayPin8vGXwx8gX33Kv5aRFzb3TSdCHpvzeadND+/gSde2MSTL2zmydWbeWnzrr3jq6vEjPENzJowkmljGzissY7DGuuZPCZ7bhpTx/iRtTTUViO5qclsKOouEeR2hzJJ1cDXgTOBlcBDkm6PiCfLit4SER/OKw6DwxrrOfu4qZx93NS9w9Zt3cWytdtYtnYbz6/bzvJ12fPTL25h7dZddHbuWm21GNtQS2NDLWPTo7G+lpEjqqmvraZhRDUj03N9bTUNe19XUVtdRU1VFbXVoqY6e86GZc+11VXUVIvaquy54yioKokqZc8STkRmOcgtEQCnAksj4jkASd8DzgXKE4FVwMTRdUwcXceC2RMOGNfa1s66bbtZs3kXL23eScvWXWzasWe/x+Yde1i/bTfL125jx542duxuY8eeNva05dvUKO1LDipJEh2JYv/E0VGu5PNov2ntG146j86TzX7lD2Ga6qRAT+ZvdsEpM3n/q4/o9+nmmQimAytK3q8ETuuk3J9Keg3we+BjEbGivICky4DLAGbNmpVDqFaqprqKyY31TG6s5zjG9uqze9ra2VmSGDqSRGt7sKetnda2oLW9nd2t2XNrWxqexu9pC1rb2mkPaI8gIohgv/cdr9uD9D5KxneMS8NKqjalraBBV8M5aPkuXlLazNr1dLovy9DqsrMBNml0XS7TzTMR9MSPgZsjYpekPwOuB84oLxQR1wDXQNZHMLAhWm90NPOMqa+tdChm1kN5HiqyCphZ8n4G+zqFAYiIdRHR0Wt5LXByjvGYmVkn8kwEDwFHSZojaQRwAXB7aQFJU0vengMsyTEeMzPrRG5NQxHRKunDwM/IDh/9TkQ8IekzQHNE3A58RNI5QCuwHrg4r3jMzKxzPqHMzKwAujuPwKeTmpkVnBOBmVnBORGYmRWcE4GZWcENuc5iSS3A8338+CSgaDf99TIXg5e5GA5lmQ+PiKbORgy5RHAoJDV31Ws+XHmZi8HLXAx5LbObhszMCs6JwMys4IqWCK6pdAAV4GUuBi9zMeSyzIXqIzAzswMVrUZgZmZlnAjMzAquMIlA0lmSnpa0VNLfVTqe/iJppqR7JT0p6QlJl6fhEyTdKemZ9Dw+DZekr6b18Jikkyq7BH0jqVrSI5LuSO/nSFqYluuWdOlzJNWl90vT+NmVjPtQSBon6VZJT0laIulVw3k7S/pY2qcfl3SzpPrhuJ0lfUfSGkmPlwzr9XaVdFEq/4yki3oTQyESgaRq4OvAm4C5wIWS5lY2qn7TCnw8IuYCrwQ+lJbt74C7I+Io4O70HrJ1cFR6XAZ8Y+BD7heXs//9Kz4PfDkijgQ2AJem4ZcCG9LwL6dyQ9VVwH9HxCuAE8iWf1huZ0nTgY8ACyJiHtml7C9geG7n64Czyob1artKmgB8iux2wKcCn+pIHj0Se+8LO3wfwKuAn5W8vwK4otJx5bSsPwLOBJ4GpqZhU4Gn0+tvAheWlN9bbqg8yO52dzfZbU3vILv3+1qgpnx7k90P41XpdU0qp0ovQx+WeSywrDz24bqd2XfP8wlpu90B/M/hup2B2cDjfd2uwIXAN0uG71fuYI9C1AjYt1N1WJmGDSupOnwisBCYHBGr06gXgcnp9XBYF18B/gZoT+8nAhsjojW9L12mvcubxm9K5YeaOUAL8G+pSexaSaMYpts5IlYB/w/4A7CabLstYvhv5w693a6HtL2LkgiGPUmjgR8AH42IzaXjIvuLMCyOE5b0FmBNRCyqdCwDrAY4CfhGRJwIbGNfcwEw7LbzeOBcsgQ4DRjFgc0nhTAQ27UoiWAVMLPk/Yw0bFiQVEuWBG6KiB+mwS913BM6Pa9Jw4f6ujgdOEfScuB7ZM1DVwHjJHXcerV0mfYubxo/Flg3kAH3k5XAyohYmN7fSpYYhut2fgOwLCJaImIP8EOybT/ct3OH3m7XQ9reRUkEDwFHpSMORpB1Ot1e4Zj6hSQB3waWRMSXSkbdDnQcOXARWd9Bx/D3pqMPXglsKqmCDnoRcUVEzIiI2WTb8Z6IeBdwL3BeKla+vB3r4bxUfsj9a46IF4EVko5Og14PPMkw3c5kTUKvlDQy7eMdyzust3OJ3m7XnwFvlDQ+1abemIb1TKU7SQawM+Zs4PfAs8AnKh1PPy7XH5NVGx8DFqfH2WTto3cDzwB3ARNSeZEdQfUs8DuyozIqvhx9XPbXAXek10cADwJLgf8A6tLw+vR+aRp/RKXjPoTlnQ80p239n8D44bydgU8DTwGPAzcCdcNxOwM3k/WD7CGr+V3al+0KvC8t/1Lgkt7E4EtMmJkVXFGahszMrAtOBGZmBedEYGZWcE4EZmYF50RgZlZwTgRmA0jS6zqumGo2WDgRmJkVnBOBWSckvVvSg5IWS/pmuv/BVklfTtfIv1tSUyo7X9ID6frwt5VcO/5ISXdJelTSw5JeliY/uuS+AjelM2fNKsaJwKyMpGOA84HTI2I+0Aa8i+zCZ80RcSzwS7LrvwPcAPxtRBxPdrZnx/CbgK9HxAnAH5GdPQrZFWI/SnZvjCPIrqFjVjE1By9iVjivB04GHkp/1hvILvrVDtySynwX+KGkscC4iPhlGn498B+SxgDTI+I2gIjYCZCm92BErEzvF5Ndi/6+/BfLrHNOBGYHEnB9RFyx30Dp/5SV6+v1WXaVvG7D30OrMDcNmR3obuA8SYfB3vvHHk72fem48uU7gfsiYhOwQdKr0/D3AL+MiC3ASklvTdOokzRyQJfCrIf8T8SsTEQ8KekfgJ9LqiK7KuSHyG4Gc2oat4asHwGyywRfnX7onwMuScPfA3xT0mfSNN4+gIth1mO++qhZD0naGhGjKx2HWX9z05CZWcG5RmBmVnCuEZiZFZwTgZlZwTkRmJkVnBOBmVnBORGYmRXc/weiKBg+QiYRZAAAAABJRU5ErkJggg==\n"
          },
          "metadata": {
            "needs_background": "light"
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Let's also visualize how the weights changed at the end of each training epoch:"
      ],
      "metadata": {
        "id": "lvo1TmBMt-CM"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "plt.plot(weights_updates[:,0], label = \"w1\")\n",
        "plt.plot(weights_updates[:,1], label = \"w2\")\n",
        "plt.plot(weights_updates[:,2], label = \"w3\")\n",
        "plt.legend()\n",
        "plt.ylabel('weights')\n",
        "plt.xlabel(\"epochs\")\n",
        "plt.title(\"Weights over training epochs - batch gradient descent\")\n",
        "plt.show()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 295
        },
        "id": "7kiO2tnDuWZ-",
        "outputId": "e93891a4-e777-49c6-d626-c5dd6717e353"
      },
      "execution_count": 248,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ],
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYoAAAEWCAYAAAB42tAoAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3deXxU1fn48c8z2SFhh7BvgiDIIuACLUjFDVultdq6tFXrUlut1dba2tpq+6vfaq21Wm3VInWpVqvWSrXWBRAXkE1AdtkEInsCJCSETDLP749zgkNIJhOSmZvleec1r7lz7pl7n7tknrnn3jlXVBVjjDGmJqGgAzDGGNO4WaIwxhgTkyUKY4wxMVmiMMYYE5MlCmOMMTFZojDGGBOTJYo6EJFLReSNOOteLiLvJTqmpkhEVojIxIau29iJyOMi8pskzOcOEfl7EubTV0RURFITPa94iMhEEcmLeh3YvpOsbZ0szT5RiMitIvJalbK1NZRdFGtaqvq0qp7ZQHG9LSJXNcS0kqWhPhhUdaiqvt3QdU39NacvOA217yQr8SaL/x8eUJf3NPtEAbwDjBORFAAR6QakASdUKRvg6xrgaJNBY/l2aZo2248al5aQKBbgEsNI/3o8MAtYU6VsvapuFZG2IvKYiGwTkU9F5DdRCeWwb1sicqaIrBGRfSLyZxGZXfUoQUR+LyJ7RGSjiEz2ZXf6eT4oIvtF5EFx7hORnSJSKCLLROT46hZIRLqLyHQRKRCRdSJydVT5ARHpEFX3BBHZLSJp/vW3RWSVj+l1EekTVVdF5DoRWQusrWbWlYl0r497rF8n7/vY84E7ROQYEZkpIvl+3k+LSLuo+XwiIqf74TtE5J8i8qSIFPnmgjFHWXeUiCz2454XkediHf7HsS5uEJENfhnuEZGQHxcSkdtEZJPfXk+KSNuo935eROaIyF4R2SIil0fNtr2IvOpjnCcix/j3xL3945Tpl79IRD4UkRFR8f1URNb7cStF5Cu+/DjgYWCs3757fXmWiNzrl3efiLwnIllR87pURDb79fTzGOu7o4j8xy/fAnH/W9H/T0fsfyJyv1+HhSKySETGR9XPEtfEs0dEVgInVplf9L4TilrufL8fdfDjKo+UL6u6HCJyNvAz4Ot+nSytYdlO8Ou5SESeAzKrjP+SiCzx+8QcERkeNe4n4j5risR9nkzy5Ski8rOobbVIRHr5cYNF5E1xnwFrRORrUdN7XEQeqmE/q/wfXuqX5+s1ba/DqGqzf+ASw01++EHg28CdVcqm+eGXgEeA1kAXYD7wHT/ucuA9P9wJKATOB1KBHwBh4KqoumHgaiAF+C6wFRA//u3Kuv71WcAioB0gwHFAtxqW5x3gz35nHAnsAk7z42YCV0fVvQd42A9PAdb5aacCtwFzouoq8CbQAciqZr59fZ3UqLLLgXLg+36aWbijszOADKCzj/ePUe/5BDjdD98BlALn+PX0W+CDutYF0oFNfjuk+e1SBvymhnUYz7qY5ddFb+DjqG37bf/e/kA28C/gKT+uD1AEXOzj6AiM9OMeB/KBk/w8nwaerev2j2N/vwO3713gY7gZ2Aik+fEXAt1xXxS/DhRXzouofTxqeg/h9tcefr2P89u2cn/4q9/uI4CDwHE1xPWsf7QChgBboudFNfsf8A2/DlOBHwHbgUw/7i7gXV+/F7AcyKth3/kB8AHQ08f+CPCPKvt1tcvh1+ffY6zvyn3vJr++L/Dr/zd+/AnATuBkv/4u87FlAIP8eugeFcsxfvjHwDJfR3xcHXGfTVuAK/x6OQHYDQypbT+LWs8D6rRPBfHBneyH39Av+eGlwEDg7CpllwG5fgfJinrvxcCsqv9EwLeAuVH1xG+86ESxLmp8K7+BuvrXb3N4ojgN92F0ChCKsSy9gAogJ6rst8DjfvgqYGaVmCb4168BV0a9LwSUAH2idqDTYsy7L9Unis21rP8vA4ujXn/C4R/+b0WNGwIcqGtdYALwKT4R+7L3qDlRxLMuzo4a/z1ghh+eAXwvatwg3AdDKnBr5X5VzTwfB6ZGvT4HWF2X7V+H/f2DKsu2DRhfQ/0lwJSq+3jUew8AI2LsDz2jyuYDF1VTN8Wvo0FRZb/hyERR4/7n6+ypjAXYUGUbXUPNiWIVMClqXLeobRZzOag9UUwg6kugL5vDZ4niL8D/q/KeNcCpuC9VO4HT8Ym8Sp0p1czv68C7VcoeAW6vbT+LWs91ShQtoekJ3Dfaz/tDzc6quha3Icf5suN9nT64bwTb/CHiXtwG6FLNNLvjPoQBULcF8qrU2R41vsQPZlcXoKrOxB3ZPATsFJFHRaRNDfMtUNWiqLJNuG97AC/img664XbgCO5bF3757o9atgJcMukRNa0t1N1h7xGRXBF51h9OFwJ/xx2B1WR71HAJrtmkpjbqmup2Bz7126HauKqo67rY5OeBf95UZVwq7otGL2B9jPlWjT8b4t/+IjLeNxnsF5EVMeYTvW9GcPtmdz+Nb0U1g+zF7f81bZ9OuCPXOi9TFZ1x6yh6nVa3faruSzeLax7c52NtGxVrd47cRjXpA7wUtcyrcF+4cuu4HNWpbt+LjqUP8KPKefv598IdRawDbsQlo53+/6ZyP6tpX+oDnFxlepcCXRtgWarVUhLFXNwOdjXwPoCqFuK+BVwNbFXVjbid7iDQSVXb+UcbVR1azTS34Q5jAdfGHP06DnpEgeoDqjoa9035WNyhZ1VbgQ4ikhNV1hv3bRpV3QO8gfvWcQnukLNyXltwzWjtoh5ZqjonVlxxjKta/n++bJiqtsE1H0iM6TaEbUAPvx0q9YpRP551Ef3+3rh1j3/uU2VcObDDT/eYo1mAeLa/qr6rqtn+Ud1+eUTs4s6t9AS2ijsP81fgeqCjqrbDNdlUrreq23I3rrnvqJYpyi7cOor+H6lu+xyavz8fcQvwNaC9j3VfVKzbOHIb1WQLMLnK9s5U1U/jiD3W/0RlHFX3vehYtgB3Vpl3K1X9B4CqPqOqn8ftUwrcHfW+6tb7FmB2lellq+p341iWo9IiEoWqHgAWAj/ks2/X4Jomfog/Sauq23AfsveKSBt/AuwYETm1msm+CgwTkS/7b7TXcXhGr80OXBs3ACJyooicLO6kczHunzNSzbJswR0N/VZEMv1JsStx39orPYNrGrvAD1d6GLhVRIb6ebYVkQvrEPMuH1P/WurlAPuBfSLSg+oTXkObi/uGeL2IpIrIFFwbbU3iWRc/FpH2/gTiD4DnfPk/gJtEpJ+IZOMS43OqWo5rDz5dRL7m4+goIiOpRbzbvw5Gi8j5ft+8EfcF6ANc+7bitiUicgXuiKLSDqCniKTDoaORacAfxF0skSLuIoaMugSjqhW4czl3iEgrERmM20djycEll11Aqoj8Eog+yvonbhu2F5GeuPNkNXkYuNMnSkSks99H4rED6OsTbnXm+jhvEJE0ETmfw/e9vwLX+u0rItJaRL4oIjkiMkhETvPrsxTXzFe53acC/09EBvr3DReRjsArwLEi8k0/vzS//xxXh+Wp7X/4MC0iUXizcU1I0deIv+vLoi+L/Rbu5NRKXHvoC7j2zMOo6m7cScHf4U4cDcElo4NxxnM/cIG4KzYewP0D/NXPc5Of5j01vPdiXLvqVtzJ99tV9a2o8dNx52G2q+qhqzRU9SXct5VnfZPQcmBynPFWNp/dCbzvD3lPqaHqr4BRuG9/r+I+IBJKVctwJ7CvBPbijmJeoYbtEee6eBl3gnkJbjke8+XTgKdw+81G3D/49/10N+PahH+Ea85agjsJWZu6bP94vIw7qtwDfBM4X1XDqroSuBf34bYDGIY/yvZmAiuA7SKy25fdjDupusAv090c3WfH9bgj++249fcPYv+/vA78D3fuZhNuPUc3Nf3Kl2/EfcF7Ksa07sf9X7whIkW4pHlynHE/75/zReTDqiOj9r3Lcevn60Tt86q6ENdy8SBue6zzdcGd0L4Ld+S2Hfd5dKsf9wdcMnwDd+HMY7jzp0XAmcBFuM+A7bhtEm/yvgN4wv8Pf622yvDZFTimnvy3jTzgUlWdFXQ8BkRkHu6Kr78dxXsVGOjbkE0CiMjduIs7Lgs6FhNbSzqiaHAicpaItPOHjT/DtZ1+EHBYLZaInCoiXX2Tz2XAcNw3UtMIiLv2f7hvRjkJd/T3UtBxmdrZrx/rZyzuHEBlU9WX/fkQE4xBuEP11rhLJy/w551M45CDa27qjmv2uhfXRGYaOWt6MsYYE5M1PRljjImp2TU9derUSfv27Rt0GMYY06QsWrRot6p2rm5cs0sUffv2ZeHChUGHYYwxTYqI1PjLdmt6MsYYE5MlCmOMMTFZojDGGBNTsztHUZ1wOExeXh6lpaVBh1IvmZmZ9OzZk7S0tKBDMca0IC0iUeTl5ZGTk0Pfvn05vIPHpkNVyc/PJy8vj379+gUdjjGmBWkRTU+lpaV07NixySYJABGhY8eOTf6oyBjT9LSIRAE06SRRqTksgzGm6Qms6cn38f8k7g5TCjyqqvdXqSO47oHPwd2l6XJVPaKbX2NM9SIaoSJSQbmWUxGpoEIrKI+UU6EVR5RXaAWqSkQjRIiAcmj4ULlGUA4frhx3WLkqEY6sE12vsvugyvGVw4c9R3UxVFsdJUZd1cPG1zS96qZV9f01xVGtmKNqHhlzXIxul3Jb53LhsXW5xUx8gjxHUQ78SFU/9HdrWyQib/r+8itNxt1XYSCu7/i/EH8f8k3Kz3/+c5588kn27NnD/v37gw7H1ENEI5SWl3Kg/ECtj4MVBymrKKOsooxwJOyGI/51RfjQcFnEv44eHwkf+sCvKQHE/BAzTZbUcMPIYZ2HNa9E4Xv13OaHi0RkFe5+xdGJYgrwpL+V5we+S+9uzbFH0HPPPZfrr7+egQMHBh1Ki6eqlJSXUHCggIKDBewt3UthWSH7w/spKiuq9rE/vJ/CssJDCaCuQhIiPZROWkoa6aF00lPcIy2U5oZ9WVZa1qHh1FAqqZJKaiiVFElxz6EUUsU9p0jKEa8r6x5Rz5eFCCEiCEJI3HBIQofKQxIi5G/0VjlcWfdQfaqpE1VeOR3xf3B4s2rVsjrVQY5ooo2uU9P7ouvGmnZl9cPKiN0sXNOHem3va0waxVVPItIXOAGYV2VUDw6/o1WeLzssUYjINcA1AL17x7ptbnDuueceMjIyuOGGG7jppptYunQpM2fOZObMmTz22GM8/fTTQYfY7JVHytl9YDc7SnawvXg7O4p3sLNkJwWlLiEUHChgz8E9FBwooCxSVuN0MlMyyUnPISc9h+z0bNpmtqVXTi+y07NpldqKrLQsslKrf7RKbUVWahaZqZlkpWaRkZJx6EPfmMYq8L3T33P4ReBGVS08mmmo6qPAowBjxoyJeaz9q/+sYOXWo5pNjYZ0b8Pt58a6zz2MHz+ee++9lxtuuIGFCxdy8OBBwuEw7777LhMmTGjQeFqqiEbYUbyDLUVb2Fy0mc1Fm8krymNH8Q62l2xn94HdRPTw21BnpGTQMbMj7TPb0ymrEwPbDzz0ukNmB9pntqd9RnvaZLQhOy2bnPQc0lPSA1pCY4IRaKLwN5J/EXhaVau7r/KnQK+o1z19WZMzevRoFi1aRGFhIRkZGYwaNYqFCxfy7rvv8sADDwQdXpMSrgizYd8G1u1dx9o9a1m/d/2hpBB9JJAWSqNHdg+6te7GuO7jyG2VS27rXPfcKpeurbvSJr1Nkzn8NyYoQV71JLibha9S1T/UUG06cL2IPIs7ib2vvucnavvmnyhpaWn069ePxx9/nHHjxjF8+HBmzZrFunXrOO644wKJqSnYX7aflfkrWbZ7GasLVrN2z1o2FW6iXMsBSJVU+rTpQ7+2/ZjQcwK9cnrRu01veuf0JrdVLimhlICXwJimL8gjis8B3wSWicgSX/YzoDeAqj4M/Bd3aew63OWxVwQQZ4MZP348v//975k2bRrDhg3jhz/8IaNHj7ZvtJ6qsm7vOj7c8SHLdi9j+e7lbNi34dCVOz2yezCw/UBO630aA9oNYGD7gfRt05e0FOvSxJhECvKqp/cgxuUAro4C1yUnosQbP348d955J2PHjqV169ZkZmYyfvx4AG655RaeeeYZSkpK6NmzJ1dddRV33HFHsAEnmKqycd9G5m+fz/zt81m0YxEFpQUAdMjswLBOwzi739kM6zSMoR2H0i6zXcARG9MyNbt7Zo8ZM0ar3rho1apVzaZ5p6kvS2l5KfO3z2f2ltnMzpvNjpIdAOS2yuWkridxYtcTGdN1DD2ze9qRljFJJCKLVHVMdeMCv+rJNH+FZYXM3DyTtza9xbxt8yitKCUrNYtx3cdxbY9rObnryfTMscRgTGNlicIkxIHyA8zeMpvXNr7Gu5++SzgSpnvr7pw/8HxO7XkqY7qOsctMjWkiLFGYBrUyfyUvfPwCr254lZLyEjpndebrg77O5H6TGdZpmB01GNMEWaIw9Xag/ACvbHiFFz5+gZX5K8lMyeSsvmdx3jHnMTp3tF2iakwTZ4nCHLX8A/k8u+ZZnl39LHsP7mVg+4H87OSf8cX+X6RNepugwzPGNBBLFKbOtu7fytRlU5m+fjoHKw4ysddELh96OaO6jLKmJWOaIUsUjUBJSQkXXngh69evJyUlhXPPPZe77ror6LCOsKtkF49+9CgvrH0BQTjvmPO4bOhl9Gtrt2Y1pjmzRNFI3HzzzXzhC1+grKyMSZMm8dprrzF58uSgwwLc5a1TP5rKM6ufoSJSwZcHfpnvDP8OXVt3DTo0Y0wSWKJIkni7GU9PT2fUqFHk5eUFHDFURCr497p/88DiB9hTuodzjzmXa4dfS682vWp/szGm2Wh5ieK1n8L2ZQ07za7DYHLspqJ4uxnfu3cv//nPf/jBD37QsDHW0dJdS/m/ef/HyvyVjOoyiodPf5jjOjbdX4QbY45ey0sUAYmnm/Hy8nIuvvhibrjhBvr37x9InCXhEv60+E88veppOrfqzN3j72Zyv8l2ktqYFqzlJYpavvknSjzdjF9zzTUMHDiQG2+8MZAYF2xfwC/f/yV5+/O4aNBF3Dj6RlqntQ4kFmNM49HyEkWAYnUzftttt7Fv3z6mTp2a9LjCFWHu+/A+nlr5FL1yejHtrGmc2PXEpMdhjGmcQkEH0JKMHz+ebdu2MXbsWHJzcw91M56Xl8edd97JypUrGTVqFCNHjkxawthSuIVvvPYNnlr5FBcNuogXz3vRkoQx5jB2RJFEkyZNIhwOH3r98ccfHxoOorv3/33yP+6YcwchCfHHiX9kUp9JSY/BGNP4WaJogSIa4cHFD/LXZX9lROcR/G7C7+ie3T3osIwxjZQlihamOFzMT9/9KW9veZuvDvwqPz/553YrUWNMTJYoWpBt+7fxvRnfY+O+jdx60q1cPPhiu+zVGFMrSxQtxIa9G7jmzWsoCZfw8BkPc0q3U4IOyRjTRFiiaAE+2vUR35vxPdJCafzt7L8xqMOgoEMyxjQhdnlsMzd361yueuMq2qS34cnJT1qSMMbUmSWKRuLss89mxIgRDB06lGuvvZaKiop6T3Petnl8f+b36ZnTkycnP0mvHOvMzxhTd5YoGol//vOfLF26lOXLl7Nr1y6ef/75ek1vwfYFXD/jenrl9GLqmVPplNWpgSI1xrQ0liiS5J577jnU+d9NN93EaaedBsDMmTO59NJLadPG3Tq0vLycsrKyel2NtHjnYq6bcR09snsw9cypdMjsUP8FMMa0WC3uZPbd8+9mdcHqBp3m4A6D+clJP4lZJ55uxs866yzmz5/P5MmTueCCC44qlvV713PdjOvIbZXL1LOm0jGr41FNxxhjKtkRRZJU7WZ87Nixh7oZHz9+PACvv/4627Zt4+DBg8ycObPO89hevJ1r37qWjJQMHj7jYWtuMsY0iBZ3RFHbN/9EiaebcYDMzEymTJnCyy+/zBlnnBH39AvLCvnuW9+lqKyIx89+nB7ZPRKxGMaYFijQIwoRmSYiO0VkeQ3jJ4rIPhFZ4h+/THaMDamym/EJEyYwfvx4Hn74YU444QSKi4vZtm0b4M5RvPrqqwwePDju6ZZHyrn57Zv5ZN8n3DfxPgZ3iP+9xhhTm6Cbnh4Hzq6lzruqOtI/fp2EmBKmpm7Gi4uLOe+88xg+fDgjR46kS5cuXHvttXFP94+L/sjcbXP5xdhfMLb72AQugTGmJQq06UlV3xGRvkHGkEyxuhlfsGDBUU3zlQ2v8MTKJ7ho0EWcP/D8esdojDFVBX1EEY+xIrJURF4TkaHVVRCRa0RkoYgs3LVrV7LjC8yK/BXcMecOxuSO4ZaTbgk6HGNMM9XYE8WHQB9VHQH8Cfh3dZVU9VFVHaOqYzp37pzUAINSWFbIj97+Ee0z23PvxHtJC1lX4caYxGjUiUJVC1V1vx/+L5AmIkd1zWcQd5BraJXLoKr8eu6v2V68nXsm3GM/qDPGJFSjThQi0lX8T5RF5CRcvPl1nU5mZib5+flNOlmoKvn5+WRmZvKvtf/i9U9e5/oTrmdkl5FBh2aMaeYCPZktIv8AJgKdRCQPuB1IA1DVh4ELgO+KSDlwALhIj+LTvmfPnuTl5dHUz19kZmZSnlPOXf+7i1O6ncK3j/920CEZY1qAoK96uriW8Q8CD9Z3PpU/dmvqwpEwl7x6Ca3SWvHb8b8lJI36gNAY00y0uF9mN2VTl01ldcFq7v/C/dY9hzEmaewraROxpmANj370KOf0O4fTep8WdDjGmBbEEkUTEI6E+cX7v6BNehtuPenWoMMxxrQw1vTUBPxt+d9YVbCK+ybeR7vMdkGHY4xpYeyIopHbUrSFR5Y+wpl9zuT0PqcHHY4xpgWyRNGIqSp3zb+L1FAqt5xoXXQYY4JhiaIRm7VlFu/kvcP3Rn6P3Na5QYdjjGmhLFE0UgfKD3D3/LsZ0G4Alxx3SdDhGGNaMDuZ3UhNWz6NrcVbmXbWNOvwzxgTKDuiaIR2luzkiRVPcGafMzmx64lBh2OMaeEsUTRCDy15iHAkzI2jbww6FGOMsUTR2KwpWMNLa1/i4sEX0yunV9DhGGOMJYrG5r5F95Gdns13hn8n6FCMMQawRNGozNs2j/e3vs93hn+Hthltgw7HGGMASxSNhqry0JKH6NKqCxcNvijocIwx5hBLFI3E3G1zWbxzMVcPu5qMlIygwzHGmEMsUTQClUcTXVt35fyB5wcdjjHGHMYSRSPw/tb3+WjXR1w97GrSU9KDDscYYw5jiSJgqsqfl/yZ7q2785UBXwk6HGOMOYIlioDN3z6fZbuXceWwK0lLsa46jDGNjyWKgE1bPo2OmR2ZMmBK0KEYY0y1LFEEaHXBauZsncM3hnzDrnQyxjRaligCNG35NFqltuJrg74WdCjGGFMjSxQBySvK441P3uDCYy+kTXqboMMxxpgaWaIIyFMrn0JE+OaQbwYdijHGxGSJIgD7y/bz73X/ZnLfyXaLU2NMo2eJIgAvr3+ZkvISu8WpMaZJsESRZBGN8OzqZxneaTjHdzo+6HCMMaZWgSYKEZkmIjtFZHkN40VEHhCRdSLykYiMSnaMDe2DrR/wSeEn1kOsMabJSA14/o8DDwJP1jB+MjDQP04G/uKfm6xnVj9Dh8wOnNX3rKBDadFUFVVQPxxRUFyZG1/Ne6imsIa6rn7N846/bk0jqitqRPE1FFU396rPcQWida9TU1mV90qNsdZ9Wke9PNXUS0tLo22HztVFVy+BJgpVfUdE+saoMgV4Ut2e+4GItBORbqq6LSkBNrAtRVt4J+8drh6e3M7/IhGltLyCA2UVHAhXUBquoDQc4UDYlYUrIoQrIpRVKOHyyOGvKyKHyipfl1cZrlA3j4qIUh5RIuqGK58PG46q+1mZujJVIhGIqBv32Qc54D/EI6qHyrSmYV/3sGEOr5N4SjrlZBAmkzIyJEwGZW6YMBkSJo1y0ignlQrSqHDPUu6H3fOh8RI1HPWcQoSQREjBPQSNPSwRQkQI+dch/0hBo4ajnsWtLEH948hhaiivHI5v/OHlKZKUjdTsrEkdRNvb5jf4dIM+oqhND2BL1Os8X3ZYohCRa4BrAHr37p204OrqhY9fICQhvnZsfD+wq4goe0vKKCguY09JmKLSMEWl5RQdLD80vL/0s+Gi0nL2HyynNOwSQmUiOFgeaZD401NCpKUIaakh0lJCpKeESAnJZw8RQiEhJcRnw/45NRQiI/WzuiHx9Q4Nf1Y3JBASQQTAPQsgleWAuJF+nPhxrty/7VC58Nn0xL9Jot4bEkjRMrLKC8ksLySjvJCMihLSIyWklZeQVlFMWkUJ6eX+uaKY1IoDpJe759RIKamRUlIiZaRWHCQlcpDUyEH/IdpwKiSViKQSCaUSkTTUDyuCSgoqIfcg9NlrQp+VSxpKVD3x9aj6WkBSCEsIJHTo49xtkM8+0hG/hBKiMp24Mj8eQSs3RuX7Kt8To65KCKihXvT7j3BkmUrVsmrqVLeyj3jfke/VOGOoblpHzvNol+fwemltEnMVZWNPFHFR1UeBRwHGjBnTKL+KhCNhpq+fzvge40mjPSu27mNHYSnb9pWyY18pu/aXUVB8kD3FYfKLD1JQXMbeA+GY337TUoSczDSyM1LJyXSP7u0yyUpPJSstRFZaCpnpKWSluUdm5fNhZSHS/Qd/5Yd/Wqoc9jo1RUgNyaEP50ZPFQ4Wwv6dsH+HexTtgOKdUFIAB/Yc+QiX1D7dlAzIyIb0bMjIgaxsSM+FtCxIzYS0TEjNgtSMz8oOlVcO+/GpmRBKgxT/qG04lEqKCCmJX3vGHKGxJ4pPgV5Rr3v6skZtX0mYTQXFfJJfwuZ897xi7xx2p+/mzfl9+M+bbx5WPyTQoXX6ocegrjl+OIMOrdLokJ1B+1ZptMlMI9snhDaZaWSkhprOh3dDKt0He7fAvi3+eTPs3QyF23xi2AnlB458XygNWnWErPbu0a4PdBsJWe0+K8tqD5ltIaMNpLc+PDFY776mhWrsiWI6cL2IPIs7ib2vMZ2f2L3/IB9vL2L19iI+3lHEmh1FbNxdzN6S8GH1cttkEOr6Hum05aLhZ9O9XWu6tvvhA7cAABqKSURBVMmka1v36JydQWqKXal8mIP7IX+de+xeC7s/dsN7N7lEES01E9r2gjbdofcpkN0FsrtCdq4fzoWcXMhsV0OTgjEmlkAThYj8A5gIdBKRPOB2IA1AVR8G/gucA6wDSoArgonUJYWlW/ayxD9Wbi0kv7js0PgOrdM5NjebLw7rRt+OrendsZV77tCK/eUFnPHCCi4behk3jbbfThwmUgH562H7R7B9mXvsXAVFWz+rIyFo1xs6DoReJ0O7Xu51295uuHVnSwDGJFDQVz1dXMt4Ba5LUjjR82XD7mLmrM9n3oZ8Fm/ey6d7XVNGSkg4NjeHScd1YVDXNgzKzWFQ1xw6ZafX2Az09JqXqdAKux+2qjsi2DIftsyDrUtgx4rPmolCadDlOOh/KnQcAJ0GQqdjoX0/185vjAlEXIlCRI4B8lT1oIhMBIbjLlvdm8jgkmn/wXL++9E25m7IZ8763ewoPAhA1zaZjO7TnsvH9WVEr3Yc36MNrdLjz6+qyktrX2J07mj6tOmTqPAbp0gEdiyHjbNdYtgy351DANfu320kjLkCug6HrsNcUki1e4Yb09jE+4n3IjBGRAbgri56GXgG1yzULJSVR7jlxY/olJ3OKf07Mu6YTow7piN9Oraq1wnjxTsXs7loM98Z8Z0GjLYR25cH62fBhlmwYTaU7Hbl7ftC/4nQ6yTXfNRlCITsGh5jmoJ4E0VEVctF5CvAn1T1TyKyOJGBJVuH1unMunkifeuZGKp6dcOrZKVmcXrv0xtsmo2KqjtqWPUKrH7FDYM7gTxgEvT/gksQbboFGaUxph7iTRRhEbkYuAw415c1u2sF+3Vq3aDTC1eEeX3T60zsNZFWaa0adNqBUoVtS2HZ8y457PkEEHfF0Zm/gWMmuXMNdoLZmGYh3kRxBXAtcKeqbhSRfsBTiQureXh/6/vsO7iPL/b7YtChNIzCbfDRc7D0Wdi1yp187j8RPn8TDDrHXYpqjGl24k0UZ6jqDZUvfLIoTVBMzcarG16lXUY7xvUYF3QoRy8SgXVvwvy/wvoZoBF3juFL98HQr7gfqBljmrV4E8VlwP1Vyi6vpsx4xeFi3t7yNlMGTCEt1ARb6Q7shcV/hwV/dU1LOd1g/I9gxMXQ8ZigozPGJFHMROHPS1wC9BOR6VGjcoCCRAbW1M3cPJPSilK+2L+JNTsV7YC5f4IF0yBcDL1OgUm/hOPOsy4sjGmhajuimIPrqbUTcG9UeRHwUaKCag5e3fAq3Vt3Z0TnEUGHEp99efD+/bDoCYiE4fivwrjvQ7cmEr8xJmFiJgpV3QRsAsYmJ5zmYW/pXj7Y9gGXD72ckDTyPpxKCuCd37smJo24pqXP32TNS8aYQ+L9Zfb5wN1AFzjUgbyqapsExtZkzdoyiwqt4Iy+ZwQdSs3CpTD/EXjnXigrgpGXwKk/cX0oGWNMlHhPZv8OOFdVVyUymObizU1v0iO7B0M6DAk6lOqtfRNe/ZHrd2ngmXD6HZA7NOiojDGNVLyJYoclifgUlRUxd9tcLhl8SeO7V0ThVvjfT2Hly64n1m+97H4HYYwxMdR21VNld6cLReQ54N/AwcrxqvqvBMbWJM3Om015pJwz+jSiZidVWPQ4vPELd6L6tNtg3A3uTmvGGFOL2o4ozo0aLgHOjHqtgCWKKt7a9BZdsrowvPPwoENxirbD9O/D2jeg36lw7h+hQ/+gozLGNCG1XfUU2I2CmqKScAnvffoe5w88v3Fc7bTqPzD9Bnc/6Mm/gxOvhlAjiMsY06TEe9XTA9UU7wMWqurLDRtS0/Xup+9ysOJg8M1OFWF483b44CHofgJ85VHofGywMRljmqx4T2ZnAoOB5/3rrwIbgREi8gVVvTERwTU1MzbNoENmB0Z1GRVcEIVb4fkrYMsHcNI1cOaddjMgY0y9xJsohgOfU9UKABH5C/Au8HlgWYJia1LCkTDvbX2PSb0nkRLUDXm2LIBnL4ayEvjqYzDsgmDiMMY0K/EmivZANq65CaA10EFVK0TkYM1vazmW7FxCUVkRp/Y8NZgAVrwEL13rOu+77BXoMjiYOIwxzU5dfnC3RETexv0qewLwfyLSGngrQbE1Ke/kvUNqKJWx3ZPc24kqvHcfzPiV68DvomegdcfkxmCMadbiShSq+piI/Bc4yRf9TFW3+uEfJySyJmZ23mxOzD2R1mkNe5e8mCIReO0W10/TsAvhvAchLTN58zfGtAgxr5UUkcH+eRTQDdjiH119mQE2F25m476NnNoric1OFeXw8nUuSYy7Ac7/qyUJY0xC1HZE8UPgGg7vYrySAqc1eERN0Dt57wAwoceE5MywvAxevBJWTYcv3AYTbrb7UxtjEqa2H9xd45+/kJxwmqbZebPp37Y/vdr0SvzMKsLwwhWw+hU467cw9nuJn6cxpkWL62e6ItJKRG4TkUf964Ei8qXEhtY07C/bz8IdC5PT7BSpgH9/1yWJyfdYkjDGJEW8/Tn8DSgDxvnXnwK/SUhETczcbXMpj5Qn/rJYVXjlRlj2vOsW/ORrEjs/Y4zx4k0Ux6jq74AwgKqW4C6TrRcROVtE1ojIOhH5aTXjLxeRXSKyxD+uqu88G9qcrXPITstO/C1P37odPnwSJvzY3YHOGGOSJN7fUZSJSBbuBDYicgxR3Y0fDRFJAR4CzgDygAUiMl1VV1ap+pyqXl+feSWKqjLn0zmc1PUkUkPxrsqjsOAxdz/rE6+CL/w8cfMxxphqxHtEcTvwP6CXiDwNzABuqee8TwLWqeoGVS0DngWm1HOaSbW5aDNbi7cyrvu42isfrbVvwn9vhoFnwdl329VNxpikizdRXAa8CvwaeAYYo6pv13PePXC/yaiU58uq+qqIfCQiL4hIEi4rit/crXMBEvdr7O3L4PnLIfd4uGAapCTwqMUYY2oQb6J4DNeD7HnAn4BHROQHCYvqM/8B+qrqcOBN4InqKonINSKyUEQW7tq1KwlhOXO2zqFHdg965SQgf5UUwLOXQGZbuOSfkJHd8PMwxpg4xJUoVHUWcCfwC+CvwBjgu/Wc96dA9CdsT18WPd98Va08FzIVGF1DfI+q6hhVHdO5c+d6hhWfcCTM/O3zGdd9XMPfGztSAS9e5e5O97WnoE23hp2+McbUQbw3LpqB6zF2Lq578RNVdWc9570AGCgi/XAJ4iLgkirz7aaq2/zL84BV9Zxng1m+eznF4eLENDu9/VtYPwO+9EfoWW1uNMaYpIm30fsj3Lf543Fdje8VkbmqeuBoZ6yq5SJyPfA6kAJMU9UVIvJr3J3zpgM3iMh5QDlQAFx+tPNraHO2ziEkIU7qelLtletizf/gnXvghG/A6MsbdtrGGHMURFXjryySg/uwvhnoqqoZCYrrqI0ZM0YXLlyY8Plc+t9LQeHpLz7dcBMt3AZ/GQdte8CVb0JaVsNN2xhjYhCRRao6prpx8Xbhcb2IPAcsxl3COg2Y3HAhNi2FZYUs3728YZudIhHXPUf4AHx1miUJY0yjUZd7Zv8BWKSq5QmMp0n4cMeHRDTCyd1ObriJznsYNsyCL90HnY9tuOkaY0w9xXvjot8nOpCmZMH2BaSH0hneeXjDTHD7ctdFx6BzYPQVDTNNY4xpIPH+jsJEWbB9ASO6jCAjpQFO0VSE3b2us9rDeX+yX14bYxodSxR1VFhWyOqC1YzJrfacT93NeQB2LIMv/gFad2qYaRpjTAOyRFFHi3csRlFO7Hpi/Se2ey28fTcMmQLH2e09jDGNkyWKOmqw8xORCEy/wV3dNPmehgnOGGMSwHqZq6MFOxYwrPOw+p+fWPQ32DwHpjwEObkNE5wxxiSAHVHUQVFZEasLVte/2ak4H2b8CvpNgJGXNkxwxhiTIJYo6mDxzsVENMKJufVMFDP/Hxzc75qc7ConY0wjZ4miDhZsX0BaKK1+5ye2LoFFj8PJ34EugxssNmOMSRRLFHWwcPtChnUaRmZq5tFNQBVeuwVadYRTf9KwwRljTIJYoohTSbiEVQWrGJ1bj26/l78IW+bB6XdAVruGCs0YYxLKEkWcVuSvoEIrGNll5NFNoPygO4HddZidwDbGNCl2eWycFu9cDMCIziOObgIL/wZ7N8M3XoSQ5WdjTNNhn1hxWrJzCf3b9qdtRtu6v7m0EN75nbsc9phJDR+cMcYkkCWKOEQ0wtJdSzmhywlHN4G5D0JJvjs3YZfDGmOaGEsUcfhk3ycUlhUeXbPT/l0w50EY8mXoYfe/NsY0PZYo4lB5fuKojijm/gnKD8BptzVwVMYYkxyWKOKwZNcS2mW0o0+bPnV7Y3E+zJ8Kx38VOg1MTHDGGJNglijisGTnEkZ2HonU9fzCBw9BuATG35yYwIwxJgksUdRiT+kePin8pO6/nygpgHmPuntNWFcdxpgmzBJFLZbuWgpQ90Qx72EoK4IJP05AVMYYkzyWKGqxZOcSUkOpDO04NP43HSxyiWLwl6Dr8YkLzhhjksASRS2W7V7GoPaD6tYR4OK/Q+k++NyNiQvMGGOSxBJFDBGNsCJ/Bcd3qsNRQUU5fPBn6HUK9GqA+2obY0zALFHE8Mm+TygOF9ctUaz+j+vTadz1iQvMGGOSyBJFDMvzlwNwfMc4E4UqzPkTtO8Hg85JYGTGGJM8gSYKETlbRNaIyDoR+Wk14zNE5Dk/fp6I9E1mfMt2LaNVaiv6te0X3xs2fwCfLoKx10EoJbHBGWNMkgSWKEQkBXgImAwMAS4WkSFVql0J7FHVAcB9wN3JjHFF/gqGdBxCSrwf+h/8GTLbwchLEhuYMcYkUZBHFCcB61R1g6qWAc8CU6rUmQI84YdfACZJnX8efXTCFWFWF6yO//xE4VZY/SqM+hakt05scMYYk0RBJooewJao13m+rNo6qloO7AM6JiO4j/d8TDgSjj9RfPgkaAWMuSKxgRljTJI1i5PZInKNiCwUkYW7du1qkGku3+1PZMeTKCrCsOhxGHA6dOjfIPM3xpjGIshE8SnQK+p1T19WbR0RSQXaAvlVJ6Sqj6rqGFUd07lz5wYJbnn+ctpntKd76+61V17zGhRtgzFXNsi8jTGmMQkyUSwABopIPxFJBy4CplepMx24zA9fAMxUVU1GcMt3L2dop6Hx9Ri7YCq07QXHnpX4wIwxJskCSxT+nMP1wOvAKuCfqrpCRH4tIuf5ao8BHUVkHfBD4IhLaBOhOFzM+r3rGdZpWO2Vd6+DjbNh9OV2SawxpllKDXLmqvpf4L9Vyn4ZNVwKXJjsuFbmr0TR+M5PfPgEhFLhhG8mPjBjjAlAsziZ3dBW5a8CYEjHqj/rqKKiHD56DgaeBTm5SYjMGGOSzxJFNVYXrKZzVmc6ZXWKXXH9DNi/w35gZ4xp1ixRVGNVwSoGd4jjrnSL/w6tOtlJbGNMs2aJoorS8lI27ttYe6IoKXCXxQ7/GqSkJSc4Y4wJgCWKKtbtXUeFVtSeKJY9D5EwjLw0OYEZY0xALFFUsarAncg+rsNxsSsu/jt0HW63OjXGNHuWKKpYnb+a7LRseuRU7XYqyo6VsP0jO5owxrQIliiqWL1nNYM6DCIkMVbN8hdBQnD8+ckLzBhjAmKJIkpFpIK1e9bGbnZSheUvQL9TIbtL8oIzxpiAWKKIsqlwEwfKD8Q+kb31Q9jzCRz/1aTFZYwxQbJEEaXyRHbMRLH8XxBKg+O+lKSojDEmWJYooqwuWE1aKI3+7Wq4p0Qk4hLFwDMgq31ygzPGmIBYooiyqmAVA9oNIC1Uww/oNs+Foq3W7GSMaVEsUXiqypqCNRzXMcaJ7OUvQlorGDQ5eYEZY0zALFF4O0p2sPfg3prPT0QqYNV0169TeuvkBmeMMQEK9H4UjUnnrM68POVl2ma0rb7ClnlQvAuOO6/68cYY00xZovBSQik1n8QGWPUKpGS4E9nGGNOCWNNTPFRh9X+g/0TIyAk6GmOMSSpLFPHYvgz2brbfThhjWiRLFPFY/Yrr22nQOUFHYowxSWeJIh6rXoHeY6F1LbdGNcaYZsgSRW0KNsDOFTDYmp2MMS2TJYrarHrFPQ/+YrBxGGNMQCxR1Obj1yF3GLTvE3QkxhgTCEsUsZTuc/07HXtm0JEYY0xgLFHEsn4WaAUMtERhjGm5LFHEsvZNyGwHPcYEHYkxxgTGEkVNIhFY+wYMmAQp1tOJMablCiRRiEgHEXlTRNb652rvAiQiFSKyxD+mJzXI7UuheKc1OxljWrygjih+CsxQ1YHADP+6OgdUdaR/JLfb1rVvAgLHTErqbI0xprEJKlFMAZ7ww08AXw4ojpqtfQN6jILszkFHYowxgQoqUeSq6jY/vB3IraFepogsFJEPRCR5yaQ4H/IWWrOTMcaQwPtRiMhbQNdqRv08+oWqqohoDZPpo6qfikh/YKaILFPV9dXM6xrgGoDevXvXM3Jg/UxAYYDde8IYYxKWKFT19JrGicgOEemmqttEpBuws4ZpfOqfN4jI28AJwBGJQlUfBR4FGDNmTE1JJ34b3oas9tB9ZL0nZYwxTV1QTU/Tgcv88GXAy1UriEh7Ecnww52AzwErEx6ZqksUfcdDKCXhszPGmMYuqERxF3CGiKwFTvevEZExIjLV1zkOWCgiS4FZwF2qmvhEkb8eCvPc3eyMMcYEc89sVc0HjrjuVFUXAlf54TnAsCSHBhtmuef+E5M+a2OMaYzsl9lVbZwNbXtBh/5BR2KMMY2CJYpokQrY+A70PxVEgo7GGGMaBUsU0bYtcV2L9/9C0JEYY0yjYYki2oa33XO/CYGGYYwxjYklimgbZkOXoZDdJehIjDGm0bBEUSl8ADZ/YFc7GWNMFZYoKpXugyHnwbFnBR2JMcY0KnZHnko5XeGrU2uvZ4wxLYwdURhjjInJEoUxxpiYLFEYY4yJyRKFMcaYmCxRGGOMickShTHGmJgsURhjjInJEoUxxpiYRLX+t5huTERkF7CpHpPoBOxuoHCaipa2zC1tecGWuaWozzL3UdXO1Y1odomivkRkoaqOCTqOZGppy9zSlhdsmVuKRC2zNT0ZY4yJyRKFMcaYmCxRHOnRoAMIQEtb5pa2vGDL3FIkZJntHIUxxpiY7IjCGGNMTJYojDHGxGSJwhORs0VkjYisE5GfBh1PQxGRXiIyS0RWisgKEfmBL+8gIm+KyFr/3N6Xi4g84NfDRyIyKtglODoikiIii0XkFf+6n4jM88v1nIik+/IM/3qdH983yLjrQ0TaicgLIrJaRFaJyNjmvJ1F5Ca/Ty8XkX+ISGZz3M4iMk1EdorI8qiyOm9XEbnM118rIpfVJQZLFLgPFeAhYDIwBLhYRIYEG1WDKQd+pKpDgFOA6/yy/RSYoaoDgRn+Nbh1MNA/rgH+kvyQG8QPgFVRr+8G7lPVAcAe4EpffiWwx5ff5+s1VfcD/1PVwcAI3PI3y+0sIj2AG4Axqno8kAJcRPPczo8DZ1cpq9N2FZEOwO3AycBJwO2VySUuqtriH8BY4PWo17cCtwYdV4KW9WXgDGAN0M2XdQPW+OFHgIuj6h+q11QeQE//z3Ma8AoguF+rplbd3sDrwFg/nOrrSdDLcBTL3BbYWDX25rqdgR7AFqCD326vAGc11+0M9AWWH+12BS4GHokqP6xebQ87onAqd7pKeb6sWfGH2ycA84BcVd3mR20Hcv1wc1gXfwRuASL+dUdgr6qW+9fRy3Roef34fb5+U9MP2AX8zTe5TRWR1jTT7ayqnwK/BzYD23DbbRHNfztXqut2rdf2tkTRQohINvAicKOqFkaPU/cVo1lcJy0iXwJ2quqioGNJslRgFPAXVT0BKOaz5gig2W3n9sAUXILsDrTmyOaZFiEZ29UShfMp0CvqdU9f1iyISBouSTytqv/yxTtEpJsf3w3Y6cub+rr4HHCeiHwCPItrfrofaCciqb5O9DIdWl4/vi2Qn8yAG0gekKeq8/zrF3CJo7lu59OBjaq6S1XDwL9w2765b+dKdd2u9dreliicBcBAf8VEOu6k2PSAY2oQIiLAY8AqVf1D1KjpQOWVD5fhzl1Uln/LXz1xCrAv6hC30VPVW1W1p6r2xW3Hmap6KTALuMBXq7q8levhAl+/yX3rVtXtwBYRGeSLJgEraabbGdfkdIqItPL7eOXyNuvtHKWu2/V14EwRae+Pxs70ZfEJ+iRNY3kA5wAfA+uBnwcdTwMu1+dxh6UfAUv84xxc++wMYC3wFtDB1xfcFWDrgWW4q0oCX46jXPaJwCt+uD8wH1gHPA9k+PJM/3qdH98/6LjrsbwjgYV+W/8baN+ctzPwK2A1sBx4CshojtsZ+AfuPEwYd+R45dFsV+DbfvnXAVfUJQbrwsMYY0xM1vRkjDEmJksUxhhjYrJEYYwxJiZLFMYYY2KyRGGMMSYmSxTGNAIiMrGyp1tjGhtLFMYYY2KyRGFMHYjIN0RkvogsEZFH/H0v9ovIff7eCDNEpLOvO1JEPvD3BXgp6p4BA0TkLRFZKiIfisgxfvLZUfeTeNr/4hgRuUvc/UQ+EpHfB7TopgWzRGFMnETkOODrwOdUdSRQAVyK65BuoaoOBWbj+v0HeBL4iaoOx/1KtrL8aeAhVR0BjMP96hZcz7434u6J0h/4nIh0BL4CDPXT+U1il9KYI1miMCZ+k4DRwAIRWeJf98d1Z/6cr/N34PMi0hZop6qzffkTwAQRyQF6qOpLAKpaqqolvs58Vc1T1Qiuq5W+uO6wS4HHROR8oLKuMUljicKY+AnwhKqO9I9BqnpHNfWOtl+cg1HDFbgb8JTj7kj2AvAl4H9HOW1jjpolCmPiNwO4QES6wKH7FvfB/R9V9lh6CfCequ4D9ojIeF/+TWC2qhYBeSLyZT+NDBFpVdMM/X1E2qrqf4GbcLc4NSapUmuvYowBUNWVInIb8IaIhHC9eV6Hu0nQSX7cTtx5DHDdPz/sE8EG4Apf/k3gERH5tZ/GhTFmmwO8LCKZuCOaHzbwYhlTK+s91ph6EpH9qpoddBzGJIo1PRljjInJjiiMMcbEZEcUxhhjYrJEYYwxJiZLFMYYY2KyRGGMMSYmSxTGGGNi+v9CF9IMM5fnzgAAAABJRU5ErkJggg==\n"
          },
          "metadata": {
            "needs_background": "light"
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Now let's take a look at the final weights of the model:"
      ],
      "metadata": {
        "id": "d535_0j4krGW"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "weights"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "L1p0GRRij-MD",
        "outputId": "703f8a1b-fa34-4334-dc46-ce0236297f45"
      },
      "execution_count": 249,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([1.00029435, 1.00010821, 2.02067723])"
            ]
          },
          "metadata": {},
          "execution_count": 249
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Out of curiosity, let's run linear regression on our data and see how the coefficients obtained with the regression compare to what we got with our implementation of gradient descent."
      ],
      "metadata": {
        "id": "R_-zUgF4lYBX"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "from sklearn.linear_model import LinearRegression\n",
        "\n",
        "reg = LinearRegression().fit(X, y)\n",
        "reg.coef_"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "abgGdSSslXET",
        "outputId": "ae1df1bf-c870-42ed-ef5e-b3e05bc240cc"
      },
      "execution_count": 230,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([1.00032104, 1.00020243, 2.02069302])"
            ]
          },
          "metadata": {},
          "execution_count": 230
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "If we compare them to the coefficients we used to simulate this dataset, we can see that they are indeed very close to one another!"
      ],
      "metadata": {
        "id": "MHrS6eKYktt3"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "model_results = pd.DataFrame(np.array([weights, reg.coef_, coeffs]).transpose(), columns = ['batch gradient descent','linear regression','true coeffs'])\n",
        "model_results"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 143
        },
        "id": "uKQBFeMpkGEB",
        "outputId": "4e5511f1-9af1-4e39-baf2-dce704d0484f"
      },
      "execution_count": 232,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "   batch gradient descent  linear regression  true coeffs\n",
              "0                1.000242           1.000321          1.0\n",
              "1                1.000179           1.000202          1.0\n",
              "2                2.020670           2.020693          2.0"
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-fcca40cc-af73-41a2-9ccb-9cc6af5b994a\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>batch gradient descent</th>\n",
              "      <th>linear regression</th>\n",
              "      <th>true coeffs</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>1.000242</td>\n",
              "      <td>1.000321</td>\n",
              "      <td>1.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>1.000179</td>\n",
              "      <td>1.000202</td>\n",
              "      <td>1.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>2.020670</td>\n",
              "      <td>2.020693</td>\n",
              "      <td>2.0</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-fcca40cc-af73-41a2-9ccb-9cc6af5b994a')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "        \n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "      \n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-fcca40cc-af73-41a2-9ccb-9cc6af5b994a button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-fcca40cc-af73-41a2-9ccb-9cc6af5b994a');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n",
              "  "
            ]
          },
          "metadata": {},
          "execution_count": 232
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Example - SGD\n",
        "\n",
        "Now we'll implement SGD. We will adapt the code from earlier, except now each batch is of size 1 ($m=1$). "
      ],
      "metadata": {
        "id": "gXVyiMtMm8SS"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# define batch size\n",
        "m = 1\n",
        "\n",
        "# reinitialize weights\n",
        "weights = np.random.randn(3)\n",
        "\n",
        "# collect avg cost at each epoch\n",
        "avg_cost_list = []\n",
        "\n",
        "# collect the updated weights at each epoch\n",
        "weights_updates = np.zeros((epochs+1, 3))\n",
        "weights_updates[0,:] = weights\n",
        "\n",
        "start = time.time()\n",
        "for iter in range(epochs):\n",
        "    print('epoch = %d' % iter)\n",
        "    \n",
        "    # collect losses for every batch in epoch\n",
        "    epoch_loss = []\n",
        "\n",
        "    for batch in range(X.shape[0]//m):\n",
        "        # predict on training set \n",
        "        y_pred = pred(X[batch,:], weights)\n",
        "\n",
        "        # calculate cost\n",
        "        cost = mse(y_pred, y[batch], m)\n",
        "        epoch_loss.append(cost)\n",
        "        \n",
        "        # update the weights\n",
        "        for i in range(weights.shape[0]):\n",
        "            weights[i] = weights[i] - lr*grad(m, y_pred, y[batch], X[batch,i])\n",
        "    \n",
        "    # save avg cost for epoch across all batches\n",
        "    avg_cost_list.append(np.average(epoch_loss))\n",
        "    weights_updates[iter+1,:] = weights\n",
        "\n",
        "end = time.time()\n",
        "\n",
        "sgd_time = end - start"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "IGEbhG8bk_8o",
        "outputId": "c2bd1a44-ad24-41cd-d48c-e50106b46cfe"
      },
      "execution_count": 233,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "epoch = 0\n",
            "epoch = 1\n",
            "epoch = 2\n",
            "epoch = 3\n",
            "epoch = 4\n",
            "epoch = 5\n",
            "epoch = 6\n",
            "epoch = 7\n",
            "epoch = 8\n",
            "epoch = 9\n",
            "epoch = 10\n",
            "epoch = 11\n",
            "epoch = 12\n",
            "epoch = 13\n",
            "epoch = 14\n",
            "epoch = 15\n",
            "epoch = 16\n",
            "epoch = 17\n",
            "epoch = 18\n",
            "epoch = 19\n",
            "epoch = 20\n",
            "epoch = 21\n",
            "epoch = 22\n",
            "epoch = 23\n",
            "epoch = 24\n",
            "epoch = 25\n",
            "epoch = 26\n",
            "epoch = 27\n",
            "epoch = 28\n",
            "epoch = 29\n",
            "epoch = 30\n",
            "epoch = 31\n",
            "epoch = 32\n",
            "epoch = 33\n",
            "epoch = 34\n",
            "epoch = 35\n",
            "epoch = 36\n",
            "epoch = 37\n",
            "epoch = 38\n",
            "epoch = 39\n",
            "epoch = 40\n",
            "epoch = 41\n",
            "epoch = 42\n",
            "epoch = 43\n",
            "epoch = 44\n",
            "epoch = 45\n",
            "epoch = 46\n",
            "epoch = 47\n",
            "epoch = 48\n",
            "epoch = 49\n",
            "epoch = 50\n",
            "epoch = 51\n",
            "epoch = 52\n",
            "epoch = 53\n",
            "epoch = 54\n",
            "epoch = 55\n",
            "epoch = 56\n",
            "epoch = 57\n",
            "epoch = 58\n",
            "epoch = 59\n",
            "epoch = 60\n",
            "epoch = 61\n",
            "epoch = 62\n",
            "epoch = 63\n",
            "epoch = 64\n",
            "epoch = 65\n",
            "epoch = 66\n",
            "epoch = 67\n",
            "epoch = 68\n",
            "epoch = 69\n",
            "epoch = 70\n",
            "epoch = 71\n",
            "epoch = 72\n",
            "epoch = 73\n",
            "epoch = 74\n",
            "epoch = 75\n",
            "epoch = 76\n",
            "epoch = 77\n",
            "epoch = 78\n",
            "epoch = 79\n",
            "epoch = 80\n",
            "epoch = 81\n",
            "epoch = 82\n",
            "epoch = 83\n",
            "epoch = 84\n",
            "epoch = 85\n",
            "epoch = 86\n",
            "epoch = 87\n",
            "epoch = 88\n",
            "epoch = 89\n",
            "epoch = 90\n",
            "epoch = 91\n",
            "epoch = 92\n",
            "epoch = 93\n",
            "epoch = 94\n",
            "epoch = 95\n",
            "epoch = 96\n",
            "epoch = 97\n",
            "epoch = 98\n",
            "epoch = 99\n",
            "epoch = 100\n",
            "epoch = 101\n",
            "epoch = 102\n",
            "epoch = 103\n",
            "epoch = 104\n",
            "epoch = 105\n",
            "epoch = 106\n",
            "epoch = 107\n",
            "epoch = 108\n",
            "epoch = 109\n",
            "epoch = 110\n",
            "epoch = 111\n",
            "epoch = 112\n",
            "epoch = 113\n",
            "epoch = 114\n",
            "epoch = 115\n",
            "epoch = 116\n",
            "epoch = 117\n",
            "epoch = 118\n",
            "epoch = 119\n",
            "epoch = 120\n",
            "epoch = 121\n",
            "epoch = 122\n",
            "epoch = 123\n",
            "epoch = 124\n",
            "epoch = 125\n",
            "epoch = 126\n",
            "epoch = 127\n",
            "epoch = 128\n",
            "epoch = 129\n",
            "epoch = 130\n",
            "epoch = 131\n",
            "epoch = 132\n",
            "epoch = 133\n",
            "epoch = 134\n",
            "epoch = 135\n",
            "epoch = 136\n",
            "epoch = 137\n",
            "epoch = 138\n",
            "epoch = 139\n",
            "epoch = 140\n",
            "epoch = 141\n",
            "epoch = 142\n",
            "epoch = 143\n",
            "epoch = 144\n",
            "epoch = 145\n",
            "epoch = 146\n",
            "epoch = 147\n",
            "epoch = 148\n",
            "epoch = 149\n",
            "epoch = 150\n",
            "epoch = 151\n",
            "epoch = 152\n",
            "epoch = 153\n",
            "epoch = 154\n",
            "epoch = 155\n",
            "epoch = 156\n",
            "epoch = 157\n",
            "epoch = 158\n",
            "epoch = 159\n",
            "epoch = 160\n",
            "epoch = 161\n",
            "epoch = 162\n",
            "epoch = 163\n",
            "epoch = 164\n",
            "epoch = 165\n",
            "epoch = 166\n",
            "epoch = 167\n",
            "epoch = 168\n",
            "epoch = 169\n",
            "epoch = 170\n",
            "epoch = 171\n",
            "epoch = 172\n",
            "epoch = 173\n",
            "epoch = 174\n",
            "epoch = 175\n",
            "epoch = 176\n",
            "epoch = 177\n",
            "epoch = 178\n",
            "epoch = 179\n",
            "epoch = 180\n",
            "epoch = 181\n",
            "epoch = 182\n",
            "epoch = 183\n",
            "epoch = 184\n",
            "epoch = 185\n",
            "epoch = 186\n",
            "epoch = 187\n",
            "epoch = 188\n",
            "epoch = 189\n",
            "epoch = 190\n",
            "epoch = 191\n",
            "epoch = 192\n",
            "epoch = 193\n",
            "epoch = 194\n",
            "epoch = 195\n",
            "epoch = 196\n",
            "epoch = 197\n",
            "epoch = 198\n",
            "epoch = 199\n",
            "epoch = 200\n",
            "epoch = 201\n",
            "epoch = 202\n",
            "epoch = 203\n",
            "epoch = 204\n",
            "epoch = 205\n",
            "epoch = 206\n",
            "epoch = 207\n",
            "epoch = 208\n",
            "epoch = 209\n",
            "epoch = 210\n",
            "epoch = 211\n",
            "epoch = 212\n",
            "epoch = 213\n",
            "epoch = 214\n",
            "epoch = 215\n",
            "epoch = 216\n",
            "epoch = 217\n",
            "epoch = 218\n",
            "epoch = 219\n",
            "epoch = 220\n",
            "epoch = 221\n",
            "epoch = 222\n",
            "epoch = 223\n",
            "epoch = 224\n",
            "epoch = 225\n",
            "epoch = 226\n",
            "epoch = 227\n",
            "epoch = 228\n",
            "epoch = 229\n",
            "epoch = 230\n",
            "epoch = 231\n",
            "epoch = 232\n",
            "epoch = 233\n",
            "epoch = 234\n",
            "epoch = 235\n",
            "epoch = 236\n",
            "epoch = 237\n",
            "epoch = 238\n",
            "epoch = 239\n",
            "epoch = 240\n",
            "epoch = 241\n",
            "epoch = 242\n",
            "epoch = 243\n",
            "epoch = 244\n",
            "epoch = 245\n",
            "epoch = 246\n",
            "epoch = 247\n",
            "epoch = 248\n",
            "epoch = 249\n",
            "epoch = 250\n",
            "epoch = 251\n",
            "epoch = 252\n",
            "epoch = 253\n",
            "epoch = 254\n",
            "epoch = 255\n",
            "epoch = 256\n",
            "epoch = 257\n",
            "epoch = 258\n",
            "epoch = 259\n",
            "epoch = 260\n",
            "epoch = 261\n",
            "epoch = 262\n",
            "epoch = 263\n",
            "epoch = 264\n",
            "epoch = 265\n",
            "epoch = 266\n",
            "epoch = 267\n",
            "epoch = 268\n",
            "epoch = 269\n",
            "epoch = 270\n",
            "epoch = 271\n",
            "epoch = 272\n",
            "epoch = 273\n",
            "epoch = 274\n",
            "epoch = 275\n",
            "epoch = 276\n",
            "epoch = 277\n",
            "epoch = 278\n",
            "epoch = 279\n",
            "epoch = 280\n",
            "epoch = 281\n",
            "epoch = 282\n",
            "epoch = 283\n",
            "epoch = 284\n",
            "epoch = 285\n",
            "epoch = 286\n",
            "epoch = 287\n",
            "epoch = 288\n",
            "epoch = 289\n",
            "epoch = 290\n",
            "epoch = 291\n",
            "epoch = 292\n",
            "epoch = 293\n",
            "epoch = 294\n",
            "epoch = 295\n",
            "epoch = 296\n",
            "epoch = 297\n",
            "epoch = 298\n",
            "epoch = 299\n",
            "epoch = 300\n",
            "epoch = 301\n",
            "epoch = 302\n",
            "epoch = 303\n",
            "epoch = 304\n",
            "epoch = 305\n",
            "epoch = 306\n",
            "epoch = 307\n",
            "epoch = 308\n",
            "epoch = 309\n",
            "epoch = 310\n",
            "epoch = 311\n",
            "epoch = 312\n",
            "epoch = 313\n",
            "epoch = 314\n",
            "epoch = 315\n",
            "epoch = 316\n",
            "epoch = 317\n",
            "epoch = 318\n",
            "epoch = 319\n",
            "epoch = 320\n",
            "epoch = 321\n",
            "epoch = 322\n",
            "epoch = 323\n",
            "epoch = 324\n",
            "epoch = 325\n",
            "epoch = 326\n",
            "epoch = 327\n",
            "epoch = 328\n",
            "epoch = 329\n",
            "epoch = 330\n",
            "epoch = 331\n",
            "epoch = 332\n",
            "epoch = 333\n",
            "epoch = 334\n",
            "epoch = 335\n",
            "epoch = 336\n",
            "epoch = 337\n",
            "epoch = 338\n",
            "epoch = 339\n",
            "epoch = 340\n",
            "epoch = 341\n",
            "epoch = 342\n",
            "epoch = 343\n",
            "epoch = 344\n",
            "epoch = 345\n",
            "epoch = 346\n",
            "epoch = 347\n",
            "epoch = 348\n",
            "epoch = 349\n",
            "epoch = 350\n",
            "epoch = 351\n",
            "epoch = 352\n",
            "epoch = 353\n",
            "epoch = 354\n",
            "epoch = 355\n",
            "epoch = 356\n",
            "epoch = 357\n",
            "epoch = 358\n",
            "epoch = 359\n",
            "epoch = 360\n",
            "epoch = 361\n",
            "epoch = 362\n",
            "epoch = 363\n",
            "epoch = 364\n",
            "epoch = 365\n",
            "epoch = 366\n",
            "epoch = 367\n",
            "epoch = 368\n",
            "epoch = 369\n",
            "epoch = 370\n",
            "epoch = 371\n",
            "epoch = 372\n",
            "epoch = 373\n",
            "epoch = 374\n",
            "epoch = 375\n",
            "epoch = 376\n",
            "epoch = 377\n",
            "epoch = 378\n",
            "epoch = 379\n",
            "epoch = 380\n",
            "epoch = 381\n",
            "epoch = 382\n",
            "epoch = 383\n",
            "epoch = 384\n",
            "epoch = 385\n",
            "epoch = 386\n",
            "epoch = 387\n",
            "epoch = 388\n",
            "epoch = 389\n",
            "epoch = 390\n",
            "epoch = 391\n",
            "epoch = 392\n",
            "epoch = 393\n",
            "epoch = 394\n",
            "epoch = 395\n",
            "epoch = 396\n",
            "epoch = 397\n",
            "epoch = 398\n",
            "epoch = 399\n",
            "epoch = 400\n",
            "epoch = 401\n",
            "epoch = 402\n",
            "epoch = 403\n",
            "epoch = 404\n",
            "epoch = 405\n",
            "epoch = 406\n",
            "epoch = 407\n",
            "epoch = 408\n",
            "epoch = 409\n",
            "epoch = 410\n",
            "epoch = 411\n",
            "epoch = 412\n",
            "epoch = 413\n",
            "epoch = 414\n",
            "epoch = 415\n",
            "epoch = 416\n",
            "epoch = 417\n",
            "epoch = 418\n",
            "epoch = 419\n",
            "epoch = 420\n",
            "epoch = 421\n",
            "epoch = 422\n",
            "epoch = 423\n",
            "epoch = 424\n",
            "epoch = 425\n",
            "epoch = 426\n",
            "epoch = 427\n",
            "epoch = 428\n",
            "epoch = 429\n",
            "epoch = 430\n",
            "epoch = 431\n",
            "epoch = 432\n",
            "epoch = 433\n",
            "epoch = 434\n",
            "epoch = 435\n",
            "epoch = 436\n",
            "epoch = 437\n",
            "epoch = 438\n",
            "epoch = 439\n",
            "epoch = 440\n",
            "epoch = 441\n",
            "epoch = 442\n",
            "epoch = 443\n",
            "epoch = 444\n",
            "epoch = 445\n",
            "epoch = 446\n",
            "epoch = 447\n",
            "epoch = 448\n",
            "epoch = 449\n",
            "epoch = 450\n",
            "epoch = 451\n",
            "epoch = 452\n",
            "epoch = 453\n",
            "epoch = 454\n",
            "epoch = 455\n",
            "epoch = 456\n",
            "epoch = 457\n",
            "epoch = 458\n",
            "epoch = 459\n",
            "epoch = 460\n",
            "epoch = 461\n",
            "epoch = 462\n",
            "epoch = 463\n",
            "epoch = 464\n",
            "epoch = 465\n",
            "epoch = 466\n",
            "epoch = 467\n",
            "epoch = 468\n",
            "epoch = 469\n",
            "epoch = 470\n",
            "epoch = 471\n",
            "epoch = 472\n",
            "epoch = 473\n",
            "epoch = 474\n",
            "epoch = 475\n",
            "epoch = 476\n",
            "epoch = 477\n",
            "epoch = 478\n",
            "epoch = 479\n",
            "epoch = 480\n",
            "epoch = 481\n",
            "epoch = 482\n",
            "epoch = 483\n",
            "epoch = 484\n",
            "epoch = 485\n",
            "epoch = 486\n",
            "epoch = 487\n",
            "epoch = 488\n",
            "epoch = 489\n",
            "epoch = 490\n",
            "epoch = 491\n",
            "epoch = 492\n",
            "epoch = 493\n",
            "epoch = 494\n",
            "epoch = 495\n",
            "epoch = 496\n",
            "epoch = 497\n",
            "epoch = 498\n",
            "epoch = 499\n",
            "epoch = 500\n",
            "epoch = 501\n",
            "epoch = 502\n",
            "epoch = 503\n",
            "epoch = 504\n",
            "epoch = 505\n",
            "epoch = 506\n",
            "epoch = 507\n",
            "epoch = 508\n",
            "epoch = 509\n",
            "epoch = 510\n",
            "epoch = 511\n",
            "epoch = 512\n",
            "epoch = 513\n",
            "epoch = 514\n",
            "epoch = 515\n",
            "epoch = 516\n",
            "epoch = 517\n",
            "epoch = 518\n",
            "epoch = 519\n",
            "epoch = 520\n",
            "epoch = 521\n",
            "epoch = 522\n",
            "epoch = 523\n",
            "epoch = 524\n",
            "epoch = 525\n",
            "epoch = 526\n",
            "epoch = 527\n",
            "epoch = 528\n",
            "epoch = 529\n",
            "epoch = 530\n",
            "epoch = 531\n",
            "epoch = 532\n",
            "epoch = 533\n",
            "epoch = 534\n",
            "epoch = 535\n",
            "epoch = 536\n",
            "epoch = 537\n",
            "epoch = 538\n",
            "epoch = 539\n",
            "epoch = 540\n",
            "epoch = 541\n",
            "epoch = 542\n",
            "epoch = 543\n",
            "epoch = 544\n",
            "epoch = 545\n",
            "epoch = 546\n",
            "epoch = 547\n",
            "epoch = 548\n",
            "epoch = 549\n",
            "epoch = 550\n",
            "epoch = 551\n",
            "epoch = 552\n",
            "epoch = 553\n",
            "epoch = 554\n",
            "epoch = 555\n",
            "epoch = 556\n",
            "epoch = 557\n",
            "epoch = 558\n",
            "epoch = 559\n",
            "epoch = 560\n",
            "epoch = 561\n",
            "epoch = 562\n",
            "epoch = 563\n",
            "epoch = 564\n",
            "epoch = 565\n",
            "epoch = 566\n",
            "epoch = 567\n",
            "epoch = 568\n",
            "epoch = 569\n",
            "epoch = 570\n",
            "epoch = 571\n",
            "epoch = 572\n",
            "epoch = 573\n",
            "epoch = 574\n",
            "epoch = 575\n",
            "epoch = 576\n",
            "epoch = 577\n",
            "epoch = 578\n",
            "epoch = 579\n",
            "epoch = 580\n",
            "epoch = 581\n",
            "epoch = 582\n",
            "epoch = 583\n",
            "epoch = 584\n",
            "epoch = 585\n",
            "epoch = 586\n",
            "epoch = 587\n",
            "epoch = 588\n",
            "epoch = 589\n",
            "epoch = 590\n",
            "epoch = 591\n",
            "epoch = 592\n",
            "epoch = 593\n",
            "epoch = 594\n",
            "epoch = 595\n",
            "epoch = 596\n",
            "epoch = 597\n",
            "epoch = 598\n",
            "epoch = 599\n",
            "epoch = 600\n",
            "epoch = 601\n",
            "epoch = 602\n",
            "epoch = 603\n",
            "epoch = 604\n",
            "epoch = 605\n",
            "epoch = 606\n",
            "epoch = 607\n",
            "epoch = 608\n",
            "epoch = 609\n",
            "epoch = 610\n",
            "epoch = 611\n",
            "epoch = 612\n",
            "epoch = 613\n",
            "epoch = 614\n",
            "epoch = 615\n",
            "epoch = 616\n",
            "epoch = 617\n",
            "epoch = 618\n",
            "epoch = 619\n",
            "epoch = 620\n",
            "epoch = 621\n",
            "epoch = 622\n",
            "epoch = 623\n",
            "epoch = 624\n",
            "epoch = 625\n",
            "epoch = 626\n",
            "epoch = 627\n",
            "epoch = 628\n",
            "epoch = 629\n",
            "epoch = 630\n",
            "epoch = 631\n",
            "epoch = 632\n",
            "epoch = 633\n",
            "epoch = 634\n",
            "epoch = 635\n",
            "epoch = 636\n",
            "epoch = 637\n",
            "epoch = 638\n",
            "epoch = 639\n",
            "epoch = 640\n",
            "epoch = 641\n",
            "epoch = 642\n",
            "epoch = 643\n",
            "epoch = 644\n",
            "epoch = 645\n",
            "epoch = 646\n",
            "epoch = 647\n",
            "epoch = 648\n",
            "epoch = 649\n",
            "epoch = 650\n",
            "epoch = 651\n",
            "epoch = 652\n",
            "epoch = 653\n",
            "epoch = 654\n",
            "epoch = 655\n",
            "epoch = 656\n",
            "epoch = 657\n",
            "epoch = 658\n",
            "epoch = 659\n",
            "epoch = 660\n",
            "epoch = 661\n",
            "epoch = 662\n",
            "epoch = 663\n",
            "epoch = 664\n",
            "epoch = 665\n",
            "epoch = 666\n",
            "epoch = 667\n",
            "epoch = 668\n",
            "epoch = 669\n",
            "epoch = 670\n",
            "epoch = 671\n",
            "epoch = 672\n",
            "epoch = 673\n",
            "epoch = 674\n",
            "epoch = 675\n",
            "epoch = 676\n",
            "epoch = 677\n",
            "epoch = 678\n",
            "epoch = 679\n",
            "epoch = 680\n",
            "epoch = 681\n",
            "epoch = 682\n",
            "epoch = 683\n",
            "epoch = 684\n",
            "epoch = 685\n",
            "epoch = 686\n",
            "epoch = 687\n",
            "epoch = 688\n",
            "epoch = 689\n",
            "epoch = 690\n",
            "epoch = 691\n",
            "epoch = 692\n",
            "epoch = 693\n",
            "epoch = 694\n",
            "epoch = 695\n",
            "epoch = 696\n",
            "epoch = 697\n",
            "epoch = 698\n",
            "epoch = 699\n",
            "epoch = 700\n",
            "epoch = 701\n",
            "epoch = 702\n",
            "epoch = 703\n",
            "epoch = 704\n",
            "epoch = 705\n",
            "epoch = 706\n",
            "epoch = 707\n",
            "epoch = 708\n",
            "epoch = 709\n",
            "epoch = 710\n",
            "epoch = 711\n",
            "epoch = 712\n",
            "epoch = 713\n",
            "epoch = 714\n",
            "epoch = 715\n",
            "epoch = 716\n",
            "epoch = 717\n",
            "epoch = 718\n",
            "epoch = 719\n",
            "epoch = 720\n",
            "epoch = 721\n",
            "epoch = 722\n",
            "epoch = 723\n",
            "epoch = 724\n",
            "epoch = 725\n",
            "epoch = 726\n",
            "epoch = 727\n",
            "epoch = 728\n",
            "epoch = 729\n",
            "epoch = 730\n",
            "epoch = 731\n",
            "epoch = 732\n",
            "epoch = 733\n",
            "epoch = 734\n",
            "epoch = 735\n",
            "epoch = 736\n",
            "epoch = 737\n",
            "epoch = 738\n",
            "epoch = 739\n",
            "epoch = 740\n",
            "epoch = 741\n",
            "epoch = 742\n",
            "epoch = 743\n",
            "epoch = 744\n",
            "epoch = 745\n",
            "epoch = 746\n",
            "epoch = 747\n",
            "epoch = 748\n",
            "epoch = 749\n",
            "epoch = 750\n",
            "epoch = 751\n",
            "epoch = 752\n",
            "epoch = 753\n",
            "epoch = 754\n",
            "epoch = 755\n",
            "epoch = 756\n",
            "epoch = 757\n",
            "epoch = 758\n",
            "epoch = 759\n",
            "epoch = 760\n",
            "epoch = 761\n",
            "epoch = 762\n",
            "epoch = 763\n",
            "epoch = 764\n",
            "epoch = 765\n",
            "epoch = 766\n",
            "epoch = 767\n",
            "epoch = 768\n",
            "epoch = 769\n",
            "epoch = 770\n",
            "epoch = 771\n",
            "epoch = 772\n",
            "epoch = 773\n",
            "epoch = 774\n",
            "epoch = 775\n",
            "epoch = 776\n",
            "epoch = 777\n",
            "epoch = 778\n",
            "epoch = 779\n",
            "epoch = 780\n",
            "epoch = 781\n",
            "epoch = 782\n",
            "epoch = 783\n",
            "epoch = 784\n",
            "epoch = 785\n",
            "epoch = 786\n",
            "epoch = 787\n",
            "epoch = 788\n",
            "epoch = 789\n",
            "epoch = 790\n",
            "epoch = 791\n",
            "epoch = 792\n",
            "epoch = 793\n",
            "epoch = 794\n",
            "epoch = 795\n",
            "epoch = 796\n",
            "epoch = 797\n",
            "epoch = 798\n",
            "epoch = 799\n",
            "epoch = 800\n",
            "epoch = 801\n",
            "epoch = 802\n",
            "epoch = 803\n",
            "epoch = 804\n",
            "epoch = 805\n",
            "epoch = 806\n",
            "epoch = 807\n",
            "epoch = 808\n",
            "epoch = 809\n",
            "epoch = 810\n",
            "epoch = 811\n",
            "epoch = 812\n",
            "epoch = 813\n",
            "epoch = 814\n",
            "epoch = 815\n",
            "epoch = 816\n",
            "epoch = 817\n",
            "epoch = 818\n",
            "epoch = 819\n",
            "epoch = 820\n",
            "epoch = 821\n",
            "epoch = 822\n",
            "epoch = 823\n",
            "epoch = 824\n",
            "epoch = 825\n",
            "epoch = 826\n",
            "epoch = 827\n",
            "epoch = 828\n",
            "epoch = 829\n",
            "epoch = 830\n",
            "epoch = 831\n",
            "epoch = 832\n",
            "epoch = 833\n",
            "epoch = 834\n",
            "epoch = 835\n",
            "epoch = 836\n",
            "epoch = 837\n",
            "epoch = 838\n",
            "epoch = 839\n",
            "epoch = 840\n",
            "epoch = 841\n",
            "epoch = 842\n",
            "epoch = 843\n",
            "epoch = 844\n",
            "epoch = 845\n",
            "epoch = 846\n",
            "epoch = 847\n",
            "epoch = 848\n",
            "epoch = 849\n",
            "epoch = 850\n",
            "epoch = 851\n",
            "epoch = 852\n",
            "epoch = 853\n",
            "epoch = 854\n",
            "epoch = 855\n",
            "epoch = 856\n",
            "epoch = 857\n",
            "epoch = 858\n",
            "epoch = 859\n",
            "epoch = 860\n",
            "epoch = 861\n",
            "epoch = 862\n",
            "epoch = 863\n",
            "epoch = 864\n",
            "epoch = 865\n",
            "epoch = 866\n",
            "epoch = 867\n",
            "epoch = 868\n",
            "epoch = 869\n",
            "epoch = 870\n",
            "epoch = 871\n",
            "epoch = 872\n",
            "epoch = 873\n",
            "epoch = 874\n",
            "epoch = 875\n",
            "epoch = 876\n",
            "epoch = 877\n",
            "epoch = 878\n",
            "epoch = 879\n",
            "epoch = 880\n",
            "epoch = 881\n",
            "epoch = 882\n",
            "epoch = 883\n",
            "epoch = 884\n",
            "epoch = 885\n",
            "epoch = 886\n",
            "epoch = 887\n",
            "epoch = 888\n",
            "epoch = 889\n",
            "epoch = 890\n",
            "epoch = 891\n",
            "epoch = 892\n",
            "epoch = 893\n",
            "epoch = 894\n",
            "epoch = 895\n",
            "epoch = 896\n",
            "epoch = 897\n",
            "epoch = 898\n",
            "epoch = 899\n",
            "epoch = 900\n",
            "epoch = 901\n",
            "epoch = 902\n",
            "epoch = 903\n",
            "epoch = 904\n",
            "epoch = 905\n",
            "epoch = 906\n",
            "epoch = 907\n",
            "epoch = 908\n",
            "epoch = 909\n",
            "epoch = 910\n",
            "epoch = 911\n",
            "epoch = 912\n",
            "epoch = 913\n",
            "epoch = 914\n",
            "epoch = 915\n",
            "epoch = 916\n",
            "epoch = 917\n",
            "epoch = 918\n",
            "epoch = 919\n",
            "epoch = 920\n",
            "epoch = 921\n",
            "epoch = 922\n",
            "epoch = 923\n",
            "epoch = 924\n",
            "epoch = 925\n",
            "epoch = 926\n",
            "epoch = 927\n",
            "epoch = 928\n",
            "epoch = 929\n",
            "epoch = 930\n",
            "epoch = 931\n",
            "epoch = 932\n",
            "epoch = 933\n",
            "epoch = 934\n",
            "epoch = 935\n",
            "epoch = 936\n",
            "epoch = 937\n",
            "epoch = 938\n",
            "epoch = 939\n",
            "epoch = 940\n",
            "epoch = 941\n",
            "epoch = 942\n",
            "epoch = 943\n",
            "epoch = 944\n",
            "epoch = 945\n",
            "epoch = 946\n",
            "epoch = 947\n",
            "epoch = 948\n",
            "epoch = 949\n",
            "epoch = 950\n",
            "epoch = 951\n",
            "epoch = 952\n",
            "epoch = 953\n",
            "epoch = 954\n",
            "epoch = 955\n",
            "epoch = 956\n",
            "epoch = 957\n",
            "epoch = 958\n",
            "epoch = 959\n",
            "epoch = 960\n",
            "epoch = 961\n",
            "epoch = 962\n",
            "epoch = 963\n",
            "epoch = 964\n",
            "epoch = 965\n",
            "epoch = 966\n",
            "epoch = 967\n",
            "epoch = 968\n",
            "epoch = 969\n",
            "epoch = 970\n",
            "epoch = 971\n",
            "epoch = 972\n",
            "epoch = 973\n",
            "epoch = 974\n",
            "epoch = 975\n",
            "epoch = 976\n",
            "epoch = 977\n",
            "epoch = 978\n",
            "epoch = 979\n",
            "epoch = 980\n",
            "epoch = 981\n",
            "epoch = 982\n",
            "epoch = 983\n",
            "epoch = 984\n",
            "epoch = 985\n",
            "epoch = 986\n",
            "epoch = 987\n",
            "epoch = 988\n",
            "epoch = 989\n",
            "epoch = 990\n",
            "epoch = 991\n",
            "epoch = 992\n",
            "epoch = 993\n",
            "epoch = 994\n",
            "epoch = 995\n",
            "epoch = 996\n",
            "epoch = 997\n",
            "epoch = 998\n",
            "epoch = 999\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Now let's look at the average MSE across each training epoch:"
      ],
      "metadata": {
        "id": "NRMV7URirUu1"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "plt.plot(avg_cost_list)\n",
        "plt.ylabel(\"Avg. cost\")\n",
        "plt.xlabel(\"epoch\")\n",
        "plt.title(\"Avg. MSE over training epochs - SGD\")\n",
        "plt.show()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 295
        },
        "id": "JADmTz0BoGVs",
        "outputId": "3612382c-e226-4971-ce60-48d2f00b942a"
      },
      "execution_count": 236,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ],
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZUAAAEWCAYAAACufwpNAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3df5wdVX3/8debkAQB+b1iSGKCkhaBQioxQhWbgmjgiwkVCkEqhJKCSh6o/VZNio2V1larFYtQhIbIb4hGwQUCASNg2gJmoTESIGb5ZX4BS4CEIAYCn/4xZ5O5l9nk7r13suzu+/l43MfOnHPmzJk7u/ez55y5M4oIzMzMmmG7nm6AmZn1HQ4qZmbWNA4qZmbWNA4qZmbWNA4qZmbWNA4qZmbWNA4qZm8hktZLenezy77VSbpb0pSeboc1zkGlH0l/uC9IGryN9heSnpW0fS5tYEqLXNqBku6Q9LykFyU9IOnYlDdO0hvpAzT/OnxbHEOtUjtXNFpPROwcEY83u6xlJA2T9GNJz0laK+khSZNz+YMkzZC0VNLLklZKuk3SR3NlnpT0iqSX0u/r/0j6tCR/nuKg0m9IGgkcAQQwYRvu+gXgmNz6MSkt72bgTuCdwDuAc4F1ufxV6QM0/7q3zEZvST5IbovtrKmuBpYDI4A9gU8Bz+Ty5wATgdOA3YF9gX8H/l9VPR+PiLener4BfBm4vNSW9xYR4Vc/eAEzgP8GvgPcktIGAy8CB+XKtQCvAO9I618CVgOrgClkQWm/GvcZwFeAH+XS5gDnZb96AbBXKrdbF3WMA1Z04zjfC9ydjmsJMCGlfwB4GhiQK/vnwOK0vB0wDXgMWAP8ENgj5Y1MbTwT+C3wi6p97pTeszeA9em1D/AP6XivIQuSU4CxwL2pfauBi4BBVe/Zfmn5CuBi4FbgJeB+4D11lv0osBRYC/wHcA8wpYv3sJb34qz0O7Ea+NvctoOB76a8VWl5cC5/IrAovR+PAeNT+t3AP5L9jr4E3AHslfJ2SO/hmvS+LQT2rvPvYD0wuou8j6TzOGwrdTwJfKQqbWw6/wfV066+9OrxBvi1jU40tAOfBQ4FXuv8owRmAV/PlTsHuD0tjyf7ID4Q2DH9YXc3qBxE9p/gbmT/+T2T0iKVEbAMuAU4vvrDgm4EFWBgOs6/AwYBR6YPqD9M+Y8BR+fK/wiYlpY/B9wHDEsfjJcC16e8zg/Sq8gCyNsK9v2mdpIFldfScW0HvC29/4cB26d6HwE+X/We5QPFmvSBtT1wLXBDd8uSBe51wCdS3udSu7oKKrW8F9en9+KPgA7Shyxwftr2HWT/oPwP8I8pbyxZUDs6vR9Dgf1T3t3p/PxBep/uBr6R8s4m683uCAxI7+Eudf4d/IwscE0C3lWV9w3g7hrqeJKqoJLSfwt8pqf/1nv61eMN8GsbnGT4UPoQ6fzP71HgC2n5I8BjubL/DZyWlmcB/5LL24/uB5X9gJnpg+HTwH921pMrN4zsP/bHyP7b+wUwKuWNS2kvVr12KtjfEWRBcLtc2vXAP6TlfwJmpeW3Ay8DI9L6I8BRue2GpPes88M/gHdv4VjHURxUfrGV9+jzwI3V71lavgKYmcs7Fni0u2XJhnLuzeWJbAioq6BSy3uxfy7/X4HL0/JjwLG5vI8BT6blS4ELutjn3cBXcuufZfM/N39FFpwObsLfwu5kwWMJ8DpZr+n9KW8mlUF7j/S7thb4fS79SYqDyn3AeWX8Dfeml+dU+ofTgTsi4rm0fl1KA7gL2FHSB9K8y2jgxpS3D9mHT6f8cndcRfbBdlparhARKyJiakS8h2yM+uWqcqsiYreq18sF+9kHWB4Rb+TSniL7jxiy4/5EulDhE8CDEfFUyhsB3JgmXl8k+2B9Hdg7V1c9x1+xjaQ/kHSLpKclrQP+mawn0ZWnc8u/A3auo2zFeYzsE3BLFxV09714Ku2jc19PdZE3nCzodLf9VwPzgBskrZL0r5IGVm8s6dTchRy3Fe0gIl6IiGkRcWA6nkXATZJE1tMbkiv7fETsRtYzquXilqHA8zWU69McVPo4SW8DTgL+NH2QPQ18AThE0iER8TrZmPkp6XVLRLyUNl9N1ovoNLzOZiwg+2PdG/ivLRWMiOVkcwMH1bGfVcDwqqtw3gWsTHU/TPYhdwzwSbIg02k5cExV4NohIlbmm7elpteYfglZT3FUROxCNlSnrRxXoyrOY/oAHdZ18Zrei/zvwrvI3nvSzxFd5C0H3tPdxkfEaxHxtYg4APgT4Diyf1Cqy10bmy/kOOZNFb25/HPAt8mC3h7AfOD9krb03hSS9H6yoLLF3+/+wEGl7zue7L/MA8h6IaPJJrMXsPkP8zrgZOBUKj9ofwicIem9knYE/r6eBqT/jD9ONmle8SEraXdJX5O0n6TtJO1FNtxxXx27up/sP9wvpUuXx6X93pArcx3ZnMGHyeZUOn0f+LqkEaldLZImdmPfzwB7Stp1K+XeTja/sV7S/sBnurGPet0K/JGk49MVaOeQXWnXlVrei7+XtKOkA4EzgNkp/XrgK2mbvcguELkm5V1O9vt0VDrXQ9N7sEWS/kzSH0kaQPbevUY2JNptkr4p6SBJ20t6O9n73x4RayLiDrKe+02p5z4o9YgO20J9u0g6jux37JqI+HU97epLHFT6vtOBH0TEbyPi6c4X2RzGqZK2j4j7yYac9gE2DRtExG3AhWR/aO1s/qDfACDp77oaZqgWEUsiYklB1qtk4/Q/I/vAeCjVPzlXZp+C76mcULCPV8mCyDHAc2RXOZ0WEY/mil0P/Cnw89xwIGSXjbYCd0h6KR3rB2o5trTvR1Pdj6dho326KPq3ZL2kl8jml2Z3Ua5p0nH+BdncxxqyfzDaSOexQC3vxT1kvxPzgW+nD2TI5q3agMXAr4EHUxoR8UuyAHQB2TzFPVT2arryTrKr6NaRDcXdQzYkVo8dyYZ3XwQeT/vPX2L/52QXjVyTyjxB9s/Wx6rquTm9N8vJrmb8Dtmx9Xuq+sfRrEuS3kv2oT84Ijb2dHusPml4cAVwakTc1c1tR5J90A7074AVcU/FtkjSn0saLGl34JvAzf4w6X0kfUzSbukihc55nHqGGM22yEHFtuZs4Fmyq3ZeZ9vMAVjzHU52Dp8jGyI8PiJe6dkmWV/k4S8zM2sa91TMzKxp+vUN7vbaa68YOXJkTzfDzKxXeeCBB56LiJaivH4dVEaOHElbW1tPN8PMrFeR9FRXeR7+MjOzpnFQMTOzpnFQMTOzpik1qEganx7L2S5pWkH+ZEkdkhal15SUPkLSgyltiaRP57Y5VNKvU50XppvjIWkPSXdKWpZ+7l7msZmZ2ZuVFlTSzd8uJrsP0wHAKZIOKCg6OyJGp9fMlLYaODwiRpPdc2ha7l5KlwB/DYxKr/EpfRowPyJGkd2P6E1BzMzMylVmT2Us2d0/H083+ruB7FGiWxURr0ZE583uBpPaKWkI2RPf7kt3u72K7C68pLqvTMtX5tLNzGwbKTOoDKXyQT4r2PywpLwTJC2WNEfSpmc0SBouaXGq45sRsSptn3+4UL7OvSNidVp+msoHCm0i6SxJbZLaOjo66jowMzMr1tMT9TcDIyPiYOBONvc0iIjlKX0/4HRJhUGiSOrFFN5/JiIui4gxETGmpaXwuztb9ZtnXuI7dyzlufVd3TnczKx/KjOorKTy6XDDUtom6cE4nZ/MM8ke20lVmVVkt1s/Im2ffypbvs5n0vBY5zDZs004hkLLnlnPhT9v5/mXXy1rF2ZmvVKZQWUhMErSvpIGAZPIHvyzSWcQSCaQPYAHScPSY3BJV3F9CFiahrfWSTosXfV1GvDTtH0rm5+7fnou3czMtpHSbtMSERslTQXmAQOAWRGxRNL5QFtEtALnSpoAbASeZ/PT/t4L/JukIHvuw7dzj+n8LHAF8DaypxR2PnnwG8APJZ1J9hzyk8o6tk6+wbOZWaVS7/0VEXOBuVVpM3LL04HpBdvdCRzcRZ1twEEF6WuAoxpsck2yb8aYmVm1np6o79Wi+FoAM7N+y0GlDu6omJkVc1AxM7OmcVBpgCfqzcwqOajUwRP1ZmbFHFQa4J6KmVklB5W6uKtiZlbEQcXMzJrGQaUB/p6KmVklB5U6eKLezKyYg0oDPFFvZlbJQaUO7qiYmRVzUDEzs6ZxUDEzs6ZxUKmDPFNvZlbIQaUBnqg3M6vkoFIH91PMzIo5qJiZWdOUGlQkjZe0VFK7pGkF+ZMldUhalF5TUvpoSfdKWiJpsaSTc9ssyJVfJemmlD5O0tpc3ozq/TWbv1FvZlaptGfUSxoAXAwcDawAFkpqjYiHq4rOjoipVWm/A06LiGWS9gEekDQvIl6MiCNy+/gx8NPcdgsi4rjmH00lz9ObmRUrs6cyFmiPiMcj4lXgBmBiLRtGxG8iYllaXgU8C7Tky0jaBTgSuKmpre4GT9SbmVUqM6gMBZbn1lektGonpCGuOZKGV2dKGgsMAh6ryjoemB8R63Jph0v6laTbJB1Y1ChJZ0lqk9TW0dHRrQPaXEddm5mZ9Xk9PVF/MzAyIg4G7gSuzGdKGgJcDZwREW9UbXsKcH1u/UFgREQcAnyPLnowEXFZRIyJiDEtLS1FRczMrE5lBpWVQL7nMSylbRIRayJiQ1qdCRzamZeGt24FzouI+/LbSdqLbHjt1lxd6yJifVqeCwxM5Urj0S8zs0plBpWFwChJ+0oaBEwCWvMFUk+k0wTgkZQ+CLgRuCoi5hTUfSJwS0T8PlfXO5W+6p6GzLYD1jTxeDa3299UMTMrVNrVXxGxUdJUYB4wAJgVEUsknQ+0RUQrcK6kCcBG4Hlgctr8JODDwJ6SOtMmR8SitDwJ+EbVLk8EPiNpI/AKMCmi3Kn0kqs3M+t1SgsqsGkYam5V2ozc8nRgesF21wDXbKHecQVpFwEXNdBcMzNrUE9P1PdOHv0yMyvkoNIAD36ZmVVyUKmDOypmZsUcVBrgeXozs0oOKmZm1jQOKnXwkx/NzIo5qDTE419mZnkOKnVwP8XMrJiDipmZNY2DSgN89ZeZWSUHlTp4nt7MrJiDSgPcUTEzq+SgUgff+t7MrJiDipmZNY2DSgM8UW9mVslBpQ6eqDczK+ag0gA/+dHMrJKDSh3cUTEzK1ZqUJE0XtJSSe2SphXkT5bUIWlRek1J6aMl3StpiaTFkk7ObXOFpCdy24xO6ZJ0YdrXYknvK/PYzMzszUp7Rr2kAcDFwNHACmChpNaIeLiq6OyImFqV9jvgtIhYJmkf4AFJ8yLixZT/xYiYU7XNMcCo9PoAcEn6WRoPfpmZVSqzpzIWaI+IxyPiVeAGYGItG0bEbyJiWVpeBTwLtGxls4nAVZG5D9hN0pD6m78FHv8yMytUZlAZCizPra9IadVOSMNVcyQNr86UNBYYBDyWS/562uYCSYO7sz9JZ0lqk9TW0dHRzUOq5Hl6M7NKPT1RfzMwMiIOBu4Ersxnpp7G1cAZEfFGSp4O7A+8H9gD+HJ3dhgRl0XEmIgY09Kytc5PMX+j3sysWJlBZSWQ73kMS2mbRMSaiNiQVmcCh3bmSdoFuBU4Lw1ndW6zOg1xbQB+QDbMVtP+zMysXGUGlYXAKEn7ShoETAJa8wWq5jwmAI+k9EHAjWRzJHOKtlH2TN/jgYdSVitwWroK7DBgbUSsbv5hbRaeqjczq1Da1V8RsVHSVGAeMACYFRFLJJ0PtEVEK3CupAnARuB5YHLa/CTgw8CekjrTJkfEIuBaSS1k0+WLgE+n/LnAsUA72dVjZ5R1bP5GvZlZsdKCCkBEzCX7sM+nzcgtTyebI6ne7hrgmi7qPLKL9ADOaaS93eaOiplZhZ6eqDczsz7EQaUOHv0yMyvmoNIAj36ZmVVyUKmDPFNvZlbIQaUB/ka9mVklBxUzM2saB5U6ePTLzKyYg0oD/I16M7NKDip1cEfFzKyYg0oDPFFvZlbJQcXMzJrGQaUOnqg3MyvmoNIAj36ZmVVyUKmLuypmZkUcVBoQnqk3M6vgoGJmZk3joFIHT9SbmRVzUGmAB7/MzCqVGlQkjZe0VFK7pGkF+ZMldUhalF5TUvpoSfdKWiJpsaSTc9tcm+p8SNIsSQNT+jhJa3N1zajeX9OOq6yKzcx6udKeUS9pAHAxcDSwAlgoqTUiHq4qOjsiplal/Q44LSKWSdoHeEDSvIh4EbgW+MtU7jpgCnBJWl8QEceVcTxmZrZ1W+2pSPpgLWkFxgLtEfF4RLwK3ABMrKVREfGbiFiWllcBzwItaX1uJMAvgWG11FkKj3+ZmVWoZfjrezWmVRsKLM+tr0hp1U5IQ1xzJA2vzpQ0FhgEPFaVPhD4FHB7LvlwSb+SdJukA4saJeksSW2S2jo6Omo4jMI66trOzKyv63L4S9LhwJ8ALZL+Jpe1CzCgSfu/Gbg+IjZIOhu4Ejgy14YhwNXA6RHxRtW2/wH8IiIWpPUHgRERsV7SscBNwKjqHUbEZcBlAGPGjGmor+Fb35uZVdpST2UQsDNZ4Hl77rUOOLGGulcC+Z7HsJS2SUSsiYgNaXUmcGhnnqRdgFuB8yLivvx2kr5KNhz2N7m61kXE+rQ8Fxgoaa8a2tlt7qeYmRXrsqcSEfcA90i6IiKeApC0HbBzRKyroe6FwChJ+5IFk0nAJ/MFJA2JiNVpdQLwSEofBNwIXBURc6q2mQJ8DDgq33uR9E7gmYiINGS2HbCmhnaamVmT1DKn8i+SdpG0E/AQ8LCkL25to4jYCEwF5pEFix9GxBJJ50uakIqdmy4b/hVwLjA5pZ8EfBiYnLtEeHTK+z6wN3Bv1aXDJwIPpbouBCZFyfdR8V1azMwq1XJJ8QERsU7SqcBtwDTgAeBbW9swDUPNrUqbkVueDkwv2O4a4Jou6ixsc0RcBFy0tTY1g+fpzcyK1dJTGZiutDoeaI2I1/DFtIB7KmZm1WoJKpcCTwI7Ab+QNIJssr7fkqfqzcwKbXX4KyIuJJuj6PSUpD8rr0lmZtZb1fKN+l0lfafzC4OS/o2s19LvefTLzKxSLcNfs4CXyK7IOols6OsHZTbqrc4T9WZmxWq5+us9EXFCbv1rkhaV1aDexE9+NDOrVEtP5RVJH+pcSTeTfKW8JpmZWW9VS0/lM8CVknZN6y+w+UuKZmZmm9Ry9dci4JB0Ly5qvEVLv+DBLzOzSrVc/fXPknZLN2xcJ2l3Sf+0LRr3VuWJejOzYrXMqRyTnrgIQES8ABxbXpN6D8/Tm5lVqiWoDJA0uHNF0tuAwVsob2Zm/VQtE/XXAvMldX435Qyyh2n1W75Ni5lZsVom6r+Zbif/kZT0jxExr9xm9RYe/zIzy6ulp0JE3E7ls+D7NU/Um5kVq2VOxbrgiXozs0oOKmZm1jSlBhVJ4yUtldQuaVpB/mRJHblHBk9J6aMl3ZseNbxY0sm5bfaVdH+qc3Z6nj2SBqf19pQ/srzjKqtmM7Pera6gIukfaigzALgYOAY4ADhF0gEFRWdHxOj0mpnSfgecFhEHAuOB70raLeV9E7ggIvYju2XMmSn9TOCFlH5BKlcqj36ZmVWqt6fyQA1lxgLtEfF4RLwK3ABMrKXyiPhNRCxLy6uAZ4EWSQKOBOakoleSPeaYVHfnpc5zgKNS+abzJcVmZsXqCioRcXMNxYYCy3PrK1JatRPSENccScOrMyWNBQYBjwF7Ai9GxMaCOjftL+WvTeVL44l6M7NKW72kWNKFBclrgbaI+GmD+78ZuD4iNkg6m6yncWRu30OAq4HTI+KNZnQ8JJ0FnAXwrne9q+H6zMxss1p6KjsAo4Fl6XUwMAw4U9J3t7DdSiDf8xiW0jaJiDURsSGtzgQO7cxLd0W+FTgvIu5LyWuA3SR1BsN8nZv2l/J3TeUrRMRlETEmIsa0tLRs6bi75Il6M7NitXz58WDggxHxOoCkS4AFwIeAX29hu4XAKEn7kn3gTwI+mS8gaUhErE6rE4BHUvog4EbgqojonD8hIkLSXcCJZHM0pwOdvaXWtH5vyv95lPxoxvBUvZlZhVqCyu7AzmRDXgA7AXtExOuSNnS1UURslDQVmAcMAGZFxBJJ55MNnbUC50qaAGwEnmfzw79OAj4M7CmpM21yerbLl4Eb0u33/xe4POVfDlwtqT3VNamGY6uLOypmZsVqCSr/CiySdDfZ5+mHgX+WtBPwsy1tGBFzgblVaTNyy9OB6QXbXQNc00Wdj5NdWVad/nvgL7ZyLGZmVqJabih5uaS5bP4g/7t0mS/AF0trWS/gq7/MzCrVcvXXzcB1QGtEvFx+k976PFFvZlaslqu/vg0cATycvktyoqQdSm5Xr+COiplZpVqGv+4B7km3XTkS+GtgFrBLyW17C3NXxcysSE3PU0mPEP44cDLwPvr5kx/NzKxYLXMqPySbpL8duAi4JyLeKLthvUHJX4MxM+t1aumpXA6ckvvy44cknRIR55TbtLcuT9SbmRWrZU5lnqQ/lnQK2ZcSnwB+UnrLzMys1+kyqEj6A+CU9HoOmA0oIv5sG7XNzMx6mS31VB4lu8fXcRHRDiDpC9ukVW9xHv0yMyu2pe+pfAJYDdwl6T8lHYU/Tyt4nt7MrFKXQSUiboqIScD+wF3A54F3SLpE0ke3VQPfikp6oKSZWa+31W/UR8TLEXFdRHyc7Pkl/0t2p+B+z7e+NzOr1K3HCUfEC+khV0eV1SAzM+u96npGfX/nwS8zs2IOKg3wRL2ZWSUHlTp4nt7MrJiDSgPcUzEzq1RqUJE0XtJSSe2SphXkT5bUIWlRek3J5d0u6UVJt1RtsyBXfpWkm1L6OElrc3kzqvdnZmblqunW9/VIz1+5GDgaWAEslNQaEQ9XFZ0dEVMLqvgWsCNwdj4xIo7I7ePHwE9z2Qsi4rhmtH9L5Kl6M7NCZfZUxgLtEfF4RLwK3ABMrHXjiJgPvNRVvqRdyB4adlOjDa2XR7/MzCqVGVSGAstz6ytSWrUTJC1Ojyoe3o36jwfmR8S6XNrhkn4l6TZJB9bR5pp4ot7MrFhPT9TfDIyMiIOBO+neEyVPAa7PrT8IjIiIQ4Dv0UUPRtJZktoktXV0dNTZ7Iwf0mVmVqnMoLISyPc8hqW0TSJiTURsSKszgUNrqVjSXmTDa7fm6loXEevT8lxgYCpXId0RYExEjGlpaenO8ZiZ2VaUGVQWAqMk7StpEDAJaM0XkDQktzoBeKTGuk8EbomI3+fqeqfSnR4ljSU7tjUNtN/MzLqptKu/ImKjpKnAPGAAMCsilkg6H2iLiFbgXEkTgI3A88Dkzu0lLSC7Q/LOklYAZ0bEvJQ9CfhG1S5PBD4jaSPwCjApSh6f8uCXmVml0oIKbBqGmluVNiO3PB2Y3sW2RxSlp7xxBWkXARfV29bu8ES9mVmxnp6o793cVTEzq+CgYmZmTeOgUgc/+dHMrJiDSgP85Eczs0oOKnVwP8XMrJiDSgP8hXozs0oOKmZm1jQOKnXwPL2ZWTEHlQZ49MvMrJKDipmZNY2DSh385Eczs2IOKg3w1V9mZpUcVOrgiXozs2IOKg3wN+rNzCo5qJiZWdM4qNTBo19mZsUcVBrgiXozs0oOKvVwV8XMrFCpQUXSeElLJbVLmlaQP1lSh6RF6TUll3e7pBcl3VK1zRWSnshtMzqlS9KFaV+LJb2vzGMDf6PezKxaac+olzQAuBg4GlgBLJTUGhEPVxWdHRFTC6r4FrAjcHZB3hcjYk5V2jHAqPT6AHBJ+mlmZttImT2VsUB7RDweEa8CNwATa904IuYDL3VjfxOBqyJzH7CbpCHdanGN/I16M7NiZQaVocDy3PqKlFbthDRcNUfS8Brr/nra5gJJg7uzP0lnSWqT1NbR0VHj7rrgmXozswo9PVF/MzAyIg4G7gSurGGb6cD+wPuBPYAvd2eHEXFZRIyJiDEtLS3dbS/gb9SbmXWlzKCyEsj3PIaltE0iYk1EbEirM4FDt1ZpRKxOQ1wbgB+QDbPVtL9mcz/FzKxSmUFlITBK0r6SBgGTgNZ8gao5jwnAI1urtHMbSQKOBx5KWa3AaekqsMOAtRGxuvHDMDOzWpV29VdEbJQ0FZgHDABmRcQSSecDbRHRCpwraQKwEXgemNy5vaQFZMNcO0taAZwZEfOAayW1kH1bZBHw6bTJXOBYoB34HXBGWcfm0S8zs2KlBRWAiJhL9mGfT5uRW55ONkdStO0RXaQf2UV6AOfU3dg6eJ7ezKxST0/U90ryTL2ZWSEHlQaEuypmZhUcVMzMrGkcVOrgwS8zs2IOKg3w4JeZWSUHlTp4nt7MrJiDSgM8T29mVslBxczMmsZBpQ6+9b2ZWTEHlQZ49MvMrJKDipmZNY2DSj3S6Je/UW9mVslBxczMmsZBpQ7+noqZWTEHFTMzaxoHFTMzaxoHlTp0jn55nt7MrJKDipmZNU2pQUXSeElLJbVLmlaQP1lSh6RF6TUll3e7pBcl3VK1zbWpzockzZI0MKWPk7Q2V9eM6v018bjKqtrMrFcr7Rn1kgYAFwNHAyuAhZJaI+LhqqKzI2JqQRXfAnYEzq5Kvxb4y7R8HTAFuCStL4iI45rR/lqEv1NvZlahzJ7KWKA9Ih6PiFeBG4CJtW4cEfOBlwrS50YC/BIY1qwGm5lZY8oMKkOB5bn1FSmt2gmSFkuaI2l4rZWnYa9PAbfnkg+X9CtJt0k6sIvtzpLUJqmto6Oj1t1V1lHXVmZmfV9PT9TfDIyMiIOBO4Eru7HtfwC/iIgFaf1BYEREHAJ8D7ipaKOIuCwixkTEmJaWlgaa7qu/zMyqlRlUVgL5nsewlLZJRKyJiA1pdSZwaC0VS/oq0AL8Ta6udRGxPi3PBQZK2qv+5m9p/2XUambW+5UZVBYCoyTtK2kQMAlozReQNCS3OgF4ZGuVpivEPgacEhFv5NLfqXRZlqSxZMe2puGj2AJ3VMzMKpV29VdEbJQ0FZgHDABmRcQSSecDbRHRCpwraQKwEXgemNy5vaQFwP7AzpJWAGdGxDzg+8BTwL0phvwkIs4HTgQ+I2kj8AowKXwbYTOzbaq0oAKbhqHmVqXNyC1PB6Z3se0RXaQXtjkiLgIuqrux3bcgnjwAAAaASURBVOAnP5qZFevpifpezf0gM7NKDip18ES9mVkxB5UG+Bv1ZmaVHFTMzKxpHFTMzKxpSr36q6+7fMET3Pjgyq0XNDN7izn5/cOZcsS7m16vg0oddhg4gM+Oew9Prnm5p5tiZlaXvXYeXEq9Dip1+tL4/Xu6CWZmbzmeUzEzs6ZxUDEzs6ZxUDEzs6ZxUDEzs6ZxUDEzs6ZxUDEzs6ZxUDEzs6ZxUDEzs6ZRf344oqQOsqdI1mMv4LkmNqc38DH3Dz7m/qGRYx4RES1FGf06qDRCUltEjOnpdmxLPub+wcfcP5R1zB7+MjOzpnFQMTOzpnFQqd9lPd2AHuBj7h98zP1DKcfsORUzM2sa91TMzKxpHFTMzKxpHFTqIGm8pKWS2iVN6+n2NIuk4ZLukvSwpCWSPpfS95B0p6Rl6efuKV2SLkzvw2JJ7+vZI6iPpAGS/lfSLWl9X0n3p+OaLWlQSh+c1ttT/siebHcjJO0maY6kRyU9IunwvnyeJX0h/U4/JOl6STv0xfMsaZakZyU9lEvr9nmVdHoqv0zS6d1pg4NKN0kaAFwMHAMcAJwi6YCebVXTbAT+f0QcABwGnJOObRowPyJGAfPTOmTvwaj0Ogu4ZNs3uSk+BzySW/8mcEFE7Ae8AJyZ0s8EXkjpF6RyvdW/A7dHxP7AIWTH3yfPs6ShwLnAmIg4CBgATKJvnucrgPFVad06r5L2AL4KfAAYC3y1MxDVJCL86sYLOByYl1ufDkzv6XaVdKw/BY4GlgJDUtoQYGlavhQ4JVd+U7ne8gKGpT+0I4FbAJF9y3j76vMNzAMOT8vbp3Lq6WOo45h3BZ6obntfPc/AUGA5sEc6b7cAH+ur5xkYCTxU73kFTgEuzaVXlNvayz2V7uv8Be20IqX1KanL/8fA/cDeEbE6ZT0N7J2W+8J78V3gS8AbaX1P4MWI2JjW88e06XhT/tpUvrfZF+gAfpCG/WZK2ok+ep4jYiXwbeC3wGqy8/YAff88d+rueW3ofDuo2JtI2hn4MfD5iFiXz4vsX5c+cR26pOOAZyPigZ5uyza2PfA+4JKI+GPgZTYPiQB97jzvDkwkC6b7ADvx5iGifmFbnFcHle5bCQzPrQ9LaX2CpIFkAeXaiPhJSn5G0pCUPwR4NqX39vfig8AESU8CN5ANgf07sJuk7VOZ/DFtOt6UvyuwZls2uElWACsi4v60PocsyPTV8/wR4ImI6IiI14CfkJ37vn6eO3X3vDZ0vh1Uum8hMCpdOTKIbMKvtYfb1BSSBFwOPBIR38lltQKdV4CcTjbX0pl+WrqK5DBgba6b/ZYXEdMjYlhEjCQ7jz+PiFOBu4ATU7Hq4+18H05M5Xvdf/MR8TSwXNIfpqSjgIfpo+eZbNjrMEk7pt/xzuPt0+c5p7vndR7wUUm7p17eR1NabXp6Uqk3voBjgd8AjwHn9XR7mnhcHyLrGi8GFqXXsWTjyfOBZcDPgD1SeZFdCfcY8Guyq2t6/DjqPPZxwC1p+d3AL4F24EfA4JS+Q1pvT/nv7ul2N3C8o4G2dK5vAnbvy+cZ+BrwKPAQcDUwuC+eZ+B6snmj18h6pGfWc16Bv0rH3w6c0Z02+DYtZmbWNB7+MjOzpnFQMTOzpnFQMTOzpnFQMTOzpnFQMTOzpnFQMeulJI3rvLOy2VuFg4qZmTWNg4pZyST9paRfSlok6dL0/Jb1ki5Iz/iYL6kllR0t6b70fIsbc8++2E/SzyT9StKDkt6Tqt8591yUa9M3xs16jIOKWYkkvRc4GfhgRIwGXgdOJbupYVtEHAjcQ/b8CoCrgC9HxMFk33LuTL8WuDgiDgH+hOxb05DdSfrzZM/2eTfZPa3Mesz2Wy9iZg04CjgUWJg6EW8ju6HfG8DsVOYa4CeSdgV2i4h7UvqVwI8kvR0YGhE3AkTE7wFSfb+MiBVpfRHZszT+q/zDMivmoGJWLgFXRsT0ikTp76vK1Xu/pA255dfx37T1MA9/mZVrPnCipHfApueFjyD72+u8Q+4ngf+KiLXAC5KOSOmfAu6JiJeAFZKOT3UMlrTjNj0Ksxr5vxqzEkXEw5K+AtwhaTuyu8eeQ/ZgrLEp71myeRfIbk3+/RQ0HgfOSOmfAi6VdH6q4y+24WGY1cx3KTbrAZLWR8TOPd0Os2bz8JeZmTWNeypmZtY07qmYmVnTOKiYmVnTOKiYmVnTOKiYmVnTOKiYmVnT/B8IczBbJyTB0AAAAABJRU5ErkJggg==\n"
          },
          "metadata": {
            "needs_background": "light"
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Let's also look at how the weights changed after each training epoch."
      ],
      "metadata": {
        "id": "1VRqKmPl3e6V"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "plt.plot(weights_updates[:,0], label = \"w1\")\n",
        "plt.plot(weights_updates[:,1], label = \"w2\")\n",
        "plt.plot(weights_updates[:,2], label = \"w3\")\n",
        "plt.ylabel('weights')\n",
        "plt.xlabel(\"epoch\")\n",
        "plt.legend()\n",
        "plt.title(\"Weights over training epochs - SGD\")\n",
        "plt.show()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 295
        },
        "id": "7tAO0GTLrreB",
        "outputId": "79fdb0a5-6529-4ea0-ee40-399508f91d23"
      },
      "execution_count": 237,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ],
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYoAAAEWCAYAAAB42tAoAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3deZwcVb338c836wAJS0LYMoEEzBUIhhBGJGgiyiKgLHLhSuQqmzcPV3gQdxAU9IoLel0QHiECAl6UTZEIwQgEJKAsEyEQApEIIhOCGQMkQAxk+T1/1BluTfdMT89M9/TM5Pt+vfqVrlOn65zqmtSvzzlVpxQRmJmZtWdArStgZma9mwOFmZmV5EBhZmYlOVCYmVlJDhRmZlaSA4WZmZXkQGGtSDpe0u/KzHuipPuqXae+SNITkvavdN7eTtJVkr5e63pYZTlQ9AOSzpZ0e0Ha0+2kHVdqWxFxbUQcXKF63SPpE5XYVk+RNFZSSBrUne1ExISIuKfSeS0jaUtJV0p6UdKrkv4s6azcekk6XdJjklanfPfk//7T8pr0+VWS5ks6S9LQ2uxV7+VA0T/cC+wnaSCApO2BwcBeBWlvS3kN6Gow6G4QsYr4PjAM2A3YAjgCWJJbfxFwJvBZYCQwGjgXOKRgO6dHxHBg+5T3OGC2JFW19n2MA0X/8DBZYJiUlqcCdwOLC9L+EhEvSNpC0hWSlklaKunruYDSqjtJ0sGSFktaKen/Sfp9YStB0nclvSzpWUmHprQLUpkXS3pN0sXpV973JS1Pv+Ael7RHWzskaQdJsyS9JGmJpP/Ipf9T0ohc3r0k/UPS4LR8sqQnU53mSNoplzcknSbpaeDpNopuCaSvpHpPSd/J/anuK4DzJe0iaa6kFansayVtmSvnr5IOTO/Pl3SDpGvSr9cnJDV0Me9kSY+kdTdKur5UV08Z38UZkp5J+/AdSQPSugGSzpX0XDpe10jaIvfZ90j6g6RXJD0v6cRcsVtJui3V8UFJu6TPlH38y/BO4OcR8XJEbIiIpyLiplTOvwCfBI6LiDsi4p8RsT4i7ouIE9vaWES8nlp1RwBTgA92sV79kgNFPxARbwIPAtNS0jRgHnBfQVrLSfAqYB1ZC2Mv4GCgqItI0tbATcDZZL/KFgP7FWR7V0rfGrgQuEKSIuKcVIfTI2JYRJyeypkG/AvZr8B/A1a0s1vXAU3ADsAxwDckvT8iXgD+CPxrLu9HgZsiYq2kI4EvAUcDo1IdflGw7aNSvXdvo9yW72vLVO8/5vbzGWBb4AJAwDdT/XYDxgDnt7MvkJ2ArgO2BGYBF3c2r6QhwM1kx29E2q8Pt7eRMr+LDwMNwGTgSODklH5ier0P2Jns13tLPXYCbgd+lLY7CXg0t83jgK8CW5H9yr8gpXfm+HfkAeACSSdJGl+w7v3A8xHR2NmNRsTfgEayHznWIiL86gcvspPUzen9AmA8WTM7n3YC2YnuDWCT3GenA3en9ycC96X3Hwf+mMsn4HngE7m8S3LrNwUC2C4t39OSNy2/H/gzsC8woMS+jAHWA8Nzad8ErkrvPwHMLajTtLR8O3BK7nMDgNXATmk5gPeXKHtsyjMol3Yi8LcOvv+jgEdyy38FDswdmztz63YH/tnZvGQn2aWAcuvvA77eTp3K+S4Oya3/JHBXen8X8MncurcDa4FBZD8cbm6nzKuAy3PLhwFPdeb4l/n3vglZEJyf6rUEODStOxd4oCB/E/AKsCa3/63+PnN5rwN+Us3/r33t5RZF/3Ev8J7UJTMqIp4G/kA2djEC2CPl2Ymsm2pZ6jZ4BbgM2KaNbe5AdhIGILL/RU0FeV7MrV+d3g5rq4IRMZfsV+klwHJJMyVt3k65L0XEq7m058j6mQF+CUxRNu4yDdhA9muZtH8/zO3bS2TBZHRuW8/Tea0+I2lbSdcp67pbBfwPWauqPS/m3q8G6tT+WEd7eXcAlqbj0Ga9CnT2u3gulUH697mCdYPIfmiMAf5SotzC+g+D8o+/pKmp2+81SU+0VUBk3UnfiIi9yVq7NwA3pr/1FWRjDvn89WTHZyjZd1DKaLLvyhIHiv7jj2TN+f8A7geIiFXACynthYh4luzE8AawdURsmV6bR8SENra5DKhvWZCk/HIZiqYmjoiL0n/u3cm6ID7fxudeAEZIGp5L25Hs1zQR8TLwO+AjZN1O1+VOns8D/ye3b1tGxCYR8YdS9SpjXWH6N1LaOyJic+Df6fgE1F3LgNHpOLQYUyJ/Od9F/vM7kn33pH93Kli3Dvh72u4uXdmBco5/RMyLrNtvWDt/l4X5V5Edj82AccBcoD4/tlMuSWOAvfnfHx6GA0W/ERH/JOtb/Qyt/8jvS2n3pnzLyE6y/y1p8zRouYuk97ax2duAd0g6Kv2iPQ3YrhPV+jtZ/zYAkt4p6V3KBp1fJ+sG2NDGvjxP1hr6pqQ6SROBU8h+tbf4OVnX2DHpfYtLgbMlTUhlbiHp2E7UuTnVaecO8g0HXgNWShpN2wGv0v5I1iV3uqRBaQxinxL5y/kuPi9pq3SC/BRwfUr/BfBpSeMkDSM7EV8fEeuAa4EDJf1bqsdISZPoQLnHvxySvpy2N0RSXar7K8DiiFhM1kq+TtJBkjZRdrFG4fhafnubpv8DtwAPAbO7Uq/+yoGif/k9WRdS/ia4eSktf1nsx4EhwCLgZbIB61ZNdYCI+AdwLNkg9QqyX4GNZC2ScvwQOCZdcXMRsDnwk1Tmc2mb32nns9PJxgteIBvAPS8i7sytn0U2DvNiRCzI1flm4NtkJ4lVwELg0DLr29J9dgFwf+qy2bedrF8lGwBeSRZQf1VuGV0V2UULR5MFzVfIWjG30s7xKPO7uIWsn/9Rsv24IqVfCfyM7O/mWbKT+v9N2/0b2djDZ8m6aB4F9ixjFzpz/DsSwE+Bf5D9jRwEfDAiXkvrTyO7RPZ7qY5NwH+RtUL/ltvOxZJeJftR8wOybs1DIqJLAay/UuvuTrP2pUsnm4DjI+LuWtfHQNKDwKUR8dMufDaA8RGxpMPMtlFzi8JKkvQBZXfBDiW7ykRklyZaDUh6r6TtUpfPCcBE4Le1rpf1b77D1DoyhWwMoKWr6qg0HmK18XayK3w2I7uv45g07mRWNe56MjOzktz1ZGZmJfXLrqett946xo4dW+tqmJn1GfPnz/9HRIxqa12/DBRjx46lsbHT07yYmW20JD3X3jp3PZmZWUkOFGZmVpIDhZmZleRAYWZmJdUsUEgaI+luSYuUPcXrU23kkaSLlD3h7DFJk2tRVzOzjVktr3paB3w2Iv6UppOeL+mOiFiUy3Mo2cRv48meMPbj9K+ZmfWQmrUoImJZRPwpvX8VeJLWD1SB7NGM10TmAWDL9LAaMzPrIb3iPgpJY8me3fxgwarRtH4CV1NKK5rbRtIMYAbAjjvu2KV6LH5pMXc8d0eXPmtmVmubDt6Uk/c4ueOMnVTzQJEeivJL4Mz0pKouiYiZwEyAhoaGLk1gdeXCK5n97GxU9QeVmZlV3shNRva/QJGedPVL4NqIaOvBL0tp/ajG+pRWFWs3rGWXLXbh10f9ulpFmJn1ObW86klkT9N6MiK+1062WcDH09VP+wIrqzmlckTQ+nHEZmZWyxbFu4GPAY9LejSlfYnsIe5ExKVkz609DFgCrAZOqmaF1sd6Bsi3lpiZ5dUsUETEfVB6MCCyh2Wc1jM1yloUAzWwp4ozM+sT/PM5ZwMb3PVkZlbAgSJnfaxngL8SM7NWfFbMiQgGDPBXYmaW57NijlsUZmbFfFbMiQhf9WRmVsBnxZwNscGBwsysgM+KOQ4UZmbFfFbMcaAwMyvms2KOA4WZWTGfFXMcKMzMivmsmLMBBwozs0I+K+ZsiA2+j8LMrIDPijnuejIzK+azYo4DhZlZMZ8VcxwozMyK+ayY40BhZlbMZ8WcDeHnUZiZFXKgyNkQG/yEOzOzAg4UOe56MjMr5rNijm+4MzMrVtOzoqQrJS2XtLCd9ftLWinp0fT6SjXr4xaFmVmxQTUu/yrgYuCaEnnmRcSHeqIyG2IDwoPZZmZ5Nf35HBH3Ai/Vsg55Hsw2MyvWF/pZpkhaIOl2SRPayyRphqRGSY3Nzc1dKsiXx5qZFevtgeJPwE4RsSfwI+DX7WWMiJkR0RARDaNGjepSYW5RmJkV69WBIiJWRcRr6f1sYLCkratYngezzcwK9OqzoqTtlPqCJO1DVt8V1Spvfax315OZWYGaXvUk6RfA/sDWkpqA84DBABFxKXAM8J+S1gH/BI6LiKhWfYJw15OZWYGaBoqImN7B+ovJLp/tEYNefS/Ll9f3VHFmZn1Cr+566mmrlh3IZuvfUetqmJn1Kg4UOYMHirXrN9S6GmZmvYoDRc6QQQN4c50DhZlZngNFzpBBA9yiMDMr4ECRM3jgAN5woDAza8WBImfIwAGsddeTmVkrDhQ5QwYN4E23KMzMWnGgyBky0GMUZmaFHChyBg/0VU9mZoUcKHKyrqeqzRBiZtYnOVDkuEVhZlas1o9C7VWGDh7A4hdXsfd/3VHrqpiZddqIzYZwx2feW/HtOlDknPzusWy16eBaV8PMrEuGDa3O+cuBImfvnUaw904jal0NM7NexWMUZmZWkgOFmZmV5EBhZmYlOVCYmVlJDhRmZlZSTQOFpCslLZe0sJ31knSRpCWSHpM0uafraGa2sav15bFXARcD17Sz/lBgfHq9C/hx+rc61qyCZQuqtnkzs6oaOAR2rPwpsqaBIiLulTS2RJYjgWsiIoAHJG0pafuIWFaVCv3uHPhTezHLzKyX22wb+PzTFd9srVsUHRkNPJ9bbkppRYFC0gxgBsCOO+7YtdLeeA02Hw0fvqxrnzczq6WBQ6qy2d4eKMoWETOBmQANDQ1dnAI2YPCmMG5qBWtmZta39farnpYCY3LL9SnNzMx6SG8PFLOAj6ern/YFVlZtfAIgAqSqbd7MrC+qadeTpF8A+wNbS2oCzgMGA0TEpcBs4DBgCbAaOKkHalX9IszM+pBaX/U0vYP1AZzWQ9UB/HQ7M7NCvb3rqWe568nMrIgDhZmZleRA0UrgMQozs9YcKMzMrCQHijyPUZiZFXGgMDOzkhwoirhFYWaW50BhZmYlOVDkeYzCzKyIA0URBwozszwHilY8hYeZWSEHirwINyjMzAo4UJiZWUkOFK14Cg8zs0IOFGZmVpIDRZ4vjzUzK+JAYWZmJTlQtOIxCjOzQg4Uhdz1ZGbWSk0DhaRDJC2WtETSWW2sP1FSs6RH0+sTVa1Q+IY7M7NCg8rJJGkXoCki3pC0PzARuCYiXulqwZIGApcABwFNwMOSZkXEooKs10fE6V0tpws167mizMz6gHJbFL8E1kt6GzATGAP8vJtl7wMsiYhnIuJN4DrgyG5us5vcojAzK1RuoNgQEeuADwM/iojPA9t3s+zRwPO55aaUVuhfJT0m6SZJY9rbmKQZkholNTY3N3etRr481sysSLmBYq2k6cAJwK0pbXB1qtTKb4CxETERuAO4ur2METEzIhoiomHUqFE9UDUzs41DuYHiJGAKcEFEPCtpHPCzbpa9lKwLq0V9SntLRKyIiDfS4uXA3t0sswO+PNbMrFBZg9nAQRFxRstCChZruln2w8D4FHSWAscBH81nkLR9RCxLi0cAT3azTDMz66RyWxQntJF2YncKTmMepwNzyALADRHxhKSvSToiZTtD0hOSFgBndLfMMirlMQozswIlWxRpXOKjwDhJs3KrhgMvdbfwiJgNzC5I+0ru/dnA2d0tx8zMuq6jrqc/AMuArYH/zqW/CjxWrUrVjscozMwKlQwUEfEc8BzZQPbGwV1PZmatlDVGIeloSU9LWilplaRXJa2qduV6nKfwMDMrUu5VTxcCh0fERnDVkVsUZmZ55V719PeNI0i4RWFmVqijq56OTm8bJV0P/BpouQGOiPhVFevW83x5rJlZkY66ng7PvV8NHJxbDqB/BQozMyvS0VVPJ/VURXoPtyjMzPLKfR7FRW0krwQaI+KWylbJzMx6k3IHs+uAScDT6TWRbBK/UyT9oEp163keozAzK1Lu5bETgXdHxHoAST8G5gHvAR6vUt3MzKwXKLdFsRUwLLe8GTAiBY432v5IX+QpPMzMCnXmhrtHJd1DdiadBnxD0mbAnVWqW22468nMrJWyAkVEXCFpNtlzrgG+FBEvpPefr0rNaiHCDQozswIlu54k7Zr+nUz2jOzn02u7lGZmZv1cRy2KzwAzaD3FeIsA3l/xGtWUp/AwMyvU0Q13M9K/7+uZ6tSYL481MytS7jTjm0o6V9LMtDxe0oeqWzUzM+sNyr089qfAm8B+aXkp8PWq1KimfHmsmVmhcgPFLhFxIbAWICJWU4EzqqRDJC2WtETSWW2sHyrp+rT+QUlju1ummZl1TrmB4k1Jm5BGeyXtQjdvtJM0ELgEOBTYHZguafeCbKcAL0fE24DvA9/uTpkd8hiFmVmRcgPFecBvgTGSrgXuAr7QzbL3AZZExDMR8SZwHXBkQZ4jgavT+5uAAySfyc3MelK5d2afANxGdrJ+BvhURPyjm2WPJrsno0UT8K728kTEOkkrgZFAd8tuh8cozMwKlduiuIJsBtkjgB8Bl0n6VNVq1QWSZkhqlNTY3NzcnQ1VrlJmZv1AWYEiIu4GLgC+DPwEaAD+s5tlLwXG5JbrU1qbeSQNArYAVrRTx5kR0RARDaNGjepajcI33JmZFSr3Poq7gPuBjwCLgXdGxK7dLPthYLykcZKGAMcBswryzCLr9gI4BpgbUe2zuVsUZmZ55XY9PUZ2H8UeZM+m2CNdBdVlEbEOOB2YAzwJ3BART0j6mqQjUrYrgJGSlpBNJ1J0CW1luUVhZlao3NljPw0gaThwItkNeNsBQ7tTeETMBmYXpH0l934NcGx3yuhkhTxGYWZWoNxnZp8OTAX2Bv4KXEn2hDszM+vnyr08tg74HjA/dRn1U7481sysULldT9+tdkXMzKx3Kncwe+PgMQozsyIOFEUcKMzM8hwoWvHlsWZmhRwoCrnrycysFQeKPE/hYWZWxIGiiFsUZmZ5DhStuEVhZlbIgSIv8BiFmVkBBwozMyvJgaIVT+FhZlbIgcLMzEpyoMjzFB5mZkUcKMzMrCQHilZ8eayZWSEHikLuejIza8WBIs9TeJiZFXGgKOIWhZlZXk0ChaQRku6Q9HT6d6t28q2X9Gh6zap+zdyiMDMrVKsWxVnAXRExHrgrLbflnxExKb2OqHqtfHmsmVmRWgWKI4Gr0/urgaNqVA8zM+tArQLFthGxLL1/Edi2nXx1kholPSCpZDCRNCPlbWxubu5itTyFh5lZoUHV2rCkO4Ht2lh1Tn4hIkJSe4MDO0XEUkk7A3MlPR4Rf2krY0TMBGYCNDQ0eLDBzKxCqhYoIuLA9tZJ+ruk7SNimaTtgeXtbGNp+vcZSfcAewFtBoqK8BiFmVmRWnU9zQJOSO9PAG4pzCBpK0lD0/utgXcDi6pfNQcKM7O8WgWKbwEHSXoaODAtI6lB0uUpz25Ao6QFwN3AtyKiyoHCPVZmZoWq1vVUSkSsAA5oI70R+ER6/wfgHT1cNXc9mZkV8J3ZeZ7Cw8ysiANFEbcozMzyHChacYvCzKyQA0WeL481MyviQGFmZiU5ULTiKTzMzAo5UJiZWUkOFHkeozAzK1KTG+56NwcKM2vb2rVraWpqYs2aNbWuSpfV1dVRX1/P4MGDy/6MA0UrvjzWzNrX1NTE8OHDGTt2LOqDvQ8RwYoVK2hqamLcuHFlf85dT4X64ME3s56xZs0aRo4c2SeDBIAkRo4c2ekWkQNFnhsUZtaBvhokWnSl/g4URfr2H4GZWaU5ULTiJoWZ9V3nnHMOY8aMYdiwYRXdrgNFni+PNbM+7PDDD+ehhx6q+HZ91ZOZWRd89TdPsOiFVRXd5u47bM55h09od/13vvMdhg4dyhlnnMGnP/1pFixYwNy5c5k7dy5XXHEF1157bUXr08ItilY8hYeZ9V5Tp05l3rx5ADQ2NvLaa6+xdu1a5s2bx7Rp06pWrlsUhRwnzKwMpX75V8vee+/N/PnzWbVqFUOHDmXy5Mk0NjYyb948LrrooqqV60CR5yfcmVkvNnjwYMaNG8dVV13Ffvvtx8SJE7n77rtZsmQJu+22W9XKrUnXk6RjJT0haYOkhhL5DpG0WNISSWf1UO16phgzsy6YOnUq3/3ud5k2bRpTp07l0ksvZa+99qrq/R21GqNYCBwN3NteBkkDgUuAQ4HdgemSdq9utdyiMLPeberUqSxbtowpU6aw7bbbUldXx9SpUwH4whe+QH19PatXr6a+vp7zzz+/ImXWpOspIp6EDu8Q3AdYEhHPpLzXAUcCi6paOV8ea2a92AEHHMDatWvfWv7zn//81vsLL7yQCy+8sOJl9uarnkYDz+eWm1JamyTNkNQoqbG5ublrJXqMwsysSNVaFJLuBLZrY9U5EXFLpcuLiJnATICGhoZunPHdojAzy6taoIiIA7u5iaXAmNxyfUqrIrcozMwK9eaup4eB8ZLGSRoCHAfMqmqJnsLDzKxIrS6P/bCkJmAKcJukOSl9B0mzASJiHXA6MAd4ErghIp6oRX3NzDZmtbrq6Wbg5jbSXwAOyy3PBmb3YM3wGIWZWWu9ueupNtz1ZGZ90OrVq/ngBz/IrrvuyoQJEzjrrMrdo+xAkefLY82sD/vc5z7HU089xSOPPML999/P7bffXpHteq6nIm5RmFkZbj8LXny8stvc7h1w6LfaXV3uNONDhgxh8uTJNDU1VaRablG04haFmfVe5U4z/sorr/Cb3/yGAw44oCLlukVRyGMUZlaOEr/8q6WcacbXrVvH9OnTOeOMM9h5550rUq4DRZ7HKMysFytnmvEZM2Ywfvx4zjzzzIqV666nIm5RmFnvVWqa8XPPPZeVK1fygx/8oKJlOlC04haFmfVu7U0z3tTUxAUXXMCiRYuYPHkykyZN4vLLL69Ime56yvMUHmbWy5WaZjyq1H3uFoWZmZXkQNGKp/AwMyvkQFHIXU9mZq04UOR5LNvMrIgDRRG3KMzM8hwoWnGTwsyskANFIY9RmFkfdcghh7DnnnsyYcIETj31VNavX1+R7TpQ5HkKDzPrw2644QYWLFjAwoULaW5u5sYbb6zIdn3DXRG3KMysY99+6Ns89dJTFd3mriN25Yv7fLHd9eVOM75u3TrefPNNVKEeErcoWnGLwsx6r3KmGf/ABz7ANttsw/DhwznmmGMqUm5NWhSSjgXOB3YD9omIxnby/RV4FVgPrIuIhqpWzFN4mFmZSv3yr5ZyphmfM2cOa9as4fjjj2fu3LkcdNBB3S63Vi2KhcDRwL1l5H1fREyqepAwM+vlCqcZnzp1atE04wB1dXUceeSR3HLLLRUptyaBIiKejIjFtSi7NHc9mVnv1t4046+//jrLli0DsjGK2267jV133bUiZfb2MYoAfidpvqQZpTJKmiGpUVJjc3Nz10t015OZ9WLtTTP++uuvc8QRRzBx4kQmTZrENttsw6mnnlqRMqs2RiHpTmC7NladExHltofeExFLJW0D3CHpqYhos7sqImYCMwEaGhq61jTY7XDYdo8ufdTMrCeUmmb84YcfrkqZVQsUEXFgBbaxNP27XNLNwD6UN67RNUfPrNqmzcz6ql7b9SRpM0nDW94DB5MNgpuZWQ+qSaCQ9GFJTcAU4DZJc1L6DpJmp2zbAvdJWgA8BNwWEb+tRX3NzFpU6ylyPaUr9a/JfRQRcTNwcxvpLwCHpffPAHv2cNXMzNpVV1fHihUrGDlyZMXueu5JEcGKFSuoq6vr1Oc8hYeZWZnq6+tpamqiW1dW1lhdXR319fWd+owDhZlZmVpueNvY9NrBbDMz6x0cKMzMrCQHCjMzK0l9/VKvtkhqBp7r4se3Bv5Rwer0Bd7n/m9j21/wPnfWThExqq0V/TJQdIekxo1tplrvc/+3se0veJ8ryV1PZmZWkgOFmZmV5EBRbGOcGdD73P9tbPsL3ueK8RiFmZmV5BaFmZmV5EBhZmYlOVAkkg6RtFjSEkln1bo+lSJpjKS7JS2S9ISkT6X0EZLukPR0+nerlC5JF6Xv4TFJk2u7B10naaCkRyTdmpbHSXow7dv1koak9KFpeUlaP7aW9e4qSVtKuknSU5KelDSlvx9nSZ9Of9cLJf1CUl1/O86SrpS0XNLCXFqnj6ukE1L+pyWd0Jk6OFCQnVCAS4BDgd2B6ZJ2r22tKmYd8NmI2B3YFzgt7dtZwF0RMR64Ky1D9h2MT68ZwI97vsoV8yngydzyt4HvR8TbgJeBU1L6KcDLKf37KV9f9EPgtxGxK9kU/U/Sj4+zpNHAGUBDROwBDASOo/8d56uAQwrSOnVcJY0AzgPeRfak0PNagktZImKjf5E9QGlObvls4Oxa16tK+3oLcBCwGNg+pW0PLE7vLwOm5/K/la8vvYD69B/o/cCtgMjuWB1UeMyBOcCU9H5Qyqda70Mn93cL4NnCevfn4wyMBp4HRqTjdivwgf54nIGxwMKuHldgOnBZLr1Vvo5eblFkWv7gWjSltH4lNbX3Ah4Eto2IZWnVi2RPFIT+8138APgCsCEtjwReiYh1aTm/X2/tc1q/MuXvS8YBzcBPU3fb5ekRwv32OEfEUuC7wN+AZWTHbT79+zi36Oxx7dbxdqDYSEgaBvwSODMiVuXXRfYTo99cJy3pQ8DyiJhf67r0oEHAZODHEbEX8Dr/2x0B9MvjvBVwJFmQ3AHYjOIumn6vJ46rA0VmKTAmt1yf0voFSYPJgsS1EfGrlPx3Sdun9dsDy1N6f/gu3g0cIemvwHVk3U8/BLaU1PKwrvx+vbXPaf0WwIqerHAFNAFNEfFgWr6JLHD05+N8IPBsRDRHxFrgV2THvj8f5xadPa7dOt4OFJmHgfHpaokhZANis2pcp4pQ9mDfK4AnI+J7uVWzgJYrH04gG7toSf94unpiX2BlronbJ0TE2RFRHxFjyY7l3Ig4HrgbOCZlK9znlu/imJS/T/3yjogXgeclvT0lHQAsoh8fZ7Iup30lbZr+zlv2ud8e55zOHscVczYAAAJeSURBVNc5wMGStkotsYNTWnlqPUjTW17AYcCfgb8A59S6PhXcr/eQNUsfAx5Nr8PI+mbvAp4G7gRGpPwiuwLsL8DjZFeU1Hw/urH/+wO3pvc7Aw8BS4AbgaEpvS4tL0nrd651vbu4r5OAxnSsfw1s1d+PM/BV4ClgIfAzYGh/O87AL8jGYNaStRxP6cpxBU5O+74EOKkzdfAUHmZmVpK7nszMrCQHCjMzK8mBwszMSnKgMDOzkhwozMysJAcKs15E0v4ts92a9RYOFGZmVpIDhVkXSPp3SQ9JelTSZenZF69J+n56PsJdkkalvJMkPZCeD3Bz7tkBb5N0p6QFkv4kaZe0+WG550pcm+46NqsZBwqzTpK0G/AR4N0RMQlYDxxPNildY0RMAH5PNv8/wDXAFyNiItndsi3p1wKXRMSewH5kd99CNsPvmWTPRtmZbP4is5oZ1HEWMytwALA38HD6sb8J2aRsG4DrU57/AX4laQtgy4j4fUq/GrhR0nBgdETcDBARawDS9h6KiKa0/CjZswjuq/5umbXNgcKs8wRcHRFnt0qUvlyQr6vz47yRe78e/z+1GnPXk1nn3QUcI2kbeOv5xTuR/X9qmbX0o8B9EbESeFnS1JT+MeD3EfEq0CTpqLSNoZI27dG9MCuTf6mYdVJELJJ0LvA7SQPIZvU8jexhQfukdcvJxjEgmwb60hQIngFOSukfAy6T9LW0jWN7cDfMyubZY80qRNJrETGs1vUwqzR3PZmZWUluUZiZWUluUZiZWUkOFGZmVpIDhZmZleRAYWZmJTlQmJlZSf8f7ebZ4Gf578kAAAAASUVORK5CYII=\n"
          },
          "metadata": {
            "needs_background": "light"
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "In the previous example of batch gradient descent, it looked like our model began to converge around 200 training epochs. Here, because we updated the model weights after evaluating each sample, we appear to have actually minimized the cost very early on in the training process. However, the weights that the model has converged on seem to be slightly less accurate for $w_1$ compared to batch gradient descent."
      ],
      "metadata": {
        "id": "OAOM4NZQreQz"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "model_results['SGD'] = weights\n",
        "model_results"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 171
        },
        "id": "cqGrtsrIrbgs",
        "outputId": "807dbe79-5fde-4b38-ad19-22139e4e126b"
      },
      "execution_count": 238,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "   batch gradient descent  linear regression  true coeffs       SGD\n",
              "0                1.000242           1.000321          1.0  1.179115\n",
              "1                1.000179           1.000202          1.0  1.008264\n",
              "2                2.020670           2.020693          2.0  1.985073"
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-6a1cf85a-d5ac-4242-9b43-ceccc7e0ab2f\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>batch gradient descent</th>\n",
              "      <th>linear regression</th>\n",
              "      <th>true coeffs</th>\n",
              "      <th>SGD</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>1.000242</td>\n",
              "      <td>1.000321</td>\n",
              "      <td>1.0</td>\n",
              "      <td>1.179115</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>1.000179</td>\n",
              "      <td>1.000202</td>\n",
              "      <td>1.0</td>\n",
              "      <td>1.008264</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>2.020670</td>\n",
              "      <td>2.020693</td>\n",
              "      <td>2.0</td>\n",
              "      <td>1.985073</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-6a1cf85a-d5ac-4242-9b43-ceccc7e0ab2f')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "        \n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "      \n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-6a1cf85a-d5ac-4242-9b43-ceccc7e0ab2f button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-6a1cf85a-d5ac-4242-9b43-ceccc7e0ab2f');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n",
              "  "
            ]
          },
          "metadata": {},
          "execution_count": 238
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Example - mini-batch stochastic gradient descent\n",
        "\n",
        "Now let's modify the code from the SGD example to accomodate mini-batch gradient descent. Let's define the batch size as $m=100$ to yield 100 batches per epoch. "
      ],
      "metadata": {
        "id": "UVOIilqd7vU-"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "def batch(X, y, batch_size):\n",
        "    '''\n",
        "    code for partitioning dataset into batcehs\n",
        "    '''\n",
        "    \n",
        "    # shuffle indices of samples \n",
        "    sample_ix = np.arange(n)\n",
        "    np.random.shuffle(sample_ix)\n",
        "    \n",
        "    batches = []\n",
        "\n",
        "    for batch_ix in range(n//m):\n",
        "        # determine which samples to pick for batch\n",
        "        samples_in_batch = sample_ix[batch_ix*m:(batch_ix*m + m )]\n",
        "        batches.append([X[samples_in_batch,:], y[samples_in_batch]])\n",
        "\n",
        "    return batches"
      ],
      "metadata": {
        "id": "owPTeiBqQ5y7"
      },
      "execution_count": 252,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# define batch size\n",
        "m = 100\n",
        "\n",
        "# reinitialize weights\n",
        "weights = np.random.randn(3)\n",
        "\n",
        "# collect avg cost at each epoch\n",
        "avg_cost_list = []\n",
        "\n",
        "# collect the updated weights at each epoch\n",
        "weights_updates = np.zeros((epochs+1, 3))\n",
        "weights_updates[0,:] = weights\n",
        "\n",
        "# get batched data\n",
        "batches = batch(X, y, m)\n",
        "\n",
        "start = time.time()\n",
        "\n",
        "for iter in range(epochs):\n",
        "    print('epoch = %d' % iter)\n",
        "    \n",
        "    # collect losses for every batch in epoch\n",
        "    epoch_loss = []\n",
        "\n",
        "    for batch_ix in range(X.shape[0]//m):\n",
        "    \n",
        "        batch_X = batches[batch_ix][0]\n",
        "        batch_y = batches[batch_ix][1]\n",
        "\n",
        "        # predict on training set \n",
        "        y_pred = pred(batch_X, weights)\n",
        "\n",
        "        # calculate cost\n",
        "        cost = mse(y_pred, batch_y, m)\n",
        "        epoch_loss.append(cost)\n",
        "        \n",
        "        # update the weights\n",
        "        for i in range(weights.shape[0]):\n",
        "            weights[i] = weights[i] - lr*grad(m, y_pred, batch_y, batch_X[:,i])\n",
        "    \n",
        "    # save avg cost for epoch across all batches\n",
        "    avg_cost_list.append(np.average(epoch_loss))\n",
        "    weights_updates[iter+1,:] = weights\n",
        "\n",
        "end = time.time()\n",
        "\n",
        "minibatch_time = end - start"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "b8meT2u5sQH5",
        "outputId": "15cae735-a5ad-4e80-f933-353252d3db06"
      },
      "execution_count": 253,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "epoch = 0\n",
            "epoch = 1\n",
            "epoch = 2\n",
            "epoch = 3\n",
            "epoch = 4\n",
            "epoch = 5\n",
            "epoch = 6\n",
            "epoch = 7\n",
            "epoch = 8\n",
            "epoch = 9\n",
            "epoch = 10\n",
            "epoch = 11\n",
            "epoch = 12\n",
            "epoch = 13\n",
            "epoch = 14\n",
            "epoch = 15\n",
            "epoch = 16\n",
            "epoch = 17\n",
            "epoch = 18\n",
            "epoch = 19\n",
            "epoch = 20\n",
            "epoch = 21\n",
            "epoch = 22\n",
            "epoch = 23\n",
            "epoch = 24\n",
            "epoch = 25\n",
            "epoch = 26\n",
            "epoch = 27\n",
            "epoch = 28\n",
            "epoch = 29\n",
            "epoch = 30\n",
            "epoch = 31\n",
            "epoch = 32\n",
            "epoch = 33\n",
            "epoch = 34\n",
            "epoch = 35\n",
            "epoch = 36\n",
            "epoch = 37\n",
            "epoch = 38\n",
            "epoch = 39\n",
            "epoch = 40\n",
            "epoch = 41\n",
            "epoch = 42\n",
            "epoch = 43\n",
            "epoch = 44\n",
            "epoch = 45\n",
            "epoch = 46\n",
            "epoch = 47\n",
            "epoch = 48\n",
            "epoch = 49\n",
            "epoch = 50\n",
            "epoch = 51\n",
            "epoch = 52\n",
            "epoch = 53\n",
            "epoch = 54\n",
            "epoch = 55\n",
            "epoch = 56\n",
            "epoch = 57\n",
            "epoch = 58\n",
            "epoch = 59\n",
            "epoch = 60\n",
            "epoch = 61\n",
            "epoch = 62\n",
            "epoch = 63\n",
            "epoch = 64\n",
            "epoch = 65\n",
            "epoch = 66\n",
            "epoch = 67\n",
            "epoch = 68\n",
            "epoch = 69\n",
            "epoch = 70\n",
            "epoch = 71\n",
            "epoch = 72\n",
            "epoch = 73\n",
            "epoch = 74\n",
            "epoch = 75\n",
            "epoch = 76\n",
            "epoch = 77\n",
            "epoch = 78\n",
            "epoch = 79\n",
            "epoch = 80\n",
            "epoch = 81\n",
            "epoch = 82\n",
            "epoch = 83\n",
            "epoch = 84\n",
            "epoch = 85\n",
            "epoch = 86\n",
            "epoch = 87\n",
            "epoch = 88\n",
            "epoch = 89\n",
            "epoch = 90\n",
            "epoch = 91\n",
            "epoch = 92\n",
            "epoch = 93\n",
            "epoch = 94\n",
            "epoch = 95\n",
            "epoch = 96\n",
            "epoch = 97\n",
            "epoch = 98\n",
            "epoch = 99\n",
            "epoch = 100\n",
            "epoch = 101\n",
            "epoch = 102\n",
            "epoch = 103\n",
            "epoch = 104\n",
            "epoch = 105\n",
            "epoch = 106\n",
            "epoch = 107\n",
            "epoch = 108\n",
            "epoch = 109\n",
            "epoch = 110\n",
            "epoch = 111\n",
            "epoch = 112\n",
            "epoch = 113\n",
            "epoch = 114\n",
            "epoch = 115\n",
            "epoch = 116\n",
            "epoch = 117\n",
            "epoch = 118\n",
            "epoch = 119\n",
            "epoch = 120\n",
            "epoch = 121\n",
            "epoch = 122\n",
            "epoch = 123\n",
            "epoch = 124\n",
            "epoch = 125\n",
            "epoch = 126\n",
            "epoch = 127\n",
            "epoch = 128\n",
            "epoch = 129\n",
            "epoch = 130\n",
            "epoch = 131\n",
            "epoch = 132\n",
            "epoch = 133\n",
            "epoch = 134\n",
            "epoch = 135\n",
            "epoch = 136\n",
            "epoch = 137\n",
            "epoch = 138\n",
            "epoch = 139\n",
            "epoch = 140\n",
            "epoch = 141\n",
            "epoch = 142\n",
            "epoch = 143\n",
            "epoch = 144\n",
            "epoch = 145\n",
            "epoch = 146\n",
            "epoch = 147\n",
            "epoch = 148\n",
            "epoch = 149\n",
            "epoch = 150\n",
            "epoch = 151\n",
            "epoch = 152\n",
            "epoch = 153\n",
            "epoch = 154\n",
            "epoch = 155\n",
            "epoch = 156\n",
            "epoch = 157\n",
            "epoch = 158\n",
            "epoch = 159\n",
            "epoch = 160\n",
            "epoch = 161\n",
            "epoch = 162\n",
            "epoch = 163\n",
            "epoch = 164\n",
            "epoch = 165\n",
            "epoch = 166\n",
            "epoch = 167\n",
            "epoch = 168\n",
            "epoch = 169\n",
            "epoch = 170\n",
            "epoch = 171\n",
            "epoch = 172\n",
            "epoch = 173\n",
            "epoch = 174\n",
            "epoch = 175\n",
            "epoch = 176\n",
            "epoch = 177\n",
            "epoch = 178\n",
            "epoch = 179\n",
            "epoch = 180\n",
            "epoch = 181\n",
            "epoch = 182\n",
            "epoch = 183\n",
            "epoch = 184\n",
            "epoch = 185\n",
            "epoch = 186\n",
            "epoch = 187\n",
            "epoch = 188\n",
            "epoch = 189\n",
            "epoch = 190\n",
            "epoch = 191\n",
            "epoch = 192\n",
            "epoch = 193\n",
            "epoch = 194\n",
            "epoch = 195\n",
            "epoch = 196\n",
            "epoch = 197\n",
            "epoch = 198\n",
            "epoch = 199\n",
            "epoch = 200\n",
            "epoch = 201\n",
            "epoch = 202\n",
            "epoch = 203\n",
            "epoch = 204\n",
            "epoch = 205\n",
            "epoch = 206\n",
            "epoch = 207\n",
            "epoch = 208\n",
            "epoch = 209\n",
            "epoch = 210\n",
            "epoch = 211\n",
            "epoch = 212\n",
            "epoch = 213\n",
            "epoch = 214\n",
            "epoch = 215\n",
            "epoch = 216\n",
            "epoch = 217\n",
            "epoch = 218\n",
            "epoch = 219\n",
            "epoch = 220\n",
            "epoch = 221\n",
            "epoch = 222\n",
            "epoch = 223\n",
            "epoch = 224\n",
            "epoch = 225\n",
            "epoch = 226\n",
            "epoch = 227\n",
            "epoch = 228\n",
            "epoch = 229\n",
            "epoch = 230\n",
            "epoch = 231\n",
            "epoch = 232\n",
            "epoch = 233\n",
            "epoch = 234\n",
            "epoch = 235\n",
            "epoch = 236\n",
            "epoch = 237\n",
            "epoch = 238\n",
            "epoch = 239\n",
            "epoch = 240\n",
            "epoch = 241\n",
            "epoch = 242\n",
            "epoch = 243\n",
            "epoch = 244\n",
            "epoch = 245\n",
            "epoch = 246\n",
            "epoch = 247\n",
            "epoch = 248\n",
            "epoch = 249\n",
            "epoch = 250\n",
            "epoch = 251\n",
            "epoch = 252\n",
            "epoch = 253\n",
            "epoch = 254\n",
            "epoch = 255\n",
            "epoch = 256\n",
            "epoch = 257\n",
            "epoch = 258\n",
            "epoch = 259\n",
            "epoch = 260\n",
            "epoch = 261\n",
            "epoch = 262\n",
            "epoch = 263\n",
            "epoch = 264\n",
            "epoch = 265\n",
            "epoch = 266\n",
            "epoch = 267\n",
            "epoch = 268\n",
            "epoch = 269\n",
            "epoch = 270\n",
            "epoch = 271\n",
            "epoch = 272\n",
            "epoch = 273\n",
            "epoch = 274\n",
            "epoch = 275\n",
            "epoch = 276\n",
            "epoch = 277\n",
            "epoch = 278\n",
            "epoch = 279\n",
            "epoch = 280\n",
            "epoch = 281\n",
            "epoch = 282\n",
            "epoch = 283\n",
            "epoch = 284\n",
            "epoch = 285\n",
            "epoch = 286\n",
            "epoch = 287\n",
            "epoch = 288\n",
            "epoch = 289\n",
            "epoch = 290\n",
            "epoch = 291\n",
            "epoch = 292\n",
            "epoch = 293\n",
            "epoch = 294\n",
            "epoch = 295\n",
            "epoch = 296\n",
            "epoch = 297\n",
            "epoch = 298\n",
            "epoch = 299\n",
            "epoch = 300\n",
            "epoch = 301\n",
            "epoch = 302\n",
            "epoch = 303\n",
            "epoch = 304\n",
            "epoch = 305\n",
            "epoch = 306\n",
            "epoch = 307\n",
            "epoch = 308\n",
            "epoch = 309\n",
            "epoch = 310\n",
            "epoch = 311\n",
            "epoch = 312\n",
            "epoch = 313\n",
            "epoch = 314\n",
            "epoch = 315\n",
            "epoch = 316\n",
            "epoch = 317\n",
            "epoch = 318\n",
            "epoch = 319\n",
            "epoch = 320\n",
            "epoch = 321\n",
            "epoch = 322\n",
            "epoch = 323\n",
            "epoch = 324\n",
            "epoch = 325\n",
            "epoch = 326\n",
            "epoch = 327\n",
            "epoch = 328\n",
            "epoch = 329\n",
            "epoch = 330\n",
            "epoch = 331\n",
            "epoch = 332\n",
            "epoch = 333\n",
            "epoch = 334\n",
            "epoch = 335\n",
            "epoch = 336\n",
            "epoch = 337\n",
            "epoch = 338\n",
            "epoch = 339\n",
            "epoch = 340\n",
            "epoch = 341\n",
            "epoch = 342\n",
            "epoch = 343\n",
            "epoch = 344\n",
            "epoch = 345\n",
            "epoch = 346\n",
            "epoch = 347\n",
            "epoch = 348\n",
            "epoch = 349\n",
            "epoch = 350\n",
            "epoch = 351\n",
            "epoch = 352\n",
            "epoch = 353\n",
            "epoch = 354\n",
            "epoch = 355\n",
            "epoch = 356\n",
            "epoch = 357\n",
            "epoch = 358\n",
            "epoch = 359\n",
            "epoch = 360\n",
            "epoch = 361\n",
            "epoch = 362\n",
            "epoch = 363\n",
            "epoch = 364\n",
            "epoch = 365\n",
            "epoch = 366\n",
            "epoch = 367\n",
            "epoch = 368\n",
            "epoch = 369\n",
            "epoch = 370\n",
            "epoch = 371\n",
            "epoch = 372\n",
            "epoch = 373\n",
            "epoch = 374\n",
            "epoch = 375\n",
            "epoch = 376\n",
            "epoch = 377\n",
            "epoch = 378\n",
            "epoch = 379\n",
            "epoch = 380\n",
            "epoch = 381\n",
            "epoch = 382\n",
            "epoch = 383\n",
            "epoch = 384\n",
            "epoch = 385\n",
            "epoch = 386\n",
            "epoch = 387\n",
            "epoch = 388\n",
            "epoch = 389\n",
            "epoch = 390\n",
            "epoch = 391\n",
            "epoch = 392\n",
            "epoch = 393\n",
            "epoch = 394\n",
            "epoch = 395\n",
            "epoch = 396\n",
            "epoch = 397\n",
            "epoch = 398\n",
            "epoch = 399\n",
            "epoch = 400\n",
            "epoch = 401\n",
            "epoch = 402\n",
            "epoch = 403\n",
            "epoch = 404\n",
            "epoch = 405\n",
            "epoch = 406\n",
            "epoch = 407\n",
            "epoch = 408\n",
            "epoch = 409\n",
            "epoch = 410\n",
            "epoch = 411\n",
            "epoch = 412\n",
            "epoch = 413\n",
            "epoch = 414\n",
            "epoch = 415\n",
            "epoch = 416\n",
            "epoch = 417\n",
            "epoch = 418\n",
            "epoch = 419\n",
            "epoch = 420\n",
            "epoch = 421\n",
            "epoch = 422\n",
            "epoch = 423\n",
            "epoch = 424\n",
            "epoch = 425\n",
            "epoch = 426\n",
            "epoch = 427\n",
            "epoch = 428\n",
            "epoch = 429\n",
            "epoch = 430\n",
            "epoch = 431\n",
            "epoch = 432\n",
            "epoch = 433\n",
            "epoch = 434\n",
            "epoch = 435\n",
            "epoch = 436\n",
            "epoch = 437\n",
            "epoch = 438\n",
            "epoch = 439\n",
            "epoch = 440\n",
            "epoch = 441\n",
            "epoch = 442\n",
            "epoch = 443\n",
            "epoch = 444\n",
            "epoch = 445\n",
            "epoch = 446\n",
            "epoch = 447\n",
            "epoch = 448\n",
            "epoch = 449\n",
            "epoch = 450\n",
            "epoch = 451\n",
            "epoch = 452\n",
            "epoch = 453\n",
            "epoch = 454\n",
            "epoch = 455\n",
            "epoch = 456\n",
            "epoch = 457\n",
            "epoch = 458\n",
            "epoch = 459\n",
            "epoch = 460\n",
            "epoch = 461\n",
            "epoch = 462\n",
            "epoch = 463\n",
            "epoch = 464\n",
            "epoch = 465\n",
            "epoch = 466\n",
            "epoch = 467\n",
            "epoch = 468\n",
            "epoch = 469\n",
            "epoch = 470\n",
            "epoch = 471\n",
            "epoch = 472\n",
            "epoch = 473\n",
            "epoch = 474\n",
            "epoch = 475\n",
            "epoch = 476\n",
            "epoch = 477\n",
            "epoch = 478\n",
            "epoch = 479\n",
            "epoch = 480\n",
            "epoch = 481\n",
            "epoch = 482\n",
            "epoch = 483\n",
            "epoch = 484\n",
            "epoch = 485\n",
            "epoch = 486\n",
            "epoch = 487\n",
            "epoch = 488\n",
            "epoch = 489\n",
            "epoch = 490\n",
            "epoch = 491\n",
            "epoch = 492\n",
            "epoch = 493\n",
            "epoch = 494\n",
            "epoch = 495\n",
            "epoch = 496\n",
            "epoch = 497\n",
            "epoch = 498\n",
            "epoch = 499\n",
            "epoch = 500\n",
            "epoch = 501\n",
            "epoch = 502\n",
            "epoch = 503\n",
            "epoch = 504\n",
            "epoch = 505\n",
            "epoch = 506\n",
            "epoch = 507\n",
            "epoch = 508\n",
            "epoch = 509\n",
            "epoch = 510\n",
            "epoch = 511\n",
            "epoch = 512\n",
            "epoch = 513\n",
            "epoch = 514\n",
            "epoch = 515\n",
            "epoch = 516\n",
            "epoch = 517\n",
            "epoch = 518\n",
            "epoch = 519\n",
            "epoch = 520\n",
            "epoch = 521\n",
            "epoch = 522\n",
            "epoch = 523\n",
            "epoch = 524\n",
            "epoch = 525\n",
            "epoch = 526\n",
            "epoch = 527\n",
            "epoch = 528\n",
            "epoch = 529\n",
            "epoch = 530\n",
            "epoch = 531\n",
            "epoch = 532\n",
            "epoch = 533\n",
            "epoch = 534\n",
            "epoch = 535\n",
            "epoch = 536\n",
            "epoch = 537\n",
            "epoch = 538\n",
            "epoch = 539\n",
            "epoch = 540\n",
            "epoch = 541\n",
            "epoch = 542\n",
            "epoch = 543\n",
            "epoch = 544\n",
            "epoch = 545\n",
            "epoch = 546\n",
            "epoch = 547\n",
            "epoch = 548\n",
            "epoch = 549\n",
            "epoch = 550\n",
            "epoch = 551\n",
            "epoch = 552\n",
            "epoch = 553\n",
            "epoch = 554\n",
            "epoch = 555\n",
            "epoch = 556\n",
            "epoch = 557\n",
            "epoch = 558\n",
            "epoch = 559\n",
            "epoch = 560\n",
            "epoch = 561\n",
            "epoch = 562\n",
            "epoch = 563\n",
            "epoch = 564\n",
            "epoch = 565\n",
            "epoch = 566\n",
            "epoch = 567\n",
            "epoch = 568\n",
            "epoch = 569\n",
            "epoch = 570\n",
            "epoch = 571\n",
            "epoch = 572\n",
            "epoch = 573\n",
            "epoch = 574\n",
            "epoch = 575\n",
            "epoch = 576\n",
            "epoch = 577\n",
            "epoch = 578\n",
            "epoch = 579\n",
            "epoch = 580\n",
            "epoch = 581\n",
            "epoch = 582\n",
            "epoch = 583\n",
            "epoch = 584\n",
            "epoch = 585\n",
            "epoch = 586\n",
            "epoch = 587\n",
            "epoch = 588\n",
            "epoch = 589\n",
            "epoch = 590\n",
            "epoch = 591\n",
            "epoch = 592\n",
            "epoch = 593\n",
            "epoch = 594\n",
            "epoch = 595\n",
            "epoch = 596\n",
            "epoch = 597\n",
            "epoch = 598\n",
            "epoch = 599\n",
            "epoch = 600\n",
            "epoch = 601\n",
            "epoch = 602\n",
            "epoch = 603\n",
            "epoch = 604\n",
            "epoch = 605\n",
            "epoch = 606\n",
            "epoch = 607\n",
            "epoch = 608\n",
            "epoch = 609\n",
            "epoch = 610\n",
            "epoch = 611\n",
            "epoch = 612\n",
            "epoch = 613\n",
            "epoch = 614\n",
            "epoch = 615\n",
            "epoch = 616\n",
            "epoch = 617\n",
            "epoch = 618\n",
            "epoch = 619\n",
            "epoch = 620\n",
            "epoch = 621\n",
            "epoch = 622\n",
            "epoch = 623\n",
            "epoch = 624\n",
            "epoch = 625\n",
            "epoch = 626\n",
            "epoch = 627\n",
            "epoch = 628\n",
            "epoch = 629\n",
            "epoch = 630\n",
            "epoch = 631\n",
            "epoch = 632\n",
            "epoch = 633\n",
            "epoch = 634\n",
            "epoch = 635\n",
            "epoch = 636\n",
            "epoch = 637\n",
            "epoch = 638\n",
            "epoch = 639\n",
            "epoch = 640\n",
            "epoch = 641\n",
            "epoch = 642\n",
            "epoch = 643\n",
            "epoch = 644\n",
            "epoch = 645\n",
            "epoch = 646\n",
            "epoch = 647\n",
            "epoch = 648\n",
            "epoch = 649\n",
            "epoch = 650\n",
            "epoch = 651\n",
            "epoch = 652\n",
            "epoch = 653\n",
            "epoch = 654\n",
            "epoch = 655\n",
            "epoch = 656\n",
            "epoch = 657\n",
            "epoch = 658\n",
            "epoch = 659\n",
            "epoch = 660\n",
            "epoch = 661\n",
            "epoch = 662\n",
            "epoch = 663\n",
            "epoch = 664\n",
            "epoch = 665\n",
            "epoch = 666\n",
            "epoch = 667\n",
            "epoch = 668\n",
            "epoch = 669\n",
            "epoch = 670\n",
            "epoch = 671\n",
            "epoch = 672\n",
            "epoch = 673\n",
            "epoch = 674\n",
            "epoch = 675\n",
            "epoch = 676\n",
            "epoch = 677\n",
            "epoch = 678\n",
            "epoch = 679\n",
            "epoch = 680\n",
            "epoch = 681\n",
            "epoch = 682\n",
            "epoch = 683\n",
            "epoch = 684\n",
            "epoch = 685\n",
            "epoch = 686\n",
            "epoch = 687\n",
            "epoch = 688\n",
            "epoch = 689\n",
            "epoch = 690\n",
            "epoch = 691\n",
            "epoch = 692\n",
            "epoch = 693\n",
            "epoch = 694\n",
            "epoch = 695\n",
            "epoch = 696\n",
            "epoch = 697\n",
            "epoch = 698\n",
            "epoch = 699\n",
            "epoch = 700\n",
            "epoch = 701\n",
            "epoch = 702\n",
            "epoch = 703\n",
            "epoch = 704\n",
            "epoch = 705\n",
            "epoch = 706\n",
            "epoch = 707\n",
            "epoch = 708\n",
            "epoch = 709\n",
            "epoch = 710\n",
            "epoch = 711\n",
            "epoch = 712\n",
            "epoch = 713\n",
            "epoch = 714\n",
            "epoch = 715\n",
            "epoch = 716\n",
            "epoch = 717\n",
            "epoch = 718\n",
            "epoch = 719\n",
            "epoch = 720\n",
            "epoch = 721\n",
            "epoch = 722\n",
            "epoch = 723\n",
            "epoch = 724\n",
            "epoch = 725\n",
            "epoch = 726\n",
            "epoch = 727\n",
            "epoch = 728\n",
            "epoch = 729\n",
            "epoch = 730\n",
            "epoch = 731\n",
            "epoch = 732\n",
            "epoch = 733\n",
            "epoch = 734\n",
            "epoch = 735\n",
            "epoch = 736\n",
            "epoch = 737\n",
            "epoch = 738\n",
            "epoch = 739\n",
            "epoch = 740\n",
            "epoch = 741\n",
            "epoch = 742\n",
            "epoch = 743\n",
            "epoch = 744\n",
            "epoch = 745\n",
            "epoch = 746\n",
            "epoch = 747\n",
            "epoch = 748\n",
            "epoch = 749\n",
            "epoch = 750\n",
            "epoch = 751\n",
            "epoch = 752\n",
            "epoch = 753\n",
            "epoch = 754\n",
            "epoch = 755\n",
            "epoch = 756\n",
            "epoch = 757\n",
            "epoch = 758\n",
            "epoch = 759\n",
            "epoch = 760\n",
            "epoch = 761\n",
            "epoch = 762\n",
            "epoch = 763\n",
            "epoch = 764\n",
            "epoch = 765\n",
            "epoch = 766\n",
            "epoch = 767\n",
            "epoch = 768\n",
            "epoch = 769\n",
            "epoch = 770\n",
            "epoch = 771\n",
            "epoch = 772\n",
            "epoch = 773\n",
            "epoch = 774\n",
            "epoch = 775\n",
            "epoch = 776\n",
            "epoch = 777\n",
            "epoch = 778\n",
            "epoch = 779\n",
            "epoch = 780\n",
            "epoch = 781\n",
            "epoch = 782\n",
            "epoch = 783\n",
            "epoch = 784\n",
            "epoch = 785\n",
            "epoch = 786\n",
            "epoch = 787\n",
            "epoch = 788\n",
            "epoch = 789\n",
            "epoch = 790\n",
            "epoch = 791\n",
            "epoch = 792\n",
            "epoch = 793\n",
            "epoch = 794\n",
            "epoch = 795\n",
            "epoch = 796\n",
            "epoch = 797\n",
            "epoch = 798\n",
            "epoch = 799\n",
            "epoch = 800\n",
            "epoch = 801\n",
            "epoch = 802\n",
            "epoch = 803\n",
            "epoch = 804\n",
            "epoch = 805\n",
            "epoch = 806\n",
            "epoch = 807\n",
            "epoch = 808\n",
            "epoch = 809\n",
            "epoch = 810\n",
            "epoch = 811\n",
            "epoch = 812\n",
            "epoch = 813\n",
            "epoch = 814\n",
            "epoch = 815\n",
            "epoch = 816\n",
            "epoch = 817\n",
            "epoch = 818\n",
            "epoch = 819\n",
            "epoch = 820\n",
            "epoch = 821\n",
            "epoch = 822\n",
            "epoch = 823\n",
            "epoch = 824\n",
            "epoch = 825\n",
            "epoch = 826\n",
            "epoch = 827\n",
            "epoch = 828\n",
            "epoch = 829\n",
            "epoch = 830\n",
            "epoch = 831\n",
            "epoch = 832\n",
            "epoch = 833\n",
            "epoch = 834\n",
            "epoch = 835\n",
            "epoch = 836\n",
            "epoch = 837\n",
            "epoch = 838\n",
            "epoch = 839\n",
            "epoch = 840\n",
            "epoch = 841\n",
            "epoch = 842\n",
            "epoch = 843\n",
            "epoch = 844\n",
            "epoch = 845\n",
            "epoch = 846\n",
            "epoch = 847\n",
            "epoch = 848\n",
            "epoch = 849\n",
            "epoch = 850\n",
            "epoch = 851\n",
            "epoch = 852\n",
            "epoch = 853\n",
            "epoch = 854\n",
            "epoch = 855\n",
            "epoch = 856\n",
            "epoch = 857\n",
            "epoch = 858\n",
            "epoch = 859\n",
            "epoch = 860\n",
            "epoch = 861\n",
            "epoch = 862\n",
            "epoch = 863\n",
            "epoch = 864\n",
            "epoch = 865\n",
            "epoch = 866\n",
            "epoch = 867\n",
            "epoch = 868\n",
            "epoch = 869\n",
            "epoch = 870\n",
            "epoch = 871\n",
            "epoch = 872\n",
            "epoch = 873\n",
            "epoch = 874\n",
            "epoch = 875\n",
            "epoch = 876\n",
            "epoch = 877\n",
            "epoch = 878\n",
            "epoch = 879\n",
            "epoch = 880\n",
            "epoch = 881\n",
            "epoch = 882\n",
            "epoch = 883\n",
            "epoch = 884\n",
            "epoch = 885\n",
            "epoch = 886\n",
            "epoch = 887\n",
            "epoch = 888\n",
            "epoch = 889\n",
            "epoch = 890\n",
            "epoch = 891\n",
            "epoch = 892\n",
            "epoch = 893\n",
            "epoch = 894\n",
            "epoch = 895\n",
            "epoch = 896\n",
            "epoch = 897\n",
            "epoch = 898\n",
            "epoch = 899\n",
            "epoch = 900\n",
            "epoch = 901\n",
            "epoch = 902\n",
            "epoch = 903\n",
            "epoch = 904\n",
            "epoch = 905\n",
            "epoch = 906\n",
            "epoch = 907\n",
            "epoch = 908\n",
            "epoch = 909\n",
            "epoch = 910\n",
            "epoch = 911\n",
            "epoch = 912\n",
            "epoch = 913\n",
            "epoch = 914\n",
            "epoch = 915\n",
            "epoch = 916\n",
            "epoch = 917\n",
            "epoch = 918\n",
            "epoch = 919\n",
            "epoch = 920\n",
            "epoch = 921\n",
            "epoch = 922\n",
            "epoch = 923\n",
            "epoch = 924\n",
            "epoch = 925\n",
            "epoch = 926\n",
            "epoch = 927\n",
            "epoch = 928\n",
            "epoch = 929\n",
            "epoch = 930\n",
            "epoch = 931\n",
            "epoch = 932\n",
            "epoch = 933\n",
            "epoch = 934\n",
            "epoch = 935\n",
            "epoch = 936\n",
            "epoch = 937\n",
            "epoch = 938\n",
            "epoch = 939\n",
            "epoch = 940\n",
            "epoch = 941\n",
            "epoch = 942\n",
            "epoch = 943\n",
            "epoch = 944\n",
            "epoch = 945\n",
            "epoch = 946\n",
            "epoch = 947\n",
            "epoch = 948\n",
            "epoch = 949\n",
            "epoch = 950\n",
            "epoch = 951\n",
            "epoch = 952\n",
            "epoch = 953\n",
            "epoch = 954\n",
            "epoch = 955\n",
            "epoch = 956\n",
            "epoch = 957\n",
            "epoch = 958\n",
            "epoch = 959\n",
            "epoch = 960\n",
            "epoch = 961\n",
            "epoch = 962\n",
            "epoch = 963\n",
            "epoch = 964\n",
            "epoch = 965\n",
            "epoch = 966\n",
            "epoch = 967\n",
            "epoch = 968\n",
            "epoch = 969\n",
            "epoch = 970\n",
            "epoch = 971\n",
            "epoch = 972\n",
            "epoch = 973\n",
            "epoch = 974\n",
            "epoch = 975\n",
            "epoch = 976\n",
            "epoch = 977\n",
            "epoch = 978\n",
            "epoch = 979\n",
            "epoch = 980\n",
            "epoch = 981\n",
            "epoch = 982\n",
            "epoch = 983\n",
            "epoch = 984\n",
            "epoch = 985\n",
            "epoch = 986\n",
            "epoch = 987\n",
            "epoch = 988\n",
            "epoch = 989\n",
            "epoch = 990\n",
            "epoch = 991\n",
            "epoch = 992\n",
            "epoch = 993\n",
            "epoch = 994\n",
            "epoch = 995\n",
            "epoch = 996\n",
            "epoch = 997\n",
            "epoch = 998\n",
            "epoch = 999\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Let's visualize the average MSE across each training epoch:\n"
      ],
      "metadata": {
        "id": "tl2it8FOh96M"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "plt.plot(avg_cost_list)\n",
        "plt.ylabel(\"Avg. cost\")\n",
        "plt.xlabel(\"epoch\")\n",
        "plt.title(\"Avg. MSE over training epochs - mini-batch gradient descent\")\n",
        "plt.show()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 295
        },
        "id": "CPtzWeTOe9Du",
        "outputId": "80ef0aab-dcd0-48bf-83e0-955db23a45d4"
      },
      "execution_count": 254,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ],
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZEAAAEWCAYAAACnlKo3AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3deZQeVZ3/8fenkxBWWUyDEEIiiwsgi2ZYBJ0oqMAPgVFUGAaQgcOM4k9wHBXQUXHX4w8UFxYFAWVTFAYYNkE2HVkSBIQAEtaENURI2ESSfH9/3PuQStPL009udae7Pq9zntO13K66tX7r3luLIgIzM7NOdA13BszMbORyEDEzs445iJiZWcccRMzMrGMOImZm1jEHETMz65iDiA1I0nOSNiyddnkn6RpJhwx3PnqSdKekacuaVtIUSSFpbMn89TGv5WpdSnpQ0s65+2hJPx2mfEyTNGc45l3KsAeRvHM9LWn8EM0vJD1ZPXAkjcvDojJsM0lXSPqrpGckzZC0Wx43TdLifMKs/rYfimVoV6kdNCJWjYj7S6e1zkTEZhFxTem0g5WPpY3rmPZQiohvRMQyB7ihDMpDQdJpkr42ULphDSKSpgDvAALYYwhn/TSwa6V/1zys6iLgt8DrgLWBTwILKuMfzSfM6u+PdWa6P53uuKNlh7fmkjRmuPPQaBExbD/gi8AfgGOBi/Ow8cAzwOaVdN3Ai8Dauf+zwGPAo8AhpCC0cZvzDOALwK8qw84DPp9WRwBMyOnW6GMa04A5g1jONwPX5OW6E9gjD98WeBwYU0n7T8DtubsLOBK4D5gH/BJYK4+bkvN4MPAwcF2Pea6S19li4Ln8Ww/4cl7eX5CC4iHANsAfc/4eA34IrNBjnW2cu08DfgT8D/AscCOwUYdp3wvcA8wHfgxcCxzSxzpsZ10cmveJx4D/rPzveOB7edyjuXt8ZfyewK15fdwH7JKHXwN8lbSPPgtcAUzI41bM63BeXm83A+t0eByclpf/0ryd/kC6ePke6eLmbmDrSvoHgZ1z95fzujgj5/FOYGpvaXuZ70Drrc/9Argu/+/zOc8f6XRd9pG3Po/xvL5OAC7J898Z+D/An/J8ZwNf7jG9/YGH8vb6fC/r8BeVtNsB/5uX+zZgWmVcf/vEwzmfreNt+16Wa6Wc/6eBmcBnqJxLSMfor4G5wAPAJ3tsj+l5GZ8Ajq2M27GS59nARyv7/ndz3p4ATgRWqp7HgE8DT+b1fVAedyjwMvD3vCwX9bmtOtnpS/2AWcDHgbflDK+Th58KfL2S7jDgsty9C+nEuxmwMulAHmwQ2Tyv0DWANXP35iwJIgLuBS4G9qLHyYFBBBFgXF7Oo4EVgHfnne+Nefx9wHsq6X8FHJm7DwduANbPO8NJwNk9TgBnkALGSr3M+1X5JB0wL+fl6so79dtIB87YPN27gCN6rLPqATwv79BjgTOBcwablhSoFwAfyOMOz/nqK4i0sy7OzuviLaSDsHWS+Er+37VJFyT/C3y1cmDOB96T18dE4E2VE8Z9wBvyeroG+FYe92+k0urKwJi8Dl/T4XFwGvBUnsaKwO9IJ5AD8rS/BlxdSf8gS58A/wbsltN+E7iht7S9zHeg9db2frEs67KXfPV7jOf1NR/YIc9nRdK+/pbcvwXpmN4rp9+UdCJ8J2nfORZYSC9BJOd5Xl6fXXlZ5gHdbewTrfU5tp9t/S3gemAtYBJwB/kYzfObQbq4XgHYELgfeF8e/0dg/9y9KrBd7p5MOqfsSzrfvBbYKo87Drgwz2810j77zcr5YSHp+BiXl/kFYM3Kev7agPtvJzt9iR8pcr7Mkih+N/Cp3L0zcF8l7R+AA3L3qa2VkPs3ZvBBZGPgp6QTwb8DP2lNp5JufdKV132kq/nrgE0qK38xKepXf6v0Mr93kA6Irsqws8lXSqQTxKm5ezXSldXk3H8XsFPl/9bN66x1UAewYT/LOo3eg8h1A6yjI4DzeztZ5B3rp5VxuwF3DzYt6QT5x8o4ka6g+goi7ayLN1XGfwc4JXffB+xWGfc+4MHcfRJwXB/zvAb4QqX/4yy5mPlXUjDaosCxcBrwk0r//wXuqvS/BXim0v8gS58Ar6yM2xR4sbe0vcy33/U2mP1iWdZlL2n7Pcbz+jpjgHX6vVZeSCfl6oXOKqQr7N6CyOeAn/eY1uXAgW3sE6312V8QuZ9cOsv9h7IkiGwLPNwj/VHAz3L3dcAx9CjB5TTn9zIvkc4n1dL/9sADuXsaqbZibGX8kywJTqfRRhAZzjaRA4ErIuKp3H9WHgZwNbCypG1zu8lWwPl53Hqkk01LtXswziCdyA7I3UuJiDkR8YmI2IgU6Z/vke7RiFijx+/5XuazHjA7IhZXhj1EuuKBtNwfyDcWfAC4JSIeyuMmA+fnhv1nSCfSRcA6lWl1svxL/Y+kN0i6WNLjkhYA3yCVFPryeKX7BdJV0WDTLrUdI+21/d0EMNh18VCeR2teD/UxbhIpyAw2/z8nnVzOkfSopO9IGtfznyXtV7nx4tJ+5vNEpfvFXvoHs45X7K2tq8dNIBtURvW63jrYLzpdlz21c4z33Ie3lXS1pLmS5pMuDlt57bmvPU8qXfRmMvCh1n6W97UdSRctg12O3vRctup+ORlYr8e8j2bJPn4wqQR0t6SbJe2eh/e13rtJJbkZleldloe3zIuIhcuwPMMTRCStBHwY+Me8gz4OfArYUtKWEbGIVM+7b/5dHBHP5n9/jFRKaJnUYTauJ+0Y6wC/7y9hRMwm1e1v3sF8HgUmSaqu6w2AR/K0Z5J2pF2BfyYFlZbZwK49AtWKEfFINXv9Zb3N4SeQSoKbRMRrSDuuBliuZbXUdpQklt6uPbWzLqr7wgakdU/+O7mPcbOBjQab+Yh4OSKOiYhNgbcDu5MuSHqmOzOW3Hix66smNIRi6ZtAHq6M6mu9DXa/6Ghd9qKdY7znPnwWqdpmUkSsTqr7b+X1seo0JK1MqvLpzWxSSaS6n60SEd9qI9/9HYstS+WFtL6r836gx7xXi4jdACLi3ojYl1Qt+23gPEmr0Pd6f4p0AbJZZXqrR0S7QaKd5Rm2kshepKvITUmljK1Ijc/Xs+RAPAv4CLAfS59YfwkcJOnNeWf4r04ykK98309q5F5qZUlaU9IxkjaW1CVpAqn64oYOZnUjKbp/Nt9KPC3P95xKmrNIdf7vJLWJtJwIfF3S5Jyvbkl7DmLeTwCvlbT6AOlWI7VPPCfpTcDHBjGPTv0P8BZJe+Wr5sNIjcl9aWdd/JeklSVtBhwEnJuHnw18If/PBFL1xi/yuFNI+9NOeVtPzOugX5LeJekt+c6gBaSqtcUD/Nvyqq/1NtB+8QSp3r6lo3XZi06O8dWAv0bE3yRtQ7ogazkP2F3SjpJWILUB9HXu+wXwfknvkzRG0or5Vvn+LnBa5pL2gf6ek/olcFQ+x6xPqrpsuQl4VtLnJK2U57+5pH8AkPQvkrpzrcYz+X8Wk9oad5b0YUljJb1W0lY53U+A4yStnacxUdL72lgWePX27dVwBZEDSfV8D0fE460fqQ1iP0ljI+JGUhXSeqS7VgCIiEuB40lVXrNYcmJ/CV55cKi/aoNXRMSdEXFnL6P+TqrfvJJ0EN2Rp//RSpr19OrnRD7Yyzz+Tgoau5KuDH5Mat+5u5LsbOAfgd9VqvcAvk+6urpC0rN5WbdtZ9nyvO/O074/F2fX6yPpf5IOumdJO925faQrJi/nh0h18PNIFxTTyduxF+2si2tJ+8RVwHcj4oo8/Gt52rcDfwZuycOIiJtIJ87jSI2117J0qaUvryOdnBaQqtauJVVxjUR9rbeB9osvA6fnfevDy7AulzLQMd6HjwNfyfvGF0kn69b07iRdpJxFKgk8TR9Vp7nWYU9SqWsu6Sr/M7RxroyIF4CvA3/I62S7XpIdQ6p5eIB0Z9fPK/+/iFSi3SqPf4rUdtu6CNwFuFPSc6TjYZ+IeDGXKncj3WX1V9LdcVvm//kceR3mKskrgTcOtCzZKcCmeVku6CuRelyEjziS3kw6yY/vUbdnI0iu7psD7BcRVw/yf6eQDrpx3gdGHx/jy7dhf2K9E5L+SdJ4SWuS6gYv8s418uQqgzWUbipo1bd3UmVoo4yP8ZFjRAYR0q25T5LuSFjE0NThW3nbk7bhU6Qqv70i4sXhzZItJ3yMjxAjvjrLzMyGz0gtiZiZ2XJgxL18b8KECTFlypThzoaZ2YgyY8aMpyKie+CUgzPigsiUKVOYPn36cGfDzGxEkfTQwKkGr7bqrPyQzk2SblP6MM4xvaQZL+lcSbMk3Zhv1TQzsxGizjaRl4B3R8SWpIdndunl4ZuDgacjYmPSA0rfrjE/ZmZWWG1BJJLncu+4/Ot5K9iewOm5+zxgp/wOJTMzGwFqvTsrv/vlVtL93r/NrzKpmkh+o2V+kGg+vbwYTdKhkqZLmj537tw6s2xmZoNQaxCJiEURsRXpjZzbSOrkLbhExMkRMTUipnZ3F7+5wMzMOjQkz4lExDOkl6nt0mPUI+TXIuc3ua5O3+/5NzOz5Uydd2d1S1ojd69E+szk3T2SXciSD1HtTXqLrR+hNzMbIeosiawLXC3pduBmUpvIxZK+ImmPnOYU0vcuZgH/ARxZV2b+8sSzHHvFPTz1XH9vkzYzs8Go7WHDiLgd2LqX4V+sdP+N9E2J2t37xHMc/7tZ7L7lekxYdfxQzNLMbNRr3LuzXFlmZlZOY4KInz4xMyuvMUGkJdr79ryZmbWhMUHEBREzs/IaE0TMzKy8xgURN6ybmZXTmCDihnUzs/IaE0RaXBIxMyunQUHERREzs9IaFEQS3+JrZlZOY4KI20TMzMprTBBpcZuImVk5jQkiLoiYmZXXmCBiZmblNSaIyI0iZmbFNSaImJlZeY0LIm5YNzMrpzFBxJVZZmblNSaItPhhQzOzchoTRNyubmZWXmOCSIvbRMzMymlMEHFJxMysvMYEkRYXRMzMymlMEJHvzzIzK64xQaQl3ChiZlZMc4KICyJmZsU1J4iYmVlxjQsirswyMyunMUHEtVlmZuU1Joi0uF3dzKycxgQRf0/EzKy8xgSRJVwUMTMrpbYgImmSpKslzZR0p6TDe0kzTdJ8Sbfm3xdry09dEzYza7CxNU57IfDpiLhF0mrADEm/jYiZPdJdHxG715iPpbhNxMysnNpKIhHxWETckrufBe4CJtY1v4G4ScTMrLwhaRORNAXYGrixl9HbS7pN0qWSNuvj/w+VNF3S9Llz5y5TXlwQMTMrp/YgImlV4NfAERGxoMfoW4DJEbEl8APggt6mEREnR8TUiJja3d3dWT7cKmJmVlytQUTSOFIAOTMiftNzfEQsiIjncvclwDhJE+rMk5mZlVPn3VkCTgHuiohj+0jzupwOSdvk/MyrK0/ghnUzs5LqvDtrB2B/4M+Sbs3DjgY2AIiIE4G9gY9JWgi8COwTNb2r3Q3rZmbl1RZEIuL3DPB4RkT8EPhhXXnoY55DOTszs1GtMU+suyBiZlZeY4JIi8shZmblNCeIuChiZlZcc4JI5iYRM7NyGhNE/LChmVl5jQkiLeFWETOzYhoTRPyciJlZeY0JImZmVl7zgohrs8zMimlMEHFtlplZeY0JIi0uiJiZldOYICK3rJuZFdeYINLihw3NzMppTBBxQcTMrLzGBJEWP2xoZlZOY4KICyJmZuU1Joi0uE3EzKycxgQRt4mYmZXXmCDS4oKImVk5DQoiLoqYmZXWoCBiZmalNS6IhFvWzcyKaUwQccO6mVl5jQkiLS6HmJmV05gg4oKImVl5jQkir3BRxMysmMYEEb8K3sysvMYEkRa/gNHMrJzGBBGXQ8zMymtMEGnxYyJmZuU0Joi4ScTMrLzagoikSZKuljRT0p2SDu8ljSQdL2mWpNslvbWu/JiZWXlja5z2QuDTEXGLpNWAGZJ+GxEzK2l2BTbJv22BE/Lf2rg6y8ysnNpKIhHxWETckrufBe4CJvZItidwRiQ3AGtIWreO/MhN62ZmxQ1Jm4ikKcDWwI09Rk0EZlf65/DqQFOUCyJmZuXUHkQkrQr8GjgiIhZ0OI1DJU2XNH3u3Lkd5qOjfzMzs34MGEQk7dDOsD7+dxwpgJwZEb/pJckjwKRK//p52FIi4uSImBoRU7u7u9uZdZ/8Kngzs3LaKYn8oM1hS1F6z8gpwF0RcWwfyS4EDsh3aW0HzI+Ix9rIk5mZLQf6vDtL0vbA24FuSf9RGfUaYEwb094B2B/4s6Rb87CjgQ0AIuJE4BJgN2AW8AJw0GAXYLBcDjEzK6e/W3xXAFbNaVarDF8A7D3QhCPi9wzwtpFIdUuHDZzNZec2ETOz8voMIhFxLXCtpNMi4iEASV3Aqp02kC8P3CRiZlZOO20i35T0GkmrAHcAMyV9puZ8FefnRMzMymsniGyaSx57AZcCrye1dZiZWcO1E0TG5Vt19wIujIiXGdHt0yM462Zmy5l2gshJwIPAKsB1kiaTGtdHFDesm5mVN+ALGCPieOD4yqCHJL2rvizVyw3rZmbltPPE+uqSjm29dkTS/yOVSkYUl0TMzMprpzrrVOBZ4MP5twD4WZ2ZqpMLImZm5bTzPZGNIuKDlf5jKk+gjxi+xdfMrLx2SiIvStqx1ZNfvvhifVmql9tEzMzKaack8jHgdEmr5/6ngY/WlqOauE3EzKy8du7OuhXYUtJrcv+Iu723KtwqYmZWTDt3Z31D0hoRsSAiFkhaU9LXhiJzJbkgYmZWXjttIrtGxDOtnoh4mvT6djMza7h2gsgYSeNbPZJWAsb3k3655oZ1M7Ny2mlYPxO4SlLr2ZCDgNPry1I93LBuZlZeOw3r35Z0G7BzHvTViLi83mzVxwURM7Ny2imJEBGXAZfVnJeauShiZlZaO20io0q4UcTMrJjGBBG3iZiZldeYIGJmZuV1FEQkfblwPmrngoiZWXmdlkRmFM3FEHKTiJlZOR0FkYi4qHRG6iY3ipiZFTfgLb6Sju9l8HxgekT8d/ksmZnZSNFOSWRFYCvg3vzbAlgfOFjS92rMWy38Fl8zs3LaedhwC2CHiFgEIOkE4HpgR+DPNeatKFdmmZmV105JZE1g1Ur/KsBaOai8VEuuauSGdTOzctopiXwHuFXSNaQL+ncC35C0CnBljXkryu3qZmbltfMCxlMkXQJskwcdHRGP5u7P1JazmrgkYmZWTjt3Z10EnAVcGBHP15+lesitImZmxbXTJvJd4B3ATEnnSdpb0oo156s2LoiYmZUzYBCJiGsj4uPAhsBJwIeBJwf6P0mnSnpS0h19jJ8mab6kW/Pvi4PN/GC4TcTMrLy2vieSP4n7fuAjwFtp78uGpwE/BM7oJ831EbF7O3koxa+CNzMrp502kV+SGtUvIwWFayNi8UD/FxHXSZqyrBk0M7PlVzttIqcAG0XEv0fE1cDbJf2o0Py3l3SbpEslbdZXIkmHSpouafrcuXMLzdrMzJZVO20ilwNbSPqOpAeBrwJ3F5j3LcDkiNgS+AFwQT95ODkipkbE1O7u7mWaqSuzzMzK6bM6S9IbgH3z7yngXEAR8a4SM46IBZXuSyT9WNKEiHiqxPR7csO6mVl5/bWJ3E16R9buETELQNKnSs1Y0uuAJyIiJG1DKhXNKzX9PrkoYmZWTH9B5APAPsDVki4DzmEQ7zGUdDYwDZggaQ7wJWAcQEScCOwNfEzSQuBFYJ+o8dYpf0/EzKy8PoNIRFwAXJDfkbUncASwdn6L7/kRcUV/E46IfQcY/0PS3V5Dyq+CNzMrp52G9ecj4qyIeD/pOyJ/Aj5Xe84KcznEzKy8QX0eNyKezndK7VRXhurmZw3NzMrp6BvrI5GbRMzMymtMEGlxQcTMrJzGBBG/Ct7MrLzGBBEzMyuvcUHEDetmZuU0Joi4Yd3MrLzGBJEWP2xoZlZOY4KICyJmZuU1Joi0uE3EzKyc5gQRF0XMzIprThDJXBAxMyunMUHEDxuamZXXmCDyCjeKmJkV05gg4udEzMzKa0wQMTOz8hoXRFyZZWZWTmOCiGuzzMzKa0wQaXG7uplZOY0JInLLuplZcY0JIi3hooiZWTGNCSIuh5iZldeYINLicoiZWTmNCSJuEjEzK68xQaTFTSJmZuU0Joj4BYxmZuU1JoiYmVl5jQsirs0yMyunOUHEtVlmZsU1J4hkftjQzKyc2oKIpFMlPSnpjj7GS9LxkmZJul3SW+vKS5pfnVM3M2umOksipwG79DN+V2CT/DsUOKHGvJiZWQ1qCyIRcR3w136S7AmcEckNwBqS1q0rPy6ImJmVN5xtIhOB2ZX+OXlYrdwkYmZWzohoWJd0qKTpkqbPnTu302kUzpWZmQ1nEHkEmFTpXz8Pe5WIODkipkbE1O7u7mWaafhJETOzYoYziFwIHJDv0toOmB8Rj9U1M5dDzMzKG1vXhCWdDUwDJkiaA3wJGAcQEScClwC7AbOAF4CD6spLldtEzMzKqS2IRMS+A4wP4LC65m9mZvUbEQ3rJbhd3cysvMYEkRbXZpmZldOYIOLviZiZldeYINLihnUzs3IaE0TcJmJmVl5jgkiLHzY0MyuncUHEzMzKaVwQcZuImVk5jQkiY7pSo4i/bGhmVk5zgkhuWV+42EHEzKyUxgSRrlwSWewgYmZWTGOCCMDYLrHI1VlmZsU0Koh0dcnVWWZmBTUqiIyRXJ1lZlZQo4LI2C6xaPFw58LMbPRoVBDp6hKLFjuKmJmV0qggMsYN62ZmRTUqiHTJ1VlmZiU1KoiMdXWWmVlRjQoiY9ywbmZWVKOCSFcXLHabiJlZMY0KImO7uvywoZlZQY0KIl3yu7PMzEpqVBBJbSIOImZmpTQsiLg6y8yspIYFETesm5mV1KwgIldnmZmV1Kwg4jYRM7OiHETMzKxjjQoiXfILGM3MSmpUEBk7xiURM7OSGhVEVhw7hhf+vmi4s2FmNmrUGkQk7SLpHkmzJB3Zy/iPSpor6db8O6TO/HSvNp6nnnupzlmYmTXK2LomLGkM8CPgPcAc4GZJF0bEzB5Jz42IT9SVj6q1VxvPvOdeYtHiYEyXhmKWZmajWm1BBNgGmBUR9wNIOgfYE+gZRIbMBq9dhcUB7/j271h5fJ2LDg5RZlbaR/5hEoe8Y8PhzsZS6jyTTgRmV/rnANv2ku6Dkt4J/AX4VETM7plA0qHAoQAbbLBBxxnac6v1eGje89w/9/mOp9GOwI33ZlbehFXHD3cWXqXey/GBXQScHREvSfo34HTg3T0TRcTJwMkAU6dO7fgMPW5MF59+7xs7/XczM+uhzob1R4BJlf7187BXRMS8iGi1dP8UeFuN+TEzs8LqDCI3A5tIer2kFYB9gAurCSStW+ndA7irxvyYmVlhtVVnRcRCSZ8ALgfGAKdGxJ2SvgJMj4gLgU9K2gNYCPwV+Ghd+TEzs/IUI+w1IFOnTo3p06cPdzbMzEYUSTMiYmrp6TbqiXUzMyvLQcTMzDrmIGJmZh1zEDEzs46NuIZ1SXOBhzr89wnAUwWzMxJ4mZvBy9wMy7LMkyOiu2RmYAQGkWUhaXoddycsz7zMzeBlboblcZldnWVmZh1zEDEzs441LYicPNwZGAZe5mbwMjfDcrfMjWoTMTOzsppWEjEzs4IcRMzMrGONCSKSdpF0j6RZko4c7vyUImmSpKslzZR0p6TD8/C1JP1W0r3575p5uCQdn9fD7ZLeOrxL0BlJYyT9SdLFuf/1km7My3Vu/vwAksbn/ll5/JThzPeykLSGpPMk3S3pLknbj+btLOlTeZ++Q9LZklYcjdtZ0qmSnpR0R2XYoLerpANz+nslHThU+W9EEJE0BvgRsCuwKbCvpE2HN1fFLAQ+HRGbAtsBh+VlOxK4KiI2Aa7K/ZDWwSb5dyhwwtBnuYjDWfr7M98GjouIjYGngYPz8IOBp/Pw43K6ker7wGUR8SZgS9Lyj8rtLGki8ElgakRsTvqcxD6Mzu18GrBLj2GD2q6S1gK+RPoE+TbAl1qBp3YRMep/wPbA5ZX+o4CjhjtfNS3rfwPvAe4B1s3D1gXuyd0nAftW0r+SbqT8SF/JvIr0KeWLAZGe4h3bc3uTvmezfe4em9NpuJehg2VeHXigZ95H63YGJgKzgbXydrsYeN9o3c7AFOCOTrcrsC9wUmX4Uunq/DWiJMKSHbJlTh42quQi/NbAjcA6EfFYHvU4sE7uHg3r4nvAZ4HFuf+1wDMRsTD3V5fpleXN4+fn9CPN64G5wM9yNd5PJa3CKN3OEfEI8F3gYeAx0nabwejfzi2D3a7Dtr2bEkRGPUmrAr8GjoiIBdVxkS5NRsW93JJ2B56MiBnDnZchNhZ4K3BCRGwNPM+SKg5g1G3nNYE9ScFzPWAVXl3l0wjL+3ZtShB5BJhU6V8/DxsVJI0jBZAzI+I3efATrW/Y579P5uEjfV3sAOwh6UHgHFKV1veBNSS1PvdcXaZXljePXx2YN5QZLmQOMCcibsz955GCymjdzjsDD0TE3Ih4GfgNaduP9u3cMtjtOmzbuylB5GZgk3xnxwqkBroLhzlPRUgScApwV0QcWxl1IdC6Q+NAUltJa/gB+S6P7YD5lWLzci8ijoqI9SNiCmk7/i4i9gOuBvbOyXoub2s97J3TL7dXdX2JiMeB2ZLemAftBMxklG5nUjXWdpJWzvt4a3lH9XauGOx2vRx4r6Q1cynuvXlY/Ya7QWkIG652A/4C3Ad8frjzU3C5diQVdW8Hbs2/3Uj1wVcB9wJXAmvl9CLdqXYf8GfS3S/DvhwdLvs04OLcvSFwEzAL+BUwPg9fMffPyuM3HO58L8PybgVMz9v6AmDN0bydgWOAu4E7gJ8D40fjdgbOJrX7vEwqcR7cyXYF/jUv/yzgoKHKv197YmZmHWtKdZaZmdXAQcTMzDrmIGJmZh1zEDEzs445iJiZWcccRMyGkKRprTcPm40GDiJmZtYxBxGzXkj6F0k3SbpV0kn5+yXPSTouf+PiKkndOe1Wkm7I33c4v/Lth40lXSnpNkm3SNooT37VyndBzsxPZJuNSA4iZj1IejPwEWCHiNgKWATsR3oJ4PSI2Ay4lvT9BvaTCQwAAAE4SURBVIAzgM9FxBakp4hbw88EfhQRWwJvJz2VDOlNy0eQvm2zIemdUGYj0tiBk5g1zk7A24CbcyFhJdIL8BYD5+Y0vwB+I2l1YI2IuDYPPx34laTVgIkRcT5ARPwNIE/vpoiYk/tvJX1L4vf1L5ZZeQ4iZq8m4PSIOGqpgdJ/9UjX6TuDXqp0L8LHoY1grs4ye7WrgL0lrQ2vfO96Mul4ab1B9p+B30fEfOBpSe/Iw/cHro2IZ4E5kvbK0xgvaeUhXQqzIeArILMeImKmpC8AV0jqIr1d9TDSh6C2yeOeJLWbQHpV94k5SNwPHJSH7w+cJOkreRofGsLFMBsSfouvWZskPRcRqw53PsyWJ67OMjOzjrkkYmZmHXNJxMzMOuYgYmZmHXMQMTOzjjmImJlZxxxEzMysY/8f/zonIrxwRxwAAAAASUVORK5CYII=\n"
          },
          "metadata": {
            "needs_background": "light"
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "And how the weights changed across each epoch:"
      ],
      "metadata": {
        "id": "DcYEjNzHiIpe"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "plt.plot(weights_updates[:,0], label = \"w1\")\n",
        "plt.plot(weights_updates[:,1], label = \"w2\")\n",
        "plt.plot(weights_updates[:,2], label = \"w3\")\n",
        "plt.ylabel('weights')\n",
        "plt.xlabel(\"epoch\")\n",
        "plt.legend()\n",
        "plt.title(\"Weights over training epochs - mini-batch gradient descent\")\n",
        "plt.show()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 295
        },
        "id": "8fCbGFRViEui",
        "outputId": "bc128380-8cd2-4ec3-a9f9-1f4ee6a9a425"
      },
      "execution_count": 255,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ],
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZYAAAEWCAYAAABFSLFOAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3deZxddX3/8dd7sg2QsAQCxCQQligQZQkRQQ21IrJYlvpDJaWyiFJaLUJdCkKF9iFV0VahoEABAaWsQg1bUQhKEAUGZV8kgJgJQUKAhC2Qmfn8/jjfG04us9zcc++cuZP38/G4ydnmnM9Z7vnc7/d7FkUEZmZmjdJWdgBmZja8OLGYmVlDObGYmVlDObGYmVlDObGYmVlDObGYmVlDrVGJRdIhkn5e47SHS7q92TG1IkkPSfpQo6cd6iRdKOkbZcdRTdKNkg5rxLSSQtLWjYuuz+UMqW0p6ZeSPpu6az5PNCGOqWkfjCxj+Y0y5BOLpBMk3Vg17PE+hh3c37wi4pKI+GiD4lp5ILaKRh20ETE9In7Z6GmtPhGxT0Rc1OhpV5ekP0r6SDPmPZgafJ4YlEQ9GCSdIukntUw75BMLcBvwfkkjACRNBEYBO1UN2zpNa0C9yaPVfymZ+RguXysklrvJEsmOqX8WcCvwWNWwJyLiGUnrSTpf0iJJCyV9I5eAVqnekvRRSY9JWirpB5J+VV0KkfRdSS9KekrSPmnYqWmZZ0p6RdKZynxP0nOSlkl6QNK7e1shSe+QNEfSC5LmS/pcbvjrksbnpt1J0vOSRqX+z0h6JMV0k6TNc9OGpM9Lehx4vJdFVxLvSynu3dI2+XWKfQlwiqStJM2VtCQt+xJJ6+eWs/KXafoVc4WkiyW9nKq+ZtY57QxJv0/jrpR0eX/VJTVsi2MkPZnW4TuS2tK4NkknSXo67a+LJa2X+9sPSrpD0kuSFkg6PLfYDSRdn2K8U9JW6W9q3v8DSdvpSkk/Sct5QNI7lZXen0sxfTQ3fb4a53BJt/d23FZP2499+9hufR4Xkn4MbAZcm46tr9a7LfvYJoem/bVE0r/0clxdlbbXMuBwSbtI+k1a7iJl39HRufntKelRZd/9MwHlxlWfJ7aR9Atl39fHJH0yN+5CSWf1cUxUvm/3pW3yqV7Wa0TaV89LehL4WNX4/s5nWys7Zy1Nf3957u+m52L+s6SvpeFtko6X9ETallconW/0Vo3GYZL+lOZ5Yhq3N/A14FNpXe7ra18BEBFD/kOWSI5L3WcCnwFOrRp2Qeq+BjgHWAfYGLgL+Ls07nDg9tS9EbAM+DgwEvgisAL4bG7aFcDngBHA3wPPAErjf1mZNvXvBdwDrE92kG4LTOxjfW4DfgC0kyXHxcCH07i5wOdy034HODt1HwDMT/MeCZwE3JGbNoBfAOOBtXpZ7tQ0zcjcsMOBLuAf0zzXIiv97QmMASakeL+f+5s/Ah9J3acAy4F903b6JvDb1Z0WGA08nfbDqLRf3gS+0cc2rGVb3Jq2xWbAH3L79jPpb7cExgJXAz9O4zYHXgZmpzg2BHZM4y4ElgC7pGVeAly2uvu/huO9sp32Ssu5GHgKODHF9Dngqdz0v6TO47aXZfe33Wo+Lopsy15i2g54BfhgOk6+m9Yxf1ytAA4k+7G8FrAzsGua91TgEeDY3Hf/ZeCgFNdxZN+B/DasnCfWARYAR6R57QQ8D2xXy3qk7bl1P9v7aOBRYEra5reS+47S//ns0nRMtJGdSz6Yho8DFgFfSsPHAe9L474I/BaYnPbjOcClVeeH/07bcAfgDWDb3Hb+SU3HcDMSQaM/aYWuSd33AdOAvauGHQZskjbEWrm/nQ3c2ssBcyjwm9x0SgdQ/uCanxu/dtrom/b2BQU+TPYl3BVo62ddpgDdwLjcsG8CF6buzwJzq2LaPfXfCByZ+7s24DVg89xB/OF+ll05cKoTy58G2P4HAr/v7QSS9s3NVSeB11d3WmB3YCHpBJiG3U7fiaWWbbF3bvw/ALek7luAf8iNexfZiWkkcELluOplmRcC5+X69wUeXZ39vxrH+y9y/fuRnVhHpP5xaf3Wrz4WWc3jtpdl97ndVue4SP11bctepv066eSXW6c3q46r2wbYpsfy1vniUFb98SOgk94Ty6eAeVXzOgc4uZb1YODEMhc4Otf/0fQ3Ixn4fHYxcC4wuWqes/P7pWrcI8Aeuf6JvHXsT03LnpwbfxdwcG4715RYWqEqDLJfRh9MRbYJEfE4cAdZ28t44N1pms3JfoEsSkXgl8gOgo17mec7yE7aAES25Tqrpnk2N/611Dm2twAjYi5Zyeks4DlJ50pat4/lvhARL+eGPQ1MSt0/BXZT1m60O9ADzEvjNgdOz63bC2Rfikm5eS1g9a3yN5I2kXRZKnovA35C9iuvL8/mul8D2tV3PXdf074DWJj2Q69xVVndbfF0Wgbp/6erxlW+yFOAJ/pZbnX8Y6H2/S9pVqpKeEXSQ/0s58+57teB5yOiO9cPfRyL1HjcKquKrMQyKzeq1+1Wx3FR17bsRfV39TWyUkJe9TH8TknXSXo2xfrvuVh7++73daxtDryvcpylY+0QYNM61qM3q8TCqsflQOezr5Id83elffmZNLy/7b45cE1ufo+Q/dDdpEHrA7RGGwvAb4D1yIr3vwaIiGVkRfzPAc9ExFNkO+gNYKOIWD991o2I6b3McxFZcRDI6sjz/TWItw2IOCMidib7Jf5O4Cu9/N0zwHhJ43LDNiP7tU5EvAj8nOyX0t+QFasry1pAVgxeP/dZKyLu6C+uGsZVD//3NOw9EbEu8Lfk6qCbZBEwKe2Hiin9TF/Ltsj//WZk2570/+ZV47rITuYLgD7r+vtTy/6PiHkRMTZ9ejsuB01kV+xVYpmXG9XXdhvouKg+jurellWqv6trkVWr5VUv+4dkVUzTUqxfy8W6iNw6pmOur2NtAfCrquNsbET8fd1rs6pVYiHb3vll93k+i4hnI+JzEfEO4O+AHyi7Am0BWTVvX+uzT9X6tEfEwhpi7e/csoqWSCwR8TrQAfwTb/16h6yq5J9IjdIRsYjspPwfktZNDVVbSfqLXmZ7PfAeSQemX8yfZ9VfIQP5M7mdJ+m9kt6nrJH9VbI68p5e1mUBWWnrm5LaJW0PHEn266/if8iK6wel7oqzgRMkTU/LXE/SJ1Yj5sUppr4OuopxZFUvSyVNovcE2Wi/Ifvl9AVJIyUdQFZv3ZdatsVXJG0gaQpZ3XKlcfNS4DhJW0gaS3bCvDwiusjqyD8i6ZMpjg0l7cgAat3/LaKv7TbQcbHKd4I6t2UvrgL2k/R+ZQ3wpzDwD51xZG2or0jahqytqeJ6YLqkj6fv/jH0/d2/DninpE9LGpU+75W0bY2xV2+TalcAx0iaLGkD4PjKiIHOZ5I+IamScF8kO/H3pJgnSjpW0hhJ4yS9L013NnCq0oUukiak71qt6zJV6WKO/rREYkl+RVYEzN+0OC8Ny19mfChZA9/DZBv7KrJ6xFVExPPAJ4DTyIrV25ElrzdqjOd04CBlV96cAaxL1uj1IllxdglZw3tvZpPVZz5D1jh3ckTcnBs/h6wd6dmIWHn1RURcA3wbuCwV7x8E9qFGqQrhVODXqSi8ax+T/iswA1hK9iW8utZl1Csi3iRrsD8SeIns1/B19LE/atwWPyNrUL+XbD3OT8MvAH5Mdtw8RZYE/jHN909k9eRfIqteu5esEXMgq7P/h7q+tttAx8U3gZPSsfXlAttyFRHxENn+uYzsF/4rwHP0/139MlmJ/2Wy/bLyiqncd/9bZPtpGqkmpJdlv0zW7nEw2ff1WbLjbkyN4Z8CXJS2ySd7Gf/fwE1k7cS/4+3btL/z2XuBOyW9QnbO+GJEPJli3pOsbe5ZsitE/zL9zelp2p9LepmsIb+SdAZyZfp/iaTf9Tdh5UqRNV7Kwp3AIRFxa9nxGEi6k+yKuB/V8bdBVg0yv/GRWZlSKfMlsv37VNnx2Nu1Uoml4STtJWl9SWN4qw72tyWHtcaS9BeSNk3VJocB2wP/V3ZcVj5J+0laW9I6ZJcbP0B2FZoNQWt0YgF2I7t64nmyYuOBqT3HyvEusiqBl8iqTw5K9cxmB5BVRT1DVnV1cLi6ZchyVZiZmTXUml5iMTOzBht2D2vbaKONYurUqWWHYWbWUu65557nI2JCI+Y17BLL1KlT6ejoKDsMM7OWIunpgaeqjavCzMysoZxYzMysoZxYzMysoZxYzMysoZxYzMysoZxYzMysoUpLLJKmSLpV0sPpJTVf7GUaSTpD2Xvh75c0o4xYzcysdmXex9IFfCkifpdeenWPpF9ExMO5afYhey7QNLJHO/+Q2h/xXLc7Ft7BQ0seYkXPCnqiVV+pYWZrmk3W2YRPvHN1XtHUHKUllvRwwUWp+2VJj5C9VjafWA4ALk4Pm/ttehLxxGY+mPDaJ67la7d/bWW/mv7iRDOzxnjPhPes2YklT9JUYCfgzqpRk1j1fdCdaVjTEsuNT93IpLGTuHr/q1lr5Fqs+qZcMzMbSOmN9+mlPT8Fjk3vsa9nHkdJ6pDUsXjx4kLxPPbiY+y8yc6sPWptJxUzszqUmljS+8F/ClwSEb29/nYhMCXXPzkNW0VEnBsRMyNi5oQJ9T9DrSd6eGH5C0xYqyHPYTMzWyOVeVWYyN6l/UhE/Gcfk80BDk1Xh+0KLG1m+8qyN5bR1dPFhmtt2KxFmJkNe2W2sXwA+DTwgKR707CvAZsBRMTZwA3AvsB84DXgiGYG9MLyFwAY3z6+mYsxMxvWyrwq7Hbo/5KrdDXY5wcnIni9K3sr8Tqj1hmsRZqZDTulN94PJcu7lwMwesTokiMxM2tdTiw5b3S/AUD7iPaSIzEza11OLDlvdGWJZczIMSVHYmbWupxYciolljFtTixmZvVyYslZmVhcYjEzq5sTS47bWMzMinNiyVnelV0V5hKLmVn9nFhy3ux5E4DRbb7c2MysXk4sOV09XQCMbBsSD302M2tJTiw5lZd6jdCIkiMxM2tdTiw53dFNm9r8uHwzswKcWHK6e7LEYmZm9fNZNKcnelwNZmZWkBNLTqUqzMzM6uezaE53dDNSviLMzKwIJ5ac7p5u2tq8SczMivBZNMdtLGZmxTmx5LiNxcysOJ9Fc7qj2yUWM7OCnFhyXBVmZlacE0uOq8LMzIrzWTSnp6fHD6A0Myuo1MQi6QJJz0l6sI/xH5K0VNK96fP1ZsbTFV0usZiZFVT2z/MLgTOBi/uZZl5E/NVgBNMTPU4sZmYFlXoWjYjbgBfKjCHPV4WZmRXXCj/Pd5N0n6QbJU3vbQJJR0nqkNSxePHiuhfUEz2MaHNiMTMrYqgnlt8Bm0fEDsB/Af/b20QRcW5EzIyImRMmTKh7Yd09LrGYmRU1pBNLRCyLiFdS9w3AKEkbNWt5vtzYzKy4IX0WlbSp0uscJe1CFu+SZi3PN0iamRVX6lVhki4FPgRsJKkTOBkYBRARZwMHAX8vqQt4HTg4IqJZ8XRHt9tYzMwKKjWxRMTsAcafSXY58qDo7ulm5Miyr8A2M2ttQ7oqbLC5KszMrDgnlpwgECo7DDOzlubEkhMRpGsFzMysTk4sOT30uMRiZlaQE0uOSyxmZsU5sVRxicXMrBgnlhw33puZFefEkuOqMDOz4pxYctx4b2ZWnBNLjkssZmbFObFUcYnFzKwYJ5Ycl1jMzIpzYslxG4uZWXFOLDkusZiZFefEUsUlFjOzYpxYcnyDpJlZcU4sOa4KMzMrzoklpyd6nFjMzApyYslxVZiZWXFOLFWcWMzMinFiyXEbi5lZcaUmFkkXSHpO0oN9jJekMyTNl3S/pBnNjMc3SJqZFVd2ieVCYO9+xu8DTEufo4AfNjMYl1jMzIobWebCI+I2SVP7meQA4OKICOC3ktaXNDEiFjUlntR4373kKV754z30dK8gArJ/zMyGtrZ1xrP+e/r7rT44Sk0sNZgELMj1d6ZhqyQWSUeRlWjYbLPN6l9awKsv/pme/5rJenTVPx8zsxI8NvJdTiyNEhHnAucCzJw5s+7iRQ89LH/mUZbHKG7c5UJGtI+lra0N3O5iZi1g3NixvKvsIBj6iWUhMCXXPzkNa4qIYMybL7FgrW3Y/2P7NWsxZmbDWtmN9wOZAxyarg7bFVjarPYVyNpY2uN1Xl9nysATm5lZr0otsUi6FPgQsJGkTuBkYBRARJwN3ADsC8wHXgOOaGY8EcFoVsDa45u5GDOzYa3sq8JmDzA+gM8PUjj09PQwkqBtHScWM7N6DfWqsEHVHd0IGLn2BmWHYmbWspxYciJ6UIDa1ys7FDOzluXEkheBgBGj28uOxMysZTmx5ARBG8GI0WPKDsXMrGU5seT0pEe3jBrlEouZWb2cWHKyZ4XBSFeFmZnVzYklL5xYzMyKcmLJqZRYRo1xG4uZWb2cWHICaAsY7RKLmVndnFhygh4ARo9Zu+RIzMxalxNLTgAiGDF6dNmhmJm1LCeWVaQbJEe6KszMrF5OLDkR2QZpG+XGezOzejmx5ATp5ZNtQ/39Z2ZmQ1dNiUXSVpLGpO4PSTpG0vrNDa0cCqBtRNlhmJm1rFpLLD8FuiVtTfZu+SnA/zQtqpKE0tvt5YKcmVm9aj2D9kREF/DXwH9FxFeAic0Lq2RS2RGYmbWsWhPLCkmzgcOA69KwUc0JqRyRHkDplGJmVkytieUIYDfg1Ih4StIWwI+bF9bg64ns5shwajEzK6TWy5/2jIhjKj0puSxvUkylWHlFmBOLmVkhtZZYDutl2OENjKN0TixmZo3Rb4kltav8DbCFpDm5UeOAF5oZ2KBLeUXhxGJmVsRAVWF3AIuAjYD/yA1/Gbi/6MIl7Q2cDowAzouIb1WNPxz4DrAwDTozIs4rutze9FBpYzEzsyL6TSwR8TTwNFnDfUNJGgGcBewJdAJ3S5oTEQ9XTXp5RHyh0cuvVrkqzFVhZmbF1Hrn/cclPS5pqaRlkl6WtKzgsncB5kfEkxHxJnAZcEDBedbNbSxmZo1Ra+P9acD+EbFeRKwbEeMiYt2Cy54ELMj1d6Zh1f6fpPslXSVpSm8zknSUpA5JHYsXL64rGJdYzMwao9bE8ueIeKSpkfTuWmBqRGwP/AK4qLeJIuLciJgZETMnTJhQaIFyYjEzK2Sgq8I+njo7JF0O/C/wRmV8RFxdYNkLyZ45VjGZtxrpK/Nfkus9j6zk1BSVGyRdYjEzK2agq8L2y3W/Bnw01x9AkcRyNzAt3cW/EDiY7NLmlSRNjIhFqXd/oGmlJrexmJk1xkBXhR3RrAVHRJekLwA3kV1ufEFEPCTp34COiJgDHCNpf6CL7L6Zw5sWjxOLmVlD1PRIF0ln9DJ4KVkC+Fm9C4+IG4AbqoZ9Pdd9AnBCvfNfzVgAt7GYmRVVa+N9O7Aj8Hj6bE/WJnKkpO83KbZBVUksfgilmVkxtT6EcnvgAxHRDSDph8A84IPAA02KbVC5KszMrDFqLbFsAIzN9a8DjE+J5o3e/6S1rEwsfsmXmVkhtZZYTgPulfRLsp/0uwP/Lmkd4OYmxTao3MZiZtYYNSWWiDhf0g1kj2EB+FpEPJO6v9KUyAbZ+Pbx/OCZTZnY83zZoZiZtbR+q8IkbZP+n0H2jvsF6bNpGjZsSKI9ghE11w6amVlvBiqx/BNwFKs+Mr8igA83PKISiaBHI8oOw8yspQ10g+RR6f+/HJxwyjUiegg33puZFVLrY/PXlnSSpHNT/zRJf9Xc0MrQQ8hVYWZmRdR6Fv0R8Cbw/tS/EPhGUyIqUVv0EG5jMTMrpNaz6FYRcRqwAiAiXmMY3kkoeuhxYjEzK6TWs+ibktYivRJe0lYMkxsj89oIV4WZmRVU6w2SJwP/B0yRdAnwAZr4pOGyyFVhZmaF1ZpYDgOuB64CngS+GBHD7k7CNnro8VVhZmaF1JpYzgdmAXsCWwG/l3RbRJzetMhKIMIlFjOzgmp9pMutkm4D3gv8JXA0MB0YdonFzMyKqfVFX7eQPdH4N2SPy39vRDzXzMDKEX66sZlZQbXW+9xPdh/Lu8nezfLudJXYsKLwi77MzIqqtSrsOABJ48iuBvsRsCkwpmmRlaDy4HwzM6tfrVVhXyBrvN8Z+CNwAVmV2LAiwnnFzKygWq8Kawf+E7gnIrqaGE+pssZ7ZxYzsyJqamOJiO9GxJ2NTiqS9pb0mKT5ko7vZfwYSZen8XdKmtrI5b+dE4uZWVGl3bQhaQRwFrAPsB0wW9J2VZMdCbwYEVsD3wO+3dSYfFWYmVlhZd4NuAswPyKejIg3gcuAA6qmOQC4KHVfBewhNffM7ztZzMyKKTOxTCJ7zXFFZxrW6zSpGm4psGH1jCQdJalDUsfixYvrDkjhqjAzs6KGxfNLIuLciJgZETMnTJhQZE6uCjMzK6jMxLIQmJLrn5yG9TqNpJHAesCSZgXkq8LMzIorM7HcDUyTtIWk0cDBwJyqaeaQPVkZ4CBgbkQ0rRlE+M57M7Oiar2PpeEioivdeHkTMAK4ICIekvRvQEdEzCF7qvKPJc0HXiBLPs2MCpdYzMyKKS2xAETEDcANVcO+nuteDnxisOLx5cZmZsUNi8b7RvLlxmZmxTix5AXIVWFmZoU4seT4IZRmZsU5sazCjfdmZkU5seSIINx4b2ZWiBNLjnL/mplZfZxYcnznvZlZcU4sq/B9LGZmRTmx5LjEYmZWnBNLjoBwXjEzK8SJZRUusZiZFeXEkuOqMDOz4pxYqrnx3sysECeWHPkJlGZmhTmxrMKXG5uZFeXEkuM2FjOz4pxYcvxIFzOz4pxYVuGqMDOzopxYclwVZmZWnBNLTnbnvROLmVkRTiw5LrGYmRXnxJIjt7GYmRVWSmKRNF7SLyQ9nv7foI/puiXdmz5zmh+Z75A0MyuqrBLL8cAtETENuCX19+b1iNgxffZvdlC+3NjMrLiyEssBwEWp+yLgwJLieDtXhZmZFVJWYtkkIhal7meBTfqYrl1Sh6TfSuoz+Ug6Kk3XsXjx4rqDEuHyiplZQSObNWNJNwOb9jLqxHxPRITU5+MfN4+IhZK2BOZKeiAinqieKCLOBc4FmDlzZt0NJVnjva9nMLPerVixgs7OTpYvX152KHVrb29n8uTJjBo1qmnLaFpiiYiP9DVO0p8lTYyIRZImAs/1MY+F6f8nJf0S2Al4W2JpULwurZhZvzo7Oxk3bhxTp05FLVhtHhEsWbKEzs5Otthii6Ytp6yf53OAw1L3YcDPqieQtIGkMal7I+ADwMPNCigC/EgXM+vP8uXL2XDDDVsyqQBIYsMNN2x6iausxPItYE9JjwMfSf1IminpvDTNtkCHpPuAW4FvRUTzEgu+KszMBtaqSaViMOJvWlVYfyJiCbBHL8M7gM+m7juA9wxiTL5B0sysAdxSnWQlFj/Sxcxa14knnsiUKVMYO3ZsqXE4sSQRKaW4xGJmLWq//fbjrrvuKjuMcqrChqIgXGIxs5r967UP8fAzyxo6z+3esS4n7ze9z/Hf+c53GDNmDMcccwzHHXcc9913H3PnzmXu3Lmcf/75XHLJJQ2Np14usSRZiSV8i6SZDVmzZs1i3rx5AHR0dPDKK6+wYsUK5s2bx+67715ydG9xiSXHVWFmVqv+ShbNsvPOO3PPPfewbNkyxowZw4wZM+jo6GDevHmcccYZgx5PX5xYkkqJxVVhZjZUjRo1ii222IILL7yQ97///Wy//fbceuutzJ8/n2233bbs8FZyVVgS6ZH5rX6NupkNb7NmzeK73/0uu+++O7NmzeLss89mp512GlLnLieWpKdSYhk6+8bM7G1mzZrFokWL2G233dhkk01ob29n1qxZAHz1q19l8uTJvPbaa0yePJlTTjmllBhdFZa89awwZxYzG7r22GMPVqxYsbL/D3/4w8ru0047jdNOO62MsFbhEktSuUFyKBUnzcxakRNL4suNzcwaw4mlwnfem5k1hBNLEgRt8uXGZmZFObEkUXnvpPOKmVkhTixJpMwiZxYzs0KcWJKInqzD77w3sxb02muv8bGPfYxtttmG6dOnc/zxx5cWi8+iSaysCzMza01f/vKXefTRR/n973/Pr3/9a2688cZS4vANksn4tUcBMH3S+iVHYmYt4cbj4dkHGjvPTd8D+3yrz9G1PjZ/9OjRzJgxg87OzsbGVyOXWJK2dJnxqBEjSo7EzKx3tT42/6WXXuLaa69ljz3e9gb4QeESy0quCjOz1dBPyaJZanlsfldXF7Nnz+aYY45hyy23HPQYwYnlLZU2Fl8UZmZDVC2PzT/qqKOYNm0axx57bGlxllIVJukTkh6S1CNpZj/T7S3pMUnzJTX5EgffyGJmQ19/j80/6aSTWLp0Kd///vdLjbGsNpYHgY8Dt/U1gaQRwFnAPsB2wGxJ2zUtopUlFicWMxu6+npsfmdnJ6eeeioPP/wwM2bMYMcdd+S8884rJcZSqsIi4hEY8KVauwDzI+LJNO1lwAHAw02KKv3vxGJmQ1d/j80fKrdNDOWrwiYBC3L9nWnY20g6SlKHpI7FixfXtzSXWMzMGqJpJRZJNwOb9jLqxIj4WSOXFRHnAucCzJw5s86U7RKLmVkjNC2xRMRHCs5iITAl1z85DWsOl1jMzBpiKFeF3Q1Mk7SFpNHAwcCc5i3OJRYzs0Yo63Ljv5bUCewGXC/ppjT8HZJuAIiILuALwE3AI8AVEfFQ04JyicXMrCHKuirsGuCaXoY/A+yb678BuGGQokr/O7GYmRUxlKvCyuESi5m1qL333psddtiB6dOnc/TRR9Pd3V1KHE4sFUPk+m8zs3pdccUV3HfffTz44IMsXryYK6+8spQ4/KywlVwVZma1+/Zd3+bRFx5t6Dy3Gb8N/7zLP/c5vtbH5nd1dfHmm28OdBN607jEUuHGezMb4mp5bP5ee+3FxhtvzLhx4zjooINKidMllrdxYjGzgfVXsmiWWh6bf9NNN7F8+XIOOeQQ5s6dy5577jnocbrEUuESi5kNcdWPzZ81a9bbHpsP0N7ezgEHHMDPftbQh5zUzIllJbexmNnQ19dj81999VUWLVoEZG0s119/Pdtss00pMeE1n8kAAAaaSURBVDqxVLjEYmYtoK/H5r/66qvsv//+bL/99uy4445svPHGHH300aXE6DaWipGjYbsDYfwWZUdiZtan/h6bf/fdd5cR0ts4sVS0rwefvKjsKMzMWp6rwszMrKGcWMzMVsNQeUtjvQYjficWM7Matbe3s2TJkpZNLhHBkiVLaG9vb+py3MZiZlajyZMn09nZSd2vQB8C2tvbmTx5clOX4cRiZlajyg2K1j9XhZmZWUM5sZiZWUM5sZiZWUOpVa9u6IukxcDTBWaxEfB8g8JpFWvaOq9p6wte5zVFkXXePCImNCKIYZdYipLUEREzy45jMK1p67ymrS94ndcUQ2WdXRVmZmYN5cRiZmYN5cTydueWHUAJ1rR1XtPWF7zOa4ohsc5uYzEzs4ZyicXMzBrKicXMzBrKiSWRtLekxyTNl3R82fE0iqQpkm6V9LCkhyR9MQ0fL+kXkh5P/2+QhkvSGWk73C9pRrlrUB9JIyT9XtJ1qX8LSXem9bpc0ug0fEzqn5/GTy0z7iIkrS/pKkmPSnpE0m7DeT9LOi4d0w9KulRS+3Dcz5IukPScpAdzw1Z7v0o6LE3/uKTDmhmzEwvZSQg4C9gH2A6YLWm7cqNqmC7gSxGxHbAr8Pm0bscDt0TENOCW1A/ZNpiWPkcBPxz8kBvii8Ajuf5vA9+LiK2BF4Ej0/AjgRfT8O+l6VrV6cD/RcQ2wA5k6z8s97OkScAxwMyIeDcwAjiY4bmfLwT2rhq2WvtV0njgZOB9wC7AyZVk1BQRscZ/gN2Am3L9JwAnlB1Xk9b1Z8CewGPAxDRsIvBY6j4HmJ2bfuV0rfIBJqcv24eB6wCR3Y08snp/AzcBu6XukWk6lb0OdazzesBT1bEP1/0MTAIWAOPTfrsO2Gu47mdgKvBgvfsVmA2ckxu+ynSN/rjEkqkcpBWdadiwkor/OwF3AptExKI06llgk9Q9HLbF94GvAj2pf0PgpYjoSv35dVq5vmn80jR9q9kCWAz8KFUBnidpHYbpfo6IhcB3gT8Bi8j22z0M//1csbr7dVD3txPLGkLSWOCnwLERsSw/LrKfMMPiunNJfwU8FxH3lB3LIBsJzAB+GBE7Aa/yVvUIMOz28wbAAWQJ9R3AOry9umiNMBT3qxNLZiEwJdc/OQ0bFiSNIksql0TE1WnwnyVNTOMnAs+l4a2+LT4A7C/pj8BlZNVhpwPrS6q82C6/TivXN41fD1gymAE3SCfQGRF3pv6ryBLNcN3PHwGeiojFEbECuJps3w/3/Vyxuvt1UPe3E0vmbmBauqJkNFkj4JySY2oISQLOBx6JiP/MjZoDVK4MOYys7aUy/NB0dcmuwNJckXvIi4gTImJyREwl249zI+IQ4FbgoDRZ9fpWtsNBafoh9euvFhHxLLBA0rvSoD2Ahxmm+5msCmxXSWunY7yyvsN6P+es7n69CfiopA1Sae+jaVhzlN0oNVQ+wL7AH4AngBPLjqeB6/VBsmLy/cC96bMvWf3yLcDjwM3A+DS9yK6QewJ4gOyqm9LXo851/xBwXereErgLmA9cCYxJw9tT//w0fsuy4y6wvjsCHWlf/y+wwXDez8C/Ao8CDwI/BsYMx/0MXErWjrSCrGR6ZD37FfhMWv/5wBHNjNmPdDEzs4ZyVZiZmTWUE4uZmTWUE4uZmTWUE4uZmTWUE4uZmTWUE4vZECLpQ5UnMpu1KicWMzNrKCcWszpI+ltJd0m6V9I56f0vr0j6XnpHyC2SJqRpd5T02/R+jGty787YWtLNku6T9DtJW6XZj829V+WSdGe5WctwYjFbTZK2BT4FfCAidgS6gUPIHoTYERHTgV+Rvf8C4GLgnyNie7K7oSvDLwHOiogdgPeT3V0N2ROojyV7N9CWZM/AMmsZIweexMyq7AHsDNydChNrkT0EsAe4PE3zE+BqSesB60fEr9Lwi4ArJY0DJkXENQARsRwgze+uiOhM/feSvYvj9uavllljOLGYrT4BF0XECasMlP6larp6n5f0Rq67G39PrcW4Ksxs9d0CHCRpY1j5/vHNyb5PlSfr/g1we0QsBV6UNCsN/zTwq4h4GeiUdGCaxxhJaw/qWpg1iX8Jma2miHhY0knAzyW1kT119vNkL9faJY17jqwdBrLHmp+dEseTwBFp+KeBcyT9W5rHJwZxNcyaxk83NmsQSa9ExNiy4zArm6vCzMysoVxiMTOzhnKJxczMGsqJxczMGsqJxczMGsqJxczMGsqJxczMGur/Awfm6XyiunL1AAAAAElFTkSuQmCC\n"
          },
          "metadata": {
            "needs_background": "light"
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "And finally, compare to the results we got from batch gradient descent to our other results:"
      ],
      "metadata": {
        "id": "tGQXMyhCiQOW"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "model_results['mini-batch'] = weights\n",
        "model_results"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 171
        },
        "id": "zqkfDUK-iLib",
        "outputId": "f505113b-2296-44f7-89d3-9fb40b1efd52"
      },
      "execution_count": 256,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "   batch gradient descent  linear regression  true coeffs       SGD  \\\n",
              "0                1.000242           1.000321          1.0  1.179115   \n",
              "1                1.000179           1.000202          1.0  1.008264   \n",
              "2                2.020670           2.020693          2.0  1.985073   \n",
              "\n",
              "   mini-batch  \n",
              "0    0.997012  \n",
              "1    1.000837  \n",
              "2    2.020861  "
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-bca31fbf-7dff-49aa-9fd4-3eb2f69d5fae\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>batch gradient descent</th>\n",
              "      <th>linear regression</th>\n",
              "      <th>true coeffs</th>\n",
              "      <th>SGD</th>\n",
              "      <th>mini-batch</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>1.000242</td>\n",
              "      <td>1.000321</td>\n",
              "      <td>1.0</td>\n",
              "      <td>1.179115</td>\n",
              "      <td>0.997012</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>1.000179</td>\n",
              "      <td>1.000202</td>\n",
              "      <td>1.0</td>\n",
              "      <td>1.008264</td>\n",
              "      <td>1.000837</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>2.020670</td>\n",
              "      <td>2.020693</td>\n",
              "      <td>2.0</td>\n",
              "      <td>1.985073</td>\n",
              "      <td>2.020861</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-bca31fbf-7dff-49aa-9fd4-3eb2f69d5fae')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "        \n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "      \n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-bca31fbf-7dff-49aa-9fd4-3eb2f69d5fae button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-bca31fbf-7dff-49aa-9fd4-3eb2f69d5fae');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n",
              "  "
            ]
          },
          "metadata": {},
          "execution_count": 256
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Let's also compare the time it took to run these three different types of gradient descent:"
      ],
      "metadata": {
        "id": "-sCa_doOQaq0"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "print(\"batch gradient descent time: %.3f\" % batch_gd_time)\n",
        "print(\"SGD time: %.3f\" % sgd_time)\n",
        "print(\"mini batch gradient descent time: %.3f\" % minibatch_time)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "RIHsS5VmQaBk",
        "outputId": "ec5a4c63-876b-42b6-9dfa-215c5dbcb169"
      },
      "execution_count": 257,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "batch gradient descent time: 35.231\n",
            "SGD time: 517.481\n",
            "mini batch gradient descent time: 6.185\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Overall, all methods managed to converge on weights that are close to the true values we used to generate the data, and also comparable to the results of using sklearn's linear regression function. However, we observed that SGD and mini-batch appeared to converge faster. Using SGD and mini-batch gradient descent, we could have probably reduced the number of epochs to reach the optimal weights in less time. SGD took very long to run because we updated the model weights after evaluating every single sample, and every single sample was evaluated 1000 times. Mini-batch gradient descent was much faster, and demonstrates that it is a good mix of the pros and cons of batch gradient descent and SGD. "
      ],
      "metadata": {
        "id": "spSLZNaAivjG"
      }
    },
    {
      "cell_type": "code",
      "source": [],
      "metadata": {
        "id": "DuL6Xu-luyNq"
      },
      "execution_count": null,
      "outputs": []
    }
  ]
}