Practice extra 6 - Using different learning schedules.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Recommender Systems 2017/18\n",
    "\n",
    "### Practice 11 - Using different learning schedules with LightFM\n",
    "\n",
    "##### The following notebook is available at [LightFM Github repo](https://github.com/lyst/lightfm)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Using different learning schedules\n",
    "`lightfm` implements two learning schedules: adagrad and adadelta. Neither is clearly superior, and, like other hyperparameter choices, the best learning schedule will differ based on the problem at hand.\n",
    "\n",
    "This example tries both at the Movielens 100k dataset.\n",
    "\n",
    "## Preliminaries\n",
    "Let's first get the data and define the evaluations functions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import data\n",
    "\n",
    "%matplotlib inline\n",
    "\n",
    "import matplotlib\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "from lightfm import LightFM\n",
    "from lightfm.datasets import fetch_movielens\n",
    "from lightfm.evaluation import auc_score\n",
    "\n",
    "movielens = fetch_movielens()\n",
    "\n",
    "train, test = movielens['train'], movielens['test']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Experiment\n",
    "To evaluate the performance of both learning schedules, let's create two models and run each for a number of epochs, measuring the ROC AUC on the test set at the end of each epoch."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "alpha = 1e-3\n",
    "epochs = 70\n",
    "\n",
    "adagrad_model = LightFM(no_components=30,\n",
    "                        loss='warp',\n",
    "                        learning_schedule='adagrad',\n",
    "                        user_alpha=alpha,\n",
    "                        item_alpha=alpha)\n",
    "adadelta_model = LightFM(no_components=30,\n",
    "                        loss='warp',\n",
    "                        learning_schedule='adadelta',\n",
    "                        user_alpha=alpha,\n",
    "                        item_alpha=alpha)\n",
    "\n",
    "adagrad_auc = []\n",
    "\n",
    "for epoch in range(epochs):\n",
    "    adagrad_model.fit_partial(train, epochs=1)\n",
    "    adagrad_auc.append(auc_score(adagrad_model, test).mean())\n",
    "    \n",
    "    \n",
    "adadelta_auc = []\n",
    "\n",
    "for epoch in range(epochs):\n",
    "    adadelta_model.fit_partial(train, epochs=1)\n",
    "    adadelta_auc.append(auc_score(adadelta_model, test).mean())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It looks like the adadelta gets to a better result at the beginning of training. However, as we keep running more epochs adagrad wins out, converging to a better final solution."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXsAAAEACAYAAABS29YJAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xl8VPW9//HXJ/u+hywkJAgE2UEUQaEErVa9P4vW64L7\nz1att25dXXrbC9feau3i8qv12qq0tLZW7a21KtpeIQiooLKvARHIypIEsq/z+f1xJmESlixMMjPJ\n5/l4nEdmzjkz85kR3/Od7/me7xFVxRhjzOAW5OsCjDHG9D8Le2OMGQIs7I0xZgiwsDfGmCHAwt4Y\nY4YAC3tjjBkCehT2InKJiOwQkV0i8sAJtieKyF9FZKOIrBGRCe712SKyXES2isgWEbnX22/AGGNM\n96S7cfYiEgzsBL4IlAAfAwtUdbvHPj8FqlX1EREZCzyjql8UkXQgXVU3iEgM8ClwhedjjTHG9L+e\ntOxnALtVda+qtgAvA/O77DMOWA6gqjuBXBFJVdVyVd3gXl8LbAcyvVa9McaYHulJ2A8HijzuF7vX\nedoIfAVARGYAOUCW5w4ikgtMA9b0rVRjjDF91ZOw78l8Co8BCSKyHrgbWA+0tW90d+G8BtznbuEb\nY4wZQCE92KcEyPa4n43Tuu+gqjXAbe33ReRzYI/7dijwF+APqvp61ycXEZucxxhj+kBVpaf79qRl\n/wkwRkRyRSQMuBZ4w3MHEYl3b0NEbgdWqGqtiAjwArBNVZ88RcEBu/zHf/yHz2uw+n1fx1CsP5Br\nHwz191a3LXtVbRWRu4F3gWDgBVXdLiJ3urc/B4wHfutupW8Bvup++PnAjcAmdxcPwEOq+k6vKzXG\nGNNnPenGQVWXAku7rHvO4/aHwNgTPG4VduKWMcb4nAXxacrPz/d1CafF6vetQK4/kGuHwK+/t7o9\nqarfCxBRX9dgjDGBRkRQLx+gNcYYE+As7I0xZgiwsDfGmCHAwt4YY4YAC3tjjBkCLOyNMWYIsLA3\nxpghwMLeGGOGAAt7Y4wZAizsjTFmCLCwN8aYIcDC3hhjhgALe2OMGQIs7I0xZgjo0cVLjDEmELhc\n0Nzs3A4KOraIOOubmo4tzc2gemwBaGuDlhZnW0vL8Utr67Ht7c/R/retzdne1uYsLlfn2jxfy+Vy\nFs/bnkv767T/TU+HJ096YdeesbA3xpyQKjQ2Qm2ts7S2dg4k1WNhGhzs/G1rg6oqZ6msdP42NBwf\nbu2BeKKltfVYaLbv2/74piY4ehSqq52/R486z98eum1tEBbmhHvXWkNDITz82NK+n+cSFOTsFxbm\n/D3REhJy7PHtf8PCnPXBwc4SEuI8V1ftr9H+BSRy7LNrXxcU5Dze8/USE0//v6eFvTEBwOVygrex\n0Qm3rq3KpiYnACsrjy1VVVBXB/X1x5b24PVsUba1Oc/hudTVOQEfEgIxMc4SGtq5tQzO83iGclAQ\nJCU54ZSY6NyOjHQCrT3I2gMuONgJy/bb7SHpGZqeQRgc7NQQH39siYuD6OjOgSsnuJyH6onXDyV2\npSpjeqn9p377T+zGxs4t2aoqJyi7BmhDw7FWcvvSHsbtf0/WCm5udoIxIsIJT8/WafsSH++Ea/uS\nmOiEdFTUsSUi4vjujfbQbX+e0FBn35gY577xT729UpWFvRn0XC7n575nq7em5lhXQfvfmho4fLjz\nUl19fEC3dwm0/9QODz++NRsTcyxA23+OR0cfayXHxDj325f2MD5ZKzgiwlqmpjML+0FEVWlua6bV\n1Uqrq5UWVwutrlaCJIjQoFDCgsMIDQ4lNCgUGSJJ0NICe/fC7t3OUlnptIbbuynq6uDIkc7BfvSo\nE6jJycdave3dEp5dBrGxkJLiLMnJzhIf76yPiXH+Rkc7jzHG1yzs/ZSq0upqpamtiea2Zupb6imr\nKaO4upiSmhJKqksorS3lYN3BTouqdgR6SFAIwUHBHV8CLa4WWtqcL4CIkAhiwmI6lsjQSARBUdo/\n39jwWKamTeWsjLOYljGNscljCQ4K7qivvqWe2uZaGlsbO75cWtpaEBHGJo8lNDh0AD4nJ8S3bIGS\nkmNLaakT8kVFMHw4jB4No0ZBamrnVnFU1LHWdfvfhAQn2I0ZTCzsfaDN1cbOip18Wvop68rWsb96\nP5UNlZ2WhpYGgiSI8JBwwoPDiQyNJCMmg+Fxwxke6yyZsZmkxaQxLHoYw6KHkRqVSmRoZLevr6o0\ntDZQ21zbsdS31AMgOP8WRITKhko2lG9gXdk61pevp6ymjPiIeGqaaqhrqSM8OJyYsBgiQiIICQrp\n+JJpdbVSWlPK+SPOZ17uPPJz8zkr4yxCgk6vidvSAnv2wPbt8PHHzvLJJ04resoUyM52gj0z0/mb\nkwMjR1o/sjFgYT9gXOridxt+x/Prn2dj+UbSY9KZnjmd6RnTGZkwkuSoZJIik0iKTCIxIpGo0KiO\nVrS/ONp4lOqmamLDY4kJizlleFfUV/D+vvdZvnc5y/cup7CikLjwOOLC44gPjycuPI7s+GxmZM7g\n3KxzmZo+lbDgMFwup2W+ezfs2uX83bnTWfbudUL8zDPh7LPhnHOcJS1t4D4DYwKVhf0A+KT0E+5+\n+24AFuYvZGbWTBIiEnxc1cBqbG2kuqma6qbqji+Nz498zvt71vD+Z2soqt9F+JFJNBxOIzSsjajo\nNiKjW4mMUrLihzNl+FhmjhnLpIyx5CbkUtdcR1VjFUcaj1DVUIWIMDltMhkxGUPmeIQxvWFh348O\n1x/m++99nzcK3+DRCx/l5ik3EyRDc8aJ2lr47LNjB0p374b1650W+7nnwqy5tQybso6U7Epio0MI\nlmCCg4IRhP1H97OzYic7K3ZSWFHIviP7iAmLISEigcTIRBIiEmh1tbKxfCPBQcFMS5/G1PSpnJ15\nNrOyZjE8briv374xPmdh3w9c6uL5dc/zg+U/4LoJ17Fo3qIh1ZJXdfrWV6+GVaucZd8+OOOMYwdK\nR4+GiROdbpjwcG+9rlJcXcyG8g2sL1/P2pK1fFT8EVGhUczKnsWsrFmMSxnHiPgRZMdnExMW450X\nNiYAWNh72bqyddz11l2EBIXw7L88y+S0yb4uqd+pOv3rBQWwfDmsWOGM8Z4zB84/H2bPhkmTfDME\nUVXZVbmLD4s+5KPij9hVuYui6iL2H91PZEgk2fHZJEcmO78SIpxfCcOihzE9czrnZJ5DfET8wBdt\nTD+wsPeSI41H+MGyH/Dqtld59MJHuWXqLYO2y6atzRnquGqV03pfscI5oWfePGeZO9cZBePPXeeq\nSkVDBcXVxVTUVzh9/+5jACXVJXxc+jEbyjeQk5DDucOdA8g58TnkJOQwIn4EiRGJdmzABBQLey/Y\ne2Qvc387l0tHX8qPL/wxSZFJvi7J62pqYMkS+Pvf4aOPnFn1Zs92Wu6BEO590dLWwuaDm1lbspbN\nBzazv3o/+486S0tbCylRKUSHRRMdGk10WDQxYTGkRqWSGZtJRkwGmbGZDI8bzqjEUSRHJfv67Zgh\nzsL+NFU2VHL+i+fzb2f/G/ece4+vy/G6PXvgl7+E3/3OabXfeKMT8Kmpvq7Mt6qbqqmor6CupY66\n5jrqWuqoaarhUP0hSmtKKaspo7S2lJLqEnZV7iIkKIS85DzGJI1hVOIosuKyyIrL6jhvIiEiwX4p\nmH5lYX8aGloauOj3F3Fe9nk8ftHjvi7Ha5qb4d134YUXnK6a226Db3zDOUnJ9J6qcqj+EIUVheyq\n2MXuyt3OWdDuM6GLq4tpdbWSHpPeaUmMSOx0lnNceBxnZ55NToL9hzC9Z2HfR22uNq5+9WoiQiL4\nw1f+EPD9821tzsHVl1+Gv/4VJkxwWvE33ODM72L6V21zLQdqD1BeW05ZbRnlteUcaTzS6SznyoZK\n1pSsISYshnm585iXO4+ZWTMJDwnvmOJCUWLCYgZlV6I5PRb2faCq3LP0HrYf3s7b179NeIiXxg76\nQGkpPPssPP88ZGXBddfBNdc4Uw8Y/6OqbDu0jWWfL2P53uV8UvoJbdqGIIgIgnC06SiRIZFMGDaB\nCanOEhka6cyrVO38oiitKSU0OLTjrO3kyOSOXxJRoVEdS/sXR3JUMsmRycSFx1l3U4CysO+Dx1c/\nzkubX+L9W98P2KF5a9fCU0/B0qWwYAHcc48zDYEJfKpKSU0JWw9uZcvBLWw9tJWmtiayYo8dI8iM\nzaRN246bk6muuY76lnrqW+upb6mnpqmGioYKKuorqGiooLG1kZz4HM4fcT6zs2cze8Rszkw5074A\nAoDXw15ELgGeBIKB51X1J122JwIvAmcAjcBtqrq1J4917+PTsH9n9zt89Y2vsuZra8iKy/JZHX21\nbBn88IdQXOwE/Fe/6szyaExPNLU2satyF6v3r2Z10WpW7V9FdVM1qdGpNLY20tjaSFNrE01tTcSF\nx5ESlUJKVAqpUakMix7GGYlnMCZpDGOSx3BG4hmEBdssdQPFq2EvIsHATuCLQAnwMbBAVbd77PNT\noFpVHxGRscAzqvrFnjzW/Xifhf2eqj3MemEWr139GnNy5vikhr766CP4/vdh/35YtMjpqrF51o03\nlNaUcrTxKBEhEYSHhBMREkFYcBjVTdUcrj/MobpDHK4/THltOZ9Vfcauyl0UVhRSXF1MalQq0WHR\nRIVGERkSSVRoVMcw1phQ58B0dFg0YcFhBElQxxISFEJKVEqnA9oJEQlUNlRyqO4QB+sOcqj+EEcb\nj9LU1tTxBdTU2kRwUHDHa0WGOn+Hxw7njMQzyIzN9LsJCL2lt2HfXTzMAHar6l73k78MzAc8A3sc\n8BiAqu4UkVwRGQaM6sFjfaauuY4r/3wl/z7n3wMq6Ddtgn//d9iwwWnR33KLzdVuvCszNpPM2Mzj\n1seExZxwfbvmtmbKa8udbqOWehpaGqhvqe8Yzup5cLqxtRGXujqWlrYW1pevp7y2vGM50niEpMgk\nUqNSSY1OJTUqlYSIBMKDwzumCg8PCcelro4T6upb6qltqaWkuoQ9VXuobKgkJyGHrLgsmlqbnIn7\nmpyJ+5rbmkmOTO547vZfLKnRnW9nxmaSHZd93PUcDtYdZOW+lby/7312Vuxkzog5/EvevzAlbYpf\ndoN1F/bDgSKP+8XAuV322Qh8BVglIjOAHCCrh4/1CVXljjfvYEraFO6ecbevy+mRkhL4wQ/g7bfh\noYfglVecS9UZ4y/CgsMYET/C12V00tDSwN4jeymuLiYyNLJjWu648DhCg0KpaKjo+LVyqP5Qx6+W\nPVV7Ou6X1JRQVlPGsOhh5CTkkBGTwZaDWzhQd4Dzs89nzog55Ofms2LfCq5+9WoaWhq4bMxlTEuf\nxuH6wxyoO+AstQdodbV2Gn4bExZDbkIuE1InMD51PDkJOf02ErC7sO9J/8pjwFMish7YDKwH2nr4\nWAAWLlzYcTs/P5/8/PyePrRPnlrzFNsObWP1bav98hvYU00NPP44/OpXcOedzqyS8YF5DNmYARcZ\nGsm41HGMSx13wu2x4bHkJuR2+zytrlZKqkvYd3QfpTWl/OALP2DisImduoiuHHclT17yJIUVhbxV\n+Bbry9eTGpVKXnIec0bMIS0mjdCgUOpajv3KqW6qZk/VHpZ9voyth7ZS1VDF6KTRHSMCVRVFyU3I\n5Rup36CgoKDPn0V3ffYzgYWqeon7/kOA60QHWj0e8zkwCZjYk8cOdJ/9qv2ruOqVq/joqx8xMnHk\ngL1ub7W1wYsvOl01F10EP/oRjPCvRpMxxsuqm6rZVbGLFldLp6vMRYZEMiltUqd9vd1n/wkwRkRy\ngVLgWmBBlxeMBxpUtVlEbgdWqGqtiHT72IHW5mrj3976N5657Bm/DvpVq+Dee53rqb75Jkyf7uuK\njDEDIS48jumZ/fM//CnDXlVbReRu4F2c4ZMvqOp2EbnTvf05YDzwWxFRYAvw1VM9tl/eRQ8t2biE\nuPA4rhp3lS/LOKniYvje92DlSvjJT5zx8n7ey2SMCRBD5qSq+pZ68v5fHq9d8xozs2b2++v1RmMj\n/Pzn8MQTcNdd8OCDNqWBMebUvN2NM2g88eETnJd9nl8FvSq88QZ885swZQp8/LEztbAxxnjbkAj7\ng3UHeeKjJ1jztTW+LqXDjh1w//3O5f3++7/h4ot9XZExZjAL7Kkde2hRwSJunHwjo5JG+boUqqqc\nlvzs2U7Ab9pkQW+M6X+DvmW/8/BOXtn2Cju+scOndbS2wnPPwX/+J1x5JWzbBsOG+bQkY8wQMujD\n/sH3HuS7533Xp5eRe/dd+Na3ICMD/vlPmDz4r1lujPEzgzrs15asZV3ZOv501Z988vq7djkhv307\n/OIXcPnlNpTSGOMbg7rP/sX1L3LX2XcRETKwk8jU1MADD8CsWU7f/Nat8OUvW9AbY3xn0IZ9Y2sj\nr257lRsm3TCgr/vKKzB2LJSXw+bNTuiHB+6Fr4wxg8Sg7cZ5s/BNpqZPJTt+YK7HV1fnTHGwciX8\n5S9Oq94YY/zFoG3ZL9m4hJsn3zwgr7VpE5x9tjPi5tNPLeiNMf5nUIb9wbqDvL/vfb4y7iv9+jqq\n8MwzcOGFzlWjfvc7iI3t15c0xpg+GZTdOC9veZnLx15ObHj/Je/+/c788gcPwgcfwJgx/fZSxhhz\n2gZly74/u3BcLqc1P306zJnjXAvWgt4Y4+8GXct+68GtlNWWccHIC7z+3Dt3wte+5gT+ypVw5ple\nfwljjOkXg65l//tNv+fGSTd69YrylZXOPPPnnw/XXmtBb4wJPIMq7Ntcbby0+SVumnKTV56vvh4e\nfdQZN19d7Yy6uftuCBpUn5oxZigYVLFVsLeAYdHDmDhs4mk9j8sFv/mN0xe/fj2sXu1MQ5yZ6aVC\njTFmgA2qPvslm5Zw0+TTa9WXl8OttzpTEf/tb874eWOMCXSDpmXf0NLA33b8jQUT+35N87fegmnT\n4JxznIt+W9AbYwaLQdOyX1uylrEpY0mLSev1Yxsa4LvfhTffdOa2mTOnHwo0xhgfGjQt+1X7VzE7\ne3avH7dtG8yYARUVsGGDBb0xZnAaPGFftIrZI3oe9qrwwgswd64z5/wf/wgJCf1YoDHG+NCg6MZp\nc7XxYdGHLLliSY/2r66Gr3/dmYJ4xQoYP76fCzTGGB8bFC37zQc3kx6TTmp0arf7btzoTHUQGwtr\n11rQG2OGhkHRsl+1v2ddOKtWwVe+Ak8+CddfPwCFGWOMnxgULftV+1cxZ8Spj6y+844T9C+9ZEFv\njBl6Aj7sVZWV+1eesmX/6qtwyy3w+utw0UUDWJwxxviJgA/7vUf24lIXZySeccLtL74I990H//gH\nnHfeABdnjDF+IuD77Nu7cETkuG2vvAKLFkFBAeTlDXxtxhjjLwZF2J+oC2f/fmeGyqVLLeiNMSbg\nu3FOdDJVWxvcdBN8+9vOMEtjjBnqAjrsK+orKK4uZnLa5E7rf/pTZ87573zHR4UZY4yfCehunNVF\nq5mZNZOQoGNv45NP4IknnL/B3rtYlTHGBLSAbtl3nfysrs4ZQ//LX0J2tg8LM8YYPxPwYT8n59jJ\nVN/8pnOd2Kuv9mFRxhjjh7oNexG5RER2iMguEXngBNtTROQdEdkgIltE5FaPbQ+JyFYR2SwifxSR\ncG8V3tDSwKYDm5gxfAbgTE/89tvw9NPeegVjjBk8Thn2IhIM/BK4BBgPLBCRcV12uxtYr6pTgXzg\n5yISIiK5wO3AWao6CQgGrvNW4WtL1jJx2ESiQqMAZ76be+5xJjgzxhjTWXct+xnAblXdq6otwMvA\n/C77lAFx7ttxQIWqtgLVQAsQJSIhQBRQ4q3CPefDKS93rhd7++3eenZjjBlcugv74UCRx/1i9zpP\nvwEmiEgpsBG4D0BVK4GfA/uBUuCIqv6vN4qGzuPrn30WrrsOkpK89ezGGDO4dBf22oPneBjYoKqZ\nwFTgGRGJEZFRwP1ALpAJxIjIDadTrKcN5RuYnjmdxkb47/925r8xxhhzYt2Nsy8BPAcxZuO07j2d\nB/wXgKp+JiKfA+OAkcAHqloBICL/4973pa4vsnDhwo7b+fn55Ofnn7Ko6qZqappqGB47nMWLnbNk\nzzyzm3dijDEBrKCggIKCgj4/XlRP3nh397XvBC7E6YpZCyxQ1e0e+/wCOKqqi0QkDfgUmIzzxfAS\ncA7QCPwWWKuqz3R5DT1VDSfyaemnfO3vX2PdHeuZMgV+9jO4+OJePYUxxgQ0EUFVj58B8iRO2bJX\n1VYRuRt4F2c0zQuqul1E7nRvfw74MbBYRDbidAt9z91fXykiS4BPABewDvh1X95UV4UVheQl57F8\nuTMPjs1Rb4wxp3bKlv2AFNCHlv2igkW0ulrZ8MQjfPnLNgrHGDP09LZlH5Bn0O6s2Elcax5r1sCN\nN/q6GmOM8X8BGfaFFYWseTuPO+6AyEhfV2OMMf4v4Ga9VFUKKwrZ+5c8frbS19UYY0xgCLiW/YG6\nA4RKBDHBieTm+roaY4wJDAHXsi+sKCRJ85hxvq8rMcaYwBFwLfvCikKkMo/Zx1921hhjzEkEZNhX\n7c7jfGvZG2NMjwVc2G8uLaS+KI8JE3xdiTHGBI6AC/stZYWclZNn15c1xpheCKiwb3O1Ud74ORdO\nG+XrUowxJqAEVNjvO7qPoIY05s2xM6mMMaY3AirsN5bspPVAHuec4+tKjDEmsARU2C/bWEhqUB5R\nUb6uxBhjAktAhf3HnxUyPi3P12UYY0zACaiw311VyJxxY31dhjHGBJyACfu2NqgMKuT/nGcte2OM\n6a2ACftPNzVA1EHOOmOEr0sxxpiAEzBh/7eVu4l3nUFwkJ1NZYwxvRUwYf/+1kJGxlsXjjHG9EXA\nhP3m0kKm51rYG2NMXwRE2BcVQWN0IeeOtrA3xpi+CIiwX70aorMLGZtsYW+MMX0REGH/wQfQGLOT\nPAt7Y4zpk4AI+x37KiCohWHRw3xdijHGBKSACPs9R3eRG5eHiPi6FGOMCUgBEfZlzYWMH2bTJBhj\nTF/5fdjX1UFTTCETM6y/3hhj+srvw764GKIy9pGbkOPrUowxJmD5fdgXFUFofCUpUSm+LsUYYwJW\nQIR9UHQFSZFJvi7FGGMCVkCEvSu8kuSoZF+XYowxAcvvw764GJqCKq1lb4wxp8Hvw35/kYsGPUJC\nRIKvSzHGmIDl92G/r/wo0aGxhASF+LoUY4wJWH4f9sWVFSRbF44xxpwWvw776mpoCakkNcYOzhpj\nzOnoNuxF5BIR2SEiu0TkgRNsTxGRd0Rkg4hsEZFbPbYliMhrIrJdRLaJyMzeFFdcDCkjbNilMcac\nrlOGvYgEA78ELgHGAwtEZFyX3e4G1qvqVCAf+LmItHewPwW8rarjgMnA9t4UV1QECRk27NIYY05X\ndy37GcBuVd2rqi3Ay8D8LvuUAXHu23FAhaq2ikg8MEdVXwRQ1VZVPdqb4oqKIDqlgqQIa9kbY8zp\n6C7shwNFHveL3es8/QaYICKlwEbgPvf6kcAhEVksIutE5DciEtWb4oqLISLRxtgbY8zp6i7stQfP\n8TCwQVUzganAMyISC4QAZwG/UtWzgDrgwd4UV1QEIbHWjWOMMaeru8HrJUC2x/1snNa9p/OA/wJQ\n1c9E5HNgrHu/YlX92L3fa5wk7BcuXNhxOz8/n/z8fMA9VcK0CpIiZ/TgrRhjzOBVUFBAQUFBnx8v\nqidvvLsPtO4ELgRKgbXAAlXd7rHPL4CjqrpIRNKAT4HJqlopIu8DX1PVQhFZCESq6gNdXkNPVsO4\ncZB836V8/8J7uXTMpX1+k8YYM9iICKra48v3nbJl7z7QejfwLhAMvKCq20XkTvf254AfA4tFZCNO\nt9D3VLXS/RT3AC+JSBjwGfB/e1qYqtOyj8KGXhpjzOnqdg4CVV0KLO2y7jmP24eBy0/y2I3AOX0p\n7OhRCAqCo83WZ2+MMafLb8+gLSqCrCyoaLCWvTHGnC7/DvvsNmqaamzGS2OMOU1+G/bFxTAs5wjx\nEfEEid+WaYwxAcFvU7SoCBKHWxeOMcZ4g1+HfWxqJcmRdnDWGGNOl1+HfWSyteyNMcYb/PbyT8XF\nEBZXSXKjteyNMeZ0+WXYt59QpVEVJGEte2OMOV1+2Y1TWQlhYVDvshOqjDHGG/wy7IuKIDsbKuqt\nz94YY7zBL8O+uNgJ+8pGm8veGGO8wS/Dvr1lX9lgQy+NMcYb/Dbss7KsG8cYY7zFb8O+o2VvB2iN\nMea0+WXYt/fZ24yXxhjjHX4Z9kVFkJbZQn1LPfHh8b4uxxhjAp7fhb2q07KPSakiISIBkR5fdcsY\nY8xJ+F3YHzoE0dHQKDbs0hhjvMXvwr5jjL0NuzTGGK/xu7C3YZfGGON9fhf2xcVO2NuwS2OM8R6/\nC/uyMsjMdA+7jLCWvTHGeIPfhX1pKWRkWMveGGO8ye/CvqNlb332xhjjNX4Z9hkZzoyXNhrHGGO8\nw+/CvrTUWvbGGONtfhX2LS1QVQWpqU6fvYW9McZ4h1+FfXm5E/TBwXaA1hhjvMmvwr794CzYjJfG\nGONNfhf2GRnQ3NZMY2sjsWGxvi7JGGMGBb8Ke88x9kmRSTbjpTHGeIlfhX17N45NgmaMMd7ld2Gf\nkWHDLo0xxtv8Kuzbx9jbsEtjjPEuvwr7jrNnbdilMcZ4lV+FffsBWpvx0hhjvKvbsBeRS0Rkh4js\nEpEHTrA9RUTeEZENIrJFRG7tsj1YRNaLyN9P9TqtrVBRAWlp1rI3xhhvCznVRhEJBn4JfBEoAT4W\nkTdUdbvHbncD61X1IRFJAXaKyB9UtdW9/T5gG3DKQfMHD0JyMoSEOAdoR8SP6Ot7Msb0AxsK7Tuq\netrPccqwB2YAu1V1L4CIvAzMBzzDvgyY7L4dB1S0B72IZAGXAf8FfOtUL9R+cBZsxktj/JU3Qsf0\njre+ZLvrxhkOFHncL3av8/QbYIKIlAIbcVry7Z4Avgu4uiuk/eAs2NBLY4zxtu5a9j35Gn8Y2KCq\n+SIyCviniEwB5gIHVXW9iOSf6gkWLlzIp586gV9QkG9DL40xpouCggIKCgr6/Hg51c8yEZkJLFTV\nS9z3HwIjwk5cAAAOzUlEQVRcqvoTj33eBv5LVVe7778HPAhcCdwEtAIROF08f1HVm7u8hqoqCxeC\nywX/+Z+Q/UQ2q29bbf32xvgREbFuHB842efuXt/jPp7uunE+AcaISK6IhAHXAm902WcHzgFcRCQN\nGAt8pqoPq2q2qo4ErgOWdQ16T57dONayN8YY7zpl2LsPtN4NvIszoubPqrpdRO4UkTvdu/0YOFtE\nNgL/C3xPVStP9HSneq32MfaNrY20ulqJDo3u9ZsxxphT2bt3L0FBQbhc3R5GHBC5ubm89957A/Ja\n3fXZo6pLgaVd1j3ncfswcHk3z7ECWHGqfTwnQbMZL40xQ4GIDFjW+c0ZtJ2mSrBhl8aYAOMvvxZO\nxi/Cvq3NOakqPd2GXRpjeu+xxx5j9OjRxMXFMWHCBF5//XUA2tra+M53vkNqaiqjRo3irbfe6vS4\nxYsXM378eOLi4hg1ahS//vWvO21//PHHyczMJCsri+eff56goCD27NkDwK233spdd93FZZddRkxM\nDAUFBbz11ltMmzaN+Ph4RowYwaJFizo93+9//3tycnJISUnhxz/+cT9+Iiegqj5dAC0rU01NVVVV\n/Z9t/6NXvHyFGmP8ixMX/unVV1/VsrIyVVX985//rNHR0VpWVqbPPvusnnnmmVpcXKyVlZWan5+v\nQUFB2tbWpqqqb731lu7Zs0dVVVesWKFRUVG6bt06VVVdunSppqen67Zt27S+vl5vuOEGFRH97LPP\nVFX1lltu0fj4eP3ggw9UVbWxsVELCgp0y5Ytqqq6adMmTUtL09dff11VVbdu3aoxMTG6cuVKbWpq\n0m9961saEhKi77333inf28k+d/f6HmetX7TsPUfilNeWkxKZ4tuCjDG9JuKdpS/+9V//lfT0dACu\nueYaxowZw9q1a3n11Vf55je/yfDhw0lMTOThhx/uNIzxsssuY+TIkQB84Qtf4OKLL2blypUAvPLK\nK9x2222MGzeOyMjI41rpAFdccQWzZs0CIDw8nLlz5zJhwgQAJk2axHXXXceKFc7hytdee43LL7+c\n2bNnExYWxiOPPEJQ0MBFsF+EvedUCSv3r2RW9izfFmSM6TVV7yx9sWTJEqZNm0ZiYiKJiYls2bKF\nw4cPU1paSnZ2dsd+I0Z0Pndn6dKlzJw5k+TkZBITE3n77bepqKgAoKysrNNjs7KyOj1WRDptB1iz\nZg3z5s1j2LBhJCQk8Nxzz3U8X2lpaafniIqKIjl54I5P+kXYt7fsVZVlny/jgpEX+LokY0yA2Ldv\nH3fccQfPPPMMlZWVVFVVMXHiRFSVjIwM9u/f37Gv5+2mpiauuuoqvve973Hw4EGqqqq47LLLOlr+\nGRkZFBUdmy3G8/bJXH/99VxxxRUUFxdz5MgRvv71r3c8X2ZmZqfnqK+v7/giGAh+FfZbD20lOiya\n3IRcX5dkjAkQdXV1iAgpKSm4XC4WL17Mli1bAKdL5+mnn6akpISqqioee+yxjsc1NzfT3NxMSkoK\nQUFBLF26lH/84x8d26+55hoWL17Mjh07qK+v55FHHun0unqCnyG1tbUkJiYSFhbG2rVr+eMf/9ix\n7aqrruLNN99k9erVNDc388Mf/nBAR/D4Rdi3d+O8t+c9Lhx5oa/LMcYEkPHjx/Ptb3+bWbNmkZ6e\nzpYtW5g9ezYiwu23386XvvQlpkyZwtlnn81VV13VMa49NjaWp59+mmuuuYakpCT+9Kc/MX/+/I7n\nveSSS7j33nuZN28eeXl5nfrm4cRj5H/1q1/xwx/+kLi4OB555BGuvfbajm0TJkzgmWee4frrrycz\nM5OkpKTjuoH60ynnxhmQAkR0/nzl5pvhd83zWTBxAddNvM6nNRljjjfU58bZvn07kyZNorm5eUAP\nrA7U3DgDorQUhqW3smLvCuuvN8b4jb/+9a80NTVRVVXFAw88wJe//OUBDXpv8ouqy8rgcNinjIgf\nwbDoYb4uxxhjAPj1r39NWloao0ePJjQ0lGeffdbXJfVZt3PjDIQDB2BTjfXXG2P8y9KlS7vfKUD4\nRcs+Lg7eL7Ihl8YY01/8IuzThjeypmQNc3Pn+roUY4wZlPwi7CPzPmDisInEhcf5uhRjjBmU/CLs\nm7Osv94YY/qTX4R9Rdx71l9vjDH9yD/CPmgr52Wf5+syjDGD3OleltBzPvtTKSgoGNCzY3vCL8I+\nL/pcIkIifF2GMcb0i9zcXJYtW+bTGvwi7M9Lt/56Y8zg5Q9TTfhF2H8pz/rrjTF911+XJfzpT3/a\ncVnCF198sdO2pqYmvvOd75CTk0N6ejp33XUXjY2Nx9V20003sX//fi6//HJiY2P52c9+BsDVV19N\nRkYGCQkJzJ07l23btnnzIzleby5r1R8LoDV1Lae8LJcxxvcYgpclTEtL061bt2pdXZ0uWLCg02UJ\n77//fp0/f75WVVVpTU2NXn755frQQw+pqury5cs1Kyuro77c3NzjLj+4ePFira2t1ebmZr3//vt1\n6tSpJ3xvJ/vc6eVlCf1i1ktf12CM6V53XRGyqI/XFOxC/+P082DatGksWrSIp556imuvvZY77rgD\ngH/+85986UtforW19YQTml155ZXMmzePe++9l9tuu4309PSOC4Pv2rWLsWPHsnv3bkaOHElsbCyb\nNm3ijDPOAODDDz/khhtuYM+ePRQUFHDTTTd1XKxk5MiRvPDCC1xwwYl7MY4cOUJSUhJHjx4lNja2\n0zZvzXrpF3PjGGMCnzdCuq+WLFnCE088wd69ewHnIiI9vSzhokWL2LVrFy6Xi/r6eiZPngw4lyU8\n55xzTvjYQ4cOUV9fz/Tp0zvWqWqPR/m4XC4efvhhXnvtNQ4dOkRQUBAiwuHDh48Le2+xsDfGBLT2\nyxIuW7aMWbNmISJMmzatx5cl/MMf/sD8+fMJDg7myiuv7HRZwpM9NiUlhcjISLZt20ZGRka3NXa9\nyMlLL73EG2+8wXvvvUdOTk5Hy74/ezn84gCtMcb0VX9elvC3v/0t27dvp76+nkWLFnVsCwoK4vbb\nb+f+++/n0KFDAJSUlHR6vKe0tDQ+++yzjvu1tbWEh4eTlJREXV0dDz/8sFc/kxOxsDfGBLT+vCzh\n/fffzwUXXEBeXh4XXnhhpxb6T37yE0aPHs3MmTOJj4/noosuorCwsGO7574PPfQQP/rRj0hMTOQX\nv/gFN998Mzk5OQwfPpyJEyd2/CLpT3aA1hjTI/4wVnwoGlSXJTTGGNO/LOyNMWYIsLA3xpghwMLe\nGGOGAAt7Y4wZAizsjTFmCLAzaI0xPdbfY8FN/+lR2IvIJcCTQDDwvKr+pMv2FOAPQLr7OX+mqr8V\nkWxgCTAMUODXqvq0F+s3xgwQG2Mf2LrtxhGRYOCXwCXAeGCBiIzrstvdwHpVnQrkAz8XkRCgBfim\nqk4AZgLfOMFjA1pBQYGvSzgtVr9vBXL9gVw7BH79vdWTPvsZwG5V3auqLcDLwPwu+5QBce7bcUCF\nqraqarmqbgBQ1VpgO5DpndL9Q6D/g7H6fSuQ6w/k2iHw6++tnnTjDAeKPO4XA+d22ec3wDIRKQVi\ngWu6PomI5ALTgDV9KdQYY0zf9aRl35OOuoeBDaqaCUwFnhGRjkmZRSQGeA24z93CN8YYM5C6u5QV\nTl/7Ox73HwIe6LLP28D5HvffA8523w4F3gXuP8nzqy222GKLLb1fenNZwp5043wCjHF3w5QC1wIL\nuuyzA/gisFpE0oCxwB5xxmm9AGxT1SdP9OS9mbXNGGNM3/RoimMRuZRjQy9fUNVHReROAFV9zj30\ncjEwAqdr6FFV/aOIzAbeBzbhfBMBPKSq73j/rRhjjDkZn89nb4wxpv/5dLoEEblERHaIyC4RecCX\ntfSEiLwoIgdEZLPHuiQR+aeIFIrIP0QkwZc1noyIZIvIchHZKiJbRORe9/pAqT9CRNaIyAYR2SYi\nj7rXB0T97UQkWETWi8jf3fcDpn4R2Ssim9z1r3WvC6T6E0TkNRHZ7v43dG6g1C8iY92fe/tyVETu\n7U39Pgv7Hp6s5W8W49Tr6UHgn6qah3Ng+sEBr6pnTnaCW0DUr6qNwDz3iXuTgXnubsKAqN/DfcA2\njnVrBlL9CuSr6jRVneFeF0j1PwW8rarjcP4N7SBA6lfVne7PfRowHagH/kpv6u/N0VxvLsAsOo/y\neRB40Ff19KLuXGCzx/0dQJr7djqww9c19vB9vI5zUD3g6geigI+BCYFUP5AF/C8wD/h7oP37AT4H\nkrusC4j6gXhgzwnWB0T9XWq+GFjZ2/p92Y1zopO1hvuoltORpqoH3LcPAGm+LKYnupzgFjD1i0iQ\niGzAqXO5qm4lgOoHngC+C7g81gVS/Qr8r4h8IiK3u9cFSv0jgUMislhE1onIb0QkmsCp39N1wJ/c\nt3tcvy/DftAdGVbn69Wv35f7BLe/4JzgVuO5zd/rV1WXOt04WcAXRGRel+1+W7+I/B/goKquB044\n3Nif63c7X51uhEtxugHneG708/pDgLOAX6nqWUAdXbo8/Lx+AEQkDLgceLXrtu7q92XYlwDZHvez\ncVr3geaAiKQDiEgGcNDH9ZyUiITiBP3vVfV19+qAqb+dqh4F3sLpuwyU+s8Dviwin+O0yi4Qkd8T\nOPWjqmXuv4dw+otnEDj1FwPFqvqx+/5rOOFfHiD1t7sU+NT93wB68fn7Muw7TtZyf1tdC7zhw3r6\n6g3gFvftW3D6wv3OKU5wC5T6U9pHGohIJHARsJ4AqV9VH1bVbFUdifMzfJmq3kSA1C8iUe1ToLi7\nPy4GNhMg9atqOVAkInnuVV8EtgJ/JwDq97CAY1040JvP38cHGi4FdgK7cU628vnBj27q/RPOWcTN\nOMcb/i+QhHPQrRD4B5Dg6zpPUvtsnL7iDTghuR5nZFGg1D8JWOeufxPwXff6gKi/y3uZC7wRSPXj\n9HlvcC9b2v9/DZT63bVOwTmwvxH4H5yDtoFUfzRwGIj1WNfj+u2kKmOMGQLsGrTGGDMEWNgbY8wQ\nYGFvjDFDgIW9McYMARb2xhgzBFjYG2PMEGBhb4wxQ4CFvTHGDAH/H5y2YvMBbmjuAAAAAElFTkSu\nQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f79d824ed10>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "x = np.arange(len(adagrad_auc))\n",
    "plt.plot(x, np.array(adagrad_auc))\n",
    "plt.plot(x, np.array(adadelta_auc))\n",
    "plt.legend(['adagrad', 'adadelta'], loc='lower right')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "We can try the same for the k-OS loss."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "alpha = 1e-3\n",
    "epochs = 70\n",
    "\n",
    "adagrad_model = LightFM(no_components=30,\n",
    "                        loss='warp-kos',\n",
    "                        learning_schedule='adagrad',\n",
    "                        user_alpha=alpha, item_alpha=alpha)\n",
    "adadelta_model = LightFM(no_components=30,\n",
    "                        loss='warp-kos',\n",
    "                        learning_schedule='adadelta',\n",
    "                        user_alpha=alpha, item_alpha=alpha)\n",
    "\n",
    "adagrad_auc = []\n",
    "\n",
    "for epoch in range(epochs):\n",
    "    adagrad_model.fit_partial(train, epochs=1)\n",
    "    adagrad_auc.append(auc_score(adagrad_model, test).mean())\n",
    "    \n",
    "    \n",
    "adadelta_auc = []\n",
    "\n",
    "for epoch in range(epochs):\n",
    "    adadelta_model.fit_partial(train, epochs=1)\n",
    "    adadelta_auc.append(auc_score(adadelta_model, test).mean())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXsAAAEACAYAAABS29YJAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xd8VHW6+PHPk0YKaRBI6FUQkCaKoAjBsqJ3kbWsLO66\nul7b3sva1rXwu1fDuuq6Rddd3b1Wdu0FGxZQRGNdRZRigFASqSkkZBLS2zy/P84kmYSQOiEz5Hm/\nXt/XzDnne855ZgjP+c73lK+oKsYYY45tQd0dgDHGmK5nyd4YY3oAS/bGGNMDWLI3xpgewJK9Mcb0\nAJbsjTGmB2g12YvIPBFJF5EdInJbM8sTRGSViGwQkTQRucJr2Q0i8p1n/g0+jt0YY0wbSUvX2YtI\nMLANOAvYD3wNLFLVrV51UoBeqnqHiCR46icCxwMvACcD1cAq4DpVzeiaj2KMMeZIWmvZTwd2quou\nVa0GXgQWNKmTDcR43scAB4FaYBzwlapWqGot8DFwoc8iN8YY02atJftBwF6v6X2eed4eByaISBaw\nEbhBnZ8L3wGni0gfEYkE/gMY7JuwjTHGtEdIK8vb8iyFJcAGVU0WkVHAahGZpKrpInI/8D5QCqwH\n3J0L1xhjTEe0luz3A0O8pofgtO69nQrcA6CqGSLyPU5//TpVfQp4CkBE7gX2NN2BiNjDeYwxpgNU\nVdpat7VunHXAcSIyXETCgIXAiiZ10nFO4CIiicBYINMz3d/zOhS4AHj+CAEHbLnrrru6PQaLv/vj\n6InxB3Lsx0L87dViy15Va0RkMfAeEAw8qapbReRaz/JHgXuBZSKyEefgcauqFng2sVxE+uJcjfNf\nqnqo3REaY4zptNa6cVDVlcDKJvMe9XqfD8w/wrqzOxugMcaYzrM7aDspOTm5u0PoFIu/ewVy/IEc\nOwR+/O3V4k1VRyUAEe3uGIwxJtCICOrDE7TGGGOOAa322RvjC6qQlQXp6ZCfD4cONZSSEhCBoCAI\nDnZeVaG4uKGUlDjbiY1tXJKSYMiQhhIT03IcxvRU1o1jWlRaCmvWQHg49OvXUHr1gvJyKChwissF\nRUVOfe+yaxds3uyU0FAYPx7693eScmys8xoV5eyrthbcbudVBKKjG0rv3k6doqKGUlgIOTmwdy/s\n2eO81h0samuhpsZ5DQpy9jlwIAwY4JTISDh40Dnw1BVVJ5bevRvK6NEwZQpMngxjxkCINY+Mn2hv\nN44le3MYVfj8c1i2DF57DaZNc5JvXh4cOOAkRnCSaJ8+EB/vvMbGOsnSuwwZAhMmOKVfv66P+9Ah\n54AREuIk/uBgJ+Hn5kJ2dkMpK4OEhIbSt69Tt6TEOUiVlDjb2rYNNmyAjRudXyZjxjifMyLCKZGR\nzoFPpHHp0wfGjXPK8cc7ByxjfMmSvWkXVSeB79zplG3bYPlyJ1n+4hfws585LeGm61RWOq39nqS4\n2Pl+ioudXzVlZc5rZaXznXiXvDzYutUp27Y5B5Rhwxp+WSQlOa/9+jUcbBISnAOJtPm/r+nJLNn3\nQIWFDUll2zbYvt3p1ggKcrpO6ooIVFQ4CaruNSfHSdqjR8OoUc7ruefC9OmWdHylthZ273a6mbKz\nne+87tW7Gyk/3/k3SUpyupzqSny8sw3vUl7uHHQOHWo4p9Gvn/PLY8wYOO44GD7c6V7bv7+hZGc7\n+/HuwqqqcvbhXUJDG+/P7Ybqaqeud6moaFyCg2HECBg5sqH069f47zA01Pn1VNf9V1DgdMuFhTV0\nn0VFOdMulxNrXamudn419enjHCD79nV+WdXUNC69ezccRPv2dT6TSOOuwsrKhl+rda+qMHhwQxky\nxH9/lVmy7wFU4bvv4O23nZKWBmPHNpQxY5z/6OD8h6yudorb3dD9EB7uvPbvD3Fx3fpxjJeKCucg\nkJXllOxsJ+HVdUnVlfBw53xHTEzDOY3cXOdAX1d273aS3KBBDWXgQCf51v2SSEhwkm9hobOfulJd\n3Xh/QUFO8vUuoaENf0t1parKOU+TmemUjIyGJF1Xamqc7q+mXYBVVY270eoOQt6JPSzMOTg0PQCE\nhDSU4GDnAFi3PD/f+UwiDZ8lONg5SHifh+rf3/m/tX8/7NvnHJz37XPq1/0iqytRUc7n7dXLKZGR\nzjYSExtKZKSzvZqahs9et44vWLI/BrndTov93/92ynvvOX/U8+fDD38Is2f77g/IGNOg7jxQ3YG3\n7hdZWZlzYK6sdEppacOvg9xcp1RXO78ggoMbftFUVDgHrLqDV9++zj7Kyxu6BisrnYOJ90UMxx0H\nv/td49gs2R8DCgpg7Vr48kunfPWV08KZOdMpZ57pnPSzbhZj/FNdi77ul4T3/JKShq60gwed5XUn\n+yMinIZbaalzkKm78iwkBH7848b7sGQfYFSda88/+QQ++8xJ7Dk5cNJJcMopTpk50/lZaIwxdSzZ\n+7nKSucyvi+/dBL8J584P9lmz4ZZs2DGDOda9ODg7o7UGOPPLNn7GVV46y14/32na2bzZucE6vTp\nToI//XQYOrS7ozTGBBpL9n5k1y647jrnpM7llztdMlOnOn1zxhjTGfYgND9QWwsPPuj0uycnw7p1\ncPPNcNppluiNMd2j1WQvIvNEJF1EdojIbc0sTxCRVSKyQUTSROQKr2V3iMhmEflORJ4XkWP+AsHv\nvnNOqK5Y4VwmefvtziVXxhjTnVpM9iISDDwMzAPGA4tEZFyTaouB9ao6BUgG/iwiISIyHLgaOFFV\nJ+IMa/gTn0bvR6qrnetgzzgDrrkGPvzQuTbWGGP8QWvP8JsO7FTVXQAi8iKwANjqVScbmOR5HwMc\n9Ixdewhn7NlIEakFIoH9Pozdb6SlwRVXOHcjfvutc4u1Mcb4k9a6cQYBe72m93nmeXscmCAiWcBG\n4AYAz6Djfwb2AFlAoap+4Iug/UVNDdx3H8yd65yIXbnSEr0xxj+11rJvy2UyS4ANqposIqOA1SIy\nCUgEbgSGA0XAKyLyU1V9rukGUlJS6t8nJycHxNiQVVXwk584z9xYt855oqExxnSV1NRUUlNTO7x+\ni5deisgMIEVV53mm7wDcqnq/V513gXtU9XPP9BrgdmAE8ANVvcoz/zJghqr+d5N9BNyllxUVcPHF\nzonXl15ynnVhjDFHk68vvVwHHCciw0UkDFgIrGhSJx04y7PzRGAskAFsA2aISISIiKfOlrYG5q/K\nymDBAueu15dftkRvjAkMLXbjeE60Lgbew7ma5klV3Soi13qWPwrcCywTkY04B49bPf31BSLyNM4B\nww18CzzWdR+l65WWOk+aHDwYnnrKhqgzxgQOu4O2jTIznbtgx4yBxx6zZ9cYY7pXe7txelzb9ONd\nH/P29repcddQ7a52XmurKakuoaiiiKLKIooqiiiuKkZVqa0JovhQEOVlQQw4qy/X/ugmkIU4P3SM\nMSYw9KiW/d+//ju//fi3XH/K9USERBASFEJocCghQSFEhUYRGx5LbK9YYsNj2b09hof/FsS6b9xc\n+Z9uLv2pm+yq7dz9yd0UlBdw5+w7uWTCJQQHtT/pp+en8/Lml7l22rUk9rZnFxtj2s8ehNaMGncN\nN793Mx9kfsBbi95iVJ9RLdZ//nm46Sb4f/8Prrqq8fNsVJUPMj/grtS7KKwoJCU5hYvHX0yQtP6Y\noT1Fe1iaupQV21dw1siz+GT3Jzz9o6c5c+SZnf2IxpgexpJ9E0UVRSxcvhBFefnil4kNj22x/hNP\nwF13OY8knjDhyPVUlfcz3ud/P/pfatw13HPGPcwbPQ9pZviovNI87vvsPv618V9cN+06fnPab4gL\nj2N1xmouf+Nyrpx6JSnJKYQEdb5XrdZdS2l1KRU1FYQGhRIWHEavkF4+2bYxxn9YsveSU5LDmU+f\nyRnDz+DBeQ+2mvAeeggeeAA++KDtz7VRVV5Pf53/+fB/SIhM4L4z72N0n9F8tuczPt3zKZ/s/oTt\nB7dzxZQr+J/Z/0NS76TDYvz56z+nvKach+Y9RE5JDlvytrA5bzObD2ymvKacUfGjnNJnFCPjR1JW\nXUZGQQYZLqdkujIpqiiitLqUqtoqokKjCA8Jp9pdTWVNJZW1lQAMix3GL6b8giunXsmgmKY3Qju/\ngNIOpAGQGJVIv6h+zX5nVbVVqCq9Qo7559oZ47cs2Xu5asVVRIdF8+C8B1ute9998OSTsGZNx+6G\nrXXX8symZ0hJTaGwopDThp7G6UNPZ/aw2UwbMK3FxOhWN/d/dj+PffsYo/uMZkK/CU7pP4HI0MiG\nxF6QQWZhJhEhEYzuM7r+ADAibgR9IvoQFRZFREhEs78uatw1bMrdxOPfPM5Lm1/i9GGnc82J19A/\nqj+pu1L5ePfHfLbnMwZEDyA0KJQDpQc4WH6QuPA4EiITqKipoLiymOKqYtzqJio0ihtOuYFbTr2F\n6F7R7f/CjDGdYsneY2veVub8cw7bf7WduPC4FuvefTe8+KLToh8woHP7rXXXAnToxO3RUlJVwktp\nL/HE+icoqSoheVgyc4bPYfaw2fSP6l9fr9Zdy8Hyg+SX5RMeEk50WDTRvaLpFdyL3UW7ufOjO3k/\n432WnL6Ea6dday19Y44iS/YeF750ITMHz+Q3p/2mxXqrVsHVV8M330D//i1WNc3YlLuJJWuWsDlv\nMxcefyEFFQUcKD1QX0b3Gc0l4y/hwnEX0i+qX3eHa8wxw5I98OW+L/nxKz9m++LtRIRGHLFeVhZM\nm+Y832b2bJ+G0ON8uvtTPt/7Of2j+teXhMgENuRs4JUtr7Byx0pOGngSF467kPjweMqqy+pLVW0V\nfSP70j+qP4lRifSP6s+Q2CFEhtqwXsYcSY9P9qrK3H/N5bJJl/GfJ/7nEevV1sJZZzmPJ77zTp/t\n3hxBeXU5K3euZMW2FVTWVhIZEklkqFNCgkI4WH6QA6UHyC3NdV5Lcjl71NlcesKlnHfceS0etI3p\niXp8sl+5YyU3v38z3/3yuxavvlm6FD7+GFavtkcf+CNXuYvXtr7G82nP8232t5w/9nxOHXwqkaGR\nRIRGEBESQWRoJCcPOpneYb27O1xjjroenezd6ubER0/krjl3ccG4C45YLzUVFi1yRpXq7AlZ0/Wy\ni7N5efPLpB1Io7ymnPKacipqKiiqKGL7we0snr6YX03/FfER8d0dqjFHTY9O9s9teo6Hv36YL678\notnLDwHy8mDqVOcyy3PO8cluTTfalr+N+z+/nze3vclVU6/i5pk32yMoTI/QY5N9dW01Yx8ey7IF\ny5gzfE6zdVSdZ9GPGwf3399sFROgdhfu5k9f/ImnNz3NwOiBjIp3bkAbGT+SEXEjGBI7hMExg0mI\nTGjToy2M8Xc9Ntm/u+Ndfvvxb/nyqi+PWOfJJ+Hhh+Grr2zQkWNVWXUZma7M+pJRkMH3hd+z79A+\n9hfvp7iymIHRAzl50MncNecuxvcb390hG9MhPfYRx8999xw/m/SzIy7PzITbb4ePPrJEfyyLDI3k\nhP4ncEL/E5pdXl5dTlZxFq+nv07yP5P5jzH/QcqcFIbF2SDC5tjW6u9ZEZknIukiskNEbmtmeYKI\nrBKRDSKSJiJXeOaPFZH1XqVIRK7vgs9ASVUJ72x/h4UTFja7vLbWGXjk9tvhhOZzgOkhIkIjGNVn\nFLecegs7frWDwdGDOfGxE7lx1Y3sP7S/u8Mzpsu0NuB4MM5YsmcB+4GvgUWqutWrTgrQS1XvEJEE\nT/1EVa3xqhPkWX+6qu5tso9Od+M8t+k5nk97nncufafZ5X/8I7zzDnz4IQRZd61pIrckl3s/vZdn\nNj3DSQNP4ueTf84Fx19AVFhUi+vVumv5bM9nrN2/lvCQ8Pr7BiJDIxkeN5yJiRPt/IDpMr7uxpkO\n7FTVXZ6NvwgsALZ61ckGJnnexwAHvRO9x1lARtNE7ystdeFs2gR/+AN8/bUletO8xN6JPHTuQ/z+\nrN+zYtsKnt70NIvfXcz5Y8/n5IEnMyxuGMNihzEsbhiRoZF8+P2HvLb1Nd7c9iaDogcxe9hsat21\nzh3BNWWUVpWSnp9OYUUhZ448k7NHns3ZI89mSOyQLom/1l1LTkkObnXX34cQHhJuBxrTSGst+4uB\nc1T1as/0z4BTVPVXXnWCgA+BMUA0cImqrmyynaeAdar692b20amW/YHSA4z52xj237z/sJZYZSVM\nnw433gi/+EWHd2F6oNySXF7Z8gqbD2xmz6E97C7cze6i3VTWVHLyoJO58PgLuWDcBYyMH3nEbewu\n3M3qzNV8kPkBH2R+QGl1af1IaDG9YogPj2fOsDksOH4BE/pNOOLlwnVq3DVszNnIJ7s/YXPeZnYV\n7mJX4S72HtpLfHg8IUEhlNeUU1ZdRmVNJeEh4USERjT6xdE7rLfzOIvIhsda1D0xNSo0iqiwKHqH\n9WZo7FBiesX4+ms1PuTTq3FE5CJgXivJ/n+ABFW9UURGAauByapa7FkehtOFM15V85rZh9511131\n08nJySQnJ7c1fv721d9Ym7WWZy545rBljzwCb78N774Lrfw/MqZVqkplrZNEO7JuWXUZhyoPUVRZ\nxKHKQxwoPcDqjNW8se0NQoNCWTB2AWeNPIvgoGAqaiqorKmkoqaCvYf28umeT/li7xcMjhnM7KGz\nmZI0hRHxIxgeN5yhsUMPi8mtbsqry+uTf10priwmryyv/kF1uSW5FFQUUFbt/CIprS6lpKqE3YW7\nGRg9kGkDpzFtwDROGngSMwfPtCebdqPU1FRSU1Prp5cuXerTZD8DSFHVeZ7pOwC3qt7vVedd4B5V\n/dwzvQa4TVXXeaYXAL+s20Yz++hUy37GEzNISU5h3ujGm6+theOPh2XLYNasDm/emC6nqmzI2cCb\n297k0z2fEizBhIeE15f+Uf2ZNXQWs4bOIiEy4ajEVOOuIT0/nW+yvuGb7G9Yu38t6fnpzBs9jwvH\nXci5o89tdhwDVW31F4rxDV+37ENwTrieCWQBazn8BO0DQJGqLhWRROAbYJKqFniWvwisVNV/HWEf\nHU72Owt2ctpTp7H/5v2HPQfnzTfhnnuca+rtb8+YzsstyeXNbW/yevrrfL7nc6YNnEaNuwZXuQtX\nhQtXuYuI0AguGX8JP5v0M04dcqol/i7k85uqRORc4C9AMPCkqt4nItcCqOqjnitwlgFDcS7lvE9V\nn/esGwXsBkbUdes0s/0OJ/vffvxb8svy+eu5fz1s2Zw58F//BQubvxrTGNMJRRVFfLH3CyJCI+gT\n0Yf48HjiI+LJL8vnhe9e4JlNz1BRU8GlEy9lxuAZCILS8P98VPwoxiaMtZPIndBj7qBVVcY+PJZn\nL3yW6YOmN1q2bh1cdBFkZEDIMXPbmDGBo65r6rnvnmNL3pb6Fr4guNVNen46rgoXpww6hZmDZzJt\n4DTc6qagvABXuYuC8gJq3DVMHzSd04ae1mgEtSPJK81jQ84G0g6kMaH/BOYOn0tocGib4s0vy+fZ\nTc+SV5rHOaPPYebgmW1et7v0mGT/9f6vufS1S9m+ePthPxUvvdQZlOTXv/ZVlMYYX8styeXLfV/y\n733/Zn3OesKCw+p/JfSJ6AM4AxF9sfcLknonMWvoLMb2HUtVbRUVNRX1ZVfRLtZnr6ekqoQpSVOY\n0G8C32R/w46CHZw/9nwuGncRZ488+7CTy2518+H3H/LEt0+waucq5o+dz9CYoazKWEWmK5MzR5zJ\nuaPP5fRhpzMqfpTfDTXaY5L9jatuJC48jpTklEbz9+6FKVOcxyPExvooSGNMt6l115J2II3P9nxG\npiuT8JBweoX0cl6DezE4ZjBTB0xlRNyIRg2/vUV7eT39dV7d+irrstYRHRbd6DLU3NJc4sPjufrE\nq7l04qWNHpGdU5LDezvfY+XOlazdv5a8sjwm9p/I5MTJTE6azOCYwfSL7EdCZAL9ovoRHRZ91M9P\n9JhkP+ZvY3j1kleZmDix0fxbb4WaGnjgAV9FaIwJdKVVpRyqPNToMtSosKg23d8AUFhRyKbcTWzM\n2cim3E1klWSRV5pHXlkeeaV5uNXNiPgRjIofxaj4UYzuM5oTB5zonK/oooNAj0j2Ne4aet/bm6Lb\nixr9NCsuhhEjnD774cN9HKgxxhxBcWWx85RVVwYZBRlkuDL4ePfHqCpXnXgVl0++nH5R/Xy6zx7x\n1Ms9RXsYED3gsD64p56CM8+0RG+MObqie0UzOcnp4qmjqnyx9wse//ZxjvvbcZw96mwuHncxMwbP\nYGjs0Da3+CtrKtlZsJMJ/Sd0KsaATPY7C3YyKn5Uo3m1tfDQQ/DCC90UlDHGeBERTht6GqcNPY2i\niiJeSHuBF9Je4IZVNyAizBg8g1MGncLxCcfXP3spPjweEWFv0V5W7lzJuzve5aNdH5E8PJk3f/Jm\n5+IJxG6cv3/9dzblbuL/fvh/9fM++QR+9SvYuNHXERpjjO+oKnuK9vDlvi9Zu38tOwp2sKtwF7uL\nduNWN30j+lJSVcI5o8/hvNHncc7oc5q9c7pHdOPsLNjJ6D6jG8179VX48Y+7KSBjjGkjEXGepBo3\njIUnNL7rs7CikAOlB7rkUs+ATPYZrgxmD5tdP+12O8l+9epuDMoYYzopLjyOuPC4Ltl2QN6r3LTP\nfu1aiIlxBhI3xhhzuIBL9m51k+nKbPQc8VdfhYsv7sagjDHGzwVcN87+Q/vrB1sAUIXly+GNN7o5\nMGOM8WMB17LPcGU06sJZvx6Cg2HSpBZWMsaYHi7gkn3TK3GWL3e6cOyx2cYYc2QBnexVnf76iy7q\n5qCMMcbPBXSy37wZKirgpJO6OShjjPFzrSZ7EZknIukiskNEbmtmeYKIrBKRDSKSJiJXeC2LE5Hl\nIrJVRLZ4xrTtFO8+++XLnVa9deEYY0zLWkz2IhIMPAzMA8YDi0Sk6dXsi4H1qjoFSAb+7Bm7FuAh\n4F1VHQdMArbSCarqXGPfx0n21oVjjDFt01rLfjqwU1V3qWo18CKwoEmdbCDG8z4GOKiqNSISC5yu\nqk8BqGqNqhZ1JtgDpQcIDwknLjyObdvg4EGYObMzWzTGmJ6htWQ/CNjrNb3PM8/b48AEEckCNgI3\neOaPAPJEZJmIfCsij4tIZGeC9e6vf/VVuPBCCAq4sw7GGHP0tXZTVVseR7kE2KCqySIyClgtIpM9\n2z4RWKyqX4vIX4DbgTubbiAlJaX+fXJyMsnJyc3uyLu//s034b772hCdMcYcA1JTU0lNTe3w+i0+\n4thzQjVFVed5pu8A3Kp6v1edd4F7VPVzz/Qa4DacXwH/VtURnvmzgNtV9YdN9tHmRxzf+dGdBEkQ\n/3t6CtHRcOAA9O7djk9rjDHHiPY+4ri1TpB1wHEiMlxEwoCFwIomddKBszw7TwTGApmqmgPsFZEx\nnnpnAZvbGlhz6rpxMjIgKckSvTHGtFWL3TieE62LgfeAYOBJVd0qItd6lj8K3AssE5GNOAePW1W1\nwLOJXwHPeQ4UGcAvOhNsXbJPWwsTOjdClzHG9CitPghNVVcCK5vMe9TrfT4w/wjrbgRO7mSM9er6\n7FdvhhNO8NVWjTHm2Bcw17IUlBdQ464hITKBtDRr2RtjTHsETLLPKMhgdJ/RiAibrWVvjDHtEjDJ\nvq6/vqoKdu6E44/v7oiMMSZwBEyyr+uv37EDhg2D8PDujsgYYwJHwCT7+itxrL/eGGPaLeCSvfXX\nG2NM+wVcsreWvTHGtF9AJPuSqhKKq4oZ0HuAteyNMaYDAiLZZxRkMDJ+JJWVwu7dcNxx3R2RMcYE\nloBI9nVdOOnpMHo0hIV1d0TGGBNYAiLZZ7gyGB1v/fXGGNNRAZHsM12ZjIwfaf31xhjTQQGR7PPL\n8u2ZOMYY0wkBkewLKwqJj4i3lr0xxnRQQCR7V4WLXhpPTg6MGtXd0RhjTOBpNdmLyDwRSReRHSJy\nWzPLE0RklYhsEJE0EbnCa9kuEdkkIutFZG1Hg3SVu8jfG8fYsRAc3NGtGGNMz9Xi4CUiEgw8jDOk\n4H7gaxFZoapbvaotBtar6h0ikgBsE5FnVbUGZ8DyZK+RqzqksKKQfTvjrQvHGGM6qLWW/XRgp6ru\nUtVq4EVgQZM62UCM530McNCT6Ou0eUDc5rjVzaHKQ3yfHmsnZ40xpoNaS/aDgL1e0/s887w9DkwQ\nkSxgI3CD1zIFPhCRdSJydUcCLK4sJiosii1pwdayN8aYDmptDFptwzaWABtUNVlERgGrRWSyqhYD\np6lqtoj088xPV9VP2xOgq8JFfLhzJY617I0xpmNaS/b7gSFe00NwWvfeTgXuAVDVDBH5HhgLrFPV\nbM/8PBF5Hadb6LBkn5KSUv8+OTmZ5OTk+unCikKiQ+P43uUMWmKMMT1RamoqqampHV5fVI/ceBeR\nEGAbcCaQBawFFnmfoBWRB4AiVV0qIonAN8AkoAIIVtViEYkC3geWqur7TfahLcXw0fcf8eu3fkvI\nsx+xtsPX8xhjzLFFRFDVNp8TbbFlr6o1IrIYeA8IBp5U1a0icq1n+aPAvcAyEdmIcw7gVlUtEJGR\nwGsiUref55om+rZwVbjQsjjrrzfGmE5orRsHVV0JrGwy71Gv9/nA/GbWywSmdDbAwopCqg7FW3+9\nMcZ0gt/fQesqd1FZFMfw4d0diTHGBC6/T/aFFYVUFMYzcGB3R2KMMYHL75O9q8JFaX48AwZ0dyTG\nGBO4/D7ZF5S7KM6Ls2RvjDGd4PfJPu9QIREST69e3R2JMcYELr9P9gdKXPSLjuvuMIwxJqD5fbJ3\nlRWSFBvf3WEYY0xA8/tkf6jKxeAES/bGGNMZfp/sS90uhva3bhxjjOkMv072FTUVqMKwgRHdHYox\nxgQ0v072rnIXITV22aUxxnSWXyf7wopCpNLunjXGmM7y62TvqnDhLrOWvTHGdJZ/J/vyQqoP2aMS\njDGms/w62e/NcxFaE094eHdHYowxgc2vk/3uAy6iQuyyS2OM6axWk72IzBORdBHZISK3NbM8QURW\nicgGEUkTkSuaLA8WkfUi8lZ7g8sqKCQ2zG6oMsaYzmox2YtIMPAwMA8YDywSkXFNqi0G1qvqFCAZ\n+LNn7NpbO9vfAAAWzElEQVQ6NwBbgCMPNHsEOUUu+kZZy94YYzqrtZb9dGCnqu5S1WrgRWBBkzrZ\nQIznfQxwUFVrAERkMHAe8ATQ5oFx6xwsKaRftLXsjTGms1pL9oOAvV7T+zzzvD0OTBCRLGAjTku+\nzoPAbwB3R4JzVbgYEG8te2OM6azWkn1bul6WABtUdSDOAOOPiEi0iPwQOKCq6+lAqx7gULWLIfYQ\nNGOM6bSQVpbvB4Z4TQ/Bad17OxW4B0BVM0Tke+B4z/zzReQ8IByIEZGnVfXnTXeSkpJS/z45OZnk\n5GQAytyFDE+0ZG+MMampqaSmpnZ4fVE9cuPdc6J1G3AmkAWsBRap6lavOg8ARaq6VEQSgW+ASapa\n4FVnDnCLqs5vZh96pBhCbhnOmss+Ys7kER36cMYYc6wSEVS1zb0mLbbsVbVGRBYD7wHBwJOqulVE\nrvUsfxS4F1gmIhtxuoVu9U703ptra1DOtqE2tJAxQ61lb4wxndViy/6oBHCElv2BvFoSH+5FbUoV\nQeLX934ZY8xR196Wvd9m0e17igiqibZEb4wxPuC3mTRjfyFhtdaFY4wxvuC3yf77bBeRQXaNvTHG\n+ILfJvu9+YX0DrGWvTHG+ILfJvusAhdx4dayN8YYX/DbZH/gkIu+kdayN8YYX/DbZJ9fWkhirLXs\njTHGF/w22RdVuhgYby17Y4zxBb9M9qpQXFPIkH6W7I0xxhf8Mtm7XBAc6aJ/jHXjGGOML/hlss/K\ngrDYQuLDrWVvjDG+4JfJPjsbgqPs0ktjjPEVv0z2WVmg4S7iI6xlb4wxvuCXyT4723m8sXXjGGOM\nb/hlst+fpVSKdeMYY4yv+GWy35dTTrCE0CukV3eHYowxx4RWk72IzBORdBHZISK3NbM8QURWicgG\nEUkTkSs888NF5CvP/C0icl9bg9qb7yI61Fr1xhjjKy0mexEJBh4G5gHjgUUiMq5JtcXAelWdAiQD\nfxaREFWtAOZ65k8C5orIrLYElVNoJ2eNMcaXWmvZTwd2quouVa0GXgQWNKmTDcR43scAB1W1BkBV\nyzzzw3DGsG1ubNpGVOHAoUISoqxlb4wxvtJash8E7PWa3ueZ5+1xYIKIZAEbgRvqFohIkIhsAHKB\nj1R1S2sBFRZCSLSLvlHWsjfGGF8JaWV5W0YjXwJsUNVkERkFrBaRyaparKpuYIqIxALviUiyqqY2\n3UBKSkr9+2HDkolLKrRuHGOM8ZKamkpqamqH128t2e8HhnhND8Fp3Xs7FbgHQFUzROR7YCywrq6C\nqhaJyDvAScBh0Xon+w8+gN4JfyWul3XjGGNMneTkZJKTk+unly5d2q71W+vGWQccJyLDRSQMWAis\naFInHTgLQEQScRJ9pucqnTjP/AjgbGB9awFlZUF4nJ2gNcYYX2qxZa+qNSKyGHgP5wTrk6q6VUSu\n9Sx/FLgXWCYiG3EOHreqaoGITAT+JSJBnvnPqOqa1gLKyYHQmELiwoe0VtUYY0wbtdaNg6quBFY2\nmfeo1/t8YH4z630HnNjegHJzQeJdxIdPau+qxhhjjsDv7qDNyQHtZSdojTHGl/wu2efmQnWIPRfH\nGGN8qdVunKMtJweqsSdeGmOML/llsg93W8veGGN8ya+SfXU1FBVBZZVdemmMMb7kV332Bw5A3341\nlFeX0zusd3eHY4wxxwy/atnn5kK/IYVUh8cSJH51HDLGmIDmV8k+JwfiBhRSbidnjTHGp/wq2efm\nQmx/F+V2ctYYY3zKr/pKcnIgsq+dnDXGGF/zq5Z9Tg5EDCpErWVvjDE+5Vct+9xcCO5dYDdUGWOM\nj/lVss/JAXdELolRid0dijHGHFP8Ktnn5kJFaC6JvS3ZG2OML/lVss/JgVK1lr0xxvia3yT7ykoo\nLQVXtbXsjTHG19qU7EVknoiki8gOEbmtmeUJIrJKRDaISJqIXOGZP0REPhKRzZ751x9pH7m50L8/\n5JbmktQ7qcMfyBhjzOFaTfYiEgw8DMwDxgOLRGRck2qLgfWqOgVIBv4sIiFANXCTqk4AZgD/3cy6\ngJPsk5Igt8S6cYwxxtfa0rKfDuxU1V2qWg28CCxoUicbiPG8jwEOqmqNquao6gYAVS0BtgIDm9tJ\nTg4kDCijqraKmF4xzVUxxhjTQW25qWoQsNdreh9wSpM6jwMfikgWEA1c0nQjIjIcmAp81dxOcnIg\nZoDTXy8ibQjLGGNMW7Ul2Wsb6iwBNqhqsoiMAlaLyGRVLQYQkd7AcuAGTwu/kZSUFD75BApq9hER\nE9Ge+I0xpkdITU0lNTW1w+uLasu5XERmACmqOs8zfQfgVtX7veq8C9yjqp97ptcAt6nqOhEJBd4G\nVqrqX5rZvqoqixdD5fA3yRn0BG8teqvDH8gYY3oCEUFV29wN0paW/TrgOE83TBawEFjUpE46cBbw\nuYgkAmOBTHH6Y54EtjSX6L3l5kL8RDs5a4y/su7V7tNao7wtWk32qlojIouB94Bg4ElV3Soi13qW\nPwrcCywTkY04J31vVdUCEZkF/AzYJCLrPZu8Q1VXNd1PTg7EhOeSZMneGL/li6Rj2sdXB9k2PfVS\nVVcCK5vMe9TrfT4wv5n1PqON1/Ln5sLQ0FwSe49pS3VjjDHt4Dd30NY9KsFuqDLGGN/zi2RfVgZV\nVVBQZX32xhjTFfwi2dffPVtqz8Uxxpiu4BfJPicHEhMhpyTHWvbGmC6za9cugoKCcLvd3R0KAMOH\nD2fNmjVHZV9+kexzc6HfgAoqaiqIsyEJjTE9hIgctUta/SLZ5+RA9IBc+kf1t2t5jTEByV9+LRyJ\nXyT73FyI7GcnZ40xHfP73/+e0aNHExMTw4QJE3jjjTcAqK2t5ZZbbqFfv36MGjWKd955p9F6y5Yt\nY/z48cTExDBq1Cgee+yxRsv/8Ic/MHDgQAYPHswTTzxBUFAQmZmZAFxxxRX88pe/5LzzzqN3796k\npqbyzjvvMHXqVGJjYxk6dChLly5ttL1nnnmGYcOGkZCQwL333tuF30gzVLVbC6DXXad67Z9X6HnP\nnafGGP/kpAv/9Morr2h2draqqr700ksaFRWl2dnZ+o9//EOPP/543bdvnxYUFGhycrIGBQVpbW2t\nqqq+8847mpmZqaqqH3/8sUZGRuq3336rqqorV67UpKQk3bJli5aVlelPf/pTFRHNyMhQVdXLL79c\nY2Nj9YsvvlBV1YqKCk1NTdW0tDRVVd20aZMmJibqG2+8oaqqmzdv1t69e+unn36qlZWVevPNN2tI\nSIiuWbOmxc92pO/dM7/NudYvWvY5OSDR1rI3JpCJ+KZ0xMUXX0xSknOPziWXXMJxxx3H2rVreeWV\nV7jpppsYNGgQ8fHxLFmypNFdwOeddx4jRowAYPbs2fzgBz/g008/BeDll1/myiuvZNy4cURERBzW\nSgf40Y9+xMyZMwHo1asXc+bMYcKECQBMnDiRn/zkJ3z88ccALF++nPnz5zNr1izCwsK4++67CQo6\neinYL5J9bi7URliyNyaQqfqmdMTTTz/N1KlTiY+PJz4+nrS0NPLz88nKymLIkCH19YYOHdpovZUr\nVzJjxgz69u1LfHw87777LgcPHgQgOzu70bqDBw9utK6INFoO8NVXXzF37lz69+9PXFwcjz76aP32\nsrKyGm0jMjKSvn37duwDd4BfJPucHKgIsbtnjTHtt3v3bq655hoeeeQRCgoKcLlcnHDCCagqAwYM\nYM+ePfV1vd9XVlZy0UUXceutt3LgwAFcLhfnnXdefct/wIAB7N3bMJSH9/sjufTSS/nRj37Evn37\nKCws5Lrrrqvf3sCBAxtto6ysrP5AcDT4RbLPzXUelWA3VBlj2qu0tBQRISEhAbfbzbJly0hLSwOc\nLp2//vWv7N+/H5fLxe9///v69aqqqqiqqiIhIYGgoCBWrlzJ+++/X7/8kksuYdmyZaSnp1NWVsbd\nd9/daL/azM+QkpIS4uPjCQsLY+3atTz//PP1yy666CLefvttPv/8c6qqqrjzzjuP6hU8fpHsAQ5W\nWjeOMab9xo8fz69//WtmzpxJUlISaWlpzJo1CxHh6quv5pxzzmHy5MmcdNJJXHTRRfWXd0dHR/PX\nv/6VSy65hD59+vDCCy+wYEHDiKvz5s3j+uuvZ+7cuYwZM6ZR3zw0f4383//+d+68805iYmK4++67\nWbhwYf2yCRMm8Mgjj3DppZcycOBA+vTpc1g3UFdqdfCSLg9AREeOVEJvOp7XFr7G+H7juzUeY0zz\nPINldHcY3Wbr1q1MnDiRqqqqo3pi9Ujfe3sHL/GLln1ioue5ONayN8b4kddff53KykpcLhe33XYb\n559//lFN9L7kF1H3G1BJaVUp8RHx3R2KMcbUe+yxx0hMTGT06NGEhobyj3/8o7tD6rA2DV4iIvOA\nv+CMVPWEeo0/61meADwLJHm2+SdV/adn2VPAfwAHVHVic9uPHXCAflH9CBK/OPYYYwzgXJp5rGg1\nu4pIMPAwMA8YDywSkXFNqi0G1qvqFCAZ+LOI1B1IlnnWPaJwe1SCMcZ0qbY0pacDO1V1l6pWAy8C\nC5rUyQZiPO9jgIOqWgOgqp8CrpZ2EBZnl10aY0xXaks3ziDA+26CfcApTeo8DnwoIllANHBJu6Lo\nbS17Y4zpSm1J9m251moJsEFVk0VkFLBaRCaranFbgvjy3ecIj6okZUMKycnJJCcnt2U1Y4zpMVJT\nU0lNTe3w+m1J9vsB7yv/h+C07r2dCtwDoKoZIvI9MBZY15YgJi6cyKQhw7hp5k1tqW6MMT1O04Zw\ncw9ma0lb+uzXAceJyHARCQMWAiua1EkHzgIQkUScRJ/Z1iBKNMf67I0xXa6zwxJ6P8++JampqUf1\n7ti2aDXZe060LgbeA7YAL6nqVhG5VkSu9VS7FzhJRDYCHwC3qmoBgIi8AHwBjBGRvSLyi6b7yK+w\nPntjzLFr+PDhfPjhh90aQ5uus1fVlcDKJvMe9XqfD8w/wrqLWtt+boldjWOMOXb5w6Mm/OIuJntU\ngjGmM7pqWMI//vGP9cMSPvXUU42WVVZWcssttzBs2DCSkpL45S9/SUVFxWGxXXbZZezZs4f58+cT\nHR3Nn/70JwB+/OMfM2DAAOLi4pgzZw5btmzx5VdyuPYMa9UVBdCQ34Zorbu2xaG5jDHdix44LGFi\nYqJu3rxZS0tLddGiRY2GJbzxxht1wYIF6nK5tLi4WOfPn6933HGHqqp+9NFHOnjw4Pr4hg8fftjw\ng8uWLdOSkhKtqqrSG2+8UadMmdLsZzvS9047hyX0i6deJv0piexfZ3drHMaYlrXWFSFLOzimYBN6\nV+dz0tSpU1m6dCkPPfQQCxcu5JprrgFg9erVnHPOOdTU1DT7QLMLLriAuXPncv3113PllVeSlJRU\nPzD4jh07GDt2LDt37mTEiBFER0ezadMmRo4cCcC///1vfvrTn5KZmUlqaiqXXXZZ/WAlI0aM4Mkn\nn+SMM85oNt7CwkL69OlDUVER0dHRjZb56qmXbeqz72rWhWNM4PNFku6op59+mgcffJBdu3YBziAi\nbR2WcOnSpezYsQO3201ZWRmTJk0CnGEJTz755GbXzcvLo6ysjGnTptXPU9U2X+XjdrtZsmQJy5cv\nJy8vj6CgIESE/Pz8w5K9r/hFsrfhCI0xHVU3LOGHH37IzJkzERGmTp3a5mEJn332WRYsWEBwcDAX\nXHBBo2EJj7RuQkICERERbNmyhQEDBrQaY9NBTp577jlWrFjBmjVrGDZsWH3Lvit7WvziBK1diWOM\n6aiuHJbwn//8J1u3bqWsrKzRTUxBQUFcffXV3HjjjeTl5QGwf//+Rut7S0xMJCMjo366pKSEXr16\n0adPH0pLS1myZIlPv5Pm+Eeyt24cY0wHdeWwhDfeeCNnnHEGY8aM4cwzz2zUQr///vsZPXo0M2bM\nIDY2lrPPPpvt27fXL/eue8cdd/C73/2O+Ph4HnjgAX7+858zbNgwBg0axAknnFD/i6Qr+cUJ2j99\n/id+feqvuzUOY0zL/OFa8Z7o2BqW0LpxjDGmS/lHsrduHGOM6VL+keytZW+MMV3KP5K9teyNMaZL\n+cUJ2praGoKDgrs1DmNMy+wEbfc4pk7QWqI3xpiu5Rd30BpjAkNXXwtuuk6rLXsRmSci6SKyQ0Ru\na2Z5goisEpENIpImIle0dV1jTOBozxMWrfj86cCd1mKyF5Fg4GFgHjAeWCQi45pUWwysV9UpQDLw\nZxEJaeO6Aa8zAwD7A4u/ewVy/IEcOwR+/O3VWst+OrBTVXepajXwIrCgSZ1sIMbzPgY4qM5Qhm1Z\nN+AF+h+Mxd+9Ajn+QI4dAj/+9mot2Q8C9npN7/PM8/Y4MEFEsoCNwA3tWNcYY8xR0Fqyb0tn0RJg\ng6oOBKYAj4hI1zyQ2RhjTMe0clJgBrDKa/oO4LYmdd4FTvOaXgOc1JZ1PfPVihUrVqy0v7TnJG9r\nl16uA44TkeFAFrAQWNSkTjpwFvC5iCQCY4FM4FAb1m3XTQHGGGM6psVkr6o1IrIYeA8IBp5U1a0i\ncq1n+aPAvcAyEdmI0y10q6oWADS3btd9FGOMMUfS7Y9LMMYY0/W69XEJgXbTlYg8JSK5IvKd17w+\nIrJaRLaLyPsiEtedMR6JiAwRkY9EZLPn5rfrPfMDJf5wEfnKc/PeFhG5zzM/IOKvIyLBIrJeRN7y\nTAdM/CKyS0Q2eeJf65kXSPHHichyEdnq+Rs6JVDiF5Gxnu+9rhSJyPXtib/bkn2A3nS1DCdeb7cD\nq1V1DM7J6duPelRtUw3cpKoTcE6e/7fn+w6I+FW1ApjruXlvEjBXRGYRIPF7uQHYgnOCDQIrfgWS\nVXWqqk73zAuk+B8C3lXVcTh/Q+kESPyqus3zvU8FpgFlwOu0J/5uvP13Jo2v1rkduL27b0tuQ9zD\nge+8ptOBRM/7JCC9u2Ns4+d4A+fEesDFD0QCXwMTAil+YDDwATAXeCvQ/n6A74G+TeYFRPxALJDZ\nzPyAiL9JzD8APm1v/N3ZjXOs3HSVqKq5nve5gN8/nN9zhdRU4CsCKH4RCRKRDThxfqSqmwmg+IEH\ngd8Abq95gRS/Ah+IyDoRudozL1DiHwHkicgyEflWRB4XkSgCJ35vPwFe8Lxvc/zdmeyPuTPD6hxe\n/fpziUhv4FXgBlUt9l7m7/GrqludbpzBwGwRmdtkud/GLyI/BA6o6nqg2cuN/Tl+j9PU6UY4F6cb\n8HTvhX4efwhwIvB3VT0RKKVJl4efxw+AiIQB84FXmi5rLf7uTPb7gSFe00NwWveBJldEkgBEZABw\noJvjOSIRCcVJ9M+o6hue2QETfx1VLQLewem7DJT4TwXOF5HvcVplZ4jIMwRO/Khqtuc1D6e/eDqB\nE/8+YJ+qfu2ZXo6T/HMCJP465wLfeP4NoB3ff3cm+/obtjxHq4XAim6Mp6NWAJd73l+O0xfud0RE\ngCeBLar6F69FgRJ/Qt2VBiISAZwNrCdA4lfVJao6RFVH4PwM/1BVLyNA4heRyLrHoHi6P34AfEeA\nxK+qOcBeERnjmXUWsBl4iwCI38siGrpwoD3ffzefaDgX2AbsBO7o7hMfbYj3BZy7gatwzjf8AuiD\nc9JtO/A+ENfdcR4h9lk4fcUbcJLkepwriwIl/onAt574NwG/8cwPiPibfJY5wIpAih+nz3uDp6TV\n/X8NlPg9sU7GObG/EXgN56RtIMUfBeQD0V7z2hy/3VRljDE9gF+MQWuMMaZrWbI3xpgewJK9Mcb0\nAJbsjTGmB7Bkb4wxPYAle2OM6QEs2RtjTA9gyd4YY3qA/w8PPvm/H0dhOgAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f79d824e950>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "x = np.arange(len(adagrad_auc))\n",
    "plt.plot(x, np.array(adagrad_auc))\n",
    "plt.plot(x, np.array(adadelta_auc))\n",
    "plt.legend(['adagrad', 'adadelta'], loc='lower right')\n",
    "plt.show()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}