Part_1_Convolutional_Neural_Networks.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# ISMIR 2018 Tutorial\n",
    "# Deep Learning for Music Information Retrieval\n",
    "\n",
    "## Part 1: Convolutional Neural Networks for Instrumental, Genre and Mood Recognition\n",
    "\n",
    "Author: Thomas Lidy\n",
    "\n",
    "This tutorial shows how different Convolutional Neural Network architectures are used for:\n",
    "* Instrumental vs. Vocal Detection:  detecting whether a piece of music is instrumental or contains vocals\n",
    "* Genre Classification\n",
    "* Mood Recognition\n",
    "\n",
    "The data set used is a subset of the [MagnaTagATune Dataset](http://mirg.city.ac.uk/codeapps/the-magnatagatune-dataset) with only 1 sample excerpt of each of the original audio files.\n",
    "\n",
    "The annotations of the original dataset contain a multitude of tags, which were preprocessed in Part 0 of this tutorial in order to create 3 groundtruth files for instrumental/vocal, genre and mood recognition.\n",
    "\n",
    "Likewise, the original audio files were preprocessed to extract Mel spectrograms as an input for this Part 1 of the tutorial; also refer to Part 0 on how this preprocessing was done."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Requirements\n",
    "\n",
    "* Python >= 3.5\n",
    "* Keras >= 2.1.1\n",
    "* Tensorflow\n",
    "* scikit-learn >= 0.18\n",
    "* Pandas\n",
    "* Librosa\n",
    "* MatplotLib"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Download Data\n",
    "\n",
    "If you haven't already (following the [README](./README.md#download-prepared-datasets)), \n",
    "please download the following prepared data (from MagnaTagaTune data set) for this tutorial:\n",
    "\n",
    "**Download prepared spectrograms:** https://owncloud.tuwien.ac.at/index.php/s/bxY87m3k4oMaoFl (96MB)\n",
    "\n",
    "Unzip the file e.g. inside this Tutorial folder, and adapt the following `SPECTROGRAM_PATH` variable:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# SET PATH OF DOWNLOADED DATA HERE\n",
    "# (can be relative path if you unzipped the files inside this tutorial's folder)\n",
    "\n",
    "SPECTROGRAM_PATH = 'ISMIR2018_tut_melspecs_subset'\n",
    "\n",
    "# included in repository\n",
    "METADATA_PATH = 'metadata'\n",
    "\n",
    "import os\n",
    "from os.path import join\n",
    "\n",
    "# here, %s will be replace by 'instrumental', 'genres' or 'moods'\n",
    "LABEL_FILE_PATTERN = join(METADATA_PATH, 'ismir2018_tut_part_1_%s_labels_subset_w_clipid.csv') \n",
    "SPECTROGRAM_FILE_PATTERN = join(SPECTROGRAM_PATH, 'ISMIR2018_tut_melspecs_part_1_%s_subset.npz')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# IF YOU USE A GPU, you may set which GPU(s) to use here:\n",
    "# (this has to be set before the import of Keras and Tensorflow)\n",
    "os.environ[\"CUDA_VISIBLE_DEVICES\"]=\"0\" #\"0,1,2,3\" "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using TensorFlow backend.\n",
      "/usr/local/Cellar/python/3.6.5_1/Frameworks/Python.framework/Versions/3.6/lib/python3.6/importlib/_bootstrap.py:219: RuntimeWarning: compiletime version 3.5 of module 'tensorflow.python.framework.fast_tensor_util' does not match runtime version 3.6\n",
      "  return f(*args, **kwds)\n"
     ]
    }
   ],
   "source": [
    "# General Imports\n",
    "\n",
    "import argparse\n",
    "import csv\n",
    "import datetime\n",
    "import glob\n",
    "import math\n",
    "import sys\n",
    "import time\n",
    "import numpy as np\n",
    "import pandas as pd # Pandas for reading CSV files and easier Data handling in preparation\n",
    "\n",
    "# Deep Learning\n",
    "\n",
    "import keras\n",
    "from keras import optimizers\n",
    "from keras import backend as K\n",
    "from keras.models import Sequential, Model\n",
    "from keras.layers import Input, Convolution2D, MaxPooling2D, Dense, Dropout, Activation, Flatten, merge\n",
    "from keras.layers.normalization import BatchNormalization\n",
    "from keras.layers.advanced_activations import ELU\n",
    "\n",
    "# Machine Learning preprocessing and evaluation\n",
    "\n",
    "from sklearn import preprocessing\n",
    "from sklearn.metrics import accuracy_score, precision_score, recall_score, classification_report, roc_auc_score, hamming_loss\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.model_selection import ShuffleSplit, StratifiedShuffleSplit"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 1) Instrumental vs. Vocal Detection\n",
    "\n",
    "This is a binary classification task to detect whether a piece of audio is instrumental or vocal (= singing or voice). The output decision is *either* 0 *or* 1."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load Audio Spectrograms\n",
    "\n",
    "We have pre-processed the audio files already and extracted Mel spectrograms. We load these from a Numpy .npz file, which contains the spectrograms and also the associated clip ids."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(1703, 80, 80)"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "task = 'instrumental'\n",
    "SPECTROGRAM_FILE = SPECTROGRAM_FILE_PATTERN % task\n",
    "\n",
    "with np.load(SPECTROGRAM_FILE) as npz:\n",
    "    spectrograms = npz[\"features\"]\n",
    "    spec_clip_ids = npz[\"clip_id\"]\n",
    "\n",
    "# check how many spectrograms we have and their dimensions\n",
    "spectrograms.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1703"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# double-check whether we have the same number of ids from spectrogram file\n",
    "len(spec_clip_ids)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>spec_id</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>clip_id</th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>37</th>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>40</th>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>172</th>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>198</th>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>253</th>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "         spec_id\n",
       "clip_id         \n",
       "37             0\n",
       "40             1\n",
       "172            2\n",
       "198            3\n",
       "253            4"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# create dataframe that associates the index order of the spectrograms with the clip_ids\n",
    "spectrograms_clip_ids = pd.DataFrame({\"spec_id\": np.arange(spectrograms.shape[0])}, index = spec_clip_ids)\n",
    "spectrograms_clip_ids.index.name = 'clip_id'\n",
    "spectrograms_clip_ids.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "# we define the same in a convenience function used later\n",
    "def load_spectrograms(spectrogram_filename):\n",
    "    # load spectrograms\n",
    "    with np.load(spectrogram_filename) as npz:\n",
    "        spectrograms = npz[\"features\"]\n",
    "        spec_clip_ids = npz[\"clip_id\"]\n",
    "    # create dataframe that associates the index order of the spectrograms with the clip_ids\n",
    "    spectrograms_clip_ids = pd.DataFrame({\"spec_id\": np.arange(spectrograms.shape[0])}, index = spec_clip_ids)\n",
    "    spectrograms_clip_ids.index.name = 'clip_id'\n",
    "    return spectrograms, spectrograms_clip_ids"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Show Mel Spectrogram (1 example just for illustration)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "# you can skip this if you do not have matplotlib installed\n",
    "\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "# take first spectrogram as an example\n",
    "i = 10\n",
    "spec = spectrograms[i]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAV0AAADnCAYAAAC9roUQAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4wLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvqOYd8AAAIABJREFUeJztvXuclmW1xn/hgDMy4NAMMjgwMYAIKiQKBVsoyEOWmmba1tLSdu6tnU/uXXtXO+1c213tzrbto51P5lnLDoqJbjFRVBIQxCEODQgTEwPOCAO/P9oW13fdzMv88vfuPp/fuv5b7/u89/u893M/z3uvta51rUF79uxRIpFIJKqDA/6vTyCRSCT+/4R86CYSiUQVkQ/dRCKRqCLyoZtIJBJVRD50E4lEoooY3N+bgwYN3SONeE6/cMiMJrNrtMvsns56/0DpDFdueE7PSZJ0SAtsvP/Yc/Cd4/Ada7bhANp/PQ44ZpTZu9dhQp/6/2Au/0YwbMYws7sX78ARuwc+aKV18kc3Dzhkl4jdD22q8CXcCzXBfqrC5/+PcLjPTfNwX1sbH8XcSdIzz/X6qyu81jOwIcbiPNdtLhz0TIVBfr95z549XB2SpEH9UcYGDWrZI/1ThcEHhrF7zjN7hLaavfS7L/QPjCwM8vLLntNzkiRdgjEvwfvTn4PvvApjXLQAB9D+6zGs+61md1+KdfC1y57z7/xbwdw9c8xeOOhBHPH0wAflOnkb3v+Zm8MuiQ/I7mFfrvAlB8G+EPZXK3z+/wgLLjPzX+ZdbvZnxn84fqb9svjaX4XDCq+tGtgQV1zm9qX/XThofYVBLl+8Z8+emaV3MryQSCQSVUS/4YX/d8A/zXfON3PdT/xteru6Afa1ha849zK36T3cgPf3B92w1/GAV8K+ecBf0XB+h9kHven5ZndceZl/4JJrMEJ75S95m4/R3QFPhp7DafhOSbplJ44Z4jY95p8VxqiIY9285HS3v7YYxw98vufrTrMX/uATfgDXzYU/LYyyyM13uTl38i/8O5afZHb3AwUPk54adse6CHYH7FuOwAvL4ndUwtk4h2sfwwE/GvCQX5/3erM3qtkPeH/hQ5dw93t54aC9MPsyt9vwfim6wOjALbBPhP0DDlBpVzsw5E43kUgkqojnfqd7ou9s557nO4FtGm72ViTq1oyc4uPNLnzHudiiLMXf2w3cKiyAPUcB7W6OPXWl2eumzsB3Dnzn9e7az5m9Q0PNvvbis81efemFPkD3ZXFQ/PPXfazT7J6ORj/+Qux8OwbFMadjZ3s+3qdXwJ1aER7L1+xJZjZ+yXcTnUsw3/dxvqcVvsMTVLPgJr3rnE+a/V2c01Mfe0UccpXvdIeN9RjtpbrC7APP6jX794rJo2Xd2OVPdbPug34N+3b5bbpz5DkY8ReKeIGbg33XOeRrnvHbufVIP/6XhSEroFe1Zteoz+ypF/8mfGbpfcjhXMMj4GH+AF5YO9bq6MKJcac7HzZ3uvRyT3t1YVBkTDUJ9r537LnTTSQSiSoiH7qJRCJRRTz34YX5bh4pD9AzvLBRziNdM3qyDzA3ur/DRjiftXsXo+ekAsEFGTk+jMnk0OYucCNLVL2KONis+fJEzRJNN7tFzllcPfIoH45uvSTNdfOYhiVmb27w38H576grcCdn+3zWjUbIoh0hCybFSomdwXC/kNA7sMbdcjnFtoC5hdcWmDUU66BVa80eoT+Y/dRYT2xKklZ5GOOEeve7p+kRs4eDa/00KJGSIqsJ0bLhDX6h+/pqzO4c7etKHQWaVBuSWPiOEU1+Xk8Nxpj7Bae2cX534P0++e+QFJOyxEgPM71o3F1mrxpboog5to30Nb9zK34r76sw5Av4giTSyErrsYzc6SYSiUQVkQ/dRCKRqCIGGF74e9iFFGebm3QpmNF8Ghn8IQgd7FwV3Z5D690NX9lGLuQY2BXcCSnw+5obPBO+pg0udQfcN20sDOrT26rfmd0eSIbAdNjtnH9FNx3lic+De/v77R5OGFIXyxl39vhktDa427hyF8MLZBJw/itjMNZF5ZW5pfCaX2euNbr+DD+UXV1/kRWUW3ABxoMG064YyhoyEsyBOj/v8XrS7A01fs082COJfFgp/hbY27oQvwnhnALDR/fA9vlrw3lzbibqiTDismEMTb3dTbCXhuEaNtf4fbexL85FzWD/8TuxtoZM8esxdJiXi3fVFSgRPeBKc732EzbJnW4ikUhUEfnQTSQSiSpigOEFqh0VyOTI8j8mJ13T5aPL8Widu6o7C6GA43Sv2SsHH40j6OrDxdsP0aGtvVBXC8I7dKlLU+mE/+ZeD1k01/p5DhVUsMJ3HskXgggcVduexG8/st7ZJCsGgy0iqQvZ3ZeCFfAEMsZRq6uQ7SXBZL6bvKY/GkayOTGk8JoT58lOGIXiCbJFHi4V4ix0sZRlyFJzPRMMR0gF5sAun2/Oxbf63uADhKU2lC9E9xafGTrMQwM9wR0uZewZ2HCWCkNEDOds0KFxSN6Lg/tnDfXhh7DAqLXGQ2GStAUMnnWbPTw2rckZKAyLdLUVwgvL29zO8EIikUj8bWKAO93hsAsBfPwztSGx0IQD1qrV7O51SIrdF79i7Vn+mbhzxT8//6hKcpFj3ZxW+6jZC6dgkIXg5W0tzAUSO1trv2g2d0nT5N/58+ln+HAlaeN2N7txjSZrhdmc76720r+4m/RWdu8i3xLSeSUedBtspxNHcRSKvATEdJIGu2jOM/qA2eQoc75/OrhU7uk1otOxIJk8Ii99VUFq8Kk12PFh+h7SMWY/03OgHxDEmHbyBakDngDWTkuN7/I75yP5eUNpPVNox+2DdKXZm3GjcQcvSQ+eiPtoIQ7A/V8r53NPxORxfUvSujVt/gJEtTad579163ZMVinxzjmfj7f7KY3PnW4ikUhUEfnQTSQSiSpigOEF6kq2xUM2u1vDMt9KPF2qOHWfFvVIH1dM/jiw9ScHsVRiipkIJYsMjAeXo+SOuVJZS6e7xCsafS5GMQHI8ywlAJGgoktHl5rJpUMmO3dYkp6a7eWwG6iU1c2sGFzT/emOgvncwiQtNffpdmJdSQqcToa26Ho2c74RYpIkDZtlZouuM3twHxJpWDYsjZWkxrGe0Ouc7fPHcM6OboTL2jBgXSGpyPWJKFKvELIYYEebP8GfB+M7PSa0tfF5ZjP0UjqvSuXfi3r9erTVtpsd+N6Sxo7zY9a9ypO0O/p8fnt7XC2tHF7Aei3p+O4DudNNJBKJKiIfuolEIlFFDDC8gHLaklsz383DwMOtVBbcvRUMiYK6F93wdUFAGI39KikZFY4JLjUZD7vY0LO9MGibWYO6/N0ahBcm6/F+TlBl9gLcsZEV5NDofm3dUhgU2fHAH65jthxMgm523ZW0BC7y/H2eYvEcYmlx4aJCDPyQle4XDp/kvNFDwdMNIQ0pNChtQvnxQd0+F8Ma/DsOKzRFDIpqYIuMn+OMiG11fpF7RqJjNnT/JRXmz/EMBMfjk6DUsJaMEZaDO7huKq5vKYZOEPYgq4hNEFgGL0kbNuJervDU29mB51zx+YF7oMJ8743c6SYSiUQVkQ/dRCKRqCIGFl6Yggx9KavX7malnmgP0Kcj8R7j/WmM58UXDRUEyEvFEZiJbX0IcwQv8ZuwSRyXQqkqiAJt492N3EQGxP6wLhAdYFkksWqLk/V3bi6IV+O6BiZHN37XdBDcS6uqDTZCAeG8g8uMkFGpTJjfi+hB0yQPDdAVHXIY+15JOy/y+Zmuh/wzICe0NPzebJaUStKmjbjO7W4ytPUMs+mvwoCl9Uy2B1zkUOYeilG28QWFNX6a/45B7vlrxDwvdw6MCaly4VIlVhHw8HpK8ymybVCZ3dleIfzQVviiEXjGzMf7DxRPT1LudBOJRKKqGNhOl1y+EjcNuyRyIZk4myXvtnrL5tf4AG3xK5icWD0YbW2Y+Dkbu6hSQqpS2WmolqW2LVsESYJOqpD/eBSiOU+wZLQC11JS+NeupCHb2uRbs/ZdbWHI3VPrw2uGOiRZZqKlUmlHztcqJTeX8AUIsJR2dxSswWaE811p1yQpJEg2TcYuFbmkTeAPr9DhYUjqtXbP9flmsi4kdbnTDXOlON/wHFprfR10DcPimlLwgNrxGjeV6HZEvehFco6tpPgEOhs2duy8ZkzWjR4D90ZSx3rM31x/cE2d6NtSel2r6/h8UUy6tsVD9oXc6SYSiUQVkQ/dRCKRqCIGFl5o349j4O7+Ab483a+gR0q+W6E8kZ85YOR2s3czg7IVSZdSySPcr6DC1EOeKDnKBc4yykp/O3OC2bVI5GwOesVA6WqhY9Lad3mp6/AaDy9U0n+VFK7zQXDhQksldlfdH+AaUD0qhoBeUOF9hfDL6qn+woH4DpbbllpDaambd54w3+wTm/0CMFFcKksNZb0IbbGL7oF1ft49WxH+KXFEGb5BOIYhjFACXRqzB9zUpb7mfzPes6Nca6s0MY7JEFo7bCTBe8Evbu9tM7upttDGiYlIKhcWogf9npMUQyulEvJ9IHe6iUQiUUXkQzeRSCSqiIGFF06DXcogt7nJ7PlICEfT5Rh2IlTGFkaVMbotu+mu1SGcMB8DlFxTuDEUR156/gv9gF0IJ/ygMCZCLeSFcm4oQE4uqy4sfAfcsxNr3N3l/NLdXTs4ij7vhqsUuuYS7bCZgZai+wXmC0MYFVFauTgPcsTZGTlwaEcWSl/XOTODn9lR42uP4bRSOKel2UNX60b6emXp/NMN/h3r2rDeg+qbwtpjmCQwN+guXxSH1A1Y87hnON9sh1RcR7yOfKa4hnwQja+t9dDLQaXvQFTjmYnOF6YSHFkuxTJratOXmEX7QO50E4lEoooY2E4X/5ZFPiZ2w0wW8d+Qjea6O/BXV+DPMllxAHiPu0cg0cDAeeygEmZiJBMN3DlwzM2FXdJI3yVRWzhoCbMKiMkMCKNIkl5eeG0vcL5D88ARsfKoE40SOd8tTb6DWTMaCaii/iiAtRMEWLjzQsKwyPPFbppNIblLZVK3CCxHzgWTXvwdJS4wz2sddknUFt6FMRpH+9rs3Mwkr+I1mO8m22aFJ0F7HDK8hnuTa42J4SIvmvc3k9x4n88L7mx5fUrnxaQtzysIak0vVD9yR74/a/5/kTvdRCKRqCLyoZtIJBJVxAB5uuTAFXiNSzzY3nSWf4ZbeQqILJnofuW6wW3hK9h2JSTS6LIsgB0lTkNZ35PMCHKMBxbjhRlxTLiNdD0pABISO3S1CtzJIVNdpIXzyzAJXafeGrj1krad6O7Yii3eHqm5CW1umPBri+fJlXb0yYU2z3ujHTZzRUiwSIoluxAQotvOxObYiXFhrHu5u5ZMfjJEVBR1AZhgWtHm1/BJeTflLRv9vBkSapgSY3Bd01nW62YQjeL8LghDSrrGzakXmkm3neuZbr6keC8G3erSeex9+OZ+bSm2CWIbp22IdW3WcT5AYa01vNznfHitX5P+5HVzp5tIJBJVRD50E4lEoooYWHihDmWqJfZChfJOdqs9XCvMpvvGzqlSQet2MH0Q8An5K0tal8h8t8PF0y38QBvsqMUqlMeSs0lmRw19Kf4sMigk7Vzn3zGqyeeL890i13tlKawUVbBG1PoXBy1WchQL56n5zu4g3zKcB6MP/I6S/4YQUSVXnxn8Elf4gNFeYs7s+Cj5fJMdMhGcW0n6tV7c73mF82z2ENFTKyDnNbjAnEHI59ijXK6L6lyhC/foyI/X1le7Pd9NcpTJiy6CLJU2PEAWetyDoRleDz4/pBjWOFH9l27fDzU0hhIk6ezaa81myOJD4RN/Qe50E4lEoorIh24ikUhUEX+diHmp9A3lg3QbGW74tt5g9rJFx/oAheKISWc8bHbnVpDD+atCBvRJviCN9HBCIHIHT4mDFpgcOCRmd5GVJsN6f8IiuCZ0lWjfi8xs+xaEUSTt/Jn/lqbz3L2dWOsp523vdDZDqZ3SU4vdJSbJPbiidDtZmFMS7j7Xzd9T/Btgue1thdthxEg/r41sqQRwvksZe4YgGCKqafKwR8fDrk4XQiuDISIvhVL3J3q9Iqi21kNbo+o9TNI9thBeOBFr/BIPa/B3sdiHoQFJapy93uy2Gr83H5ztraBWyJk0RClcRnxQHzO7Uourrp/FB92iMzwEEUqH9Z/7HC93uolEIlFFDGynyxYV+1H6Ng0CN9xBUtdz4yzfSaz+bRS7DCWjBHmjTPh1xN0deYzUXg1cvR/sR5PODtcfZaKGiZ6wE+DVKQlv4LcxWcfWRtx5bWuKO7HVc73FTBQt8mtGwZDQdkhS7Yz+d/nkzAa+ZqD1LuILUrfvPpjkIr+ba/GZQuLtmR5/bWO9lw6zDJiJs9KY07FNv3+knzfnu4Mc2v1pmwXHoWuzb30PHOPzvfoJ7BD3o+ScvGauA67FUqJyWo0/H0JpNu4r3iO8T5kglKSH4Da9Q18wmxq9X9FbfICCsBfXzjn6odn3x4/8GbnTTSQSiSoiH7qJRCJRRQwsvMAKu1KJ3s/cPHCyb/8ZGqALwjK+vqOin0OVpq0og+zahcA3Ew8lPV0cE0IYLFfsZjIu8gPpAjfD3WWZatDTJUotQXANYpkqWr/AjR+hP4Qhx41zF5khIoYGlugYs+lmStEtpEsX3Eqso5C07Sh0loWbzdAKXX2GG1giLUkbuj0Zd3i988rJtSb3t5TMY+Jm3Jh2s5lknDD5t2avXoWQW6n9VEj8+uRsGoP53oxkXKymDeuP15lrj+poJQ4twwGhjVAFMHlXCi/wvluF8BefQYdirS7bjOS+4m97CPdAf8idbiKRSFQR+dBNJBKJKuKv6wZcEgNHhp3Zcqoy0f2l21kCy2W71oFJQE7nhbBLHM/+ZIGkyBtdQAZEoRRTzh/u1Qf6/QqWUQbWRWG+WaZKLjDdILrxpW61DN8wNEA3nW7jhoJLvbXPf9thNe768ztDKIVRphKTA59htpwZZ4ZWyNuVpEcHexdiuqpc3wxLlTigPA+6t80Ic4Qu0VwXpfACgWNYtrpsMFzoUgjOK1/1P23H+wuI+JTWFsF2UgzX8JpWWmslEXN+5m6UYTOkwWdUaX7Jb29tiiG1fSF3uolEIlFF5EM3kUgkqoiBhRfmwy6wFw6Y7u4uM5zM0NO1Yma81E2VWX+StNe1oacRPY5SF2Mcc5zuNXvNYfBnA2G9UIqJ37IMJYpUAAvKZsxAF0jwB9b13w2VqmIEmQdSzADTHaN7vPRf0Sm50A14wgzPwDOcEK4zi1EWwi6tXISIlszqP6O8P/3MCBLt+RmGG0rZ9HYo1G3a7iGf4fXuDjOkMXjeg2Yv+23Mrodw2WEe/qJLXXdYp9k9cxvjmCxDB8OBwv+cX963krSpt/+yarr2ZJhwfnlPSdISXLOLdJXZ7Px9pS4x+4D5/kyTpDc2XW021+8d4RN7jdfPe4lEIpF4jjGwnS6PLojR7L7FO/Fe9SZXwGGge6bY9saxZnHMmDRM9S/u24UdSqXzLP1q7NqDgAV5i2GMnXxB1PU9Am1DFuilZodd0Vyq2cStbkuD72S5i7oT7kmlMkpJOgF6o0Oxe16EjMnYT640e92K2D2VyQx6QEHIhNeMibVS8mi2v0h+8e06ud9z4u+WpJom38E8gFp47nSZ2GErGClypbu3+u54Sz0SZzwnegXD4tobMtOv2c7lLlbzxERPYA0ejDFLPF3KACCxS++FiTQmzSSpqdZ33Bu6Du33O27W6WYz6VjyjPnaL+FG8V5nMpreZGmMLb28Zu8Nn3kWudNNJBKJKiIfuolEIlFFDCy8wMRZiSsJD/hM3WB25DU65zOoTRXOcHqtE20X9bzID6BryvMkj1cKSavHNkJ1ie4WPf3RaBEkhYRd5HA6R5nuLstD1wyOE875ZPhgDhKCwdXqi61OH61xbipLsx/tc3es8/PQMy6UKzdN9jF4HpPRtmnh4JN8AK6DAo+0cbS7quRz0xWl9mpJi5X6ubxmlTi2pdLXR+XzewBceyZxKyXnmsbF0tlwjeo8vEDub/dS6OcWQochUoJ7hr+VibOSOuDqNa5opx7cR1AZe6nuNLvS9ZHi/DF8QHDM4xruDcdcoivNXlvrJeX7Di7kTjeRSCSqinzoJhKJRBUxsPACuZKvKhyDbON0PdTvkFRUIg+y9ehYXkf1qCV1/pkeCoqTs1hSR4OKGDvido92Vkbg0JZmEtnell53/UfW9i8G/rned/sAhfN+eru7UwfV+3mz3JPfsaMmlqlWEhg/p8YFmxe8d36/x0vSqbrNbCp8hawzIymz3TxgZuROboVQN2mhZHZwLq7QpWHMdVc6E+OMi79v9sSuNWbPb3D3l79TklbIXepnmj3ERi4q3WNyUUtC6Z3LEfK5xs2tM9BSqR0DlMqAeQxCbGS5kAtMNokkTRrnjB5ekzu6T8Np+Y1HRbBS6THDGh/TB83muviccN8VQLH6/WkT9Cxyp5tIJBJVxF/H0y1xJUd65cu/6lNmk9fI1iU3fuO1GC9+xZYzfmN2V0cFQRDudEvtTbA7riFvkb+Vc1Fq19MOs7atcNBfwF1/VztEZAtCPbvO9fkkj/GxLf4PPKvJNX4XPoyElSTd4uaii3xX+brm75lNkaJli2OF1IoZ/r3kszLpVck72b0UnoekYbOfMps7HO6KuIMs8YtJ3X36YvcMuEw45uOFRoo/2e4le90LPIm17VQfY22v75a3gdc7s7nQsbQdNjyFkEziEKVk87vcHDL2j2ZvhJgS13O4xoo70yA2A1yvM81mYo086j+dh98j30IzXJ4XE4K/1kvCmOfKPZ6VK47GEZ8Jn3kWudNNJBKJKiIfuolEIlFFDCy8cD7sgps+eqK3sWEih3qutIe8yl2WnQucX1hC3Qh3B3rGQqyD4Qa4Wn8axE12Tw3cU4Y99oPXSNeTWrdN5HRyzOg5qafb3bHpDZ643Njk88sk2YSjXYhGklYv9XYwxzU7T/HarrPMrq3zxNlpM34cxvz6Gu+wOnqcJ0xeorv9A2zP0xaGDBhV78IwTV1dZg9t8ERPSLoMLmgi/8yFjBhK6QX1lO15SvrQwyBos2OmX0PeM6t6vHx290IPrdw/cl74Dl0EG2tp3WiEUqgXXUjajvuoixq/UjeZTbecczE+xDxix+ZpesTsRWO95JyJtFt1itnPi32Kwn22SM7rZ/KO7XxO1u1hTPKxWyY7bzd5uolEIvE3gnzoJhKJRBUxsPACuZMFLl/Higlmz5z8abNLKkB74/Cmx81uP6stHENeXc8qhBPINJgLu/Sr8VtGj/FMd8dU/11B7xUZ/9KYlZS2QpkkQxglhgRAruTputlsckBDexRJD5w3o9/z6kH2/JJxV5gdWu9IenScl6X2Vir/ZniBdiG7zvLYTQ2+LnheDPeooCalKR53Ypl1nZsaNcnd5VK7nrfqK2a3N7eZzRLdvl1YsGSxMOwnxRZVZIMwXMZQYYGDfyKoHLN0v9nUbub8lkItLDHnPbEbCoKc/yOh3FfSL+Yz52z0HXrhEl9Mb5n+n2aXyobJuirxsfeF3OkmEolEFZEP3UQikagiBhZeaBv4p29DdrFS2d43Fr/NBxgRM8rHT7zVXxgJEeeRUCoiW4GdZaXANBiJEsaOkQgv0D0rMSIQDjhq02qz149y95cut+rw26cWUsqr3C9sH9NmdqX2R49uh1i7pO52J+vPO+pnZg8Z5iEMlkCybYskbdnu53Fova+D4BbS3WUJeqEwh67pmLXegqap0e0V9cjgby7QcXBdQ3gM4YWWSf7C4yj5laQrer3cmMU9c8ctMDuI9HP9lkJbDCdg/o6efJ/ZD/fghxbubd6bP5xyjtlr6p9v9pxObyu0sbEhjMn5vFfH+QHr/JqsGOPz+ViXr73DGmJHZ4bU2K5HTrhS63RfR+GcFJXI2NJK+nz4zLPInW4ikUhUEQPb6fIftqCbqmG+BaE2aMV/CJ7RA7Hh49CJT4fXDEy63Fc8yoHfEnRQyZmlvm4pyXUu7E2wnUKrLcicNYz1f+iua/nDFASGuNtjue3aPg/4c1crKZSEPn2Ui+rUIuH08zee4R8oJXba3ax9k7/A3cjoee4V9M3zhUGhHymWoQqaOH2jfI8REmmla4gd9RNsOePTG9pRkSMqFcrW290zqxnnu7/h4KF3fayC+JIkLYCNdRKacHJnXGjXw+aVM+q91RZLeJ9udDt4cpJeiUTv09DDvbnNy9qZvBvZ4B5pKaFFXd+X6C6z55zpzyhesw1CCyFJffgeJmk5nXsjd7qJRCJRReRDN5FIJKqIgYUX4KKUSjOHoCR0llxdilt3unhNR7u78NjRUacyqBVR6mkwEmnkF5dUxhA+oBZo3XR3rXoWghtccvFYfhxpihjCXVOqSWl2oUwVoH4xeY031bzS7Kajoh+5dmz/WrcH4hoz2TT2BO8OLEmbu9ylZtiJXODnI0xy/xokM1bF9kiTT3BXtWNKTNzsDZapamyhozOSsoF3W0FlrNSiZsI456I/M87dbrr+IZHGJGKpcy8Bzji7QFOfePfcqOLW0+FrfkWDJ7XGbPB7ZE2Lh64WF+rYF6BbNTnKuzv8PFqafT2zHJedqqWYpF36E49/rTrL435tyKwNL8SdztT1ZvPeLYm0PYvc6SYSiUQVkQ/dRCKRqCL+ujLgAt+1r81doWvlgs3kdNJ1XXgjRLVLQunn3GPmMGR3uxk/aMPnS4pg8CCYjQylmO34fGkmGXLAaTHUQuYBxanv3xVdJ+3qX8Sc7AVm+DtuB/9YCr7RiPf6Dxla4+5aJ+Zu4xb0yZG0s8dd6CMbvHzzdp1s9v3fhHIW1x5DN4qMkxHbXWVsa33/4QZ1Fzo6Q32LXYsZ6WJoIDAkFEM+C2/3Nd96sq+DEfU+/93XgnFSaj/F9DlCgywX370O4YT2wpjXuNlRh7Xzj27yejBkJ0W2wc/XOK+fbbSWHO0XhKLlpbJrnsfcs35hNtsIPSrnrt+7MfJ0GeYIzJkUMU8kEom/DQxsp8t/VGpwKnIKCQpUMHCuqUgW3Rd5utTl7F6Hf/42fADnfcCU2NSQ//TkYzY1ebaiYzSJnYvBAAAgAElEQVR0fgsdU8JrU90cOcr/+bfVeoKKXkDj6LhTYJKFuyj+8/Nfv2NKQahjne/4eM3YPJTzu5MJQEWdZSY8yNMNHs5YvNATE1RMDlGMZnSD73znj1pg9rFHsexNehBqSYG/DfGZGVP9opd2XuSmHzDd1yPFVFb0oeVPoYVVAHaI9LJ2gA/bOH292Z3T0dhSitfk5bChC9O4wT/AVlGSdHitew4HjXMv6sYOb99F75AJrV+HyjBprbxSrgmZR65vCmqNGBmz5N9/+B/MrmvrDMfsC7nTTSQSiSoiH7qJRCJRRfx13YCjN6a+sb79H1XrHDhu3ds13gdYhXBCIekVuHgoPaZIhlyvRbs3Rw4iuY4bzvMEVMfDSBqQiEd3TpJ+Bt4nWqgM3b7b7B21Hk54qMvd+J4HwA2WdMgJv/Mx4M4u6vLWJAHrCskjuK9MclGD94j3ubDJ/FCDKq2A8AvDN0GDl1zqS+v6f1/SMVfD11+LA5BHG77dM4BP1xc60SJ0wkQwBW+Gd2EtFnJ3bAdDLJbrGXc+AFef1eCF+zCEILC+2am38wZ8R0lEh/ci74GPFz6zF7jeJeneWk9SPdGLuUFCsHVW/611GEqQpK1bnEN741avz398oodvmNA+sKags4zlelyD887viJ/4M3Knm0gkElVEPnQTiUSiivjrVMbYBqeAVrn7y/ACOXJB19MPl1TKIMPXJKcTHsuk9z4cxlx569Fms5x2xNHuTzx8Ic6zxF6oc9d9zxH+NjvJsjPyOQ0/NPvm+c7BlaQR+oPZExHnOKbBXW6GHx7eVRACbneT880xXoxOvqX2JmyRslXPMztolpKHS+WytvAVIUSx0710PVMHlbFaZ1kse+KYOCjukMPJ0wUj5bEG1+gtzQXL2BnuqmlGbTGrULG+G89eL6KzA+ECuOksWz32LI9RPLi1cHMziQ9lvvUtHv46COW3/E5JOgZMjYtqXev2qrM8JtcMqT7qXpdwSzd6Dy33EOaBEz1cxvMstbQ6Yp6H1K7ve7XZ/THCc6ebSCQSVUQ+dBOJRKKKGFh4gS50IYM8bLK7okFFDGlUtuuh6zRk5B/Dd5A8/vPDIKLtTUsDe2Hl0qMVAPfrxjYnZYeySLp8hUIRZncHIdNdh0T3UXLh7k0zvZy2sx2qWJKGTuzfhWOBBVWZGmZHekjXWP9eKoL9UN6m5evfeKcPMD+qoR070Uu32aWY4ZxbprzGz/MwP88aKssphkGGgMwwpM+z5wfO8GKJt038jzDmDTrT7JN1ux+AOpsXdLrC2sjG6P5SeYy/jev7sbnOmNjRjYKXmliQ1BkFvQzb+vy+XLMAN953Ch+6zE262GOWo0DAI4tqbIxr7biZvrY4v1Qmo2IY54pMG0kaO67d7HW7PN7I9c3CEZbSS9KyG481+5dnsD34T8NnnkXudBOJRKKKGNhOdz7sq+IhW+Y6QbCl6fdmk+fYjZ3wASN869DWhK5xiqIidaOhdTsCfFa2FSq1GcJMHHGU/4sv2+r/bIGKWtLoZeIBO9s9oP5SNjWIpdTFpAzFZUY0+ZeyNJZJsK51UZzmgGG+m2Dyc8NG/PMj6fWyiTeFMe/d7omyX9a74ApLMTl3XT8AObWgITvjvXDFcJp7UHE+BNN5f0GLdd1iNK9Eco600EHY2G5sjPN7IlwxikIRB6I8/KBavz5BhEfSmtHYueIatdX4fdXZg8Qbk9FS8AaXLcI9Qfo7koyFnKK+oHeYfZNc7/lX230HeXH9lWY/JE9+sh2YFLWcD5noPFwmNlk7UBJw4vr8uP4NB+RON5FIJP4mkA/dRCKRqCIGFl5oh31JPGTnElffuvcEdyuZODsUdZS7l7iPsnJ0THq1Hv0Vs3s2O+cz/CpyPksavSj3DOpQDB+QcljS6OX3YoxBTPQgOffZ099j9gVjvhm+4kBwCJ8AKZkcw8BrLIQsdrf7NZje7Cf6h2b/8a3nuLv2cX0wjPnq+uvMXoUy4FN0m3+AyVCi4P6yGy0zlYPYRQg86V2jYuderotQto5ruBMdoNn6RYprnm2Zhjb3Hz64Ts4JDR2KFZPLw85+yr8DerpDZnvCeudhUNGTYiId90APwnZ13nQ3hHsk6TDwyjejfrl7uSsI9s5wXWYmjpkUk6RFDV4Kv/KJF5g9cqLfE0zW7VxQmAuUWXM++0PudBOJRKKKyIduIpFIVBEDCy9gS03xZUnaDQ4hyzvpKlGUOGSl2XZEsUNww1jnnnaNRKa7HQN8Po5Jd3XNhW3+ApWcboBNQefS96I2cPsk/8+r3+Q80hVy9aOS60RFJIZF6HoGRsTyAu0Cq4ItVViKyY7PQZhe0u9UEEvfC1TWCkJcDCeMjJ176WqO2QLeKDuqIHTwDn0xjLl2ll/oMH8Ys6/C3ElRaJ6tX/gdvO6H3ONxqB1zQACX1DDT412ttWgBhPR7LcJMfSPivT3xPA8FzNRiH4M63lwGmxRw7T2v9xfABrl4ht+sLXI2FDs6Pw2OrRRDmmQvUKmsVx7CKLWG4mu8D/tD7nQTiUSiisiHbiKRSFQRAwsvwGs/sJD5Hgxi/cX6mtlXgvIQVJhYTlsIL9Dt7kWn2cA0oOt/bnRN9TFXBJs6xtXPls59oR/PcAPLgqXoIsNjo8rVY63uir5HnzU7FBAo9is7R65M1osUPV3XYbM9qy1J3Qs9Y7wE3/HYdg/vLKr3ooJSxn7LFo9NHdnkvyW44WQvkHHSFsXXV43zCT+qwcuqA3kf4YVn6FZK2gIfOZDvER3bUO83SWA7KDJMyEBhDzW+P3yOl/3yfpCklloPOSz9pq/foRf4fdrdgdjh12Jvwsn/8bjZLOkP7BCUAZfQM8dtzh/DZVy/DJNwvUtSG+J8LI3foEPNZpm27gtDauyH/cf+l7wU/vvxI39G7nQTiUSiihjYThfB455C19dh6JzJf3omO/gvVDcWJb3tsUUNdWfDebBr8TAIsGwutKjBznVDH0iFHJMzV9rpMvmDhrdNh7rgyuglbp85zzudshxXip4C55viHUwqNNVHQZZu+U53VZ/vzEbVe0aEXMtSO5qdHc51rGnihAIs1f4Y7PnxIwtO8BfP+N3P/QAkdvbgEt9DTV/F+QreBryXCWs9gbWpNZaQhkQNwJ0Yr/H1cn3YUosack0pYPPkBdiBb8bOtnB5mJQNutakOU+DXaC7frj+MrMfQEL1jo+fZvakD7gWNne+JX1d3sud16DkmdLBbL1V2Omuu9XLw+89lWsHa28v5E43kUgkqoh86CYSiUQVMaDwQsNUd526rmVbUql7prumayd7goQuCRXDaus8adBT0PV8cm6bv9CBcAG4vi+a+Guz7799XhwU3N0za9y1/8boi/2Aw/CdVBSTwuzugbs1hFRIjy4EbdE7Cz413SlyQJmgoqsauttKIXnZeZ+7Y81zPE7CZMfIggTYaoz54C736V5xtJcJB/eWJedUsCoBAmFrWnxtPq8PXNVCWxZqqQY+JqIH61s9HFZK7LBceWufh9yaa3x+364vmM1OvqWWQJ/S+8xe8zZ//6W60+zbp7sObVB1U4GjDKzBvb8/pbFcn+Td6nxPenO9r1zj63fouP0ox0Woau6TvzB71WQPo3S0Qw5Q0rD5noC+RF/FEbE0+1nkTjeRSCSqiHzoJhKJRBUxoPACW6TMO+9n4ZjF2z37SHeXmVlyQNmKpFRe+5IG7z67dpa7KMtGuLgyeY6nnfzjMOYvu04wm51M/2mcK5t9/Vzn5Q0ZHdsKUXFtEGijO+H+DkEzWpZQs3OqJP3DJmcE3jHq78xmCSnDCfydkjT6TX6iLPd8g75l9muW3GL2U9Nj3eQ/n/wZs+/WS8wmK2DLJ51qcBEU80P5uKTHyVdFR59xK8FJRlnqlS3vCmNub/F9ybW1EBxHRp7ru8T95bnv6HaGyXR0cCbjh7+zNBe87g8O83AO78uza71b8zcu9S68UrxGbLE0brnP706nv4YSaUn6WP0HzCYT48PjLjebJecnjHNC9wK9NHzHMzV+DToRqmLIiCHPLeeCwyxpYv0TZjPEUxIdfBa5000kEokqYkA73c5VnlAZPjkKsHCnura+/0RaEwLjFN7YGf9kwj/7HyjsOczLlzaC18vmjJI0GLv4TciQTJT/swU+8ZLIJ2YV1fa5ELhZ6QI3FMBh9RP/gSXpwVG+K1qhw/v9DHdNpeTIZHnlEccIPFwkvfgdkrQFaknkD9MbOUG/MpvXmFVdkjQRBMvtozDfG3y+w+ovcFM31vra4U6WJtsjBS6rYrKIPPOtDf5b37nFE2k727G9Lmyrjj3VieeTTnB+K+czNPWsq5xU3CrXsV4/xe+BUV1+j2ypj/dIaI2De5XcdO4o54e+WRHc1a/rcBfz5u3eIujIet/R1wyOC2NtH0SgauIzZV/InW4ikUhUEfnQTSQSiSrir2rXc8vy18RjEA6oaXb/i+IcLAO+qx2Zs+XxK5441V3LjtvBo0O3mJUjvOXPynfFFkCciTtP9oD8j+65wA9gJ+RrohiNvuNJq/pvw71FWWp9l78/Ys4f/Cu3x+TG52/4V7PPOM8Ta0yckTvZviUKsqy8xefnRRd43xWWg06beb/ZS38LcSApChdNcf7l4+M8LPKTFef78T/A59vjV3zj6teZXX8T5pt6r+D6vn3Sp8OYFD95tz7nB6DLUGOrx5RGtUS3M4Qc2p3zvQJzsXMzwgn0dqegzF0xPLbuy8javtXNlTfingCvV5KWfsqv69Jhbn9i+Ef9AxC8GXMEL4D0gVmup8tE2NZeD4P01vr1+OBGJ91Obo6dkZc9gQw1wjEMia4Y7M+onmtjWKQH6/ndH/Z18aHwib8gd7qJRCJRReRDN5FIJKqIgYUXqLYzv3AMXB+252FmmyWjU4/6jdlLl0dX9XX6ntmfHPkRP8CFiaJq0IIwpHSimzyvI+d4+OCuzyMMcmKhnJZuINr1hO6oaP1C17b7B15mKUnDznVuJPVFWTJKbuW2psheWD32KLNfhnJkXsPAqihoIBMTxjlDIrAo2vEBloPPjmMGzjHbxWC170R4ocTkWKJjwmsGeJ4dLX6RK5XO/ulDbpJju2GyLxTypkvdgEeBhLxuuYcX2PaGITjNLJxnG2wqv/0DbKz3PYhwSLEkes1iSPNhbmae6i2J7x3sXHb+bklaO9rv5e7z/T66qNljhVQ6e5Dtv6QwFyzZz/BCIpFI/I0gH7qJRCJRRQwsvHAL7HPjIcPGurtLV5TbcKoMUbGKLYJKYwaFLwqKs8AiimAFd3Zjn2d/D61Bx1WWJxeEjnUhsuHfxvv9N8gNRPEz3hSbgNAVpTt7DMIJDDeUyPucL4Y5WJiwbLGXXStWh8dwgYtaxWtKEfMLYVMgXgWVMOpZIyQxBMn05obomrKQplIIY/QGl4p7tIVK3rHAYthpfs8MxwLmNaOI/6otsVCktQlqaPjtS3q9/P5f5nm57Wf04TCmRoMlcXZs6WMAMWZQoRtwU6NfpKNn+I308KJCHGnvU2jy8mUWp0jShnqvR14JkX5eD4Y8QqsoKYQsP3gCYy2v0r6QO91EIpGoIga204XWx7jJkUTLpArLDUfAHoWdRGgzwiaTisk5TcFf0Tp8aAEGYHsOKSS9amr83y/sxLh7LvCJQ3aOuzPo6XZM6j8JUxK8oQgJEyRMBB2B47kTlqTrj/bdL5Nv3P1tneFz0zQjtkxhOedjGz3xeHrzTWbf/x3XPD7ikw+aXUqY0DMQZJM7Rvn8Nnf6rpRNVKUCrxm/Y9wU36X+sdk5t4+GnjVxjF27/J7hNaH98787wwfEfSlJx73XPaC3n+elxPQKuI7unRdbFwUtYebv2I8UT5c1rTERfKZct/qNutrs98zy5qyUDeB5l/Shx8PNWrnck4YjTnVXmbUDT768LYzZM9szqD8/7IxwzL6QO91EIpGoIvKhm0gkElXEgMILdZd45mHNE5PDMYdMdBekGW4g3a0Z4BwGFNrgMARxyBj/jqcecE3OQ672esSn1sMNlaSlHpI4B3Wn3+59gx9PZadScu4W5/uxm/JOnAaTAJy7q3pjGfD1tWea/Q60diHnkIm3kp7uxbrS7NfedqMf4JKmes9F3qrkN62xl8735CW67cPazKaK2ys+6e17GIa6TaeG76AKFnMqozd5OGF7g+85FhfJqY6gMtb7cPnA/0VwyQtj9FzlruqO93oih9xrvR8DFpLNbBNEbu8qxAau0KVmL9nuiTZJOrDeJ5Qhih6s5zqEz57fCT1jSZ9vfLfZw5DYfaDL129zg68DrptSp2Uq2HE6xyMusgXZUbYQk6RnFviP2/3++nDMvpA73UQikagi8qGbSCQSVcSAwgsnN6ActOF54RiKaDPzTV4o2/UwO3nAVLbMle5BGxuO+RSyuafoVrOfGBN5jQt3zTebbuHra71Fzbcv9HDDxItZaxz5lUjUagiqZxv/6CyMaa2Pmr2hlnXDBc4m3MbzUDJNRkRJcJwtUURPE6Juna0emiH3WoosllH1Hjoh8+ABuPo1oJc8dauHkCRp2anIXIORshNiXfWbXIXs160vDmN+acW/mL0UnWKhj62Dul09rbUhhhe4Xg8439c4lfjIdph6hpfKB8aPpF/J2089jvuSLJaHwHLp7YldjO/u8RZLO7/jE1rHMl9Ea0o83ZpGD7WwRVVPh4de2MqI64bqalIsk37TR79kdmifBJ7u8NrIZT+x2dsErb3a1/z914SP/Bm5000kEokqIh+6iUQiUUUMKLwwTe7uMiMqSVejXvPUPld5fqzGXUBmnE+BKvTTzd5LS4pZ0/frU2ZfO8/jC6frZrNDBlqSxrlJt5z9ytprvcaRHXIl6Vad4i8gnNCDaMHaeq99fanuNDu4/ZJ+qHPMpsg2+1h9RW8x+2x5GaUkfUseOtnR4u7WhhYvq7xfs8xmqbcknasfms2eaXQL6YKvfjuUz74IRoViSGj9eHdNGUp5rMHXIs9JkiZNdnYCWQG8g25ueIXZdxak+NjXq7fZx/zkClfNe9lk/63vk5eXl0JEX9A7zL5+o7NcPtTs33GmbjD7bU3ugktxfh99Jwo/XOtenaM87PTEqBjWm46Cn/lY88+b7FQDMms2yNfilUKrX0kvwn3D4geW/U7TI2aX5vdVmC/Ozfzwib8gd7qJRCJRRQxop8sn/vUFUYebN55u9oebXUhjM3YTTJD8EqWzj3TFMsppDb7jfgTcX3ar/bTeZ/ZBKO2UpIWLTjL7ilnOW3wSyQzyjUOHXMUdIBMLO+p9J0Ce7lVyXu7mIBAbd7JX6R/NZhKGu/yv6eIwJnedC/C//Y31fl4TxjhXsqQhS67kgu0+5uvrXQ1o9Y2+s6VG789/G8suX3zU3Wa3dDqv/KBGTzgxeVQS/6E3dzuUeo7VMrPZkTiUJhfGuGuNJ70+OtnX3iKsI3KeS/Nd6hy9Nz4Nsi/X3v/cc3z4zNo5K83euAVJKyTSmEgucZY/J+fpskv0jTe+1uzDz3AdZt6HzxR4urw3v7nozWa/GaXG7Dh8y4rYlmzbZJ9z6m9Lr9e+kDvdRCKRqCLyoZtIJBJVxIDCC4+DPxjK6yS1NrsLQc4mg9gsQ6VLPnRYDAW8WO5GfkT/bvbq2901XY1fOfqE1WFM6mPeOsuTYHQ92VakdUZ0nd6ir/gLEAnra/ETY0CfXFUmGaWYtKJbyTJgunzX6qwwZkwKLPADxrhJXinDPZI0We4WXljvalLUkOXK/PT1bzebJdJSdE0HgUPbeJ/zoM88wYnTs2pjopKufQuJuRC4o6tPlbLSa2PHtZtN5Sxy2X9yKzolF3DaqT82e3Ozh6YYZvpnXWH2/XMQGivglCasx3vcHFnn1/ShxiPCGJfie8nzf/SMF5hNvjFLdktJXJZRr17lz4eXzvLkHXm61AiXYthuwUpPoPanNJw73UQikagi8qGbSCQSVcSAwgsMBZRcPLoHLOudIe/mSVeL2cimmijfRb7wCP3B7MYT15vdudT94a1dkXfH9jAn6ldm/xJllQeMjeXJBEugBSGibZAd429/pVzYu539TxQF3UevdSWtEa3uWi3Si8zmXEqRpULRcio7lYSjibtWeH+j0yf7b2OZ8E9O9fDOyb1o81QbS43JCgjdmCGyXf8lLwOe1BDbGLe+1l97oB6tiUD5ZoiIbqgUwyAsQ2UoiyE50oknzYpKZ1SbO2nlQj+Aty7m6sXz7hbB9Xk5W/osdXMQKM3HbnCmh6Qg5H9sox9zzaQ3ms31eoK8HLfE5AivtbnJ8AOPrxkcef2h9Hr/RcZyp5tIJBLVxIB2ui9BAutQJhUUeYrcMXIXxX91/pOVxFPiGB6wH1rj/0IvPtobOpZ26HfOm2/2Yfgd3M2Nb243+0T840qFXQ54jBPWuijvtlbftTIhxaSBJJ0L3d+PtX7AbM4fPYtrNnmFmiS9apRX45EfzH957hRY3SRJOyb77o6JHCZAurHbuL3Wd7HUf5UKCT/mxZAbWj3ThWhLu6S75SIv5HDOrfE2Qlz/odGq4s729i7/bRsbnP/Ka3hAm28P2Y5GihWU6sIBcJruaPm7MEalMelZXDD1R/4B7GLXz/QKQUm6Wr6TZRKXXGpW45EHXUpc8poMm+6JMV73/fHc6PVf1PJFHPF27Qu5000kEokqIh+6iUQiUUUMKLzAcEJpK09X57A+39o/UOPcU4YX+PmT9fPwHXS3mOhhaOBwJJtK583yYyYA6f4ydFAq95y9BAkOurvo5nP0dV5mqXe6GE1JK5TtSZhsYyjg9F4X/9F7w5C64XQvMw1NiNHs951zvx4HAa4733mMdPl+hfmnW8n5ZvseKYqf6D4cwPDOSg/vfGzSeYUxXZWI7Y64PCfM9DGHToprjbzbUxuc78pSVnKDf9fsWsIPBcHj6CJPn/k/biPJy9BhaBGkWOr6LZS6XtDi4YWdiCY0bfeybEnaUO/zy3uZiWLeZ4MRphpe7JvleFW9h78YXggdh2udNy3Fa8JnTH/InW4ikUhUEfnQTSQSiSqiQnihXnunfL8nL597nb5b8QtW1bjCD91dZkSZ1S4pP7HslGWTbPnBTGOJ70ru5G3QwmUWmpzFj+mDYcxA7mD18RzYLtAWwiB0taTohtPFZtkvS13rm6LLp3fBZtK5ATYroGO1p/rO71/1ivNL941ZbbYpkgprxSVk9fDxHl+gQltJN5U4jrWuCFnsAcGkFHbimi+pb+0Nru8xD/g1GzFtQfjMklpfFw+v9xDEo4N9/d7WHLsrE7wmQVsY4ZwhUNVbOym2LWao8MqNrnq3e5UTYFvn+FyR7cTwhBTvoxfgM1RYYyiL8y/FZ8xPFrE0O+oRP4vc6SYSiUQVkQ/dRCKRqCIqhBdqpL0ye5vgApaKDJjtpbvLrfozcFGY2S25fE3IUD4BkWK6b8xGUjhdiipYdHfp2lOE+7H6SKied8T9/sKPcQDICgw3MMRBUr0UXVO62I/KVZoYmrn387Hg4jUzbzF7zfmHmE2mAVXfQjmuohD669AOidn0V673Ds7HjvHy8QdvnRu+44JTv+ovgC1ydJNP+M6p/v7tDfG8r+5z8v5ba77sB+AWeKTR4w0sC5Zitpwl5nSZqTb3yMyo4kZcK7TEfr8L5p/87evMvnu7d0LuvtSvuSQ1f9Vd96X3vNAPQBkw52bCCR0ijpzqz4NbVrlg+FgIp5M9QqYHnz9SfF78c+9/mP3vtd66iEp9VJqTpGVvRzn42eGQfSJ3uolEIlFFVNjpdmtvkcwj5H3uyW2VYrCd9g3IbrBpHHmnTHBJUisC4+QU8jv5fqm0mLsPNmykZu+Ieh+T5aGStGa87xbGNUCXE3/Kf2waYvbd8t1H6bzPQcNH7nw53+RvUqBIkm4933fYnIs79VKz2RxzUW/cGXT9wJMoMy7wNjghmXSf78weHOE720mnRpEXejzhkmBTv6jBdyuLSZxWFFyq1JiSiUu2efrTMTWw+78NmYy7u89Lkzs7orfCFko6103eE8fV+w7ysa9Gz4335ovmoBOlV/yHhOuaqXH3TBw9x7NxD/92ttnbjvL79NvgCp9VaLRKb3larXsS9L65vkul3HTAj5jn5eAFaZ8/I3e6iUQiUUXkQzeRSCSqiArhhaGS/uKCjYeSVqk7Ld1Vuu1M9Mzc7tvyxfWeNCiVI9LdvVmvNJvuBF3/Ei+SLhyPmRVqeB2luZj9JFxgeN3LZ40zmwmTX/Z6+GZlLUihkg65Cm1u3BtT+1TnJDMhWFIuq0Ubppf0eaJsMUq5mQja0R3dsUkXxHDA3qC28OiznNS8os/bIx28fGcY40NT/tVfeAkO8IbDmtvoa+/2ltjdmjzcjpnwmXFNOZ+ltcYSaLZUYlsm3gNLapyD+8yI2AH3B4gnXH+qh5lOkScq5y73ufjsFO+YK0W+e9BiZm4Tc8d2X5L06Y9c5vYJbl8+51/M5nw+ttbn7jetyI4qXpMrPvAhPwARo95/9xdK6nPtH23r9zsyvJBIJBJ/I8iHbiKRSFQRFcILB0uDT/qzdRDax7ykUB5HMDN4PbLp2+rdrWSGtJSxH73SFZlXTfKsNbf6FcsXJd0JHikzzHTH7gWpdiSltyTpd7BRcdu2fY3ZB9W7+3Zd7avNHtqLFiGSdAdstKSZ9nF3AakUxdY7UuRKH7zMXXlyK+eAO9nUFOeCLjL52wwZvfmOb/oA9Nci1VrnTflefHFvkBdNFbIo4hay/qNboAaOMbnWSplvhm94j4xe6t8xdKpfd7rYQ+vjunjhIifNvnAZSLRcm3gSvOjfYjiNa3zKB3z9Bto+puqk7WgZJMXrivV7yhwPtYRwDaptZ76PZGFpEK87KrlZ5k7OeGDFKMoZ7E8J+bPInW4ikUhUEfnQTSQSiSqicnHErr+4BIuRXS/1ZmI4gKphdKWYtWgkYBEAABYkSURBVKYo8Ra2PpXUg6ICfudBKJ5gGV/o5Kn4W1guy0IFCkfTrZSkpnnujk1a6Z1la+Aij/uwF0/UXu5u6PW1kM2SNPl7zkaguDfVkCjGXnKLOF8dUz1jzzEZoigVtHx6+WX+AqeLER+Ic6083ts1s6xYiq7nFMH9pfAT+4bFpq+6Y4r3DuN6Zql3KdNNcP3duAY1pI0uIs9wGLvw8pwk6bOznH3wnu0okYbu+XXTXWSeKnpSLK//7L+5An7dd/ABivlxvqVQ+r66xYtoWApPZkfNp/23l8KRx9Z6DGPNAhQtbfL7jvdEqQjsjbra7Ekv9Hv7TeETf0HudBOJRKKKqLDT/aO0Fzd3LYiPpYQU/w0PRNKArXQoUEFRl8AFlLSk/miz+U/PXSe1V0vJDQppPKGJZi/q893fgTXO/S3t7oK+K2Z7yPX4AAL+o5/0rUHN+Jg94vytwnmTN82dbkn8h/qiHDMmxfwcivqwlDkFZ3bPf7r9z40fxXnSA4r8YnZPfvkxXqZ6xyjfta4d5buiklgKE6pcO7Ne5jvdSlrPUnk3tjdubHmZ2S/Wr81u/EiPf2BeHOPb87w89vLjne9K3jk54p+81UVgJIkVzY8d5aXCC4733fLPJvmJleaXHiPL1O9YcZrZoyb7QvqC3mH2ND0SvmPDdE/asu3V8FF+n7J0u+QZN/dhQfeEQ/aJ3OkmEolEFZEP3UQikagiBtQNmK4VtSyl6AayvJbalnS1KikuSQVNWCR23qBvmc0wSEkRjMFyusgtNR6yYMIkdImVNHotMgds2opk0foLnTBIV4u2FBOADMf803ZP0ry1/itmv/kT4MNKqv03DwlRN/mV8o7CDMW8Zrnr8UrSg/O8h8+xN3ty485Gd/3J1+b8ljR7w9pBsm5Eq4dFGP75sC4PY65Z7OXHb57xWbPrXOZXffM8HFG6ZpuZHO52dTmGx1g+O/vVXlL926kTwndUalnF9XsTSum1PAwZuNF33fJys3ve5u/zHhpWaL11tVyveOGKk/wA0G6fnuzPoB9ucYW7x5uiat5IXWk2tYYZWuG6KIXgPrTqCv/eRxEyGuSJtb2RO91EIpGoIvKhm0gkElXEgMILDA2UlLWoYnXB8h+Z3THFOZ/kxJEPW8pSM/N6ppwGQNfzcJwTs5dSzFCyqyjPY22vuxNbayPftaPVfyuzpP9c621DWGp8lS4y+xgtCd/Ba0C38aL6q8wOzI3vhyH1Dz/Gi+RXbnfz2MEeKthR4GMeex/qPdF56Pib/sftaW7/ZryrR9UixCEV2qqgGpluIsNMXN+StMYvgY58CGyE5/M7PLxG9oMU19qbjnICMUMryxCi+MhUF9SnapkkLbsL7WS+5ubw77s63fOo5tcehtS493rMYc1ID73sqHfheTKXSq2LyIyBkKGGnPtHs9twYu9u+pzZZEdJkW3z0195ef2w2c7TPbveS9JLzKRjJvu9yCYH0j+FzzyL3OkmEolEFVFhpztcssof/3dk9ZhU2Jnij76507dBvY3+r85ge6naho0mK7XrKQlWENyR8B/119iBz6j1DEqpOo86qadtcXWalhY/Twb4mUAp7XSZqDlTN5jN+Xya/9ofD0NKrCxqqGC79K2GFlbV6qleaTRhNpoUYsyV470CjZVJpQpA6vpKl5nF6sYv6O1mP7496r3K8yVhnfA8uV5LetA890o85/uxg+fn7w7CwdKL5jlH+aGprsFLvjYFiJ78IsvJpC/prWZ/b855ZjducLLq1hb3LN4jT0JK8Rky4a2/NZvJeorR0HObKWQ2Fe+RSSd4IvJTch1mXg9636UxH9vIhGnudBOJROJvAvnQTSQSiSqiQnjhAGkvd7QFdaol158c2lsmHW82NTmZ5GLZ5BK5WyRFXdlttZ48okvNBAtFYaSYXGNCalOvvz+01kV1SkInr9zufNWnWvrXDmaiYRT4sReDbyhJo29C1gpu+snz3I2crMfN/v7pZ4Qx7z7d3dXPbndhkx/W/73Z5GKXEyYe4vny6e6qHrzQNXvJ/f3wXZ/xAaFNLEmHnomQg9MvdfxaJOt63N4Tc7YahETZxfq82VxLdIeZkJVieID3zPP6PCSxrcbXFu+7UqJnySbnPS8c5Ym1uSu9PQ8Tqsf9e+Sdn/Z9D4+d9ny39zgVW0MhPHVPH9RtJB38JbRdwhi/fZlzkCcs9bDUhHt+4h8A3ViShrf48+Ad+qLZUYSrsmgR79225nazEXEz5E43kUgkqoh86CYSiUQVUSG8METSmD9bLfqZvcsttiQdg+zicDAeyCud/YBnPH8/010n8n4laUOtZxfp0jEjygxye1/MzI6oiVnmvXFkrbsgdCtLZcB1UBGrm+lzMXmKu/pkHlAdbcT2AgH2B7ChdjR3AtxICMMNHeUuoBQzwHVow1Iz3ueXbiSz7ZL0k++eb/Zbz/uy2bPv8ozyy7/v2fdAViho3542yd3dh6d6O9qjr4SMmy9VDYpDimScF3/cOyO/RG5/TRebfQKJp4pKfCw7Pfh37nKfMN7HIG+3xNOVT6/mLsQ6QFucLWjf89qLboxj3gYbZJpBKGsf3YT1WtLTxXnwmKM+Dkcd4R5Bna6o9vVONy/a/g2zt9Z7TI4qe9TwLeFH68+peMyzyJ1uIpFIVBH50E0kEokqokJ4Yauk6/5sUREsktEjO4GKVMwMzprpItAUEKYCkBRdeZLJ+RmGKNbuiiLSbTXu55CE/ZXtbzH7yHonyZdaelwgL4GWV+Tq/C5kXqnX/kYvnb3r4heF72j7np/3uOu9pPGOVs9ic64YwpAKyli1ntoeoT+YzXXwh0ILoCPOc/eW8zV7k4cXmMXW+9xcPx4tXCU1bXdKw2G93kbouotdZJuKbKVCBoauSJRvg39MRTYWuJQQSoV/7Obc63zu5nYhVDA7jrn9a76fqv/EbgzqZhOm8zctXnYtSdO/5JJfQxbjAIaADoZdCAmFSBQJOh4h0lPznAF0yBUesiuQneK9Wc9TcFkBdmsuFVeRdXXAYP9xmG0/tp/3EolEIvEco8JO90Dt3aNjFgRZSm1HqNtJTmI7drKHLPV/qqFTPSnD8kRJmrLUGw6umOo7Gu50jwAPb0dt5JFSeIfJoaZ638Hvj7DJTnAGhyCPE2Q6XwMbO4XSXCxGqfF7ZnoDQs738Sudm6pJC8KYwbtYhAPQH5OJoDewF4+kb8nbx4SWPl7lq/86xcso+XnyjSVpcr3vPpjE4lxciaTXnVteGsY8rsm9KnoBLKclZ/kNfa7tLEmLanx7x3smlFmTP8y7lsklSV+udc9s1uXuUbIc/CFsEUuNKQ9s8ATTM8d7VvZS1ExzvX5bbwhjTkc2bu3l/kwhz/nDTzpf+47T3ZOb1cvFGney9FamrPXnCXWXS8+5jeCRn9zs6+Cn4RN/Qe50E4lEoorIh24ikUhUERXCCyMknf5n6xSoDJWUnuhqvvwm51v+4nRE8O9zs2Wqj/lMoeMwa+xqp7rbQ57pe9a6y31Lq5cml3Dabc75HHrKP5pNpbNS66Ih4Ld2Xu56owxpfAXze8XKD5ld6izLxKV+5ebECz30onvcbG2JbUXa6tv9Bejnsi0OQwXUUJakF0/xjrZb9Tw/APxM8rnP0/fMLpVqMizyFnlrIq5Nts1pbYpdjFm+/PR2d8uH1/uYTLoc/ADKXCW1zvLvCQk8hJ0evNWzimzFw47PknQDYkDX4Lc/dpOHpQ493e+71+m7Ycyb93oWSFFDluEFhuiorytJF0Pol658kAGALPPk8R5Sql8WU1hTFnn4YMrVbrN71+h3+mJ85ylfD2PynuBjqsj5/l/kTjeRSCSqiHzoJhKJRBVRIbzQq73r9OgOs5xRKrSDQWY1lA6Dj0m3saj4g609mQR0Y/7Y4t1WSwjuLuirzHzzO0p812nPdx4oXT66u5ybzkkejqAId+kzJB48hgmeV+tZ7BIWmHC99K+1rqzFlijMUu+MJAtNeB9Ey58PG9M3B1xsroNSqyi24xnZ6cyYtY2+3Pk7SiEiuu7T6z3bznXAEtJSdKylz135TTW4j2LzWQN556FNkSKPNKwd3JdU2St15aa495aNfg2eafb5Y6iA/O7SMbwPyQZ56hTn6TLU8spJsRN13SfwAm9VNgdn1PSaMGRscxVp4/tE7nQTiUSiiqiw092mvTvFfVnO/Stp3T6x3flr7dPbzOa/5ZFzPDnE3TQrwySFChx+hrvS99R4m5BvrEe3QUnjxrT7ec707do1eqPZT3b5d7yhIXJTH6jtvzKOngLbrrwGpUkLNkYe6dnN3kRv1lTfyd6mU81+c/03za57VxhStx//Kn8BoiRH/adnMo96PjKbMb8aq5EgsKLpblJLuBdbxlJz0cPkScN7Gl1DlkI81LG9ts/bJUlS3y7nX/fVus21xp3bw9NRUqW4aw+61PD+uAPnd+7PDnJFl+/Y3z3dt3/flbfeeaY37vrPqf2h2bs7vLTr0Ob+2xD9HrYk/Yu8Oeu9233n2t3hO/SzJ/p651wuqo9Vm/Pm+T1x3Vu9MnG+7jS7XS6IdexNyN5JCtRd0prv0D6RO91EIpGoIvKhm0gkElXEoD179uz7zUEte/buannBHnfpWvT78Bm2WeHWne7up/R+s5mIC4k5SS/pdHfhoUb3x5gIovtGt1KKSS1yYpmsYIsUlhpKkUNLl3mbPCnwCX2g3zE/0/Vv4Tu2NXiyrXElBEVZesykQWyeGoNO1ChlDosV0CVhk9fCBr14+fRxhQ/9BVM29c+tlKTvtJ5l9jQ9YvaRXT4Zmxo8+1FKznFd3CtvOUPXn+vkxdDbleJ1ZbjgpHsWmr1yjnccpkv9Q0Ut15fivmO4gWuRwj2f07vDmG8F75mdv1++yTn5q0d5B+iScAz57vxtV8lDgW+FUPAuLL65myAGJKlzlN8jnIujl/i6WD3dz3txQXSLIjpMoH5p0PsW79mzJ35QudNNJBKJqiIfuolEIlFFVGAvDJX0lwzwrL20daXYiudPn3DFJPLuqIV71F3IfMPD2z4p/i8Mgu7stOmeXayFGhI1eqmj+qfzdAYEXUCGKOhaMXMuSS/e7m7itnoPJ7Cr8WldSHmSr1nIiDZuqRBOQClycPMjkUOd4xGy+L5/x8rXurtLcC5LaGttN3vKPQgfuHRrKP8stWU5/yLXJ+4BC2DINW6PGez6u2NqCy2GEcaYMMe/4zfTXXeWIYpToMwnxRDQiEkoA8ZvnzTeS7XXt3hYhKW0kjThNvCgK4WMEBL61Jn/GsYc/QBqtdlqh3MldO7dhXMqnRc4tFtehvZeS6G7zHuE60aSvFNUKE/mPTMeHPItje1hSDKRSiHLfSF3uolEIlFF5EM3kUgkqogK4YVd0l4iwlSTGq/28Alm5CnyHMBGmyDJ1+8qNL6AOzBkgtsHNnhpJkMaJeUyhgvGPOmu5vm97launuIZThLBJWlDvR8zYS3cK7rMbFKM9zvPrxNR2+u/tf6/MF9ocxPKFymYLalxMHxPiG8197mw9Laa4f2+L0kHP+RqW3twzUKnWHaLmaPKwGpmF+P173S3vKR6RZClwnLaVWDrsABgcF+ByoEuuocwTEdGCVzuMVMRBkH7GUlaf4r/Vmb5m7d7W6cd9f1n+CVp40ycCHLzZOsUS/g5JopcahGjIHtEaBKtl7m5B2L4JbRuh7IewiSDULgzuSF2JK+p8evKjsEFXbI/I3e6iUQiUUVU2Ok+I2n9ny0GoA/vQhJM0oYG/6ef8GSFgH6JJ1rpDL8DGwH8SadHjVjDrpiFGV2LrRZ0fplomFDrv6tlVEwS9A3u/z9t9ct8J7wWKiRbJ3lCijt2SdqK1kP192EXRC0U7q7Bly2+hrk4+Nu+ax1+un/nIO5apbC7G8QE33/CfiNsbrxKXOAKfOERLbHx5N44sCd6VX0NPig1pCmyQ07nwf8e9XTDrp5gnpdOFOw/ToiCTmPuwjqAR7MTXlXjUtwTU10MSIoJUibSD9ng9sMt7j1+D6XGpTGpycvy75A4QxPPQQWxoMb6QtZ1b/Bex2Pt4EviNTy2DjdSBZGivZE73UQikagi8qGbSCQSVUSF8ILjCl1q9lcadoRjGDxvH99m9p1ypazvdb7JB2DoIHZQ0UoE0ydxa8/kEPM6sXo5JiPokVAsCm1v6ti+Q5IOhrsKDvLiVndFmVRgCeTDT0Td1GUT4Sd2wa38CD5At7wU3oFK2Jab3G5CxfMg8jVjDiZyOmH/FBzkV0RxrspgnhF5n/ouXI9Kbr6k0TwI3u7W17p7HJJzcH8laQfmdyjmYudytzsx381Y7wevLoQwyNfGPTGEYSYsm8bV0SVvbEYIjYfgHmhv8bVJbqskXY040uFr3Lf/yjhvkzXlXS4B0AwObknhbg9DW+y6jbnYibU45O/jmOF7Svf/PpA73UQikagi8qGbSCQSVcSAwgssddvQFfuy9Gz2st9XTbze7NAShfxLuLIlTIJHvRKu/iRvqhtLYUsZe7oLB8Omu0ZXttCWJVRJ4zPMzD4g79D68MNQa49V15qy08tntyD00oRSWLpBj9Htl3QkuNJNOI2FmO+5OD7M975e2wsvIvOAfMyTYZeuIat4eU3oUnP1F/iu4byxLiZ2+fw/04D1Te61pKEVwmFDEJ55BBGOkxgWKZW+cj3zOymoxvdL80vdeH4G80lJgG/r9WHIwAde5UyM8ePa/Su4ThA6YBsiSVqFdTGJ1/QENxe5kJnmxirryAoq3Ef7Qu50E4lEoorIh24ikUhUEQMKL6y+6yh/oSQmNcxF0U/XzWazVJB4EmyFTpatSpqBEtFH8P6y17l9ItzGoVTakqKrNAw2Z4quaym8UCEEwZLHWma+MZdaPih+B9ypJhDnV8L1bIR7Br0lSdJPwSB5BVxkTtUvXu32iYXOqINYngmXugnt9v4brIpZyOi/4JXxOwZ8zSq9L8WQA67p1gb/0onoMMx+Z5K0A+GZe7DGTwLr4o/4/C/g/s4phEWGzsMLFVTFAkpkf36Grj5CFqNAGxrGkl4VmhTcgAPg+jehF9mTCC+MLbCdyO1Y6BFPzZ3iNn/6dwoMlJPwWvP+lKn/L3Knm0gkElXEgHa6IWBPURJJB4z24Dm1bIPWKv6lxyMps4aBckmP4Tzm4v1rYC9D8ug9hX+uIRfiBew2Ag+Pu4vSTHLHh13oDJBkyXG+aaJH5zs2UyVGWt7qbW6mTPPEzmE47/tZJcwdkaTvIIn1CyQJTsLv+G8kbb5YkKU9D7u5pvfiAPByj8BOF5tDNd+sgGbufrlz5U640s5XiusAxzAx3McDCvfIUIi07MLO6xdI9JyE8/whyt65viXpHeRfY8cYSovJZS8IIRU9gb2BueJ6Lolf3dx3ur8w380gJIV1Mh7b0kcKScUX4LwWY34XwYuahh38IwWv4Kuwz+MC7Qe5000kEokqIh+6iUQiUUVU6AY86ClJa/Z5QCKRSCRKGLdnz55DSm/0+9BNJBKJxHOLDC8kEolEFZEP3UQikagi8qGbSCQSVUQ+dBOJRKKKyIduIpFIVBH/D999JNBhSogTAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# plot it \n",
    "fig = plt.imshow(spec, origin='lower', aspect='auto')\n",
    "fig.set_cmap('jet')\n",
    "fig.axes.get_xaxis().set_visible(False)\n",
    "fig.axes.get_yaxis().set_visible(False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Standardization\n",
    "\n",
    "<b>Always standardize</b> the data before feeding it into the Neural Network! (unless you use BatchNormalization in your Neural Network)\n",
    "\n",
    "We use <b>Zero-mean Unit-variance standardization</b> (also known as Z-score normalization).\n",
    "Here, we use <b>attribute-wise standardization</b>, i.e. each pixel is standardized individually, as opposed to computing a single mean and single standard deviation of all values.\n",
    "\n",
    "('Flat' standardization would also be possible, but we have seen benefits of attribute-wise standardization in our experiments).\n",
    "\n",
    "We use the StandardScaler from the scikit-learn package for our purpose.\n",
    "As it works typically on vector data, we have to vectorize (i.e. reshape) our matrices first, and then reshape again to the original shape. We created a convenience function for that:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "def standardize(data):\n",
    "    # vectorize before standardization (cause scaler can't do it in that format)\n",
    "    N, ydim, xdim = data.shape\n",
    "    data = data.reshape(N, xdim*ydim)\n",
    "\n",
    "    # standardize\n",
    "    scaler = preprocessing.StandardScaler()\n",
    "    data = scaler.fit_transform(data)\n",
    "\n",
    "    # reshape to original shape\n",
    "    return data.reshape(N, ydim, xdim)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(1703, 80, 80)"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "spectrograms = standardize(spectrograms)\n",
    "spectrograms.shape # verify that the shape is again the same as before"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load the Metadata"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(1680, 1)"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# use META_FILE_PATTERN to load the correct metadata file. set correct METADATA_PATH above\n",
    "task = 'instrumental'\n",
    "csv_file = LABEL_FILE_PATTERN % task\n",
    "\n",
    "metadata = pd.read_csv(csv_file, index_col=0) #, sep='\\t')\n",
    "metadata.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>instrumental</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>clip_id</th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>37</th>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>40</th>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>172</th>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>198</th>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>253</th>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "         instrumental\n",
       "clip_id              \n",
       "37                0.0\n",
       "40                0.0\n",
       "172               1.0\n",
       "198               0.0\n",
       "253               0.0"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "metadata.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "instrumental    420.0\n",
       "dtype: float64"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# how many instrumental tracks\n",
    "metadata.sum()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "instrumental    1260.0\n",
       "dtype: float64"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# how many vocal tracks\n",
    "(1-metadata).sum()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.75"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# baseline:\n",
    "1260/len(metadata)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Align Metadata and Spectrograms"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1680"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(metadata)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1680"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# check if we find all metadata clip ids in our spectrogram data\n",
    "len(set(metadata.index).intersection(set(spec_clip_ids)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(1703, 80, 80)"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# we may have more spectrograms than metadata\n",
    "spectrograms.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create Train X and Y: data and classes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**get the correct spectrogram indices given the metadata's clip_ids in a sorted way**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "meta_clip_ids = metadata.index\n",
    "spec_indices = spectrograms_clip_ids.loc[meta_clip_ids]['spec_id']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**and select a correctly sorted subset of the original spectrograms for this task**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(1680, 80, 80)"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data = spectrograms[spec_indices,:]\n",
    "data.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[0.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       ...,\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [1.]])"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# for training convert from Pandas DataFrame to numpy array\n",
    "classes = metadata.values\n",
    "classes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "# number of classes is number of columns in metaddata\n",
    "n_classes = metadata.shape[1]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Convolutional Neural Networks\n",
    "\n",
    "A Convolutional Neural Network (ConvNet or CNN) is a type of (Deep) Neural Network that is well-suited for 2D axes data, such as images or spectrograms, as it is optimized for learning from spatial proximity. Its core elements are 2D filter kernels which essentially learn the weights of the Neural Network, and down-scaling functions such as Max Pooling.\n",
    "\n",
    "A CNN can have one or more Convolution layers, each of them having an arbitrary number of N filters (which define the depth of the CNN layer), typically followed by a pooling step, which aggregates neighboring pixels together and thus reduces the image resolution by retaining only the average or maximum values of neighboring pixels."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Preparing the Data\n",
    "\n",
    "### Adding the channel\n",
    "\n",
    "As CNNs were initially made for image data, we need to add a dimension for the color channel to the data. RGB images typically have a 3rd dimension with the color. \n",
    "\n",
    "<b>Spectrograms, however, are considered like greyscale images, as in the previous tutorial.\n",
    "Likewise we need to add an extra dimension for compatibility with the CNN implementation.</b>\n",
    "\n",
    "For greyscale images, we add the number 1 as the depth of the additional dimension of the input shape (for RGB color images, the number of channels is 3)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "def add_channel(data, n_channels=1):\n",
    "    # n_channels: 1 for grey-scale, 3 for RGB, but usually already present in the data\n",
    "    \n",
    "    N, ydim, xdim = data.shape\n",
    "\n",
    "    if keras.backend.image_data_format() == 'channels_last':  # TENSORFLOW\n",
    "        # Tensorflow ordering (~/.keras/keras.json: \"image_dim_ordering\": \"tf\")\n",
    "        data = data.reshape(N, ydim, xdim, n_channels)\n",
    "    else: # THEANO\n",
    "        # Theano ordering (~/.keras/keras.json: \"image_dim_ordering\": \"th\")\n",
    "        data = data.reshape(N, n_channels, ydim, xdim)\n",
    "        \n",
    "    return data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(1680, 80, 80)"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(1680, 80, 80, 1)"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data = add_channel(data, n_channels=1)\n",
    "data.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(80, 80, 1)"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# we store the new shape of the images in the 'input_shape' variable.\n",
    "# take all dimensions except the 0th one (which is the number of files)\n",
    "input_shape = data.shape[1:]  \n",
    "input_shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Train & Test Set Split\n",
    "\n",
    "We split the original full data set into two parts: Train Set (75%) and Test Set (25%).\n",
    "\n",
    "Note: \n",
    "For demo purposes we use only 1 split here. A better way to do it is to use **Cross-Validation**, doing the split multiple times, iterating training and testing over the splits and averaging the results."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [],
   "source": [
    "# use 75% of data for train, 25% for test set\n",
    "testset_size = 0.25"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Stratified Split retains the class balance in both sets\n",
    "\n",
    "splitter = StratifiedShuffleSplit(n_splits=1, test_size=testset_size, random_state=0)\n",
    "splits = splitter.split(data, classes)\n",
    "\n",
    "for train_index, test_index in splits:\n",
    "    #print(\"TRAIN INDEX:\", train_index)\n",
    "    #print(\"TEST INDEX:\", test_index)\n",
    "    #print(\"# of instances TRAIN:\", len(train_index))\n",
    "    #print(\"# of instances TEST:\", len(test_index))\n",
    "    train_set = data[train_index]\n",
    "    test_set = data[test_index]\n",
    "    train_classes = classes[train_index]\n",
    "    test_classes = classes[test_index]\n",
    "# Note: this for loop is only executed once if n_splits==1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(1260, 80, 80, 1)\n",
      "(420, 80, 80, 1)\n"
     ]
    }
   ],
   "source": [
    "print(train_set.shape)\n",
    "print(test_set.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Creating Neural Network Models in Keras\n",
    "\n",
    "## Sequential Models\n",
    "\n",
    "In Keras, one can choose between a Sequential model and a Graph model. Sequential models are simple concatenations of layers. Graph models can also handle those but also more complex neural network architectures. Keras now recommends to use the Graph models by default, but for a simple entry into the topic we are going to start with Sequential models first:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercise:** Try different configurations by uncommenting various lines of code in the following code box:\n",
    "* 1 Layer CNN\n",
    "* add 2nd Layer\n",
    "* increase number of conv_filters\n",
    "* add Dropout\n",
    "\n",
    "Observe how the number of parameters in the model changes, and also the speed of training."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [],
   "source": [
    "#np.random.seed(0) # make results repeatable\n",
    "\n",
    "model = Sequential()\n",
    "\n",
    "conv_filters = 16   # number of convolution filters (= CNN depth)\n",
    "# UNCOMMENT TO INCREASE FILTERS\n",
    "#conv_filters = 32   # number of convolution filters (= CNN depth)\n",
    "\n",
    "# 1st Layer\n",
    "model.add(Convolution2D(conv_filters, (3, 3), input_shape=input_shape))\n",
    "model.add(MaxPooling2D(pool_size=(2, 2))) \n",
    "\n",
    "# # UNCOMMENT TO ADD 2nd LAYEER\n",
    "#model.add(Convolution2D(conv_filters, (3, 3)))\n",
    "#model.add(MaxPooling2D(pool_size=(2, 2))) \n",
    "\n",
    "# UNCOMMENT TO ADD DROPOUT\n",
    "#model.add(Dropout(0.25)) \n",
    "\n",
    "# After Convolution, we have a conv_filters*y*x matrix output\n",
    "# In order to feed this to a Full (Dense) layer, we need to flatten all data\n",
    "# Note: Keras does automatic shape inference, i.e. it knows how many (flat) input units the next layer will need,\n",
    "# so no parameter is needed for the Flatten() layer.\n",
    "model.add(Flatten()) \n",
    "\n",
    "# Full layer\n",
    "model.add(Dense(256, activation='sigmoid')) \n",
    "\n",
    "# Output layer\n",
    "# For binary/2-class problems use ONE sigmoid unit, \n",
    "# for multi-class/multi-label problems use n output units and activation='softmax!'\n",
    "model.add(Dense(n_classes,activation='sigmoid'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Model.summary() gives a nice overview of the model architecture and the number of weights (parameters) in the NN**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "conv2d_1 (Conv2D)            (None, 78, 78, 16)        160       \n",
      "_________________________________________________________________\n",
      "max_pooling2d_1 (MaxPooling2 (None, 39, 39, 16)        0         \n",
      "_________________________________________________________________\n",
      "flatten_1 (Flatten)          (None, 24336)             0         \n",
      "_________________________________________________________________\n",
      "dense_1 (Dense)              (None, 256)               6230272   \n",
      "_________________________________________________________________\n",
      "dense_2 (Dense)              (None, 1)                 257       \n",
      "=================================================================\n",
      "Total params: 6,230,689\n",
      "Trainable params: 6,230,689\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Training the CNN"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We have to define:\n",
    "\n",
    "* loss function: binary crossentropy for binary or multi-label problems, categorical crossentropy for single class problems (custom loss functions are also possible)\n",
    "* optimizer: classic Stochastic Gradient Descent, or derivations thereof (e.g. Adam, ...)\n",
    "* metric: one or multiple metrics for evaluation on the train, validation and test sets\n",
    "* epochs: number of iterations to train the network (in the default case, in each epoch the full dataset is presented once to the network)\n",
    "* batch_size: how many instances are presented as one batch to the network, before a weight update (= Back Propagation) takes place"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define a loss function \n",
    "loss = 'binary_crossentropy'  # 'categorical_crossentropy' for multi-class problems\n",
    "\n",
    "# Optimizer = Stochastic Gradient Descent\n",
    "optimizer = 'sgd' \n",
    "\n",
    "# Which metric to evaluate\n",
    "metrics = ['accuracy']\n",
    "\n",
    "# Batch size\n",
    "batch_size = 32\n",
    "\n",
    "# Compiling the model\n",
    "model.compile(loss=loss, optimizer=optimizer, metrics=metrics)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/10\n",
      "1260/1260 [==============================] - 3s 3ms/step - loss: 0.5481 - acc: 0.7516\n",
      "Epoch 2/10\n",
      "1260/1260 [==============================] - 3s 2ms/step - loss: 0.5137 - acc: 0.7675\n",
      "Epoch 3/10\n",
      "1260/1260 [==============================] - 3s 2ms/step - loss: 0.4886 - acc: 0.7802\n",
      "Epoch 4/10\n",
      "1260/1260 [==============================] - 3s 2ms/step - loss: 0.4770 - acc: 0.8008\n",
      "Epoch 5/10\n",
      "1260/1260 [==============================] - 3s 2ms/step - loss: 0.4642 - acc: 0.8016\n",
      "Epoch 6/10\n",
      "1260/1260 [==============================] - 3s 2ms/step - loss: 0.4517 - acc: 0.8119\n",
      "Epoch 7/10\n",
      "1260/1260 [==============================] - 3s 2ms/step - loss: 0.4455 - acc: 0.8206\n",
      "Epoch 8/10\n",
      "1260/1260 [==============================] - 3s 2ms/step - loss: 0.4352 - acc: 0.8198\n",
      "Epoch 9/10\n",
      "1260/1260 [==============================] - 3s 2ms/step - loss: 0.4275 - acc: 0.8222\n",
      "Epoch 10/10\n",
      "1260/1260 [==============================] - 3s 3ms/step - loss: 0.4161 - acc: 0.8341\n"
     ]
    }
   ],
   "source": [
    "# TRAINING the model\n",
    "# (execute multiple times to train more epochs)\n",
    "epochs = 10\n",
    "history = model.fit(train_set, train_classes, batch_size=batch_size, epochs=epochs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Verifying Accuracy on Test Set"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [],
   "source": [
    "# always execute this, and then one of the boxes of accuracy_score below to print the result\n",
    "test_pred = model.predict_classes(test_set)\n",
    "# Note: we use model.predict_classes (only available in the Sequential model) to already round the prediction value to 0 or 1\n",
    "# model.predict(test_set) gives you the raw values\n",
    "#test_pred = model.predict(test_set)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [],
   "source": [
    "# show first 10 predictions\n",
    "#test_pred[0:10]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.8119047619047619"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 1 layer\n",
    "accuracy_score(test_classes, test_pred)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.8119047619047619"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 2 layers\n",
    "accuracy_score(test_classes, test_pred)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.8119047619047619"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 2 layers + 32 convolution filters\n",
    "accuracy_score(test_classes, test_pred)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.8119047619047619"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 2 layer + 32 convolution filters + Dropout\n",
    "accuracy_score(test_classes, test_pred)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Additional Parameters & Techniques\n",
    "\n",
    "**Exercise:** Try out more parameters and techniques: comment/uncomment appropriate lines of code below:\n",
    "* add ReLU activation\n",
    "* add Batch normalization\n",
    "* add Dropout on multiple layers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = Sequential()\n",
    "\n",
    "conv_filters = 16   # number of convolution filters (= CNN depth)\n",
    "filter_size = (3,3)\n",
    "pool_size = (2,2)\n",
    "\n",
    "# Layer 1\n",
    "model.add(Convolution2D(conv_filters, filter_size, padding='valid', input_shape=input_shape))\n",
    "#model.add(BatchNormalization())\n",
    "#model.add(Activation('relu')) \n",
    "model.add(MaxPooling2D(pool_size=pool_size)) \n",
    "#model.add(Dropout(0.3))\n",
    "\n",
    "# Layer 2\n",
    "model.add(Convolution2D(conv_filters, filter_size, padding='valid', input_shape=input_shape))\n",
    "#model.add(BatchNormalization())\n",
    "#model.add(Activation('relu')) \n",
    "model.add(MaxPooling2D(pool_size=pool_size)) \n",
    "#model.add(Dropout(0.1))\n",
    "\n",
    "# In order to feed this to a Full(Dense) layer, we need to flatten all data\n",
    "model.add(Flatten()) \n",
    "\n",
    "# Full layer\n",
    "model.add(Dense(256))  \n",
    "#model.add(Activation('relu'))\n",
    "#model.add(Dropout(0.1))\n",
    "\n",
    "# Output layer\n",
    "# For binary/2-class problems use ONE sigmoid unit, \n",
    "# for multi-class/multi-label problems use n output units and activation='softmax!'\n",
    "model.add(Dense(n_classes,activation='sigmoid'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "conv2d_2 (Conv2D)            (None, 78, 78, 16)        160       \n",
      "_________________________________________________________________\n",
      "max_pooling2d_2 (MaxPooling2 (None, 39, 39, 16)        0         \n",
      "_________________________________________________________________\n",
      "conv2d_3 (Conv2D)            (None, 37, 37, 16)        2320      \n",
      "_________________________________________________________________\n",
      "max_pooling2d_3 (MaxPooling2 (None, 18, 18, 16)        0         \n",
      "_________________________________________________________________\n",
      "flatten_2 (Flatten)          (None, 5184)              0         \n",
      "_________________________________________________________________\n",
      "dense_3 (Dense)              (None, 256)               1327360   \n",
      "_________________________________________________________________\n",
      "dense_4 (Dense)              (None, 1)                 257       \n",
      "=================================================================\n",
      "Total params: 1,330,097\n",
      "Trainable params: 1,330,097\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "model.summary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Compile the model\n",
    "model.compile(loss=loss, optimizer=optimizer, metrics=metrics)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/10\n",
      "1260/1260 [==============================] - 3s 3ms/step - loss: 0.5737 - acc: 0.7389\n",
      "Epoch 2/10\n",
      "1260/1260 [==============================] - 3s 3ms/step - loss: 0.5063 - acc: 0.7722\n",
      "Epoch 3/10\n",
      "1260/1260 [==============================] - 3s 2ms/step - loss: 0.4759 - acc: 0.7849\n",
      "Epoch 4/10\n",
      "1260/1260 [==============================] - 4s 3ms/step - loss: 0.4527 - acc: 0.7968\n",
      "Epoch 5/10\n",
      "1260/1260 [==============================] - 4s 3ms/step - loss: 0.4437 - acc: 0.8008\n",
      "Epoch 6/10\n",
      "1260/1260 [==============================] - 3s 3ms/step - loss: 0.4280 - acc: 0.8111\n",
      "Epoch 7/10\n",
      "1260/1260 [==============================] - 3s 2ms/step - loss: 0.4069 - acc: 0.8254\n",
      "Epoch 8/10\n",
      "1260/1260 [==============================] - 3s 2ms/step - loss: 0.3942 - acc: 0.8302\n",
      "Epoch 9/10\n",
      "1260/1260 [==============================] - 3s 3ms/step - loss: 0.3892 - acc: 0.8238\n",
      "Epoch 10/10\n",
      "1260/1260 [==============================] - 4s 3ms/step - loss: 0.3725 - acc: 0.8389\n"
     ]
    }
   ],
   "source": [
    "# Train the model\n",
    "epochs = 10\n",
    "history = model.fit(train_set, train_classes, batch_size=32, epochs=epochs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.7761904761904762"
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Verify Accuracy on Test Set\n",
    "test_pred = model.predict_classes(test_set)\n",
    "accuracy_score(test_classes, test_pred)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 2) Genre Classification\n",
    "\n",
    "In this Genre classification task, we have multiple classes, but the decision has to be made for 1 target class.\n",
    "This is called a single-label / multi-class task (as opposed to a multi-label task)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load Audio Spectrograms\n",
    "\n",
    "We prepared already the Mel spectrograms for the audio files used in this task."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(1998, 80, 80)"
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "task = 'genres'\n",
    "\n",
    "# load Mel spectrograms\n",
    "spectrogram_file = SPECTROGRAM_FILE_PATTERN % task\n",
    "spectrograms, spectrograms_clip_ids = load_spectrograms(spectrogram_file)\n",
    "\n",
    "# standardize\n",
    "data = standardize(spectrograms)\n",
    "data.shape # verify the shape of the loaded & standardize spectrograms"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load Metadata"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(1998, 8)"
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# use META_FILE_PATTERN to load the correct metadata file. set correct METADATA_PATH above\n",
    "csv_file = LABEL_FILE_PATTERN % task\n",
    "metadata = pd.read_csv(csv_file, index_col=0) #, sep='\\t')\n",
    "metadata.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>classical</th>\n",
       "      <th>country</th>\n",
       "      <th>jazz</th>\n",
       "      <th>pop</th>\n",
       "      <th>rock</th>\n",
       "      <th>techno</th>\n",
       "      <th>blues</th>\n",
       "      <th>dance</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>clip_id</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>41797</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>38338</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>34335</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25542</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>38344</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "         classical  country  jazz  pop  rock  techno  blues  dance\n",
       "clip_id                                                           \n",
       "41797            0        0     0    0     1       0      0      0\n",
       "38338            1        0     0    0     0       0      0      0\n",
       "34335            0        0     0    0     1       0      0      0\n",
       "25542            1        0     0    0     0       0      0      0\n",
       "38344            0        0     0    0     0       1      0      0"
      ]
     },
     "execution_count": 49,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "metadata.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "classical    999\n",
       "country       72\n",
       "jazz          90\n",
       "pop           94\n",
       "rock         379\n",
       "techno       341\n",
       "blues          9\n",
       "dance         14\n",
       "dtype: int64"
      ]
     },
     "execution_count": 50,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# how many tracks per genre\n",
    "metadata.sum()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Baseline:\n",
    "\n",
    "A 'dumb' classifier could assign all predictions to the biggest class. The number of tracks belonging to the biggest class divided by the total number of tracks in the dataset is our baseline accuracy in %."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.5"
      ]
     },
     "execution_count": 51,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# baseline: \n",
    "metadata.sum().max() / len(metadata)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Align Metadata and Spectrograms"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0"
      ]
     },
     "execution_count": 52,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# check if we find all metadata clip ids in our spectrogram data\n",
    "len(set(metadata.index).intersection(set(spectrograms_clip_ids)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(1998, 80, 80)"
      ]
     },
     "execution_count": 53,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "spec_indices = spectrograms_clip_ids.loc[metadata.index]['spec_id']\n",
    "data = spectrograms[spec_indices,:]\n",
    "data.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create Train X and Y: data and classes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[0, 0, 0, ..., 0, 0, 0],\n",
       "       [1, 0, 0, ..., 0, 0, 0],\n",
       "       [0, 0, 0, ..., 0, 0, 0],\n",
       "       ...,\n",
       "       [0, 0, 1, ..., 0, 0, 0],\n",
       "       [0, 0, 0, ..., 1, 0, 0],\n",
       "       [0, 0, 1, ..., 0, 0, 0]])"
      ]
     },
     "execution_count": 54,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# classes needs to be a \"1-hot encoded\" numpy array (which our groundtruth already is! we just convert pandas to numpy)\n",
    "classes = metadata.values\n",
    "classes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [],
   "source": [
    "n_classes = metadata.shape[1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(1998, 80, 80, 1)"
      ]
     },
     "execution_count": 56,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# add channel (see above)\n",
    "data = add_channel(data)\n",
    "data.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(80, 80, 1)"
      ]
     },
     "execution_count": 57,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# input_shape: we store the new shape of the images in the 'input_shape' variable.\n",
    "# take all dimensions except the 0th one (which is the number of files)\n",
    "input_shape = data.shape[1:]  \n",
    "input_shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Train & Test Set Split\n",
    "\n",
    "We split the original full data set into two parts: Train Set (75%) and Test Set (25%)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [],
   "source": [
    "testset_size = 0.25 # % portion of whole data set to keep for testing, i.e. 75% is used for training"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Stratified Split retains the class balance in both sets\n",
    "\n",
    "splitter = StratifiedShuffleSplit(n_splits=1, test_size=testset_size, random_state=0)\n",
    "splits = splitter.split(data, classes)\n",
    "\n",
    "for train_index, test_index in splits:\n",
    "    train_set = data[train_index]\n",
    "    test_set = data[test_index]\n",
    "    train_classes = classes[train_index]\n",
    "    test_classes = classes[test_index]\n",
    "# Note: this for loop is only executed once if n_splits==1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(1498, 80, 80, 1)\n",
      "(500, 80, 80, 1)\n"
     ]
    }
   ],
   "source": [
    "print(train_set.shape)\n",
    "print(test_set.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Model: Compact CNN\n",
    "\n",
    "This is a 5 layer Convolutional Neural Network inspired and adapted from Keunwoo Choi (https://github.com/keunwoochoi/music-auto_tagging-keras)\n",
    "\n",
    "* It is specified using Keras' functional Model **Graph API** (https://keras.io/models/model/).\n",
    "* It allows to specify 3, 4 or 5 Convolutional Layers.\n",
    "* It adapts the Pooling sizes according to the number of Mel bands use in the input.\n",
    "* It uses Batch Normalization."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [],
   "source": [
    "def CompactCNN(input_shape, nb_conv, nb_filters, normalize, nb_hidden, dense_units, \n",
    "               output_shape, activation, dropout, multiple_segments=False, input_tensor=None):\n",
    "    \n",
    "    melgram_input = Input(shape=input_shape)\n",
    "    \n",
    "    n_mels = input_shape[0]\n",
    "\n",
    "    if n_mels >= 256:\n",
    "        poolings = [(2, 4), (4, 4), (4, 5), (2, 4), (4, 4)]\n",
    "    elif n_mels >= 128:\n",
    "        poolings = [(2, 4), (4, 4), (2, 5), (2, 4), (4, 4)]\n",
    "    elif n_mels >= 96:\n",
    "        poolings = [(2, 4), (3, 4), (2, 5), (2, 4), (4, 4)]\n",
    "    elif n_mels >= 72:\n",
    "        poolings = [(2, 4), (3, 4), (2, 5), (2, 4), (3, 4)]\n",
    "    elif n_mels >= 64:\n",
    "        poolings = [(2, 4), (2, 4), (2, 5), (2, 4), (4, 4)]\n",
    "\n",
    "    # Determine input axis\n",
    "    if keras.backend.image_dim_ordering() == 'th':\n",
    "        channel_axis = 1\n",
    "        freq_axis = 2\n",
    "        time_axis = 3\n",
    "    else:\n",
    "        channel_axis = 3\n",
    "        freq_axis = 1\n",
    "        time_axis = 2\n",
    "            \n",
    "    # Input block\n",
    "    #x = BatchNormalization(axis=time_axis, name='bn_0_freq')(melgram_input)\n",
    "        \n",
    "    if normalize == 'batch':\n",
    "        x = BatchNormalization(axis=freq_axis, name='bn_0_freq')(melgram_input)\n",
    "    elif normalize in ('data_sample', 'time', 'freq', 'channel'):\n",
    "        x = Normalization2D(normalize, name='nomalization')(melgram_input)\n",
    "    elif normalize in ('no', 'False'):\n",
    "        x = melgram_input\n",
    "\n",
    "    # Conv block 1\n",
    "    x = Convolution2D(nb_filters[0], (3, 3), padding='same')(x)\n",
    "    x = BatchNormalization(axis=channel_axis, name='bn1')(x)\n",
    "    x = ELU()(x)\n",
    "    x = MaxPooling2D(pool_size=poolings[0], name='pool1')(x)\n",
    "        \n",
    "    # Conv block 2\n",
    "    x = Convolution2D(nb_filters[1], (3, 3), padding='same')(x)\n",
    "    x = BatchNormalization(axis=channel_axis, name='bn2')(x)\n",
    "    x = ELU()(x)\n",
    "    x = MaxPooling2D(pool_size=poolings[1], name='pool2')(x)\n",
    "        \n",
    "    # Conv block 3\n",
    "    x = Convolution2D(nb_filters[2], (3, 3), padding='same')(x)\n",
    "    x = BatchNormalization(axis=channel_axis, name='bn3')(x)\n",
    "    x = ELU()(x)\n",
    "    x = MaxPooling2D(pool_size=poolings[2], name='pool3')(x)\n",
    "    \n",
    "    # Conv block 4\n",
    "    if nb_conv > 3:        \n",
    "        x = Convolution2D(nb_filters[3], (3, 3), padding='same')(x)\n",
    "        x = BatchNormalization(axis=channel_axis, name='bn4')(x)\n",
    "        x = ELU()(x)   \n",
    "        x = MaxPooling2D(pool_size=poolings[3], name='pool4')(x)\n",
    "        \n",
    "    # Conv block 5\n",
    "    if nb_conv == 5:\n",
    "        x = Convolution2D(nb_filters[4], (3, 3), padding='same')(x)\n",
    "        x = BatchNormalization(axis=channel_axis, name='bn5')(x)\n",
    "        x = ELU()(x)\n",
    "        x = MaxPooling2D(pool_size=poolings[4], name='pool5')(x)\n",
    "\n",
    "    # Flatten the outout of the last Conv Layer\n",
    "    x = Flatten()(x)\n",
    "      \n",
    "    if nb_hidden == 1:\n",
    "        x = Dropout(dropout)(x)\n",
    "        x = Dense(dense_units, activation='relu')(x)\n",
    "    elif nb_hidden == 2:\n",
    "        x = Dropout(dropout)(x)\n",
    "        x = Dense(dense_units[0], activation='relu')(x)\n",
    "        x = Dropout(dropout)(x)\n",
    "        x = Dense(dense_units[1], activation='relu')(x) \n",
    "    else:\n",
    "        raise ValueError(\"More than 2 hidden units not supported at the moment.\")\n",
    "    \n",
    "    # Output Layer\n",
    "    x = Dense(output_shape, activation=activation, name = 'output')(x)\n",
    "    \n",
    "    # Create model\n",
    "    model = Model(melgram_input, x)\n",
    "    \n",
    "    return model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Set model parameters\n",
    "\n",
    "**Exercise:** Try to experiment with the following parameters:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [],
   "source": [
    "# number of Convolutional Layers (3, 4 or 5)\n",
    "nb_conv_layers = 3\n",
    "\n",
    "# number of Filters in each layer (# of elements must correspond to nb_conv_layers)\n",
    "nb_filters = [32,64,64,128,128]\n",
    "\n",
    "# number of hidden layers at the end of the model\n",
    "nb_hidden = 1 # 2\n",
    "\n",
    "# how many neurons in each hidden layer (# of elements must correspond to nb_hidden)\n",
    "dense_units = 128 #[128,56]\n",
    "\n",
    "# how many output units\n",
    "output_shape = n_classes\n",
    "\n",
    "# which activation function to use for OUTPUT layer\n",
    "# IN A SINGLE LABEL MULTI-CLASS TASK with N classes we use softmax activation to BALANCE best between the classes \n",
    "# and find the best decision for ONE class\n",
    "# (in a binary *or* multi-label task we use 'sigmoid')\n",
    "output_activation = 'softmax'\n",
    "\n",
    "# which type of normalization\n",
    "normalization = 'batch'\n",
    "\n",
    "# how much dropout to use on the hidden dense layers\n",
    "dropout = 0.2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = CompactCNN(input_shape, nb_conv = nb_conv_layers, nb_filters= nb_filters, \n",
    "                           normalize=normalization, \n",
    "                           nb_hidden = nb_hidden, dense_units = dense_units, \n",
    "                           output_shape = output_shape, activation = output_activation, \n",
    "                           dropout = dropout)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(80, 80, 1)"
      ]
     },
     "execution_count": 64,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "input_shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "input_1 (InputLayer)         (None, 80, 80, 1)         0         \n",
      "_________________________________________________________________\n",
      "bn_0_freq (BatchNormalizatio (None, 80, 80, 1)         320       \n",
      "_________________________________________________________________\n",
      "conv2d_4 (Conv2D)            (None, 80, 80, 32)        320       \n",
      "_________________________________________________________________\n",
      "bn1 (BatchNormalization)     (None, 80, 80, 32)        128       \n",
      "_________________________________________________________________\n",
      "elu_1 (ELU)                  (None, 80, 80, 32)        0         \n",
      "_________________________________________________________________\n",
      "pool1 (MaxPooling2D)         (None, 40, 20, 32)        0         \n",
      "_________________________________________________________________\n",
      "conv2d_5 (Conv2D)            (None, 40, 20, 64)        18496     \n",
      "_________________________________________________________________\n",
      "bn2 (BatchNormalization)     (None, 40, 20, 64)        256       \n",
      "_________________________________________________________________\n",
      "elu_2 (ELU)                  (None, 40, 20, 64)        0         \n",
      "_________________________________________________________________\n",
      "pool2 (MaxPooling2D)         (None, 13, 5, 64)         0         \n",
      "_________________________________________________________________\n",
      "conv2d_6 (Conv2D)            (None, 13, 5, 64)         36928     \n",
      "_________________________________________________________________\n",
      "bn3 (BatchNormalization)     (None, 13, 5, 64)         256       \n",
      "_________________________________________________________________\n",
      "elu_3 (ELU)                  (None, 13, 5, 64)         0         \n",
      "_________________________________________________________________\n",
      "pool3 (MaxPooling2D)         (None, 6, 1, 64)          0         \n",
      "_________________________________________________________________\n",
      "flatten_3 (Flatten)          (None, 384)               0         \n",
      "_________________________________________________________________\n",
      "dropout_1 (Dropout)          (None, 384)               0         \n",
      "_________________________________________________________________\n",
      "dense_5 (Dense)              (None, 128)               49280     \n",
      "_________________________________________________________________\n",
      "output (Dense)               (None, 8)                 1032      \n",
      "=================================================================\n",
      "Total params: 107,016\n",
      "Trainable params: 106,536\n",
      "Non-trainable params: 480\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Training Parameters\n",
    "\n",
    "In contrast with the binary Instrumental vs. Vocal task above we have to do some **important changes**:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Change #1: Loss"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {},
   "outputs": [],
   "source": [
    "# the loss for a single label classification task is CATEGORICAL crossentropy\n",
    "loss = 'categorical_crossentropy' "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Change #2: Output activation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "# which activation function to use for OUTPUT layer\n",
    "# IN A SINGLE LABEL MULTI-CLASS TASK with N classes we use softmax activation to BALANCE best between the classes \n",
    "# and find the best decision for ONE class\n",
    "output_activation = 'softmax'\n",
    "\n",
    "# Note that this has been set already above in the CompactCNN model definition (changing it here will be impactless)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Optimizer\n",
    "\n",
    "We have used **Stochastic Gradient Descent (SGD)** in our first experiments. This is the standard optimizer. A number of advanced algorithms are available.\n",
    "\n",
    "**Exercise:** Try various optimizers and their parameters and observe the impact on training convergence."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Optimizers\n",
    "\n",
    "# we define a couple of optimizers here\n",
    "sgd = optimizers.SGD() # standard\n",
    "sgd_momentum = optimizers.SGD(momentum=0.9, nesterov=True)\n",
    "rmsprop = optimizers.RMSprop(lr=0.001, rho=0.9, epsilon=1e-08, decay=0.01)#lr=0.001 decay = 0.03\n",
    "adagrad = optimizers.Adagrad(lr=0.01, epsilon=1e-08, decay=0.0)\n",
    "adam = optimizers.Adam(lr=0.003, beta_1=0.9, beta_2=0.999, epsilon=1e-07, decay=0.01)\n",
    "nadam = optimizers.Nadam(lr=0.002, beta_1=0.9, beta_2=0.999, epsilon=1e-07, schedule_decay=0.004)\n",
    "\n",
    "# PLEASE CHOOSE ONE:\n",
    "optimizer = adam"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Metrics\n",
    "\n",
    "In addition to accuracy, we evaluate precision and recall here."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Metrics\n",
    "def precision(y_true, y_pred):\n",
    "    true_positives = K.sum(K.round(K.clip(y_true * y_pred, 0, 1)))\n",
    "    predicted_positives = K.sum(K.round(K.clip(y_pred, 0, 1)))\n",
    "    precision = true_positives / (predicted_positives + K.epsilon())\n",
    "    return precision\n",
    "\n",
    "def recall(y_true, y_pred):\n",
    "    true_positives = K.sum(K.round(K.clip(y_true * y_pred, 0, 1)))\n",
    "    possible_positives = K.sum(K.round(K.clip(y_true, 0, 1)))\n",
    "    recall = true_positives / (possible_positives + K.epsilon())\n",
    "    return recall\n",
    "\n",
    "metrics = ['accuracy', precision, recall]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Other Parameters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {},
   "outputs": [],
   "source": [
    "batch_size = 32 \n",
    "\n",
    "validation_split=0.1 \n",
    "\n",
    "random_seed = 0\n",
    "\n",
    "callbacks = None\n",
    "\n",
    "epochs = 10"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Tensorboard (optional)\n",
    "\n",
    "Tensorboard (included in Tensorflow) is a web-based visualization to observe your training process."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {},
   "outputs": [],
   "source": [
    "from keras.callbacks import TensorBoard"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {},
   "outputs": [],
   "source": [
    "# set PATH where to store tensorboard files\n",
    "cwd = os.getcwd()\n",
    "TB_LOGDIR = join(cwd, \"tensorboard\")\n",
    "\n",
    "# make a subdir for each task and another subdir for each run using date/time\n",
    "from time import strftime, localtime\n",
    "experiment_name = task #join(task, strftime(\"%Y-%m-%d_%H-%M-%S\", localtime()))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'/Users/tom/Documents/Code/Tutorials/ismir2018_tutorial/tensorboard/genres'"
      ]
     },
     "execution_count": 73,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tb_logdir_cur = os.path.join(TB_LOGDIR, experiment_name)\n",
    "tb_logdir_cur"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Execute the following in a terminal:\n",
      "\n",
      "tensorboard --logdir=/Users/tom/Documents/Code/Tutorials/ismir2018_tutorial/tensorboard\n"
     ]
    }
   ],
   "source": [
    "print(\"Execute the following in a terminal:\\n\")\n",
    "print(\"tensorboard --logdir=\" + TB_LOGDIR)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {},
   "outputs": [],
   "source": [
    "# initialize TensorBoard in Python\n",
    "tensorboard = TensorBoard(log_dir = tb_logdir_cur)\n",
    "\n",
    "# add to Keras callbacks\n",
    "callbacks = [tensorboard]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then open Tensorboard in browser:\n",
    "\n",
    "http://localhost:6006"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Training"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "categorical_crossentropy\n",
      "<keras.optimizers.Adam object at 0x12e46e9e8>\n",
      "['accuracy', <function precision at 0x1214b2400>, <function recall at 0x1214b2598>]\n",
      "Batch size: 32 \n",
      "Epochs: 10\n"
     ]
    }
   ],
   "source": [
    "# Summary of Training options\n",
    "\n",
    "print(loss)\n",
    "print(optimizer)\n",
    "print(metrics)\n",
    "print(\"Batch size:\", batch_size, \"\\nEpochs:\", epochs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {},
   "outputs": [],
   "source": [
    "# COMPILE MODEL\n",
    "model.compile(loss=loss, metrics=metrics, optimizer=optimizer)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {},
   "outputs": [],
   "source": [
    "# past_epochs is only for the case that we execute the next code box multiple times (so that Tensorboard is displaying properly)\n",
    "past_epochs = 0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 1348 samples, validate on 150 samples\n",
      "Epoch 1/10\n",
      "1348/1348 [==============================] - 25s 19ms/step - loss: 1.1836 - acc: 0.6728 - precision: 0.7482 - recall: 0.6039 - val_loss: 0.8025 - val_acc: 0.7867 - val_precision: 0.8674 - val_recall: 0.7000\n",
      "Epoch 2/10\n",
      "1348/1348 [==============================] - 22s 16ms/step - loss: 0.7148 - acc: 0.7611 - precision: 0.8509 - recall: 0.7040 - val_loss: 0.7611 - val_acc: 0.7600 - val_precision: 0.8531 - val_recall: 0.7333\n",
      "Epoch 3/10\n",
      "1348/1348 [==============================] - 27s 20ms/step - loss: 0.6427 - acc: 0.7856 - precision: 0.8749 - recall: 0.7315 - val_loss: 0.6881 - val_acc: 0.7867 - val_precision: 0.8592 - val_recall: 0.7067\n",
      "Epoch 4/10\n",
      "1348/1348 [==============================] - 26s 20ms/step - loss: 0.5837 - acc: 0.8131 - precision: 0.8828 - recall: 0.7530 - val_loss: 0.6508 - val_acc: 0.8133 - val_precision: 0.9158 - val_recall: 0.7200\n",
      "Epoch 5/10\n",
      "1348/1348 [==============================] - 26s 19ms/step - loss: 0.5594 - acc: 0.8175 - precision: 0.8992 - recall: 0.7470 - val_loss: 0.9836 - val_acc: 0.6800 - val_precision: 0.8430 - val_recall: 0.5400\n",
      "Epoch 6/10\n",
      "1348/1348 [==============================] - 25s 19ms/step - loss: 0.5164 - acc: 0.8205 - precision: 0.9073 - recall: 0.7715 - val_loss: 0.5509 - val_acc: 0.8133 - val_precision: 0.8942 - val_recall: 0.7800\n",
      "Epoch 7/10\n",
      "1348/1348 [==============================] - 22s 16ms/step - loss: 0.4841 - acc: 0.8390 - precision: 0.9089 - recall: 0.7730 - val_loss: 0.5437 - val_acc: 0.8133 - val_precision: 0.9052 - val_recall: 0.7600\n",
      "Epoch 8/10\n",
      "1348/1348 [==============================] - 22s 16ms/step - loss: 0.4749 - acc: 0.8323 - precision: 0.9036 - recall: 0.7752 - val_loss: 0.5360 - val_acc: 0.8200 - val_precision: 0.8681 - val_recall: 0.7867\n",
      "Epoch 9/10\n",
      "1348/1348 [==============================] - 21s 16ms/step - loss: 0.4303 - acc: 0.8472 - precision: 0.9208 - recall: 0.8093 - val_loss: 0.5283 - val_acc: 0.8133 - val_precision: 0.8752 - val_recall: 0.7933\n",
      "Epoch 10/10\n",
      "1348/1348 [==============================] - 21s 16ms/step - loss: 0.4209 - acc: 0.8524 - precision: 0.9197 - recall: 0.8027 - val_loss: 0.6106 - val_acc: 0.8067 - val_precision: 0.8626 - val_recall: 0.7867\n"
     ]
    }
   ],
   "source": [
    "# START TRAINING\n",
    "\n",
    "history = model.fit(train_set, train_classes, \n",
    "                     validation_split=validation_split,\n",
    "                     #validation_data=(X_test,y_test), # option to provide separate validation set\n",
    "                     epochs=epochs, \n",
    "                     initial_epoch=past_epochs,\n",
    "                     batch_size=batch_size, \n",
    "                     callbacks=callbacks\n",
    "                     )\n",
    "\n",
    "past_epochs += epochs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Testing / Evaluation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[9.90408182e-01, 3.86757753e-03, 1.39946991e-03, 6.48082874e-04,\n",
       "        1.70425541e-04, 2.63284822e-03, 7.75201013e-04, 9.82842175e-05],\n",
       "       [1.94947876e-04, 5.59557346e-04, 4.76227608e-04, 4.31460468e-03,\n",
       "        2.87526008e-02, 9.56318080e-01, 3.57154677e-05, 9.34836455e-03],\n",
       "       [3.14175314e-03, 7.39955856e-03, 2.04317528e-03, 1.57288276e-02,\n",
       "        5.59136152e-01, 3.63210738e-01, 5.20799775e-04, 4.88189794e-02],\n",
       "       [9.65618849e-01, 6.35169586e-03, 2.37884163e-03, 3.14985169e-03,\n",
       "        4.68866638e-04, 1.98720805e-02, 1.53004937e-03, 6.29787450e-04],\n",
       "       [2.73255585e-03, 1.31402854e-02, 2.27193534e-02, 4.51066792e-02,\n",
       "        1.73882186e-01, 6.97263122e-01, 1.43500825e-03, 4.37207259e-02],\n",
       "       [2.04648823e-02, 8.87840614e-02, 1.92894250e-01, 1.06734954e-01,\n",
       "        8.91334563e-02, 4.36084837e-01, 5.30699501e-03, 6.05966076e-02],\n",
       "       [2.29632924e-03, 7.51652755e-03, 3.46888509e-03, 3.51347066e-02,\n",
       "        1.73782021e-01, 7.47896731e-01, 3.92064219e-04, 2.95127034e-02],\n",
       "       [1.49739143e-02, 2.68744212e-02, 5.97740524e-03, 8.54737386e-02,\n",
       "        6.01390600e-01, 2.41949797e-01, 8.40716588e-04, 2.25194525e-02],\n",
       "       [9.82153654e-01, 5.57385210e-04, 1.94950041e-03, 5.62786416e-04,\n",
       "        7.33113557e-05, 1.41208889e-02, 4.00457880e-04, 1.81972588e-04],\n",
       "       [8.54870975e-01, 3.49389552e-03, 6.72898144e-02, 1.01314699e-02,\n",
       "        1.48331830e-02, 4.31356244e-02, 1.72124640e-03, 4.52378066e-03]],\n",
       "      dtype=float32)"
      ]
     },
     "execution_count": 80,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# compute probabilities for the classes (= get outputs of output layer)\n",
    "test_pred_prob = model.predict(test_set)\n",
    "test_pred_prob[0:10]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0, 5, 4, 0, 5, 5, 5, 4, 0, 0, 5, 5, 4, 1, 4, 4, 4, 4, 4, 5])"
      ]
     },
     "execution_count": 81,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# for a multi-class SINGLE LABEL OUTPUT classification task, we use ARG MAX to determine \n",
    "# the most probable class per instance (we take the ARG MAX of the row vectors)\n",
    "test_pred = np.argmax(test_pred_prob, axis=1)\n",
    "test_pred[0:20]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0, 5, 4, 0, 4, 2, 4, 4, 0, 2, 3, 3, 4, 1, 4, 4, 4, 4, 4, 5])"
      ]
     },
     "execution_count": 82,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# do the same for groundtruth\n",
    "test_gt = np.argmax(test_classes, axis=1)\n",
    "test_gt[0:20]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.788"
      ]
     },
     "execution_count": 83,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# evaluate Accuracy\n",
    "accuracy_score(test_gt, test_pred)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 84,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.788"
      ]
     },
     "execution_count": 84,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# evaluate Precision\n",
    "precision_score(test_gt, test_pred, average='micro')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 85,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.788"
      ]
     },
     "execution_count": 85,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# evaluate Recall\n",
    "recall_score(test_gt, test_pred, average='micro')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 86,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "             precision    recall  f1-score   support\n",
      "\n",
      "  classical       0.96      0.91      0.93       250\n",
      "    country       0.50      0.33      0.40        18\n",
      "       jazz       1.00      0.13      0.23        23\n",
      "        pop       0.00      0.00      0.00        24\n",
      "       rock       0.85      0.85      0.85        95\n",
      "     techno       0.50      0.91      0.64        85\n",
      "      blues       0.00      0.00      0.00         2\n",
      "      dance       0.00      0.00      0.00         3\n",
      "\n",
      "avg / total       0.79      0.79      0.76       500\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/usr/local/lib/python3.6/site-packages/sklearn/metrics/classification.py:1135: UndefinedMetricWarning: Precision and F-score are ill-defined and being set to 0.0 in labels with no predicted samples.\n",
      "  'precision', 'predicted', average, warn_for)\n"
     ]
    }
   ],
   "source": [
    "print(classification_report(test_gt, test_pred, target_names=metadata.columns))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 3) Mood Recognition\n",
    "\n",
    "This is a multi-label classification task: multiple categories to detect, any of them can be 0 or 1."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load Audio Spectrograms\n",
    "\n",
    "We prepared already the Mel spectrograms for the audio files used in this task."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 87,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(719, 80, 80)"
      ]
     },
     "execution_count": 87,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "task = 'moods'\n",
    "\n",
    "# load Mel spectrograms\n",
    "spectrogram_file = SPECTROGRAM_FILE_PATTERN % task\n",
    "spectrograms, spectrograms_clip_ids = load_spectrograms(spectrogram_file)\n",
    "\n",
    "# standardize\n",
    "data = standardize(spectrograms)\n",
    "data.shape # verify the shape of the loaded & standardize spectrograms"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load Metadata"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 88,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(719, 4)"
      ]
     },
     "execution_count": 88,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# use META_FILE_PATTERN to load the correct metadata file. set correct METADATA_PATH above\n",
    "csv_file = LABEL_FILE_PATTERN % task\n",
    "metadata = pd.read_csv(csv_file, index_col=0) #, sep='\\t')\n",
    "metadata.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 89,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>loud</th>\n",
       "      <th>quiet</th>\n",
       "      <th>soft</th>\n",
       "      <th>strange</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>clip_id</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>30064</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5862</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>38362</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>44901</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16246</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "         loud  quiet  soft  strange\n",
       "clip_id                            \n",
       "30064       0      0     0        1\n",
       "5862        0      0     0        0\n",
       "38362       1      0     0        0\n",
       "44901       0      0     0        1\n",
       "16246       0      0     1        1"
      ]
     },
     "execution_count": 89,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "metadata.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 90,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "loud       209\n",
       "quiet      177\n",
       "soft       200\n",
       "strange    120\n",
       "dtype: int64"
      ]
     },
     "execution_count": 90,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# how many tracks per mood\n",
    "metadata.sum()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 91,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "3"
      ]
     },
     "execution_count": 91,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# maximum number of moods per track\n",
    "metadata.sum(axis=1).max()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Align Metadata and Spectrograms"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 92,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "spec_indices = spectrograms_clip_ids.loc[metadata.index]['spec_id']\n",
    "data = spectrograms[spec_indices,:]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create Train X and Y: data and classes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 93,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[0, 0, 0, 1],\n",
       "       [0, 0, 0, 0],\n",
       "       [1, 0, 0, 0],\n",
       "       ...,\n",
       "       [0, 0, 0, 1],\n",
       "       [0, 0, 0, 0],\n",
       "       [1, 0, 0, 0]])"
      ]
     },
     "execution_count": 93,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# classes needs to be a \"1-hot encoded\" numpy array (which our groundtruth already is! we just convert pandas to numpy)\n",
    "classes = metadata.values\n",
    "classes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 94,
   "metadata": {},
   "outputs": [],
   "source": [
    "n_classes = metadata.shape[1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 95,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(719, 80, 80, 1)"
      ]
     },
     "execution_count": 95,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# add channel (see above)\n",
    "data = add_channel(data)\n",
    "data.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 96,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(80, 80, 1)"
      ]
     },
     "execution_count": 96,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# input_shape: we store the new shape of the images in the 'input_shape' variable.\n",
    "# take all dimensions except the 0th one (which is the number of files)\n",
    "input_shape = data.shape[1:]  \n",
    "input_shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Train & Test Set Split\n",
    "\n",
    "We split the original full data set into two parts: Train Set (75%) and Test Set (25%)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Change: We cannot use Stratified Split here as it does not make sense for a Multi-Label task!\n",
    "\n",
    "We use a random ShuffleSplit instead."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 97,
   "metadata": {},
   "outputs": [],
   "source": [
    "# use ShuffleSplit INSTEAD OF StratifiedShuffleSplit \n",
    "\n",
    "splitter = ShuffleSplit(n_splits=1, test_size=testset_size, random_state=0)\n",
    "splits = splitter.split(data, classes)\n",
    "\n",
    "for train_index, test_index in splits:\n",
    "    train_set = data[train_index]\n",
    "    test_set = data[test_index]\n",
    "    train_classes = classes[train_index]\n",
    "    test_classes = classes[test_index]\n",
    "# Note: this for loop is only executed once if n_splits==1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 98,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(539, 80, 80, 1)\n",
      "(180, 80, 80, 1)\n"
     ]
    }
   ],
   "source": [
    "print(train_set.shape)\n",
    "print(test_set.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Model and Training Parameters\n",
    "\n",
    "we use the same model as for Instrumental vs. Vocal and Genres above\n",
    "\n",
    "with a few changes in the Training parameters"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Change #1: Loss"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 99,
   "metadata": {},
   "outputs": [],
   "source": [
    "# the loss for a MULTI label classification task is BINARY crossentropy\n",
    "loss = 'binary_crossentropy' "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Change #2: Output activation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 100,
   "metadata": {},
   "outputs": [],
   "source": [
    "# which activation function to use for OUTPUT layer\n",
    "# IN A MULTI-LABEL TASK with N classes we use SIGMOID activation same as with a BINARY task\n",
    "# as EACH of the classes can be 0 or 1 \n",
    "\n",
    "output_activation = 'sigmoid'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Model\n",
    "\n",
    "We are reusing the **CompactCNN** from above.\n",
    "\n",
    "**Exercise:** Adapt the parameters of the CompactCNN model:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 101,
   "metadata": {},
   "outputs": [],
   "source": [
    "# number of Convolutional Layers (3, 4 or 5)\n",
    "nb_conv_layers = 3\n",
    "\n",
    "# number of Filters in each layer (# of elements must correspond to nb_conv_layers)\n",
    "nb_filters = [32,64,64,128,128]\n",
    "\n",
    "# number of hidden layers at the end of the model\n",
    "nb_hidden = 1 # 2\n",
    "\n",
    "# how many neurons in each hidden layer (# of elements must correspond to nb_hidden)\n",
    "dense_units = 128 #[128,56]\n",
    "\n",
    "# how many output units\n",
    "output_shape = n_classes\n",
    "\n",
    "# which type of normalization\n",
    "normalization = 'batch'\n",
    "\n",
    "# how much dropout to use on the hidden dense layers\n",
    "dropout = 0.2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 102,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = CompactCNN(input_shape, nb_conv = nb_conv_layers, nb_filters= nb_filters, \n",
    "                           normalize=normalization, \n",
    "                           nb_hidden = nb_hidden, dense_units = dense_units, \n",
    "                           output_shape = output_shape, activation = output_activation, \n",
    "                           dropout = dropout)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 103,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "input_2 (InputLayer)         (None, 80, 80, 1)         0         \n",
      "_________________________________________________________________\n",
      "bn_0_freq (BatchNormalizatio (None, 80, 80, 1)         320       \n",
      "_________________________________________________________________\n",
      "conv2d_7 (Conv2D)            (None, 80, 80, 32)        320       \n",
      "_________________________________________________________________\n",
      "bn1 (BatchNormalization)     (None, 80, 80, 32)        128       \n",
      "_________________________________________________________________\n",
      "elu_4 (ELU)                  (None, 80, 80, 32)        0         \n",
      "_________________________________________________________________\n",
      "pool1 (MaxPooling2D)         (None, 40, 20, 32)        0         \n",
      "_________________________________________________________________\n",
      "conv2d_8 (Conv2D)            (None, 40, 20, 64)        18496     \n",
      "_________________________________________________________________\n",
      "bn2 (BatchNormalization)     (None, 40, 20, 64)        256       \n",
      "_________________________________________________________________\n",
      "elu_5 (ELU)                  (None, 40, 20, 64)        0         \n",
      "_________________________________________________________________\n",
      "pool2 (MaxPooling2D)         (None, 13, 5, 64)         0         \n",
      "_________________________________________________________________\n",
      "conv2d_9 (Conv2D)            (None, 13, 5, 64)         36928     \n",
      "_________________________________________________________________\n",
      "bn3 (BatchNormalization)     (None, 13, 5, 64)         256       \n",
      "_________________________________________________________________\n",
      "elu_6 (ELU)                  (None, 13, 5, 64)         0         \n",
      "_________________________________________________________________\n",
      "pool3 (MaxPooling2D)         (None, 6, 1, 64)          0         \n",
      "_________________________________________________________________\n",
      "flatten_4 (Flatten)          (None, 384)               0         \n",
      "_________________________________________________________________\n",
      "dropout_2 (Dropout)          (None, 384)               0         \n",
      "_________________________________________________________________\n",
      "dense_6 (Dense)              (None, 128)               49280     \n",
      "_________________________________________________________________\n",
      "output (Dense)               (None, 4)                 516       \n",
      "=================================================================\n",
      "Total params: 106,500\n",
      "Trainable params: 106,020\n",
      "Non-trainable params: 480\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### TensorBoard setup (optional)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 104,
   "metadata": {},
   "outputs": [],
   "source": [
    "experiment_name = task\n",
    "\n",
    "tb_logdir_cur = os.path.join(TB_LOGDIR, experiment_name)\n",
    "\n",
    "# initialize TensorBoard in Python\n",
    "tensorboard = TensorBoard(log_dir = tb_logdir_cur)\n",
    "\n",
    "# + add to callbacks\n",
    "callbacks = [tensorboard]\n",
    "\n",
    "# otherwise assign:\n",
    "# callbacks = None"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Rest of Parameters\n",
    "\n",
    "stay essentially the same (or similar)\n",
    "\n",
    "**Excercise:** change the optimizer (see same exercise in Genre model)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 105,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Optimizer\n",
    "optimizer = adam\n",
    "\n",
    "metrics = ['accuracy']\n",
    "\n",
    "random_seed = 0\n",
    "\n",
    "batch_size = 32 \n",
    "\n",
    "validation_split = 0.1 \n",
    "\n",
    "epochs = 10"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Training"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 106,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "binary_crossentropy\n",
      "<keras.optimizers.Adam object at 0x12e46e9e8>\n",
      "['accuracy']\n",
      "Batch size: 32 \n",
      "Epochs: 10\n"
     ]
    }
   ],
   "source": [
    "# Summary of Training options\n",
    "\n",
    "print(loss)\n",
    "print(optimizer)\n",
    "print(metrics)\n",
    "print(\"Batch size:\", batch_size, \"\\nEpochs:\", epochs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 107,
   "metadata": {},
   "outputs": [],
   "source": [
    "# COMPILE MODEL\n",
    "model.compile(loss=loss, metrics=metrics, optimizer=optimizer)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 108,
   "metadata": {},
   "outputs": [],
   "source": [
    "# past_epochs is only for the case that we execute the next code box multiple times (so that Tensorboard is displaying properly)\n",
    "past_epochs = 0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 109,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 485 samples, validate on 54 samples\n",
      "Epoch 1/10\n",
      "485/485 [==============================] - 8s 17ms/step - loss: 0.5095 - acc: 0.7711 - val_loss: 0.4958 - val_acc: 0.7963\n",
      "Epoch 2/10\n",
      "485/485 [==============================] - 7s 15ms/step - loss: 0.4131 - acc: 0.8165 - val_loss: 0.4223 - val_acc: 0.8194\n",
      "Epoch 3/10\n",
      "485/485 [==============================] - 7s 15ms/step - loss: 0.3736 - acc: 0.8299 - val_loss: 0.3737 - val_acc: 0.8194\n",
      "Epoch 4/10\n",
      "485/485 [==============================] - 7s 15ms/step - loss: 0.3563 - acc: 0.8376 - val_loss: 0.4074 - val_acc: 0.8194\n",
      "Epoch 5/10\n",
      "485/485 [==============================] - 8s 17ms/step - loss: 0.3536 - acc: 0.8294 - val_loss: 0.3742 - val_acc: 0.8102\n",
      "Epoch 6/10\n",
      "485/485 [==============================] - 8s 16ms/step - loss: 0.3546 - acc: 0.8423 - val_loss: 0.3738 - val_acc: 0.8102\n",
      "Epoch 7/10\n",
      "485/485 [==============================] - 8s 17ms/step - loss: 0.3310 - acc: 0.8490 - val_loss: 0.3842 - val_acc: 0.8056\n",
      "Epoch 8/10\n",
      "485/485 [==============================] - 8s 16ms/step - loss: 0.3164 - acc: 0.8567 - val_loss: 0.3818 - val_acc: 0.8102\n",
      "Epoch 9/10\n",
      "485/485 [==============================] - 8s 16ms/step - loss: 0.3277 - acc: 0.8495 - val_loss: 0.3515 - val_acc: 0.8241\n",
      "Epoch 10/10\n",
      "485/485 [==============================] - 7s 15ms/step - loss: 0.3120 - acc: 0.8598 - val_loss: 0.3878 - val_acc: 0.8056\n"
     ]
    }
   ],
   "source": [
    "# START TRAINING\n",
    "\n",
    "history = model.fit(train_set, train_classes, \n",
    "                     validation_split=validation_split,\n",
    "                     #validation_data=(X_test,y_test), \n",
    "                     epochs=epochs, \n",
    "                     initial_epoch=past_epochs,\n",
    "                     batch_size=batch_size, \n",
    "                     callbacks=callbacks\n",
    "                     )\n",
    "\n",
    "past_epochs += epochs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Evaluation on Test Set"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 110,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[0.95713997, 0.0125533 , 0.0104373 , 0.03815303],\n",
       "       [0.09826187, 0.24977514, 0.34378156, 0.5955263 ],\n",
       "       [0.00465956, 0.36048868, 0.6711814 , 0.06887353],\n",
       "       [0.04967834, 0.14146991, 0.6209346 , 0.13409252],\n",
       "       [0.17552362, 0.2713198 , 0.40802023, 0.13511126],\n",
       "       [0.01442199, 0.25579467, 0.4835568 , 0.22557716],\n",
       "       [0.02002936, 0.22810721, 0.6371216 , 0.05681899],\n",
       "       [0.7855748 , 0.02367456, 0.10101752, 0.22913985],\n",
       "       [0.98808986, 0.02013061, 0.00150874, 0.01755911],\n",
       "       [0.20514971, 0.05430627, 0.21276605, 0.5311777 ]], dtype=float32)"
      ]
     },
     "execution_count": 110,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# compute probabilities for the classes (= get outputs of output layer)\n",
    "test_pred_prob = model.predict(test_set)\n",
    "test_pred_prob[0:10]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Change: In a multi-label task we have to round each prediction probability to 0 or 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 111,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[1., 0., 0., 0.],\n",
       "       [0., 0., 0., 1.],\n",
       "       [0., 0., 1., 0.],\n",
       "       [0., 0., 1., 0.],\n",
       "       [0., 0., 0., 0.],\n",
       "       [0., 0., 0., 0.],\n",
       "       [0., 0., 1., 0.],\n",
       "       [1., 0., 0., 0.],\n",
       "       [1., 0., 0., 0.],\n",
       "       [0., 0., 0., 1.]], dtype=float32)"
      ]
     },
     "execution_count": 111,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# to get the predicted class(es) we have to round 0 < 0.5 > 1\n",
    "test_pred = np.round(test_pred_prob)\n",
    "test_pred[0:10]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 112,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[1, 0, 0, 0],\n",
       "       [0, 0, 0, 0],\n",
       "       [0, 1, 1, 0],\n",
       "       [0, 0, 0, 0],\n",
       "       [0, 0, 1, 0],\n",
       "       [0, 0, 1, 0],\n",
       "       [0, 0, 0, 1],\n",
       "       [0, 0, 0, 1],\n",
       "       [1, 0, 0, 0],\n",
       "       [0, 1, 0, 0]])"
      ]
     },
     "execution_count": 112,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# groundtruth\n",
    "test_classes[0:10]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Evaluation Metrics\n",
    "\n",
    "In addition to Accuracy, common metrics for multi-label classification are ROC AUC score and Hamming Loss (among others)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 113,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.5"
      ]
     },
     "execution_count": 113,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Accuracy\n",
    "accuracy_score(test_classes, test_pred)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 114,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.7214879034654973"
      ]
     },
     "execution_count": 114,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Area Under the Receiver Operating Characteristic Curve (ROC AUC) \n",
    "roc_auc_score(test_classes, test_pred)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 115,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.16944444444444445"
      ]
     },
     "execution_count": 115,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Hamming loss is the fraction of labels that are incorrectly predicted.\n",
    "hamming_loss(test_classes, test_pred)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}