audio_classification.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Energy auto-encoder: classification"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Technically akin to \"transductive learning\" because the sparse auto-encoder dictionary is learned over the whole dataset, not only the training data. The LeCun paper we compare with does the same. This could be solved by including the dictionary learning step in the classifier. Technical solutions:\n",
    "    1. Compute the dictionary in our custom classifier.\n",
    "    2. Create a scikit-learn Pipeline which includes the whole preprocessing and feature extraction steps.\n",
    "    In either case the ability to import functions from other notebooks would help. This would be very slow due to the tremendous amount of time needed to train the auto-encoder.\n",
    "* We should use \"grid search\" to find the optimal hyper-parameters (auto-encoders, frames, feature vectors, SVM).\n",
    "* We may use a validation set to mitigate the leak of the testing set in the model as we tune the hyper-parameters.\n",
    "* Even if not stated in LeCun's paper we should rescale the data before SVM classification.\n",
    "\n",
    "For the auto-encoder learning to converge we should rescale beforehand. As the transformation preserves energy, there is no need to rescale again."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Hyper-parameters"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* `scale`: scaling (None, minmax, std)\n",
    "* `Nvectors`: number of feature vectors per song.\n",
    "* `svm_type`: C-SVM (C) or nu-SVM (nu).\n",
    "* `kernel`: C-SVM kernel (linear, rbf).\n",
    "* `C`: penalty parameter C of the error term.\n",
    "* `nu`: an upper bound on the fraction of training errors and a lower bound of the fraction of support vectors.\n",
    "* `majority_voting`: When `True`, each of the 2`Nvectors` votes for one label and the accuracy is computed on the classification of the whole clips. When `False`, the accuracy is computed on the classification of the feature vectors.\n",
    "* `test_size`: proportion of testing data for cross-validation.\n",
    "* `Ncv`: number of cross-validation runs, in multiple of 10.\n",
    "* `dataset_classification`: the dataset to use for classification (X, Z). It allows to compare with the baseline, i.e. spectrograms.\n",
    "* `Ngenres, Nclips, Nframes`: a way to reduce the size of the dataset.\n",
    "* `folder`: relative path to HDF5 files.\n",
    "* `filename_*`: name of the HDF5 file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "if 'p' in globals().keys():\n",
    "    # Hyper-parameters passed by the experiment runner.\n",
    "    for key, value in p.items():\n",
    "        globals()[key] = value\n",
    "else:\n",
    "    scale = 'minmax'\n",
    "    Nvectors = 6\n",
    "    svm_type = 'C'\n",
    "    kernel = 'linear'\n",
    "    C = 1\n",
    "    nu = 0.5\n",
    "    majority_voting = True\n",
    "    test_size = 0.1\n",
    "    Ncv = 20\n",
    "    dataset_classification = 'Z'\n",
    "    Ngenres, Nclips, Nframes = 10, 100, 644\n",
    "    folder = 'data'\n",
    "    filename_features = 'features.hdf5'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import os, time\n",
    "import numpy as np\n",
    "import sklearn\n",
    "from sklearn import svm\n",
    "from sklearn import cross_validation\n",
    "from sklearn import metrics\n",
    "from sklearn import preprocessing\n",
    "import h5py\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "\n",
    "print('Software versions:')\n",
    "for pkg in [np, sklearn]:\n",
    "    print('  {}: {}'.format(pkg.__name__, pkg.__version__))\n",
    "\n",
    "toverall = time.time()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Input data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1. Retrieve data from the HDF5 data store.\n",
    "2. Choose the data we want to work with:\n",
    "    * raw audio $X_a$,\n",
    "    * CQT spectrograms $X_s$,\n",
    "    * normalized spectrograms $X_n$,\n",
    "    * sparse codes $Z$.\n",
    "3. Eventually reduce the number $N_{genres} \\cdot N_{clips}$ of samples for quicker analysis."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "def datinfo(X, name='Dataset'):\n",
    "    r\"\"\"Print dataset size and dimensionality\"\"\"\n",
    "    print('{}:\\n'\n",
    "          '  size: N={:,} x n={} -> {:,} floats\\n'\n",
    "          '  dim: {:,} features per clip\\n'\n",
    "          '  shape: {}'\n",
    "          .format(name, np.prod(X.shape[:-1]), X.shape[-1],\n",
    "                  np.prod(X.shape), np.prod(X.shape[2:]), X.shape))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "filename = os.path.join(folder, filename_features)\n",
    "with h5py.File(filename, 'r') as audio:\n",
    "\n",
    "    # Display HDF5 attributes.\n",
    "    print('Attributes:')\n",
    "    for attr in audio.attrs:\n",
    "        print('  {} = {}'.format(attr, audio.attrs[attr]))\n",
    "    labels = audio.attrs['labels']\n",
    "\n",
    "    # Show datasets, their dimensionality and data type.\n",
    "    print('Datasets:')\n",
    "    for dname, dset in audio.items():\n",
    "        print('  {:2}: {:24}, {}'.format(dname, dset.shape, dset.dtype))\n",
    "\n",
    "    # Choose dataset: Xa, Xs, Z.\n",
    "    X = audio.get(dataset_classification)\n",
    "\n",
    "    # Full dataset.\n",
    "    n = X.shape[-1]\n",
    "    datinfo(X, 'Full dataset')\n",
    "    print(type(X))\n",
    "\n",
    "    # Load data into memory as a standard NumPy array.\n",
    "    X = X[:Ngenres,:Nclips,:Nframes,...]\n",
    "    datinfo(X, 'Reduced dataset')\n",
    "    print(type(X))\n",
    "\n",
    "    # Resize in place without memory loading via hyperslab.\n",
    "    # Require chunked datasets.\n",
    "    #X.resize((Ngenres, Nclips, Nframes, 2, n))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Feature vectors through aggregation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Yet another (hopefully intelligent) dimensionality reduction:\n",
    "* Aggregation of features from various frames to make up $2N_{vectors} = 12$ feature vectors per clip. Each vector represents approximatly 5 seconds of audio which is way longer than single frames while shorter than the whole clip.\n",
    "* There is again a 50% overlap between those feature vectors.\n",
    "* Absolute value rectification to prevent components of different sign from canceling each other out. May be worth disabling when working with raw audio ($X_a$). It then makes sense to rescale with 'std' instead of 'minmax'.\n",
    "* Can be thought of as an histogram of used dictionary atoms (if using $Z$) or frequency bins (if using $X_s$) along the chosen time window."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Flatten consecutive frames in time.\n",
    "X.resize((Ngenres, Nclips, 2*Nframes, n))\n",
    "#assert np.all(X1[1,4,3,:] == X[1,4,1,1,:])\n",
    "datinfo(X, 'Flattened frames')\n",
    "\n",
    "# Parameters.\n",
    "Nframes_per_vector = int(np.floor(2 * Nframes / (Nvectors+0.5)))\n",
    "\n",
    "def aggregate(X, absrect=True):\n",
    "    # Truncate.\n",
    "    X = X[:,:,:Nvectors*Nframes_per_vector,:]\n",
    "    # Group.\n",
    "    X = X.reshape((Ngenres, Nclips, Nvectors, Nframes_per_vector, n))\n",
    "    datinfo(X, 'Truncated and grouped')\n",
    "    # Aggregate.\n",
    "    if absrect:\n",
    "        return np.sum(np.abs(X), axis=3)\n",
    "    else:\n",
    "        return np.sum(X, axis=3)\n",
    "\n",
    "# Feature vectors.\n",
    "Y = np.empty((Ngenres, Nclips, Nvectors, 2, n))\n",
    "Y[:,:,:,0,:] = aggregate(X)  # Aligned.\n",
    "Y[:,:,:,1,:] = aggregate(X[:,:,Nframes_per_vector/2:,:])  # Ovelapped.\n",
    "datinfo(Y, 'Feature vectors')\n",
    "\n",
    "# Free memory.\n",
    "del(X)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Feature vectors visualization"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Visualize all feature vectors of a given clip.\n",
    "\n",
    "Observations:\n",
    "* Classical music seems to have a much denser spectrum than blues, which may explain why these two classes are easily identifiable using $X_s$.\n",
    "* Country seems to have strong low frequencies."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "genre, clip = 0, 7\n",
    "fig = plt.figure(figsize=(8,5))\n",
    "fig.suptitle('12 feature vectors each covering 5 seconds with 50% overlap')\n",
    "for vector in range(Nvectors):\n",
    "    for k in range(2):\n",
    "        i = vector*2+k\n",
    "        ax = fig.add_subplot(4, 3, i)\n",
    "        ax.plot(Y[genre,clip,vector,k,:])\n",
    "        ax.set_xlim((0, n))\n",
    "        ax.set_xticks([])\n",
    "        ax.set_yticks([])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data preparation for classification"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1. Rearrange dataset as a 2D array: number of samples x dimensionality.\n",
    "2. Optionally scale the data.\n",
    "3. Generate labels.\n",
    "4. Optionally split in training and testing sets.\n",
    "5. Optionally randomize labels for testing.\n",
    "\n",
    "Observations:\n",
    "* Scaling is necessary for classification performance (both accuracy and speed). 'std' scaling is not well suited to our histogram-like feature vectors which are not at all Gaussian distributions. Prefer 'minmax', i.e. scale features in [0,1]. Moreover this scaling will preserve the sparsity when dealing with sparse codes $Z$."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "def prepdata(a, b, c, test_size=None, scale=None, rand=False):\n",
    "    \"\"\"Prepare data for classification.\"\"\"\n",
    "    \n",
    "    # Squeeze dataset to a 2D array.\n",
    "    data = Y.reshape((a*b), c)\n",
    "    if c == n:\n",
    "        assert np.all(data[31,:] == Y[0,2,3,1,:])\n",
    "    elif c == Nvectors*2*n:\n",
    "        assert np.all(data[Nclips+2,:] == Y[1,2,:,:,:].reshape(-1))\n",
    "\n",
    "    # Independently rescale each feature.\n",
    "    # To be put in an sklearn Pipeline to avoid transductive learning.\n",
    "    if scale is 'std':\n",
    "        # Features have zero norm and unit standard deviation.\n",
    "        data = preprocessing.scale(data, axis=0)\n",
    "    elif scale is 'minmax':\n",
    "        # Features in [0,1].\n",
    "        data -= np.min(data, axis=0)\n",
    "        data /= np.max(data, axis=0)\n",
    "    #print(np.min(data, axis=0))\n",
    "    #print(np.max(data, axis=0))\n",
    "    \n",
    "    # Labels.\n",
    "    target = np.empty((a, b), dtype=np.uint8)\n",
    "    for genre in range(Ngenres):\n",
    "        target[genre,:] = genre\n",
    "    target.resize(data.shape[0])\n",
    "    print('{} genres: {}'.format(Ngenres, ', '.join(labels[:Ngenres])))\n",
    "\n",
    "    # Be sure that classification with random labels is no better than random.\n",
    "    if rand:\n",
    "        target = np.floor(np.random.uniform(0, Ngenres, target.shape))\n",
    "        print('Balance: {} {}'.format(np.sum(target == 0), np.sum(target == 1)))\n",
    "\n",
    "    # Training and testing sets.\n",
    "    if test_size is not None:\n",
    "        X_train, X_test, y_train, y_test = cross_validation.train_test_split(\n",
    "            data, target, test_size=test_size)  # random_state=1\n",
    "        print('Training data: {}, {}'.format(X_train.shape, X_train.dtype))\n",
    "        print('Testing data: {}, {}'.format(X_test.shape, X_test.dtype))\n",
    "        print('Training labels: {}, {}'.format(y_train.shape, y_train.dtype))\n",
    "        print('Testing labels: {}, {}'.format(y_test.shape, y_test.dtype))\n",
    "        return X_train, X_test, y_train, y_test\n",
    "    else:\n",
    "        print('Data: {}, {}'.format(data.shape, data.dtype))\n",
    "        print('Labels: {}, {}'.format(target.shape, target.dtype))\n",
    "        return data, target"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Linear SVM"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Each feature vector gets a genre label.\n",
    "* Classification with linear Vector Support Machine (SVM).\n",
    "* Fast to train.\n",
    "* Scale well to large dataset.\n",
    "* Two implementations: liblinear (sklearn LinearSVC) and libsvm (sklearn SVC and NuSVC)\n",
    "* Multi-class: \"one-vs-one\" approach (Knerr et al., 1990) (sklearn SVC and NuSVC) and \"one-vs-the-rest\" (sklearn LinearSVC)\n",
    "\n",
    "Observations:\n",
    "* We can predict genre labels of individual frames with good accuracy using CQT spectrograms only.\n",
    "* SVC vs NuSVC vs LinearSVC:\n",
    "    * 10-fold cross-validation with 10 classes (default $C=1$ and $\\nu=0.5$):\n",
    "        * SVC (0.56) yields better accuracy than LinearSVC (0.53) than NuSVC (0.51)\n",
    "        * SVC (303s) and LinearSVC (296s) faster than NuSVC (501s)\n",
    "    * SVC does often not converge if data is not scaled\n",
    "    * LinearSVC may be more scalable (in the number of samples)\n",
    "* Hyper-parameters:\n",
    "    * $C$ seems to have little impact.\n",
    "    * $\\nu$ has a great impact on speed: lower is slower\n",
    "\n",
    "Open questions:\n",
    "* Which multi-class strategy to adopt: one-vs-all or one-vs-one ?\n",
    "    * sklearn states that one-vs-all is the most common strategy\n",
    "* Determine $C$ or $\\nu$."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Instantiate a classifier.\n",
    "\n",
    "if svm_type is 'C':\n",
    "    clf_svm = svm.SVC(kernel=kernel, C=C)\n",
    "elif svm_type is 'nu':\n",
    "    clf_svm = svm.NuSVC(kernel=kernel, nu=nu)\n",
    "#clf_svm = svm.LinearSVC(C=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Try the single feature vector classifier (linear SVM).\n",
    "if True:\n",
    "    \n",
    "    # Split data.\n",
    "    X_train, X_test, y_train, y_test = prepdata(\n",
    "        Ngenres, Nclips*Nvectors*2, n, test_size=0.4,\n",
    "        scale=scale, rand=False)\n",
    "    \n",
    "    # Train.\n",
    "    clf_svm.fit(X_train, y_train)\n",
    "    \n",
    "    # Test.\n",
    "    y_predict = clf_svm.predict(X_test)\n",
    "    acc = metrics.accuracy_score(y_test, y_predict)\n",
    "    print('Accuracy: {:.1f} %'.format(acc*100))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Majority voting"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Final dimensionality reduction step:\n",
    "* Each of the 12 feature vectors of a clip gives a vote. We choose the genre with the highest number of votes.\n",
    "* Implemented as a custom classifier which embeds an SVM for individual feature vectors classification.\n",
    "* Alternative implementation: insert in a sklearn pipeline after SVC.\n",
    "\n",
    "Observations:\n",
    "* Accuracy on whole clips is indeed better than accuracy on individual feature vectors.\n",
    "* *clf_svm_vote.confidence* is useful to observe if a class is harder to differentiate."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Define and instantiate our custom classifier.\n",
    "class svm_vote(sklearn.base.BaseEstimator):\n",
    "    \n",
    "    def __init__(self, svm):\n",
    "        self.svm = svm\n",
    "    \n",
    "    def _vectors(self, X, y=None):\n",
    "        \"\"\"Rearrange data in feature vectors for SVM.\"\"\"\n",
    "        X = X.reshape(X.shape[0]*Nvectors*2, n)\n",
    "        if y is not None:\n",
    "            y = np.repeat(y, Nvectors*2, axis=0)\n",
    "            assert y.shape[0] == X.shape[0]\n",
    "            return (X, y)\n",
    "        else:\n",
    "            return (X,)\n",
    "    \n",
    "    def fit(self, X, y):\n",
    "        \"\"\"Fit the embedded SVC.\"\"\"\n",
    "        self.svm.fit(*self._vectors(X, y))\n",
    "    \n",
    "    def svm_score(self, X, y):\n",
    "        \"\"\"Return SVC accuracy on feature vectors.\"\"\"\n",
    "        return self.svm.score(*self._vectors(X, y))\n",
    "    \n",
    "    def svm_predict(self, X):\n",
    "        \"\"\"Return SVC predictions on feature vectors.\"\"\"\n",
    "        y = self.svm.predict(*self._vectors(X))\n",
    "        y.resize(X.shape[0], Nvectors*2)\n",
    "        return y\n",
    "        \n",
    "    def confidence(self, X):\n",
    "        \"\"\"Return the number of votes for each class.\"\"\"\n",
    "        def bincount(x):\n",
    "            return np.bincount(x, minlength=Ngenres)\n",
    "        y = np.apply_along_axis(bincount, 1, self.svm_predict(X))\n",
    "        assert np.all(np.sum(y, axis=1) == Nvectors*2)\n",
    "        return y\n",
    "        \n",
    "    def predict(self, X):\n",
    "        \"\"\"Return predictions on whole clips.\"\"\"\n",
    "        y = self.svm_predict(X)\n",
    "        return np.apply_along_axis(lambda x: np.bincount(x).argmax(), 1, y)\n",
    "        #return np.zeros(X.shape[0])  # Pretty bad prediction.\n",
    "    \n",
    "    def score(self, X, y):\n",
    "        \"\"\"Return the accuracy score. Used by sklearn cross-validation.\"\"\"\n",
    "        return metrics.accuracy_score(y, self.predict(X))\n",
    "\n",
    "clf_svm_vote = svm_vote(clf_svm)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Try the whole clip classifier (linear SVM and majority voting).\n",
    "if True:\n",
    "    \n",
    "    # Split data.\n",
    "    X_train, X_test, y_train, y_test = prepdata(\n",
    "        Ngenres, Nclips, Nvectors*2*n, test_size=0.4,\n",
    "        scale=scale, rand=False)\n",
    "    \n",
    "    # Train.\n",
    "    clf_svm_vote.fit(X_train, y_train)\n",
    "    \n",
    "    # Test on single vectors.\n",
    "    acc = clf_svm_vote.svm_score(X_test, y_test)\n",
    "    print('Feature vectors accuracy: {:.1f} %'.format(acc*100))\n",
    "    \n",
    "    # Observe individual votes.\n",
    "    #print(clf_svm_vote.svm_predict(X_test))\n",
    "    #print(clf_svm_vote.confidence(X_test))\n",
    "    \n",
    "    # Test on whole clips.\n",
    "    y_predict = clf_svm_vote.predict(X_test)\n",
    "    acc = metrics.accuracy_score(y_test, y_predict)\n",
    "    assert acc == clf_svm_vote.score(X_test, y_test)\n",
    "    print('Clips accuracy: {:.1f} %'.format(acc*100))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Cross-validation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* 10-fold cross-validation.\n",
    "* 100 randomly chosen clips per fold.\n",
    "* 9 folds (900 clips) for training, 1 fold (100 clips) for testing.\n",
    "* Determine a classification accuracy using testing set.\n",
    "* Repeat 10 times: mean and standard deviation.\n",
    "\n",
    "Observations:\n",
    "* Data should be shuffled as samples with the same label are contiguous, i.e. data ordering is not arbitrary.\n",
    "* *ShuffleSplit*, *StratifiedShuffleSplit*, *KFold* and *StratifiedKFold* yields similar results as long as data is shuffeld.\n",
    "* (Lots of variance between runs.)\n",
    "* Data should be rescaled for good performance (both accuracy and speed).\n",
    "\n",
    "Results:\n",
    "* With $X_a$ (best observed)\n",
    "    * Accuracy of 89 (+/- 5.0) for 2 genres (SVC, abs, minmax) (50s)\n",
    "    * Accuracy of 60 (+/- 7.9) for 2 genres (SVC, noabs, minmax) (100s)\n",
    "    * Accuracy of 64 (+/- 11.1) for 2 genres (SVC, noabs, std) (1000s)\n",
    "* With $X_s$ (best observed)\n",
    "    * Accuracy of 96 (+/- 4.7) for 2 genres (SVC, minmax) (2s, CDK 1s)\n",
    "    * Accuracy of 81 (+/- 4) for 4 genres (SVC, minmax) (14s)\n",
    "    * Accuracy of 76 (+/- 3.1) for 5 genres (SVC, minmax) (CDK 12s)\n",
    "    * Accuracy of 56 (+/- 5) for 10 genres (SVC, minmax) (300s)\n",
    "    * Accuracy of 53 (+/- 3) for 10 genres (LinearSVC, minmax) (300s)\n",
    "    * Accuracy of 51 (+/- 5) for 10 genres (NuSVC, minmax) (500s)\n",
    "* With $Z$ (best observed) (all with SVC, no normalization if not mentioned)\n",
    "    * Accuracy of 96 (+/- 3.2) for 2 genres (ld=10, m=128, minmax, 10 outer) (CDK 1s)\n",
    "    * Accuracy of 98 (+/- 2.4) for 2 genres (ld=10, m=128, 10 outer) (CDK 1s)\n",
    "    * Accuracy of 98 (+/- 2.5) for 2 genres (ld=10, m=128, encoder, 10 outer) (CDK 1s)\n",
    "    * Accuracy of 98 (+/- 2.5) for 2 genres (ld=10, m=128, 20 outer) (CDK 1s)\n",
    "    * Accuracy of 98 (+/- 2.5) for 2 genres (ld=100, m=128, 15 outer) (CDK 1s)\n",
    "    * Accuracy of 58 (+/- 10.5) for 2 genres (ld=1, m=128, 15 outer) (CDK 4s)\n",
    "    * Accuracy of 99 (+/- 3.2) for 2 genres (ld=10, m=512, 15 outer) (CDK 2s)\n",
    "    * Accuracy of 79 (+/- 2.7) for 5 genres (ld=10, m=512, 20 outer) (CDK 28s)\n",
    "    * Accuracy of 65 (+/- 3.6) for 10 genres (ld=10, m=512, 15 outer) (CDK 167s)\n",
    "\n",
    "Ideas:\n",
    "* Use the area under the receiver operating characteristing (ROC) curve (AUC). Not sure if applicable to multi-class."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "if majority_voting:\n",
    "    clf = clf_svm_vote\n",
    "    b = Nclips\n",
    "    c = Nvectors*2*n\n",
    "else:\n",
    "    clf = clf_svm\n",
    "    b = Nclips*Nvectors*2\n",
    "    c = n\n",
    "\n",
    "data, target = prepdata(Ngenres, b, c, scale=scale)\n",
    "print('Ratio: {} training, {} testing'.format(\n",
    "        (1-test_size)*target.size, test_size*target.size))\n",
    "\n",
    "tstart = time.time()\n",
    "scores = np.empty(shape=(Ncv, 10))\n",
    "\n",
    "# Cross-validation iterators.\n",
    "cv = cross_validation.ShuffleSplit(target.size, n_iter=10, test_size=test_size)\n",
    "#cv = cross_validation.StratifiedShuffleSplit(target.size, n_iter=10, test_size=test_size)\n",
    "#cv = cross_validation.KFold(target.size, shuffle=True, n_folds=10)\n",
    "#cv = cross_validation.StratifiedKFold(target, shuffle=True, n_folds=10)\n",
    "\n",
    "for i in range(Ncv):\n",
    "    \n",
    "    scores[i,:] = cross_validation.cross_val_score(\n",
    "                  clf, data, target, cv=cv, n_jobs=1)\n",
    "    \n",
    "    # Performance: accuracy.\n",
    "    mean, std = scores[i,:].mean()*100, scores[i,:].std()*100\n",
    "    print(' {:3.0f} (+/-{:4.1f}) <- {}'.format(mean, std, (scores[i,:]*100).astype(np.int)))\n",
    "\n",
    "accuracy, accuracy_std = scores.mean()*100, scores.std()*100\n",
    "print('Accuracy: {:.1f} (+/- {:.2f})'.format(accuracy, accuracy_std))\n",
    "\n",
    "meantime = (time.time() - tstart) / Ncv\n",
    "print('Mean time ({} cv): {:.2f} seconds'.format(Ncv, meantime))\n",
    "print('Overall time: {:.2f} seconds'.format(time.time() - toverall))"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}