src/doc/split.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "_xnMOsbqHz61"
   },
   "source": [
    "# Stratified splitting\n",
    "\n",
    "This notebook provides several tutorials on how to apply any algorithm introduced in the **straSplit** package for partitioning a multi-label dataset using [stratified strategy](https://bit.ly/3s3IDA8). Please install\n",
    "[anaconda](https://www.anaconda.com/) package and other modules listed in [requirement.txt](../../requirements.txt) file. **The data is assumed to have been written in [sparse matrix format](https://docs.scipy.org/doc/scipy/reference/sparse.html)**."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "_xnMOsbqHz61"
   },
   "source": [
    "## Load modules and datasets\n",
    "\n",
    "First, let us change the current directory to the `model` folder."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import os\n",
    "os.chdir('../model')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then, load the following modules."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "from IPython import display\n",
    "\n",
    "import pickle as pkl\n",
    "import pandas as pd\n",
    "\n",
    "## load utilities\n",
    "from utils import DATASET_PATH,RESULT_PATH, data_properties\n",
    "from utils import check_type, custom_shuffle, data_properties, LabelBinarizer\n",
    "\n",
    "## load modules\n",
    "from naive2split import NaiveStratification\n",
    "from iterative2split import IterativeStratification\n",
    "from extreme2split import ExtremeStratification\n",
    "from plssvd2split import ClusterStratification\n",
    "from eigencluster2split import ClusteringEigenStratification\n",
    "from comm2split import CommunityStratification\n",
    "from enhance2split import LabelEnhancementStratification\n",
    "from active2split import ActiveStratification\n",
    "\n",
    "## multi-label model\n",
    "from sklearn.multioutput import ClassifierChain\n",
    "from sklearn.svm import LinearSVC\n",
    "\n",
    "## evaluation metrics\n",
    "from sklearn.metrics import f1_score\n",
    "\n",
    "## Set dataframe to maxwidth\n",
    "pd.set_option('display.max_columns', None)\n",
    "pd.set_option('display.max_rows', None)\n",
    "pd.set_option('display.max_colwidth', None)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Ensure that **DATASET_PATH** (dataset folder) and **RESULT_PATH** (results folder) are properly set in the [utils.py](utils.py) module.\n",
    "\n",
    "Now, let us assign arbitrary values to the following arguments."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "split_type = \"extreme\"\n",
    "split_size = 0.80\n",
    "batch_size = 500\n",
    "num_epochs = 50\n",
    "num_jobs = 2\n",
    "num_clusters = 5\n",
    "use_solver = False"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "where `use_solver` suggests whether to utilize the sklearn based optimization algorithm ([SGDClassifier](https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.SGDClassifier.html)) or the custom build optimization algorithm for the `active2split` module. `split_type` takes only: `extreme\"`, `\"iterative\"`, and `\"naive\"` while `split_size` represents the proportion of the dataset to include in the training set."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Load the \"yeast\" dataset\n",
    "\n",
    "Let us use the well-known [*yeast*](https://www.openml.org/d/40597) multi-label data which consists of micro-array expression data, as well as phylogenetic profiles of yeast. The data is comprised of 2417 genes with 103 variables and each gene can be tagged using a subset of 14 distinct classes. This data is provided in the `sample` folder."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Size of the data:  (2417, 103)\n",
      "Label size of the data:  14\n"
     ]
    }
   ],
   "source": [
    "dsname=\"yeast\"\n",
    "X_name = dsname + \"_X.pkl\"\n",
    "y_name = dsname + \"_y.pkl\"\n",
    "file_path = os.path.join(DATASET_PATH, y_name)\n",
    "with open(file_path, mode=\"rb\") as f_in:\n",
    "    y = pkl.load(f_in)\n",
    "    idx = list(set(y.nonzero()[0]))\n",
    "    y = y[idx]\n",
    "\n",
    "file_path = os.path.join(DATASET_PATH, X_name)\n",
    "with open(file_path, mode=\"rb\") as f_in:\n",
    "    X = pkl.load(f_in)\n",
    "    X = X[idx]\n",
    "\n",
    "print(\"Size of the data: \", X.shape)\n",
    "print(\"Label size of the data: \", y.shape[1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Define the ClassifierChain model\n",
    "\n",
    "To evaluate the performance of training and test sets, we construct `Classifier Chains`([paper](https://link.springer.com/chapter/10.1007/978-3-642-04174-7_17)) model that arranges binary classifiers into a chain. This method is expected to presevre label correlations given a multi-label data.\n",
    "\n",
    "For the base estimator, we apply [LinearSVC](https://scikit-learn.org/stable/modules/generated/sklearn.svm.LinearSVC.html#sklearn.svm.LinearSVC) (linear support vector classification) using the default settings."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = ClassifierChain(LinearSVC(), order='random', random_state=12345)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For performance evaluation, in this notebook we use F1-score performance metric as implemented in the `score` method."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "def score(y_true, y_pred, tag:str = \"training set\"):\n",
    "    f1_samples_average = f1_score(y_true, y_pred, average='samples')\n",
    "    f1_samples_micro = f1_score(y_true, y_pred, average='micro')\n",
    "    f1_samples_macro = f1_score(y_true, y_pred, average='macro')\n",
    "    print('\\t>> F1-score for {0}...'.format(tag))\n",
    "    print('\\t\\t--> Average sample f1-score: {0:.4f}'.format(f1_samples_average))\n",
    "    print('\\t\\t--> Micro f1-score: {0:.4f}'.format(f1_samples_micro))\n",
    "    print('\\t\\t--> Macro f1-score: {0:.4f}'.format(f1_samples_macro))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Using the above data, the chain model, and configuration arguments, we will explore the performance of each splitting strategies."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Algorithms\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Naive approach\n",
    "\n",
    "This splitter was proposed in the [paper](https://doi.org/10.1371/journal.pcbi.1008174) and integrated into the metabolic pathway prediction software [mlLGPR](https://github.com/hallamlab/mlLGPR). This is an iterative procedure where at first it selects a label, independently of others, then finds examples associated with this label. Next, the algorithm splits data based on the `split_size` parameter and assigns examples to training and test sets accordingly. If an example was already being added to the test or training sets then continue the process by selecting another label at random. This process iterates until all examples are consumed in the splitting process. In most cases, the resulted training set may not be equal to the `split_size` parameter. This approach is scalable to large-scale data, although it suffers from the class imbalance problem and does not consider label-correlations to split a dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "## Configuration parameters to naive based stratified multi-label dataset\n",
      "   splitting:\n",
      "\t\t1. Shuffle the dataset? True\n",
      "\t\t2. Split size: 0.8\n",
      "\t\t3. Number of examples to use in each iteration: 500\n",
      "\t\t4. Number of parallel workers: 2\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\t>> Perform splitting...\n",
      "\t\t--> Splitting progress: 100.00%...\n"
     ]
    }
   ],
   "source": [
    "st = NaiveStratification(shuffle=True, split_size=split_size, batch_size=batch_size,\n",
    "                         num_jobs=num_jobs)\n",
    "training_idx, test_idx = st.fit(y=y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "where *training_idx* and *test_idx* are two lists corresponding the indices of yeast data.\n",
    "\n",
    "Let us explore some properties of the resulted training and test sets."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Properties for yeast</th>\n",
       "      <th>Complete set</th>\n",
       "      <th>Training set</th>\n",
       "      <th>Test set</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Number of examples</td>\n",
       "      <td>2417</td>\n",
       "      <td>1935</td>\n",
       "      <td>482</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Number of labels</td>\n",
       "      <td>10241</td>\n",
       "      <td>8181</td>\n",
       "      <td>2060</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Label cardinality</td>\n",
       "      <td>4.237071</td>\n",
       "      <td>4.227907</td>\n",
       "      <td>4.273859</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Label density</td>\n",
       "      <td>0.302648</td>\n",
       "      <td>0.301993</td>\n",
       "      <td>0.305276</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Distinct labels</td>\n",
       "      <td>14</td>\n",
       "      <td>14</td>\n",
       "      <td>14</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>Distinct label sets</td>\n",
       "      <td>198</td>\n",
       "      <td>182</td>\n",
       "      <td>102</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>Frequency of distinct label sets</td>\n",
       "      <td>0.08192</td>\n",
       "      <td>0.08192</td>\n",
       "      <td>0.08192</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>Mean imbalance ratio intra-class for all labels</td>\n",
       "      <td>9.578575</td>\n",
       "      <td>10.232741</td>\n",
       "      <td>8.081865</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>Mean imbalance ratio inter-class for all labels</td>\n",
       "      <td>7.196811</td>\n",
       "      <td>7.652081</td>\n",
       "      <td>6.187154</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>Mean imbalance ratio labelsets for all labels</td>\n",
       "      <td>0.468229</td>\n",
       "      <td>0.398937</td>\n",
       "      <td>0.674251</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>Labels having less than or equal to 5 examples</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>Labels having more than 6 examples</td>\n",
       "      <td>14</td>\n",
       "      <td>14</td>\n",
       "      <td>14</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>KL difference between complete and data partition</td>\n",
       "      <td>0</td>\n",
       "      <td>0.000095</td>\n",
       "      <td>0.001474</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                 Properties for yeast Complete set  \\\n",
       "0                                  Number of examples         2417   \n",
       "1                                    Number of labels        10241   \n",
       "2                                   Label cardinality     4.237071   \n",
       "3                                       Label density     0.302648   \n",
       "4                                     Distinct labels           14   \n",
       "5                                 Distinct label sets          198   \n",
       "6                    Frequency of distinct label sets      0.08192   \n",
       "7     Mean imbalance ratio intra-class for all labels     9.578575   \n",
       "8     Mean imbalance ratio inter-class for all labels     7.196811   \n",
       "9       Mean imbalance ratio labelsets for all labels     0.468229   \n",
       "10     Labels having less than or equal to 5 examples            0   \n",
       "11                 Labels having more than 6 examples           14   \n",
       "12  KL difference between complete and data partition            0   \n",
       "\n",
       "   Training set  Test set  \n",
       "0          1935       482  \n",
       "1          8181      2060  \n",
       "2      4.227907  4.273859  \n",
       "3      0.301993  0.305276  \n",
       "4            14        14  \n",
       "5           182       102  \n",
       "6       0.08192   0.08192  \n",
       "7     10.232741  8.081865  \n",
       "8      7.652081  6.187154  \n",
       "9      0.398937  0.674251  \n",
       "10            0         0  \n",
       "11           14        14  \n",
       "12     0.000095  0.001474  "
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model_name = \"naive2split\"\n",
    "df, chart = data_properties(y=y, selected_examples=[training_idx, test_idx], \n",
    "                            num_tails=5, dataset_name=dsname, model_name=model_name, \n",
    "                            rspath=RESULT_PATH, display_dataframe=True, \n",
    "                            display_figure=True)\n",
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "where *Label cardinality* is defined as the mean number of labels associated with an example, *Label density* is defined as cardinality divided by the number of labels, *Distinct label sets* is defined as the number of label combinations appearing in the dataset, *Ferequency of distinct label sets* is defined as the number of appearances of distinct labels divided by the total number of examples, and *KL* (Kullback-divergence) difference between complete and partitioned data measures the difference between two probability distributions where a low KL number entails that a partitioned data is closely approximating the complete data label distributions. *Mean imbalance ratio intra-class for all labels* represents the degree of imbalance within a label, *Mean imbalance ratio inter-class for all labels* represents the degree of imbalance among labels, and *Mean imbalance ratio labelsets for all labels* represents the degree of imbalance among labelsets.\n",
    "\n",
    "From the table, we observe that both training and test sets are close associated with the complete data in terms the KL metric. However, the three imbalance ratio metrics indicate that these datasets differ from each other. In fact, the test set contains less number of labelsets than yeast complete data. Unfortunately, this limitation was not addressed in this approach. \n",
    "\n",
    "Let us plot the resulted data in terms of the frequency of examples for each label."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "<div id=\"altair-viz-2986f308c82144d2a065ce85a4e6a383\"></div>\n",
       "<script type=\"text/javascript\">\n",
       "  (function(spec, embedOpt){\n",
       "    let outputDiv = document.currentScript.previousElementSibling;\n",
       "    if (outputDiv.id !== \"altair-viz-2986f308c82144d2a065ce85a4e6a383\") {\n",
       "      outputDiv = document.getElementById(\"altair-viz-2986f308c82144d2a065ce85a4e6a383\");\n",
       "    }\n",
       "    const paths = {\n",
       "      \"vega\": \"https://cdn.jsdelivr.net/npm//vega@5?noext\",\n",
       "      \"vega-lib\": \"https://cdn.jsdelivr.net/npm//vega-lib?noext\",\n",
       "      \"vega-lite\": \"https://cdn.jsdelivr.net/npm//vega-lite@4.8.1?noext\",\n",
       "      \"vega-embed\": \"https://cdn.jsdelivr.net/npm//vega-embed@6?noext\",\n",
       "    };\n",
       "\n",
       "    function loadScript(lib) {\n",
       "      return new Promise(function(resolve, reject) {\n",
       "        var s = document.createElement('script');\n",
       "        s.src = paths[lib];\n",
       "        s.async = true;\n",
       "        s.onload = () => resolve(paths[lib]);\n",
       "        s.onerror = () => reject(`Error loading script: ${paths[lib]}`);\n",
       "        document.getElementsByTagName(\"head\")[0].appendChild(s);\n",
       "      });\n",
       "    }\n",
       "\n",
       "    function showError(err) {\n",
       "      outputDiv.innerHTML = `<div class=\"error\" style=\"color:red;\">${err}</div>`;\n",
       "      throw err;\n",
       "    }\n",
       "\n",
       "    function displayChart(vegaEmbed) {\n",
       "      vegaEmbed(outputDiv, spec, embedOpt)\n",
       "        .catch(err => showError(`Javascript Error: ${err.message}<br>This usually means there's a typo in your chart specification. See the javascript console for the full traceback.`));\n",
       "    }\n",
       "\n",
       "    if(typeof define === \"function\" && define.amd) {\n",
       "      requirejs.config({paths});\n",
       "      require([\"vega-embed\"], displayChart, err => showError(`Error loading script: ${err.message}`));\n",
       "    } else if (typeof vegaEmbed === \"function\") {\n",
       "      displayChart(vegaEmbed);\n",
       "    } else {\n",
       "      loadScript(\"vega\")\n",
       "        .then(() => loadScript(\"vega-lite\"))\n",
       "        .then(() => loadScript(\"vega-embed\"))\n",
       "        .catch(showError)\n",
       "        .then(() => displayChart(vegaEmbed));\n",
       "    }\n",
       "  })({\"data\": {\"name\": \"data-faeee7e1cbca148f6845fb37fa46a179\"}, \"mark\": {\"type\": \"bar\", \"color\": \"grey\"}, \"config\": {\"axis\": {\"labelFontSize\": 12, \"labelLimit\": 500, \"labelPadding\": 5, \"titleFontSize\": 20}, \"axisY\": {\"grid\": false}, \"header\": {\"labelFontSize\": 15, \"titleFontSize\": 20}, \"legend\": {\"cornerRadius\": 10, \"fillColor\": \"white\", \"padding\": 10, \"strokeColor\": \"gray\"}}, \"encoding\": {\"color\": {\"type\": \"nominal\", \"field\": \"Dataset\", \"scale\": {\"range\": [\"red\", \"black\", \"blue\"]}}, \"x\": {\"type\": \"ordinal\", \"field\": \"Label\", \"sort\": \"ascending\", \"title\": \"Label ID\"}, \"y\": {\"type\": \"quantitative\", \"field\": \"Sum\", \"stack\": null, \"title\": \"Number of Examples\"}}, \"height\": 350, \"resolve\": {\"scale\": {\"x\": \"independent\"}}, \"title\": {\"text\": \"Number of examples for each label given yeast data.\", \"subtitle\": [\"The horizontal axis indicates the indices of labels while the vertical axis represents the number of associated examples\"]}, \"width\": 600, \"$schema\": \"https://vega.github.io/schema/vega-lite/v4.8.1.json\", \"datasets\": {\"data-faeee7e1cbca148f6845fb37fa46a179\": [{\"Label\": 1, \"Dataset\": \"Complete\", \"Sum\": 34}, {\"Label\": 2, \"Dataset\": \"Complete\", \"Sum\": 178}, {\"Label\": 3, \"Dataset\": \"Complete\", \"Sum\": 253}, {\"Label\": 4, \"Dataset\": \"Complete\", \"Sum\": 289}, {\"Label\": 5, \"Dataset\": \"Complete\", \"Sum\": 428}, {\"Label\": 6, \"Dataset\": \"Complete\", \"Sum\": 480}, {\"Label\": 7, \"Dataset\": \"Complete\", \"Sum\": 597}, {\"Label\": 8, \"Dataset\": \"Complete\", \"Sum\": 722}, {\"Label\": 9, \"Dataset\": \"Complete\", \"Sum\": 762}, {\"Label\": 10, \"Dataset\": \"Complete\", \"Sum\": 862}, {\"Label\": 11, \"Dataset\": \"Complete\", \"Sum\": 983}, {\"Label\": 12, \"Dataset\": \"Complete\", \"Sum\": 1038}, {\"Label\": 13, \"Dataset\": \"Complete\", \"Sum\": 1799}, {\"Label\": 14, \"Dataset\": \"Complete\", \"Sum\": 1816}, {\"Label\": 1, \"Dataset\": \"Train\", \"Sum\": 24}, {\"Label\": 2, \"Dataset\": \"Train\", \"Sum\": 141}, {\"Label\": 3, \"Dataset\": \"Train\", \"Sum\": 205}, {\"Label\": 4, \"Dataset\": \"Train\", \"Sum\": 238}, {\"Label\": 5, \"Dataset\": \"Train\", \"Sum\": 354}, {\"Label\": 6, \"Dataset\": \"Train\", \"Sum\": 391}, {\"Label\": 7, \"Dataset\": \"Train\", \"Sum\": 482}, {\"Label\": 8, \"Dataset\": \"Train\", \"Sum\": 574}, {\"Label\": 9, \"Dataset\": \"Train\", \"Sum\": 610}, {\"Label\": 10, \"Dataset\": \"Train\", \"Sum\": 683}, {\"Label\": 11, \"Dataset\": \"Train\", \"Sum\": 774}, {\"Label\": 12, \"Dataset\": \"Train\", \"Sum\": 824}, {\"Label\": 13, \"Dataset\": \"Train\", \"Sum\": 1434}, {\"Label\": 14, \"Dataset\": \"Train\", \"Sum\": 1447}, {\"Label\": 1, \"Dataset\": \"Test\", \"Sum\": 10}, {\"Label\": 2, \"Dataset\": \"Test\", \"Sum\": 37}, {\"Label\": 3, \"Dataset\": \"Test\", \"Sum\": 48}, {\"Label\": 4, \"Dataset\": \"Test\", \"Sum\": 51}, {\"Label\": 5, \"Dataset\": \"Test\", \"Sum\": 74}, {\"Label\": 6, \"Dataset\": \"Test\", \"Sum\": 89}, {\"Label\": 7, \"Dataset\": \"Test\", \"Sum\": 115}, {\"Label\": 8, \"Dataset\": \"Test\", \"Sum\": 148}, {\"Label\": 9, \"Dataset\": \"Test\", \"Sum\": 152}, {\"Label\": 10, \"Dataset\": \"Test\", \"Sum\": 179}, {\"Label\": 11, \"Dataset\": \"Test\", \"Sum\": 209}, {\"Label\": 12, \"Dataset\": \"Test\", \"Sum\": 214}, {\"Label\": 13, \"Dataset\": \"Test\", \"Sum\": 365}, {\"Label\": 14, \"Dataset\": \"Test\", \"Sum\": 369}]}}, {\"mode\": \"vega-lite\"});\n",
       "</script>"
      ],
      "text/plain": [
       "alt.Chart(...)"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "chart"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The above figure follows the power law distribution where $50\\%$ of labels are found to occur in less than $25\\% \\sim 30\\%$ of examples while we observe that the last two labels $13$ and $14$ are frequently appearing in the yeast data. This figure demonstrates the well-known class-imbalance problem."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Lets see the performance of splitted data by using ClassifierChain model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "ClassifierChain(base_estimator=LinearSVC(), order='random', random_state=12345)"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# train\n",
    "model.fit(X[training_idx].toarray(), y[training_idx].toarray())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\t>> F1-score for training set...\n",
      "\t\t--> Average sample f1-score: 0.6364\n",
      "\t\t--> Micro f1-score: 0.6586\n",
      "\t\t--> Macro f1-score: 0.4053\n",
      "\t>> F1-score for test set...\n",
      "\t\t--> Average sample f1-score: 0.6198\n",
      "\t\t--> Micro f1-score: 0.6444\n",
      "\t\t--> Macro f1-score: 0.3566\n"
     ]
    }
   ],
   "source": [
    "# evaluate\n",
    "y_pred = model.predict(X[training_idx].toarray())\n",
    "score(y_true=y[training_idx], y_pred=y_pred, tag=\"training set\")\n",
    "\n",
    "y_pred = model.predict(X[test_idx].toarray())\n",
    "score(y_true=y[test_idx], y_pred=y_pred, tag=\"test set\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It seems the model performs similar to both datasets albeit these two data differ from each other with regard to the imbalance ratio."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Iterative approach\n",
    "\n",
    "This is a modified algorithm from the [paper](https://link.springer.com/chapter/10.1007/978-3-642-23808-6_10), which performs iterative splitting to the dataset. The algorithm starts by calculating the desired number of examples and proportions for training and test sets. This is followed by estimating the desired number of examples of each label at each partition. Then, for each example of a selected label, the algorithm chooses appropriate partitions. Once the appropriate subset is selected, the example is added to the partition while decrementing the number of desired examples for each label of this example as well as the total number of desired examples for that group."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "## Configuration parameters to iteratively stratifying a multi-label\n",
      "   dataset splitting:\n",
      "\t\t1. Shuffle the dataset? True\n",
      "\t\t2. Split size: 0.8\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\t>> Perform splitting (iterative)...\n",
      "\t\t--> Splitting progress: 100.00%...\r"
     ]
    }
   ],
   "source": [
    "st = IterativeStratification(shuffle=True, split_size=split_size)\n",
    "training_idx, test_idx = st.fit(y=y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Properties for yeast</th>\n",
       "      <th>Complete set</th>\n",
       "      <th>Training set</th>\n",
       "      <th>Test set</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Number of examples</td>\n",
       "      <td>2417</td>\n",
       "      <td>1940</td>\n",
       "      <td>477</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Number of labels</td>\n",
       "      <td>10241</td>\n",
       "      <td>8182</td>\n",
       "      <td>2059</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Label cardinality</td>\n",
       "      <td>4.237071</td>\n",
       "      <td>4.217526</td>\n",
       "      <td>4.316562</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Label density</td>\n",
       "      <td>0.302648</td>\n",
       "      <td>0.301252</td>\n",
       "      <td>0.308326</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Distinct labels</td>\n",
       "      <td>14</td>\n",
       "      <td>14</td>\n",
       "      <td>14</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>Distinct label sets</td>\n",
       "      <td>198</td>\n",
       "      <td>174</td>\n",
       "      <td>109</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>Frequency of distinct label sets</td>\n",
       "      <td>0.08192</td>\n",
       "      <td>0.08192</td>\n",
       "      <td>0.08192</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>Mean imbalance ratio intra-class for all labels</td>\n",
       "      <td>9.578575</td>\n",
       "      <td>10.097934</td>\n",
       "      <td>8.090211</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>Mean imbalance ratio inter-class for all labels</td>\n",
       "      <td>7.196811</td>\n",
       "      <td>7.56304</td>\n",
       "      <td>6.156702</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>Mean imbalance ratio labelsets for all labels</td>\n",
       "      <td>0.468229</td>\n",
       "      <td>0.602303</td>\n",
       "      <td>1.985403</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>Labels having less than or equal to 5 examples</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>Labels having more than 6 examples</td>\n",
       "      <td>14</td>\n",
       "      <td>14</td>\n",
       "      <td>14</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>KL difference between complete and data partition</td>\n",
       "      <td>0</td>\n",
       "      <td>0.000085</td>\n",
       "      <td>0.00131</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                 Properties for yeast Complete set  \\\n",
       "0                                  Number of examples         2417   \n",
       "1                                    Number of labels        10241   \n",
       "2                                   Label cardinality     4.237071   \n",
       "3                                       Label density     0.302648   \n",
       "4                                     Distinct labels           14   \n",
       "5                                 Distinct label sets          198   \n",
       "6                    Frequency of distinct label sets      0.08192   \n",
       "7     Mean imbalance ratio intra-class for all labels     9.578575   \n",
       "8     Mean imbalance ratio inter-class for all labels     7.196811   \n",
       "9       Mean imbalance ratio labelsets for all labels     0.468229   \n",
       "10     Labels having less than or equal to 5 examples            0   \n",
       "11                 Labels having more than 6 examples           14   \n",
       "12  KL difference between complete and data partition            0   \n",
       "\n",
       "   Training set  Test set  \n",
       "0          1940       477  \n",
       "1          8182      2059  \n",
       "2      4.217526  4.316562  \n",
       "3      0.301252  0.308326  \n",
       "4            14        14  \n",
       "5           174       109  \n",
       "6       0.08192   0.08192  \n",
       "7     10.097934  8.090211  \n",
       "8       7.56304  6.156702  \n",
       "9      0.602303  1.985403  \n",
       "10            0         0  \n",
       "11           14        14  \n",
       "12     0.000085   0.00131  "
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model_name = \"iterative2split\"\n",
    "df, chart = data_properties(y=y, selected_examples=[training_idx, test_idx], \n",
    "                            num_tails=5, dataset_name=dsname, model_name=model_name, \n",
    "                            rspath=RESULT_PATH, display_dataframe=True, \n",
    "                            display_figure=True)\n",
    "df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "<div id=\"altair-viz-7689058a03f84cd88ad15622d1ac097b\"></div>\n",
       "<script type=\"text/javascript\">\n",
       "  (function(spec, embedOpt){\n",
       "    let outputDiv = document.currentScript.previousElementSibling;\n",
       "    if (outputDiv.id !== \"altair-viz-7689058a03f84cd88ad15622d1ac097b\") {\n",
       "      outputDiv = document.getElementById(\"altair-viz-7689058a03f84cd88ad15622d1ac097b\");\n",
       "    }\n",
       "    const paths = {\n",
       "      \"vega\": \"https://cdn.jsdelivr.net/npm//vega@5?noext\",\n",
       "      \"vega-lib\": \"https://cdn.jsdelivr.net/npm//vega-lib?noext\",\n",
       "      \"vega-lite\": \"https://cdn.jsdelivr.net/npm//vega-lite@4.8.1?noext\",\n",
       "      \"vega-embed\": \"https://cdn.jsdelivr.net/npm//vega-embed@6?noext\",\n",
       "    };\n",
       "\n",
       "    function loadScript(lib) {\n",
       "      return new Promise(function(resolve, reject) {\n",
       "        var s = document.createElement('script');\n",
       "        s.src = paths[lib];\n",
       "        s.async = true;\n",
       "        s.onload = () => resolve(paths[lib]);\n",
       "        s.onerror = () => reject(`Error loading script: ${paths[lib]}`);\n",
       "        document.getElementsByTagName(\"head\")[0].appendChild(s);\n",
       "      });\n",
       "    }\n",
       "\n",
       "    function showError(err) {\n",
       "      outputDiv.innerHTML = `<div class=\"error\" style=\"color:red;\">${err}</div>`;\n",
       "      throw err;\n",
       "    }\n",
       "\n",
       "    function displayChart(vegaEmbed) {\n",
       "      vegaEmbed(outputDiv, spec, embedOpt)\n",
       "        .catch(err => showError(`Javascript Error: ${err.message}<br>This usually means there's a typo in your chart specification. See the javascript console for the full traceback.`));\n",
       "    }\n",
       "\n",
       "    if(typeof define === \"function\" && define.amd) {\n",
       "      requirejs.config({paths});\n",
       "      require([\"vega-embed\"], displayChart, err => showError(`Error loading script: ${err.message}`));\n",
       "    } else if (typeof vegaEmbed === \"function\") {\n",
       "      displayChart(vegaEmbed);\n",
       "    } else {\n",
       "      loadScript(\"vega\")\n",
       "        .then(() => loadScript(\"vega-lite\"))\n",
       "        .then(() => loadScript(\"vega-embed\"))\n",
       "        .catch(showError)\n",
       "        .then(() => displayChart(vegaEmbed));\n",
       "    }\n",
       "  })({\"data\": {\"name\": \"data-c062a8b6c40cbaa34e9877281738f488\"}, \"mark\": {\"type\": \"bar\", \"color\": \"grey\"}, \"config\": {\"axis\": {\"labelFontSize\": 12, \"labelLimit\": 500, \"labelPadding\": 5, \"titleFontSize\": 20}, \"axisY\": {\"grid\": false}, \"header\": {\"labelFontSize\": 15, \"titleFontSize\": 20}, \"legend\": {\"cornerRadius\": 10, \"fillColor\": \"white\", \"padding\": 10, \"strokeColor\": \"gray\"}}, \"encoding\": {\"color\": {\"type\": \"nominal\", \"field\": \"Dataset\", \"scale\": {\"range\": [\"red\", \"black\", \"blue\"]}}, \"x\": {\"type\": \"ordinal\", \"field\": \"Label\", \"sort\": \"ascending\", \"title\": \"Label ID\"}, \"y\": {\"type\": \"quantitative\", \"field\": \"Sum\", \"stack\": null, \"title\": \"Number of Examples\"}}, \"height\": 350, \"resolve\": {\"scale\": {\"x\": \"independent\"}}, \"title\": {\"text\": \"Number of examples for each label given yeast data.\", \"subtitle\": [\"The horizontal axis indicates the indices of labels while the vertical axis represents the number of associated examples\"]}, \"width\": 600, \"$schema\": \"https://vega.github.io/schema/vega-lite/v4.8.1.json\", \"datasets\": {\"data-c062a8b6c40cbaa34e9877281738f488\": [{\"Label\": 1, \"Dataset\": \"Complete\", \"Sum\": 34}, {\"Label\": 2, \"Dataset\": \"Complete\", \"Sum\": 178}, {\"Label\": 3, \"Dataset\": \"Complete\", \"Sum\": 253}, {\"Label\": 4, \"Dataset\": \"Complete\", \"Sum\": 289}, {\"Label\": 5, \"Dataset\": \"Complete\", \"Sum\": 428}, {\"Label\": 6, \"Dataset\": \"Complete\", \"Sum\": 480}, {\"Label\": 7, \"Dataset\": \"Complete\", \"Sum\": 597}, {\"Label\": 8, \"Dataset\": \"Complete\", \"Sum\": 722}, {\"Label\": 9, \"Dataset\": \"Complete\", \"Sum\": 762}, {\"Label\": 10, \"Dataset\": \"Complete\", \"Sum\": 862}, {\"Label\": 11, \"Dataset\": \"Complete\", \"Sum\": 983}, {\"Label\": 12, \"Dataset\": \"Complete\", \"Sum\": 1038}, {\"Label\": 13, \"Dataset\": \"Complete\", \"Sum\": 1799}, {\"Label\": 14, \"Dataset\": \"Complete\", \"Sum\": 1816}, {\"Label\": 1, \"Dataset\": \"Train\", \"Sum\": 25}, {\"Label\": 2, \"Dataset\": \"Train\", \"Sum\": 141}, {\"Label\": 3, \"Dataset\": \"Train\", \"Sum\": 201}, {\"Label\": 4, \"Dataset\": \"Train\", \"Sum\": 229}, {\"Label\": 5, \"Dataset\": \"Train\", \"Sum\": 333}, {\"Label\": 6, \"Dataset\": \"Train\", \"Sum\": 373}, {\"Label\": 7, \"Dataset\": \"Train\", \"Sum\": 473}, {\"Label\": 8, \"Dataset\": \"Train\", \"Sum\": 582}, {\"Label\": 9, \"Dataset\": \"Train\", \"Sum\": 605}, {\"Label\": 10, \"Dataset\": \"Train\", \"Sum\": 702}, {\"Label\": 11, \"Dataset\": \"Train\", \"Sum\": 800}, {\"Label\": 12, \"Dataset\": \"Train\", \"Sum\": 826}, {\"Label\": 13, \"Dataset\": \"Train\", \"Sum\": 1439}, {\"Label\": 14, \"Dataset\": \"Train\", \"Sum\": 1453}, {\"Label\": 1, \"Dataset\": \"Test\", \"Sum\": 9}, {\"Label\": 2, \"Dataset\": \"Test\", \"Sum\": 37}, {\"Label\": 3, \"Dataset\": \"Test\", \"Sum\": 52}, {\"Label\": 4, \"Dataset\": \"Test\", \"Sum\": 60}, {\"Label\": 5, \"Dataset\": \"Test\", \"Sum\": 95}, {\"Label\": 6, \"Dataset\": \"Test\", \"Sum\": 107}, {\"Label\": 7, \"Dataset\": \"Test\", \"Sum\": 124}, {\"Label\": 8, \"Dataset\": \"Test\", \"Sum\": 140}, {\"Label\": 9, \"Dataset\": \"Test\", \"Sum\": 157}, {\"Label\": 10, \"Dataset\": \"Test\", \"Sum\": 160}, {\"Label\": 11, \"Dataset\": \"Test\", \"Sum\": 183}, {\"Label\": 12, \"Dataset\": \"Test\", \"Sum\": 212}, {\"Label\": 13, \"Dataset\": \"Test\", \"Sum\": 360}, {\"Label\": 14, \"Dataset\": \"Test\", \"Sum\": 363}]}}, {\"mode\": \"vega-lite\"});\n",
       "</script>"
      ],
      "text/plain": [
       "alt.Chart(...)"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "chart"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "As can be observed that this algorithm produces less optimum results (KL metric) in comparison to the Naive based approach."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\t>> F1-score for training set...\n",
      "\t\t--> Average sample f1-score: 0.6435\n",
      "\t\t--> Micro f1-score: 0.6637\n",
      "\t\t--> Macro f1-score: 0.3979\n",
      "\t>> F1-score for test set...\n",
      "\t\t--> Average sample f1-score: 0.5976\n",
      "\t\t--> Micro f1-score: 0.6209\n",
      "\t\t--> Macro f1-score: 0.3545\n"
     ]
    }
   ],
   "source": [
    "# train\n",
    "model.fit(X[training_idx].toarray(), y[training_idx].toarray())\n",
    "# evaluate\n",
    "y_pred = model.predict(X[training_idx].toarray())\n",
    "score(y_true=y[training_idx], y_pred=y_pred, tag=\"training set\")\n",
    "\n",
    "y_pred = model.predict(X[test_idx].toarray())\n",
    "score(y_true=y[test_idx], y_pred=y_pred, tag=\"test set\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The results suggest this approach is less optimum than the naive approach."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Stratifying XML data approach\n",
    "\n",
    "Stratifying XML, proposed in the [paper](https://arxiv.org/abs/2103.03494), starts by randomly allocating each example to the training or test sets according to the `split_size` parameter. Then, it performs iterative stratification for `num_epochs` rounds. At the beginning, the algorithm computes the number of examples of each label being assigned to each partition. Then, at each iteration, it computes a score for each label describing the proximity to which a label's train size diverges from the `split_size` parameter. Next, it calculates a score associated with each example according to the scores of its labels for training and test sets. A high example score entails that many of its labels have too many examples associated with a partition, and, preferably, that example should be exchanged to the other partition. This process iterates for `num_epochs` rounds, and at the end, the algorithm terminates while producing the resulted partitions.\n",
    "\n",
    "Let's apply the algorithm with default settings."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "id": "iuGVPOo7Cce0"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "## Configuration parameters to stratifying a large scale multi-label\n",
      "   dataset splitting:\n",
      "\t\t1. A hyper-parameter for extreme stratification: 0.1\n",
      "\t\t2. A hyper-parameter for extreme stratification: 0.1\n",
      "\t\t3. A hyper-parameter for extreme stratification: 0.1\n",
      "\t\t4. Shuffle the dataset? True\n",
      "\t\t5. Split size: 0.8\n",
      "\t\t6. Number of loops over a dataset: 50\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\t>> Perform splitting (extreme)...\n",
      "\t\t--> Starting score: 96\n",
      "\t\t--> Splitting progress: 100.00%; score: -2.19\n"
     ]
    }
   ],
   "source": [
    "st = ExtremeStratification(swap_probability=0.1, threshold_proportion=0.1, decay=0.1,\n",
    "                           shuffle=True, split_size=split_size, num_epochs=num_epochs)\n",
    "training_idx, test_idx = st.fit(X=X, y=y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Properties for yeast</th>\n",
       "      <th>Complete set</th>\n",
       "      <th>Training set</th>\n",
       "      <th>Test set</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Number of examples</td>\n",
       "      <td>2417</td>\n",
       "      <td>1931</td>\n",
       "      <td>486</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Number of labels</td>\n",
       "      <td>10241</td>\n",
       "      <td>8142</td>\n",
       "      <td>2099</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Label cardinality</td>\n",
       "      <td>4.237071</td>\n",
       "      <td>4.216468</td>\n",
       "      <td>4.31893</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Label density</td>\n",
       "      <td>0.302648</td>\n",
       "      <td>0.301176</td>\n",
       "      <td>0.308495</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Distinct labels</td>\n",
       "      <td>14</td>\n",
       "      <td>14</td>\n",
       "      <td>14</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>Distinct label sets</td>\n",
       "      <td>198</td>\n",
       "      <td>177</td>\n",
       "      <td>110</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>Frequency of distinct label sets</td>\n",
       "      <td>0.08192</td>\n",
       "      <td>0.08192</td>\n",
       "      <td>0.08192</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>Mean imbalance ratio intra-class for all labels</td>\n",
       "      <td>9.578575</td>\n",
       "      <td>10.479014</td>\n",
       "      <td>7.77096</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>Mean imbalance ratio inter-class for all labels</td>\n",
       "      <td>7.196811</td>\n",
       "      <td>7.86333</td>\n",
       "      <td>5.868194</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>Mean imbalance ratio labelsets for all labels</td>\n",
       "      <td>0.468229</td>\n",
       "      <td>0.371309</td>\n",
       "      <td>0.690851</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>Labels having less than or equal to 5 examples</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>Labels having more than 6 examples</td>\n",
       "      <td>14</td>\n",
       "      <td>14</td>\n",
       "      <td>14</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>KL difference between complete and data partition</td>\n",
       "      <td>0</td>\n",
       "      <td>0.000204</td>\n",
       "      <td>0.002993</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                 Properties for yeast Complete set  \\\n",
       "0                                  Number of examples         2417   \n",
       "1                                    Number of labels        10241   \n",
       "2                                   Label cardinality     4.237071   \n",
       "3                                       Label density     0.302648   \n",
       "4                                     Distinct labels           14   \n",
       "5                                 Distinct label sets          198   \n",
       "6                    Frequency of distinct label sets      0.08192   \n",
       "7     Mean imbalance ratio intra-class for all labels     9.578575   \n",
       "8     Mean imbalance ratio inter-class for all labels     7.196811   \n",
       "9       Mean imbalance ratio labelsets for all labels     0.468229   \n",
       "10     Labels having less than or equal to 5 examples            0   \n",
       "11                 Labels having more than 6 examples           14   \n",
       "12  KL difference between complete and data partition            0   \n",
       "\n",
       "   Training set  Test set  \n",
       "0          1931       486  \n",
       "1          8142      2099  \n",
       "2      4.216468   4.31893  \n",
       "3      0.301176  0.308495  \n",
       "4            14        14  \n",
       "5           177       110  \n",
       "6       0.08192   0.08192  \n",
       "7     10.479014   7.77096  \n",
       "8       7.86333  5.868194  \n",
       "9      0.371309  0.690851  \n",
       "10            0         0  \n",
       "11           14        14  \n",
       "12     0.000204  0.002993  "
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model_name = \"extreme2split\"\n",
    "df, chart = data_properties(y=y, selected_examples=[training_idx, test_idx], \n",
    "                            num_tails=5, dataset_name=dsname, model_name=model_name, \n",
    "                            rspath=RESULT_PATH, display_dataframe=True, \n",
    "                            display_figure=True)\n",
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Again, we train the chain classifier using the resulted training and test sets."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "<div id=\"altair-viz-e40628adc53242faa6e9527c69710481\"></div>\n",
       "<script type=\"text/javascript\">\n",
       "  (function(spec, embedOpt){\n",
       "    let outputDiv = document.currentScript.previousElementSibling;\n",
       "    if (outputDiv.id !== \"altair-viz-e40628adc53242faa6e9527c69710481\") {\n",
       "      outputDiv = document.getElementById(\"altair-viz-e40628adc53242faa6e9527c69710481\");\n",
       "    }\n",
       "    const paths = {\n",
       "      \"vega\": \"https://cdn.jsdelivr.net/npm//vega@5?noext\",\n",
       "      \"vega-lib\": \"https://cdn.jsdelivr.net/npm//vega-lib?noext\",\n",
       "      \"vega-lite\": \"https://cdn.jsdelivr.net/npm//vega-lite@4.8.1?noext\",\n",
       "      \"vega-embed\": \"https://cdn.jsdelivr.net/npm//vega-embed@6?noext\",\n",
       "    };\n",
       "\n",
       "    function loadScript(lib) {\n",
       "      return new Promise(function(resolve, reject) {\n",
       "        var s = document.createElement('script');\n",
       "        s.src = paths[lib];\n",
       "        s.async = true;\n",
       "        s.onload = () => resolve(paths[lib]);\n",
       "        s.onerror = () => reject(`Error loading script: ${paths[lib]}`);\n",
       "        document.getElementsByTagName(\"head\")[0].appendChild(s);\n",
       "      });\n",
       "    }\n",
       "\n",
       "    function showError(err) {\n",
       "      outputDiv.innerHTML = `<div class=\"error\" style=\"color:red;\">${err}</div>`;\n",
       "      throw err;\n",
       "    }\n",
       "\n",
       "    function displayChart(vegaEmbed) {\n",
       "      vegaEmbed(outputDiv, spec, embedOpt)\n",
       "        .catch(err => showError(`Javascript Error: ${err.message}<br>This usually means there's a typo in your chart specification. See the javascript console for the full traceback.`));\n",
       "    }\n",
       "\n",
       "    if(typeof define === \"function\" && define.amd) {\n",
       "      requirejs.config({paths});\n",
       "      require([\"vega-embed\"], displayChart, err => showError(`Error loading script: ${err.message}`));\n",
       "    } else if (typeof vegaEmbed === \"function\") {\n",
       "      displayChart(vegaEmbed);\n",
       "    } else {\n",
       "      loadScript(\"vega\")\n",
       "        .then(() => loadScript(\"vega-lite\"))\n",
       "        .then(() => loadScript(\"vega-embed\"))\n",
       "        .catch(showError)\n",
       "        .then(() => displayChart(vegaEmbed));\n",
       "    }\n",
       "  })({\"data\": {\"name\": \"data-668f8040dcfe2e60425ef04684308304\"}, \"mark\": {\"type\": \"bar\", \"color\": \"grey\"}, \"config\": {\"axis\": {\"labelFontSize\": 12, \"labelLimit\": 500, \"labelPadding\": 5, \"titleFontSize\": 20}, \"axisY\": {\"grid\": false}, \"header\": {\"labelFontSize\": 15, \"titleFontSize\": 20}, \"legend\": {\"cornerRadius\": 10, \"fillColor\": \"white\", \"padding\": 10, \"strokeColor\": \"gray\"}}, \"encoding\": {\"color\": {\"type\": \"nominal\", \"field\": \"Dataset\", \"scale\": {\"range\": [\"red\", \"black\", \"blue\"]}}, \"x\": {\"type\": \"ordinal\", \"field\": \"Label\", \"sort\": \"ascending\", \"title\": \"Label ID\"}, \"y\": {\"type\": \"quantitative\", \"field\": \"Sum\", \"stack\": null, \"title\": \"Number of Examples\"}}, \"height\": 350, \"resolve\": {\"scale\": {\"x\": \"independent\"}}, \"title\": {\"text\": \"Number of examples for each label given yeast data.\", \"subtitle\": [\"The horizontal axis indicates the indices of labels while the vertical axis represents the number of associated examples\"]}, \"width\": 600, \"$schema\": \"https://vega.github.io/schema/vega-lite/v4.8.1.json\", \"datasets\": {\"data-668f8040dcfe2e60425ef04684308304\": [{\"Label\": 1, \"Dataset\": \"Complete\", \"Sum\": 34}, {\"Label\": 2, \"Dataset\": \"Complete\", \"Sum\": 178}, {\"Label\": 3, \"Dataset\": \"Complete\", \"Sum\": 253}, {\"Label\": 4, \"Dataset\": \"Complete\", \"Sum\": 289}, {\"Label\": 5, \"Dataset\": \"Complete\", \"Sum\": 428}, {\"Label\": 6, \"Dataset\": \"Complete\", \"Sum\": 480}, {\"Label\": 7, \"Dataset\": \"Complete\", \"Sum\": 597}, {\"Label\": 8, \"Dataset\": \"Complete\", \"Sum\": 722}, {\"Label\": 9, \"Dataset\": \"Complete\", \"Sum\": 762}, {\"Label\": 10, \"Dataset\": \"Complete\", \"Sum\": 862}, {\"Label\": 11, \"Dataset\": \"Complete\", \"Sum\": 983}, {\"Label\": 12, \"Dataset\": \"Complete\", \"Sum\": 1038}, {\"Label\": 13, \"Dataset\": \"Complete\", \"Sum\": 1799}, {\"Label\": 14, \"Dataset\": \"Complete\", \"Sum\": 1816}, {\"Label\": 1, \"Dataset\": \"Train\", \"Sum\": 23}, {\"Label\": 2, \"Dataset\": \"Train\", \"Sum\": 142}, {\"Label\": 3, \"Dataset\": \"Train\", \"Sum\": 201}, {\"Label\": 4, \"Dataset\": \"Train\", \"Sum\": 237}, {\"Label\": 5, \"Dataset\": \"Train\", \"Sum\": 354}, {\"Label\": 6, \"Dataset\": \"Train\", \"Sum\": 397}, {\"Label\": 7, \"Dataset\": \"Train\", \"Sum\": 480}, {\"Label\": 8, \"Dataset\": \"Train\", \"Sum\": 564}, {\"Label\": 9, \"Dataset\": \"Train\", \"Sum\": 608}, {\"Label\": 10, \"Dataset\": \"Train\", \"Sum\": 664}, {\"Label\": 11, \"Dataset\": \"Train\", \"Sum\": 765}, {\"Label\": 12, \"Dataset\": \"Train\", \"Sum\": 824}, {\"Label\": 13, \"Dataset\": \"Train\", \"Sum\": 1434}, {\"Label\": 14, \"Dataset\": \"Train\", \"Sum\": 1449}, {\"Label\": 1, \"Dataset\": \"Test\", \"Sum\": 11}, {\"Label\": 2, \"Dataset\": \"Test\", \"Sum\": 36}, {\"Label\": 3, \"Dataset\": \"Test\", \"Sum\": 52}, {\"Label\": 4, \"Dataset\": \"Test\", \"Sum\": 52}, {\"Label\": 5, \"Dataset\": \"Test\", \"Sum\": 74}, {\"Label\": 6, \"Dataset\": \"Test\", \"Sum\": 83}, {\"Label\": 7, \"Dataset\": \"Test\", \"Sum\": 117}, {\"Label\": 8, \"Dataset\": \"Test\", \"Sum\": 158}, {\"Label\": 9, \"Dataset\": \"Test\", \"Sum\": 154}, {\"Label\": 10, \"Dataset\": \"Test\", \"Sum\": 198}, {\"Label\": 11, \"Dataset\": \"Test\", \"Sum\": 218}, {\"Label\": 12, \"Dataset\": \"Test\", \"Sum\": 214}, {\"Label\": 13, \"Dataset\": \"Test\", \"Sum\": 365}, {\"Label\": 14, \"Dataset\": \"Test\", \"Sum\": 367}]}}, {\"mode\": \"vega-lite\"});\n",
       "</script>"
      ],
      "text/plain": [
       "alt.Chart(...)"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "chart"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\t>> F1-score for training set...\n",
      "\t\t--> Average sample f1-score: 0.6366\n",
      "\t\t--> Micro f1-score: 0.6569\n",
      "\t\t--> Macro f1-score: 0.3870\n",
      "\t>> F1-score for test set...\n",
      "\t\t--> Average sample f1-score: 0.6161\n",
      "\t\t--> Micro f1-score: 0.6356\n",
      "\t\t--> Macro f1-score: 0.3630\n"
     ]
    }
   ],
   "source": [
    "# train\n",
    "model.fit(X[training_idx].toarray(), y[training_idx].toarray())\n",
    "# evaluate\n",
    "y_pred = model.predict(X[training_idx].toarray())\n",
    "score(y_true=y[training_idx], y_pred=y_pred, tag=\"training set\")\n",
    "\n",
    "y_pred = model.predict(X[test_idx].toarray())\n",
    "score(y_true=y[test_idx], y_pred=y_pred, tag=\"test set\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The results show better performance than the iterative approach while having more labelsets in the test data."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Clustering based strategy\n",
    "\n",
    "This approach is inspired by the [Partial Least Squares (PLS)](https://www.sciencedirect.com/science/article/abs/pii/S0169743901001551) to find the directions (eigenvectors) of maximum cross-covariance between the explanatory and the response variables by [Singular Value Decomposition (SVD)](https://epubs.siam.org/doi/abs/10.1137/s0895479896305696). Then, the left eigenvector is used to project explanatory variables onto it to extract a low-dimensional expression matrix, which is subsequently fed into the [K-means clustering algorithm](https://web.cse.msu.edu/~cse802/notes/ConstrainedKmeans.pdf). The clusters are used to perform the remapping of response variables to their nearest centroids. Finally, any of the following algorithms: Naive, Iterative, or Stratifying XML, can be applied to perform splitting. This approach aims to improve partitioning data while preserving correlations between response and attribute variables during the partitioning process."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "## Configuration parameters to stratifying a multi-label dataset splitting\n",
      "   based on clustering the covariance of X and y using PLSSVD:\n",
      "\t\t1. Number of clusters to form: 5\n",
      "\t\t2. A hyper-parameter: 0.1\n",
      "\t\t3. A hyper-parameter: 0.1\n",
      "\t\t4. A hyper-parameter: 0.1\n",
      "\t\t5. Shuffle the dataset? True\n",
      "\t\t6. Split size: 0.8\n",
      "\t\t7. Number of examples to use in each iteration: 500\n",
      "\t\t8. Number of loops over training set: 50\n",
      "\t\t9. Learning rate: 0.0001\n",
      "\t\t10. Number of parallel workers: 2\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\t>> Computing the covariance of X and y using PLSSVD: 100.00%...\n",
      "\t>> Projecting examples onto the obtained low dimensional U orthonormal basis...\n",
      "\t>> Clustering the resulted low dimensional examples...\n",
      "\t>> Perform splitting (extreme)...\n",
      "\t\t--> Starting score: -21\n",
      "\t\t--> Splitting progress: 100.00%; score: -5.23\n"
     ]
    }
   ],
   "source": [
    "st = ClusterStratification(num_clusters=num_clusters, swap_probability=0.1, threshold_proportion=0.1,\n",
    "                           decay=0.1, shuffle=True, split_size=split_size, batch_size=batch_size,\n",
    "                           num_epochs=num_epochs, lr=0.0001, num_jobs=num_jobs)\n",
    "training_idx, test_idx = st.fit(X=X, y=y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's train the chain classifier using the resulted training and test sets."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Properties for yeast</th>\n",
       "      <th>Complete set</th>\n",
       "      <th>Training set</th>\n",
       "      <th>Test set</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Number of examples</td>\n",
       "      <td>2417</td>\n",
       "      <td>1924</td>\n",
       "      <td>493</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Number of labels</td>\n",
       "      <td>10241</td>\n",
       "      <td>8095</td>\n",
       "      <td>2146</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Label cardinality</td>\n",
       "      <td>4.237071</td>\n",
       "      <td>4.20738</td>\n",
       "      <td>4.352941</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Label density</td>\n",
       "      <td>0.302648</td>\n",
       "      <td>0.300527</td>\n",
       "      <td>0.310924</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Distinct labels</td>\n",
       "      <td>14</td>\n",
       "      <td>14</td>\n",
       "      <td>14</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>Distinct label sets</td>\n",
       "      <td>198</td>\n",
       "      <td>181</td>\n",
       "      <td>105</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>Frequency of distinct label sets</td>\n",
       "      <td>0.08192</td>\n",
       "      <td>0.08192</td>\n",
       "      <td>0.08192</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>Mean imbalance ratio intra-class for all labels</td>\n",
       "      <td>9.578575</td>\n",
       "      <td>10.022388</td>\n",
       "      <td>8.368533</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>Mean imbalance ratio inter-class for all labels</td>\n",
       "      <td>7.196811</td>\n",
       "      <td>7.45949</td>\n",
       "      <td>6.51829</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>Mean imbalance ratio labelsets for all labels</td>\n",
       "      <td>0.468229</td>\n",
       "      <td>0.550249</td>\n",
       "      <td>2.050292</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>Labels having less than or equal to 5 examples</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>Labels having more than 6 examples</td>\n",
       "      <td>14</td>\n",
       "      <td>14</td>\n",
       "      <td>14</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>KL difference between complete and data partition</td>\n",
       "      <td>0</td>\n",
       "      <td>0.000084</td>\n",
       "      <td>0.001191</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                 Properties for yeast Complete set  \\\n",
       "0                                  Number of examples         2417   \n",
       "1                                    Number of labels        10241   \n",
       "2                                   Label cardinality     4.237071   \n",
       "3                                       Label density     0.302648   \n",
       "4                                     Distinct labels           14   \n",
       "5                                 Distinct label sets          198   \n",
       "6                    Frequency of distinct label sets      0.08192   \n",
       "7     Mean imbalance ratio intra-class for all labels     9.578575   \n",
       "8     Mean imbalance ratio inter-class for all labels     7.196811   \n",
       "9       Mean imbalance ratio labelsets for all labels     0.468229   \n",
       "10     Labels having less than or equal to 5 examples            0   \n",
       "11                 Labels having more than 6 examples           14   \n",
       "12  KL difference between complete and data partition            0   \n",
       "\n",
       "   Training set  Test set  \n",
       "0          1924       493  \n",
       "1          8095      2146  \n",
       "2       4.20738  4.352941  \n",
       "3      0.300527  0.310924  \n",
       "4            14        14  \n",
       "5           181       105  \n",
       "6       0.08192   0.08192  \n",
       "7     10.022388  8.368533  \n",
       "8       7.45949   6.51829  \n",
       "9      0.550249  2.050292  \n",
       "10            0         0  \n",
       "11           14        14  \n",
       "12     0.000084  0.001191  "
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model_name = \"plssvd2split\"\n",
    "df, chart = data_properties(y=y, selected_examples=[training_idx, test_idx], \n",
    "                            num_tails=5, dataset_name=dsname, model_name=model_name, \n",
    "                            rspath=RESULT_PATH, display_dataframe=True, \n",
    "                            display_figure=True)\n",
    "df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "<div id=\"altair-viz-090dbf68326f4a97ad6804d5f1a83f6e\"></div>\n",
       "<script type=\"text/javascript\">\n",
       "  (function(spec, embedOpt){\n",
       "    let outputDiv = document.currentScript.previousElementSibling;\n",
       "    if (outputDiv.id !== \"altair-viz-090dbf68326f4a97ad6804d5f1a83f6e\") {\n",
       "      outputDiv = document.getElementById(\"altair-viz-090dbf68326f4a97ad6804d5f1a83f6e\");\n",
       "    }\n",
       "    const paths = {\n",
       "      \"vega\": \"https://cdn.jsdelivr.net/npm//vega@5?noext\",\n",
       "      \"vega-lib\": \"https://cdn.jsdelivr.net/npm//vega-lib?noext\",\n",
       "      \"vega-lite\": \"https://cdn.jsdelivr.net/npm//vega-lite@4.8.1?noext\",\n",
       "      \"vega-embed\": \"https://cdn.jsdelivr.net/npm//vega-embed@6?noext\",\n",
       "    };\n",
       "\n",
       "    function loadScript(lib) {\n",
       "      return new Promise(function(resolve, reject) {\n",
       "        var s = document.createElement('script');\n",
       "        s.src = paths[lib];\n",
       "        s.async = true;\n",
       "        s.onload = () => resolve(paths[lib]);\n",
       "        s.onerror = () => reject(`Error loading script: ${paths[lib]}`);\n",
       "        document.getElementsByTagName(\"head\")[0].appendChild(s);\n",
       "      });\n",
       "    }\n",
       "\n",
       "    function showError(err) {\n",
       "      outputDiv.innerHTML = `<div class=\"error\" style=\"color:red;\">${err}</div>`;\n",
       "      throw err;\n",
       "    }\n",
       "\n",
       "    function displayChart(vegaEmbed) {\n",
       "      vegaEmbed(outputDiv, spec, embedOpt)\n",
       "        .catch(err => showError(`Javascript Error: ${err.message}<br>This usually means there's a typo in your chart specification. See the javascript console for the full traceback.`));\n",
       "    }\n",
       "\n",
       "    if(typeof define === \"function\" && define.amd) {\n",
       "      requirejs.config({paths});\n",
       "      require([\"vega-embed\"], displayChart, err => showError(`Error loading script: ${err.message}`));\n",
       "    } else if (typeof vegaEmbed === \"function\") {\n",
       "      displayChart(vegaEmbed);\n",
       "    } else {\n",
       "      loadScript(\"vega\")\n",
       "        .then(() => loadScript(\"vega-lite\"))\n",
       "        .then(() => loadScript(\"vega-embed\"))\n",
       "        .catch(showError)\n",
       "        .then(() => displayChart(vegaEmbed));\n",
       "    }\n",
       "  })({\"data\": {\"name\": \"data-02b1c59029dbbe23f392357820fe527a\"}, \"mark\": {\"type\": \"bar\", \"color\": \"grey\"}, \"config\": {\"axis\": {\"labelFontSize\": 12, \"labelLimit\": 500, \"labelPadding\": 5, \"titleFontSize\": 20}, \"axisY\": {\"grid\": false}, \"header\": {\"labelFontSize\": 15, \"titleFontSize\": 20}, \"legend\": {\"cornerRadius\": 10, \"fillColor\": \"white\", \"padding\": 10, \"strokeColor\": \"gray\"}}, \"encoding\": {\"color\": {\"type\": \"nominal\", \"field\": \"Dataset\", \"scale\": {\"range\": [\"red\", \"black\", \"blue\"]}}, \"x\": {\"type\": \"ordinal\", \"field\": \"Label\", \"sort\": \"ascending\", \"title\": \"Label ID\"}, \"y\": {\"type\": \"quantitative\", \"field\": \"Sum\", \"stack\": null, \"title\": \"Number of Examples\"}}, \"height\": 350, \"resolve\": {\"scale\": {\"x\": \"independent\"}}, \"title\": {\"text\": \"Number of examples for each label given yeast data.\", \"subtitle\": [\"The horizontal axis indicates the indices of labels while the vertical axis represents the number of associated examples\"]}, \"width\": 600, \"$schema\": \"https://vega.github.io/schema/vega-lite/v4.8.1.json\", \"datasets\": {\"data-02b1c59029dbbe23f392357820fe527a\": [{\"Label\": 1, \"Dataset\": \"Complete\", \"Sum\": 34}, {\"Label\": 2, \"Dataset\": \"Complete\", \"Sum\": 178}, {\"Label\": 3, \"Dataset\": \"Complete\", \"Sum\": 253}, {\"Label\": 4, \"Dataset\": \"Complete\", \"Sum\": 289}, {\"Label\": 5, \"Dataset\": \"Complete\", \"Sum\": 428}, {\"Label\": 6, \"Dataset\": \"Complete\", \"Sum\": 480}, {\"Label\": 7, \"Dataset\": \"Complete\", \"Sum\": 597}, {\"Label\": 8, \"Dataset\": \"Complete\", \"Sum\": 722}, {\"Label\": 9, \"Dataset\": \"Complete\", \"Sum\": 762}, {\"Label\": 10, \"Dataset\": \"Complete\", \"Sum\": 862}, {\"Label\": 11, \"Dataset\": \"Complete\", \"Sum\": 983}, {\"Label\": 12, \"Dataset\": \"Complete\", \"Sum\": 1038}, {\"Label\": 13, \"Dataset\": \"Complete\", \"Sum\": 1799}, {\"Label\": 14, \"Dataset\": \"Complete\", \"Sum\": 1816}, {\"Label\": 1, \"Dataset\": \"Train\", \"Sum\": 25}, {\"Label\": 2, \"Dataset\": \"Train\", \"Sum\": 146}, {\"Label\": 3, \"Dataset\": \"Train\", \"Sum\": 195}, {\"Label\": 4, \"Dataset\": \"Train\", \"Sum\": 222}, {\"Label\": 5, \"Dataset\": \"Train\", \"Sum\": 337}, {\"Label\": 6, \"Dataset\": \"Train\", \"Sum\": 387}, {\"Label\": 7, \"Dataset\": \"Train\", \"Sum\": 465}, {\"Label\": 8, \"Dataset\": \"Train\", \"Sum\": 578}, {\"Label\": 9, \"Dataset\": \"Train\", \"Sum\": 610}, {\"Label\": 10, \"Dataset\": \"Train\", \"Sum\": 684}, {\"Label\": 11, \"Dataset\": \"Train\", \"Sum\": 767}, {\"Label\": 12, \"Dataset\": \"Train\", \"Sum\": 830}, {\"Label\": 13, \"Dataset\": \"Train\", \"Sum\": 1417}, {\"Label\": 14, \"Dataset\": \"Train\", \"Sum\": 1432}, {\"Label\": 1, \"Dataset\": \"Test\", \"Sum\": 9}, {\"Label\": 2, \"Dataset\": \"Test\", \"Sum\": 32}, {\"Label\": 3, \"Dataset\": \"Test\", \"Sum\": 58}, {\"Label\": 4, \"Dataset\": \"Test\", \"Sum\": 67}, {\"Label\": 5, \"Dataset\": \"Test\", \"Sum\": 91}, {\"Label\": 6, \"Dataset\": \"Test\", \"Sum\": 93}, {\"Label\": 7, \"Dataset\": \"Test\", \"Sum\": 132}, {\"Label\": 8, \"Dataset\": \"Test\", \"Sum\": 144}, {\"Label\": 9, \"Dataset\": \"Test\", \"Sum\": 152}, {\"Label\": 10, \"Dataset\": \"Test\", \"Sum\": 178}, {\"Label\": 11, \"Dataset\": \"Test\", \"Sum\": 216}, {\"Label\": 12, \"Dataset\": \"Test\", \"Sum\": 208}, {\"Label\": 13, \"Dataset\": \"Test\", \"Sum\": 382}, {\"Label\": 14, \"Dataset\": \"Test\", \"Sum\": 384}]}}, {\"mode\": \"vega-lite\"});\n",
       "</script>"
      ],
      "text/plain": [
       "alt.Chart(...)"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "chart"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\t>> F1-score for training set...\n",
      "\t\t--> Average sample f1-score: 0.6330\n",
      "\t\t--> Micro f1-score: 0.6542\n",
      "\t\t--> Macro f1-score: 0.3901\n",
      "\t>> F1-score for test set...\n",
      "\t\t--> Average sample f1-score: 0.6307\n",
      "\t\t--> Micro f1-score: 0.6470\n",
      "\t\t--> Macro f1-score: 0.3740\n"
     ]
    }
   ],
   "source": [
    "# train\n",
    "model.fit(X[training_idx].toarray(), y[training_idx].toarray())\n",
    "# evaluate\n",
    "y_pred = model.predict(X[training_idx].toarray())\n",
    "score(y_true=y[training_idx], y_pred=y_pred, tag=\"training set\")\n",
    "\n",
    "y_pred = model.predict(X[test_idx].toarray())\n",
    "score(y_true=y[test_idx], y_pred=y_pred, tag=\"test set\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The results are seen to be improved in relation to Stratifying XML. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Clustering eigenvalues based approach\n",
    "\n",
    "Similar to the above method, this algorithm builds an adjacency matrix from the response variables and then, decomposes this matrix to extract eigenvectors, which is subsequently fed into the [K-means clustering algorithm](https://web.cse.msu.edu/~cse802/notes/ConstrainedKmeans.pdf). The clusters are then used to perform the remapping of response variables to their nearest centroids. As with the previous approach, any of the following algorithms: Naive, Iterative, or Stratifying XML, can be applied to perform splitting."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "## Configuration parameters to stratifying a multi-label dataset splitting\n",
      "   based on clustering eigen values of the label adjacency matrix:\n",
      "\t\t1. Subsampling input size: 10000\n",
      "\t\t2. Number of communities: 5\n",
      "\t\t3. Constant that scales the amount of laplacian norm regularization: 2\n",
      "\t\t4. A hyper-parameter: 0.1\n",
      "\t\t5. A hyper-parameter: 0.1\n",
      "\t\t6. A hyper-parameter: 0.1\n",
      "\t\t7. Shuffle the dataset? True\n",
      "\t\t8. Split size: 0.8\n",
      "\t\t9. Number of examples to use in each iteration: 500\n",
      "\t\t10. Number of loops over training set: 50\n",
      "\t\t11. Number of parallel workers: 2\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\t>> Extracting clusters...\n",
      "\t>> Perform splitting (extreme)...\n",
      "\t\t--> Starting score: 359\n",
      "\t\t--> Splitting progress: 100.00%; score: 136.40\n"
     ]
    }
   ],
   "source": [
    "st = ClusteringEigenStratification(num_subsamples=10000, num_clusters=num_clusters, sigma=2, \n",
    "                                   swap_probability=0.1, threshold_proportion=0.1, \n",
    "                                   decay=0.1, shuffle=True, split_size=split_size,\n",
    "                                   batch_size=batch_size, num_epochs=num_epochs, \n",
    "                                   num_jobs=num_jobs)\n",
    "training_idx, test_idx = st.fit(X=X, y=y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Properties for yeast</th>\n",
       "      <th>Complete set</th>\n",
       "      <th>Training set</th>\n",
       "      <th>Test set</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Number of examples</td>\n",
       "      <td>2417</td>\n",
       "      <td>1954</td>\n",
       "      <td>463</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Number of labels</td>\n",
       "      <td>10241</td>\n",
       "      <td>8288</td>\n",
       "      <td>1953</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Label cardinality</td>\n",
       "      <td>4.237071</td>\n",
       "      <td>4.241556</td>\n",
       "      <td>4.218143</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Label density</td>\n",
       "      <td>0.302648</td>\n",
       "      <td>0.302968</td>\n",
       "      <td>0.301296</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Distinct labels</td>\n",
       "      <td>14</td>\n",
       "      <td>14</td>\n",
       "      <td>14</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>Distinct label sets</td>\n",
       "      <td>198</td>\n",
       "      <td>179</td>\n",
       "      <td>109</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>Frequency of distinct label sets</td>\n",
       "      <td>0.08192</td>\n",
       "      <td>0.08192</td>\n",
       "      <td>0.08192</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>Mean imbalance ratio intra-class for all labels</td>\n",
       "      <td>9.578575</td>\n",
       "      <td>9.512279</td>\n",
       "      <td>10.007766</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>Mean imbalance ratio inter-class for all labels</td>\n",
       "      <td>7.196811</td>\n",
       "      <td>7.160984</td>\n",
       "      <td>7.457191</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>Mean imbalance ratio labelsets for all labels</td>\n",
       "      <td>0.468229</td>\n",
       "      <td>0.377509</td>\n",
       "      <td>1.042197</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>Labels having less than or equal to 5 examples</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>Labels having more than 6 examples</td>\n",
       "      <td>14</td>\n",
       "      <td>14</td>\n",
       "      <td>14</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>KL difference between complete and data partition</td>\n",
       "      <td>0</td>\n",
       "      <td>0.00012</td>\n",
       "      <td>0.002088</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                 Properties for yeast Complete set  \\\n",
       "0                                  Number of examples         2417   \n",
       "1                                    Number of labels        10241   \n",
       "2                                   Label cardinality     4.237071   \n",
       "3                                       Label density     0.302648   \n",
       "4                                     Distinct labels           14   \n",
       "5                                 Distinct label sets          198   \n",
       "6                    Frequency of distinct label sets      0.08192   \n",
       "7     Mean imbalance ratio intra-class for all labels     9.578575   \n",
       "8     Mean imbalance ratio inter-class for all labels     7.196811   \n",
       "9       Mean imbalance ratio labelsets for all labels     0.468229   \n",
       "10     Labels having less than or equal to 5 examples            0   \n",
       "11                 Labels having more than 6 examples           14   \n",
       "12  KL difference between complete and data partition            0   \n",
       "\n",
       "   Training set   Test set  \n",
       "0          1954        463  \n",
       "1          8288       1953  \n",
       "2      4.241556   4.218143  \n",
       "3      0.302968   0.301296  \n",
       "4            14         14  \n",
       "5           179        109  \n",
       "6       0.08192    0.08192  \n",
       "7      9.512279  10.007766  \n",
       "8      7.160984   7.457191  \n",
       "9      0.377509   1.042197  \n",
       "10            0          0  \n",
       "11           14         14  \n",
       "12      0.00012   0.002088  "
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model_name = \"eigencluster2split\"\n",
    "df, chart = data_properties(y=y, selected_examples=[training_idx, test_idx], \n",
    "                            num_tails=5, dataset_name=dsname, model_name=model_name, \n",
    "                            rspath=RESULT_PATH, display_dataframe=True, \n",
    "                            display_figure=True)\n",
    "df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "<div id=\"altair-viz-b9b0dc2b47334219a75cc89f5325c681\"></div>\n",
       "<script type=\"text/javascript\">\n",
       "  (function(spec, embedOpt){\n",
       "    let outputDiv = document.currentScript.previousElementSibling;\n",
       "    if (outputDiv.id !== \"altair-viz-b9b0dc2b47334219a75cc89f5325c681\") {\n",
       "      outputDiv = document.getElementById(\"altair-viz-b9b0dc2b47334219a75cc89f5325c681\");\n",
       "    }\n",
       "    const paths = {\n",
       "      \"vega\": \"https://cdn.jsdelivr.net/npm//vega@5?noext\",\n",
       "      \"vega-lib\": \"https://cdn.jsdelivr.net/npm//vega-lib?noext\",\n",
       "      \"vega-lite\": \"https://cdn.jsdelivr.net/npm//vega-lite@4.8.1?noext\",\n",
       "      \"vega-embed\": \"https://cdn.jsdelivr.net/npm//vega-embed@6?noext\",\n",
       "    };\n",
       "\n",
       "    function loadScript(lib) {\n",
       "      return new Promise(function(resolve, reject) {\n",
       "        var s = document.createElement('script');\n",
       "        s.src = paths[lib];\n",
       "        s.async = true;\n",
       "        s.onload = () => resolve(paths[lib]);\n",
       "        s.onerror = () => reject(`Error loading script: ${paths[lib]}`);\n",
       "        document.getElementsByTagName(\"head\")[0].appendChild(s);\n",
       "      });\n",
       "    }\n",
       "\n",
       "    function showError(err) {\n",
       "      outputDiv.innerHTML = `<div class=\"error\" style=\"color:red;\">${err}</div>`;\n",
       "      throw err;\n",
       "    }\n",
       "\n",
       "    function displayChart(vegaEmbed) {\n",
       "      vegaEmbed(outputDiv, spec, embedOpt)\n",
       "        .catch(err => showError(`Javascript Error: ${err.message}<br>This usually means there's a typo in your chart specification. See the javascript console for the full traceback.`));\n",
       "    }\n",
       "\n",
       "    if(typeof define === \"function\" && define.amd) {\n",
       "      requirejs.config({paths});\n",
       "      require([\"vega-embed\"], displayChart, err => showError(`Error loading script: ${err.message}`));\n",
       "    } else if (typeof vegaEmbed === \"function\") {\n",
       "      displayChart(vegaEmbed);\n",
       "    } else {\n",
       "      loadScript(\"vega\")\n",
       "        .then(() => loadScript(\"vega-lite\"))\n",
       "        .then(() => loadScript(\"vega-embed\"))\n",
       "        .catch(showError)\n",
       "        .then(() => displayChart(vegaEmbed));\n",
       "    }\n",
       "  })({\"data\": {\"name\": \"data-4226a1881d9009b3672c50191b7ac720\"}, \"mark\": {\"type\": \"bar\", \"color\": \"grey\"}, \"config\": {\"axis\": {\"labelFontSize\": 12, \"labelLimit\": 500, \"labelPadding\": 5, \"titleFontSize\": 20}, \"axisY\": {\"grid\": false}, \"header\": {\"labelFontSize\": 15, \"titleFontSize\": 20}, \"legend\": {\"cornerRadius\": 10, \"fillColor\": \"white\", \"padding\": 10, \"strokeColor\": \"gray\"}}, \"encoding\": {\"color\": {\"type\": \"nominal\", \"field\": \"Dataset\", \"scale\": {\"range\": [\"red\", \"black\", \"blue\"]}}, \"x\": {\"type\": \"ordinal\", \"field\": \"Label\", \"sort\": \"ascending\", \"title\": \"Label ID\"}, \"y\": {\"type\": \"quantitative\", \"field\": \"Sum\", \"stack\": null, \"title\": \"Number of Examples\"}}, \"height\": 350, \"resolve\": {\"scale\": {\"x\": \"independent\"}}, \"title\": {\"text\": \"Number of examples for each label given yeast data.\", \"subtitle\": [\"The horizontal axis indicates the indices of labels while the vertical axis represents the number of associated examples\"]}, \"width\": 600, \"$schema\": \"https://vega.github.io/schema/vega-lite/v4.8.1.json\", \"datasets\": {\"data-4226a1881d9009b3672c50191b7ac720\": [{\"Label\": 1, \"Dataset\": \"Complete\", \"Sum\": 34}, {\"Label\": 2, \"Dataset\": \"Complete\", \"Sum\": 178}, {\"Label\": 3, \"Dataset\": \"Complete\", \"Sum\": 253}, {\"Label\": 4, \"Dataset\": \"Complete\", \"Sum\": 289}, {\"Label\": 5, \"Dataset\": \"Complete\", \"Sum\": 428}, {\"Label\": 6, \"Dataset\": \"Complete\", \"Sum\": 480}, {\"Label\": 7, \"Dataset\": \"Complete\", \"Sum\": 597}, {\"Label\": 8, \"Dataset\": \"Complete\", \"Sum\": 722}, {\"Label\": 9, \"Dataset\": \"Complete\", \"Sum\": 762}, {\"Label\": 10, \"Dataset\": \"Complete\", \"Sum\": 862}, {\"Label\": 11, \"Dataset\": \"Complete\", \"Sum\": 983}, {\"Label\": 12, \"Dataset\": \"Complete\", \"Sum\": 1038}, {\"Label\": 13, \"Dataset\": \"Complete\", \"Sum\": 1799}, {\"Label\": 14, \"Dataset\": \"Complete\", \"Sum\": 1816}, {\"Label\": 1, \"Dataset\": \"Train\", \"Sum\": 28}, {\"Label\": 2, \"Dataset\": \"Train\", \"Sum\": 135}, {\"Label\": 3, \"Dataset\": \"Train\", \"Sum\": 211}, {\"Label\": 4, \"Dataset\": \"Train\", \"Sum\": 241}, {\"Label\": 5, \"Dataset\": \"Train\", \"Sum\": 347}, {\"Label\": 6, \"Dataset\": \"Train\", \"Sum\": 389}, {\"Label\": 7, \"Dataset\": \"Train\", \"Sum\": 478}, {\"Label\": 8, \"Dataset\": \"Train\", \"Sum\": 591}, {\"Label\": 9, \"Dataset\": \"Train\", \"Sum\": 596}, {\"Label\": 10, \"Dataset\": \"Train\", \"Sum\": 702}, {\"Label\": 11, \"Dataset\": \"Train\", \"Sum\": 805}, {\"Label\": 12, \"Dataset\": \"Train\", \"Sum\": 836}, {\"Label\": 13, \"Dataset\": \"Train\", \"Sum\": 1458}, {\"Label\": 14, \"Dataset\": \"Train\", \"Sum\": 1471}, {\"Label\": 1, \"Dataset\": \"Test\", \"Sum\": 6}, {\"Label\": 2, \"Dataset\": \"Test\", \"Sum\": 43}, {\"Label\": 3, \"Dataset\": \"Test\", \"Sum\": 42}, {\"Label\": 4, \"Dataset\": \"Test\", \"Sum\": 48}, {\"Label\": 5, \"Dataset\": \"Test\", \"Sum\": 81}, {\"Label\": 6, \"Dataset\": \"Test\", \"Sum\": 91}, {\"Label\": 7, \"Dataset\": \"Test\", \"Sum\": 119}, {\"Label\": 8, \"Dataset\": \"Test\", \"Sum\": 131}, {\"Label\": 9, \"Dataset\": \"Test\", \"Sum\": 166}, {\"Label\": 10, \"Dataset\": \"Test\", \"Sum\": 160}, {\"Label\": 11, \"Dataset\": \"Test\", \"Sum\": 178}, {\"Label\": 12, \"Dataset\": \"Test\", \"Sum\": 202}, {\"Label\": 13, \"Dataset\": \"Test\", \"Sum\": 341}, {\"Label\": 14, \"Dataset\": \"Test\", \"Sum\": 345}]}}, {\"mode\": \"vega-lite\"});\n",
       "</script>"
      ],
      "text/plain": [
       "alt.Chart(...)"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "chart"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As usual, we train the chain classifier using the resulted training and test sets."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\t>> F1-score for training set...\n",
      "\t\t--> Average sample f1-score: 0.6392\n",
      "\t\t--> Micro f1-score: 0.6600\n",
      "\t\t--> Macro f1-score: 0.4021\n",
      "\t>> F1-score for test set...\n",
      "\t\t--> Average sample f1-score: 0.6212\n",
      "\t\t--> Micro f1-score: 0.6398\n",
      "\t\t--> Macro f1-score: 0.3764\n"
     ]
    }
   ],
   "source": [
    "# train\n",
    "model.fit(X[training_idx].toarray(), y[training_idx].toarray())\n",
    "# evaluate\n",
    "y_pred = model.predict(X[training_idx].toarray())\n",
    "score(y_true=y[training_idx], y_pred=y_pred, tag=\"training set\")\n",
    "\n",
    "y_pred = model.predict(X[test_idx].toarray())\n",
    "score(y_true=y[test_idx], y_pred=y_pred, tag=\"test set\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The results are seen to be improved in relation to Stratifying XML."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "hvX8sKsfMaio"
   },
   "source": [
    "### Community based splitting strategy\n",
    "\n",
    "Inspired by [community detection](https://www.pnas.org/content/99/12/7821.short), this approach identifies communities over a graph that is constructed using response variables.  Then communities are used to remap response variables to their communities. Then, any of the following algorithms: Naive, Iterative, or Stratifying XML, can be applied to perform splitting."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "## Configuration parameters to stratifying a multi-label dataset splitting\n",
      "   based on community detection approach:\n",
      "\t\t1. Subsampling input size: 20000\n",
      "\t\t2. Number of communities: 5\n",
      "\t\t3. Constant that scales the amount of laplacian norm regularization: 2\n",
      "\t\t4. A hyper-parameter: 0.1\n",
      "\t\t5. A hyper-parameter: 0.1\n",
      "\t\t6. A hyper-parameter: 0.1\n",
      "\t\t7. Shuffle the dataset? True\n",
      "\t\t8. Split size: 0.8\n",
      "\t\t9. Number of examples to use in each iteration: 500\n",
      "\t\t10. Number of loops over training set: 50\n",
      "\t\t11. Number of parallel workers: 2\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\t>> Building Graph...\n",
      "\t>> Perform splitting (extreme)...\n",
      "\t\t--> Starting score: 72\n",
      "\t\t--> Splitting progress: 100.00%; score: 15.08\n"
     ]
    }
   ],
   "source": [
    "st = CommunityStratification(num_subsamples=20000, num_communities=num_clusters, sigma=2, \n",
    "                             swap_probability=0.1, threshold_proportion=0.1, decay=0.1, \n",
    "                             shuffle=True, split_size=split_size, batch_size=batch_size, \n",
    "                             num_epochs=num_epochs, num_jobs=num_jobs)\n",
    "training_idx, test_idx = st.fit(X=X, y=y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Properties for yeast</th>\n",
       "      <th>Complete set</th>\n",
       "      <th>Training set</th>\n",
       "      <th>Test set</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Number of examples</td>\n",
       "      <td>2417</td>\n",
       "      <td>1937</td>\n",
       "      <td>480</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Number of labels</td>\n",
       "      <td>10241</td>\n",
       "      <td>8250</td>\n",
       "      <td>1991</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Label cardinality</td>\n",
       "      <td>4.237071</td>\n",
       "      <td>4.259164</td>\n",
       "      <td>4.147917</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Label density</td>\n",
       "      <td>0.302648</td>\n",
       "      <td>0.304226</td>\n",
       "      <td>0.29628</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Distinct labels</td>\n",
       "      <td>14</td>\n",
       "      <td>14</td>\n",
       "      <td>14</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>Distinct label sets</td>\n",
       "      <td>198</td>\n",
       "      <td>187</td>\n",
       "      <td>92</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>Frequency of distinct label sets</td>\n",
       "      <td>0.08192</td>\n",
       "      <td>0.08192</td>\n",
       "      <td>0.08192</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>Mean imbalance ratio intra-class for all labels</td>\n",
       "      <td>9.578575</td>\n",
       "      <td>9.092897</td>\n",
       "      <td>12.569147</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>Mean imbalance ratio inter-class for all labels</td>\n",
       "      <td>7.196811</td>\n",
       "      <td>6.802069</td>\n",
       "      <td>9.61016</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>Mean imbalance ratio labelsets for all labels</td>\n",
       "      <td>0.468229</td>\n",
       "      <td>0.883835</td>\n",
       "      <td>0.677507</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>Labels having less than or equal to 5 examples</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>Labels having more than 6 examples</td>\n",
       "      <td>14</td>\n",
       "      <td>14</td>\n",
       "      <td>13</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>KL difference between complete and data partition</td>\n",
       "      <td>0</td>\n",
       "      <td>0.000362</td>\n",
       "      <td>0.006983</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                 Properties for yeast Complete set  \\\n",
       "0                                  Number of examples         2417   \n",
       "1                                    Number of labels        10241   \n",
       "2                                   Label cardinality     4.237071   \n",
       "3                                       Label density     0.302648   \n",
       "4                                     Distinct labels           14   \n",
       "5                                 Distinct label sets          198   \n",
       "6                    Frequency of distinct label sets      0.08192   \n",
       "7     Mean imbalance ratio intra-class for all labels     9.578575   \n",
       "8     Mean imbalance ratio inter-class for all labels     7.196811   \n",
       "9       Mean imbalance ratio labelsets for all labels     0.468229   \n",
       "10     Labels having less than or equal to 5 examples            0   \n",
       "11                 Labels having more than 6 examples           14   \n",
       "12  KL difference between complete and data partition            0   \n",
       "\n",
       "   Training set   Test set  \n",
       "0          1937        480  \n",
       "1          8250       1991  \n",
       "2      4.259164   4.147917  \n",
       "3      0.304226    0.29628  \n",
       "4            14         14  \n",
       "5           187         92  \n",
       "6       0.08192    0.08192  \n",
       "7      9.092897  12.569147  \n",
       "8      6.802069    9.61016  \n",
       "9      0.883835   0.677507  \n",
       "10            0          1  \n",
       "11           14         13  \n",
       "12     0.000362   0.006983  "
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model_name = \"comm2split\"\n",
    "df, chart = data_properties(y=y, selected_examples=[training_idx, test_idx], \n",
    "                            num_tails=5, dataset_name=dsname, model_name=model_name, \n",
    "                            rspath=RESULT_PATH, display_dataframe=True, \n",
    "                            display_figure=True)\n",
    "df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "<div id=\"altair-viz-ddb4461f960f4587af4957800b7c261c\"></div>\n",
       "<script type=\"text/javascript\">\n",
       "  (function(spec, embedOpt){\n",
       "    let outputDiv = document.currentScript.previousElementSibling;\n",
       "    if (outputDiv.id !== \"altair-viz-ddb4461f960f4587af4957800b7c261c\") {\n",
       "      outputDiv = document.getElementById(\"altair-viz-ddb4461f960f4587af4957800b7c261c\");\n",
       "    }\n",
       "    const paths = {\n",
       "      \"vega\": \"https://cdn.jsdelivr.net/npm//vega@5?noext\",\n",
       "      \"vega-lib\": \"https://cdn.jsdelivr.net/npm//vega-lib?noext\",\n",
       "      \"vega-lite\": \"https://cdn.jsdelivr.net/npm//vega-lite@4.8.1?noext\",\n",
       "      \"vega-embed\": \"https://cdn.jsdelivr.net/npm//vega-embed@6?noext\",\n",
       "    };\n",
       "\n",
       "    function loadScript(lib) {\n",
       "      return new Promise(function(resolve, reject) {\n",
       "        var s = document.createElement('script');\n",
       "        s.src = paths[lib];\n",
       "        s.async = true;\n",
       "        s.onload = () => resolve(paths[lib]);\n",
       "        s.onerror = () => reject(`Error loading script: ${paths[lib]}`);\n",
       "        document.getElementsByTagName(\"head\")[0].appendChild(s);\n",
       "      });\n",
       "    }\n",
       "\n",
       "    function showError(err) {\n",
       "      outputDiv.innerHTML = `<div class=\"error\" style=\"color:red;\">${err}</div>`;\n",
       "      throw err;\n",
       "    }\n",
       "\n",
       "    function displayChart(vegaEmbed) {\n",
       "      vegaEmbed(outputDiv, spec, embedOpt)\n",
       "        .catch(err => showError(`Javascript Error: ${err.message}<br>This usually means there's a typo in your chart specification. See the javascript console for the full traceback.`));\n",
       "    }\n",
       "\n",
       "    if(typeof define === \"function\" && define.amd) {\n",
       "      requirejs.config({paths});\n",
       "      require([\"vega-embed\"], displayChart, err => showError(`Error loading script: ${err.message}`));\n",
       "    } else if (typeof vegaEmbed === \"function\") {\n",
       "      displayChart(vegaEmbed);\n",
       "    } else {\n",
       "      loadScript(\"vega\")\n",
       "        .then(() => loadScript(\"vega-lite\"))\n",
       "        .then(() => loadScript(\"vega-embed\"))\n",
       "        .catch(showError)\n",
       "        .then(() => displayChart(vegaEmbed));\n",
       "    }\n",
       "  })({\"data\": {\"name\": \"data-fc48e8234a6671fd3b36ccbd640e1d3f\"}, \"mark\": {\"type\": \"bar\", \"color\": \"grey\"}, \"config\": {\"axis\": {\"labelFontSize\": 12, \"labelLimit\": 500, \"labelPadding\": 5, \"titleFontSize\": 20}, \"axisY\": {\"grid\": false}, \"header\": {\"labelFontSize\": 15, \"titleFontSize\": 20}, \"legend\": {\"cornerRadius\": 10, \"fillColor\": \"white\", \"padding\": 10, \"strokeColor\": \"gray\"}}, \"encoding\": {\"color\": {\"type\": \"nominal\", \"field\": \"Dataset\", \"scale\": {\"range\": [\"red\", \"black\", \"blue\"]}}, \"x\": {\"type\": \"ordinal\", \"field\": \"Label\", \"sort\": \"ascending\", \"title\": \"Label ID\"}, \"y\": {\"type\": \"quantitative\", \"field\": \"Sum\", \"stack\": null, \"title\": \"Number of Examples\"}}, \"height\": 350, \"resolve\": {\"scale\": {\"x\": \"independent\"}}, \"title\": {\"text\": \"Number of examples for each label given yeast data.\", \"subtitle\": [\"The horizontal axis indicates the indices of labels while the vertical axis represents the number of associated examples\"]}, \"width\": 600, \"$schema\": \"https://vega.github.io/schema/vega-lite/v4.8.1.json\", \"datasets\": {\"data-fc48e8234a6671fd3b36ccbd640e1d3f\": [{\"Label\": 1, \"Dataset\": \"Complete\", \"Sum\": 34}, {\"Label\": 2, \"Dataset\": \"Complete\", \"Sum\": 178}, {\"Label\": 3, \"Dataset\": \"Complete\", \"Sum\": 253}, {\"Label\": 4, \"Dataset\": \"Complete\", \"Sum\": 289}, {\"Label\": 5, \"Dataset\": \"Complete\", \"Sum\": 428}, {\"Label\": 6, \"Dataset\": \"Complete\", \"Sum\": 480}, {\"Label\": 7, \"Dataset\": \"Complete\", \"Sum\": 597}, {\"Label\": 8, \"Dataset\": \"Complete\", \"Sum\": 722}, {\"Label\": 9, \"Dataset\": \"Complete\", \"Sum\": 762}, {\"Label\": 10, \"Dataset\": \"Complete\", \"Sum\": 862}, {\"Label\": 11, \"Dataset\": \"Complete\", \"Sum\": 983}, {\"Label\": 12, \"Dataset\": \"Complete\", \"Sum\": 1038}, {\"Label\": 13, \"Dataset\": \"Complete\", \"Sum\": 1799}, {\"Label\": 14, \"Dataset\": \"Complete\", \"Sum\": 1816}, {\"Label\": 1, \"Dataset\": \"Train\", \"Sum\": 29}, {\"Label\": 2, \"Dataset\": \"Train\", \"Sum\": 150}, {\"Label\": 3, \"Dataset\": \"Train\", \"Sum\": 224}, {\"Label\": 4, \"Dataset\": \"Train\", \"Sum\": 254}, {\"Label\": 5, \"Dataset\": \"Train\", \"Sum\": 352}, {\"Label\": 6, \"Dataset\": \"Train\", \"Sum\": 395}, {\"Label\": 7, \"Dataset\": \"Train\", \"Sum\": 483}, {\"Label\": 8, \"Dataset\": \"Train\", \"Sum\": 580}, {\"Label\": 9, \"Dataset\": \"Train\", \"Sum\": 589}, {\"Label\": 10, \"Dataset\": \"Train\", \"Sum\": 693}, {\"Label\": 11, \"Dataset\": \"Train\", \"Sum\": 791}, {\"Label\": 12, \"Dataset\": \"Train\", \"Sum\": 826}, {\"Label\": 13, \"Dataset\": \"Train\", \"Sum\": 1435}, {\"Label\": 14, \"Dataset\": \"Train\", \"Sum\": 1449}, {\"Label\": 1, \"Dataset\": \"Test\", \"Sum\": 5}, {\"Label\": 2, \"Dataset\": \"Test\", \"Sum\": 28}, {\"Label\": 3, \"Dataset\": \"Test\", \"Sum\": 29}, {\"Label\": 4, \"Dataset\": \"Test\", \"Sum\": 35}, {\"Label\": 5, \"Dataset\": \"Test\", \"Sum\": 76}, {\"Label\": 6, \"Dataset\": \"Test\", \"Sum\": 85}, {\"Label\": 7, \"Dataset\": \"Test\", \"Sum\": 114}, {\"Label\": 8, \"Dataset\": \"Test\", \"Sum\": 142}, {\"Label\": 9, \"Dataset\": \"Test\", \"Sum\": 173}, {\"Label\": 10, \"Dataset\": \"Test\", \"Sum\": 169}, {\"Label\": 11, \"Dataset\": \"Test\", \"Sum\": 192}, {\"Label\": 12, \"Dataset\": \"Test\", \"Sum\": 212}, {\"Label\": 13, \"Dataset\": \"Test\", \"Sum\": 364}, {\"Label\": 14, \"Dataset\": \"Test\", \"Sum\": 367}]}}, {\"mode\": \"vega-lite\"});\n",
       "</script>"
      ],
      "text/plain": [
       "alt.Chart(...)"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "chart"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we did before, let's train the chain classifier."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\t>> F1-score for training set...\n",
      "\t\t--> Average sample f1-score: 0.6372\n",
      "\t\t--> Micro f1-score: 0.6596\n",
      "\t\t--> Macro f1-score: 0.4107\n",
      "\t>> F1-score for test set...\n",
      "\t\t--> Average sample f1-score: 0.6162\n",
      "\t\t--> Micro f1-score: 0.6350\n",
      "\t\t--> Macro f1-score: 0.3542\n"
     ]
    }
   ],
   "source": [
    "# train\n",
    "model.fit(X[training_idx].toarray(), y[training_idx].toarray())\n",
    "# evaluate\n",
    "y_pred = model.predict(X[training_idx].toarray())\n",
    "score(y_true=y[training_idx], y_pred=y_pred, tag=\"training set\")\n",
    "\n",
    "y_pred = model.predict(X[test_idx].toarray())\n",
    "score(y_true=y[test_idx], y_pred=y_pred, tag=\"test set\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Label enhancement based strategy\n",
    "\n",
    "Label enhancement is the process to recover label distributions from logical or given labels ([paper](https://crad.ict.ac.cn/EN/Y2017/V54/I6/1171)). This idea can be utilized to perform splitting. In particular, this algorithm is based on an [iterative label propagation technique](https://ieeexplore.ieee.org/abstract/document/7373329) to recovers the label distributions. Then, we apply the community detection method, similar to the above method, to reassign response labels to their associated communities. Afterward, an algorithm from: Naive, Iterative, or Stratifying XML, is applied to perform splitting."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "## Configuration parameters to stratifying a multi-label dataset splitting\n",
      "   based on label enhancement approach:\n",
      "\t\t1. Subsampling input size: 10000\n",
      "\t\t2. Number of communities: 5\n",
      "\t\t3. Constant that scales the amount of laplacian norm regularization: 2\n",
      "\t\t4. A hyperparameter to balancing parameterwhich controls the fraction of the information inherited from the label propagation and the label matrix.: 0.2\n",
      "\t\t5. A hyper-parameter: 0.1\n",
      "\t\t6. A hyper-parameter: 0.1\n",
      "\t\t7. A hyper-parameter: 0.1\n",
      "\t\t8. Shuffle the dataset? True\n",
      "\t\t9. Split size: 0.8\n",
      "\t\t10. Number of examples to use in each iteration: 500\n",
      "\t\t11. Number of loops over training set: 50\n",
      "\t\t12. Number of parallel workers: 2\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\t>> Building Graph...\n",
      "\t>> Perform splitting (extreme)...\n",
      "\t\t--> Starting score: 28\n",
      "\t\t--> Splitting progress: 100.00%; score: -0.40\n"
     ]
    }
   ],
   "source": [
    "st = LabelEnhancementStratification(num_subsamples=10000, num_communities=num_clusters, \n",
    "                                    sigma=2, alpha=0.2, swap_probability=0.1, \n",
    "                                    threshold_proportion=0.1, decay=0.1, shuffle=True, \n",
    "                                    split_size=split_size, batch_size=batch_size, \n",
    "                                    num_epochs=num_epochs, num_jobs=num_jobs)\n",
    "training_idx, test_idx = st.fit(X=X, y=y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Properties for yeast</th>\n",
       "      <th>Complete set</th>\n",
       "      <th>Training set</th>\n",
       "      <th>Test set</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Number of examples</td>\n",
       "      <td>2417</td>\n",
       "      <td>1910</td>\n",
       "      <td>507</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Number of labels</td>\n",
       "      <td>10241</td>\n",
       "      <td>8064</td>\n",
       "      <td>2177</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Label cardinality</td>\n",
       "      <td>4.237071</td>\n",
       "      <td>4.22199</td>\n",
       "      <td>4.293886</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Label density</td>\n",
       "      <td>0.302648</td>\n",
       "      <td>0.301571</td>\n",
       "      <td>0.306706</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Distinct labels</td>\n",
       "      <td>14</td>\n",
       "      <td>14</td>\n",
       "      <td>14</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>Distinct label sets</td>\n",
       "      <td>198</td>\n",
       "      <td>174</td>\n",
       "      <td>108</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>Frequency of distinct label sets</td>\n",
       "      <td>0.08192</td>\n",
       "      <td>0.08192</td>\n",
       "      <td>0.08192</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>Mean imbalance ratio intra-class for all labels</td>\n",
       "      <td>9.578575</td>\n",
       "      <td>9.552217</td>\n",
       "      <td>9.717836</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>Mean imbalance ratio inter-class for all labels</td>\n",
       "      <td>7.196811</td>\n",
       "      <td>7.096647</td>\n",
       "      <td>7.60943</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>Mean imbalance ratio labelsets for all labels</td>\n",
       "      <td>0.468229</td>\n",
       "      <td>0.537607</td>\n",
       "      <td>1.70713</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>Labels having less than or equal to 5 examples</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>Labels having more than 6 examples</td>\n",
       "      <td>14</td>\n",
       "      <td>14</td>\n",
       "      <td>14</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>KL difference between complete and data partition</td>\n",
       "      <td>0</td>\n",
       "      <td>0.000136</td>\n",
       "      <td>0.001846</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                 Properties for yeast Complete set  \\\n",
       "0                                  Number of examples         2417   \n",
       "1                                    Number of labels        10241   \n",
       "2                                   Label cardinality     4.237071   \n",
       "3                                       Label density     0.302648   \n",
       "4                                     Distinct labels           14   \n",
       "5                                 Distinct label sets          198   \n",
       "6                    Frequency of distinct label sets      0.08192   \n",
       "7     Mean imbalance ratio intra-class for all labels     9.578575   \n",
       "8     Mean imbalance ratio inter-class for all labels     7.196811   \n",
       "9       Mean imbalance ratio labelsets for all labels     0.468229   \n",
       "10     Labels having less than or equal to 5 examples            0   \n",
       "11                 Labels having more than 6 examples           14   \n",
       "12  KL difference between complete and data partition            0   \n",
       "\n",
       "   Training set  Test set  \n",
       "0          1910       507  \n",
       "1          8064      2177  \n",
       "2       4.22199  4.293886  \n",
       "3      0.301571  0.306706  \n",
       "4            14        14  \n",
       "5           174       108  \n",
       "6       0.08192   0.08192  \n",
       "7      9.552217  9.717836  \n",
       "8      7.096647   7.60943  \n",
       "9      0.537607   1.70713  \n",
       "10            0         0  \n",
       "11           14        14  \n",
       "12     0.000136  0.001846  "
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model_name = \"enhance2split\"\n",
    "df, chart = data_properties(y=y, selected_examples=[training_idx, test_idx], \n",
    "                            num_tails=5, dataset_name=dsname, model_name=model_name, \n",
    "                            rspath=RESULT_PATH, display_dataframe=True, \n",
    "                            display_figure=True)\n",
    "df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "<div id=\"altair-viz-c834389c39fc475a92cd7a074adbab64\"></div>\n",
       "<script type=\"text/javascript\">\n",
       "  (function(spec, embedOpt){\n",
       "    let outputDiv = document.currentScript.previousElementSibling;\n",
       "    if (outputDiv.id !== \"altair-viz-c834389c39fc475a92cd7a074adbab64\") {\n",
       "      outputDiv = document.getElementById(\"altair-viz-c834389c39fc475a92cd7a074adbab64\");\n",
       "    }\n",
       "    const paths = {\n",
       "      \"vega\": \"https://cdn.jsdelivr.net/npm//vega@5?noext\",\n",
       "      \"vega-lib\": \"https://cdn.jsdelivr.net/npm//vega-lib?noext\",\n",
       "      \"vega-lite\": \"https://cdn.jsdelivr.net/npm//vega-lite@4.8.1?noext\",\n",
       "      \"vega-embed\": \"https://cdn.jsdelivr.net/npm//vega-embed@6?noext\",\n",
       "    };\n",
       "\n",
       "    function loadScript(lib) {\n",
       "      return new Promise(function(resolve, reject) {\n",
       "        var s = document.createElement('script');\n",
       "        s.src = paths[lib];\n",
       "        s.async = true;\n",
       "        s.onload = () => resolve(paths[lib]);\n",
       "        s.onerror = () => reject(`Error loading script: ${paths[lib]}`);\n",
       "        document.getElementsByTagName(\"head\")[0].appendChild(s);\n",
       "      });\n",
       "    }\n",
       "\n",
       "    function showError(err) {\n",
       "      outputDiv.innerHTML = `<div class=\"error\" style=\"color:red;\">${err}</div>`;\n",
       "      throw err;\n",
       "    }\n",
       "\n",
       "    function displayChart(vegaEmbed) {\n",
       "      vegaEmbed(outputDiv, spec, embedOpt)\n",
       "        .catch(err => showError(`Javascript Error: ${err.message}<br>This usually means there's a typo in your chart specification. See the javascript console for the full traceback.`));\n",
       "    }\n",
       "\n",
       "    if(typeof define === \"function\" && define.amd) {\n",
       "      requirejs.config({paths});\n",
       "      require([\"vega-embed\"], displayChart, err => showError(`Error loading script: ${err.message}`));\n",
       "    } else if (typeof vegaEmbed === \"function\") {\n",
       "      displayChart(vegaEmbed);\n",
       "    } else {\n",
       "      loadScript(\"vega\")\n",
       "        .then(() => loadScript(\"vega-lite\"))\n",
       "        .then(() => loadScript(\"vega-embed\"))\n",
       "        .catch(showError)\n",
       "        .then(() => displayChart(vegaEmbed));\n",
       "    }\n",
       "  })({\"data\": {\"name\": \"data-94c5c0930d3d56be1cbbbdac586bfd2d\"}, \"mark\": {\"type\": \"bar\", \"color\": \"grey\"}, \"config\": {\"axis\": {\"labelFontSize\": 12, \"labelLimit\": 500, \"labelPadding\": 5, \"titleFontSize\": 20}, \"axisY\": {\"grid\": false}, \"header\": {\"labelFontSize\": 15, \"titleFontSize\": 20}, \"legend\": {\"cornerRadius\": 10, \"fillColor\": \"white\", \"padding\": 10, \"strokeColor\": \"gray\"}}, \"encoding\": {\"color\": {\"type\": \"nominal\", \"field\": \"Dataset\", \"scale\": {\"range\": [\"red\", \"black\", \"blue\"]}}, \"x\": {\"type\": \"ordinal\", \"field\": \"Label\", \"sort\": \"ascending\", \"title\": \"Label ID\"}, \"y\": {\"type\": \"quantitative\", \"field\": \"Sum\", \"stack\": null, \"title\": \"Number of Examples\"}}, \"height\": 350, \"resolve\": {\"scale\": {\"x\": \"independent\"}}, \"title\": {\"text\": \"Number of examples for each label given yeast data.\", \"subtitle\": [\"The horizontal axis indicates the indices of labels while the vertical axis represents the number of associated examples\"]}, \"width\": 600, \"$schema\": \"https://vega.github.io/schema/vega-lite/v4.8.1.json\", \"datasets\": {\"data-94c5c0930d3d56be1cbbbdac586bfd2d\": [{\"Label\": 1, \"Dataset\": \"Complete\", \"Sum\": 34}, {\"Label\": 2, \"Dataset\": \"Complete\", \"Sum\": 178}, {\"Label\": 3, \"Dataset\": \"Complete\", \"Sum\": 253}, {\"Label\": 4, \"Dataset\": \"Complete\", \"Sum\": 289}, {\"Label\": 5, \"Dataset\": \"Complete\", \"Sum\": 428}, {\"Label\": 6, \"Dataset\": \"Complete\", \"Sum\": 480}, {\"Label\": 7, \"Dataset\": \"Complete\", \"Sum\": 597}, {\"Label\": 8, \"Dataset\": \"Complete\", \"Sum\": 722}, {\"Label\": 9, \"Dataset\": \"Complete\", \"Sum\": 762}, {\"Label\": 10, \"Dataset\": \"Complete\", \"Sum\": 862}, {\"Label\": 11, \"Dataset\": \"Complete\", \"Sum\": 983}, {\"Label\": 12, \"Dataset\": \"Complete\", \"Sum\": 1038}, {\"Label\": 13, \"Dataset\": \"Complete\", \"Sum\": 1799}, {\"Label\": 14, \"Dataset\": \"Complete\", \"Sum\": 1816}, {\"Label\": 1, \"Dataset\": \"Train\", \"Sum\": 27}, {\"Label\": 2, \"Dataset\": \"Train\", \"Sum\": 144}, {\"Label\": 3, \"Dataset\": \"Train\", \"Sum\": 203}, {\"Label\": 4, \"Dataset\": \"Train\", \"Sum\": 228}, {\"Label\": 5, \"Dataset\": \"Train\", \"Sum\": 322}, {\"Label\": 6, \"Dataset\": \"Train\", \"Sum\": 369}, {\"Label\": 7, \"Dataset\": \"Train\", \"Sum\": 464}, {\"Label\": 8, \"Dataset\": \"Train\", \"Sum\": 565}, {\"Label\": 9, \"Dataset\": \"Train\", \"Sum\": 612}, {\"Label\": 10, \"Dataset\": \"Train\", \"Sum\": 684}, {\"Label\": 11, \"Dataset\": \"Train\", \"Sum\": 785}, {\"Label\": 12, \"Dataset\": \"Train\", \"Sum\": 836}, {\"Label\": 13, \"Dataset\": \"Train\", \"Sum\": 1406}, {\"Label\": 14, \"Dataset\": \"Train\", \"Sum\": 1419}, {\"Label\": 1, \"Dataset\": \"Test\", \"Sum\": 7}, {\"Label\": 2, \"Dataset\": \"Test\", \"Sum\": 34}, {\"Label\": 3, \"Dataset\": \"Test\", \"Sum\": 50}, {\"Label\": 4, \"Dataset\": \"Test\", \"Sum\": 61}, {\"Label\": 5, \"Dataset\": \"Test\", \"Sum\": 106}, {\"Label\": 6, \"Dataset\": \"Test\", \"Sum\": 111}, {\"Label\": 7, \"Dataset\": \"Test\", \"Sum\": 133}, {\"Label\": 8, \"Dataset\": \"Test\", \"Sum\": 157}, {\"Label\": 9, \"Dataset\": \"Test\", \"Sum\": 150}, {\"Label\": 10, \"Dataset\": \"Test\", \"Sum\": 178}, {\"Label\": 11, \"Dataset\": \"Test\", \"Sum\": 198}, {\"Label\": 12, \"Dataset\": \"Test\", \"Sum\": 202}, {\"Label\": 13, \"Dataset\": \"Test\", \"Sum\": 393}, {\"Label\": 14, \"Dataset\": \"Test\", \"Sum\": 397}]}}, {\"mode\": \"vega-lite\"});\n",
       "</script>"
      ],
      "text/plain": [
       "alt.Chart(...)"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "chart"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's train the chain classifier to evaluate the performance of partitions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\t>> F1-score for training set...\n",
      "\t\t--> Average sample f1-score: 0.6332\n",
      "\t\t--> Micro f1-score: 0.6535\n",
      "\t\t--> Macro f1-score: 0.3919\n",
      "\t>> F1-score for test set...\n",
      "\t\t--> Average sample f1-score: 0.6251\n",
      "\t\t--> Micro f1-score: 0.6506\n",
      "\t\t--> Macro f1-score: 0.3823\n"
     ]
    }
   ],
   "source": [
    "# train\n",
    "model.fit(X[training_idx].toarray(), y[training_idx].toarray())\n",
    "# evaluate\n",
    "y_pred = model.predict(X[training_idx].toarray())\n",
    "score(y_true=y[training_idx], y_pred=y_pred, tag=\"training set\")\n",
    "\n",
    "y_pred = model.predict(X[test_idx].toarray())\n",
    "score(y_true=y[test_idx], y_pred=y_pred, tag=\"test set\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The results are seen to be improved in relation to Stratifying XML."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "aKUZnDiqQrAh"
   },
   "source": [
    "### Active learning based splitting strategy\n",
    "\n",
    "The subsampling approach aims to select the most informative examples using an appropriate acquisition function, such as entropy, then train on selected examples. An extended version was introduced in the [paper](https://www.biorxiv.org/content/10.1101/2020.09.14.297424v1) and integrated into the [leADS](https://github.com/hallamlab/leADS) software. Here, the aim is to measures the uncertainty of an example over labels using either entropy or [normalized propensity scored precision (nPSP)](https://www.biorxiv.org/content/10.1101/2020.09.14.297424v1) and then use the scores to calibrate the examples score in Stratified XML based splitting."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "## Configuration parameters to estimating examples predictive uncertainty\n",
      "   scores to group example with high informativeness into training set\n",
      "   using a modified approach to splitting an extreme large scale multi-\n",
      "   label dataset:\n",
      "\t\t1. Subsampling labels: 10\n",
      "\t\t2. The acquisition function for estimating the predictive uncertainty: entropy\n",
      "\t\t3. Apply sklearn optimizers? False\n",
      "\t\t4. The loss function: hinge\n",
      "\t\t5. A hyper-parameter for extreme stratification: 0.1\n",
      "\t\t6. A hyper-parameter for extreme stratification: 0.1\n",
      "\t\t7. A hyper-parameter for extreme stratification: 0.1\n",
      "\t\t8. The penalty (aka regularization term): elasticnet\n",
      "\t\t9. Constant controlling the elastic term: 0.0001\n",
      "\t\t10. The elastic net mixing parameter: 0.65\n",
      "\t\t11. A cutoff threshold between two consecutive rounds: 0.05\n",
      "\t\t12. Shuffle the dataset? True\n",
      "\t\t13. Split size: 0.8\n",
      "\t\t14. Number of examples to use in each iteration: 500\n",
      "\t\t15. Number of loops over training set: 50\n",
      "\t\t16. Learning rate: 0.001\n",
      "\t\t17. How often to evaluate? 1\n",
      "\t\t18. Number of parallel workers: 2\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\t>> Training to learn a model...\n",
      "\t   1)- Epoch count (1/50)...\n",
      "  \t\t<<<------------<<<------------<<<\n",
      "  \t\t>> Feed-Backward...\n",
      "\t\t\t--> Optimizing Theta: 100.00%...\n",
      "  \t\t>>>------------>>>------------>>>\n",
      "  \t\t>> Feed-Forward...\n",
      "  \t\t>> Predictive uncertainty using entropy...0000%...\n",
      "  \t\t>> Compute cost...\n",
      "\t\t\t--> Calculating cost: 100.00%...\n",
      "\t\t\t--> New cost: 0.8492; Old cost: inf\n",
      "\t\t\t--> Epoch 1 took 0.232 seconds...\n",
      "\t   2)- Epoch count (2/50)...\n",
      "  \t\t<<<------------<<<------------<<<\n",
      "  \t\t>> Feed-Backward...\n",
      "\t\t\t--> Optimizing Theta: 100.00%...\n",
      "  \t\t>>>------------>>>------------>>>\n",
      "  \t\t>> Feed-Forward...\n",
      "  \t\t>> Predictive uncertainty using entropy...\n",
      "  \t\t>> Compute cost...\n",
      "\t\t\t--> Calculating cost: 100.00%...\n",
      "\t\t\t--> New cost: 0.8542; Old cost: 0.8492\n",
      "\t\t\t--> Epoch 2 took 0.272 seconds...\n",
      "\t   3)- Epoch count (3/50)...\n",
      "  \t\t<<<------------<<<------------<<<\n",
      "  \t\t>> Feed-Backward...\n",
      "\t\t\t--> Optimizing Theta: 100.00%...\n",
      "  \t\t>>>------------>>>------------>>>\n",
      "  \t\t>> Feed-Forward...\n",
      "  \t\t>> Predictive uncertainty using entropy...0000%...\n",
      "  \t\t>> Compute cost...\n",
      "\t\t\t--> Calculating cost: 100.00%...\n",
      "\t\t\t--> New cost: 0.8498; Old cost: 0.8492\n",
      "\t\t\t--> Epoch 3 took 0.337 seconds...\n",
      "\t   4)- Epoch count (4/50)...\n",
      "  \t\t<<<------------<<<------------<<<\n",
      "  \t\t>> Feed-Backward...\n",
      "\t\t\t--> Optimizing Theta: 100.00%...\n",
      "  \t\t>>>------------>>>------------>>>\n",
      "  \t\t>> Feed-Forward...\n",
      "  \t\t>> Predictive uncertainty using entropy...0000%...\n",
      "  \t\t>> Compute cost...\n",
      "\t\t\t--> Calculating cost: 100.00%...\n",
      "\t\t\t--> New cost: 0.8534; Old cost: 0.8492\n",
      "\t\t\t--> Epoch 4 took 0.266 seconds...\n",
      "\t   5)- Epoch count (5/50)...\n",
      "  \t\t<<<------------<<<------------<<<\n",
      "  \t\t>> Feed-Backward...\n",
      "\t\t\t--> Optimizing Theta: 100.00%...\n",
      "  \t\t>>>------------>>>------------>>>\n",
      "  \t\t>> Feed-Forward...\n",
      "  \t\t>> Predictive uncertainty using entropy...0000%...\n",
      "  \t\t>> Compute cost...\n",
      "\t\t\t--> Calculating cost: 100.00%...\n",
      "\t\t\t--> New cost: 0.8492; Old cost: 0.8492\n",
      "\t\t\t--> Epoch 5 took 0.311 seconds...\n",
      "\t   6)- Epoch count (6/50)...\n",
      "  \t\t<<<------------<<<------------<<<\n",
      "  \t\t>> Feed-Backward...\n",
      "\t\t\t--> Optimizing Theta: 100.00%...\n",
      "  \t\t>>>------------>>>------------>>>\n",
      "  \t\t>> Feed-Forward...\n",
      "  \t\t>> Predictive uncertainty using entropy...0000%...\n",
      "  \t\t>> Compute cost...\n",
      "\t\t\t--> Calculating cost: 100.00%...\n",
      "\t\t\t--> New cost: 0.8489; Old cost: 0.8492\n",
      "\t\t\t--> Epoch 6 took 0.262 seconds...\n",
      "\t   7)- Epoch count (7/50)...\n",
      "  \t\t<<<------------<<<------------<<<\n",
      "  \t\t>> Feed-Backward...\n",
      "\t\t\t--> Optimizing Theta: 100.00%...\n",
      "  \t\t>>>------------>>>------------>>>\n",
      "  \t\t>> Feed-Forward...\n",
      "  \t\t>> Predictive uncertainty using entropy...0000%...\n",
      "  \t\t>> Compute cost...\n",
      "\t\t\t--> Calculating cost: 100.00%...\n",
      "\t\t\t--> New cost: 0.8493; Old cost: 0.8489\n",
      "\t\t\t--> Epoch 7 took 0.292 seconds...\n",
      "\t   8)- Epoch count (8/50)...\n",
      "  \t\t<<<------------<<<------------<<<\n",
      "  \t\t>> Feed-Backward...\n",
      "\t\t\t--> Optimizing Theta: 100.00%...\n",
      "  \t\t>>>------------>>>------------>>>\n",
      "  \t\t>> Feed-Forward...\n",
      "  \t\t>> Predictive uncertainty using entropy...\n",
      "  \t\t>> Compute cost...\n",
      "\t\t\t--> Calculating cost: 100.00%...\n",
      "\t\t\t--> New cost: 0.8516; Old cost: 0.8489\n",
      "\t\t\t--> Epoch 8 took 0.241 seconds...\n",
      "\t   9)- Epoch count (9/50)...\n",
      "  \t\t<<<------------<<<------------<<<\n",
      "  \t\t>> Feed-Backward...\n",
      "\t\t\t--> Optimizing Theta: 100.00%...\n",
      "  \t\t>>>------------>>>------------>>>\n",
      "  \t\t>> Feed-Forward...\n",
      "  \t\t>> Predictive uncertainty using entropy...\n",
      "  \t\t>> Compute cost...\n",
      "\t\t\t--> Calculating cost: 100.00%...\n",
      "\t\t\t--> New cost: 0.8495; Old cost: 0.8489\n",
      "\t\t\t--> Epoch 9 took 0.257 seconds...\n",
      "\t   10)- Epoch count (10/50)...\n",
      "  \t\t<<<------------<<<------------<<<\n",
      "  \t\t>> Feed-Backward...\n",
      "\t\t\t--> Optimizing Theta: 100.00%...\n",
      "  \t\t>>>------------>>>------------>>>\n",
      "  \t\t>> Feed-Forward...\n",
      "  \t\t>> Predictive uncertainty using entropy...0000%...\n",
      "  \t\t>> Compute cost...\n",
      "\t\t\t--> Calculating cost: 100.00%...\n",
      "\t\t\t--> New cost: 0.8526; Old cost: 0.8489\n",
      "\t\t\t--> Epoch 10 took 0.305 seconds...\n",
      "\t   11)- Epoch count (11/50)...\n",
      "  \t\t<<<------------<<<------------<<<\n",
      "  \t\t>> Feed-Backward...\n",
      "\t\t\t--> Optimizing Theta: 100.00%...\n",
      "  \t\t>>>------------>>>------------>>>\n",
      "  \t\t>> Feed-Forward...\n",
      "  \t\t>> Predictive uncertainty using entropy...0000%...\n",
      "  \t\t>> Compute cost...\n",
      "\t\t\t--> Calculating cost: 100.00%...\n",
      "\t\t\t--> New cost: 0.8531; Old cost: 0.8489\n",
      "\t\t\t--> Epoch 11 took 0.254 seconds...\n",
      "\t   12)- Epoch count (12/50)...\n",
      "  \t\t<<<------------<<<------------<<<\n",
      "  \t\t>> Feed-Backward...\n",
      "\t\t\t--> Optimizing Theta: 100.00%...\n",
      "  \t\t>>>------------>>>------------>>>\n",
      "  \t\t>> Feed-Forward...\n",
      "  \t\t>> Predictive uncertainty using entropy...\n",
      "  \t\t>> Compute cost...\n",
      "\t\t\t--> Calculating cost: 100.00%...\n",
      "\t\t\t--> New cost: 0.8510; Old cost: 0.8489\n",
      "\t\t\t--> Epoch 12 took 0.3 seconds...\n",
      "\t   13)- Epoch count (13/50)...\n",
      "  \t\t<<<------------<<<------------<<<\n",
      "  \t\t>> Feed-Backward...\n",
      "\t\t\t--> Optimizing Theta: 100.00%...\n",
      "  \t\t>>>------------>>>------------>>>\n",
      "  \t\t>> Feed-Forward...\n",
      "  \t\t>> Predictive uncertainty using entropy...\n",
      "  \t\t>> Compute cost...\n",
      "\t\t\t--> Calculating cost: 100.00%...\n",
      "\t\t\t--> New cost: 0.8548; Old cost: 0.8489\n",
      "\t\t\t--> Epoch 13 took 0.248 seconds...\n",
      "\t   14)- Epoch count (14/50)...\n",
      "  \t\t<<<------------<<<------------<<<\n",
      "  \t\t>> Feed-Backward...\n",
      "\t\t\t--> Optimizing Theta: 100.00%...\n",
      "  \t\t>>>------------>>>------------>>>\n",
      "  \t\t>> Feed-Forward...\n",
      "  \t\t>> Predictive uncertainty using entropy...\n",
      "  \t\t>> Compute cost...\n",
      "\t\t\t--> Calculating cost: 100.00%...\n",
      "\t\t\t--> New cost: 0.8533; Old cost: 0.8489\n",
      "\t\t\t--> Epoch 14 took 0.291 seconds...\n",
      "\t   15)- Epoch count (15/50)...\n",
      "  \t\t<<<------------<<<------------<<<\n",
      "  \t\t>> Feed-Backward...\n",
      "\t\t\t--> Optimizing Theta: 100.00%...\n",
      "  \t\t>>>------------>>>------------>>>\n",
      "  \t\t>> Feed-Forward...\n",
      "  \t\t>> Predictive uncertainty using entropy...0000%...\n",
      "  \t\t>> Compute cost...\n",
      "\t\t\t--> Calculating cost: 100.00%...\n",
      "\t\t\t--> New cost: 0.8532; Old cost: 0.8489\n",
      "\t\t\t--> Epoch 15 took 0.289 seconds...\n",
      "\t   16)- Epoch count (16/50)...\n",
      "  \t\t<<<------------<<<------------<<<\n",
      "  \t\t>> Feed-Backward...\n",
      "\t\t\t--> Optimizing Theta: 100.00%...\n",
      "  \t\t>>>------------>>>------------>>>\n",
      "  \t\t>> Feed-Forward...\n",
      "  \t\t>> Predictive uncertainty using entropy...0000%...\n",
      "  \t\t>> Compute cost...\n",
      "\t\t\t--> Calculating cost: 100.00%...\n",
      "\t\t\t--> New cost: 0.8568; Old cost: 0.8489\n",
      "\t\t\t--> Epoch 16 took 0.243 seconds...\n",
      "\t   17)- Epoch count (17/50)...\n",
      "  \t\t<<<------------<<<------------<<<\n",
      "  \t\t>> Feed-Backward...\n",
      "\t\t\t--> Optimizing Theta: 100.00%...\n",
      "  \t\t>>>------------>>>------------>>>\n",
      "  \t\t>> Feed-Forward...\n",
      "  \t\t>> Predictive uncertainty using entropy...\n",
      "  \t\t>> Compute cost...\n",
      "\t\t\t--> Calculating cost: 100.00%...\n",
      "\t\t\t--> New cost: 0.8554; Old cost: 0.8489\n",
      "\t\t\t--> Epoch 17 took 0.254 seconds...\n",
      "\t   18)- Epoch count (18/50)...\n",
      "  \t\t<<<------------<<<------------<<<\n",
      "  \t\t>> Feed-Backward...\n",
      "\t\t\t--> Optimizing Theta: 100.00%...\n",
      "  \t\t>>>------------>>>------------>>>\n",
      "  \t\t>> Feed-Forward...\n",
      "  \t\t>> Predictive uncertainty using entropy...\n",
      "  \t\t>> Compute cost...\n",
      "\t\t\t--> Calculating cost: 100.00%...\n",
      "\t\t\t--> New cost: 0.8562; Old cost: 0.8489\n",
      "\t\t\t--> Epoch 18 took 0.249 seconds...\n",
      "\t   19)- Epoch count (19/50)...\n",
      "  \t\t<<<------------<<<------------<<<\n",
      "  \t\t>> Feed-Backward...\n",
      "\t\t\t--> Optimizing Theta: 100.00%...\n",
      "  \t\t>>>------------>>>------------>>>\n",
      "  \t\t>> Feed-Forward...\n",
      "  \t\t>> Predictive uncertainty using entropy...0000%...\n",
      "  \t\t>> Compute cost...\n",
      "\t\t\t--> Calculating cost: 100.00%...\n",
      "\t\t\t--> New cost: 0.8552; Old cost: 0.8489\n",
      "\t\t\t--> Epoch 19 took 0.318 seconds...\n",
      "\t   20)- Epoch count (20/50)...\n",
      "  \t\t<<<------------<<<------------<<<\n",
      "  \t\t>> Feed-Backward...\n",
      "\t\t\t--> Optimizing Theta: 100.00%...\n",
      "  \t\t>>>------------>>>------------>>>\n",
      "  \t\t>> Feed-Forward...\n",
      "  \t\t>> Predictive uncertainty using entropy...\n",
      "  \t\t>> Compute cost...\n",
      "\t\t\t--> Calculating cost: 100.00%...\n",
      "\t\t\t--> New cost: 0.8567; Old cost: 0.8489\n",
      "\t\t\t--> Epoch 20 took 0.257 seconds...\n",
      "\t   21)- Epoch count (21/50)...\n",
      "  \t\t<<<------------<<<------------<<<\n",
      "  \t\t>> Feed-Backward...\n",
      "\t\t\t--> Optimizing Theta: 100.00%...\n",
      "  \t\t>>>------------>>>------------>>>\n",
      "  \t\t>> Feed-Forward...\n",
      "  \t\t>> Predictive uncertainty using entropy...\n",
      "  \t\t>> Compute cost...\n",
      "\t\t\t--> Calculating cost: 100.00%...\n",
      "\t\t\t--> New cost: 0.8564; Old cost: 0.8489\n",
      "\t\t\t--> Epoch 21 took 0.305 seconds...\n",
      "\t   22)- Epoch count (22/50)...\n",
      "  \t\t<<<------------<<<------------<<<\n",
      "  \t\t>> Feed-Backward...\n",
      "\t\t\t--> Optimizing Theta: 100.00%...\n",
      "  \t\t>>>------------>>>------------>>>\n",
      "  \t\t>> Feed-Forward...\n",
      "  \t\t>> Predictive uncertainty using entropy...\n",
      "  \t\t>> Compute cost...\n",
      "\t\t\t--> Calculating cost: 100.00%...\n",
      "\t\t\t--> New cost: 0.8579; Old cost: 0.8489\n",
      "\t\t\t--> Epoch 22 took 0.245 seconds...\n",
      "\t   23)- Epoch count (23/50)...\n",
      "  \t\t<<<------------<<<------------<<<\n",
      "  \t\t>> Feed-Backward...\n",
      "\t\t\t--> Optimizing Theta: 100.00%...\n",
      "  \t\t>>>------------>>>------------>>>\n",
      "  \t\t>> Feed-Forward...\n",
      "  \t\t>> Predictive uncertainty using entropy...\n",
      "  \t\t>> Compute cost...\n",
      "\t\t\t--> Calculating cost: 100.00%...\n",
      "\t\t\t--> New cost: 0.8586; Old cost: 0.8489\n",
      "\t\t\t--> Epoch 23 took 0.312 seconds...\n",
      "\t   24)- Epoch count (24/50)...\n",
      "  \t\t<<<------------<<<------------<<<\n",
      "  \t\t>> Feed-Backward...\n",
      "\t\t\t--> Optimizing Theta: 100.00%...\n",
      "  \t\t>>>------------>>>------------>>>\n",
      "  \t\t>> Feed-Forward...\n",
      "  \t\t>> Predictive uncertainty using entropy...0000%...\n",
      "  \t\t>> Compute cost...\n",
      "\t\t\t--> Calculating cost: 100.00%...\n",
      "\t\t\t--> New cost: 0.8581; Old cost: 0.8489\n",
      "\t\t\t--> Epoch 24 took 0.252 seconds...\n",
      "\t   25)- Epoch count (25/50)...\n",
      "  \t\t<<<------------<<<------------<<<\n",
      "  \t\t>> Feed-Backward...\n",
      "\t\t\t--> Optimizing Theta: 100.00%...\n",
      "  \t\t>>>------------>>>------------>>>\n",
      "  \t\t>> Feed-Forward...\n",
      "  \t\t>> Predictive uncertainty using entropy...\n",
      "  \t\t>> Compute cost...\n",
      "\t\t\t--> Calculating cost: 100.00%...\n",
      "\t\t\t--> New cost: 0.8559; Old cost: 0.8489\n",
      "\t\t\t--> Epoch 25 took 0.252 seconds...\n",
      "\t   26)- Epoch count (26/50)...\n",
      "  \t\t<<<------------<<<------------<<<\n",
      "  \t\t>> Feed-Backward...\n",
      "\t\t\t--> Optimizing Theta: 100.00%...\n",
      "  \t\t>>>------------>>>------------>>>\n",
      "  \t\t>> Feed-Forward...\n",
      "  \t\t>> Predictive uncertainty using entropy...0000%...\n",
      "  \t\t>> Compute cost...\n",
      "\t\t\t--> Calculating cost: 100.00%...\n",
      "\t\t\t--> New cost: 0.8583; Old cost: 0.8489\n",
      "\t\t\t--> Epoch 26 took 0.244 seconds...\n",
      "\t   27)- Epoch count (27/50)...\n",
      "  \t\t<<<------------<<<------------<<<\n",
      "  \t\t>> Feed-Backward...\n",
      "\t\t\t--> Optimizing Theta: 100.00%...\n",
      "  \t\t>>>------------>>>------------>>>\n",
      "  \t\t>> Feed-Forward...\n",
      "  \t\t>> Predictive uncertainty using entropy...\n",
      "  \t\t>> Compute cost...\n",
      "\t\t\t--> Calculating cost: 100.00%...\n",
      "\t\t\t--> New cost: 0.8577; Old cost: 0.8489\n",
      "\t\t\t--> Epoch 27 took 0.245 seconds...\n",
      "\t   28)- Epoch count (28/50)...\n",
      "  \t\t<<<------------<<<------------<<<\n",
      "  \t\t>> Feed-Backward...\n",
      "\t\t\t--> Optimizing Theta: 100.00%...\n",
      "  \t\t>>>------------>>>------------>>>\n",
      "  \t\t>> Feed-Forward...\n",
      "  \t\t>> Predictive uncertainty using entropy...0000%...\n",
      "  \t\t>> Compute cost...\n",
      "\t\t\t--> Calculating cost: 100.00%...\n",
      "\t\t\t--> New cost: 0.8593; Old cost: 0.8489\n",
      "\t\t\t--> Epoch 28 took 0.3 seconds...\n",
      "\t   29)- Epoch count (29/50)...\n",
      "  \t\t<<<------------<<<------------<<<\n",
      "  \t\t>> Feed-Backward...\n",
      "\t\t\t--> Optimizing Theta: 100.00%...\n",
      "  \t\t>>>------------>>>------------>>>\n",
      "  \t\t>> Feed-Forward...\n",
      "  \t\t>> Predictive uncertainty using entropy...\n",
      "  \t\t>> Compute cost...\n",
      "\t\t\t--> Calculating cost: 100.00%...\n",
      "\t\t\t--> New cost: 0.8577; Old cost: 0.8489\n",
      "\t\t\t--> Epoch 29 took 0.248 seconds...\n",
      "\t   30)- Epoch count (30/50)...\n",
      "  \t\t<<<------------<<<------------<<<\n",
      "  \t\t>> Feed-Backward...\n",
      "\t\t\t--> Optimizing Theta: 100.00%...\n",
      "  \t\t>>>------------>>>------------>>>\n",
      "  \t\t>> Feed-Forward...\n",
      "  \t\t>> Predictive uncertainty using entropy...0000%...\n",
      "  \t\t>> Compute cost...\n",
      "\t\t\t--> Calculating cost: 100.00%...\n",
      "\t\t\t--> New cost: 0.8583; Old cost: 0.8489\n",
      "\t\t\t--> Epoch 30 took 0.3 seconds...\n",
      "\t   31)- Epoch count (31/50)...\n",
      "  \t\t<<<------------<<<------------<<<\n",
      "  \t\t>> Feed-Backward...\n",
      "\t\t\t--> Optimizing Theta: 100.00%...\n",
      "  \t\t>>>------------>>>------------>>>\n",
      "  \t\t>> Feed-Forward...\n",
      "  \t\t>> Predictive uncertainty using entropy...0000%...\n",
      "  \t\t>> Compute cost...\n",
      "\t\t\t--> Calculating cost: 100.00%...\n",
      "\t\t\t--> New cost: 0.8592; Old cost: 0.8489\n",
      "\t\t\t--> Epoch 31 took 0.237 seconds...\n",
      "\t   32)- Epoch count (32/50)...\n",
      "  \t\t<<<------------<<<------------<<<\n",
      "  \t\t>> Feed-Backward...\n",
      "\t\t\t--> Optimizing Theta: 100.00%...\n",
      "  \t\t>>>------------>>>------------>>>\n",
      "  \t\t>> Feed-Forward...\n",
      "  \t\t>> Predictive uncertainty using entropy...0000%...\n",
      "  \t\t>> Compute cost...\n",
      "\t\t\t--> Calculating cost: 100.00%...\n",
      "\t\t\t--> New cost: 0.8591; Old cost: 0.8489\n",
      "\t\t\t--> Epoch 32 took 0.294 seconds...\n",
      "\t   33)- Epoch count (33/50)...\n",
      "  \t\t<<<------------<<<------------<<<\n",
      "  \t\t>> Feed-Backward...\n",
      "\t\t\t--> Optimizing Theta: 100.00%...\n",
      "  \t\t>>>------------>>>------------>>>\n",
      "  \t\t>> Feed-Forward...\n",
      "  \t\t>> Predictive uncertainty using entropy...0000%...\n",
      "  \t\t>> Compute cost...\n",
      "\t\t\t--> Calculating cost: 100.00%...\n",
      "\t\t\t--> New cost: 0.8587; Old cost: 0.8489\n",
      "\t\t\t--> Epoch 33 took 0.279 seconds...\n",
      "\t   34)- Epoch count (34/50)...\n",
      "  \t\t<<<------------<<<------------<<<\n",
      "  \t\t>> Feed-Backward...\n",
      "\t\t\t--> Optimizing Theta: 100.00%...\n",
      "  \t\t>>>------------>>>------------>>>\n",
      "  \t\t>> Feed-Forward...\n",
      "  \t\t>> Predictive uncertainty using entropy...\n",
      "  \t\t>> Compute cost...\n",
      "\t\t\t--> Calculating cost: 100.00%...\n",
      "\t\t\t--> New cost: 0.8589; Old cost: 0.8489\n",
      "\t\t\t--> Epoch 34 took 0.254 seconds...\n",
      "\t   35)- Epoch count (35/50)...\n",
      "  \t\t<<<------------<<<------------<<<\n",
      "  \t\t>> Feed-Backward...\n",
      "\t\t\t--> Optimizing Theta: 100.00%...\n",
      "  \t\t>>>------------>>>------------>>>\n",
      "  \t\t>> Feed-Forward...\n",
      "  \t\t>> Predictive uncertainty using entropy...\n",
      "  \t\t>> Compute cost...\n",
      "\t\t\t--> Calculating cost: 100.00%...\n",
      "\t\t\t--> New cost: 0.8611; Old cost: 0.8489\n",
      "\t\t\t--> Epoch 35 took 0.255 seconds...\n",
      "\t   36)- Epoch count (36/50)...\n",
      "  \t\t<<<------------<<<------------<<<\n",
      "  \t\t>> Feed-Backward...\n",
      "\t\t\t--> Optimizing Theta: 100.00%...\n",
      "  \t\t>>>------------>>>------------>>>\n",
      "  \t\t>> Feed-Forward...\n",
      "  \t\t>> Predictive uncertainty using entropy...0000%...\n",
      "  \t\t>> Compute cost...\n",
      "\t\t\t--> Calculating cost: 100.00%...\n",
      "\t\t\t--> New cost: 0.8595; Old cost: 0.8489\n",
      "\t\t\t--> Epoch 36 took 0.257 seconds...\n",
      "\t   37)- Epoch count (37/50)...\n",
      "  \t\t<<<------------<<<------------<<<\n",
      "  \t\t>> Feed-Backward...\n",
      "\t\t\t--> Optimizing Theta: 100.00%...\n",
      "  \t\t>>>------------>>>------------>>>\n",
      "  \t\t>> Feed-Forward...\n",
      "  \t\t>> Predictive uncertainty using entropy...\n",
      "  \t\t>> Compute cost...\n",
      "\t\t\t--> Calculating cost: 100.00%...\n",
      "\t\t\t--> New cost: 0.8607; Old cost: 0.8489\n",
      "\t\t\t--> Epoch 37 took 0.301 seconds...\n",
      "\t   38)- Epoch count (38/50)...\n",
      "  \t\t<<<------------<<<------------<<<\n",
      "  \t\t>> Feed-Backward...\n",
      "\t\t\t--> Optimizing Theta: 100.00%...\n",
      "  \t\t>>>------------>>>------------>>>\n",
      "  \t\t>> Feed-Forward...\n",
      "  \t\t>> Predictive uncertainty using entropy...\n",
      "  \t\t>> Compute cost...\n",
      "\t\t\t--> Calculating cost: 100.00%...\n",
      "\t\t\t--> New cost: 0.8605; Old cost: 0.8489\n",
      "\t\t\t--> Epoch 38 took 0.255 seconds...\n",
      "\t   39)- Epoch count (39/50)...\n",
      "  \t\t<<<------------<<<------------<<<\n",
      "  \t\t>> Feed-Backward...\n",
      "\t\t\t--> Optimizing Theta: 100.00%...\n",
      "  \t\t>>>------------>>>------------>>>\n",
      "  \t\t>> Feed-Forward...\n",
      "  \t\t>> Predictive uncertainty using entropy...\n",
      "  \t\t>> Compute cost...\n",
      "\t\t\t--> Calculating cost: 100.00%...\n",
      "\t\t\t--> New cost: 0.8613; Old cost: 0.8489\n",
      "\t\t\t--> Epoch 39 took 0.328 seconds...\n",
      "\t   40)- Epoch count (40/50)...\n",
      "  \t\t<<<------------<<<------------<<<\n",
      "  \t\t>> Feed-Backward...\n",
      "\t\t\t--> Optimizing Theta: 100.00%...\n",
      "  \t\t>>>------------>>>------------>>>\n",
      "  \t\t>> Feed-Forward...\n",
      "  \t\t>> Predictive uncertainty using entropy...\n",
      "  \t\t>> Compute cost...\n",
      "\t\t\t--> Calculating cost: 100.00%...\n",
      "\t\t\t--> New cost: 0.8612; Old cost: 0.8489\n",
      "\t\t\t--> Epoch 40 took 0.261 seconds...\n",
      "\t   41)- Epoch count (41/50)...\n",
      "  \t\t<<<------------<<<------------<<<\n",
      "  \t\t>> Feed-Backward...\n",
      "\t\t\t--> Optimizing Theta: 100.00%...\n",
      "  \t\t>>>------------>>>------------>>>\n",
      "  \t\t>> Feed-Forward...\n",
      "  \t\t>> Predictive uncertainty using entropy...\n",
      "  \t\t>> Compute cost...\n",
      "\t\t\t--> Calculating cost: 100.00%...\n",
      "\t\t\t--> New cost: 0.8609; Old cost: 0.8489\n",
      "\t\t\t--> Epoch 41 took 0.3 seconds...\n",
      "\t   42)- Epoch count (42/50)...\n",
      "  \t\t<<<------------<<<------------<<<\n",
      "  \t\t>> Feed-Backward...\n",
      "\t\t\t--> Optimizing Theta: 100.00%...\n",
      "  \t\t>>>------------>>>------------>>>\n",
      "  \t\t>> Feed-Forward...\n",
      "  \t\t>> Predictive uncertainty using entropy...\n",
      "  \t\t>> Compute cost...\n",
      "\t\t\t--> Calculating cost: 100.00%...\n",
      "\t\t\t--> New cost: 0.8589; Old cost: 0.8489\n",
      "\t\t\t--> Epoch 42 took 0.253 seconds...\n",
      "\t   43)- Epoch count (43/50)...\n",
      "  \t\t<<<------------<<<------------<<<\n",
      "  \t\t>> Feed-Backward...\n",
      "\t\t\t--> Optimizing Theta: 100.00%...\n",
      "  \t\t>>>------------>>>------------>>>\n",
      "  \t\t>> Feed-Forward...\n",
      "  \t\t>> Predictive uncertainty using entropy...\n",
      "  \t\t>> Compute cost...\n",
      "\t\t\t--> Calculating cost: 100.00%...\n",
      "\t\t\t--> New cost: 0.8619; Old cost: 0.8489\n",
      "\t\t\t--> Epoch 43 took 0.241 seconds...\n",
      "\t   44)- Epoch count (44/50)...\n",
      "  \t\t<<<------------<<<------------<<<\n",
      "  \t\t>> Feed-Backward...\n",
      "\t\t\t--> Optimizing Theta: 100.00%...\n",
      "  \t\t>>>------------>>>------------>>>\n",
      "  \t\t>> Feed-Forward...\n",
      "  \t\t>> Predictive uncertainty using entropy...0000%...\n",
      "  \t\t>> Compute cost...\n",
      "\t\t\t--> Calculating cost: 100.00%...\n",
      "\t\t\t--> New cost: 0.8597; Old cost: 0.8489\n",
      "\t\t\t--> Epoch 44 took 0.255 seconds...\n",
      "\t   45)- Epoch count (45/50)...\n",
      "  \t\t<<<------------<<<------------<<<\n",
      "  \t\t>> Feed-Backward...\n",
      "\t\t\t--> Optimizing Theta: 100.00%...\n",
      "  \t\t>>>------------>>>------------>>>\n",
      "  \t\t>> Feed-Forward...\n",
      "  \t\t>> Predictive uncertainty using entropy...0000%...\n",
      "  \t\t>> Compute cost...\n",
      "\t\t\t--> Calculating cost: 100.00%...\n",
      "\t\t\t--> New cost: 0.8618; Old cost: 0.8489\n",
      "\t\t\t--> Epoch 45 took 0.268 seconds...\n",
      "\t   46)- Epoch count (46/50)...\n",
      "  \t\t<<<------------<<<------------<<<\n",
      "  \t\t>> Feed-Backward...\n",
      "\t\t\t--> Optimizing Theta: 100.00%...\n",
      "  \t\t>>>------------>>>------------>>>\n",
      "  \t\t>> Feed-Forward...\n",
      "  \t\t>> Predictive uncertainty using entropy...\n",
      "  \t\t>> Compute cost...\n",
      "\t\t\t--> Calculating cost: 100.00%...\n",
      "\t\t\t--> New cost: 0.8616; Old cost: 0.8489\n",
      "\t\t\t--> Epoch 46 took 0.321 seconds...\n",
      "\t   47)- Epoch count (47/50)...\n",
      "  \t\t<<<------------<<<------------<<<\n",
      "  \t\t>> Feed-Backward...\n",
      "\t\t\t--> Optimizing Theta: 100.00%...\n",
      "  \t\t>>>------------>>>------------>>>\n",
      "  \t\t>> Feed-Forward...\n",
      "  \t\t>> Predictive uncertainty using entropy...\n",
      "  \t\t>> Compute cost...\n",
      "\t\t\t--> Calculating cost: 100.00%...\n",
      "\t\t\t--> New cost: 0.8604; Old cost: 0.8489\n",
      "\t\t\t--> Epoch 47 took 0.255 seconds...\n",
      "\t   48)- Epoch count (48/50)...\n",
      "  \t\t<<<------------<<<------------<<<\n",
      "  \t\t>> Feed-Backward...\n",
      "\t\t\t--> Optimizing Theta: 100.00%...\n",
      "  \t\t>>>------------>>>------------>>>\n",
      "  \t\t>> Feed-Forward...\n",
      "  \t\t>> Predictive uncertainty using entropy...0000%...\n",
      "  \t\t>> Compute cost...\n",
      "\t\t\t--> Calculating cost: 100.00%...\n",
      "\t\t\t--> New cost: 0.8604; Old cost: 0.8489\n",
      "\t\t\t--> Epoch 48 took 0.32 seconds...\n",
      "\t   49)- Epoch count (49/50)...\n",
      "  \t\t<<<------------<<<------------<<<\n",
      "  \t\t>> Feed-Backward...\n",
      "\t\t\t--> Optimizing Theta: 100.00%...\n",
      "  \t\t>>>------------>>>------------>>>\n",
      "  \t\t>> Feed-Forward...\n",
      "  \t\t>> Predictive uncertainty using entropy...0000%...\n",
      "  \t\t>> Compute cost...\n",
      "\t\t\t--> Calculating cost: 100.00%...\n",
      "\t\t\t--> New cost: 0.8613; Old cost: 0.8489\n",
      "\t\t\t--> Epoch 49 took 0.249 seconds...\n",
      "\t   50)- Epoch count (50/50)...\n",
      "  \t\t<<<------------<<<------------<<<\n",
      "  \t\t>> Feed-Backward...\n",
      "\t\t\t--> Optimizing Theta: 100.00%...\n",
      "  \t\t>>>------------>>>------------>>>\n",
      "  \t\t>> Feed-Forward...\n",
      "  \t\t>> Predictive uncertainty using entropy...\n",
      "  \t\t>> Compute cost...\n",
      "\t\t\t--> Calculating cost: 100.00%...\n",
      "\t\t\t--> New cost: 0.8621; Old cost: 0.8489\n",
      "\t\t\t--> Epoch 50 took 0.314 seconds...\n",
      "\t  --> Training consumed 0.32 mintues\n",
      "\t>> Perform splitting (extreme)...\n",
      "\t\t--> Starting score: 526.0\n",
      "\t\t--> Splitting progress: 100.00%; score: 488.44\n"
     ]
    }
   ],
   "source": [
    "st = ActiveStratification(subsample_labels_size=10, acquisition_type=\"entropy\", \n",
    "                          top_k=5, calc_ads=False, ads_percent=0.7, \n",
    "                          use_solver=use_solver, loss_function=\"hinge\", \n",
    "                          swap_probability=0.1, threshold_proportion=0.1, decay=0.1, \n",
    "                          penalty='elasticnet', alpha_elastic=0.0001, l1_ratio=0.65, \n",
    "                          alpha_l21=0.01, loss_threshold=0.05, shuffle=True,\n",
    "                          split_size=split_size, batch_size=batch_size, num_epochs=num_epochs, \n",
    "                          lr=1e-3, display_interval=1, num_jobs=num_jobs)\n",
    "training_idx, test_idx = st.fit(X=X, y=y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Properties for yeast</th>\n",
       "      <th>Complete set</th>\n",
       "      <th>Training set</th>\n",
       "      <th>Test set</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Number of examples</td>\n",
       "      <td>2417</td>\n",
       "      <td>1925</td>\n",
       "      <td>492</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Number of labels</td>\n",
       "      <td>10241</td>\n",
       "      <td>8120</td>\n",
       "      <td>2121</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Label cardinality</td>\n",
       "      <td>4.237071</td>\n",
       "      <td>4.218182</td>\n",
       "      <td>4.310976</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Label density</td>\n",
       "      <td>0.302648</td>\n",
       "      <td>0.301299</td>\n",
       "      <td>0.307927</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Distinct labels</td>\n",
       "      <td>14</td>\n",
       "      <td>14</td>\n",
       "      <td>14</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>Distinct label sets</td>\n",
       "      <td>198</td>\n",
       "      <td>172</td>\n",
       "      <td>113</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>Frequency of distinct label sets</td>\n",
       "      <td>0.08192</td>\n",
       "      <td>0.08192</td>\n",
       "      <td>0.08192</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>Mean imbalance ratio intra-class for all labels</td>\n",
       "      <td>9.578575</td>\n",
       "      <td>9.397859</td>\n",
       "      <td>11.079869</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>Mean imbalance ratio inter-class for all labels</td>\n",
       "      <td>7.196811</td>\n",
       "      <td>7.108199</td>\n",
       "      <td>8.107221</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>Mean imbalance ratio labelsets for all labels</td>\n",
       "      <td>0.468229</td>\n",
       "      <td>0.525651</td>\n",
       "      <td>0.749716</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>Labels having less than or equal to 5 examples</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>Labels having more than 6 examples</td>\n",
       "      <td>14</td>\n",
       "      <td>14</td>\n",
       "      <td>13</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>KL difference between complete and data partition</td>\n",
       "      <td>0</td>\n",
       "      <td>0.000216</td>\n",
       "      <td>0.003074</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                 Properties for yeast Complete set  \\\n",
       "0                                  Number of examples         2417   \n",
       "1                                    Number of labels        10241   \n",
       "2                                   Label cardinality     4.237071   \n",
       "3                                       Label density     0.302648   \n",
       "4                                     Distinct labels           14   \n",
       "5                                 Distinct label sets          198   \n",
       "6                    Frequency of distinct label sets      0.08192   \n",
       "7     Mean imbalance ratio intra-class for all labels     9.578575   \n",
       "8     Mean imbalance ratio inter-class for all labels     7.196811   \n",
       "9       Mean imbalance ratio labelsets for all labels     0.468229   \n",
       "10     Labels having less than or equal to 5 examples            0   \n",
       "11                 Labels having more than 6 examples           14   \n",
       "12  KL difference between complete and data partition            0   \n",
       "\n",
       "   Training set   Test set  \n",
       "0          1925        492  \n",
       "1          8120       2121  \n",
       "2      4.218182   4.310976  \n",
       "3      0.301299   0.307927  \n",
       "4            14         14  \n",
       "5           172        113  \n",
       "6       0.08192    0.08192  \n",
       "7      9.397859  11.079869  \n",
       "8      7.108199   8.107221  \n",
       "9      0.525651   0.749716  \n",
       "10            0          1  \n",
       "11           14         13  \n",
       "12     0.000216   0.003074  "
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model_name = \"active2split\"\n",
    "df, chart = data_properties(y=y, selected_examples=[training_idx, test_idx], \n",
    "                            num_tails=5, dataset_name=dsname, model_name=model_name, \n",
    "                            rspath=RESULT_PATH, display_dataframe=True, \n",
    "                            display_figure=True)\n",
    "df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "<div id=\"altair-viz-50a37a47fa6c4aa1b4fc8d3b073ef269\"></div>\n",
       "<script type=\"text/javascript\">\n",
       "  (function(spec, embedOpt){\n",
       "    let outputDiv = document.currentScript.previousElementSibling;\n",
       "    if (outputDiv.id !== \"altair-viz-50a37a47fa6c4aa1b4fc8d3b073ef269\") {\n",
       "      outputDiv = document.getElementById(\"altair-viz-50a37a47fa6c4aa1b4fc8d3b073ef269\");\n",
       "    }\n",
       "    const paths = {\n",
       "      \"vega\": \"https://cdn.jsdelivr.net/npm//vega@5?noext\",\n",
       "      \"vega-lib\": \"https://cdn.jsdelivr.net/npm//vega-lib?noext\",\n",
       "      \"vega-lite\": \"https://cdn.jsdelivr.net/npm//vega-lite@4.8.1?noext\",\n",
       "      \"vega-embed\": \"https://cdn.jsdelivr.net/npm//vega-embed@6?noext\",\n",
       "    };\n",
       "\n",
       "    function loadScript(lib) {\n",
       "      return new Promise(function(resolve, reject) {\n",
       "        var s = document.createElement('script');\n",
       "        s.src = paths[lib];\n",
       "        s.async = true;\n",
       "        s.onload = () => resolve(paths[lib]);\n",
       "        s.onerror = () => reject(`Error loading script: ${paths[lib]}`);\n",
       "        document.getElementsByTagName(\"head\")[0].appendChild(s);\n",
       "      });\n",
       "    }\n",
       "\n",
       "    function showError(err) {\n",
       "      outputDiv.innerHTML = `<div class=\"error\" style=\"color:red;\">${err}</div>`;\n",
       "      throw err;\n",
       "    }\n",
       "\n",
       "    function displayChart(vegaEmbed) {\n",
       "      vegaEmbed(outputDiv, spec, embedOpt)\n",
       "        .catch(err => showError(`Javascript Error: ${err.message}<br>This usually means there's a typo in your chart specification. See the javascript console for the full traceback.`));\n",
       "    }\n",
       "\n",
       "    if(typeof define === \"function\" && define.amd) {\n",
       "      requirejs.config({paths});\n",
       "      require([\"vega-embed\"], displayChart, err => showError(`Error loading script: ${err.message}`));\n",
       "    } else if (typeof vegaEmbed === \"function\") {\n",
       "      displayChart(vegaEmbed);\n",
       "    } else {\n",
       "      loadScript(\"vega\")\n",
       "        .then(() => loadScript(\"vega-lite\"))\n",
       "        .then(() => loadScript(\"vega-embed\"))\n",
       "        .catch(showError)\n",
       "        .then(() => displayChart(vegaEmbed));\n",
       "    }\n",
       "  })({\"data\": {\"name\": \"data-583e7bb18a5393d41904f7e05cddc588\"}, \"mark\": {\"type\": \"bar\", \"color\": \"grey\"}, \"config\": {\"axis\": {\"labelFontSize\": 12, \"labelLimit\": 500, \"labelPadding\": 5, \"titleFontSize\": 20}, \"axisY\": {\"grid\": false}, \"header\": {\"labelFontSize\": 15, \"titleFontSize\": 20}, \"legend\": {\"cornerRadius\": 10, \"fillColor\": \"white\", \"padding\": 10, \"strokeColor\": \"gray\"}}, \"encoding\": {\"color\": {\"type\": \"nominal\", \"field\": \"Dataset\", \"scale\": {\"range\": [\"red\", \"black\", \"blue\"]}}, \"x\": {\"type\": \"ordinal\", \"field\": \"Label\", \"sort\": \"ascending\", \"title\": \"Label ID\"}, \"y\": {\"type\": \"quantitative\", \"field\": \"Sum\", \"stack\": null, \"title\": \"Number of Examples\"}}, \"height\": 350, \"resolve\": {\"scale\": {\"x\": \"independent\"}}, \"title\": {\"text\": \"Number of examples for each label given yeast data.\", \"subtitle\": [\"The horizontal axis indicates the indices of labels while the vertical axis represents the number of associated examples\"]}, \"width\": 600, \"$schema\": \"https://vega.github.io/schema/vega-lite/v4.8.1.json\", \"datasets\": {\"data-583e7bb18a5393d41904f7e05cddc588\": [{\"Label\": 1, \"Dataset\": \"Complete\", \"Sum\": 34}, {\"Label\": 2, \"Dataset\": \"Complete\", \"Sum\": 178}, {\"Label\": 3, \"Dataset\": \"Complete\", \"Sum\": 253}, {\"Label\": 4, \"Dataset\": \"Complete\", \"Sum\": 289}, {\"Label\": 5, \"Dataset\": \"Complete\", \"Sum\": 428}, {\"Label\": 6, \"Dataset\": \"Complete\", \"Sum\": 480}, {\"Label\": 7, \"Dataset\": \"Complete\", \"Sum\": 597}, {\"Label\": 8, \"Dataset\": \"Complete\", \"Sum\": 722}, {\"Label\": 9, \"Dataset\": \"Complete\", \"Sum\": 762}, {\"Label\": 10, \"Dataset\": \"Complete\", \"Sum\": 862}, {\"Label\": 11, \"Dataset\": \"Complete\", \"Sum\": 983}, {\"Label\": 12, \"Dataset\": \"Complete\", \"Sum\": 1038}, {\"Label\": 13, \"Dataset\": \"Complete\", \"Sum\": 1799}, {\"Label\": 14, \"Dataset\": \"Complete\", \"Sum\": 1816}, {\"Label\": 1, \"Dataset\": \"Train\", \"Sum\": 29}, {\"Label\": 2, \"Dataset\": \"Train\", \"Sum\": 129}, {\"Label\": 3, \"Dataset\": \"Train\", \"Sum\": 196}, {\"Label\": 4, \"Dataset\": \"Train\", \"Sum\": 223}, {\"Label\": 5, \"Dataset\": \"Train\", \"Sum\": 340}, {\"Label\": 6, \"Dataset\": \"Train\", \"Sum\": 372}, {\"Label\": 7, \"Dataset\": \"Train\", \"Sum\": 461}, {\"Label\": 8, \"Dataset\": \"Train\", \"Sum\": 562}, {\"Label\": 9, \"Dataset\": \"Train\", \"Sum\": 625}, {\"Label\": 10, \"Dataset\": \"Train\", \"Sum\": 681}, {\"Label\": 11, \"Dataset\": \"Train\", \"Sum\": 771}, {\"Label\": 12, \"Dataset\": \"Train\", \"Sum\": 830}, {\"Label\": 13, \"Dataset\": \"Train\", \"Sum\": 1445}, {\"Label\": 14, \"Dataset\": \"Train\", \"Sum\": 1456}, {\"Label\": 1, \"Dataset\": \"Test\", \"Sum\": 5}, {\"Label\": 2, \"Dataset\": \"Test\", \"Sum\": 49}, {\"Label\": 3, \"Dataset\": \"Test\", \"Sum\": 57}, {\"Label\": 4, \"Dataset\": \"Test\", \"Sum\": 66}, {\"Label\": 5, \"Dataset\": \"Test\", \"Sum\": 88}, {\"Label\": 6, \"Dataset\": \"Test\", \"Sum\": 108}, {\"Label\": 7, \"Dataset\": \"Test\", \"Sum\": 136}, {\"Label\": 8, \"Dataset\": \"Test\", \"Sum\": 160}, {\"Label\": 9, \"Dataset\": \"Test\", \"Sum\": 137}, {\"Label\": 10, \"Dataset\": \"Test\", \"Sum\": 181}, {\"Label\": 11, \"Dataset\": \"Test\", \"Sum\": 212}, {\"Label\": 12, \"Dataset\": \"Test\", \"Sum\": 208}, {\"Label\": 13, \"Dataset\": \"Test\", \"Sum\": 354}, {\"Label\": 14, \"Dataset\": \"Test\", \"Sum\": 360}]}}, {\"mode\": \"vega-lite\"});\n",
       "</script>"
      ],
      "text/plain": [
       "alt.Chart(...)"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "chart"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we did before, we train the chain classifier to evaluate the performance of partitions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\t>> F1-score for training set...\n",
      "\t\t--> Average sample f1-score: 0.6465\n",
      "\t\t--> Micro f1-score: 0.6668\n",
      "\t\t--> Macro f1-score: 0.3997\n",
      "\t>> F1-score for test set...\n",
      "\t\t--> Average sample f1-score: 0.5778\n",
      "\t\t--> Micro f1-score: 0.6015\n",
      "\t\t--> Macro f1-score: 0.3328\n"
     ]
    }
   ],
   "source": [
    "# train\n",
    "model.fit(X[training_idx].toarray(), y[training_idx].toarray())\n",
    "# evaluate\n",
    "y_pred = model.predict(X[training_idx].toarray())\n",
    "score(y_true=y[training_idx], y_pred=y_pred, tag=\"training set\")\n",
    "\n",
    "y_pred = model.predict(X[test_idx].toarray())\n",
    "score(y_true=y[test_idx], y_pred=y_pred, tag=\"test set\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Given results from the table and F-score, it seems this approach was able to perform splitting much better than Statified XML."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ABGiHY6fE02b"
   },
   "source": [
    "## Next steps\n",
    "\n",
    "Examples provided here are constrained to specific settings to keep time complexity reasonable for this tutorial. Splitting and performance may be less optimum. Therefore, as a next step, you could try to:\n",
    "\n",
    "- improve algorithms or analyze results on a large number of multi-label data.\n",
    "- rerun algorithms using different configurations. For instance, you could try setting the `split_type` parameter to `\"iterative\"` or `\"naive\"` or use a range of split size values (`split_size` $\\in (0,1)$) and document performance results. If you choose to apply `comom2split.py` or `enhance2split.py` then **use a small scale data**.\n",
    "- apply different metrics in the `../model/mlmetrics.py` module to evaluate the results. As can be seen, one metric can not fully justify the performance of splitters.\n",
    "- use alternative classification models. For example, try [leADS](https://github.com/hallamlab/leADS). \n",
    "- use deep learning or Bayesian approach to splitting data?"
   ]
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "collapsed_sections": [],
   "name": "cyclegan.ipynb",
   "toc_visible": true
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.8"
  },
  "toc-autonumbering": false,
  "toc-showmarkdowntxt": true
 },
 "nbformat": 4,
 "nbformat_minor": 4
}