notebooks/parameter_tuning_sol_03.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# \ud83d\udcc3 Solution for Exercise M3.02\n",
    "\n",
    "The goal is to find the best set of hyperparameters which maximize the\n",
    "generalization performance on a training set.\n",
    "\n",
    "Here again with limit the size of the training set to make computation\n",
    "run faster. Feel free to increase the `train_size` value if your computer\n",
    "is powerful enough."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "\n",
    "adult_census = pd.read_csv(\"../datasets/adult-census.csv\")\n",
    "\n",
    "target_name = \"class\"\n",
    "target = adult_census[target_name]\n",
    "data = adult_census.drop(columns=[target_name, \"education-num\"])\n",
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "data_train, data_test, target_train, target_test = train_test_split(\n",
    "    data, target, train_size=0.2, random_state=42)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lines_to_next_cell": 0
   },
   "source": [
    "In this exercise, we will progressively define the classification pipeline\n",
    "and later tune its hyperparameters.\n",
    "\n",
    "Our pipeline should:\n",
    "* preprocess the categorical columns using a `OneHotEncoder` and use a\n",
    "  `StandardScaler` to normalize the numerical data.\n",
    "* use a `LogisticRegression` as a predictive model.\n",
    "\n",
    "Start by defining the columns and the preprocessing pipelines to be applied\n",
    "on each group of columns."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.compose import make_column_selector as selector\n",
    "\n",
    "# solution\n",
    "categorical_columns_selector = selector(dtype_include=object)\n",
    "categorical_columns = categorical_columns_selector(data)\n",
    "\n",
    "numerical_columns_selector = selector(dtype_exclude=object)\n",
    "numerical_columns = numerical_columns_selector(data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.preprocessing import OneHotEncoder\n",
    "from sklearn.preprocessing import StandardScaler\n",
    "\n",
    "# solution\n",
    "categorical_processor = OneHotEncoder(handle_unknown=\"ignore\")\n",
    "numerical_processor = StandardScaler()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Subsequently, create a `ColumnTransformer` to redirect the specific columns\n",
    "a preprocessing pipeline."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.compose import ColumnTransformer\n",
    "\n",
    "# solution\n",
    "preprocessor = ColumnTransformer(\n",
    "    [('cat_preprocessor', categorical_processor, categorical_columns),\n",
    "     ('num_preprocessor', numerical_processor, numerical_columns)]\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Assemble the final pipeline by combining the above preprocessor\n",
    "with a logistic regression classifier. Force the maximum number of\n",
    "iterations to `10_000` to ensure that the model will converge."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.pipeline import make_pipeline\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "\n",
    "# solution\n",
    "model = make_pipeline(preprocessor, LogisticRegression(max_iter=10_000))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Use `RandomizedSearchCV` with `n_iter=20` to find the best set of\n",
    "hyperparameters by tuning the following parameters of the `model`:\n",
    "\n",
    "- the parameter `C` of the `LogisticRegression` with values ranging from\n",
    "  0.001 to 10. You can use a log-uniform distribution\n",
    "  (i.e. `scipy.stats.loguniform`);\n",
    "- the parameter `with_mean` of the `StandardScaler` with possible values\n",
    "  `True` or `False`;\n",
    "- the parameter `with_std` of the `StandardScaler` with possible values\n",
    "  `True` or `False`.\n",
    "\n",
    "Once the computation has completed, print the best combination of parameters\n",
    "stored in the `best_params_` attribute."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import RandomizedSearchCV\n",
    "from scipy.stats import loguniform\n",
    "\n",
    "# solution\n",
    "param_distributions = {\n",
    "    \"logisticregression__C\": loguniform(0.001, 10),\n",
    "    \"columntransformer__num_preprocessor__with_mean\": [True, False],\n",
    "    \"columntransformer__num_preprocessor__with_std\": [True, False],\n",
    "}\n",
    "\n",
    "model_random_search = RandomizedSearchCV(\n",
    "    model, param_distributions=param_distributions,\n",
    "    n_iter=20, error_score=np.nan, n_jobs=2, verbose=1, random_state=1)\n",
    "model_random_search.fit(data_train, target_train)\n",
    "model_random_search.best_params_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": [
     "solution"
    ]
   },
   "source": [
    "\n",
    "So the best hyperparameters give a model where the features are scaled but\n",
    "not centered and the final model is regularized.\n",
    "\n",
    "Getting the best parameter combinations is the main outcome of the\n",
    "hyper-parameter optimization procedure. However it is also interesting to\n",
    "assess the sensitivity of the best models to the choice of those parameters.\n",
    "The following code, not required to answer the quiz question shows how to\n",
    "conduct such an interactive analysis for this this pipeline using a parallel\n",
    "coordinate plot using the `plotly` library.\n",
    "\n",
    "We could use `cv_results = model_random_search.cv_results_` to make a\n",
    "parallel coordinate plot as we did in the previous notebook (you are more\n",
    "than welcome to try!). Instead we are going to load the results obtained from\n",
    "a similar search with many more iterations (1,000 instead of 20)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "solution"
    ]
   },
   "outputs": [],
   "source": [
    "cv_results = pd.read_csv(\n",
    "    \"../figures/randomized_search_results_logistic_regression.csv\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": [
     "solution"
    ]
   },
   "source": [
    "To simplify the axis of the plot, we will rename the column of the dataframe\n",
    "and only select the mean test score and the value of the hyperparameters."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "solution"
    ]
   },
   "outputs": [],
   "source": [
    "column_name_mapping = {\n",
    "    \"param_logisticregression__C\": \"C\",\n",
    "    \"param_columntransformer__num_preprocessor__with_mean\": \"centering\",\n",
    "    \"param_columntransformer__num_preprocessor__with_std\": \"scaling\",\n",
    "    \"mean_test_score\": \"mean test accuracy\",\n",
    "}\n",
    "\n",
    "cv_results = cv_results.rename(columns=column_name_mapping)\n",
    "cv_results = cv_results[column_name_mapping.values()].sort_values(\n",
    "    \"mean test accuracy\", ascending=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": [
     "solution"
    ]
   },
   "source": [
    "In addition, the parallel coordinate plot from `plotly` expects all data to\n",
    "be numeric. Thus, we convert the boolean indicator informing whether or not\n",
    "the data were centered or scaled into an integer, where True is mapped to 1\n",
    "and False is mapped to 0.\n",
    "\n",
    "We also take the logarithm of the `C` values to span the data on a broader\n",
    "range for a better visualization."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "solution"
    ]
   },
   "outputs": [],
   "source": [
    "column_scaler = [\"centering\", \"scaling\"]\n",
    "cv_results[column_scaler] = cv_results[column_scaler].astype(np.int64)\n",
    "cv_results['log C'] = np.log10(cv_results['C'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "solution"
    ]
   },
   "outputs": [],
   "source": [
    "import plotly.express as px\n",
    "\n",
    "fig = px.parallel_coordinates(\n",
    "    cv_results,\n",
    "    color=\"mean test accuracy\",\n",
    "    dimensions=[\"log C\", \"centering\", \"scaling\", \"mean test accuracy\"],\n",
    "    color_continuous_scale=px.colors.diverging.Tealrose,\n",
    ")\n",
    "fig.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": [
     "solution"
    ]
   },
   "source": [
    "We recall that it is possible to select a range of results by clicking and\n",
    "holding on any axis of the parallel coordinate plot. You can then slide\n",
    "(move) the range selection and cross two selections to see the intersections.\n",
    "\n",
    "Selecting the best performing models (i.e. above an accuracy of ~0.845), we\n",
    "observe the following pattern:\n",
    "\n",
    "- scaling the data is important. All the best performing models are scaling\n",
    "  the data;\n",
    "- centering the data does not have a strong impact. Both approaches,\n",
    "  centering and not centering, can lead to good models;\n",
    "- using some regularization is fine but using too much is a problem. Recall\n",
    "  that a smaller value of C means a stronger regularization. In particular\n",
    "  no pipeline with C lower than 0.001 can be found among the best\n",
    "  models."
   ]
  }
 ],
 "metadata": {
  "jupytext": {
   "main_language": "python"
  },
  "kernelspec": {
   "display_name": "Python 3",
   "name": "python3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}