notebooks/linear_models_sol_05.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# \ud83d\udcc3 Solution for Exercise M4.05\n",
    "In the previous notebook we set `penalty=\"none\"` to disable regularization\n",
    "entirely. This parameter can also control the **type** of regularization to use,\n",
    "whereas the regularization **strength** is set using the parameter `C`.\n",
    "Setting`penalty=\"none\"` is equivalent to an infinitely large value of `C`.\n",
    "In this exercise, we ask you to train a logistic regression classifier using the\n",
    "`penalty=\"l2\"` regularization (which happens to be the default in scikit-learn)\n",
    "to find by yourself the effect of the parameter `C`.\n",
    "\n",
    "We will start by loading the dataset and create the helper function to show\n",
    "the decision separation as in the previous code."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"admonition note alert alert-info\">\n",
    "<p class=\"first admonition-title\" style=\"font-weight: bold;\">Note</p>\n",
    "<p class=\"last\">If you want a deeper overview regarding this dataset, you can refer to the\n",
    "Appendix - Datasets description section at the end of this MOOC.</p>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "\n",
    "penguins = pd.read_csv(\"../datasets/penguins_classification.csv\")\n",
    "# only keep the Adelie and Chinstrap classes\n",
    "penguins = penguins.set_index(\"Species\").loc[\n",
    "    [\"Adelie\", \"Chinstrap\"]].reset_index()\n",
    "\n",
    "culmen_columns = [\"Culmen Length (mm)\", \"Culmen Depth (mm)\"]\n",
    "target_column = \"Species\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "penguins_train, penguins_test = train_test_split(penguins, random_state=0)\n",
    "\n",
    "data_train = penguins_train[culmen_columns]\n",
    "data_test = penguins_test[culmen_columns]\n",
    "\n",
    "target_train = penguins_train[target_column]\n",
    "target_test = penguins_test[target_column]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First, let's create our predictive model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.pipeline import make_pipeline\n",
    "from sklearn.preprocessing import StandardScaler\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "\n",
    "logistic_regression = make_pipeline(\n",
    "    StandardScaler(), LogisticRegression(penalty=\"l2\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Given the following candidates for the `C` parameter, find out the impact of\n",
    "`C` on the classifier decision boundary. You can import the helper class with\n",
    "`from helpers.plotting import DecisionBoundaryDisplay` to plot the decision\n",
    "function boundary. Use the method `from_estimator` from this class."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "Cs = [0.01, 0.1, 1, 10]\n",
    "\n",
    "# solution\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns\n",
    "from helpers.plotting import DecisionBoundaryDisplay\n",
    "\n",
    "for C in Cs:\n",
    "    logistic_regression.set_params(logisticregression__C=C)\n",
    "    logistic_regression.fit(data_train, target_train)\n",
    "    accuracy = logistic_regression.score(data_test, target_test)\n",
    "\n",
    "    DecisionBoundaryDisplay.from_estimator(\n",
    "        logistic_regression,\n",
    "        data_test,\n",
    "        response_method=\"predict\",\n",
    "        cmap=\"RdBu_r\",\n",
    "        alpha=0.5,\n",
    "    )\n",
    "    sns.scatterplot(\n",
    "        data=penguins_test, x=culmen_columns[0], y=culmen_columns[1],\n",
    "        hue=target_column, palette=[\"tab:red\", \"tab:blue\"])\n",
    "    plt.legend(bbox_to_anchor=(1.05, 0.8), loc=\"upper left\")\n",
    "    plt.title(f\"C: {C} \\n Accuracy on the test set: {accuracy:.2f}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Look at the impact of the `C` hyperparameter on the magnitude of the weights."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# solution\n",
    "weights_ridge = []\n",
    "for C in Cs:\n",
    "    logistic_regression.set_params(logisticregression__C=C)\n",
    "    logistic_regression.fit(data_train, target_train)\n",
    "    coefs = logistic_regression[-1].coef_[0]\n",
    "    weights_ridge.append(pd.Series(coefs, index=culmen_columns))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "solution"
    ]
   },
   "outputs": [],
   "source": [
    "weights_ridge = pd.concat(\n",
    "    weights_ridge, axis=1, keys=[f\"C: {C}\" for C in Cs])\n",
    "weights_ridge.plot.barh()\n",
    "_ = plt.title(\"LogisticRegression weights depending of C\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": [
     "solution"
    ]
   },
   "source": [
    "We see that a small `C` will shrink the weights values toward zero. It means\n",
    "that a small `C` provides a more regularized model. Thus, `C` is the inverse\n",
    "of the `alpha` coefficient in the `Ridge` model.\n",
    "\n",
    "Besides, with a strong penalty (i.e. small `C` value), the weight of the\n",
    "feature \"Culmen Depth (mm)\" is almost zero. It explains why the decision\n",
    "separation in the plot is almost perpendicular to the \"Culmen Length (mm)\"\n",
    "feature."
   ]
  }
 ],
 "metadata": {
  "jupytext": {
   "main_language": "python"
  },
  "kernelspec": {
   "display_name": "Python 3",
   "name": "python3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}