notebooks/logistic_regression.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Linear model for classification\n",
    "In regression, we saw that the target to be predicted was a continuous\n",
    "variable. In classification, this target will be discrete (e.g. categorical).\n",
    "\n",
    "We will go back to our penguin dataset. However, this time we will try to\n",
    "predict the penguin species using the culmen information. We will also\n",
    "simplify our classification problem by selecting only 2 of the penguin\n",
    "species to solve a binary classification problem."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"admonition note alert alert-info\">\n",
    "<p class=\"first admonition-title\" style=\"font-weight: bold;\">Note</p>\n",
    "<p class=\"last\">If you want a deeper overview regarding this dataset, you can refer to the\n",
    "Appendix - Datasets description section at the end of this MOOC.</p>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "\n",
    "penguins = pd.read_csv(\"../datasets/penguins_classification.csv\")\n",
    "\n",
    "# only keep the Adelie and Chinstrap classes\n",
    "penguins = penguins.set_index(\"Species\").loc[\n",
    "    [\"Adelie\", \"Chinstrap\"]].reset_index()\n",
    "culmen_columns = [\"Culmen Length (mm)\", \"Culmen Depth (mm)\"]\n",
    "target_column = \"Species\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can quickly start by visualizing the feature distribution by class:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "\n",
    "for feature_name in culmen_columns:\n",
    "    plt.figure()\n",
    "    # plot the histogram for each specie\n",
    "    penguins.groupby(\"Species\")[feature_name].plot.hist(alpha=0.5, legend=True)\n",
    "    plt.xlabel(feature_name)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can observe that we have quite a simple problem. When the culmen\n",
    "length increases, the probability that the penguin is a Chinstrap is closer\n",
    "to 1. However, the culmen depth is not helpful for predicting the penguin\n",
    "species.\n",
    "\n",
    "For model fitting, we will separate the target from the data and\n",
    "we will create a training and a testing set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "penguins_train, penguins_test = train_test_split(penguins, random_state=0)\n",
    "\n",
    "data_train = penguins_train[culmen_columns]\n",
    "data_test = penguins_test[culmen_columns]\n",
    "\n",
    "target_train = penguins_train[target_column]\n",
    "target_test = penguins_test[target_column]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "The linear regression that we previously saw will predict a continuous\n",
    "output. When the target is a binary outcome, one can use the logistic\n",
    "function to model the probability. This model is known as logistic\n",
    "regression.\n",
    "\n",
    "Scikit-learn provides the class `LogisticRegression` which implements this\n",
    "algorithm."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import sklearn\n",
    "sklearn.set_config(display=\"diagram\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.pipeline import make_pipeline\n",
    "from sklearn.preprocessing import StandardScaler\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "\n",
    "logistic_regression = make_pipeline(\n",
    "    StandardScaler(), LogisticRegression(penalty=\"none\")\n",
    ")\n",
    "logistic_regression.fit(data_train, target_train)\n",
    "accuracy = logistic_regression.score(data_test, target_test)\n",
    "print(f\"Accuracy on test set: {accuracy:.3f}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "Since we are dealing with a classification problem containing only 2\n",
    "features, it is then possible to observe the decision function boundary.\n",
    "The boundary is the rule used by our predictive model to affect a class label\n",
    "given the feature values of the sample.\n",
    "\n",
    "<div class=\"admonition note alert alert-info\">\n",
    "<p class=\"first admonition-title\" style=\"font-weight: bold;\">Note</p>\n",
    "<p class=\"last\">Here, we will use the class <tt class=\"docutils literal\">DecisionBoundaryDisplay</tt>. We provide this class\n",
    "to allow making plots of the decision function boundary in a 2 dimensional\n",
    "space. The implementation can be found <a class=\"reference external\" href=\"https://github.com/INRIA/scikit-learn-mooc/blob/main/python_scripts/helpers/plotting.py\">here</a>.\n",
    "This class is intended to be part of the <tt class=\"docutils literal\"><span class=\"pre\">scikit-learn</span></tt> package in the future\n",
    "as it is proposed in the following <a class=\"reference external\" href=\"https://github.com/scikit-learn/scikit-learn/pull/16061\">Pull-Request</a>.</p>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import seaborn as sns\n",
    "from helpers.plotting import DecisionBoundaryDisplay\n",
    "\n",
    "DecisionBoundaryDisplay.from_estimator(\n",
    "    logistic_regression, data_test, response_method=\"predict\", cmap=\"RdBu_r\", alpha=0.5\n",
    ")\n",
    "sns.scatterplot(\n",
    "    data=penguins_test, x=culmen_columns[0], y=culmen_columns[1],\n",
    "    hue=target_column, palette=[\"tab:red\", \"tab:blue\"])\n",
    "_ = plt.title(\"Decision boundary of the trained\\n LogisticRegression\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Thus, we see that our decision function is represented by a line separating\n",
    "the 2 classes. We should also note that we did not impose any regularization\n",
    "by setting the parameter `penalty` to `'none'`.\n",
    "\n",
    "Since the line is oblique, it means that we used a combination of both\n",
    "features:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "coefs = logistic_regression[-1].coef_[0]  # the coefficients is a 2d array\n",
    "weights = pd.Series(coefs, index=culmen_columns)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "weights.plot.barh()\n",
    "_ = plt.title(\"Weights of the logistic regression\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Indeed, both coefficients are non-null."
   ]
  }
 ],
 "metadata": {
  "jupytext": {
   "main_language": "python"
  },
  "kernelspec": {
   "display_name": "Python 3",
   "name": "python3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}