notebooks/logistic_regression_non_linear.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Beyond linear separation in classification\n",
    "\n",
    "As we saw in the regression section, the linear classification model\n",
    "expects the data to be linearly separable. When this assumption does not\n",
    "hold, the model is not expressive enough to properly fit the data.\n",
    "Therefore, we need to apply the same tricks as in regression: feature\n",
    "augmentation (potentially using expert-knowledge) or using a\n",
    "kernel-based method.\n",
    "\n",
    "We will provide examples where we will use a kernel support vector machine\n",
    "to perform classification on some toy-datasets where it is impossible to\n",
    "find a perfect linear separation.\n",
    "\n",
    "We will generate a first dataset where the data are represented as two\n",
    "interlaced half circle. This dataset is generated using the function\n",
    "[`sklearn.datasets.make_moons`](https://scikit-learn.org/stable/modules/generated/sklearn.datasets.make_moons.html)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "from sklearn.datasets import make_moons\n",
    "\n",
    "feature_names = [\"Feature #0\", \"Features #1\"]\n",
    "target_name = \"class\"\n",
    "\n",
    "X, y = make_moons(n_samples=100, noise=0.13, random_state=42)\n",
    "\n",
    "# We store both the data and target in a dataframe to ease plotting\n",
    "moons = pd.DataFrame(np.concatenate([X, y[:, np.newaxis]], axis=1),\n",
    "                     columns=feature_names + [target_name])\n",
    "data_moons, target_moons = moons[feature_names], moons[target_name]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Since the dataset contains only two features, we can make a scatter plot to\n",
    "have a look at it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns\n",
    "\n",
    "sns.scatterplot(data=moons, x=feature_names[0], y=feature_names[1],\n",
    "                hue=target_moons, palette=[\"tab:red\", \"tab:blue\"])\n",
    "_ = plt.title(\"Illustration of the moons dataset\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "From the intuitions that we got by studying linear model, it should be\n",
    "obvious that a linear classifier will not be able to find a perfect decision\n",
    "function to separate the two classes.\n",
    "\n",
    "Let's try to see what is the decision boundary of such a linear classifier.\n",
    "We will create a predictive model by standardizing the dataset followed by\n",
    "a linear support vector machine classifier."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import sklearn\n",
    "sklearn.set_config(display=\"diagram\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.pipeline import make_pipeline\n",
    "from sklearn.preprocessing import StandardScaler\n",
    "from sklearn.svm import SVC\n",
    "\n",
    "linear_model = make_pipeline(StandardScaler(), SVC(kernel=\"linear\"))\n",
    "linear_model.fit(data_moons, target_moons)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"admonition warning alert alert-danger\">\n",
    "<p class=\"first admonition-title\" style=\"font-weight: bold;\">Warning</p>\n",
    "<p class=\"last\">Be aware that we fit and will check the boundary decision of the classifier\n",
    "on the same dataset without splitting the dataset into a training set and a\n",
    "testing set. While this is a bad practice, we use it for the sake of\n",
    "simplicity to depict the model behavior. Always use cross-validation when\n",
    "you want to assess the generalization performance of a machine-learning model.</p>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's check the decision boundary of such a linear model on this dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from helpers.plotting import DecisionBoundaryDisplay\n",
    "\n",
    "DecisionBoundaryDisplay.from_estimator(\n",
    "    linear_model, data_moons, response_method=\"predict\", cmap=\"RdBu\", alpha=0.5\n",
    ")\n",
    "sns.scatterplot(data=moons, x=feature_names[0], y=feature_names[1],\n",
    "                hue=target_moons, palette=[\"tab:red\", \"tab:blue\"])\n",
    "_ = plt.title(\"Decision boundary of a linear model\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As expected, a linear decision boundary is not enough flexible to split the\n",
    "two classes.\n",
    "\n",
    "To push this example to the limit, we will create another dataset where\n",
    "samples of a class will be surrounded by samples from the other class."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.datasets import make_gaussian_quantiles\n",
    "\n",
    "feature_names = [\"Feature #0\", \"Features #1\"]\n",
    "target_name = \"class\"\n",
    "\n",
    "X, y = make_gaussian_quantiles(\n",
    "    n_samples=100, n_features=2, n_classes=2, random_state=42)\n",
    "gauss = pd.DataFrame(np.concatenate([X, y[:, np.newaxis]], axis=1),\n",
    "                     columns=feature_names + [target_name])\n",
    "data_gauss, target_gauss = gauss[feature_names], gauss[target_name]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ax = sns.scatterplot(data=gauss, x=feature_names[0], y=feature_names[1],\n",
    "                     hue=target_gauss, palette=[\"tab:red\", \"tab:blue\"])\n",
    "_ = plt.title(\"Illustration of the Gaussian quantiles dataset\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here, this is even more obvious that a linear decision function is not\n",
    "adapted. We can check what decision function, a linear support vector machine\n",
    "will find."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "linear_model.fit(data_gauss, target_gauss)\n",
    "DecisionBoundaryDisplay.from_estimator(\n",
    "    linear_model, data_gauss, response_method=\"predict\", cmap=\"RdBu\", alpha=0.5\n",
    ")\n",
    "sns.scatterplot(data=gauss, x=feature_names[0], y=feature_names[1],\n",
    "                hue=target_gauss, palette=[\"tab:red\", \"tab:blue\"])\n",
    "_ = plt.title(\"Decision boundary of a linear model\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As expected, a linear separation cannot be used to separate the classes\n",
    "properly: the model will under-fit as it will make errors even on\n",
    "the training set.\n",
    "\n",
    "In the section about linear regression, we saw that we could use several\n",
    "tricks to make a linear model more flexible by augmenting features or\n",
    "using a kernel. Here, we will use the later solution by using a radial basis\n",
    "function (RBF) kernel together with a support vector machine classifier.\n",
    "\n",
    "We will repeat the two previous experiments and check the obtained decision\n",
    "function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "kernel_model = make_pipeline(StandardScaler(), SVC(kernel=\"rbf\", gamma=5))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "kernel_model.fit(data_moons, target_moons)\n",
    "DecisionBoundaryDisplay.from_estimator(\n",
    "    kernel_model, data_moons, response_method=\"predict\", cmap=\"RdBu\", alpha=0.5\n",
    ")\n",
    "sns.scatterplot(data=moons, x=feature_names[0], y=feature_names[1],\n",
    "                hue=target_moons, palette=[\"tab:red\", \"tab:blue\"])\n",
    "_ = plt.title(\"Decision boundary with a model using an RBF kernel\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We see that the decision boundary is not anymore a straight line. Indeed,\n",
    "an area is defined around the red samples and we could imagine that this\n",
    "classifier should be able to generalize on unseen data.\n",
    "\n",
    "Let's check the decision function on the second dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "kernel_model.fit(data_gauss, target_gauss)\n",
    "DecisionBoundaryDisplay.from_estimator(\n",
    "    kernel_model, data_gauss, response_method=\"predict\", cmap=\"RdBu\", alpha=0.5\n",
    ")\n",
    "ax = sns.scatterplot(data=gauss, x=feature_names[0], y=feature_names[1],\n",
    "                     hue=target_gauss, palette=[\"tab:red\", \"tab:blue\"])\n",
    "_ = plt.title(\"Decision boundary with a model using an RBF kernel\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We observe something similar than in the previous case. The decision function\n",
    "is more flexible and does not underfit anymore.\n",
    "\n",
    "Thus, kernel trick or feature expansion are the tricks to make a linear\n",
    "classifier more expressive, exactly as we saw in regression.\n",
    "\n",
    "Keep in mind that adding flexibility to a model can also risk increasing\n",
    "overfitting by making the decision function to be sensitive to individual\n",
    "(possibly noisy) data points of the training set. Here we can observe that\n",
    "the decision functions remain smooth enough to preserve good generalization.\n",
    "If you are curious, you can try to repeat the above experiment with\n",
    "`gamma=100` and look at the decision functions."
   ]
  }
 ],
 "metadata": {
  "jupytext": {
   "main_language": "python"
  },
  "kernelspec": {
   "display_name": "Python 3",
   "name": "python3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}