tutorial_time_series_forecasting_with_xgboost.ipynb

{
  "metadata": {
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "name": "python",
      "version": "3.6.6",
      "mimetype": "text/x-python",
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "pygments_lexer": "ipython3",
      "nbconvert_exporter": "python",
      "file_extension": ".py"
    },
    "kaggle": {
      "accelerator": "none",
      "dataSources": [
        {
          "sourceId": 87794,
          "sourceType": "datasetVersion",
          "datasetId": 48149
        }
      ],
      "dockerImageVersionId": 13801,
      "isInternetEnabled": false,
      "language": "python",
      "sourceType": "notebook",
      "isGpuEnabled": false
    },
    "colab": {
      "provenance": [],
      "include_colab_link": true
    }
  },
  "nbformat_minor": 0,
  "nbformat": 4,
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/Shakilkhan24/Playground_DL/blob/main/tutorial_time_series_forecasting_with_xgboost.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# reference notebook for TIME SERIES ANALYSIS\n",
        "\n",
        "# COPIED FROM ROB MULLA , KAGGLE\n",
        "\n",
        "# ALL CREDITS FOR HIM ...."
      ],
      "metadata": {
        "id": "ONzEAkqNwYt3"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Hourly Time Series Forecasting using XGBoost\n",
        "\n",
        "[If you haven't already first check out my previous notebook forecasting on the same data using Prophet](https://www.kaggle.com/robikscube/hourly-time-series-forecasting-with-prophet)\n",
        "\n",
        "In this notebook we will walk through time series forecasting using XGBoost. The data we will be using is hourly energy consumption."
      ],
      "metadata": {
        "_uuid": "9bb54771ab0e8de3f40d12e7f47d3c86fe58645b",
        "id": "__Zztk7UwUQU"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import numpy as np # linear algebra\n",
        "import pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n",
        "import seaborn as sns\n",
        "import matplotlib.pyplot as plt\n",
        "import xgboost as xgb\n",
        "from xgboost import plot_importance, plot_tree\n",
        "from sklearn.metrics import mean_squared_error, mean_absolute_error\n",
        "plt.style.use('fivethirtyeight')"
      ],
      "metadata": {
        "_uuid": "8f2839f25d086af736a60e9eeb907d3b93b6e0e5",
        "_cell_guid": "b1076dfc-b9ad-4769-8c92-a6c4dae69d19",
        "trusted": true,
        "id": "70_2PVO6wUQc"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Data\n",
        "The data we will be using is hourly power consumption data from PJM. Energy consumtion has some unique charachteristics. It will be interesting to see how prophet picks them up.\n",
        "\n",
        "Pulling the `PJM East` which has data from 2002-2018 for the entire east region."
      ],
      "metadata": {
        "_uuid": "102c0bcb9cfc3be70922d8a308d4e29f02002c3e",
        "id": "Ag3YiK3SwUQg"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "pjme = pd.read_csv('../input/PJME_hourly.csv', index_col=[0], parse_dates=[0])"
      ],
      "metadata": {
        "_cell_guid": "79c7e3d0-c299-4dcb-8224-4455121ee9b0",
        "_uuid": "d629ff2d2480ee46fbb7e2d37f6b5fab8052498a",
        "trusted": true,
        "id": "mg9zWi2EwUQi"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "color_pal = [\"#F8766D\", \"#D39200\", \"#93AA00\", \"#00BA38\", \"#00C19F\", \"#00B9E3\", \"#619CFF\", \"#DB72FB\"]\n",
        "_ = pjme.plot(style='.', figsize=(15,5), color=color_pal[0], title='PJM East')"
      ],
      "metadata": {
        "_uuid": "c9ed25d7b8010577d04385c9c4077b6fffd7d409",
        "trusted": true,
        "id": "GbEjPurYwUQj"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Train/Test Split\n",
        "Cut off the data after 2015 to use as our validation set."
      ],
      "metadata": {
        "_uuid": "1c1c9cf0b6c70a8edd221810bb289b8837889288",
        "trusted": true,
        "id": "TvSkIpOUwUQk"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "split_date = '01-Jan-2015'\n",
        "pjme_train = pjme.loc[pjme.index <= split_date].copy()\n",
        "pjme_test = pjme.loc[pjme.index > split_date].copy()"
      ],
      "metadata": {
        "_uuid": "9e5c8c6fd78d2ac35c628315a7f12ad19c84dddb",
        "trusted": true,
        "id": "1iqXcLwxwUQl"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "_ = pjme_test \\\n",
        "    .rename(columns={'PJME_MW': 'TEST SET'}) \\\n",
        "    .join(pjme_train.rename(columns={'PJME_MW': 'TRAINING SET'}), how='outer') \\\n",
        "    .plot(figsize=(15,5), title='PJM East', style='.')"
      ],
      "metadata": {
        "_uuid": "e1a730acd8b68e569ce5ff1abe18b047ca85ab91",
        "trusted": true,
        "id": "qMFkBLM-wUQm"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Create Time Series Features"
      ],
      "metadata": {
        "_uuid": "a40373b87cfcbbfac9be205b3dbacf56bb9bc3d0",
        "id": "AeLC9TYqwUQn"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "def create_features(df, label=None):\n",
        "    \"\"\"\n",
        "    Creates time series features from datetime index\n",
        "    \"\"\"\n",
        "    df['date'] = df.index\n",
        "    df['hour'] = df['date'].dt.hour\n",
        "    df['dayofweek'] = df['date'].dt.dayofweek\n",
        "    df['quarter'] = df['date'].dt.quarter\n",
        "    df['month'] = df['date'].dt.month\n",
        "    df['year'] = df['date'].dt.year\n",
        "    df['dayofyear'] = df['date'].dt.dayofyear\n",
        "    df['dayofmonth'] = df['date'].dt.day\n",
        "    df['weekofyear'] = df['date'].dt.weekofyear\n",
        "\n",
        "    X = df[['hour','dayofweek','quarter','month','year',\n",
        "           'dayofyear','dayofmonth','weekofyear']]\n",
        "    if label:\n",
        "        y = df[label]\n",
        "        return X, y\n",
        "    return X"
      ],
      "metadata": {
        "_uuid": "53212882b070962fad91503aec7d2e550ac401c7",
        "trusted": true,
        "id": "1ale-13kwUQu"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "X_train, y_train = create_features(pjme_train, label='PJME_MW')\n",
        "X_test, y_test = create_features(pjme_test, label='PJME_MW')"
      ],
      "metadata": {
        "_uuid": "6bb80315201ecc19305b67e4ba8177bcbb9514b4",
        "trusted": true,
        "id": "QkuY8ju_wUQv"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Create XGBoost Model"
      ],
      "metadata": {
        "_uuid": "60ffc6c21d16280ee05faead916fd09c34fa490c",
        "id": "HJJwJJT8wUQw"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "reg = xgb.XGBRegressor(n_estimators=1000)\n",
        "reg.fit(X_train, y_train,\n",
        "        eval_set=[(X_train, y_train), (X_test, y_test)],\n",
        "        early_stopping_rounds=50,\n",
        "       verbose=False) # Change verbose to True if you want to see it train"
      ],
      "metadata": {
        "_uuid": "f608b0b457225f175c2eb1c4a74450747b095e93",
        "trusted": true,
        "id": "EiKFYHtkwUQx"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Feature Importances\n",
        "Feature importance is a great way to get a general idea about which features the model is relying on most to make the prediction. This is a metric that simply sums up how many times each feature is split on.\n",
        "\n",
        "We can see that the day of year was most commonly used to split trees, while hour and year came in next. Quarter has low importance due to the fact that it could be created by different dayofyear splits."
      ],
      "metadata": {
        "_uuid": "bf8d62f71d3037055bb267d283d58e510b2f58ff",
        "id": "e0RhaDYqwUQx"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "_ = plot_importance(reg, height=0.9)"
      ],
      "metadata": {
        "_uuid": "bf9e389874f0463e33fc844a4821b27f32353e2c",
        "trusted": true,
        "id": "7rM9MpLywUQy"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Forecast on Test Set"
      ],
      "metadata": {
        "_uuid": "7d16d64d0081f8ab9d5aff6970115f82a2e455a1",
        "id": "m0GDVXaTwUQz"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "pjme_test['MW_Prediction'] = reg.predict(X_test)\n",
        "pjme_all = pd.concat([pjme_test, pjme_train], sort=False)"
      ],
      "metadata": {
        "_uuid": "ab0386238f329132254a5d2fd1ab15cc05c2ce4e",
        "trusted": true,
        "id": "wjhNGAC2wUQ0"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "_ = pjme_all[['PJME_MW','MW_Prediction']].plot(figsize=(15, 5))"
      ],
      "metadata": {
        "_uuid": "46c23648943927364a6e35428ac15dbb52a3ee7b",
        "trusted": true,
        "id": "5J1WuLL8wUQ1"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Look at first month of predictions"
      ],
      "metadata": {
        "_uuid": "7b0c23f8ab69549bcf64c154a4ab6cb86c0fe204",
        "id": "O1RgZ8ZjwUQ2"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Plot the forecast with the actuals\n",
        "f, ax = plt.subplots(1)\n",
        "f.set_figheight(5)\n",
        "f.set_figwidth(15)\n",
        "_ = pjme_all[['MW_Prediction','PJME_MW']].plot(ax=ax,\n",
        "                                              style=['-','.'])\n",
        "ax.set_xbound(lower='01-01-2015', upper='02-01-2015')\n",
        "ax.set_ylim(0, 60000)\n",
        "plot = plt.suptitle('January 2015 Forecast vs Actuals')"
      ],
      "metadata": {
        "_uuid": "b07c99d3ceffa3634714f23ab3e27b7d1c6fb3e1",
        "trusted": true,
        "id": "r1rAVuIZwUQ2"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [],
      "metadata": {
        "_uuid": "7c46fb18df1f2a56cf2ab082814d890d34e54e4e",
        "id": "NazD3aA4wUQ3"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Plot the forecast with the actuals\n",
        "f, ax = plt.subplots(1)\n",
        "f.set_figheight(5)\n",
        "f.set_figwidth(15)\n",
        "_ = pjme_all[['MW_Prediction','PJME_MW']].plot(ax=ax,\n",
        "                                              style=['-','.'])\n",
        "ax.set_xbound(lower='01-01-2015', upper='01-08-2015')\n",
        "ax.set_ylim(0, 60000)\n",
        "plot = plt.suptitle('First Week of January Forecast vs Actuals')"
      ],
      "metadata": {
        "_uuid": "153d79e6a408c5705b5c4a19ef100449d0c38d8b",
        "trusted": true,
        "id": "GlBFy1cawUQ3"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "f, ax = plt.subplots(1)\n",
        "f.set_figheight(5)\n",
        "f.set_figwidth(15)\n",
        "_ = pjme_all[['MW_Prediction','PJME_MW']].plot(ax=ax,\n",
        "                                              style=['-','.'])\n",
        "ax.set_ylim(0, 60000)\n",
        "ax.set_xbound(lower='07-01-2015', upper='07-08-2015')\n",
        "plot = plt.suptitle('First Week of July Forecast vs Actuals')"
      ],
      "metadata": {
        "_uuid": "8e24f91a3a71acd6d16eb930381d5b9baabffae2",
        "trusted": true,
        "id": "SyQZGJrmwUQ4"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Error Metrics On Test Set\n",
        "Our RMSE error is 13780445  \n",
        "Our MAE error is 2848.89  \n",
        "Our MAPE error is 8.9%"
      ],
      "metadata": {
        "_uuid": "31964c9653b64c3e3f6fe48f5b3d7eb1987eaf84",
        "trusted": true,
        "id": "qllDQHLiwUQ4"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "mean_squared_error(y_true=pjme_test['PJME_MW'],\n",
        "                   y_pred=pjme_test['MW_Prediction'])"
      ],
      "metadata": {
        "_uuid": "baf33e07988ad1860ab3fee31817cba797110879",
        "trusted": true,
        "id": "-AsigySQwUQ5"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "mean_absolute_error(y_true=pjme_test['PJME_MW'],\n",
        "                   y_pred=pjme_test['MW_Prediction'])"
      ],
      "metadata": {
        "_uuid": "4943521452099c82efb7fd00da4305b457a52fc0",
        "trusted": true,
        "id": "zuMuIr8MwUQ5"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "I like using mean absolute percent error because it gives an easy to interperate percentage showing how off the predictions are.\n",
        "MAPE isn't included in sklearn so we need to use a custom function."
      ],
      "metadata": {
        "_uuid": "3bfa58efee593e8ea8a8accdcd981b6b9cde5146",
        "id": "k-s2eecSwUQ6"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "def mean_absolute_percentage_error(y_true, y_pred):\n",
        "    \"\"\"Calculates MAPE given y_true and y_pred\"\"\"\n",
        "    y_true, y_pred = np.array(y_true), np.array(y_pred)\n",
        "    return np.mean(np.abs((y_true - y_pred) / y_true)) * 100"
      ],
      "metadata": {
        "_uuid": "32fcb6d7905e2847a18e89e310b0a61bfc6c3321",
        "trusted": true,
        "id": "OAuJxJthwUQ6"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "mean_absolute_percentage_error(y_true=pjme_test['PJME_MW'],\n",
        "                   y_pred=pjme_test['MW_Prediction'])"
      ],
      "metadata": {
        "_uuid": "eb095dfe68dccdc10f545d2345e3ed10865102dd",
        "trusted": true,
        "id": "dquztIYSwUQ8"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Look at Worst and Best Predicted Days"
      ],
      "metadata": {
        "_uuid": "5f5225264a6ae80a24b4c2802eff73f576aa4867",
        "id": "IhfpXtIkwUQ8"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "pjme_test['error'] = pjme_test['PJME_MW'] - pjme_test['MW_Prediction']\n",
        "pjme_test['abs_error'] = pjme_test['error'].apply(np.abs)\n",
        "error_by_day = pjme_test.groupby(['year','month','dayofmonth']) \\\n",
        "    .mean()[['PJME_MW','MW_Prediction','error','abs_error']]"
      ],
      "metadata": {
        "_uuid": "fb1fcc67506951b7a9708cce9463f899b98a7b37",
        "trusted": true,
        "id": "Rnar9H8XwUQ8"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Over forecasted days\n",
        "error_by_day.sort_values('error', ascending=True).head(10)"
      ],
      "metadata": {
        "_uuid": "6364555e8bed275076f6f6ee93c8537da3741aba",
        "trusted": true,
        "id": "IfP5eQwQwUQ8"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Notice anything about the over forecasted days?\n",
        "- #1 worst day - July 4th, 2016 - is a holiday.\n",
        "- #3 worst day - December 25, 2015 - Christmas\n",
        "- #5 worst day - July 4th, 2016 - is a holiday.   \n",
        "Looks like our model may benefit from adding a holiday indicator."
      ],
      "metadata": {
        "_uuid": "adcc6c413889583c4ab195d3b9933827ff630762",
        "id": "aGHHjFk5wUQ9"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Worst absolute predicted days\n",
        "error_by_day.sort_values('abs_error', ascending=False).head(10)"
      ],
      "metadata": {
        "_uuid": "72ee8685aea837bc274cec93c2fabb108b6a107d",
        "trusted": true,
        "id": "nD0fQXx7wUQ-"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "The best predicted days seem to be a lot of october (not many holidays and mild weather) Also early may"
      ],
      "metadata": {
        "_uuid": "0f4c1097312214cb35781d1e692a50b9a4c75654",
        "id": "C15h1F58wUQ-"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Best predicted days\n",
        "error_by_day.sort_values('abs_error', ascending=True).head(10)"
      ],
      "metadata": {
        "_uuid": "faf0e5530e1f02fb82022813780562c42a911493",
        "trusted": true,
        "id": "3_t90EeywUQ_"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Plotting some best/worst predicted days"
      ],
      "metadata": {
        "_uuid": "0f78f528ca59a1e1cfb253ab69651149a5018b3b",
        "id": "qULBCH8mwUQ_"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "f, ax = plt.subplots(1)\n",
        "f.set_figheight(5)\n",
        "f.set_figwidth(10)\n",
        "_ = pjme_all[['MW_Prediction','PJME_MW']].plot(ax=ax,\n",
        "                                              style=['-','.'])\n",
        "ax.set_ylim(0, 60000)\n",
        "ax.set_xbound(lower='08-13-2016', upper='08-14-2016')\n",
        "plot = plt.suptitle('Aug 13, 2016 - Worst Predicted Day')"
      ],
      "metadata": {
        "_uuid": "fd9d1df9db6153690488549f11cfd32c87cce960",
        "trusted": true,
        "id": "7A_cyNaKwURA"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "This one is pretty impressive. SPOT ON!"
      ],
      "metadata": {
        "_uuid": "2ca961106be98ebbf703ba52b9d10a2d58ff8a56",
        "id": "05KqBKhgwURA"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "f, ax = plt.subplots(1)\n",
        "f.set_figheight(5)\n",
        "f.set_figwidth(10)\n",
        "_ = pjme_all[['MW_Prediction','PJME_MW']].plot(ax=ax,\n",
        "                                              style=['-','.'])\n",
        "ax.set_ylim(0, 60000)\n",
        "ax.set_xbound(lower='10-03-2016', upper='10-04-2016')\n",
        "plot = plt.suptitle('Oct 3, 2016 - Best Predicted Day')"
      ],
      "metadata": {
        "_uuid": "39eac1134b1278e7dd5f848ea03e7246d730b97f",
        "trusted": true,
        "id": "RTgP_9FfwURC"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "f, ax = plt.subplots(1)\n",
        "f.set_figheight(5)\n",
        "f.set_figwidth(10)\n",
        "_ = pjme_all[['MW_Prediction','PJME_MW']].plot(ax=ax,\n",
        "                                              style=['-','.'])\n",
        "ax.set_ylim(0, 60000)\n",
        "ax.set_xbound(lower='08-13-2016', upper='08-14-2016')\n",
        "plot = plt.suptitle('Aug 13, 2016 - Worst Predicted Day')"
      ],
      "metadata": {
        "_uuid": "b20a85a8224428cbcb9888c62fa466757b0e04f7",
        "trusted": true,
        "id": "Pr2yYtkNwURC"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Up next?\n",
        "- Add Lag variables\n",
        "- Add holiday indicators.\n",
        "- Add weather data source."
      ],
      "metadata": {
        "_uuid": "4cd7f2212c6ca87a2ffb2b8ea12a3581e4ee4027",
        "id": "Wn--Xv_jwURD"
      }
    }
  ]
}