DetailedWalkThrough3.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Appendix: Hyperparameter Optimization"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Unlike real like, all that matters in a Kaggle competition is the metric under consideration. Even if the metric improves by the most insignificant amount, it matters in a Kaggle competition. The way to squeeze out maximum performance from a model is hyperparameter optimization\n",
    "\n",
    "I was getting pretty good results with XGBoost. I now wanted to find the optimal hyperparameters. To do this, I decided to use [Bayesian hyperparameter optimization methods](https://en.wikipedia.org/wiki/Hyperparameter_optimization#Bayesian_optimization). After looking around for a bit, I couldn't find a readily available R package for hyperparameter optimization. So I decided to use a Python package called [hyperopt](https://github.com/hyperopt/hyperopt). \n",
    "\n",
    "The way I structured the optimization was to have a python wrapper to my R code. The python wrapper use [rpy2](http://rpy2.bitbucket.org/) to execute the R code."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Python Wrapper\n",
    "\n",
    "#### [Full source](hyperopt/xgbDirect-hyperopt.py)\n",
    "#### Code below cannot be executed. It is for illustrative purposes only"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Let's import some modules first\n",
    "\n",
    "```python\n",
    "import rpy2.robjects as ro\n",
    "import pandas as pd\n",
    "from rpy2.robjects import pandas2ri\n",
    "from hyperopt import fmin, tpe, hp\n",
    "```\n",
    "\n",
    "#### The function below is the objective function for the optimizer. Bayesian optimization is a black box optimizer. It will suggest new hyperparameters to explore based on the result obtained from the previous trials.\n",
    "\n",
    "```python\n",
    "def objective (params):\n",
    "    eta, max_depth, subsample, col_sample_bytree, min_child_weight, gamma = params\n",
    "\n",
    "#####Pass functions to R by generating a list\n",
    "    rParams = ro.ListVector ({\n",
    "        'eta': ro.FloatVector ([eta]),\n",
    "        'max_depth': ro.IntVector ([max_depth]),\n",
    "        'subsample': ro.FloatVector ([subsample]),\n",
    "        'col_sample_bytree': ro.FloatVector ([col_sample_bytree]),\n",
    "        'min_child_weight': ro.FloatVector ([min_child_weight]),\n",
    "        'gamma': ro.FloatVector ([gamma])})\n",
    "\n",
    "    ro.globalenv['params'] = rParams\n",
    "    rmse = ro.r('fitFunc (fitFormula, trainData, params)')\n",
    "    \n",
    "###Additional code to log the trial omitted for simplicity. \n",
    "###But all that portion does is to log the trial to a file\n",
    "```    \n",
    "\n",
    "#### Generate hyperparameter search space. The optimizer needs to know the bounds of the space to explore and the type of parameter\n",
    "\n",
    "```python\n",
    "space = (\n",
    "    hp.loguniform ('eta', -6, 0),\n",
    "    hp.randint ('max_depth', 6) + 2,\n",
    "    hp.uniform ('subsample', 0.5, 1),\n",
    "    hp.uniform ('col_sample_bytree', 0.5, 1),\n",
    "    hp.uniform ('min_child_weight', 0, 3), \n",
    "    hp.uniform ('gamma', 0, 3)\n",
    "    )\n",
    "    \n",
    "pandas2ri.activate()\n",
    "```\n",
    "\n",
    "#### Source R file with actual XGBoost code\n",
    "\n",
    "```python\n",
    "ro.r('source (\"xgbDirect-hyperopt.R\")')\n",
    "       \n",
    "best = fmin (objective,\n",
    "             space = space,\n",
    "             algo = tpe.suggest,\n",
    "             max_evals= 100)\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### R Program\n",
    "\n",
    "#### [Full Source](hyperopt/xgbDirect-hyperopt.R)"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "suppressMessages (library (caret))\n",
    "suppressMessages (library (lubridate))\n",
    "suppressMessages (library (dplyr))\n",
    "suppressMessages (library (xgboost))\n",
    "\n",
    "source (\"../support.R\")\n",
    "trainFile <- \"../data/train.csv\"\n",
    "trainCol <- \"registered\"\n",
    "\n",
    "#The function below performs the actual fit and returns the error metric (RMSLE).\n",
    "\n",
    "fitFunc <- function (fitFormula, trainData, params)\n",
    "{\n",
    "       \n",
    "        set.seed (4322)\n",
    "        fit <- xgb.cv(params, booster=\"gbtree\", objective=\"reg:linear\", eval_metric=\"rmse\", \n",
    "                      data=trainData, nround = 10000, nfold = 5, early.stop.round = 5,\n",
    "                      verbose=FALSE)\n",
    "        \n",
    "        return (fit$\\$$test.rmse.mean[length (fit$test.rmse.mean)])\n",
    "}\n",
    "\n",
    "train.df <- read.csv (trainFile)\n",
    "\n",
    "train.df <- formatData (train.df, logTransform = TRUE) %>% tbl_df()\n",
    "train.df$month <- factor (train.df$month)\n",
    "train.df$year <- factor (train.df$year)\n",
    "\n",
    "trainFormula <- paste0 (trainCol, \" ~ season + holiday + workingday + weather + temp + atemp +\",\n",
    "                        \"humidity + windspeed + year + month + wday + day + hour\")\n",
    "\n",
    "trainFormula <- as.formula (trainFormula)\n",
    "\n",
    "trainData <- xgb.DMatrix (model.matrix (trainFormula, train.df), label=train.df$registered)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "R",
   "language": "R",
   "name": "ir"
  },
  "language_info": {
   "codemirror_mode": "r",
   "file_extension": ".r",
   "mimetype": "text/x-r-source",
   "name": "R",
   "pygments_lexer": "r",
   "version": "3.2.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}