06b-cnns-transfer.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# The Fashion MNIST - transfer learning\n",
    "This is the last time we will see the **fashion MNIST dataset**. This time we will use a pretrained CNN to process the images. For every image we want to classify it into **10 distinct clothing categories**. We will use the first layers from a known CNN, **ResNet50** and only use 10% of the dataset to speed up processing.\n",
    "\n",
    "The **ResNet50** expects the input to have **3 colors**, but the Fashion MNIST dataset is only grayscale. We will \"fix\" this issue by copying the grayscale values as the colors Red, Green, Blue. These images will undboutably look strange, but this allows us to use the ResNet50 model. Furthermore, the minimal image input size is **32 by 32** pixels, and our images are 28 by 28. To fix this, we will resize our images. Thus, our training set will have dimensions `(6000, 32, 32, 1)`. Lastly, we will do our normal **Min/Max scaling** and map the labels to their **one-hot encoding representation**. \n",
    "\n",
    "We will then download the ResNet50 model and chop off the last layers. We will then construct a new model which uses the first layers of the ResNet50 as the input layer and **freeze** the weights, so we do not train them. After feeding the input to the ResNet50 we will use the computed representation as input to a **dense layer** and then to a **softmax layer**. This will allow us to output a 10 class probability distribution specific to our problem and train the CNN using **categorical_crossentropy**. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's load the data set and inspect the dimensions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "library(tensorflow)\n",
    "library(keras)\n",
    "library(abind)\n",
    "source(\"06-helpers.R\")\n",
    "\n",
    "use_multi_cpu(4L)\n",
    "\n",
    "data <- dataset_fashion_mnist()\n",
    "dim(data$train$x)\n",
    "length(data$train$y)\n",
    "dim(data$test$x)\n",
    "length(data$test$y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1.1 Reshape and subsampling\n",
    "For our purposes we will only work with a small number of examples this time, 6000 training examples and 1000 testing examples. Otherwise processing times might become too long. Our first step is to reshape the data, to add one extra dimension for the colors. We reshape a subsample of the data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "x_train_reshaped <- array_reshape(x = data$train$x[1:6000,,], dim = c(6000, 28, 28, 1))\n",
    "x_test_reshaped <- array_reshape(x = data$test$x[1:1000,,], dim = c(1000, 28, 28, 1))\n",
    "dim(x_train_reshaped)\n",
    "dim(x_test_reshaped)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1.2 Stacking/concatenating arrays\n",
    "Since we only have one color value now and the ResNet expects 3 colors, we will copy the color value we have two more times. We do this by stacking the original color values together. To understand what is happening, think of a book, the book represents a single black and white picture. Then duplicate the book until we have 3 copies. Then we glue them together to create one book which is three times thicker by stacking the books in order. This is effectively what we are doing.\n",
    "\n",
    "To do this we use the function `abind`. Run the code below to get an idea how this works."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "vector <- array(1:5, dim = c(5))\n",
    "dim(vector)\n",
    "vector\n",
    "long_vector <- abind(vector, vector)\n",
    "dim(long_vector)\n",
    "long_vector"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can see that we just stacked/concatinated this vector to create a longer vector. Let's now try stacking these vectors so that we add the other vector as a column."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "two_dimension <- array(1:5, dim = c(5, 1))\n",
    "dim(two_dimension)\n",
    "two_dimension\n",
    "two_dimension_duplicated <- abind(two_dimension, two_dimension)\n",
    "dim(two_dimension_duplicated)\n",
    "two_dimension_duplicated"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we added the vector as a new column. This then generalises to more dimensions, and in the cell below we just stack our reshaped data along the last axis."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dim(x_train_reshaped)\n",
    "x_train_stacked <- abind(x_train_reshaped, x_train_reshaped, x_train_reshaped)\n",
    "x_test_stacked <- abind(x_test_reshaped, x_test_reshaped, x_test_reshaped)\n",
    "dim(x_train_stacked)\n",
    "dim(x_test_stacked)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1.3 Resizing\n",
    "Now we need to make our images slightly larger by resizing them to 32 by 32 pixels. This is the minimal image size accepted. We use the `image_array_resize` function provided with Keras for this.\n",
    "\n",
    "This cell will take about 50 seconds to run."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Just to time the function\n",
    "start <- Sys.time()\n",
    "\n",
    "# Initialise our output arrays with the correct dimensions\n",
    "x_train_resized <- array(0, dim = c(6000, 32, 32, 3))\n",
    "x_test_resized <- array(0, dim = c(1000, 32, 32, 3))\n",
    "\n",
    "# We iterate through the image sequentially - poor performance\n",
    "for (image_index in 1:dim(x_train_stacked)[1]) {\n",
    "    x_train_resized[image_index,,,] <- image_array_resize(x_train_stacked[image_index,,,],\n",
    "                                                          32, 32, data_format = c(\"channels_last\"))\n",
    "}\n",
    "for (image_index in 1:dim(x_test_stacked)[1]) {\n",
    "    x_test_resized[image_index,,,] <- image_array_resize(x_test_stacked[image_index,,,],\n",
    "                                                          32, 32, data_format = c(\"channels_last\"))\n",
    "}\n",
    "\n",
    "end <- Sys.time()\n",
    "end - start\n",
    "# Check our work\n",
    "dim(x_train_resized)\n",
    "dim(x_test_resized)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1.4 Scaling\n",
    "Next we scale the image using the Min/Max approach like before. Since all our features are pixel values, we just divide each feature by 255."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "x_train <- x_train_resized/255\n",
    "x_test <- x_test_resized/255"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1.5 One-hot encoding\n",
    "Lastly, the one-hot encoding."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_train = to_categorical(y = data$train$y[1:6000], num_classes = 10)\n",
    "y_test = to_categorical(y = data$test$y[1:1000], num_classes = 10)\n",
    "dim(y_train)\n",
    "dim(y_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2.1 ResNet50\n",
    "Now we want to use the first layers of the ResNet50 model, pretrained, as feature extractors for our final model. We start by downloading the model description along with the weights. \n",
    "\n",
    "The ResNet50 model which we download has been trained with a known dataset called [ImageNet](http://www.image-net.org/). This is a very large dataset (14 million images) thus we can benefitted from someone else spending a long time training on this dataset. The model was trained in the multiclass classification task over 1000 classes. We do not care to classify our clothing images as these classes so we specify `include_top = FALSE`, to indicate that we will not use the last layer of the model, which is the softmax layer over 1000 classes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "start <- Sys.time()\n",
    "\n",
    "k_clear_session()\n",
    "resnet_50_model <- application_resnet50(weights = 'imagenet', include_top = FALSE, input_shape = c(32, 32, 3))\n",
    "\n",
    "end <- Sys.time()\n",
    "end - start"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let us check the layers on this model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "summary(resnet_50_model)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Wow! This is a very big model, 23 million parameters with lots of layers. In fact, processing our images through this network takes some time and we also expect the first features learned by the network to be beneficial to our problem so in the next cell we only use the layers up to the layer `activation_6`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "smaller_resnet_50 <- keras_model(inputs = resnet_50_model$input, \n",
    "                                 outputs = get_layer(resnet_50_model, 'activation_6')$output)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Furthermore, we do not want to train those layers. Let's make sure that we do not train all of these layers by freezing the weights."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "freeze_weights(smaller_resnet_50)\n",
    "summary(smaller_resnet_50)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can see that there are no trainable parameters in this model. Good!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2.2 Our model\n",
    "Let us add a dense layer and our own softmax layer to the `smaller_resnet_50` model and start training it.\n",
    "\n",
    "Take notice of\n",
    "- The first \"layer\" in our network is the `smaller_resnet_50` model. It takes the initial input and outputs some features.\n",
    "- The small learning rate. A small learning rate is very common when training with pretrained networks.\n",
    "- Long training time."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model <- keras_model_sequential() %>%\n",
    "    smaller_resnet_50 %>%\n",
    "    layer_flatten() %>%\n",
    "    layer_dense(units = 32, activation = \"relu\") %>%\n",
    "    layer_dense(units = 10, activation = \"softmax\")\n",
    "\n",
    "cat(summary(model))\n",
    "\n",
    "model %>% compile(\n",
    "    optimizer = optimizer_adam(lr = 0.0001),\n",
    "    loss = \"categorical_crossentropy\",\n",
    "    metric = \"acc\"\n",
    ")\n",
    "\n",
    "history <- model %>% fit(\n",
    "    x = x_train,\n",
    "    y = y_train,\n",
    "    validation_split = 0.2,\n",
    "    epochs = 20,\n",
    "    batch_size = 64,\n",
    "    callbacks=list(Progress$new())\n",
    ")\n",
    "\n",
    "plot(history)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "From the plot we can see that our model is fairly expressive and we could maybe continue to train it and seen some performance improvements but we leave this is a bonus exercise."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model %>% evaluate(x_test, y_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this notebook we saw how to use a pretrained CNN and how we could add our layers to the model so that we can benefit from the generic features learned by the model. Some notes about this notebook.\n",
    "- Our images are only grayscale, but the model assumed colors. We can expect this to have some negative (or at least wasted computation) on our performance.\n",
    "- We needed to resize our images in order for them to fit. We can also expect this to have negative impact on performance."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Bonus Excercise\n",
    "Try improving the model above. Consider the options below\n",
    "- Adding more layers from the ResNet50 model\n",
    "- Adding dropout.\n",
    "- Adding more dense layers/more dense units."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "R",
   "language": "R",
   "name": "ir"
  },
  "language_info": {
   "codemirror_mode": "r",
   "file_extension": ".r",
   "mimetype": "text/x-r-source",
   "name": "R",
   "pygments_lexer": "r",
   "version": "3.5.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}