BatchNorm_experimental-v1.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#### Author: Sameer Kesava"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Examining the Batch Normalization method\n",
    "\n",
    "    * From the paper \"Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift\" by\n",
    "      Sergey Ioffe and Christian Szegedy\n",
    "      \n",
    "    * Coding the method using keras API layers and custom layers in tensorflow 2.1.0 to understand the math and comparing against the API\n",
    "    \n",
    "    * Using MNIST data set\n",
    "\n",
    "    * This document uses only the existing Keras BatchNorm layers. v2 will use layers with batch normalization explicitly coded.\n",
    "\n",
    "    * From analysis, it appears that during training, the mini-batch mean and variance are used in the calculation of batch normalization,\n",
    "      and not the moving mean and variance.\n",
    "\n",
    "    * The definition of the arguments 'training' and 'trainable' need to be differentiated, can be confusing if not understood."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import tensorflow as tf"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": "'2.1.0'"
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tf.__version__"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 341,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": "'2.2.4-tf'"
     },
     "execution_count": 341,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tf.keras.__version__"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 144,
   "metadata": {},
   "outputs": [],
   "source": [
    "# garbage collection clear memory\n",
    "import gc"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Loading MNIST data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "(x_train, y_train), (x_test, y_test) = tf.keras.datasets.mnist.load_data()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": "(60000, 28, 28) (10000, 28, 28)\n"
    }
   ],
   "source": [
    "print(x_train.shape, x_test.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Checking min and max values of the input"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": "255"
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.max(x_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": "0"
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.min(x_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Scaling the data using the Standard Scaler"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.preprocessing import StandardScaler"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "stdscaler = StandardScaler(with_mean=True, with_std=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "stdscaler_fit = stdscaler.fit(x_train.reshape(-1, 28*28))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": "(784,)"
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "stdscaler_fit.mean_.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": "5.772211140440891"
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# average mean\n",
    "np.sqrt(stdscaler_fit.mean_.mean())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": "60000"
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "stdscaler_fit.n_samples_seen_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": "(784,)"
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "stdscaler_fit.scale_.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": "66.12879201995706"
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# average std\n",
    "np.sqrt(stdscaler_fit.var_.mean())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": "(60000, 28, 28)"
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x_train_scaled =  stdscaler_fit.transform(x_train.reshape(-1,28*28)).reshape(-1,28,28)\n",
    "x_train_scaled.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": "(10000, 28, 28)"
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x_test_scaled =  stdscaler_fit.transform(x_test.reshape(-1,28*28)).reshape(-1,28,28)\n",
    "x_test_scaled.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Checking the min and max values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": "-1.2742078920822268"
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.min(x_train_scaled)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": "244.94693302873063"
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.max(x_train_scaled)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": "0.9145408163265558"
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.var(x_train_scaled)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": "-2.1974863349995617e-18"
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.mean(x_train_scaled)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### One-hot encoding the y labels"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": "(array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9], dtype=uint8),\n array([5923, 6742, 5958, 6131, 5842, 5421, 5918, 6265, 5851, 5949]))"
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.unique(y_train, return_counts=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "    * The labels are more or less balanced"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": "(array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9], dtype=uint8),\n array([ 980, 1135, 1032, 1010,  982,  892,  958, 1028,  974, 1009]))"
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.unique(y_test, return_counts=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": "(60000, 10)"
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_train_coded =  tf.keras.utils.to_categorical(y_train, num_classes=10)\n",
    "y_train_coded.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": "Label:  5 \n One-hot encoded:  [0. 0. 0. 0. 0. 1. 0. 0. 0. 0.]\n"
    }
   ],
   "source": [
    "print(\"Label: \", y_train[0],'\\n',\"One-hot encoded: \", y_train_coded[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": "(10000, 10)"
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_test_coded =  tf.keras.utils.to_categorical(y_test, num_classes=10)\n",
    "y_test_coded.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Creating validation data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.utils import shuffle"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "# for reproducibility\n",
    "random_seed = 100 "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": "(60000, 28, 28)"
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x_train_scaled.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": "(60000, 10)"
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_train_coded.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [],
   "source": [
    "x_train_scaled, y_train_coded = shuffle(x_train_scaled, y_train_coded, random_state = random_seed) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": "(60000, 28, 28)"
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x_train_scaled.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": "(60000, 10)"
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_train_coded.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [],
   "source": [
    "x_valid_scaled = x_train_scaled[:5000]\n",
    "y_valid_coded = y_train_coded[:5000]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": "(5000, 28, 28)"
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x_valid_scaled.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": "(5000, 10)"
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_valid_coded.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [],
   "source": [
    "x_train_scaled = x_train_scaled[5000:]\n",
    "y_train_coded = y_train_coded[5000:]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Creating a tf dataset for training on a Model with dense layers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_dataset = tf.data.Dataset.from_tensor_slices((x_train_scaled.reshape(-1, 784), y_train_coded))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [],
   "source": [
    "valid_dataset = tf.data.Dataset.from_tensor_slices((x_valid_scaled.reshape(-1, 784), y_valid_coded))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [],
   "source": [
    "test_dataset = tf.data.Dataset.from_tensor_slices((x_test_scaled.reshape(-1,784), y_test_coded))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [],
   "source": [
    "minibatch =  60 # in the paper"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": "60000"
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "buffer_size = len(y_train)\n",
    "buffer_size"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [],
   "source": [
    "# shuffle first, batch 2nd, then prefetch\n",
    "train_dataset = train_dataset.shuffle(buffer_size=buffer_size, seed=random_seed, \n",
    "                                      reshuffle_each_iteration=True).batch(batch_size=minibatch, \n",
    "                                                                           drop_remainder=True).prefetch(buffer_size=tf.data.experimental.AUTOTUNE)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [],
   "source": [
    "valid_dataset = valid_dataset.shuffle(buffer_size=buffer_size, seed=random_seed, \n",
    "                                      reshuffle_each_iteration=False).batch(batch_size=minibatch).prefetch(buffer_size=tf.data.experimental.AUTOTUNE)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [],
   "source": [
    "test_dataset = test_dataset.shuffle(buffer_size=buffer_size, seed=random_seed, \n",
    "                                      reshuffle_each_iteration=False).batch(batch_size=minibatch).prefetch(buffer_size=tf.data.experimental.AUTOTUNE)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": "(784,) \n\ntf.Tensor([0. 1. 0. 0. 0. 0. 0. 0. 0. 0.], shape=(10,), dtype=float32)\n"
    }
   ],
   "source": [
    "for i in test_dataset.take(1):\n",
    "\n",
    "    print(i[0][0].shape, '\\n')\n",
    "    print(i[1][0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Building a Model using Dense Layers and Batch Norm from Keras"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "    * Using the model architecture from the paper"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [],
   "source": [
    "tf.keras.backend.clear_session()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [],
   "source": [
    "input_layer = tf.keras.Input(shape = (784,), name = 'input')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [],
   "source": [
    "units1, units2, units3 = 100,100,100"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [],
   "source": [
    "kerasdense1 = tf.keras.layers.Dense(units = units1, activation=None, \n",
    "                                    kernel_initializer=tf.keras.initializers.Orthogonal(gain=1,seed=random_seed))(input_layer)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [],
   "source": [
    "# mean and variance are calculated for the minibatch.\n",
    "# setting momentum = 0 for now. Population statistics will be calculated separately for inference.\n",
    "# Initializing moving mean and variance using the statistics from the StandardScaler fit\n",
    "\n",
    "# \"\"\"bn_layer = tf.keras.layers.BatchNormalization(axis = [-1], momentum=0.0, epsilon=0.001, center=True, scale=True, \n",
    "#                                              beta_initializer = tf.keras.initializers.zeros(), \n",
    "#                                               gamma_initializer = tf.keras.initializers.ones(), \n",
    "#                                              moving_mean_initializer = tf.keras.initializers.zeros(), \n",
    "#                                              moving_variance_initializer = tf.keras.initializers.ones(), trainable=True)\"\"\"\n",
    "# the \"bn_layer\" object can be used only once. Have to write a new batch_norm for every layer. Hence, will use a function"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [],
   "source": [
    "def bn_layer(axis = [-1]):\n",
    "    \n",
    "    \"\"\"\n",
    "    returns a batch norm layer\n",
    "    \n",
    "    Parameters:\n",
    "    axis: list of integers. Default is [-1] which is confusing because the mean and averages are\n",
    "    calculated across the minibatch rather than the features.   \n",
    "    \n",
    "    Anyways, will be cross-checking against custom code later\n",
    "    \n",
    "    \"\"\"\n",
    "    \n",
    "    \n",
    "    return tf.keras.layers.BatchNormalization(axis = axis, momentum=0.99, epsilon=0.001, center=True, scale=True, \n",
    "                                             beta_initializer = tf.keras.initializers.zeros(), \n",
    "                                              gamma_initializer = tf.keras.initializers.ones(), \n",
    "                                             moving_mean_initializer = tf.keras.initializers.zeros(), \n",
    "                                             moving_variance_initializer = tf.keras.initializers.ones(), trainable=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [],
   "source": [
    "kerasdensebn1 = bn_layer(axis = [-1])(kerasdense1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [],
   "source": [
    "activation1 = tf.keras.layers.Activation(activation = tf.nn.tanh)(kerasdensebn1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [],
   "source": [
    "kerasdense2 = tf.keras.layers.Dense(units = units2, activation=None, \n",
    "                                    kernel_initializer=tf.keras.initializers.Orthogonal(gain=1,seed=random_seed))(activation1)\n",
    "kerasdensebn2 = bn_layer(axis = [-1])(kerasdense2, training=True)\n",
    "\n",
    "activation2 = tf.keras.layers.Activation(activation = tf.nn.tanh)(kerasdensebn2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [],
   "source": [
    "kerasdense3 = tf.keras.layers.Dense(units = units3, activation=None, \n",
    "                                    kernel_initializer=tf.keras.initializers.Orthogonal(gain=1,seed=random_seed))(activation2)\n",
    "kerasdensebn3 = bn_layer(axis = [-1])(kerasdense3, training=True)\n",
    "activation3 = tf.keras.layers.Activation(activation = tf.nn.tanh)(kerasdensebn3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [],
   "source": [
    "output_layer = tf.keras.layers.Dense(units = 10, activation=None,\n",
    "                                    kernel_initializer=tf.keras.initializers.Orthogonal(gain=1,seed=random_seed))(activation3)\n",
    "# no softmax activation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [],
   "source": [
    "keras_fc_model = tf.keras.Model(inputs = [input_layer], outputs = [output_layer], name = 'kerasmodel1')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": "Model: \"kerasmodel1\"\n_________________________________________________________________\nLayer (type)                 Output Shape              Param #   \n=================================================================\ninput (InputLayer)           [(None, 784)]             0         \n_________________________________________________________________\ndense (Dense)                (None, 100)               78500     \n_________________________________________________________________\nbatch_normalization (BatchNo (None, 100)               400       \n_________________________________________________________________\nactivation (Activation)      (None, 100)               0         \n_________________________________________________________________\ndense_1 (Dense)              (None, 100)               10100     \n_________________________________________________________________\nbatch_normalization_1 (Batch (None, 100)               400       \n_________________________________________________________________\nactivation_1 (Activation)    (None, 100)               0         \n_________________________________________________________________\ndense_2 (Dense)              (None, 100)               10100     \n_________________________________________________________________\nbatch_normalization_2 (Batch (None, 100)               400       \n_________________________________________________________________\nactivation_2 (Activation)    (None, 100)               0         \n_________________________________________________________________\ndense_3 (Dense)              (None, 10)                1010      \n=================================================================\nTotal params: 100,910\nTrainable params: 100,310\nNon-trainable params: 600\n_________________________________________________________________\n"
    }
   ],
   "source": [
    "keras_fc_model.summary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 346,
   "metadata": {},
   "outputs": [],
   "source": [
    "# saving model to csv\n",
    "model_config = keras_fc_model.get_config()\n",
    "with open('Issues/test.csv', 'w') as f:\n",
    "    for key in model_config.keys():\n",
    "        f.write(\"%s,%s\\n\"%(key,model_config[key]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 348,
   "metadata": {},
   "outputs": [],
   "source": [
    "import json"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 349,
   "metadata": {},
   "outputs": [],
   "source": [
    "# saving as json config\n",
    "model_config_json = keras_fc_model.to_json()\n",
    "with open('Issues/model.json', 'w') as f:\n",
    "    json.dump(model_config_json, f)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": "[]"
     },
     "execution_count": 60,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "keras_fc_model.losses"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": "20"
     },
     "execution_count": 61,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(keras_fc_model.weights)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": "[<tf.Variable 'batch_normalization/gamma:0' shape=(100,) dtype=float32, numpy=\n array([1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,\n        1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,\n        1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,\n        1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,\n        1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,\n        1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.],\n       dtype=float32)>,\n <tf.Variable 'batch_normalization/beta:0' shape=(100,) dtype=float32, numpy=\n array([0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n        0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n        0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n        0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n        0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n        0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],\n       dtype=float32)>,\n <tf.Variable 'batch_normalization/moving_mean:0' shape=(100,) dtype=float32, numpy=\n array([0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n        0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n        0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n        0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n        0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n        0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],\n       dtype=float32)>,\n <tf.Variable 'batch_normalization/moving_variance:0' shape=(100,) dtype=float32, numpy=\n array([1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,\n        1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,\n        1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,\n        1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,\n        1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,\n        1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.],\n       dtype=float32)>]"
     },
     "execution_count": 62,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# batch norm layer variables\n",
    "keras_fc_model.layers[2].weights"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Compiling"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [],
   "source": [
    "keras_fc_model.compile(optimizer =  tf.keras.optimizers.Adam(learning_rate=0.001), \n",
    "                      loss = tf.keras.losses.CategoricalCrossentropy(from_logits=True), \n",
    "                      metrics = [tf.keras.metrics.CategoricalAccuracy()])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [],
   "source": [
    "# USing tf.keras.metrics.AUC() give:\n",
    "\n",
    "# \"\"\"InvalidArgumentError:  assertion failed: [predictions must be >= 0] [Condition x >= y did not hold element-wise:] \n",
    "# [x (kerasmodel1/dense_3/BiasAdd:0) = ] [[0.7673769 2.2196033 1.78565049...]...] [y (metrics/auc/Cast_1/x:0) = ] [0]\n",
    "#\t [[{{node metrics/auc/assert_greater_equal/Assert/AssertGuard/else/_1/Assert}}]] [Op:__inference_distributed_function_103831]\"\"\"\n",
    "\n",
    "# To use this metric, the predictions have to be between 0 and 1, implies softmax needs to be used.\n",
    "                      \n",
    "   "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": "'kerasmodel1'"
     },
     "execution_count": 65,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "keras_fc_model.name"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Callbacks"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Learning rate\n",
    "    * Adam optimizer can do adaptive learning rate feature"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Not setting EarlyStopping"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Save best model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {},
   "outputs": [],
   "source": [
    "keras_savedmodels = 'keras_savedmodels'\n",
    "\n",
    "if os.path.exists(keras_savedmodels):\n",
    "    pass\n",
    "else:\n",
    "    os.mkdir(keras_savedmodels)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {},
   "outputs": [],
   "source": [
    "cb_savemodel = tf.keras.callbacks.ModelCheckpoint(os.path.join(keras_savedmodels, 'model_{epoch}-{val_loss:.3f}.h5'), mode = 'min',\n",
    "                                                  monitor = 'val_loss',\n",
    "                                                 verbose = 1, save_best_only=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Tensorboard"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {},
   "outputs": [],
   "source": [
    "keras_models_logs = 'keras_models_logs'\n",
    "\n",
    "if os.path.exists(keras_models_logs):\n",
    "    pass\n",
    "else:\n",
    "    os.mkdir(keras_models_logs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {},
   "outputs": [],
   "source": [
    "cb_tboard = tf.keras.callbacks.TensorBoard(log_dir=keras_models_logs,histogram_freq=1, write_graph=True,\n",
    "                                          write_images=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {},
   "outputs": [],
   "source": [
    "cblist = [cb_savemodel, cb_tboard]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Fitting"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {},
   "outputs": [],
   "source": [
    "epochs1=10"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": "Train for 916 steps, validate for 84 steps\nEpoch 1/10\n910/916 [============================>.] - ETA: 0s - loss: 0.3041 - categorical_accuracy: 0.9111\nEpoch 00001: val_loss improved from inf to 0.21771, saving model to keras_savedmodels/model_1-0.218.h5\n916/916 [==============================] - 7s 8ms/step - loss: 0.3034 - categorical_accuracy: 0.9113 - val_loss: 0.2177 - val_categorical_accuracy: 0.9344\nEpoch 2/10\n910/916 [============================>.] - ETA: 0s - loss: 0.1610 - categorical_accuracy: 0.9516\nEpoch 00002: val_loss improved from 0.21771 to 0.16106, saving model to keras_savedmodels/model_2-0.161.h5\n916/916 [==============================] - 5s 5ms/step - loss: 0.1611 - categorical_accuracy: 0.9515 - val_loss: 0.1611 - val_categorical_accuracy: 0.9526\nEpoch 3/10\n907/916 [============================>.] - ETA: 0s - loss: 0.1176 - categorical_accuracy: 0.9635\nEpoch 00003: val_loss improved from 0.16106 to 0.15292, saving model to keras_savedmodels/model_3-0.153.h5\n916/916 [==============================] - 5s 5ms/step - loss: 0.1173 - categorical_accuracy: 0.9636 - val_loss: 0.1529 - val_categorical_accuracy: 0.9540\nEpoch 4/10\n912/916 [============================>.] - ETA: 0s - loss: 0.0904 - categorical_accuracy: 0.9716\nEpoch 00004: val_loss improved from 0.15292 to 0.13310, saving model to keras_savedmodels/model_4-0.133.h5\n916/916 [==============================] - 5s 5ms/step - loss: 0.0904 - categorical_accuracy: 0.9715 - val_loss: 0.1331 - val_categorical_accuracy: 0.9606\nEpoch 5/10\n914/916 [============================>.] - ETA: 0s - loss: 0.0745 - categorical_accuracy: 0.9764\nEpoch 00005: val_loss improved from 0.13310 to 0.13114, saving model to keras_savedmodels/model_5-0.131.h5\n916/916 [==============================] - 5s 5ms/step - loss: 0.0746 - categorical_accuracy: 0.9763 - val_loss: 0.1311 - val_categorical_accuracy: 0.9640\nEpoch 6/10\n912/916 [============================>.] - ETA: 0s - loss: 0.0608 - categorical_accuracy: 0.9806\nEpoch 00006: val_loss improved from 0.13114 to 0.12137, saving model to keras_savedmodels/model_6-0.121.h5\n916/916 [==============================] - 5s 5ms/step - loss: 0.0610 - categorical_accuracy: 0.9805 - val_loss: 0.1214 - val_categorical_accuracy: 0.9652\nEpoch 7/10\n914/916 [============================>.] - ETA: 0s - loss: 0.0546 - categorical_accuracy: 0.9821\nEpoch 00007: val_loss did not improve from 0.12137\n916/916 [==============================] - 5s 5ms/step - loss: 0.0545 - categorical_accuracy: 0.9822 - val_loss: 0.1346 - val_categorical_accuracy: 0.9658\nEpoch 8/10\n913/916 [============================>.] - ETA: 0s - loss: 0.0463 - categorical_accuracy: 0.9848\nEpoch 00008: val_loss did not improve from 0.12137\n916/916 [==============================] - 5s 5ms/step - loss: 0.0462 - categorical_accuracy: 0.9848 - val_loss: 0.1303 - val_categorical_accuracy: 0.9660\nEpoch 9/10\n912/916 [============================>.] - ETA: 0s - loss: 0.0410 - categorical_accuracy: 0.9863\nEpoch 00009: val_loss did not improve from 0.12137\n916/916 [==============================] - 5s 5ms/step - loss: 0.0411 - categorical_accuracy: 0.9862 - val_loss: 0.1264 - val_categorical_accuracy: 0.9670\nEpoch 10/10\n908/916 [============================>.] - ETA: 0s - loss: 0.0350 - categorical_accuracy: 0.9883\nEpoch 00010: val_loss improved from 0.12137 to 0.11878, saving model to keras_savedmodels/model_10-0.119.h5\n916/916 [==============================] - 5s 5ms/step - loss: 0.0349 - categorical_accuracy: 0.9884 - val_loss: 0.1188 - val_categorical_accuracy: 0.9702\n"
    }
   ],
   "source": [
    "keras_history1 = keras_fc_model.fit(train_dataset, epochs=epochs1, verbose = 1, callbacks=cblist,\n",
    "                                   validation_data=valid_dataset, shuffle = True)"
   ]
  },
  {
   "cell_type": "markdown",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#### Test data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": "167/167 [==============================] - 0s 3ms/step - loss: 0.1114 - categorical_accuracy: 0.9711\n"
    },
    {
     "data": {
      "text/plain": "[0.11143299131751797, 0.9711]"
     },
     "execution_count": 80,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "keras_fc_model.evaluate(test_dataset)"
   ]
  },
  {
   "cell_type": "markdown",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "### Testing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 337,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": "tf.Tensor(7, shape=(), dtype=int64)\n"
    }
   ],
   "source": [
    "# Taking the first batch in the test dataset\n",
    "for i in test_dataset.take(1):\n",
    "    test_x = i[0]\n",
    "    test_y = i[1]\n",
    "    print(tf.argmax(i[1][-1]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 104,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": "TensorShape([60, 784])"
     },
     "execution_count": 104,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_x.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 106,
   "metadata": {},
   "outputs": [],
   "source": [
    "test_prediction = keras_fc_model.predict(test_x, batch_size=60)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 107,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": "(60, 10)"
     },
     "execution_count": 107,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_prediction.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 108,
   "metadata": {},
   "outputs": [],
   "source": [
    "test_labels = tf.argmax(test_prediction, axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 109,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": "<tf.Tensor: shape=(60,), dtype=int64, numpy=\narray([1, 9, 7, 6, 1, 9, 5, 9, 1, 6, 2, 2, 4, 4, 7, 1, 4, 5, 8, 6, 4, 3,\n       3, 5, 6, 1, 2, 5, 9, 5, 4, 8, 2, 5, 3, 3, 4, 5, 0, 1, 2, 4, 2, 1,\n       2, 8, 3, 2, 1, 1, 8, 3, 8, 4, 8, 3, 6, 5, 4, 7])>"
     },
     "execution_count": 109,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_labels"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 120,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": "<tf.Tensor: shape=(1, 10), dtype=float32, numpy=\narray([[-1.6399686 , -1.8865588 ,  0.6251957 ,  2.1267636 , -1.6664417 ,\n        -0.23559336, -1.7577692 , -0.3431458 ,  2.1472526 ,  0.9643423 ]],\n      dtype=float32)>"
     },
     "execution_count": 120,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# call output for input of batch size = 1\n",
    "keras_fc_model(tf.reshape(test_x[-1], shape=(1,-1)), training = False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 123,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": "array([-0.47502184, -2.1236982 ,  0.6776048 ,  0.5870802 , -4.3017907 ,\n        1.2738266 , -7.350608  , 16.93022   , -6.412077  ,  0.8857768 ],\n      dtype=float32)"
     },
     "execution_count": 123,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# different output using bs=60\n",
    "test_prediction[-1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 121,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": "<tf.Tensor: shape=(1,), dtype=int64, numpy=array([8])>"
     },
     "execution_count": 121,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# output label\n",
    "tf.argmax(keras_fc_model(tf.reshape(test_x[-1], shape=(1,-1)), training = False), axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 122,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": "<tf.Tensor: shape=(1, 10), dtype=float32, numpy=\narray([[-1.6399686 , -1.8865588 ,  0.6251957 ,  2.1267636 , -1.6664417 ,\n        -0.23559336, -1.7577692 , -0.3431458 ,  2.1472526 ,  0.9643423 ]],\n      dtype=float32)>"
     },
     "execution_count": 122,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# setting training = True does not change the output\n",
    "keras_fc_model(tf.reshape(test_x[-1], shape=(1,-1)), training = True)"
   ]
  },
  {
   "cell_type": "markdown",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "    * No change in model output between training = True and False"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 124,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": "[]"
     },
     "execution_count": 124,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "keras_fc_model.state_updates"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": "[<tensorflow.python.keras.engine.input_layer.InputLayer at 0x7f33f44998d0>,\n <tensorflow.python.keras.layers.core.Dense at 0x7f340c27a8d0>,\n <tensorflow.python.keras.layers.normalization_v2.BatchNormalization at 0x7f340c228dd8>,\n <tensorflow.python.keras.layers.core.Activation at 0x7f340c228f60>,\n <tensorflow.python.keras.layers.core.Dense at 0x7f340c1faeb8>,\n <tensorflow.python.keras.layers.normalization_v2.BatchNormalization at 0x7f340c1fe048>,\n <tensorflow.python.keras.layers.core.Activation at 0x7f340c1fe2b0>,\n <tensorflow.python.keras.layers.core.Dense at 0x7f340c1a4d68>,\n <tensorflow.python.keras.layers.normalization_v2.BatchNormalization at 0x7f340c1ac198>,\n <tensorflow.python.keras.layers.core.Activation at 0x7f340c1ac438>,\n <tensorflow.python.keras.layers.core.Dense at 0x7f340c1fab70>]"
     },
     "execution_count": 74,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "keras_fc_model.layers"
   ]
  },
  {
   "cell_type": "markdown",
   "execution_count": 142,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": "[<tf.Variable 'batch_normalization/gamma:0' shape=(100,) dtype=float32, numpy=\n array([1.2269483, 1.1349252, 1.3410976, 0.9179288, 1.2856462, 1.3862823,\n        1.3595926, 1.2745571, 1.5197902, 1.3166919, 1.4408834, 1.2638034,\n        1.2629784, 1.2740654, 1.415783 , 1.4326637, 1.4493729, 1.3970267,\n        1.3195894, 1.3734947, 1.3420596, 1.1721978, 1.2968619, 1.5249897,\n        1.0370381, 1.3912894, 1.5404277, 1.3771281, 1.4850557, 1.3972007,\n        1.4163164, 1.1762176, 1.1790816, 1.3884237, 1.3342997, 1.303082 ,\n        1.1480755, 1.2126306, 1.2919899, 1.1597459, 1.485534 , 1.2900969,\n        1.221908 , 1.2401632, 1.4433057, 1.4404896, 1.508135 , 1.3095189,\n        1.4880893, 1.3983922, 1.2017698, 1.2355157, 1.4851977, 1.5640281,\n        1.5173646, 1.3068354, 1.62453  , 1.5176066, 1.4924428, 1.2629938,\n        1.327193 , 1.4941337, 1.6335568, 1.4370104, 1.5315871, 1.3623064,\n        1.284233 , 1.1915743, 1.4893486, 1.3606577, 1.1597953, 1.3838465,\n        1.2619832, 1.2759168, 1.5620822, 1.2270515, 1.5284364, 1.2064832,\n        1.3470151, 1.4076749, 1.3609585, 1.3630152, 1.6558574, 1.6179266,\n        1.2785853, 1.1306003, 1.3955864, 1.4067265, 1.5867912, 1.4267366,\n        1.3758888, 1.4742678, 1.2202123, 1.0020919, 1.2884294, 1.423675 ,\n        1.3156143, 1.4312552, 1.415758 , 1.4510306], dtype=float32)>,\n <tf.Variable 'batch_normalization/beta:0' shape=(100,) dtype=float32, numpy=\n array([ 0.0725309 ,  0.13827226, -0.06939456,  0.12959993, -0.7255127 ,\n        -0.33006316, -0.07126302,  0.57206756, -0.02050218,  0.01971257,\n        -0.14860351,  0.14233395, -0.5124093 , -0.48969638, -0.34047705,\n        -0.1610549 ,  0.1095884 ,  0.22338244,  0.478024  ,  0.23863536,\n        -0.6544056 , -0.61530656,  0.43131748,  0.25865248, -0.05738853,\n        -0.19319026, -0.1278918 ,  0.20257951, -0.34966147, -0.22343421,\n        -0.28003773,  0.06845671, -0.18589702,  0.24006306, -0.13137347,\n        -0.18610711, -0.09652063,  0.07730827, -0.13674106, -0.36124447,\n        -0.2068936 ,  0.1938605 , -0.07766312,  0.31542403, -0.39941385,\n        -0.13848141, -0.24340703,  0.22813042,  0.3291286 , -0.40002584,\n        -0.39932287,  0.27706632,  0.41098887, -0.23980504, -0.19397306,\n         0.01697474,  0.17772289, -0.23378782, -0.08268411, -0.39916015,\n         0.5646259 ,  0.11323285, -0.29715964,  0.48129022, -0.3476707 ,\n        -0.19937208,  0.07189935, -0.1534567 , -0.239148  ,  0.3631805 ,\n         0.2086617 , -0.43803284,  0.36308923,  0.2902676 ,  0.5195069 ,\n        -0.12038191, -0.41274363, -0.03893393, -0.17440654,  0.27123487,\n        -0.19878754,  0.00880557, -0.51359844,  0.01657882, -0.18577358,\n         0.5174917 ,  0.00808138, -0.25319344, -0.13443038, -0.16466412,\n         0.4555682 ,  0.4716943 , -0.46857092, -0.19754733,  0.11025815,\n        -0.80802745,  0.20232354,  0.60316294,  0.07676253, -0.2766438 ],\n       dtype=float32)>]"
     },
     "execution_count": 142,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "### Exploring the weights of the 1st dense and the following bnorm layer to see what's happening with the variables"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 258,
   "metadata": {},
   "outputs": [],
   "source": [
    "del keras_fc_model\n",
    "tf.keras.backend.clear_session()\n",
    "gc.collect()\n",
    "\n",
    "# loading the best model\n",
    "keras_fc_model = tf.keras.models.load_model(\n",
    "    os.path.join(keras_savedmodels,'model_10-0.119.h5'), compile=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 222,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": "True"
     },
     "execution_count": 222,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "keras_fc_model.trainable"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 223,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": "True\nTrue\nTrue\nTrue\nTrue\nTrue\nTrue\nTrue\nTrue\nTrue\nTrue\n"
    }
   ],
   "source": [
    "for layer in keras_fc_model.layers:\n",
    "    print(layer.trainable)"
   ]
  },
  {
   "cell_type": "markdown",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "    * All layers are trainable"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 224,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": "Model: \"kerasmodel1\"\n_________________________________________________________________\nLayer (type)                 Output Shape              Param #   \n=================================================================\ninput (InputLayer)           [(None, 784)]             0         \n_________________________________________________________________\ndense (Dense)                (None, 100)               78500     \n_________________________________________________________________\nbatch_normalization (BatchNo (None, 100)               400       \n_________________________________________________________________\nactivation (Activation)      (None, 100)               0         \n_________________________________________________________________\ndense_1 (Dense)              (None, 100)               10100     \n_________________________________________________________________\nbatch_normalization_1 (Batch (None, 100)               400       \n_________________________________________________________________\nactivation_1 (Activation)    (None, 100)               0         \n_________________________________________________________________\ndense_2 (Dense)              (None, 100)               10100     \n_________________________________________________________________\nbatch_normalization_2 (Batch (None, 100)               400       \n_________________________________________________________________\nactivation_2 (Activation)    (None, 100)               0         \n_________________________________________________________________\ndense_3 (Dense)              (None, 10)                1010      \n=================================================================\nTotal params: 100,910\nTrainable params: 100,310\nNon-trainable params: 600\n_________________________________________________________________\n"
    }
   ],
   "source": [
    "keras_fc_model.summary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 259,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Copying the weights of the 1st Batch Norm layer before calling or prediction\n",
    "bn1wghts_before = [i.numpy() for i in keras_fc_model.layers[2].weights]"
   ]
  },
  {
   "cell_type": "markdown",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "    * Doing \"layer.weights.copy()\" does not work as still only a pointer between the tensors of the weights and the dummy variable is created.\n",
    "      Using the copy module also does not work. The right way is to used the .numpy() method or convert the tensor to a list using the .to_list()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 260,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": "list"
     },
     "execution_count": 260,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "type(bn1wghts_before)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 261,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": "[array([1.2269483, 1.1349252, 1.3410976, 0.9179288, 1.2856462, 1.3862823,\n        1.3595926, 1.2745571, 1.5197902, 1.3166919, 1.4408834, 1.2638034,\n        1.2629784, 1.2740654, 1.415783 , 1.4326637, 1.4493729, 1.3970267,\n        1.3195894, 1.3734947, 1.3420596, 1.1721978, 1.2968619, 1.5249897,\n        1.0370381, 1.3912894, 1.5404277, 1.3771281, 1.4850557, 1.3972007,\n        1.4163164, 1.1762176, 1.1790816, 1.3884237, 1.3342997, 1.303082 ,\n        1.1480755, 1.2126306, 1.2919899, 1.1597459, 1.485534 , 1.2900969,\n        1.221908 , 1.2401632, 1.4433057, 1.4404896, 1.508135 , 1.3095189,\n        1.4880893, 1.3983922, 1.2017698, 1.2355157, 1.4851977, 1.5640281,\n        1.5173646, 1.3068354, 1.62453  , 1.5176066, 1.4924428, 1.2629938,\n        1.327193 , 1.4941337, 1.6335568, 1.4370104, 1.5315871, 1.3623064,\n        1.284233 , 1.1915743, 1.4893486, 1.3606577, 1.1597953, 1.3838465,\n        1.2619832, 1.2759168, 1.5620822, 1.2270515, 1.5284364, 1.2064832,\n        1.3470151, 1.4076749, 1.3609585, 1.3630152, 1.6558574, 1.6179266,\n        1.2785853, 1.1306003, 1.3955864, 1.4067265, 1.5867912, 1.4267366,\n        1.3758888, 1.4742678, 1.2202123, 1.0020919, 1.2884294, 1.423675 ,\n        1.3156143, 1.4312552, 1.415758 , 1.4510306], dtype=float32),\n array([ 0.0725309 ,  0.13827226, -0.06939456,  0.12959993, -0.7255127 ,\n        -0.33006316, -0.07126302,  0.57206756, -0.02050218,  0.01971257,\n        -0.14860351,  0.14233395, -0.5124093 , -0.48969638, -0.34047705,\n        -0.1610549 ,  0.1095884 ,  0.22338244,  0.478024  ,  0.23863536,\n        -0.6544056 , -0.61530656,  0.43131748,  0.25865248, -0.05738853,\n        -0.19319026, -0.1278918 ,  0.20257951, -0.34966147, -0.22343421,\n        -0.28003773,  0.06845671, -0.18589702,  0.24006306, -0.13137347,\n        -0.18610711, -0.09652063,  0.07730827, -0.13674106, -0.36124447,\n        -0.2068936 ,  0.1938605 , -0.07766312,  0.31542403, -0.39941385,\n        -0.13848141, -0.24340703,  0.22813042,  0.3291286 , -0.40002584,\n        -0.39932287,  0.27706632,  0.41098887, -0.23980504, -0.19397306,\n         0.01697474,  0.17772289, -0.23378782, -0.08268411, -0.39916015,\n         0.5646259 ,  0.11323285, -0.29715964,  0.48129022, -0.3476707 ,\n        -0.19937208,  0.07189935, -0.1534567 , -0.239148  ,  0.3631805 ,\n         0.2086617 , -0.43803284,  0.36308923,  0.2902676 ,  0.5195069 ,\n        -0.12038191, -0.41274363, -0.03893393, -0.17440654,  0.27123487,\n        -0.19878754,  0.00880557, -0.51359844,  0.01657882, -0.18577358,\n         0.5174917 ,  0.00808138, -0.25319344, -0.13443038, -0.16466412,\n         0.4555682 ,  0.4716943 , -0.46857092, -0.19754733,  0.11025815,\n        -0.80802745,  0.20232354,  0.60316294,  0.07676253, -0.2766438 ],\n       dtype=float32),\n array([-9.47964936e-03, -1.13305561e-02, -2.65359320e-02,  1.08358869e-03,\n        -2.73278710e-02, -3.11168749e-03, -7.59504922e-03,  6.60309009e-03,\n         4.54964191e-02, -6.23745052e-03, -2.85433084e-02,  4.74635232e-03,\n        -1.88576318e-02,  2.24377774e-02,  5.44967549e-03, -4.31050472e-02,\n         1.90221705e-04, -6.47694916e-02,  2.13363282e-02, -2.00563092e-02,\n        -5.59500493e-02, -2.07057185e-02,  1.85878631e-02,  1.05127068e-02,\n        -2.63166633e-02,  1.57331135e-02,  1.02630835e-02,  4.82410332e-03,\n        -2.12563518e-02,  1.59315709e-02, -3.26706059e-02, -6.98142685e-05,\n        -7.33800139e-03, -1.47310020e-02,  1.48518737e-02, -5.48786521e-02,\n        -5.33562191e-02,  6.19289353e-02,  8.21460597e-03,  8.23432300e-03,\n        -1.64685398e-02, -4.61975485e-02,  4.18895744e-02, -3.18473727e-02,\n        -2.41696835e-03, -2.99536195e-02, -2.16178652e-02,  2.30958760e-02,\n        -2.75954381e-02, -1.89153329e-02,  2.34767981e-03,  7.07078679e-03,\n         4.26627770e-02, -5.05843237e-02,  7.47829769e-03,  4.66979817e-02,\n         6.60671107e-03,  7.11089559e-03,  6.16222806e-02,  2.83963196e-02,\n         2.39989739e-02, -2.10756809e-03, -4.02762555e-02, -1.02800550e-03,\n         1.32320039e-02,  5.29733375e-02,  1.30893067e-02, -1.21751130e-02,\n         2.57658795e-03, -5.31097241e-02,  1.52281215e-02,  2.39445362e-03,\n         1.92800257e-03, -7.33795464e-02,  2.03121584e-02, -9.88201052e-03,\n         4.01057228e-02, -2.86189560e-02,  4.13046107e-02,  4.34740968e-02,\n         1.47780683e-02, -2.54178140e-02, -1.52519392e-03,  4.88437191e-02,\n         9.54913255e-03,  4.79647778e-02, -5.81578352e-03, -4.73721744e-03,\n         1.91070717e-02, -2.10116152e-03, -1.37595655e-02,  8.68080929e-03,\n         3.35385129e-02,  1.28508126e-02,  3.66466679e-03, -8.42949469e-03,\n         3.74097340e-02,  3.71622294e-02, -3.23811844e-02, -2.26809038e-03],\n       dtype=float32),\n array([21.076986 , 16.843124 , 20.834011 , 24.038845 , 40.036083 ,\n        27.389042 , 21.442293 , 34.84173  , 34.220356 , 16.27224  ,\n        23.450117 , 17.219156 , 30.438965 , 20.586555 , 27.593739 ,\n        21.050295 , 23.088821 , 19.461712 , 39.986233 , 22.584919 ,\n        27.14464  , 24.02813  , 18.352482 , 23.075844 , 15.904074 ,\n        22.048882 , 26.024933 , 21.055819 , 22.949402 , 24.86849  ,\n        26.507677 , 20.462847 , 16.049488 , 17.37114  , 36.425774 ,\n        25.451834 , 25.587803 , 22.594397 , 16.445166 , 25.267582 ,\n        24.626537 , 23.133387 , 15.845271 , 15.977093 , 26.684368 ,\n        18.19778  , 24.9619   , 29.210503 , 34.12977  , 23.537685 ,\n        23.798798 , 21.184074 , 28.373373 , 26.478884 , 16.513794 ,\n        19.778822 , 27.966188 , 19.807663 , 23.658226 , 25.400915 ,\n        28.438196 , 24.261097 , 26.656666 , 18.69155  , 20.17362  ,\n        15.1383705, 14.348465 , 17.397808 , 25.866411 , 19.936626 ,\n        23.50689  , 16.371773 , 20.483273 , 21.956085 , 68.508286 ,\n        18.651371 , 31.916445 , 16.893143 , 25.34363  , 25.841879 ,\n        22.599669 , 24.660942 , 25.784834 , 30.056602 , 23.253328 ,\n        16.411661 , 23.85958  , 16.93443  , 24.809193 , 37.38166  ,\n        22.852282 , 68.457695 , 23.873999 , 15.34342  , 23.76838  ,\n        23.517864 , 28.502975 , 27.204023 , 24.015827 , 17.425133 ],\n       dtype=float32)]"
     },
     "execution_count": 261,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "bn1wghts_before"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 262,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": "array([[-1.6399686 , -1.8865588 ,  0.6251957 ,  2.1267636 , -1.6664417 ,\n        -0.23559336, -1.7577692 , -0.3431458 ,  2.1472526 ,  0.9643423 ]],\n      dtype=float32)"
     },
     "execution_count": 262,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# running a prediction bs=1\n",
    "keras_fc_model.predict(test_x[-1].numpy().reshape(1,-1), batch_size=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 235,
   "metadata": {},
   "outputs": [],
   "source": [
    "keras_fc_model.reset_states()# does nothing as there are no \"states\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 263,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": "<tf.Tensor: shape=(4, 100), dtype=bool, numpy=\narray([[ True,  True,  True,  True,  True,  True,  True,  True,  True,\n         True,  True,  True,  True,  True,  True,  True,  True,  True,\n         True,  True,  True,  True,  True,  True,  True,  True,  True,\n         True,  True,  True,  True,  True,  True,  True,  True,  True,\n         True,  True,  True,  True,  True,  True,  True,  True,  True,\n         True,  True,  True,  True,  True,  True,  True,  True,  True,\n         True,  True,  True,  True,  True,  True,  True,  True,  True,\n         True,  True,  True,  True,  True,  True,  True,  True,  True,\n         True,  True,  True,  True,  True,  True,  True,  True,  True,\n         True,  True,  True,  True,  True,  True,  True,  True,  True,\n         True,  True,  True,  True,  True,  True,  True,  True,  True,\n         True],\n       [ True,  True,  True,  True,  True,  True,  True,  True,  True,\n         True,  True,  True,  True,  True,  True,  True,  True,  True,\n         True,  True,  True,  True,  True,  True,  True,  True,  True,\n         True,  True,  True,  True,  True,  True,  True,  True,  True,\n         True,  True,  True,  True,  True,  True,  True,  True,  True,\n         True,  True,  True,  True,  True,  True,  True,  True,  True,\n         True,  True,  True,  True,  True,  True,  True,  True,  True,\n         True,  True,  True,  True,  True,  True,  True,  True,  True,\n         True,  True,  True,  True,  True,  True,  True,  True,  True,\n         True,  True,  True,  True,  True,  True,  True,  True,  True,\n         True,  True,  True,  True,  True,  True,  True,  True,  True,\n         True],\n       [False, False, False, False, False, False, False, False, False,\n        False, False, False, False, False, False, False, False, False,\n        False, False, False, False, False, False, False, False, False,\n        False, False, False, False, False, False, False, False, False,\n        False, False, False, False, False, False, False, False, False,\n        False, False, False, False, False, False, False, False, False,\n        False, False, False, False, False, False, False, False, False,\n        False, False, False, False, False, False, False, False, False,\n        False, False, False, False, False, False, False, False, False,\n        False, False, False, False, False, False, False, False, False,\n        False, False, False, False, False, False, False, False, False,\n        False],\n       [False, False, False, False, False, False, False, False, False,\n        False, False, False, False, False, False, False, False, False,\n        False, False, False, False, False, False, False, False, False,\n        False, False, False, False, False, False, False, False, False,\n        False, False, False, False, False, False, False, False, False,\n        False, False, False, False, False, False, False, False, False,\n        False, False, False, False, False, False, False, False, False,\n        False, False, False, False, False, False, False, False, False,\n        False, False, False, False, False, False, False, False, False,\n        False, False, False, False, False, False, False, False, False,\n        False, False, False, False, False, False, False, False, False,\n        False]])>"
     },
     "execution_count": 263,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tf.equal(bn1wghts_before, keras_fc_model.layers[2].weights)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 265,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": "<tf.Tensor: shape=(4,), dtype=bool, numpy=array([ True,  True, False, False])>"
     },
     "execution_count": 265,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Checking if the weights of the bn1 layer changed  \n",
    "tf.reduce_all(tf.equal(bn1wghts_before, keras_fc_model.layers[2].weights), axis=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "    * So the moving mean and variance have changed."
   ]
  },
  {
   "cell_type": "markdown",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#### Changing the layer properties to False"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 267,
   "metadata": {},
   "outputs": [],
   "source": [
    "keras_fc_model.trainable = False"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 268,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": "False\nFalse\nFalse\nFalse\nFalse\nFalse\nFalse\nFalse\nFalse\nFalse\nFalse\n"
    }
   ],
   "source": [
    "for layer in keras_fc_model.layers:\n",
    "    print(layer.trainable)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 269,
   "metadata": {},
   "outputs": [],
   "source": [
    "# again, saving the weights of the 1st bn layer before calling/prediction\n",
    "bn1wghts_before = [i.numpy() for i in keras_fc_model.layers[2].weights]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 270,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": "array([20.866217, 16.674694, 20.625671, 23.798456, 39.635723, 27.115152,\n       21.22787 , 34.49331 , 33.87815 , 16.109518, 23.215616, 17.046965,\n       30.134575, 20.38069 , 27.3178  , 20.839792, 22.857933, 19.267096,\n       39.58637 , 22.35907 , 26.873194, 23.787848, 18.168957, 22.845085,\n       15.745033, 21.828392, 25.764683, 20.84526 , 22.719908, 24.619804,\n       26.2426  , 20.258219, 15.888993, 17.197428, 36.061516, 25.197315,\n       25.331924, 22.368452, 16.280714, 25.014906, 24.380272, 22.902052,\n       15.686818, 15.817322, 26.417524, 18.015802, 24.712282, 28.918398,\n       33.78847 , 23.302309, 23.56081 , 20.972233, 28.08964 , 26.214094,\n       16.348656, 19.581034, 27.686527, 19.609587, 23.421644, 25.146906,\n       28.153814, 24.018486, 26.390099, 18.504633, 19.971884, 14.986987,\n       14.20498 , 17.22383 , 25.607748, 19.73726 , 23.27182 , 16.208055,\n       20.27844 , 21.736525, 67.823204, 18.464857, 31.59728 , 16.72421 ,\n       25.090193, 25.58346 , 22.373672, 24.414333, 25.526985, 29.756037,\n       23.020796, 16.247545, 23.620983, 16.765085, 24.5611  , 37.007843,\n       22.623758, 67.77312 , 23.635258, 15.189986, 23.530695, 23.282686,\n       28.217945, 26.931984, 23.77567 , 17.250881], dtype=float32)"
     },
     "execution_count": 270,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# moving variance weights\n",
    "bn1wghts_before[-1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 271,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": "array([[-1.6399693 , -1.886558  ,  0.62519604,  2.1267643 , -1.6664422 ,\n        -0.23559408, -1.757769  , -0.34314552,  2.1472523 ,  0.9643422 ]],\n      dtype=float32)"
     },
     "execution_count": 271,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# prediction on input with bs=1\n",
    "keras_fc_model.predict(test_x[-2].numpy().reshape(1,-1))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 272,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": "<tf.Variable 'batch_normalization/moving_variance:0' shape=(100,) dtype=float32, numpy=\narray([20.866217, 16.674694, 20.625671, 23.798456, 39.635723, 27.115152,\n       21.22787 , 34.49331 , 33.87815 , 16.109518, 23.215616, 17.046965,\n       30.134575, 20.38069 , 27.3178  , 20.839792, 22.857933, 19.267096,\n       39.58637 , 22.35907 , 26.873194, 23.787848, 18.168957, 22.845085,\n       15.745033, 21.828392, 25.764683, 20.84526 , 22.719908, 24.619804,\n       26.2426  , 20.258219, 15.888993, 17.197428, 36.061516, 25.197315,\n       25.331924, 22.368452, 16.280714, 25.014906, 24.380272, 22.902052,\n       15.686818, 15.817322, 26.417524, 18.015802, 24.712282, 28.918398,\n       33.78847 , 23.302309, 23.56081 , 20.972233, 28.08964 , 26.214094,\n       16.348656, 19.581034, 27.686527, 19.609587, 23.421644, 25.146906,\n       28.153814, 24.018486, 26.390099, 18.504633, 19.971884, 14.986987,\n       14.20498 , 17.22383 , 25.607748, 19.73726 , 23.27182 , 16.208055,\n       20.27844 , 21.736525, 67.823204, 18.464857, 31.59728 , 16.72421 ,\n       25.090193, 25.58346 , 22.373672, 24.414333, 25.526985, 29.756037,\n       23.020796, 16.247545, 23.620983, 16.765085, 24.5611  , 37.007843,\n       22.623758, 67.77312 , 23.635258, 15.189986, 23.530695, 23.282686,\n       28.217945, 26.931984, 23.77567 , 17.250881], dtype=float32)>"
     },
     "execution_count": 272,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 1st bn layer moving variance weights after prediction\n",
    "keras_fc_model.layers[2].weights[-1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 273,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": "<tf.Tensor: shape=(100,), dtype=bool, numpy=\narray([ True,  True,  True,  True,  True,  True,  True,  True,  True,\n        True,  True,  True,  True,  True,  True,  True,  True,  True,\n        True,  True,  True,  True,  True,  True,  True,  True,  True,\n        True,  True,  True,  True,  True,  True,  True,  True,  True,\n        True,  True,  True,  True,  True,  True,  True,  True,  True,\n        True,  True,  True,  True,  True,  True,  True,  True,  True,\n        True,  True,  True,  True,  True,  True,  True,  True,  True,\n        True,  True,  True,  True,  True,  True,  True,  True,  True,\n        True,  True,  True,  True,  True,  True,  True,  True,  True,\n        True,  True,  True,  True,  True,  True,  True,  True,  True,\n        True,  True,  True,  True,  True,  True,  True,  True,  True,\n        True])>"
     },
     "execution_count": 273,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tf.equal(bn1wghts_before[-1],keras_fc_model.layers[2].weights[-1] )"
   ]
  },
  {
   "cell_type": "markdown",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "    * With setting trainable=False, the moving mean and variance have not changed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 275,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Loading the best model again\n",
    "del keras_fc_model\n",
    "tf.keras.backend.clear_session()\n",
    "gc.collect()\n",
    "\n",
    "keras_fc_model = tf.keras.models.load_model(\n",
    "    os.path.join(keras_savedmodels,'model_10-0.119.h5'), compile=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 276,
   "metadata": {},
   "outputs": [],
   "source": [
    "keras_fc_model.trainable = False"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 296,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": "False\nFalse\nFalse\nFalse\nFalse\nFalse\nFalse\nFalse\nFalse\nFalse\nFalse\n"
    }
   ],
   "source": [
    "for layer in keras_fc_model.layers:\n",
    "    print(layer.trainable)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 277,
   "metadata": {},
   "outputs": [],
   "source": [
    "# copying the weights of the 1st dense and bn layers\n",
    "dns1wght_before = [i.numpy() for i in keras_fc_model.layers[1].weights]\n",
    "bn1wghts_before = [i.numpy() for i in keras_fc_model.layers[2].weights]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 416,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": "(784, 100)"
     },
     "execution_count": 416,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dns1wght_before[0].shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 417,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": "(100,)"
     },
     "execution_count": 417,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dns1wght_before[1].shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 278,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": "<tf.Tensor: shape=(60,), dtype=int64, numpy=\narray([1, 9, 7, 6, 1, 9, 5, 9, 1, 6, 2, 2, 4, 4, 7, 1, 4, 5, 8, 6, 4, 3,\n       3, 5, 6, 1, 2, 5, 9, 5, 4, 8, 2, 5, 3, 3, 4, 5, 0, 1, 2, 4, 2, 1,\n       2, 8, 3, 2, 1, 1, 8, 3, 8, 4, 8, 3, 6, 5, 4, 7])>"
     },
     "execution_count": 278,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# prediction with bs=60\n",
    "tf.argmax(keras_fc_model.predict(test_x), axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 274,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": "<tf.Tensor: shape=(60,), dtype=int64, numpy=\narray([1, 9, 7, 6, 1, 9, 5, 9, 1, 6, 2, 2, 4, 4, 7, 1, 4, 5, 8, 6, 4, 3,\n       3, 5, 6, 1, 2, 5, 9, 5, 4, 8, 2, 5, 3, 3, 4, 5, 0, 1, 2, 4, 2, 1,\n       2, 8, 3, 2, 1, 1, 8, 3, 8, 4, 8, 3, 6, 5, 4, 7])>"
     },
     "execution_count": 274,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_labels # from prediction immediately after fitting (above)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 339,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": "<tf.Tensor: shape=(60,), dtype=int64, numpy=\narray([1, 9, 7, 6, 1, 9, 5, 9, 1, 6, 2, 2, 4, 4, 7, 1, 4, 5, 8, 6, 4, 3,\n       3, 5, 6, 1, 2, 5, 9, 5, 9, 8, 2, 5, 3, 3, 4, 5, 0, 1, 2, 4, 2, 1,\n       2, 8, 3, 2, 1, 1, 8, 3, 8, 4, 8, 3, 6, 5, 4, 7])>"
     },
     "execution_count": 339,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tf.argmax(test_y, axis=1) # true labels."
   ]
  },
  {
   "cell_type": "markdown",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 285,
   "metadata": {},
   "outputs": [],
   "source": [
    "# copying the weights of the 1st dense and bn layers after prediction\n",
    "dns1wght_afterpred = [i.numpy() for i in keras_fc_model.layers[1].weights]\n",
    "bn1wghts_afterpred = [i.numpy() for i in keras_fc_model.layers[2].weights]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 289,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": "<tf.Tensor: shape=(), dtype=bool, numpy=True>"
     },
     "execution_count": 289,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# comparing the weights of the dense layer before and after prediction. No change.\n",
    "tf.reduce_all(tf.equal(dns1wght_before[0], dns1wght_afterpred[0]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 290,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": "<tf.Tensor: shape=(4,), dtype=bool, numpy=array([ True,  True,  True,  True])>"
     },
     "execution_count": 290,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# comparing the weights of the bn layer before and after prediction. No change, because trainable was set to False\n",
    "tf.reduce_all(tf.equal(bn1wghts_before, bn1wghts_afterpred), axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 340,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": "[<tf.Tensor: shape=(1,), dtype=int64, numpy=array([8])>,\n <tf.Tensor: shape=(1,), dtype=int64, numpy=array([8])>,\n <tf.Tensor: shape=(1,), dtype=int64, numpy=array([8])>,\n <tf.Tensor: shape=(1,), dtype=int64, numpy=array([8])>,\n <tf.Tensor: shape=(1,), dtype=int64, numpy=array([8])>,\n <tf.Tensor: shape=(1,), dtype=int64, numpy=array([8])>,\n <tf.Tensor: shape=(1,), dtype=int64, numpy=array([8])>,\n <tf.Tensor: shape=(1,), dtype=int64, numpy=array([8])>,\n <tf.Tensor: shape=(1,), dtype=int64, numpy=array([8])>,\n <tf.Tensor: shape=(1,), dtype=int64, numpy=array([8])>,\n <tf.Tensor: shape=(1,), dtype=int64, numpy=array([8])>,\n <tf.Tensor: shape=(1,), dtype=int64, numpy=array([8])>,\n <tf.Tensor: shape=(1,), dtype=int64, numpy=array([8])>,\n <tf.Tensor: shape=(1,), dtype=int64, numpy=array([8])>,\n <tf.Tensor: shape=(1,), dtype=int64, numpy=array([8])>,\n <tf.Tensor: shape=(1,), dtype=int64, numpy=array([8])>,\n <tf.Tensor: shape=(1,), dtype=int64, numpy=array([8])>,\n <tf.Tensor: shape=(1,), dtype=int64, numpy=array([8])>,\n <tf.Tensor: shape=(1,), dtype=int64, numpy=array([8])>,\n <tf.Tensor: shape=(1,), dtype=int64, numpy=array([8])>,\n <tf.Tensor: shape=(1,), dtype=int64, numpy=array([8])>,\n <tf.Tensor: shape=(1,), dtype=int64, numpy=array([8])>,\n <tf.Tensor: shape=(1,), dtype=int64, numpy=array([8])>,\n <tf.Tensor: shape=(1,), dtype=int64, numpy=array([8])>,\n <tf.Tensor: shape=(1,), dtype=int64, numpy=array([8])>,\n <tf.Tensor: shape=(1,), dtype=int64, numpy=array([8])>,\n <tf.Tensor: shape=(1,), dtype=int64, numpy=array([8])>,\n <tf.Tensor: shape=(1,), dtype=int64, numpy=array([8])>,\n <tf.Tensor: shape=(1,), dtype=int64, numpy=array([8])>,\n <tf.Tensor: shape=(1,), dtype=int64, numpy=array([8])>,\n <tf.Tensor: shape=(1,), dtype=int64, numpy=array([8])>,\n <tf.Tensor: shape=(1,), dtype=int64, numpy=array([8])>,\n <tf.Tensor: shape=(1,), dtype=int64, numpy=array([8])>,\n <tf.Tensor: shape=(1,), dtype=int64, numpy=array([8])>,\n <tf.Tensor: shape=(1,), dtype=int64, numpy=array([8])>,\n <tf.Tensor: shape=(1,), dtype=int64, numpy=array([8])>,\n <tf.Tensor: shape=(1,), dtype=int64, numpy=array([8])>,\n <tf.Tensor: shape=(1,), dtype=int64, numpy=array([8])>,\n <tf.Tensor: shape=(1,), dtype=int64, numpy=array([8])>,\n <tf.Tensor: shape=(1,), dtype=int64, numpy=array([8])>,\n <tf.Tensor: shape=(1,), dtype=int64, numpy=array([8])>,\n <tf.Tensor: shape=(1,), dtype=int64, numpy=array([8])>,\n <tf.Tensor: shape=(1,), dtype=int64, numpy=array([8])>,\n <tf.Tensor: shape=(1,), dtype=int64, numpy=array([8])>,\n <tf.Tensor: shape=(1,), dtype=int64, numpy=array([8])>,\n <tf.Tensor: shape=(1,), dtype=int64, numpy=array([8])>,\n <tf.Tensor: shape=(1,), dtype=int64, numpy=array([8])>,\n <tf.Tensor: shape=(1,), dtype=int64, numpy=array([8])>,\n <tf.Tensor: shape=(1,), dtype=int64, numpy=array([8])>,\n <tf.Tensor: shape=(1,), dtype=int64, numpy=array([8])>,\n <tf.Tensor: shape=(1,), dtype=int64, numpy=array([8])>,\n <tf.Tensor: shape=(1,), dtype=int64, numpy=array([8])>,\n <tf.Tensor: shape=(1,), dtype=int64, numpy=array([8])>,\n <tf.Tensor: shape=(1,), dtype=int64, numpy=array([8])>,\n <tf.Tensor: shape=(1,), dtype=int64, numpy=array([8])>,\n <tf.Tensor: shape=(1,), dtype=int64, numpy=array([8])>,\n <tf.Tensor: shape=(1,), dtype=int64, numpy=array([8])>,\n <tf.Tensor: shape=(1,), dtype=int64, numpy=array([8])>,\n <tf.Tensor: shape=(1,), dtype=int64, numpy=array([8])>,\n <tf.Tensor: shape=(1,), dtype=int64, numpy=array([8])>]"
     },
     "execution_count": 340,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Predicting on each sample individually to compare against running on a batch\n",
    "test_out = []\n",
    "for i in range(test_x.shape[0]):\n",
    "    # print(i.numpy().mean())\n",
    "    test_out.append(tf.argmax(keras_fc_model.predict(test_x[i:i+1]), axis=1))\n",
    "\n",
    "test_out"
   ]
  },
  {
   "cell_type": "markdown",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "    * The output does not make sense. Each different sample has resulted in the same output. This suggests that despite setting trainable=False,\n",
    "      the batch norm layer still use batch mean and batch variance and not the moving mean and moving variance to calculate the output.\n",
    "\n",
    "    * This implies that using a batch size of greater than 1 is important for prediction. Even in this case, the model uses batch mean and batch\n",
    "      variance rather than the original population mean and variance. If the bn layer trainable is set to True, the moving mean and var, i.e.the\n",
    "      population stats, are updated, else not."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 318,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": "True\nTrue\nTrue\nTrue\nTrue\nTrue\nTrue\nTrue\nTrue\nTrue\nTrue\n"
    }
   ],
   "source": [
    "# Setting the layers as trainable\n",
    "keras_fc_model.trainable = True\n",
    "for layer in keras_fc_model.layers:\n",
    "    print(layer.trainable)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 319,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": "<tf.Tensor: shape=(60,), dtype=int64, numpy=\narray([1, 9, 7, 6, 1, 9, 5, 9, 1, 6, 2, 2, 4, 4, 7, 1, 4, 5, 8, 6, 4, 3,\n       3, 5, 6, 1, 2, 5, 9, 5, 4, 8, 2, 5, 3, 3, 4, 5, 0, 1, 2, 4, 2, 1,\n       2, 8, 3, 2, 1, 1, 8, 3, 8, 4, 8, 3, 6, 5, 4, 7])>"
     },
     "execution_count": 319,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# on a batch of size = 60\n",
    "tf.argmax(keras_fc_model(test_x), axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 329,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": "<tf.Tensor: shape=(5,), dtype=int64, numpy=array([3, 6, 5, 4, 7])>"
     },
     "execution_count": 329,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# on a batch of size = 5, predictions remain similar\n",
    "tf.argmax(keras_fc_model(test_x[55:]), axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 330,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": "<tf.Tensor: shape=(3,), dtype=int64, numpy=array([5, 4, 7])>"
     },
     "execution_count": 330,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# on a batch of size = 3, predictions remain similar\n",
    "tf.argmax(keras_fc_model(test_x[57:]), axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 331,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": "<tf.Tensor: shape=(2,), dtype=int64, numpy=array([4, 3])>"
     },
     "execution_count": 331,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# on a batch of size = 2, prediction accuracy is 50%\n",
    "tf.argmax(keras_fc_model(test_x[58:]), axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 332,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": "<tf.Tensor: shape=(1,), dtype=int64, numpy=array([8])>"
     },
     "execution_count": 332,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tf.argmax(keras_fc_model(test_x[59:]), axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 304,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": "<tf.Tensor: shape=(1,), dtype=int64, numpy=array([8])>"
     },
     "execution_count": 304,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# on a batch of size = 1, prediction is as before, spitting out '8' for all inputs\n",
    "tf.argmax(keras_fc_model(tf.expand_dims(test_x[-1], axis=0)), axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 320,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": "<tf.Tensor: shape=(4,), dtype=bool, numpy=array([ True,  True, False, False])>"
     },
     "execution_count": 320,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Checking if the weights of the 1st bn layer have changed\n",
    "tf.reduce_all(tf.equal(bn1wghts_before, keras_fc_model.layers[2].weights), axis=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "### So what is happening when using only one sample?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 350,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": "TensorShape([784])"
     },
     "execution_count": 350,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_sample = test_x[-1]\n",
    "test_sample.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 351,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": "TensorShape([1, 784])"
     },
     "execution_count": 351,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# reshaping into bs x input = 1 x 784\n",
    "test_sample = tf.expand_dims(test_sample, axis=0)\n",
    "test_sample.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 352,
   "metadata": {},
   "outputs": [],
   "source": [
    "x = keras_fc_model.layers[1](test_sample)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 353,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": "<tf.Tensor: shape=(1, 100), dtype=float32, numpy=\narray([[ 7.2454110e-02,  1.6121851e-01, -5.9279698e-01, -3.5625021e+00,\n         5.6486158e+00,  1.8131368e+00,  2.2250998e+00, -9.2394991e+00,\n         6.8580999e+00, -3.2521808e+00, -3.8725848e+00, -4.8101954e+00,\n         7.5291615e+00, -1.2260950e+01, -2.0900023e+00,  1.2195963e+00,\n         6.7551619e-01,  8.5417891e+00, -4.6665339e+00,  4.9510393e+00,\n        -3.3225839e+00,  5.9273677e+00,  1.6514000e+00, -3.2149615e+00,\n        -6.2358837e+00, -1.3371571e+01,  3.7234601e-01, -6.7863178e+00,\n         1.5272391e+00, -7.8243756e+00,  5.1648498e+00,  7.5447268e+00,\n         1.5534180e+00,  2.4607325e+00, -7.1062841e+00,  2.2235289e+00,\n         5.1989765e+00, -2.8489923e+00,  3.7170084e+00, -4.5380349e+00,\n         3.2168820e+00,  2.8787351e-01,  1.9058392e+00,  4.1891546e+00,\n         5.2643147e+00,  2.4832835e+00,  5.3992143e+00, -9.1668167e+00,\n        -5.2137561e+00,  1.1860637e+01,  4.8817563e+00, -7.7666149e+00,\n        -5.1408153e+00,  3.6978636e+00,  6.5479164e+00, -9.2205362e+00,\n        -8.2373619e+00,  4.6396565e+00,  7.2580934e+00,  9.8909298e-03,\n        -3.1045222e+00, -6.7756546e-01,  3.4100225e+00, -3.6700037e+00,\n        -5.8807821e+00,  2.8058200e+00, -1.4055607e+00, -9.7930479e-01,\n         6.9504099e+00, -5.4594240e+00, -1.0238970e+01, -2.0173113e+00,\n         6.3875929e-02,  6.3440555e-01,  2.0938821e+00, -4.0404968e+00,\n         4.2832246e+00,  3.1141212e+00,  3.0902274e+00,  1.1832256e+00,\n         5.0429672e-01,  5.1996791e-01, -2.6988006e+00, -4.6864676e+00,\n         1.7786427e+00, -2.1827197e+00, -6.5343451e+00, -2.8798950e+00,\n        -7.7858605e+00, -7.4602594e+00, -6.6749349e+00, -3.8071394e-01,\n        -3.3673732e+00, -3.1014171e+00,  3.9711661e+00,  6.0375791e+00,\n        -7.1257219e+00, -5.1079063e+00, -3.2053969e+00, -3.3552625e+00]],\n      dtype=float32)>"
     },
     "execution_count": 353,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 354,
   "metadata": {},
   "outputs": [],
   "source": [
    "bn1wghts_before = [i.numpy() for i in keras_fc_model.layers[2].weights]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 355,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": "<tf.Tensor: shape=(1, 100), dtype=float32, numpy=\narray([[ 0.15085383,  0.31743085, -0.16629188, -0.65695924,  0.97057295,\n         0.3213714 ,  0.5426107 , -2.1190796 ,  1.9093887 , -1.4385841 ,\n        -1.6424987 , -1.710846  ,  1.8375623 , -4.998596  , -1.2291566 ,\n         0.42653888,  0.29996243,  3.8062656 , -0.8616546 ,  2.1698096 ,\n        -2.015799  ,  1.3508562 ,  1.2605891 , -1.0878319 , -2.2595797 ,\n        -5.1802454 ,  0.01518461, -2.780314  ,  0.22559097, -2.7845726 ,\n         1.7277735 ,  2.6498253 ,  0.5636972 ,  1.3715785 , -2.3905523 ,\n         0.8962747 ,  1.680484  , -0.9752423 ,  1.486685  , -2.0054598 ,\n         0.94583297,  0.6109275 ,  0.54913896,  2.0541725 ,  1.6639935 ,\n         0.944753  ,  2.1851091 , -2.839921  , -1.5892298 ,  4.1117544 ,\n         1.4021511 , -2.4831624 , -1.4514474 ,  1.4872153 ,  3.000867  ,\n        -3.6977327 , -3.2329707 ,  2.2383838 ,  2.717575  , -0.5416679 ,\n        -0.55600256, -0.22708406,  1.176469  , -1.3580976 , -2.952281  ,\n         1.2185371 , -0.48507637, -0.3555448 ,  2.4226036 , -1.6188681 ,\n        -3.076171  , -1.4092741 ,  0.31892136,  0.62292224,  1.2838175 ,\n        -1.7029567 ,  0.94957954,  1.2568707 ,  1.143868  ,  0.796368  ,\n         0.06236662,  0.27620366, -1.8543824 , -1.7517853 ,  0.46764246,\n        -0.38578704, -2.4551926 , -1.5845742 , -3.694496  , -2.39157   ,\n        -2.131837  ,  0.579357  , -1.5333937 , -1.2250006 ,  1.6112467 ,\n         1.3571093 , -2.3655915 , -1.3148227 , -1.0736322 , -1.481328  ]],\n      dtype=float32)>"
     },
     "execution_count": 355,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x_bn = keras_fc_model.layers[2](x, training = False)\n",
    "x_bn"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 356,
   "metadata": {},
   "outputs": [],
   "source": [
    "bn1wghts_aftercall = [i.numpy() for i in keras_fc_model.layers[2].weights]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 358,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": "<tf.Tensor: shape=(4,), dtype=bool, numpy=array([ True,  True,  True,  True])>"
     },
     "execution_count": 358,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tf.reduce_all(tf.equal(bn1wghts_aftercall, bn1wghts_before), axis=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "    * No change in the moving mean/var courtesy of setting training=False (I presume)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 359,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": "TensorShape([1, 100])"
     },
     "execution_count": 359,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x_activ = keras_fc_model.layers[3](x_bn)\n",
    "x_activ.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 360,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": "<tf.Tensor: shape=(1, 100), dtype=float32, numpy=\narray([[ 0.14971982,  0.307182  , -0.16477582, -0.57633626,  0.74895597,\n         0.31074643,  0.49496162, -0.9715424 ,  0.957034  , -0.8934123 ,\n        -0.92782104, -0.9367512 ,  0.95056057, -0.999909  , -0.84233457,\n         0.40242475,  0.2912782 ,  0.9990122 , -0.6971092 ,  0.97425276,\n        -0.96512705,  0.8742553 ,  0.85122645, -0.7960855 , -0.9784387 ,\n        -0.9999367 ,  0.01518344, -0.9923367 ,  0.22184041, -0.9924014 ,\n         0.9387922 ,  0.990063  ,  0.5107154 ,  0.87905145, -0.98336613,\n         0.71447915,  0.9329244 , -0.750999  ,  0.90271294, -0.96441114,\n         0.7378908 ,  0.54477966,  0.4998746 ,  0.9676615 ,  0.93075305,\n         0.7373984 ,  0.975019  , -0.993195  , -0.92003113,  0.99946356,\n         0.8858157 , -0.9861591 , -0.8959787 ,  0.902811  ,  0.9950633 ,\n        -0.9987726 , -0.99689376,  0.97751546,  0.9913167 , -0.49424943,\n        -0.5050055 , -0.22325954,  0.826335  , -0.87595093, -0.9945609 ,\n         0.83922213, -0.45029986, -0.34128374,  0.98439074, -0.9244599 ,\n        -0.99575204, -0.8873399 ,  0.30853125,  0.55315936,  0.85749876,\n        -0.9357776 ,  0.7395927 ,  0.8501991 ,  0.8157125 ,  0.6620014 ,\n         0.06228587,  0.26938784, -0.952157  , -0.94157845,  0.4362925 ,\n        -0.3677226 , -0.98536855, -0.9193133 , -0.9987647 , -0.9833997 ,\n        -0.97224945,  0.5221979 , -0.91100365, -0.84112304,  0.923344  ,\n         0.87572086, -0.9825221 , -0.8654905 , -0.7908258 , -0.9017165 ]],\n      dtype=float32)>"
     },
     "execution_count": 360,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x_activ"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 363,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": "<function tensorflow.python.keras.activations.tanh(x)>"
     },
     "execution_count": 363,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "keras_fc_model.layers[3].activation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 364,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": "TensorShape([1, 100])"
     },
     "execution_count": 364,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x2 = keras_fc_model.layers[4](x_activ)\n",
    "x2.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 365,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": "TensorShape([1, 100])"
     },
     "execution_count": 365,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x2_bn = keras_fc_model.layers[5](x2, training = False)\n",
    "x2_bn.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 366,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": "<tf.Tensor: shape=(1, 100), dtype=float32, numpy=\narray([[ 1.4469844 ,  0.81933117,  0.08188009,  5.5216475 ,  1.6466796 ,\n         3.1818552 ,  6.1577783 , -6.268153  ,  0.13917148,  2.8146408 ,\n         1.6124916 ,  2.125453  ,  1.1283283 , -0.6871972 ,  0.27393034,\n         1.8312399 , -0.2513807 ,  0.63431036,  2.1236696 ,  2.640796  ,\n        -1.4139752 , -1.0399275 , -0.04609195,  0.09193619, -1.9930353 ,\n         0.9907995 , -0.5948348 ,  3.446775  ,  3.2425377 ,  0.26977125,\n        -1.5345777 ,  2.3626957 , -0.34507048,  0.826348  , -3.725133  ,\n         2.1427503 ,  3.036732  , -2.370662  , -1.422682  ,  1.0785854 ,\n        -0.8406631 , -1.0621421 , -1.2442622 , -3.2096956 , -0.37212515,\n         0.9509739 ,  0.20337313,  2.1173682 ,  1.1640146 , -3.1060717 ,\n         2.06516   , -1.8573284 , -4.5774794 ,  2.2338474 ,  1.2312582 ,\n        -4.235283  , -1.2487199 , -3.3013825 ,  2.0299222 , -0.6114552 ,\n         1.9935496 ,  0.41533   , -4.1426725 ,  1.5959333 , -1.4516155 ,\n         0.3862956 ,  2.0928628 , -0.9713255 ,  0.4667313 ,  3.0295286 ,\n         0.39003024, -0.6042428 , -3.5324564 ,  1.2642055 , -0.12422729,\n        -1.2656407 ,  1.017278  ,  0.28193396, -0.8429899 ,  1.2441393 ,\n         0.11159265,  2.8635287 , -1.7469139 ,  3.969672  , -1.8412215 ,\n        -2.4179213 , -1.6794939 ,  2.7932186 , -0.11464133, -0.53945875,\n        -1.4187484 , -4.022732  , -3.3950155 ,  1.0791916 ,  3.1329916 ,\n         2.0015762 ,  1.4866371 , -0.8220953 ,  1.593153  ,  1.5754225 ]],\n      dtype=float32)>"
     },
     "execution_count": 366,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x2_bn"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 367,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": "<tf.Tensor: shape=(1, 100), dtype=float32, numpy=\narray([[ 0.895095  ,  0.6747057 ,  0.08169758,  0.99996805,  0.92840064,\n         0.99656   ,  0.9999912 , -0.99999285,  0.13827984,  0.99284345,\n         0.9235274 ,  0.9718979 ,  0.8104464 , -0.5961785 ,  0.2672782 ,\n         0.94994724, -0.2462161 ,  0.56101316,  0.97179884,  0.9898829 ,\n        -0.8883353 , -0.77785945, -0.04605933,  0.09167803, -0.9635322 ,\n         0.7577031 , -0.53336394,  0.99797344,  0.9969525 ,  0.26341194,\n        -0.9112047 ,  0.9824213 , -0.33199653,  0.67851025, -0.99883795,\n         0.9728405 ,  0.9954043 , -0.9826969 , -0.89015716,  0.79267395,\n        -0.68616015, -0.7864824 , -0.8466669 , -0.996746  , -0.35584915,\n         0.7402237 ,  0.20061485,  0.97144616,  0.82234395, -0.9959981 ,\n         0.9683534 , -0.9524313 , -0.99978864,  0.97731274,  0.8429439 ,\n        -0.9995809 , -0.84792435, -0.99729043,  0.9660816 , -0.5451507 ,\n         0.9635689 ,  0.39298886, -0.99949574,  0.9210541 , -0.89601177,\n         0.3681623 ,  0.9700335 , -0.7492862 ,  0.4355545 ,  0.99533767,\n         0.37138626, -0.5400618 , -0.9982922 ,  0.85221946, -0.12359215,\n        -0.8526117 ,  0.76875556,  0.27469406, -0.6873895 ,  0.84663194,\n         0.11113172,  0.9935079 , -0.9410232 ,  0.99928725, -0.95091224,\n        -0.9842452 , -0.9327959 ,  0.99253136, -0.11414173, -0.49257815,\n        -0.8893376 , -0.9993591 , -0.99775267,  0.79289913,  0.9962075 ,\n         0.96413875,  0.9027041 , -0.67620873,  0.92063135,  0.91788405]],\n      dtype=float32)>"
     },
     "execution_count": 367,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x2_activ = keras_fc_model.layers[6](x2_bn)\n",
    "x2_activ"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 368,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": "<function tensorflow.python.keras.activations.tanh(x)>"
     },
     "execution_count": 368,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "keras_fc_model.layers[6].activation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 369,
   "metadata": {},
   "outputs": [],
   "source": [
    "x3 = keras_fc_model.layers[7](x2_activ)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 370,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": "<tf.Tensor: shape=(1, 100), dtype=float32, numpy=\narray([[-2.6575303 ,  4.0253205 , -4.5723295 , -0.8328315 , -2.1877398 ,\n         5.7970123 , -2.1550064 ,  1.343811  ,  4.4645624 , -3.1028416 ,\n        -0.01754965, -0.06544566, -2.495442  , -0.9473952 , -3.1625395 ,\n        -4.8681808 ,  5.214006  ,  3.11476   , -2.796949  ,  4.2898436 ,\n        -2.6451263 , -5.971562  ,  2.9875793 ,  2.7249525 , -2.207279  ,\n         3.0586452 , -3.7173393 , -5.3977556 , -2.6516745 , -3.4322805 ,\n        -1.1940911 , -0.8222678 ,  3.7170627 , -6.393489  , -1.188656  ,\n         2.353908  , -3.9905925 , -3.1419573 ,  2.200204  ,  4.343115  ,\n         3.1492703 , -5.1185412 , -1.9381793 , -0.46158677,  2.03314   ,\n         0.10080343, -0.5024499 ,  1.4315917 ,  2.958827  ,  4.1813436 ,\n         5.5145955 , -3.4113038 ,  0.01222157, -3.2777324 , -0.01305201,\n         0.6775641 ,  1.1302983 ,  2.2492542 , -3.693972  , -2.5032449 ,\n         4.2869644 ,  2.4097655 ,  0.623179  ,  2.629727  , -0.61606425,\n        -2.9371173 , -4.148166  ,  3.9615362 ,  3.9864044 , -1.442079  ,\n        -2.4448576 , -3.5733826 , -3.7040915 , -1.8602548 , -1.0515144 ,\n        -4.232032  , -1.2483096 , -0.4200025 ,  2.8162067 , -3.1672773 ,\n         0.4743244 ,  3.0609243 ,  2.0378203 ,  3.287813  ,  0.6042713 ,\n        -6.3054175 ,  0.46171454, -0.5369231 ,  0.27971873, -1.1871147 ,\n         5.407876  ,  3.53885   , -3.938532  , -4.425593  ,  0.26109427,\n        -0.1026352 , -3.517859  ,  5.531349  ,  1.3756092 ,  3.9457374 ]],\n      dtype=float32)>"
     },
     "execution_count": 370,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x3_bn = keras_fc_model.layers[8](x3, training = False)\n",
    "x3_bn"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 371,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": "<tf.Tensor: shape=(1, 100), dtype=float32, numpy=\narray([[-0.99021417,  0.99936235, -0.9997865 , -0.68199354, -0.97514844,\n         0.9999816 , -0.97348946,  0.8725846 ,  0.99973506, -0.9959723 ,\n        -0.01754784, -0.06535237, -0.9864926 , -0.73860157, -0.99642485,\n        -0.9998818 ,  0.9999408 ,  0.99606705, -0.9925866 ,  0.9996244 ,\n        -0.98996955, -0.9999869 ,  0.99493057,  0.9914434 , -0.9760894 ,\n         0.9956008 , -0.9988198 , -0.99995905, -0.99009955, -0.9979139 ,\n        -0.83184344, -0.6763024 ,  0.9988191 , -0.99999446, -0.8301616 ,\n         0.9821125 , -0.9993166 , -0.99627477,  0.97575283,  0.9996622 ,\n         0.99632883, -0.9999283 , -0.9593894 , -0.43137658,  0.9662956 ,\n         0.10046338, -0.46404174,  0.8919924 ,  0.9946314 ,  0.9995333 ,\n         0.9999677 , -0.99782467,  0.01222096, -0.9971594 , -0.01305127,\n         0.5899336 ,  0.81112134,  0.97799367, -0.99876344, -0.9867003 ,\n         0.99962217,  0.98398817,  0.5533376 ,  0.9896575 , -0.54838175,\n        -0.9943939 , -0.9995012 ,  0.99927557,  0.99931073, -0.8941155 ,\n        -0.98506534, -0.99842644, -0.9987882 , -0.95270234, -0.78239447,\n        -0.9995783 , -0.8478089 , -0.39693248,  0.99286574, -0.99645835,\n         0.44168675,  0.9956208 ,  0.96660435,  0.9972159 ,  0.5400819 ,\n        -0.99999344,  0.43148056, -0.49065536,  0.27264473, -0.8296819 ,\n         0.9999598 ,  0.9983139 , -0.99924135, -0.9997136 ,  0.25531873,\n        -0.10227631, -0.9982418 ,  0.9999687 ,  0.8799642 ,  0.99925256]],\n      dtype=float32)>"
     },
     "execution_count": 371,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x3_activ = keras_fc_model.layers[9](x3_bn)\n",
    "x3_activ"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 372,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": "<function tensorflow.python.keras.activations.tanh(x)>"
     },
     "execution_count": 372,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "keras_fc_model.layers[9].activation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 374,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": "<tf.Tensor: shape=(1, 10), dtype=float32, numpy=\narray([[-1.0399033 ,  0.8363577 ,  0.57764983, -1.0306426 , -3.062043  ,\n         1.4964693 , -5.39317   , 17.652716  , -9.008466  , -0.6458832 ]],\n      dtype=float32)>"
     },
     "execution_count": 374,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x_out = keras_fc_model.layers[10](x3_activ)\n",
    "x_out"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 377,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": "<tf.Tensor: shape=(1,), dtype=int64, numpy=array([7])>"
     },
     "execution_count": 377,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x_label = tf.argmax(x_out, axis=1)\n",
    "x_label"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 387,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": "<tf.Tensor: shape=(), dtype=int64, numpy=7>"
     },
     "execution_count": 387,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tf.squeeze(x_label)"
   ]
  },
  {
   "cell_type": "markdown",
   "execution_count": 378,
   "metadata": {},
   "outputs": [],
   "source": [
    "    * Agrees with the prediction using batch size > 2. \n",
    "\n",
    "    * Testing on other inputs for confirmation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 396,
   "metadata": {},
   "outputs": [],
   "source": [
    "def pred_fn(x_input, model, training = False):\n",
    "    \"\"\"\n",
    "    Parameters:\n",
    "    x_input: tensor or np.array of shape: (784,)\n",
    "\n",
    "    model: keras model for prediction\n",
    "    \"\"\"\n",
    "    model_layers = model.layers\n",
    "\n",
    "    x_input = tf.expand_dims(x_input, axis=0) \n",
    "    # reshaping to batchsize x input: 1 * 784\n",
    "    x = model_layers[1](x_input) # dense1\n",
    "    x = model_layers[2](x, training=training) # bn1\n",
    "    x = model_layers[3](x) # activation1\n",
    "\n",
    "    x = model_layers[4](x) # dense2\n",
    "    x = model_layers[5](x, training=training) # bn2\n",
    "    x = model_layers[6](x) # activation2\n",
    "\n",
    "    x = model_layers[7](x) # dense3\n",
    "    x = model_layers[8](x, training=training) # bn3\n",
    "    x = model_layers[9](x) # activation3\n",
    "\n",
    "    x_out = model_layers[10](x)\n",
    "    label = tf.argmax(x_out, axis=1) # output of shape (1,) - a vector, \n",
    "                                     # need a scalar of shape: ()\n",
    "    \n",
    "    return tf.squeeze(label).numpy()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 380,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": "<tf.Tensor: shape=(60,), dtype=int64, numpy=\narray([1, 9, 7, 6, 1, 9, 5, 9, 1, 6, 2, 2, 4, 4, 7, 1, 4, 5, 8, 6, 4, 3,\n       3, 5, 6, 1, 2, 5, 9, 5, 4, 8, 2, 5, 3, 3, 4, 5, 0, 1, 2, 4, 2, 1,\n       2, 8, 3, 2, 1, 1, 8, 3, 8, 4, 8, 3, 6, 5, 4, 7])>"
     },
     "execution_count": 380,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_labels"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 392,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": "[1, 9, 7, 6, 1]"
     },
     "execution_count": 392,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pred_labels = []\n",
    "for i in test_x:\n",
    "    pred_labels.append(pred_fn(i, keras_fc_model, training=False))\n",
    "pred_labels[:5]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 393,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": "<tf.Tensor: shape=(60,), dtype=bool, numpy=\narray([ True,  True,  True,  True,  True,  True,  True,  True,  True,\n        True,  True,  True,  True,  True,  True,  True, False,  True,\n        True,  True,  True,  True,  True,  True,  True,  True,  True,\n        True, False,  True,  True,  True,  True,  True,  True,  True,\n        True,  True,  True,  True,  True,  True,  True,  True,  True,\n        True,  True,  True,  True,  True,  True,  True,  True,  True,\n        True,  True,  True,  True,  True,  True])>"
     },
     "execution_count": 393,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tf.equal(test_labels, pred_labels)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 394,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": "<tf.Tensor: shape=(), dtype=bool, numpy=False>"
     },
     "execution_count": 394,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Not all are true but atleast majority are\n",
    "tf.reduce_all(tf.equal(test_labels, pred_labels))"
   ]
  },
  {
   "cell_type": "markdown",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "    * Running again and setting training = True"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 397,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": "[8, 8, 8, 8, 8]"
     },
     "execution_count": 397,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pred_labels = []\n",
    "for i in test_x:\n",
    "    pred_labels.append(pred_fn(i, keras_fc_model, training=True))\n",
    "pred_labels[:5]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 398,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": "<tf.Tensor: shape=(1, 10), dtype=float32, numpy=\narray([[-1.6399686 , -1.8865588 ,  0.6251957 ,  2.1267636 , -1.6664417 ,\n        -0.23559336, -1.7577692 , -0.3431458 ,  2.1472526 ,  0.9643423 ]],\n      dtype=float32)>"
     },
     "execution_count": 398,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "keras_fc_model(tf.expand_dims(test_x[-1], axis=0), training = False)"
   ]
  },
  {
   "cell_type": "markdown",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "    * This has happened because in the original model definition, the training arg of the batch norm layers has been set to true.\n",
    "      Hence, in the pred_fn(), when the training argument was to False, the prediction was correct, and when set to true,\n",
    "      the outputs were as before, i.e. 8. \n",
    "\n",
    "    * This also implies that during training, minibatch statistics are used, i.e. minibatch mean/var, rather than moving mean and\n",
    "      variance. Why is this the case? Why not moving mean/var?"
   ]
  },
  {
   "cell_type": "markdown",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "### Redefining the model correctly"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 399,
   "metadata": {},
   "outputs": [],
   "source": [
    "input_layer = tf.keras.Input(shape = (784,), name = 'input')\n",
    "kerasdense1 = tf.keras.layers.Dense(units = units1, activation=None, \n",
    "                                    kernel_initializer=tf.keras.initializers.Orthogonal(gain=1,seed=random_seed))(input_layer)\n",
    "kerasdensebn1 = bn_layer(axis = [-1])(kerasdense1)\n",
    "activation1 = tf.keras.layers.Activation(activation = tf.nn.tanh)(kerasdensebn1)\n",
    "\n",
    "kerasdense2 = tf.keras.layers.Dense(units = units2, activation=None, \n",
    "                                    kernel_initializer=tf.keras.initializers.Orthogonal(gain=1,seed=random_seed))(activation1)\n",
    "kerasdensebn2 = bn_layer(axis = [-1])(kerasdense2)\n",
    "activation2 = tf.keras.layers.Activation(activation = tf.nn.tanh)(kerasdensebn2)\n",
    "\n",
    "kerasdense3 = tf.keras.layers.Dense(units = units3, activation=None, \n",
    "                                    kernel_initializer=tf.keras.initializers.Orthogonal(gain=1,seed=random_seed))(activation2)\n",
    "kerasdensebn3 = bn_layer(axis = [-1])(kerasdense3)\n",
    "activation3 = tf.keras.layers.Activation(activation = tf.nn.tanh)(kerasdensebn3)\n",
    "\n",
    "output_layer = tf.keras.layers.Dense(units = 10, activation=None,\n",
    "                                    kernel_initializer=tf.keras.initializers.Orthogonal(gain=1,seed=random_seed))(activation3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 400,
   "metadata": {},
   "outputs": [],
   "source": [
    "keras_fc_model2 = tf.keras.Model(inputs = [input_layer], outputs = [output_layer], name = 'kerasmodel2')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 402,
   "metadata": {},
   "outputs": [],
   "source": [
    "modelconfig = keras_fc_model2.get_config()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 403,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": "286785"
     },
     "execution_count": 403,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "del keras_fc_model2\n",
    "tf.keras.backend.clear_session()\n",
    "gc.collect()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 407,
   "metadata": {},
   "outputs": [],
   "source": [
    "# initializing a new model without any weights\n",
    "keras_fc_model2 = tf.keras.Model.from_config(modelconfig)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 409,
   "metadata": {},
   "outputs": [],
   "source": [
    "# loading the weights of the best model\n",
    "keras_fc_model2.load_weights(os.path.join(keras_savedmodels, 'model_10-0.119.h5'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 410,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": "True"
     },
     "execution_count": 410,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "keras_fc_model2.trainable"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 411,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": "False\nFalse\nFalse\nFalse\nFalse\nFalse\nFalse\nFalse\nFalse\nFalse\nFalse\n"
    }
   ],
   "source": [
    "# Setting all layers' trainable to False\n",
    "keras_fc_model2.trainable = False\n",
    "for layer in keras_fc_model2.layers:\n",
    "    print(layer.trainable)"
   ]
  },
  {
   "cell_type": "markdown",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#### Testing on one sample"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 413,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": "<tf.Tensor: shape=(1, 10), dtype=float32, numpy=\narray([[-1.0326976 , -1.3163404 ,  1.1134243 ,  1.1836618 , -4.406678  ,\n         0.57405937, -7.777385  , 16.786484  , -6.2431746 ,  1.109576  ]],\n      dtype=float32)>"
     },
     "execution_count": 413,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "last_elem = keras_fc_model2(tf.expand_dims(test_x[-1], axis=0))\n",
    "last_elem"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 414,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": "<tf.Tensor: shape=(1,), dtype=int64, numpy=array([7])>"
     },
     "execution_count": 414,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tf.argmax(last_elem, axis=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "    * Matches with the main prediction. The mistake was setting the training argument to True when defining the batch\n",
    "      norm layers"
   ]
  },
  {
   "cell_type": "markdown",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## Conclusion\n",
    "\n",
    "    * The lesson learnt is that the training argument of layers such as BatchNorm and Dropout must not be set to True in the original definition\n",
    "      of the model. It is automatically set to the appropriate value by Keras during fitting and inference.\n",
    "    \n",
    "    * The difference between training and trainable needs to be understood.\n",
    "\n",
    "    * The Keras BatchNorm layer uses mini-batch stats rather than moving/running stats during training. During inference, it uses running stats and\n",
    "      updates the same if trainable is set to True."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python (datascience)",
   "language": "python",
   "name": "datascience"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.7-final"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}