Deciphering Code with Character-Level RNN.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Deciphering Code with Character-Level RNN\n",
    "\n",
    "In this notebook, we'll look at how to build a recurrent neural network and train it to decipher strings encrypted with a certain cipher.\n",
    "\n",
    "This exercise will make you familiar with the techniques of preprocessing and model-building that will come in handy when you start building more advanced models for machine translation, text summarization, and beyond.\n",
    "\n",
    "## Dataset\n",
    "The dataset we have consists of 10,000 encrypted phrases and the plaintext version of each encrypted phrase.\n",
    "\n",
    "Let's start by loading up the dataset to get more familiar with it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 122,
   "metadata": {},
   "outputs": [],
   "source": [
    "import helper\n",
    "\n",
    "codes = helper.load_data('cipher.txt')\n",
    "plaintext = helper.load_data('plaintext.txt')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now `codes` and `plaintext` are both arrays with each element being a phrase. The first three encoded phrases are:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 123,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['YMJ QNRJ NX MJW QJFXY QNPJI KWZNY , GZY YMJ GFSFSF NX RD QJFXY QNPJI .',\n",
       " 'MJ XFB F TQI DJQQTB YWZHP .',\n",
       " 'NSINF NX WFNSD IZWNSL OZSJ , FSI NY NX XTRJYNRJX BFWR NS STAJRGJW .',\n",
       " 'YMFY HFY BFX RD RTXY QTAJI FSNRFQ .',\n",
       " 'MJ INXQNPJX LWFUJKWZNY , QNRJX , FSI QJRTSX .']"
      ]
     },
     "execution_count": 123,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "codes[:5]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And their plaintext versions are:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 124,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['THE LIME IS HER LEAST LIKED FRUIT , BUT THE BANANA IS MY LEAST LIKED .',\n",
       " 'HE SAW A OLD YELLOW TRUCK .',\n",
       " 'INDIA IS RAINY DURING JUNE , AND IT IS SOMETIMES WARM IN NOVEMBER .',\n",
       " 'THAT CAT WAS MY MOST LOVED ANIMAL .',\n",
       " 'HE DISLIKES GRAPEFRUIT , LIMES , AND LEMONS .']"
      ]
     },
     "execution_count": 124,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "plaintext[:5]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 125,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "10001"
      ]
     },
     "execution_count": 125,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(codes)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 126,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "10001"
      ]
     },
     "execution_count": 126,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(plaintext)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 136,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "101"
      ]
     },
     "execution_count": 136,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "max([len(sentence) for sentence in codes])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Model Overview: Character-Level RNN\n",
    "The model we will use here is a character-level RNN since the cipher seems to work on the characer level. In a machine translation scenario, a word-level RNN is the more common choice.\n",
    "\n",
    "A character-level RNN will take as input an integer referring to a specific character and output another integer. To be able to get our model to work, we'll need to preprocess our dataset in the following steps:\n",
    " 1. Isolating each character as an array element (instead of an entire phrase, or word being the element of the array)\n",
    " 1. Tokenizing the characters so we can turn them from letters to integers and vice-versa\n",
    " 1. Padding the strings so that all the inputs and outputs can fit in matrix form\n",
    " \n",
    "To visualize this processing, let's assume either our source sequences (`codes` in this case) or target sequences (`plaintext` in this case) look like this (a list of strings):\n",
    "\n",
    "<img src=\"list_1.png\" />\n",
    "\n",
    "Since this model will be working on the character level, we'll need to separate each string into a list of characters (implicitly done by the tokenizer in this notebook):\n",
    "\n",
    "<img src=\"list_2.png\" />\n",
    "\n",
    "Then, the process of tokenization will turn each character into an integer.  Note that when you're working on the a word-level RNN (as in most machine translation examples), the tokenizer will assign an integer to each word rather than each letter, and each cell would represent a word rather than a character.\n",
    "\n",
    "<img src=\"list_3.png\" />\n",
    "\n",
    "Most machine learning platforms expect the input to be a matrix rather than a list of lists. To turn the input into a matrix, we need to find the longest member of the list, and pad all shorter sequences with 0. Assuming 'and two' is the longest sequence in this example, the matrix ends up looking like this:\n",
    "\n",
    "<img src=\"padded_list.png\" />\n",
    " \n",
    "## Preprocessing (IMPLEMENT)\n",
    "For a neural network to predict on text data, it first has to be turned into data it can understand. Text data like \"dog\" is a sequence of ASCII character encodings.  Since a neural network is a series of multiplication and addition operations, the input data needs to be number(s).\n",
    "\n",
    "We can turn each character into a number or each word into a number.  These are called character and word ids, respectively.  Character ids are used for character level models that generate text predictions for each character.  A word level model uses word ids that generate text predictions for each word.  Word level models tend to learn better.\n",
    "\n",
    "Turn each sentence into a sequence of words ids using Keras's [`Tokenizer`](https://keras.io/preprocessing/text/#tokenizer) function. Since we're working on the character level, make sure to set the `char_level` flag to the appropriate value. Then, fit the tokenizer on x."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 127,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{' ': 1, 'e': 2, 'o': 3, 'i': 4, 's': 5, 'h': 6, 'r': 7, 'y': 8, 'u': 9, 'c': 10, 'n': 11, 't': 12, 'a': 13, 'p': 14, '.': 15, 'T': 16, 'q': 17, 'k': 18, 'w': 19, 'f': 20, 'x': 21, 'm': 22, 'v': 23, 'l': 24, 'z': 25, 'd': 26, 'g': 27, 'b': 28, 'j': 29, 'B': 30, 'J': 31, ',': 32}\n",
      "\n",
      "Sequence 1 in x\n",
      "  Input:  The quick brown fox jumps over the lazy dog .\n",
      "  Output: [16, 6, 2, 1, 17, 9, 4, 10, 18, 1, 28, 7, 3, 19, 11, 1, 20, 3, 21, 1, 29, 9, 22, 14, 5, 1, 3, 23, 2, 7, 1, 12, 6, 2, 1, 24, 13, 25, 8, 1, 26, 3, 27, 1, 15]\n",
      "Sequence 2 in x\n",
      "  Input:  By Jove , my quick study of lexicography won a prize .\n",
      "  Output: [30, 8, 1, 31, 3, 23, 2, 1, 32, 1, 22, 8, 1, 17, 9, 4, 10, 18, 1, 5, 12, 9, 26, 8, 1, 3, 20, 1, 24, 2, 21, 4, 10, 3, 27, 7, 13, 14, 6, 8, 1, 19, 3, 11, 1, 13, 1, 14, 7, 4, 25, 2, 1, 15]\n",
      "Sequence 3 in x\n",
      "  Input:  This is a short sentence .\n",
      "  Output: [16, 6, 4, 5, 1, 4, 5, 1, 13, 1, 5, 6, 3, 7, 12, 1, 5, 2, 11, 12, 2, 11, 10, 2, 1, 15]\n"
     ]
    }
   ],
   "source": [
    "from keras.preprocessing.text import Tokenizer\n",
    "\n",
    "\n",
    "def tokenize(x):\n",
    "    \"\"\"\n",
    "    Tokenize x\n",
    "    :param x: List of sentences/strings to be tokenized\n",
    "    :return: Tuple of (tokenized x data, tokenizer used to tokenize x)\n",
    "    \n",
    "    tf.keras.preprocessing.text.Tokenizer(num_words=None,\n",
    "    filters='!\"#$%&()*+,-./:;<=>?@[\\\\]^_`{|}~\\t\\n',\n",
    "    lower=True,\n",
    "    split=\" \",\n",
    "    char_level=False,\n",
    "    oov_token=None,\n",
    "    document_count=0,\n",
    "    **kwargs\n",
    "    )\n",
    "    \"\"\"\n",
    "    # TODO: Implement\n",
    "    x_tk = Tokenizer(char_level=True)\n",
    "    x_tk.fit_on_texts(x)                 # because input is text, not sequence (list of integer tokens)\n",
    "\n",
    "    return x_tk.texts_to_sequences(x), x_tk\n",
    "\n",
    "# Tokenize Example output\n",
    "text_sentences = [\n",
    "    'The quick brown fox jumps over the lazy dog .',\n",
    "    'By Jove , my quick study of lexicography won a prize .',\n",
    "    'This is a short sentence .']\n",
    "text_tokenized, text_tokenizer = tokenize(text_sentences)\n",
    "print(text_tokenizer.word_index)\n",
    "print()\n",
    "for sample_i, (sent, token_sent) in enumerate(zip(text_sentences, text_tokenized)):\n",
    "    print('Sequence {} in x'.format(sample_i + 1))\n",
    "    print('  Input:  {}'.format(sent))\n",
    "    print('  Output: {}'.format(token_sent))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Padding (IMPLEMENTATION)\n",
    "When batching the sequence of word ids together, each sequence needs to be the same length.  Since sentences are dynamic in length, we can add padding to the end of the sequences to make them the same length.\n",
    "\n",
    "Make sure all the cipher sequences have the same length and all the plaintext sequences have the same length by adding padding to the **end** of each sequence using Keras's [`pad_sequences`](https://keras.io/preprocessing/sequence/#pad_sequences) function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 128,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Sequence 1 in x\n",
      "  Input:  [16  6  2  1 17  9  4 10 18  1 28  7  3 19 11  1 20  3 21  1 29  9 22 14  5\n",
      "  1  3 23  2  7  1 12  6  2  1 24 13 25  8  1 26  3 27  1 15]\n",
      "  Output: [16  6  2  1 17  9  4 10 18  1 28  7  3 19 11  1 20  3 21  1 29  9 22 14  5\n",
      "  1  3 23  2  7  1 12  6  2  1 24 13 25  8  1 26  3 27  1 15  0  0  0  0  0\n",
      "  0  0  0  0]\n",
      "Sequence 2 in x\n",
      "  Input:  [30  8  1 31  3 23  2  1 32  1 22  8  1 17  9  4 10 18  1  5 12  9 26  8  1\n",
      "  3 20  1 24  2 21  4 10  3 27  7 13 14  6  8  1 19  3 11  1 13  1 14  7  4\n",
      " 25  2  1 15]\n",
      "  Output: [30  8  1 31  3 23  2  1 32  1 22  8  1 17  9  4 10 18  1  5 12  9 26  8  1\n",
      "  3 20  1 24  2 21  4 10  3 27  7 13 14  6  8  1 19  3 11  1 13  1 14  7  4\n",
      " 25  2  1 15]\n",
      "Sequence 3 in x\n",
      "  Input:  [16  6  4  5  1  4  5  1 13  1  5  6  3  7 12  1  5  2 11 12  2 11 10  2  1\n",
      " 15]\n",
      "  Output: [16  6  4  5  1  4  5  1 13  1  5  6  3  7 12  1  5  2 11 12  2 11 10  2  1\n",
      " 15  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0\n",
      "  0  0  0  0]\n"
     ]
    }
   ],
   "source": [
    "import numpy as np\n",
    "from keras.preprocessing.sequence import pad_sequences\n",
    "\n",
    "\n",
    "def pad(x, length=None):\n",
    "    \"\"\"\n",
    "    Pad x\n",
    "    :param x: List of sequences.\n",
    "    :param length: Length to pad the sequence to.  If None, use length of longest sequence in x.\n",
    "    :return: Padded numpy array of sequences\n",
    "    \n",
    "    tf.keras.preprocessing.sequence.pad_sequences(\n",
    "    sequences, maxlen=None, dtype='int32', padding='pre', truncating='pre',\n",
    "    value=0.0)\n",
    "    \"\"\"\n",
    "    # TODO: Implement\n",
    "    # Find the length of the longest string in the dataset.\n",
    "    if length is None:\n",
    "        length = max([len(sentence) for sentence in x])\n",
    "    # Then, pass it to pad_sentences as the maxlen parameter\n",
    "    \n",
    "    return pad_sequences(x, maxlen=length, padding=\"post\", truncating=\"post\",)\n",
    "\n",
    "# Pad Tokenized output\n",
    "test_pad = pad(text_tokenized)\n",
    "for sample_i, (token_sent, pad_sent) in enumerate(zip(text_tokenized, test_pad)):\n",
    "    print('Sequence {} in x'.format(sample_i + 1))\n",
    "    print('  Input:  {}'.format(np.array(token_sent)))\n",
    "    print('  Output: {}'.format(pad_sent))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Preprocess Pipeline\n",
    "Your focus for this project is to build neural network architecture, so we won't ask you to create a preprocess pipeline.  Instead, we've provided you with the implementation of the `preprocess` function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 129,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Data Preprocessed\n"
     ]
    }
   ],
   "source": [
    "def preprocess(x, y):\n",
    "    \"\"\"\n",
    "    Preprocess x and y\n",
    "    :param x: Feature List of sentences\n",
    "    :param y: Label List of sentences\n",
    "    :return: Tuple of (Preprocessed x, Preprocessed y, x tokenizer, y tokenizer)\n",
    "    \"\"\"\n",
    "    preprocess_x, x_tk = tokenize(x)\n",
    "    preprocess_y, y_tk = tokenize(y)\n",
    "\n",
    "    preprocess_x = pad(preprocess_x)\n",
    "    preprocess_y = pad(preprocess_y)\n",
    "\n",
    "    # Keras's sparse_categorical_crossentropy function requires the labels to be in 3 dimensions\n",
    "    preprocess_y = preprocess_y.reshape(*preprocess_y.shape, 1)\n",
    "\n",
    "    return preprocess_x, preprocess_y, x_tk, y_tk\n",
    "\n",
    "preproc_code_sentences, preproc_plaintext_sentences, code_tokenizer, plaintext_tokenizer =\\\n",
    "    preprocess(codes, plaintext)\n",
    "\n",
    "print('Data Preprocessed')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 130,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ 5, 14,  3,  1, 10,  2, 13,  3,  1,  2,  4,  1, 14,  3,  6,  1, 10,\n",
       "        3,  8,  4,  5,  1, 10,  2, 25,  3, 11,  1, 20,  6,  9,  2,  5,  1,\n",
       "       18,  1, 17,  9,  5,  1,  5, 14,  3,  1, 17,  8,  7,  8,  7,  8,  1,\n",
       "        2,  4,  1, 13, 15,  1, 10,  3,  8,  4,  5,  1, 10,  2, 25,  3, 11,\n",
       "        1, 19,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,\n",
       "        0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0], dtype=int32)"
      ]
     },
     "execution_count": 130,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "preproc_code_sentences[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 131,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "32"
      ]
     },
     "execution_count": 131,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(code_tokenizer.word_index)+1  # +1 for 0?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 132,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "32"
      ]
     },
     "execution_count": 132,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(plaintext_tokenizer.word_index)+1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 133,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{' ': 1,\n",
       " 'I': 2,\n",
       " 'E': 3,\n",
       " 'S': 4,\n",
       " 'T': 5,\n",
       " 'R': 6,\n",
       " 'N': 7,\n",
       " 'A': 8,\n",
       " 'U': 9,\n",
       " 'L': 10,\n",
       " 'D': 11,\n",
       " 'O': 12,\n",
       " 'M': 13,\n",
       " 'H': 14,\n",
       " 'Y': 15,\n",
       " 'G': 16,\n",
       " 'B': 17,\n",
       " ',': 18,\n",
       " '.': 19,\n",
       " 'F': 20,\n",
       " 'P': 21,\n",
       " 'C': 22,\n",
       " 'V': 23,\n",
       " 'W': 24,\n",
       " 'K': 25,\n",
       " 'J': 26,\n",
       " 'X': 27,\n",
       " 'Q': 28,\n",
       " 'Z': 29,\n",
       " '?': 30,\n",
       " \"'\": 31}"
      ]
     },
     "execution_count": 133,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "plaintext_tokenizer.word_index"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "By default, the output of a RNN layer contains a single vector per sample. This vector is the RNN cell output corresponding to the last timestep, containing information about the entire input sequence. The shape of this output is (batch_size, units) where units corresponds to the units argument passed to the layer's constructor.\n",
    "\n",
    "A RNN layer can also return the entire sequence of outputs for each sample (one vector per timestep per sample), if you set return_sequences=True. The shape of this output is (batch_size, timesteps, units).\n",
    "\n",
    "In addition, a RNN layer can return its final internal state(s). The returned states can be used to resume the RNN execution later, or to initialize another RNN. This setting is commonly used in the encoder-decoder sequence-to-sequence model, where the encoder final state is used as the initial state of the decoder.\n",
    "\n",
    "To configure a RNN layer to return its internal state, set the return_state parameter to True when creating the layer. Note that LSTM has 2 state tensors, but GRU only has one.\n",
    "\n",
    "To configure the initial state of the layer, just call the layer with additional keyword argument initial_state. Note that the shape of the state needs to match the unit size of the layer\n",
    "\n",
    "Return states in addition to output:\n",
    "- output, state_h, state_c = layers.LSTM(64, return_state=True, name=\"encoder\")(x)\n",
    "- encoder_state = [state_h, state_c]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 134,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(10001, 101)"
      ]
     },
     "execution_count": 134,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "preproc_code_sentences.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 135,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(10001, 101, 1)"
      ]
     },
     "execution_count": 135,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "preproc_plaintext_sentences.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# LSTM"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 137,
   "metadata": {},
   "outputs": [],
   "source": [
    "from keras.layers import GRU, Input, Dense, TimeDistributed, RNN, LSTM\n",
    "from keras.models import Model, Sequential\n",
    "from keras.layers import Activation\n",
    "from keras.optimizers import Adam\n",
    "from keras.losses import sparse_categorical_crossentropy\n",
    "import tensorflow\n",
    "\n",
    "\n",
    "def simple_model(input_shape, output_sequence_length, code_vocab_size, plaintext_vocab_size):\n",
    "    \"\"\"\n",
    "    Build and train a basic RNN on x and y\n",
    "    :param input_shape: Tuple of input shape\n",
    "    :param output_sequence_length: Length of output sequence\n",
    "    :param code_vocab_size: Number of unique code characters in the dataset\n",
    "    :param plaintext_vocab_size: Number of unique plaintext characters in the dataset\n",
    "    :return: Keras model built, but not trained\n",
    "    \"\"\"\n",
    "    # TODO: Build the model\n",
    "    \n",
    "    x = Input(shape=input_shape[1:])   # shape(none,101,1) ie \n",
    "    \n",
    "    seq = LSTM(units= 64, return_sequences = True, activation=\"tanh\", name='Layer1')(x)  # output must be batchsize x timesteps x units\n",
    "    \n",
    "    output = TimeDistributed(Dense(units = plaintext_vocab_size, activation='softmax', name='Layer2'))(seq)\n",
    "    \n",
    "    model = Model(inputs = x, outputs = output)\n",
    "    \n",
    "    model.compile(optimizer='adam', loss= sparse_categorical_crossentropy, metrics=['accuracy'])\n",
    "    \n",
    "    model.summary()\n",
    "    \n",
    "    return model\n",
    "\n",
    "\n",
    "# Reshaping the input to work with a basic RNN\n",
    "tmp_x = pad(preproc_code_sentences, preproc_plaintext_sentences.shape[1]) # pad code sequences with maxlen 54: shape=10001x54\n",
    "tmp_x = tmp_x.reshape((-1, preproc_plaintext_sentences.shape[-2], 1))     # reshape padded code seq in 10001 x 54 x 1\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 138,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(10001, 101, 1)"
      ]
     },
     "execution_count": 138,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tmp_x.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 149,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "input_21 (InputLayer)        (None, 101, 1)            0         \n",
      "_________________________________________________________________\n",
      "Layer1 (LSTM)                (None, 101, 64)           16896     \n",
      "_________________________________________________________________\n",
      "time_distributed_18 (TimeDis (None, 101, 32)           2080      \n",
      "=================================================================\n",
      "Total params: 18,976\n",
      "Trainable params: 18,976\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "# Train the neural network\n",
    "simple_rnn_model = simple_model(\n",
    "    tmp_x.shape,\n",
    "    preproc_plaintext_sentences.shape[1],\n",
    "    len(code_tokenizer.word_index)+1,\n",
    "    len(plaintext_tokenizer.word_index)+1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 150,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "TensorShape([Dimension(None), Dimension(None), Dimension(64)])"
      ]
     },
     "execution_count": 150,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "simple_rnn_model.get_layer(name=\"Layer1\").output.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 153,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 8000 samples, validate on 2001 samples\n",
      "Epoch 1/4\n",
      "8000/8000 [==============================] - 36s 5ms/step - loss: 0.3267 - acc: 0.9262 - val_loss: 0.2858 - val_acc: 0.9362\n",
      "Epoch 2/4\n",
      "8000/8000 [==============================] - 36s 5ms/step - loss: 0.2529 - acc: 0.9448 - val_loss: 0.2237 - val_acc: 0.9532\n",
      "Epoch 3/4\n",
      "8000/8000 [==============================] - 36s 5ms/step - loss: 0.2005 - acc: 0.9575 - val_loss: 0.1791 - val_acc: 0.9611\n",
      "Epoch 4/4\n",
      "8000/8000 [==============================] - 36s 4ms/step - loss: 0.1613 - acc: 0.9665 - val_loss: 0.1451 - val_acc: 0.9706\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.callbacks.History at 0x7f1b078e1e48>"
      ]
     },
     "execution_count": 153,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "simple_rnn_model.fit(tmp_x, preproc_plaintext_sentences, batch_size=32, epochs=4, validation_split=0.2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 154,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "`logits_to_text` function loaded.\n",
      "T H E   L I M E   I S   H E R   L E A S T   L I K E D   F R U I T   ,   B U T   T H E   G A N A N A   I S   M O   L E A S T   L I K E D   . <PAD> <PAD> <PAD> <PAD> <PAD> <PAD> <PAD> <PAD> <PAD> <PAD> <PAD> <PAD> <PAD> <PAD> <PAD> <PAD> <PAD> <PAD> <PAD> <PAD> <PAD> <PAD> <PAD> <PAD> <PAD> <PAD> <PAD> <PAD> <PAD> <PAD> <PAD>\n"
     ]
    }
   ],
   "source": [
    "def logits_to_text(logits, tokenizer):\n",
    "    \"\"\"\n",
    "    Turn logits from a neural network into text using the tokenizer\n",
    "    :param logits: Logits from a neural network\n",
    "    :param tokenizer: Keras Tokenizer fit on the labels\n",
    "    :return: String that represents the text of the logits\n",
    "    \"\"\"\n",
    "    index_to_words = {id: word for word, id in tokenizer.word_index.items()}\n",
    "    index_to_words[0] = '<PAD>'\n",
    "\n",
    "    return ' '.join([index_to_words[prediction] for prediction in np.argmax(logits, 1)])\n",
    "\n",
    "print('`logits_to_text` function loaded.')\n",
    "\n",
    "print(logits_to_text(simple_rnn_model.predict(tmp_x[:1])[0], plaintext_tokenizer))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 89,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'THE LIME IS HER LEAST LIKED FRUIT , BUT THE BANANA IS MY LEAST LIKED .'"
      ]
     },
     "execution_count": 89,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "plaintext[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And there it is. The RNN was able to learn this basic character-level cipher (which was a simple [Caesar cipher](https://en.wikipedia.org/wiki/Caesar_cipher). If you want a bigger cryptography challenge, check out [Learning the Enigma with Recurrent Neural Networks](https://greydanus.github.io/2017/01/07/enigma-rnn/). "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# GRU"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 155,
   "metadata": {},
   "outputs": [],
   "source": [
    "def simple_model1(input_shape, output_sequence_length, code_vocab_size, plaintext_vocab_size):\n",
    "    \"\"\"\n",
    "    Build and train a basic RNN on x and y\n",
    "    :param input_shape: Tuple of input shape\n",
    "    :param output_sequence_length: Length of output sequence\n",
    "    :param code_vocab_size: Number of unique code characters in the dataset\n",
    "    :param plaintext_vocab_size: Number of unique plaintext characters in the dataset\n",
    "    :return: Keras model built, but not trained\n",
    "    \"\"\"\n",
    "    # TODO: Build the model\n",
    "    \n",
    "    x = Input(shape=input_shape[1:])   # shape(none,54,1) ie \n",
    "    \n",
    "    seq = GRU(units= 64, return_sequences = True, activation=\"tanh\", name='Layer1')(x)  # output must be batchsize x timesteps x units\n",
    "    \n",
    "    output = TimeDistributed(Dense(units = plaintext_vocab_size, activation='softmax', name='Layer2'))(seq)\n",
    "    \n",
    "    model = Model(inputs = x, outputs = output)\n",
    "    \n",
    "    model.compile(optimizer='adam', loss= sparse_categorical_crossentropy, metrics=['accuracy'])\n",
    "    \n",
    "    model.summary()\n",
    "    \n",
    "    return model\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 156,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "input_22 (InputLayer)        (None, 101, 1)            0         \n",
      "_________________________________________________________________\n",
      "Layer1 (GRU)                 (None, 101, 64)           12672     \n",
      "_________________________________________________________________\n",
      "time_distributed_19 (TimeDis (None, 101, 32)           2080      \n",
      "=================================================================\n",
      "Total params: 14,752\n",
      "Trainable params: 14,752\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "# Train the neural network\n",
    "simple_rnn_model1 = simple_model1(\n",
    "    tmp_x.shape,\n",
    "    preproc_plaintext_sentences.shape[1],\n",
    "    len(code_tokenizer.word_index)+1,\n",
    "    len(plaintext_tokenizer.word_index)+1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 157,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 8000 samples, validate on 2001 samples\n",
      "Epoch 1/16\n",
      "8000/8000 [==============================] - 21s 3ms/step - loss: 2.5627 - acc: 0.4277 - val_loss: 1.6755 - val_acc: 0.4654\n",
      "Epoch 2/16\n",
      "8000/8000 [==============================] - 20s 2ms/step - loss: 1.4602 - acc: 0.5440 - val_loss: 1.2939 - val_acc: 0.6118\n",
      "Epoch 3/16\n",
      "8000/8000 [==============================] - 20s 2ms/step - loss: 1.1693 - acc: 0.6538 - val_loss: 1.0674 - val_acc: 0.6844\n",
      "Epoch 4/16\n",
      "8000/8000 [==============================] - 20s 2ms/step - loss: 0.9792 - acc: 0.7207 - val_loss: 0.9052 - val_acc: 0.7441\n",
      "Epoch 5/16\n",
      "8000/8000 [==============================] - 20s 2ms/step - loss: 0.8352 - acc: 0.7654 - val_loss: 0.7752 - val_acc: 0.7864\n",
      "Epoch 6/16\n",
      "8000/8000 [==============================] - 20s 2ms/step - loss: 0.7156 - acc: 0.8070 - val_loss: 0.6637 - val_acc: 0.8269\n",
      "Epoch 7/16\n",
      "8000/8000 [==============================] - 20s 2ms/step - loss: 0.6139 - acc: 0.8448 - val_loss: 0.5714 - val_acc: 0.8597\n",
      "Epoch 8/16\n",
      "8000/8000 [==============================] - 20s 2ms/step - loss: 0.5301 - acc: 0.8711 - val_loss: 0.4947 - val_acc: 0.8798\n",
      "Epoch 9/16\n",
      "8000/8000 [==============================] - 20s 3ms/step - loss: 0.4594 - acc: 0.8908 - val_loss: 0.4295 - val_acc: 0.8994\n",
      "Epoch 10/16\n",
      "8000/8000 [==============================] - 20s 3ms/step - loss: 0.4001 - acc: 0.9074 - val_loss: 0.3752 - val_acc: 0.9135\n",
      "Epoch 11/16\n",
      "8000/8000 [==============================] - 20s 2ms/step - loss: 0.3504 - acc: 0.9200 - val_loss: 0.3294 - val_acc: 0.9253\n",
      "Epoch 12/16\n",
      "8000/8000 [==============================] - 20s 2ms/step - loss: 0.3081 - acc: 0.9327 - val_loss: 0.2902 - val_acc: 0.9381\n",
      "Epoch 13/16\n",
      "8000/8000 [==============================] - 20s 2ms/step - loss: 0.2730 - acc: 0.9433 - val_loss: 0.2581 - val_acc: 0.9491\n",
      "Epoch 14/16\n",
      "8000/8000 [==============================] - 20s 2ms/step - loss: 0.2436 - acc: 0.9517 - val_loss: 0.2308 - val_acc: 0.9552\n",
      "Epoch 15/16\n",
      "8000/8000 [==============================] - 20s 2ms/step - loss: 0.2184 - acc: 0.9573 - val_loss: 0.2072 - val_acc: 0.9609\n",
      "Epoch 16/16\n",
      "8000/8000 [==============================] - 20s 2ms/step - loss: 0.1966 - acc: 0.9629 - val_loss: 0.1868 - val_acc: 0.9648\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.callbacks.History at 0x7f1b0bbe1eb8>"
      ]
     },
     "execution_count": 157,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "simple_rnn_model1.fit(tmp_x, preproc_plaintext_sentences, batch_size=128, epochs=16, validation_split=0.2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 158,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "T H E   L I M E   I S   H E R   L E A S T   L I K E D   F R U I T   ,   B U T   T H E   B A N A N A   I S   M H   L E A S T   L I K E D   . <PAD> <PAD> <PAD> <PAD> <PAD> <PAD> <PAD> <PAD> <PAD> <PAD> <PAD> <PAD> <PAD> <PAD> <PAD> <PAD> <PAD> <PAD> <PAD> <PAD> <PAD> <PAD> <PAD> <PAD> <PAD> <PAD> <PAD> <PAD> <PAD> <PAD> <PAD>\n"
     ]
    }
   ],
   "source": [
    "print(logits_to_text(simple_rnn_model1.predict(tmp_x[:1])[0], plaintext_tokenizer))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# SimpleRNN"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 165,
   "metadata": {},
   "outputs": [],
   "source": [
    "from keras.layers import SimpleRNN \n",
    "def simple_model2(input_shape, output_sequence_length, code_vocab_size, plaintext_vocab_size):\n",
    "    \"\"\"\n",
    "    Build and train a basic RNN on x and y\n",
    "    :param input_shape: Tuple of input shape\n",
    "    :param output_sequence_length: Length of output sequence\n",
    "    :param code_vocab_size: Number of unique code characters in the dataset\n",
    "    :param plaintext_vocab_size: Number of unique plaintext characters in the dataset\n",
    "    :return: Keras model built, but not trained\n",
    "    \"\"\"\n",
    "    # TODO: Build the model\n",
    "    \n",
    "    x = Input(shape=input_shape[1:])   # shape(none,54,1) ie \n",
    "    \n",
    "    seq = SimpleRNN(units= 64, return_sequences = True, activation=\"tanh\", name='Layer1')(x)  # output must be batchsize x timesteps x units\n",
    "    \n",
    "    output = TimeDistributed(Dense(units = plaintext_vocab_size, activation='softmax', name='Layer2'))(seq)\n",
    "    \n",
    "    model = Model(inputs = x, outputs = output)\n",
    "    \n",
    "    model.compile(optimizer='adam', loss= sparse_categorical_crossentropy, metrics=['accuracy'])\n",
    "    \n",
    "    model.summary()\n",
    "    \n",
    "    return model\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 166,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "input_25 (InputLayer)        (None, 101, 1)            0         \n",
      "_________________________________________________________________\n",
      "Layer1 (SimpleRNN)           (None, 101, 64)           4224      \n",
      "_________________________________________________________________\n",
      "time_distributed_21 (TimeDis (None, 101, 32)           2080      \n",
      "=================================================================\n",
      "Total params: 6,304\n",
      "Trainable params: 6,304\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "# Train the neural network\n",
    "simple_rnn_model2 = simple_model2(\n",
    "    tmp_x.shape,\n",
    "    preproc_plaintext_sentences.shape[1],\n",
    "    len(code_tokenizer.word_index)+1,\n",
    "    len(plaintext_tokenizer.word_index)+1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 167,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 8000 samples, validate on 2001 samples\n",
      "Epoch 1/16\n",
      "8000/8000 [==============================] - 12s 2ms/step - loss: 2.1848 - acc: 0.4288 - val_loss: 1.5287 - val_acc: 0.5425\n",
      "Epoch 2/16\n",
      "8000/8000 [==============================] - 11s 1ms/step - loss: 1.3699 - acc: 0.5847 - val_loss: 1.2494 - val_acc: 0.6178\n",
      "Epoch 3/16\n",
      "8000/8000 [==============================] - 11s 1ms/step - loss: 1.1558 - acc: 0.6431 - val_loss: 1.0809 - val_acc: 0.6626\n",
      "Epoch 4/16\n",
      "8000/8000 [==============================] - 11s 1ms/step - loss: 1.0124 - acc: 0.6913 - val_loss: 0.9570 - val_acc: 0.7188\n",
      "Epoch 5/16\n",
      "8000/8000 [==============================] - 11s 1ms/step - loss: 0.9021 - acc: 0.7386 - val_loss: 0.8567 - val_acc: 0.7708\n",
      "Epoch 6/16\n",
      "8000/8000 [==============================] - 11s 1ms/step - loss: 0.8102 - acc: 0.7834 - val_loss: 0.7715 - val_acc: 0.7980\n",
      "Epoch 7/16\n",
      "8000/8000 [==============================] - 11s 1ms/step - loss: 0.7315 - acc: 0.8086 - val_loss: 0.6982 - val_acc: 0.8148\n",
      "Epoch 8/16\n",
      "8000/8000 [==============================] - 11s 1ms/step - loss: 0.6633 - acc: 0.8232 - val_loss: 0.6343 - val_acc: 0.8317\n",
      "Epoch 9/16\n",
      "8000/8000 [==============================] - 11s 1ms/step - loss: 0.6036 - acc: 0.8434 - val_loss: 0.5779 - val_acc: 0.8511\n",
      "Epoch 10/16\n",
      "8000/8000 [==============================] - 11s 1ms/step - loss: 0.5508 - acc: 0.8589 - val_loss: 0.5281 - val_acc: 0.8623\n",
      "Epoch 11/16\n",
      "8000/8000 [==============================] - 11s 1ms/step - loss: 0.5036 - acc: 0.8696 - val_loss: 0.4902 - val_acc: 0.8743\n",
      "Epoch 12/16\n",
      "8000/8000 [==============================] - 11s 1ms/step - loss: 0.4617 - acc: 0.8819 - val_loss: 0.4507 - val_acc: 0.8885\n",
      "Epoch 13/16\n",
      "8000/8000 [==============================] - 11s 1ms/step - loss: 0.4248 - acc: 0.8931 - val_loss: 0.4082 - val_acc: 0.8957\n",
      "Epoch 14/16\n",
      "8000/8000 [==============================] - 11s 1ms/step - loss: 0.3913 - acc: 0.9037 - val_loss: 0.3836 - val_acc: 0.9079\n",
      "Epoch 15/16\n",
      "8000/8000 [==============================] - 11s 1ms/step - loss: 0.3619 - acc: 0.9118 - val_loss: 0.3560 - val_acc: 0.9134\n",
      "Epoch 16/16\n",
      "8000/8000 [==============================] - 11s 1ms/step - loss: 0.3356 - acc: 0.9196 - val_loss: 0.3311 - val_acc: 0.9227\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.callbacks.History at 0x7f1b06464668>"
      ]
     },
     "execution_count": 167,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "simple_rnn_model2.fit(tmp_x, preproc_plaintext_sentences, batch_size=128, epochs=16, validation_split=0.2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 168,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "T H E   L I M E   I S   M E R   L E A S T   L I K E D   F R U I T   ,   B U T   T H E   . A N A N A   I S   M H   D E A S T   L I K E D   . <PAD> <PAD> <PAD> <PAD> <PAD> <PAD> <PAD> <PAD> <PAD> <PAD> <PAD> <PAD> <PAD> <PAD> <PAD> <PAD> <PAD> <PAD> <PAD> <PAD> <PAD> <PAD> <PAD> <PAD> <PAD> <PAD> <PAD> <PAD> <PAD> <PAD> <PAD>\n"
     ]
    }
   ],
   "source": [
    "print(logits_to_text(simple_rnn_model2.predict(tmp_x[:1])[0], plaintext_tokenizer))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Overall:\n",
    "- the GRU has the least errors (only 1) but required higher training time with 16 epochs for 128 batch size\n",
    "- SimpleRNN with the same training has much more errors\n",
    "- Finally LSTM has 2 errors only and was trained with only 4 epochs and 64 batch size which is twice as less as the GRU\n",
    "\n",
    "Conclusion: best performance for LSTM with GRU next"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}