n-gram-corpus-preprocessing (1).ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name=\"corpus-preprocessing\"></a>\n",
    "# N-grams Corpus preprocessing\n",
    "\n",
    "The input corpus in this week's assignment is a continuous text that needs some preprocessing so that you can start calculating the n-gram probabilities.\n",
    "\n",
    "Some common preprocessing steps for the language models include:\n",
    "- lowercasing the text\n",
    "- remove special characters\n",
    "- split text to list of sentences\n",
    "- split sentence into list words\n",
    "\n",
    "Can you note the similarities and differences among the preprocessing steps shown during the Course 1 of this specialization?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[nltk_data] Downloading package punkt to\n",
      "[nltk_data]     C:\\Users\\anish\\AppData\\Roaming\\nltk_data...\n",
      "[nltk_data]   Unzipping tokenizers\\punkt.zip.\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import nltk               # NLP toolkit\n",
    "import re                 # Library for Regular expression operations\n",
    "\n",
    "nltk.download('punkt')    # Download the Punkt sentence tokenizer "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name=\"lowercase\"></a>\n",
    "### Lowercase\n",
    "\n",
    "Words at the beginning of a sentence and names start with a capital letter. However, when counting words, you want to treat them the same as if they appeared in the middle of a sentence. \n",
    "\n",
    "You can do that by converting the text to lowercase using [str.lowercase]\n",
    "(https://docs.python.org/3/library/stdtypes.html?highlight=split#str.lower).\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "learning% makes 'me' happy. i am happy be-cause i am learning! :)\n"
     ]
    }
   ],
   "source": [
    "# change the corpus to lowercase\n",
    "corpus = \"Learning% makes 'me' happy. I am happy be-cause I am learning! :)\"\n",
    "corpus = corpus.lower()\n",
    "\n",
    "# note that word \"learning\" will now be the same regardless of its position in the sentence\n",
    "print(corpus)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name=\"special-characters\"></a>\n",
    "### Remove special charactes\n",
    "\n",
    "Some of the characters may need to be removed from the corpus before we start processing the text to find n-grams. \n",
    "\n",
    "Often, the special characters such as double quotes '\"' or dash '-' are removed, and the interpunction such as full stop '.' or question mark '?' are left in the corpus."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "learning makes me happy. i am happy because i am learning! \n"
     ]
    }
   ],
   "source": [
    "# remove special characters\n",
    "corpus = \"learning% makes 'me' happy. i am happy be-cause i am learning! :)\"\n",
    "corpus = re.sub(r'[^a-zA-Z0-9.?! ]+', '', corpus)\n",
    "print(corpus)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that this process gets rid of the happy face made with punctuations :). Remember that for sentiment analysis, this emoticon was very important. However, we will not consider it here."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name=\"text-splitting\"></a>\n",
    "### Text splitting\n",
    "\n",
    "In the assignment, the sentences in the corpus are separated by a special delimiter \\n. You will need to split the corpus into an array of sentences using this delimiter. \n",
    "One way to do that is by using the [str.split](https://docs.python.org/3/library/stdtypes.html?highlight=split#str.split) method.\n",
    "\n",
    "\n",
    "The following examples illustrate how to use this method. The code shows:\n",
    "- how to split a string containing a date into an array of date parts\n",
    "- how to split a string with time into an array containing hours, minutes and seconds \n",
    "\n",
    "Also, note what happens if there are several back-to-back delimiters like between \"May\" and \"9\"."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "date parts = ['Sat', 'May', '', '9', '07:33:35', 'CEST', '2020']\n",
      "time parts = ['07', '33', '35']\n"
     ]
    }
   ],
   "source": [
    "# split text by a delimiter to array\n",
    "input_date=\"Sat May  9 07:33:35 CEST 2020\"\n",
    "\n",
    "# get the date parts in array\n",
    "date_parts = input_date.split(' ')\n",
    "print(f\"date parts = {date_parts}\")\n",
    "\n",
    "#get the time parts in array\n",
    "time_parts = date_parts[4].split(':')\n",
    "print(f\"time parts = {time_parts}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " This text splitting is more complicated than the tokenization process used for sentiment analysis."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name=\"sentence-tokenizing\"></a>\n",
    "### Sentence tokenizing\n",
    "\n",
    "Once you have a list of sentences, the next step is to split each sentence into a list of words.\n",
    "\n",
    "This process could be done in several ways, even using the str.split method described above, but we will use the NLTK library [nltk](https://www.nltk.org/) to help us with that.\n",
    "\n",
    "In the code assignment, you will use the method [word_tokenize](https://www.nltk.org/api/nltk.tokenize.html#nltk.tokenize.punkt.PunktLanguageVars.word_tokenize) to split your sentence into a list of words. Let us try the method in an example."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "i am happy because i am learning. -> ['i', 'am', 'happy', 'because', 'i', 'am', 'learning', '.']\n"
     ]
    }
   ],
   "source": [
    "# tokenize the sentence into an array of words\n",
    "\n",
    "sentence = 'i am happy because i am learning.'\n",
    "tokenized_sentence = nltk.word_tokenize(sentence)\n",
    "print(f'{sentence} -> {tokenized_sentence}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that the sentence is tokenized, you can work with each word in the sentence separately. This will be useful later when creating and counting N-grams. In the following code example, you will see how to find the length of each word."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " Lengths of the words: \n",
      "[('i', 1), ('am', 2), ('happy', 5), ('because', 7), ('i', 1), ('am', 2), ('learning', 8), ('.', 1)]\n"
     ]
    }
   ],
   "source": [
    "# find length of each word in the tokenized sentence\n",
    "sentence = ['i', 'am', 'happy', 'because', 'i', 'am', 'learning', '.']\n",
    "word_lengths = [(word, len(word)) for word in sentence] # Create a list with the word lengths using a list comprehension\n",
    "print(f' Lengths of the words: \\n{word_lengths}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The previous result produces a list of pairs. This is not equivalent to a dictionary."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name=\"n-grams\"></a>\n",
    "## N-grams\n",
    "<a name=\"sentence-to-ngram\"></a>\n",
    "### Sentence to n-gram\n",
    "\n",
    "The next step is to build n-grams from the tokenized sentences. \n",
    "\n",
    "A sliding window of size n-words can generate the n-grams. The window scans the list of words starting at the sentence beginning, moving by a step of one word until it reaches the end of the sentence.\n",
    "\n",
    "Here is an example method that prints all trigrams in the given sentence."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "List all trigrams of sentence: ['i', 'am', 'happy', 'because', 'i', 'am', 'learning', '.']\n",
      "\n",
      "['i', 'am', 'happy']\n",
      "['am', 'happy', 'because']\n",
      "['happy', 'because', 'i']\n",
      "['because', 'i', 'am']\n",
      "['i', 'am', 'learning']\n",
      "['am', 'learning', '.']\n"
     ]
    }
   ],
   "source": [
    "def sentence_to_trigram(tokenized_sentence):\n",
    "    \"\"\"\n",
    "    Prints all trigrams in the given tokenized sentence.\n",
    "    \n",
    "    Args:\n",
    "        tokenized_sentence: The words list.\n",
    "    \n",
    "    Returns:\n",
    "        No output\n",
    "    \"\"\"\n",
    "    # note that the last position of i is 3rd to the end\n",
    "    for i in range(len(tokenized_sentence) - 3 + 1):\n",
    "        # the sliding window starts at position i and contains 3 words\n",
    "        trigram = tokenized_sentence[i : i + 3]\n",
    "        print(trigram)\n",
    "\n",
    "tokenized_sentence = ['i', 'am', 'happy', 'because', 'i', 'am', 'learning', '.']\n",
    "\n",
    "print(f'List all trigrams of sentence: {tokenized_sentence}\\n')\n",
    "sentence_to_trigram(tokenized_sentence)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name=\"ngram-prefix\"></a>\n",
    "### Prefix of an n-gram\n",
    "\n",
    "As you saw in the lecture, the n-gram probability is often calculated based on the (n-1)-gram counts. The prefix is needed in the formula to calculate the probability of an n-gram.\n",
    "\n",
    "\\begin{equation*}\n",
    "P(w_n|w_1^{n-1})=\\frac{C(w_1^n)}{C(w_1^{n-1})}\n",
    "\\end{equation*}\n",
    "\n",
    "The following code shows how to get an (n-1)-gram  prefix from n-gram on an example of getting trigram from a 4-gram."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['i', 'am', 'happy']\n"
     ]
    }
   ],
   "source": [
    "# get trigram prefix from a 4-gram\n",
    "fourgram = ['i', 'am', 'happy','because']\n",
    "trigram = fourgram[0:-1] # Get the elements from 0, included, up to the last element, not included.\n",
    "print(trigram)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name=\"start-end-of-sentence\"></a>\n",
    "### Start and end of sentence word $<s>$ and $</s>$\n",
    "You could see in the lecture that we must add some special characters at the beginning and the end of each sentence: \n",
    "* $<s>$  at beginning\n",
    "* $</s>$ at the end\n",
    "\n",
    "For n-grams, we must prepend n-1 of characters at the begining of the sentence. \n",
    "\n",
    "Let us have a look at how you can implement this in code.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['<s>', '<s>', 'i', 'am', 'happy', 'because', 'i', 'am', 'learning', '.', '</s>']\n"
     ]
    }
   ],
   "source": [
    "# when working with trigrams, you need to prepend 2 <s> and append one </s>\n",
    "n = 3\n",
    "tokenized_sentence = ['i', 'am', 'happy', 'because', 'i', 'am', 'learning', '.']\n",
    "tokenized_sentence = ['<s>'] * (n - 1) + tokenized_sentence + ['</s>']\n",
    "print(tokenized_sentence)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "That's all for the lab for \"N-gram\" lesson of week 3."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}