n_gram_corpus_preprocessing_(1).ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/dr-mushtaq/Natural-language-processing/blob/main/n_gram_corpus_preprocessing_(1).ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "oi4p28_AjAZ9"
      },
      "source": [
        "\n",
        "#<p style=\"font-family:JetBrains Mono; font-weight:normal; letter-spacing: 1px; color:#207d06; font-size:100%; text-align:left;padding: 0px; border-bottom: 3px solid #207d06;\">**N-grams Corpus preprocessing**</p>\n",
        "\n",
        "The input corpus in this week's assignment is a continuous text that needs some preprocessing so that you can start calculating the n-gram probabilities.\n",
        "\n",
        "Some common preprocessing steps for the language models include:\n",
        "- lowercasing the text\n",
        "- remove special characters\n",
        "- split text to list of sentences\n",
        "- split sentence into list words\n",
        "\n",
        "Can you note the similarities and differences among the preprocessing steps shown during the Course 1 of this specialization?"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "metadata": {
        "id": "1oIdDk39jAaC",
        "outputId": "862fedc4-8d1d-4432-873b-a632b778b14e",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "[nltk_data] Downloading package punkt to /root/nltk_data...\n",
            "[nltk_data]   Unzipping tokenizers/punkt.zip.\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "True"
            ]
          },
          "metadata": {},
          "execution_count": 1
        }
      ],
      "source": [
        "import nltk               # NLP toolkit\n",
        "import re                 # Library for Regular expression operations\n",
        "\n",
        "nltk.download('punkt')    # Download the Punkt sentence tokenizer"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cgr-QPfZjAaI"
      },
      "source": [
        "#<p style=\"font-family:JetBrains Mono; font-weight:normal; letter-spacing: 1px; color:#207d06; font-size:100%; text-align:left;padding: 0px; border-bottom: 3px solid #207d06;\">**Lowercase**</p>\n",
        "\n",
        "Words at the beginning of a sentence and names start with a capital letter. However, when counting words, you want to treat them the same as if they appeared in the middle of a sentence.\n",
        "\n",
        "You can do that by converting the text to lowercase using [str.lowercase]\n",
        "(https://docs.python.org/3/library/stdtypes.html?highlight=split#str.lower).\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 2,
      "metadata": {
        "scrolled": true,
        "id": "VlV5hSnkjAaJ",
        "outputId": "effa8293-ddb6-487d-c2d7-a5facef12077",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "learning% makes 'me' happy. i am happy be-cause i am learning! :)\n"
          ]
        }
      ],
      "source": [
        "# change the corpus to lowercase\n",
        "corpus = \"Learning% makes 'me' happy. I am happy be-cause I am learning! :)\"\n",
        "corpus = corpus.lower()\n",
        "\n",
        "# note that word \"learning\" will now be the same regardless of its position in the sentence\n",
        "print(corpus)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Ka5dzxAzjAaK"
      },
      "source": [
        "#<p style=\"font-family:JetBrains Mono; font-weight:normal; letter-spacing: 1px; color:#207d06; font-size:100%; text-align:left;padding: 0px; border-bottom: 3px solid #207d06;\">**Remove special charactes**</p>\n",
        "\n",
        "Some of the characters may need to be removed from the corpus before we start processing the text to find n-grams.\n",
        "\n",
        "Often, the special characters such as double quotes '\"' or dash '-' are removed, and the interpunction such as full stop '.' or question mark '?' are left in the corpus."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 3,
      "metadata": {
        "id": "q00r6BPYjAaK",
        "outputId": "80ebf9e9-a835-4152-e4ef-11a02378769d",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "learning makes me happy. i am happy because i am learning! \n"
          ]
        }
      ],
      "source": [
        "# remove special characters\n",
        "corpus = \"learning% makes 'me' happy. i am happy be-cause i am learning! :)\"\n",
        "corpus = re.sub(r'[^a-zA-Z0-9.?! ]+', '', corpus)\n",
        "print(corpus)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "riaOAWZsjAaK"
      },
      "source": [
        "Note that this process gets rid of the happy face made with punctuations :). Remember that for sentiment analysis, this emoticon was very important. However, we will not consider it here."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "in9pDtLOjAaL"
      },
      "source": [
        "#<p style=\"font-family:JetBrains Mono; font-weight:normal; letter-spacing: 1px; color:#207d06; font-size:100%; text-align:left;padding: 0px; border-bottom: 3px solid #207d06;\">**Text splitting**</p>\n",
        "\n",
        "In the assignment, the sentences in the corpus are separated by a special delimiter \\n. You will need to split the corpus into an array of sentences using this delimiter.\n",
        "One way to do that is by using the [str.split](https://docs.python.org/3/library/stdtypes.html?highlight=split#str.split) method.\n",
        "\n",
        "\n",
        "The following examples illustrate how to use this method. The code shows:\n",
        "- how to split a string containing a date into an array of date parts\n",
        "- how to split a string with time into an array containing hours, minutes and seconds\n",
        "\n",
        "Also, note what happens if there are several back-to-back delimiters like between \"May\" and \"9\"."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 4,
      "metadata": {
        "id": "tAK7_ipwjAaL",
        "outputId": "9cf3f8b9-c3dc-42de-b6f7-e2b5d8f1f0d4",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "date parts = ['Sat', 'May', '', '9', '07:33:35', 'CEST', '2020']\n",
            "time parts = ['07', '33', '35']\n"
          ]
        }
      ],
      "source": [
        "# split text by a delimiter to array\n",
        "input_date=\"Sat May  9 07:33:35 CEST 2020\"\n",
        "\n",
        "# get the date parts in array\n",
        "date_parts = input_date.split(' ')\n",
        "print(f\"date parts = {date_parts}\")\n",
        "\n",
        "#get the time parts in array\n",
        "time_parts = date_parts[4].split(':')\n",
        "print(f\"time parts = {time_parts}\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "YpHtKy4GjAaM"
      },
      "source": [
        " This text splitting is more complicated than the tokenization process used for sentiment analysis."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "BnoyCZAMjAaM"
      },
      "source": [
        "\n",
        "#<p style=\"font-family:JetBrains Mono; font-weight:normal; letter-spacing: 1px; color:#207d06; font-size:100%; text-align:left;padding: 0px; border-bottom: 3px solid #207d06;\">**Sentence tokenizing**</p>\n",
        "\n",
        "Once you have a list of sentences, the next step is to split each sentence into a list of words.\n",
        "\n",
        "This process could be done in several ways, even using the str.split method described above, but we will use the NLTK library [nltk](https://www.nltk.org/) to help us with that.\n",
        "\n",
        "In the code assignment, you will use the method [word_tokenize](https://www.nltk.org/api/nltk.tokenize.html#nltk.tokenize.punkt.PunktLanguageVars.word_tokenize) to split your sentence into a list of words. Let us try the method in an example."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 5,
      "metadata": {
        "id": "9nFSrzjIjAaN",
        "outputId": "f8de89ff-975b-4b85-ace0-c09d217f3653",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "i am happy because i am learning. -> ['i', 'am', 'happy', 'because', 'i', 'am', 'learning', '.']\n"
          ]
        }
      ],
      "source": [
        "# tokenize the sentence into an array of words\n",
        "\n",
        "sentence = 'i am happy because i am learning.'\n",
        "tokenized_sentence = nltk.word_tokenize(sentence)\n",
        "print(f'{sentence} -> {tokenized_sentence}')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3w94RS3QjAaN"
      },
      "source": [
        "Now that the sentence is tokenized, you can work with each word in the sentence separately. This will be useful later when creating and counting N-grams. In the following code example, you will see how to find the length of each word."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "adUjXIBdjAaO",
        "outputId": "3dbc745a-481d-4082-ffe3-ba164523e3e8"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            " Lengths of the words: \n",
            "[('i', 1), ('am', 2), ('happy', 5), ('because', 7), ('i', 1), ('am', 2), ('learning', 8), ('.', 1)]\n"
          ]
        }
      ],
      "source": [
        "# find length of each word in the tokenized sentence\n",
        "sentence = ['i', 'am', 'happy', 'because', 'i', 'am', 'learning', '.']\n",
        "word_lengths = [(word, len(word)) for word in sentence] # Create a list with the word lengths using a list comprehension\n",
        "print(f' Lengths of the words: \\n{word_lengths}')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Wbdjg4ZTjAaO"
      },
      "source": [
        "The previous result produces a list of pairs. This is not equivalent to a dictionary."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "YSlx8428jAaO"
      },
      "source": [
        "\n",
        "#<p style=\"font-family:JetBrains Mono; font-weight:normal; letter-spacing: 1px; color:#207d06; font-size:100%; text-align:left;padding: 0px; border-bottom: 3px solid #207d06;\">**N-grams**</p>\n",
        "\n",
        "The next step is to build n-grams from the tokenized sentences.\n",
        "\n",
        "A sliding window of size n-words can generate the n-grams. The window scans the list of words starting at the sentence beginning, moving by a step of one word until it reaches the end of the sentence.\n",
        "\n",
        "Here is an example method that prints all trigrams in the given sentence."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 6,
      "metadata": {
        "id": "_XBuDh-NjAaO",
        "outputId": "895c79ac-89fe-4408-ceae-ec6c32e0c769",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "List all trigrams of sentence: ['i', 'am', 'happy', 'because', 'i', 'am', 'learning', '.']\n",
            "\n",
            "['i', 'am', 'happy']\n",
            "['am', 'happy', 'because']\n",
            "['happy', 'because', 'i']\n",
            "['because', 'i', 'am']\n",
            "['i', 'am', 'learning']\n",
            "['am', 'learning', '.']\n"
          ]
        }
      ],
      "source": [
        "def sentence_to_trigram(tokenized_sentence):\n",
        "    \"\"\"\n",
        "    Prints all trigrams in the given tokenized sentence.\n",
        "\n",
        "    Args:\n",
        "        tokenized_sentence: The words list.\n",
        "\n",
        "    Returns:\n",
        "        No output\n",
        "    \"\"\"\n",
        "    # note that the last position of i is 3rd to the end\n",
        "    for i in range(len(tokenized_sentence) - 3 + 1):\n",
        "        # the sliding window starts at position i and contains 3 words\n",
        "        trigram = tokenized_sentence[i : i + 3]\n",
        "        print(trigram)\n",
        "\n",
        "tokenized_sentence = ['i', 'am', 'happy', 'because', 'i', 'am', 'learning', '.']\n",
        "\n",
        "print(f'List all trigrams of sentence: {tokenized_sentence}\\n')\n",
        "sentence_to_trigram(tokenized_sentence)\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jpm9alD5jAaP"
      },
      "source": [
        "\n",
        "#<p style=\"font-family:JetBrains Mono; font-weight:normal; letter-spacing: 1px; color:#207d06; font-size:100%; text-align:left;padding: 0px; border-bottom: 3px solid #207d06;\">**Prefix of an n-gram**</p>\n",
        "\n",
        "As you saw in the lecture, the n-gram probability is often calculated based on the (n-1)-gram counts. The prefix is needed in the formula to calculate the probability of an n-gram.\n",
        "\n",
        "\\begin{equation*}\n",
        "P(w_n|w_1^{n-1})=\\frac{C(w_1^n)}{C(w_1^{n-1})}\n",
        "\\end{equation*}\n",
        "\n",
        "The following code shows how to get an (n-1)-gram  prefix from n-gram on an example of getting trigram from a 4-gram."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 7,
      "metadata": {
        "scrolled": true,
        "id": "8MAQLOwYjAaP",
        "outputId": "bbc4d14c-4fe5-4540-de8b-fc631f753dca",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "['i', 'am', 'happy']\n"
          ]
        }
      ],
      "source": [
        "# get trigram prefix from a 4-gram\n",
        "fourgram = ['i', 'am', 'happy','because']\n",
        "trigram = fourgram[0:-1] # Get the elements from 0, included, up to the last element, not included.\n",
        "print(trigram)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "PnrTmRVjjAaQ"
      },
      "source": [
        "<a name=\"start-end-of-sentence\"></a>\n",
        "### Start and end of sentence word $<s>$ and $</s>$\n",
        "You could see in the lecture that we must add some special characters at the beginning and the end of each sentence:\n",
        "* $<s>$  at beginning\n",
        "* $</s>$ at the end\n",
        "\n",
        "For n-grams, we must prepend n-1 of characters at the begining of the sentence.\n",
        "\n",
        "Let us have a look at how you can implement this in code.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 8,
      "metadata": {
        "id": "hz0GiTmZjAaQ",
        "outputId": "467499a7-f783-4843-e635-7ba603477362",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "['<s>', '<s>', 'i', 'am', 'happy', 'because', 'i', 'am', 'learning', '.', '</s>']\n"
          ]
        }
      ],
      "source": [
        "# when working with trigrams, you need to prepend 2 <s> and append one </s>\n",
        "n = 3\n",
        "tokenized_sentence = ['i', 'am', 'happy', 'because', 'i', 'am', 'learning', '.']\n",
        "tokenized_sentence = ['<s>'] * (n - 1) + tokenized_sentence + ['</s>']\n",
        "print(tokenized_sentence)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ridFZ02HjAaQ"
      },
      "source": [
        "That's all for the lab for \"N-gram\" lesson of week 3."
      ]
    }
  ],
  "metadata": {
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.7.8"
    },
    "colab": {
      "provenance": [],
      "toc_visible": true,
      "include_colab_link": true
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}