Practical_4_Word_Embeddings.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "orig_nbformat": 2,
    "deepnote_notebook_id": "9f9dc0ff-66a4-4ba9-9fa2-60906bad0cd6",
    "deepnote_execution_queue": [
      {
        "cellId": "00066-1e11fbaa-a239-4289-87f6-bff904bd6076",
        "sessionId": "d02c71a7-eb6c-4ab3-afc4-327e36b1c242",
        "msgId": "9e8804f9-9529-4275-a730-038d26b6e7b0"
      }
    ],
    "colab": {
      "name": "Practical_4_Word_Embeddings.ipynb",
      "provenance": []
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "tags": [],
        "cell_id": "00000-22249393-0b65-4e24-b87d-6ddb638964e5",
        "deepnote_cell_type": "markdown",
        "id": "os-1hxYkKS4N"
      },
      "source": [
        "# Word embeddings modelling"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "tags": [],
        "cell_id": "00001-a3906043-950b-4bb7-8203-828054593c18",
        "deepnote_to_be_reexecuted": false,
        "source_hash": "5600ace6",
        "execution_millis": 1059,
        "execution_start": 1610716244596,
        "deepnote_cell_type": "code",
        "id": "OiKRM1FfKS4Q"
      },
      "source": [
        "import nltk"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "tags": [],
        "cell_id": "00001-60a47584-2cf9-46fc-a3b2-b1a44e89adae",
        "deepnote_cell_type": "markdown",
        "id": "rIkf6jVMKS4U"
      },
      "source": [
        "## Language models"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "tags": [],
        "cell_id": "00002-3abbc42f-9063-40c8-a06d-8eeab62cd0e2",
        "deepnote_cell_type": "markdown",
        "id": "JNJO6T0QKS4V"
      },
      "source": [
        "Language models assign **probability values** to sequences of words. In essence, they are trying to \n",
        "“fill in the blank” based on context. \n",
        "\n",
        "Given a sentence “The hand-held gaming device is powered by small solar /---/ \", a language model may complete this sentence by saying that the word **\"panels\"** would fill the gap 80% of the time and the word **\"batteries\"** 20% of the time.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "tags": [],
        "cell_id": "00003-6a1cec31-fd71-4277-a323-8249809c7faf",
        "deepnote_cell_type": "markdown",
        "id": "WXiT8CpzKS4W"
      },
      "source": [
        "\n",
        "Two types:\n",
        "\n",
        "- **statistical** (N-grams, Hidden Markov Models, linguistc rules)\n",
        "- **neural** (neural networks)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "tags": [],
        "cell_id": "00004-18e03455-894a-463e-8696-a2804cb28521",
        "deepnote_cell_type": "markdown",
        "id": "nDsQuIoqKS4X"
      },
      "source": [
        "## Word vectors/embeddings"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "tags": [],
        "cell_id": "00005-0c57ca3a-83eb-46ba-a934-bcd1e7c724c3",
        "deepnote_cell_type": "markdown",
        "id": "fy6hU9DWKS4Y"
      },
      "source": [
        "When we want to process natural language and mine it for useful information using machine learning techniques we have to map textual data to some numerical representation (this process is called **vectorisation** as we create vectors of numeric values). Word vectors are often referred to as **\"word embeddings\"**. "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "tags": [],
        "cell_id": "00006-5b8d8f33-e5dc-47c2-9ac0-41bce5d68a16",
        "deepnote_cell_type": "markdown",
        "id": "qWEP09FHKS4Z"
      },
      "source": [
        "Do you remember what a **vector** is?"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "tags": [],
        "cell_id": "00007-a15797f3-5c30-4a19-9e32-a34519420a02",
        "deepnote_cell_type": "markdown",
        "id": "GXH6FxINKS4Z"
      },
      "source": [
        "- Geometry: **an object with magnitude and direction**\n",
        "\n",
        "- Computer science: **one-dimensional array** (e.g. $$[.44, .26, .07, -.89, -.15].$$ )"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "tags": [],
        "cell_id": "00008-2bbf8e6c-b61e-4998-b76e-ca398e44877a",
        "deepnote_cell_type": "markdown",
        "id": "c-HSkqBUKS4a"
      },
      "source": [
        "![Picture title](image-20201006-124756.png)\n",
        "Image source and refresher on vectors: https://www.mathsisfun.com/algebra/vectors-cross-product.html"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "tags": [],
        "cell_id": "00009-ecf93e3f-cfd0-4d43-8373-ae17b91400a0",
        "deepnote_cell_type": "markdown",
        "id": "dH7yFVa4KS4b"
      },
      "source": [
        "## Data representation"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "tags": [],
        "cell_id": "00010-4e558b86-2279-4cdf-af8e-b25fd976a63a",
        "deepnote_cell_type": "markdown",
        "id": "-M1Lx3ePKS4b"
      },
      "source": [
        "How do we vectorise words? There are quite a few ways to represent text data as numbers, depending on what information we want them to contain."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "tags": [],
        "cell_id": "00011-388b8b9b-86a4-499b-a207-34b547654460",
        "deepnote_cell_type": "markdown",
        "id": "5moH2IWvKS4c"
      },
      "source": [
        "Types of data representation\n",
        "- one-hot (presence/absence)\n",
        "- frequency-based (occurence frequency)\n",
        "- distributed (read on!)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "tags": [],
        "cell_id": "00012-879dbdef-63f3-4e3a-90b1-03cfa8729a09",
        "deepnote_cell_type": "markdown",
        "id": "URr269l3KS4c"
      },
      "source": [
        "### One-hot encoding"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "tags": [],
        "cell_id": "00013-e67d8479-d276-4880-a6ce-a50c3dd8b944",
        "deepnote_cell_type": "markdown",
        "id": "SD3NH6NRKS4d"
      },
      "source": [
        "**One-hot encoding**  is the simplest method  (“1-of-N” encoding). The resulting embeddings (vectors) are composed of a single \"one\" and a number of \"0 (zeros)\".\n",
        "This encoding method marks a particular vector **index** with a value of true (1) if the token occurs in a document and false (0) if it does not. \n",
        "In other words, each element of a one-hot encoded vector reflects either the presence or absence of the token in the analysed text.\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "tags": [],
        "output_cleared": false,
        "cell_id": "00014-00f5b699-fa1a-41f6-9edd-aba4ce6ce3cd",
        "deepnote_to_be_reexecuted": false,
        "source_hash": "d705a0a",
        "execution_millis": 44,
        "execution_start": 1610716245656,
        "deepnote_cell_type": "code",
        "id": "nhiYwT52KS4d"
      },
      "source": [
        "import numpy as np\n",
        "sentence1 = \"We value talking to a human being at the other end of a conversation\".lower().split()\n",
        "sentence2 = \"Trump is being given a steroid that is usually used for severe cases of covid-19\".lower().split()\n",
        "\n",
        "vocab = set(sentence1+sentence2)\n",
        "vocab = sorted(vocab)\n",
        "print (\"vocabulary (two sentences combined): \", vocab)\n",
        "\n",
        "#encoding words in the sentece based on their index (position) in the vocabulary\n",
        "integer_encoded = []\n",
        "for i in sentence1:\n",
        "    print (np.array(vocab)==i)\n",
        "    v = np.where( np.array(vocab) == i)[0][0]\n",
        "    print ('v: ', v)\n",
        "    integer_encoded.append(v)\n",
        "print (\"sentence 1 encoded: \",integer_encoded)\n",
        "\n",
        "integer_encoded = []\n",
        "for i in sentence2:\n",
        "    v = np.where( np.array(vocab) == i)[0][0]\n",
        "    integer_encoded.append(v)\n",
        "print (\"sentence 2 encoded: \",integer_encoded)\n",
        "\n",
        "def get_vec(len_vocab,word):\n",
        "    empty_vector = [0] * len_vocab\n",
        "    vect = 0\n",
        "    find = np.where(np.array(vocab) == word)[0][0]\n",
        "    empty_vector[find] = 1\n",
        "    return empty_vector\n",
        "\n",
        "def get_matrix(vocab, sentence):\n",
        "    mat = []\n",
        "    len_vocab = len(vocab)\n",
        "    for i in sentence:\n",
        "        vec = get_vec(len_vocab,i)\n",
        "        mat.append(vec)\n",
        "        \n",
        "    return np.asarray(mat)\n",
        "\n",
        "print (\"MATRIX Sentence 1 :\")\n",
        "print (get_matrix(vocab, sentence1))   \n",
        "print (\"MATRIX Sentence 2 :\")\n",
        "print (get_matrix(vocab, sentence2))   "
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "tags": [],
        "cell_id": "00015-c157e3f7-d781-446d-a21e-1bde3b91e5e5",
        "deepnote_cell_type": "markdown",
        "id": "UVzNABq8KS4e"
      },
      "source": [
        "**CODEIT** write a code snippet to extract the one-hot matrix representation of the following three sentences:\n",
        "\n",
        "1. NLP is now the most popular subfield of machine learning.\n",
        "2. My washing machine is not working properly now.\n",
        "3. Analysis of language using artificial intelligence methods have risen dramatically.\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "tags": [],
        "output_cleared": false,
        "cell_id": "00016-73f54d86-5752-454b-9798-a6e3187e4034",
        "deepnote_to_be_reexecuted": false,
        "source_hash": "457eaf7a",
        "execution_millis": 0,
        "execution_start": 1610716245700,
        "deepnote_cell_type": "code",
        "id": "x9KaJD6-KS4e"
      },
      "source": [
        "##Insert your code here\n",
        "sentence_1 = \"NLP is now the most popular subfield of machine learning .\".lower().split()\n",
        "sentence_2 = \"My washing machine is not working properly now .\".lower().split()\n",
        "sentence_3 = \"Analysis of language using artificial intelligence methods have risen dramatically .\".lower().split()\n",
        "\n",
        "import numpy as np\n",
        "\n",
        "vocab = set(sentence_1+sentence_2+sentence_3)\n",
        "vocab = sorted(vocab)\n",
        "print (\"vocabulary (three sentences combined): \", vocab)\n",
        "#Insert your code here"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "tags": [],
        "cell_id": "00017-c07bbbac-543d-46ca-ae39-1ca1857be263",
        "deepnote_cell_type": "markdown",
        "id": "T0OoA-bwKS4f"
      },
      "source": [
        "**OBSERVE AND REFLECT:**  Using the examples above explain why one-hot vector representation is **not** the best method for analysing semantic similarity? \n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "tags": [],
        "cell_id": "00018-9b358ad8-a4ef-43bf-872a-2c80048f2b21",
        "deepnote_cell_type": "markdown",
        "id": "jT9zzxayKS4f"
      },
      "source": [
        "### Write your answer here ###"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "tags": [],
        "cell_id": "00019-e6a6a448-3136-480f-bf34-8c906dec8fce",
        "deepnote_cell_type": "markdown",
        "id": "14HmzsjTKS4f"
      },
      "source": [
        "---"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "tags": [],
        "cell_id": "00020-9060eb9e-6b1a-4314-81ec-19067401dddb",
        "deepnote_cell_type": "markdown",
        "id": "wg0died1KS4f"
      },
      "source": [
        "What are the **main problems with this one-hot representation**?\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "tags": [],
        "cell_id": "00021-ca5046d4-5bcc-4113-bf4a-53884551fc1e",
        "deepnote_cell_type": "markdown",
        "id": "-wR_N3nuKS4g"
      },
      "source": [
        "- **sparsity and size**: the representation size grows with the corpus (imagine a corpus with the 300,000 word vocabulary where each word vector will will have 300,000 dimensions (float values) with all but one being a zero) (computationally expensive!).\n",
        "- **each vector is equally distant from every other vector** (does not reflect their position in relation to each other)\n",
        "- **no contextual/semantic information** is embedded  - therefore they are not suitable for NLP tasks like POS tagging, named-entity recognition etc.\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "tags": [],
        "cell_id": "00022-a717753b-b9a1-4519-bca2-2a92b2a7eb84",
        "deepnote_cell_type": "markdown",
        "id": "30ChGg7qKS4g"
      },
      "source": [
        "### Distributed representation"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "tags": [],
        "cell_id": "00023-13dcd47a-9b2a-4cda-9ea5-b65f8c6ccb1d",
        "deepnote_cell_type": "markdown",
        "id": "MlQjLlunKS4g"
      },
      "source": [
        "An alternative is called **distributed representation**. \n",
        "\n",
        "Please read here UNTIL (and including) Figure 3 (up until \"While this shape example is oversimplified, it serves as a great high-level, abstract introduction to distributed representations\"\n",
        "to get familiar with this concept. https://www.oreilly.com/content/how-neural-networks-learn-distributed-representations/\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "tags": [],
        "cell_id": "00024-2896d0f8-70ae-493d-9be1-01caf00a1ae1",
        "deepnote_cell_type": "markdown",
        "id": "s_9Wki8CKS4g"
      },
      "source": [
        "## Training a simple neural language model"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "tags": [],
        "cell_id": "00025-b63779ed-2fcb-42b4-97a0-7965e44edfc0",
        "deepnote_cell_type": "markdown",
        "id": "BzLdI9FJKS4h"
      },
      "source": [
        "1. represent words with **one-hot vectors**\n",
        "2. encode input words (create **word embeddings**):\n",
        "- take the  one-hot vector representing the input word\n",
        "- multiply it by a matrix of size (N,200) (200 is the vector size - number of dimensions - which is chosen **arbitrarily**).\n",
        "This multiplication results in a vector of size 200 (word embedding). \n",
        "\n",
        "<img  src=\"http://mccormickml.com/assets/word2vec/matrix_mult_w_one_hot.png\"/>\n",
        "\n",
        "3. Now we have a representation of the input word. \n",
        "We multiply it by a matrix of size (200,N) (**output embedding**).  \n",
        "As a result, we get a vector of size N and then pass it through **softmax function**.\n",
        "Softmax normalises values of the vector into a probability distribution (each one of the values is between 0 and 1, and their sum is 1). \n",
        "This decoding step takes a word representation and returns a distribution which represents the model’s predictions of the next word. \n",
        "\n",
        "\n",
        "\n",
        "<center><img src = \"http://mccormickml.com/assets/word2vec/output_weights_function.png\"></center>\n",
        "\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "tags": [],
        "cell_id": "00026-3f228a93-b1d3-4d5d-9d8e-d4c8c1bff8ad",
        "deepnote_cell_type": "markdown",
        "id": "4f7eylZfKS4h"
      },
      "source": [
        "**QUICK Softmax refresher**: https://victorzhou.com/blog/softmax/"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "tags": [],
        "cell_id": "00027-3ea67cb8-91ef-4a64-a209-3f2202a5eb3c",
        "deepnote_cell_type": "markdown",
        "id": "U8tDYJpOKS4h"
      },
      "source": [
        "**Data needed for training**: pairs of input and target output words\n",
        "\n",
        "**Data generation**: take every pair of neighboring words from the text and use the first one as the input word and the second one as the target output word. \n",
        "Example: “The cat is on the mat”.\n",
        "\n",
        "**Word pairs for training**: (The, cat), (cat, is), (is, on), (on, the), (the, mat).\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "tags": [],
        "cell_id": "00028-e86b339f-f0d6-4038-a0a7-7c8b935cbcb7",
        "deepnote_cell_type": "markdown",
        "id": "rEMmb_fSKS4h"
      },
      "source": [
        "**Training process**: using gradient descent to update the model during training and loss measures to calculate\n",
        "the distance between the output distribution predicted by the model and the target distribution for each pair of training words. \n",
        "The target distribution for each pair is a one-hot vector representing the target word.\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "tags": [],
        "cell_id": "00029-bd7f5166-b74e-4ff3-b5de-09f391e19919",
        "deepnote_cell_type": "markdown",
        "id": "7XxiHxJ3KS4i"
      },
      "source": [
        "Please check this page for more info on the algorithm architecture: http://mccormickml.com/2016/04/19/word2vec-tutorial-the-skip-gram-model/"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "tags": [],
        "cell_id": "00030-487a5420-bd5a-4415-9afd-948c4ce062aa",
        "deepnote_cell_type": "markdown",
        "id": "3c4B9I8SKS4i"
      },
      "source": [
        "**Model performance evaluation**: Let's talk about PERPLEXITY again :) \n",
        "\n",
        "<img  src=\"https://miro.medium.com/max/616/1*vV0XMYe69LPMlH3fFouDtw.png\"/>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "tags": [],
        "cell_id": "00031-2cda670a-db48-4f1e-8cc5-192e9ab7681d",
        "deepnote_cell_type": "markdown",
        "id": "3uQntLJTKS4j"
      },
      "source": [
        "## How to improve the performance of a simple model?"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "tags": [],
        "cell_id": "00032-6cd9926b-0f21-4b6f-92d9-660437d6eafb",
        "deepnote_cell_type": "markdown",
        "id": "OMVnPpw1KS4j"
      },
      "source": [
        "Can you think what the biggest problem of this simple model is?"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "tags": [],
        "cell_id": "00033-e3f18755-3215-48e5-ac77-24addda84368",
        "deepnote_cell_type": "markdown",
        "id": "EJDYbzEMKS4k"
      },
      "source": [
        "To predict the next word in the sentence, it only uses ONE preceding word. In real life, we consider much more context when reading and understanding a text. \n",
        "A model that could be taught to \"remember\" more than one preceding word would be more efficient!"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "tags": [],
        "cell_id": "00034-639493ff-48b6-423c-b75e-c7bd4786dd0b",
        "deepnote_cell_type": "markdown",
        "id": "ko9c8aDfKS4l"
      },
      "source": [
        "**Example:** what words follow the word \"eat\"? \n",
        "\n",
        "We can answer “cookies”, “nuts” or \"eucalyptus\", and the model could also reply that these words may have high probability of being the target ones. However, if we knew that the actual word sequence was “Koalas eat\" would it change our opinion about the most probable answer?\n",
        "\n",
        "\n",
        "![ChessUrl](https://media.giphy.com/media/eDUHhtooZxyhi/giphy.gif)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "tags": [],
        "cell_id": "00035-b8470a7c-b6aa-4cfa-983a-575cdc58deeb",
        "deepnote_cell_type": "markdown",
        "id": "O2GeiDlwKS4l"
      },
      "source": [
        "## WORD EMBEDDINGS\n",
        "\n",
        "Words get their embeddings by us looking at which other words they tend to appear next to. The mechanics of that is that\n",
        "\n",
        "1. We get a lot of text data (say, all Wikipedia articles, for example). then\n",
        "\n",
        "\n",
        "2. We have a window (say, of three words) that we slide against all of that text.\n",
        "\n",
        "\n",
        "3. The sliding window generates training samples for our model\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "tags": [],
        "cell_id": "00036-525f151c-e079-417c-b674-fc14ae0ba2af",
        "deepnote_cell_type": "markdown",
        "id": "fRx8HJ7oKS4l"
      },
      "source": [
        "### Word2vec\n",
        "\n",
        "A method of creating word embeddings"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "tags": [],
        "cell_id": "00037-0eb7f0ca-a1cb-4625-aff7-d6c5ef40ea5f",
        "deepnote_cell_type": "markdown",
        "id": "moRBW7QZKS4m"
      },
      "source": [
        "http://jalammar.github.io/illustrated-word2vec/\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "tags": [],
        "cell_id": "00038-aa13eac1-95cf-43e3-8c1d-396acfeae9d5",
        "deepnote_cell_type": "markdown",
        "id": "kBJckxoDKS4m"
      },
      "source": [
        "### GloVe (Global Vectors)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "tags": [],
        "cell_id": "00039-efa3aae0-3f20-41fd-9f8c-fea16a2457d1",
        "deepnote_cell_type": "markdown",
        "id": "JxOP3yeiKS4m"
      },
      "source": [
        "Disadvantage of skipgram models: they do not operate directly on the co-occurrence statistics.\n",
        "They scan context windows across the entire corpus and fail to take advantage of the vast amount of repetition in the data.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "tags": [],
        "cell_id": "00040-17560e4f-3148-4ff4-8d7c-b8145c1dac03",
        "deepnote_cell_type": "markdown",
        "id": "DW5M7s-FKS4m"
      },
      "source": [
        "**GloVe (Global Vectors)** is a **count-based model**. It learns word embeddings by dimensionality reduction of a **co-occurrence counts matrix**.\n",
        "\n",
        "1. Build a co-occurence matrix (each row = how often does a word occur with every other word in some defined context-size in a large corpus).\n",
        "\n",
        "2. Factorise this matrix (=> a lower-dimensional matrix: rows = word vectors).\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "tags": [],
        "cell_id": "00041-24b2e3ff-20c6-43ff-b414-b16bb05b5f42",
        "deepnote_cell_type": "markdown",
        "id": "niHQ6CjYKS4m"
      },
      "source": [
        "### Problems with word2vec and GloVe "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "tags": [],
        "cell_id": "00042-0c101a0c-db65-467c-8193-240f485c94e3",
        "deepnote_cell_type": "markdown",
        "id": "FB1MDOW1KS4m"
      },
      "source": [
        "They create one vector for different meanings of a polysemous word (and about 40% of English words are polysemous!).\n",
        "\n",
        "Example: any occurence of the word \"bank\" (river bank or financial institution) - will be mapped to the same vector.\n",
        "\n",
        "Words exist in context and their meanings are defined by the contextual use. Would not it be beneficial to learn representations that reflect this?"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "tags": [],
        "cell_id": "00043-49b6d1aa-b0eb-4234-a0ac-9d2f52f0237a",
        "deepnote_cell_type": "markdown",
        "id": "k3LHmzqGKS4n"
      },
      "source": [
        "### BERT (Bidirectional Encoder Representations from Transformers)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "tags": [],
        "cell_id": "00044-2741451a-88f7-4f20-b8ef-3414996ac079",
        "deepnote_cell_type": "markdown",
        "id": "BhdmtlLaKS4n"
      },
      "source": [
        "Release of BERT model was described as marking the beginning of a new era in NLP. **Bidirectional Encoder Representations from Transformers (BERT)** is a language model that looks both to the left and the right of a word to pre-train representations.\n",
        "\n",
        "![BertUrl](https://media.giphy.com/media/umMYB9u0rpJyE/giphy.gif)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "tags": [],
        "cell_id": "00045-f90c3810-ed09-4ec5-8fc6-b02a79cd3787",
        "deepnote_cell_type": "markdown",
        "id": "xPZSE1Y3KS4o"
      },
      "source": [
        "\n",
        "Key technical innovation:\n",
        "- applying bidirectional training of Transformer, a popular attention model, to language modelling\n",
        "- deeper understanding on a word's context\n",
        "- reads the entire sequence of words at once =>  learns context of a word based on all of its surroundings \n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "tags": [],
        "cell_id": "00046-aa350828-71c1-428d-9514-b9cfb426ce13",
        "deepnote_cell_type": "markdown",
        "id": "X37eJ3ptKS4o"
      },
      "source": [
        "### Fastext (by Facebook Research)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "tags": [],
        "cell_id": "00047-c2f1d502-6419-43ed-8b69-aa9b4bf4293b",
        "deepnote_cell_type": "markdown",
        "id": "3mt9dcbAKS4o"
      },
      "source": [
        "- represents each word as an n-gram of characters.\n",
        "Example: \"artificial\" with n=3 <ar, art, rti, tif, ifi, fic, ici, ial, al> (the angular brackets mean the beginning and end of the word).   \n",
        "- capture the meaning of shorter words and suffixes & prefixes\n",
        "- works well with rare words"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "tags": [],
        "cell_id": "00048-b0eca7df-b3ab-4ef7-bd61-588c559417c6",
        "deepnote_cell_type": "markdown",
        "id": "Lg2ouYAMKS4o"
      },
      "source": [
        "## Wait, there is more!"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "tags": [],
        "cell_id": "00049-038b8818-f3b1-4026-9c84-a82acde67f66",
        "deepnote_cell_type": "markdown",
        "id": "xzmNyJ7_KS4p"
      },
      "source": [
        "If you want to learn about the most recent models please check out the following links:"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "tags": [],
        "cell_id": "00050-0aae958d-e301-4bb2-8909-3e006c025289",
        "deepnote_cell_type": "markdown",
        "id": "m5DQuZuoKS4p"
      },
      "source": [
        "- Word2vec: http://jalammar.github.io/illustrated-word2vec/\n",
        "- Transformers: http://jalammar.github.io/illustrated-transformer/\n",
        "- GPT2: http://jalammar.github.io/illustrated-gpt2/  & https://openai.com/blog/gpt-2-1-5b-release/ (the model was initially not release to public out of fear it would be used to spread fake news, spam, and disinformation. )\n",
        "- GPT3 (2020): can generate computer code, prose and poetry; has been called \"amazing\", \"spooky\", \"humbling\", and \"more than a little terrifying\". \n",
        "- GPT3 use examples: https://gpt3examples.com/#examples\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "tags": [],
        "cell_id": "00051-893cca1e-7887-4f9c-8bd0-386323d5a62a",
        "deepnote_cell_type": "markdown",
        "id": "p1G2d3PHKS4q"
      },
      "source": [
        "# Let's see how word2vec models work"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "tags": [],
        "cell_id": "00052-6afb0a45-9c3e-45c7-be5e-6fe85c1cdf01",
        "deepnote_cell_type": "markdown",
        "id": "-G-CsO8JKS4q"
      },
      "source": [
        "Choose a word embedding in a language of your preference and download it:http://vectors.nlpl.eu/repository/#\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "tags": [],
        "cell_id": "00053-5c43d1b8-7425-4f24-b499-cd10d79646b7",
        "deepnote_cell_type": "markdown",
        "id": "D59Ll3oaKS4q"
      },
      "source": [
        "**Gensim** is a python library which implements various natural language processing methods and algorithms."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "tags": [],
        "output_cleared": false,
        "cell_id": "00054-3470baf3-7890-47a8-9a3c-6ed186793764",
        "deepnote_to_be_reexecuted": false,
        "source_hash": "9162bcaf",
        "execution_millis": 2512,
        "execution_start": 1610716245701,
        "deepnote_cell_type": "code",
        "id": "3wQuMkGVKS4r"
      },
      "source": [
        "!pip install gensim\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "tags": [],
        "cell_id": "00055-9a8f2fa4-385d-4a60-9e8e-53fdd7e16a50",
        "deepnote_cell_type": "markdown",
        "id": "moxT_4tWKS4s"
      },
      "source": [
        "In the following code we read the brown corpus from nltk library and train(build) a Word2Vec language model using Gensim library.\n",
        "\n",
        "**Note:** Running the following code takes a few minutes"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "tags": [],
        "output_cleared": false,
        "cell_id": "00056-52a969df-7f46-44b6-a1c3-524dd2cf7800",
        "deepnote_to_be_reexecuted": false,
        "source_hash": "5f463216",
        "execution_millis": 46455,
        "execution_start": 1610716372916,
        "deepnote_cell_type": "code",
        "id": "EqPpgVqYKS4s"
      },
      "source": [
        "import gensim\n",
        "import logging\n",
        "from nltk.corpus import brown \n",
        "\n",
        "\n",
        "nltk.download('brown')\n",
        "logging.basicConfig(format='%(asctime)s : %(levelname)s : %(message)s', level=logging.INFO)\n",
        "sentences = brown.sents()\n",
        "model = gensim.models.Word2Vec(sentences, min_count=1)\n",
        "\n",
        "model.save('brown_model.bin')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "tags": [],
        "cell_id": "00057-b23bd006-9e42-47c3-966b-32013824bbcc",
        "deepnote_cell_type": "markdown",
        "id": "3xZRNkmBKS4t"
      },
      "source": [
        "Using the following code you can access vectors of words in your gensim model."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "tags": [],
        "output_cleared": false,
        "cell_id": "00058-29b94963-660d-4e2e-b26f-ea583870af74",
        "deepnote_to_be_reexecuted": false,
        "source_hash": "11d403da",
        "execution_millis": 7,
        "execution_start": 1610716292752,
        "deepnote_cell_type": "code",
        "id": "_3XjtBPBKS4t"
      },
      "source": [
        "import numpy as np\n",
        "import nltk\n",
        "\n",
        "# Access vectors for specific words with a keyed lookup:\n",
        "vector = model['year']\n",
        "print(vector)\n",
        "# see the shape of the vector (300,)\n",
        "print(vector.shape)\n",
        "# Processing sentences is not as simple as with Spacy:\n",
        "vectors = [model[x] for x in \"This is some text I am processing with text analysis library\".split(' ')]\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "tags": [],
        "cell_id": "00059-42932642-b183-47b3-858a-a650760c29b8",
        "deepnote_cell_type": "markdown",
        "id": "3la7g1_6KS4u"
      },
      "source": [
        "## Using a pretrained Word Embedding model\n",
        "\n",
        "In the above we learnt who to use Gensim Library to train a language model from text.\n",
        "\n",
        "In this section we focus on using the language models which are already built and trained with huge amount of data such as the whole corpus of Wikipedia.\n",
        "\n",
        "Download a word2vec model in english on Wikipedia from [this link](http://vectors.nlpl.eu/repository/20/3.zip) (596 MB file)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "tags": [],
        "cell_id": "00060-1ba89a2b-c684-4943-970c-decd9852c440",
        "deepnote_cell_type": "markdown",
        "id": "n7P050iUKS4u"
      },
      "source": [
        " <p style=\"color:red\"> IMPORTANT NOTE: do not run the following cell if you haven't downloaded a word embedding model</p>\n",
        "\n",
        "\n",
        " If you didn't download a model you can continue with the current small model.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "tags": [],
        "cell_id": "00061-bc944bbf-ee77-4069-b6b3-c2954536589b",
        "deepnote_cell_type": "markdown",
        "id": "fAolxsINKS4u"
      },
      "source": [
        "The following cell code loads an already trained word2vec model using gensim library. (a pretrained model)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "tags": [],
        "output_cleared": false,
        "cell_id": "00062-a00bd2c8-f01e-49f9-83ef-6a7bb2c08975",
        "deepnote_to_be_reexecuted": false,
        "source_hash": "7b2695b",
        "execution_millis": 2689,
        "execution_start": 1610716292762,
        "deepnote_cell_type": "code",
        "id": "ITMRfRa-KS4u"
      },
      "source": [
        "# Load vectors directly from the file\n",
        "\n",
        "#Put the address of your downloded language model here \n",
        "from gensim.models import KeyedVectors\n",
        "# Load vectors directly from the file\n",
        "address_of_your_model =\"model.bin\"\n",
        "model = KeyedVectors.load_word2vec_format(address_of_your_model, binary=True)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "tags": [],
        "cell_id": "00063-f6f9d0a5-b982-4862-99d8-ce2a915fd2a1",
        "deepnote_cell_type": "markdown",
        "id": "Ty_XtHsvKS4v"
      },
      "source": [
        "The word2vec class in gensim library has a function for identifuing the most similar or dissimlar words to a word in it's vocabulary.\n",
        "\n",
        "Try it by running th e following code cell:"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "tags": [],
        "cell_id": "00064-995cd490-b21a-43a5-a9ca-ce0f8eecee58",
        "deepnote_cell_type": "markdown",
        "id": "RtNVi4p8KS4v"
      },
      "source": [
        " <p style=\"color:red\">If you didn't download and load the pretrained model , you can still run the following codes. However it's very probable that your model doesn't work well or doesn't know some words, since it has been trained on a very small corpus</p>"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "tags": [],
        "cell_id": "00066-1e11fbaa-a239-4289-87f6-bff904bd6076",
        "deepnote_to_be_reexecuted": false,
        "source_hash": "a02d5af9",
        "execution_start": 1610716580156,
        "deepnote_cell_type": "code",
        "id": "nw1X_QE_KS4v"
      },
      "source": [
        "# Load vectors directly from the file\n",
        "!wget https://s3.amazonaws.com/dl4j-distribution/GoogleNews-vectors-negative300.bin.gz\n",
        "!gunzip GoogleNews-vectors-negative300.bin\n",
        "#Put the address of your downloded language model here\n",
        "from gensim.models import KeyedVectors\n",
        "# Load vectors directly from the file\n",
        "address_of_your_model =\"GoogleNews-vectors-negative300.bin\"\n",
        "model = KeyedVectors.load_word2vec_format(address_of_your_model, binary=True)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "tags": [],
        "output_cleared": false,
        "cell_id": "00065-0653a1d2-4a2a-44ae-ab90-26eaaa60123e",
        "deepnote_to_be_reexecuted": false,
        "source_hash": "36b589fa",
        "execution_millis": 474,
        "execution_start": 1610716295455,
        "deepnote_cell_type": "code",
        "id": "rshTCwsIKS4w"
      },
      "source": [
        "model.similar_by_word('music')\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "tags": [],
        "cell_id": "00066-990a3ebd-132b-4291-a0cd-044c09469e53",
        "deepnote_cell_type": "markdown",
        "id": "TrMPfL7AKS4w"
      },
      "source": [
        "**CODE IT** Using the function `similarity` from gensim library. print the similarity measures of two sets of words according to your model. \n",
        "\n",
        "Cat and Dog \n",
        "\n",
        "Cat and King"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "tags": [],
        "output_cleared": false,
        "cell_id": "00067-4e3bdf0f-bc20-48df-b637-ccb9acedee50",
        "deepnote_to_be_reexecuted": false,
        "source_hash": "a940b6fb",
        "execution_millis": 60,
        "execution_start": 1610716295933,
        "deepnote_cell_type": "code",
        "id": "Xw_-ufqZKS4w"
      },
      "source": [
        "x = 'Cat'\n",
        "y = 'Dog'\n",
        "z = 'King'\n",
        "print(model.similarity(x,y))\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "tags": [],
        "cell_id": "00068-c74618b3-2c8b-48a8-8997-d2c771cb1526",
        "deepnote_cell_type": "markdown",
        "id": "cw4v1Xj_KS4x"
      },
      "source": [
        "### Analogies:\n",
        "\n",
        "Gensim library provides functionalities for getting analogies from word2vec models.\n",
        "\n",
        "\n",
        "The `king-man+woman = queen` is a very typical example of how vord embeddings capture semantic dimentions.\n",
        "\n",
        "Imagine a dimention in a 300 dimentional embedding is storing the concept of Royalty in the word `King`. And one other dimention is storing `gender`.\n",
        "\n",
        "What would happen if we substract the vector of `Man` from `King` (getting a vector which keeps the `Royalty` but subtracts `masculinity` from gender) and then add `Women` to the result. We excpect to get Queen (Royality+ feminine) which actualy happens in word embeddings trained are huge amount of text.\n",
        "\n",
        "\n",
        "In the following we see how we can use the analogy funtionality in gensim library."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "tags": [],
        "cell_id": "00069-d585592f-e3a7-44e4-abe9-ae9a0c4952db",
        "deepnote_cell_type": "markdown",
        "id": "_xRSLAOmKS4x"
      },
      "source": [
        "\n",
        "<img src=\"https://cdn-images-1.medium.com/max/600/1*LdviucnshWgIIcQvhTTF-g.png\" >"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "tags": [],
        "cell_id": "00070-0277a8ba-0bd6-4d45-ad6b-08c701571b99",
        "deepnote_cell_type": "markdown",
        "id": "vAvERCmiKS4y"
      },
      "source": [
        "The following code performs the above vector calculations. King-Man +Woman = Queen"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "tags": [],
        "output_cleared": false,
        "cell_id": "00071-ff82cdc3-4de2-4092-8f35-a442a06d5768",
        "deepnote_to_be_reexecuted": false,
        "source_hash": "756abe0a",
        "execution_millis": 57,
        "execution_start": 1610716295995,
        "deepnote_cell_type": "code",
        "id": "e0oMbzb3KS4z"
      },
      "source": [
        "model.wv.most_similar(positive=[\"king\", \"woman\"], negative=[\"man\"], topn=3)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "tags": [],
        "cell_id": "00072-3f95311f-2cf8-4ea1-b618-4987084889f8",
        "deepnote_cell_type": "markdown",
        "id": "V7gTftyAKS4z"
      },
      "source": [
        "**CODEIT** Using the above example, write a code line which can give the Capital of Belgium as output by knowing the Capital of France. \n",
        "\n",
        "or in other words:   **France** to **Paris** is **Belgium** to ...\n",
        "\n",
        "**NOTE:** if you could not load the language model file **write the code as you think it's correct** and get the answer using this demo :https://rare-technologies.com/word2vec-tutorial/#app\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "tags": [],
        "output_cleared": false,
        "cell_id": "00073-5d7c6fd7-eef8-4fcd-97a2-494612bf867b",
        "deepnote_to_be_reexecuted": false,
        "source_hash": "1e919cbe",
        "execution_millis": 66,
        "execution_start": 1610716296055,
        "deepnote_cell_type": "code",
        "id": "ZzSDfltgKS4z"
      },
      "source": [
        "#insert you code here\n",
        "\n",
        "model.wv.most_similar(positive=[\"Paris\", \"Belgium\"], negative=[\"France\"], topn=1)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "tags": [],
        "cell_id": "00074-b6177933-10bc-45b2-b039-af51f1fcc46a",
        "deepnote_cell_type": "markdown",
        "id": "4dKBzvAAKS40"
      },
      "source": [
        "**CODEIT**    Using the same code try: **Man** is to **Actor** as **Woman** is to ...\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "tags": [],
        "output_cleared": false,
        "cell_id": "00075-38a0fe00-69b4-4f4e-8d7d-3bafed16c8fb",
        "deepnote_to_be_reexecuted": false,
        "source_hash": "6d3f81be",
        "execution_start": 1610716296125,
        "execution_millis": 151,
        "deepnote_cell_type": "code",
        "id": "2MmD1KTgKS40"
      },
      "source": [
        "#insert your code here\n",
        "model.wv.most_similar(positive=[\"actor\", \"woman\"], negative=[\"man\"], topn=10)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "tags": [],
        "cell_id": "00076-fd40b2c9-8af7-492b-b94b-b745fc0ecc27",
        "deepnote_cell_type": "markdown",
        "id": "_avHTMi0KS41"
      },
      "source": [
        "**CODEIT**    Using the same code try: **go** is to **going** as **come** is to ...\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "tags": [],
        "output_cleared": false,
        "cell_id": "00077-ca1553fe-7f7d-4f14-af8c-1bed76a9c0b9",
        "deepnote_to_be_reexecuted": false,
        "source_hash": "e9d95ceb",
        "execution_start": 1610716296278,
        "execution_millis": 86,
        "deepnote_cell_type": "code",
        "id": "-nrV_qE4KS41"
      },
      "source": [
        "model.wv.most_similar(positive=[\"going\", \"come\"], negative=[\"go\"], topn=10)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "tags": [],
        "cell_id": "00078-432391bb-2787-4a4f-ae24-ab31567a25ee",
        "deepnote_cell_type": "markdown",
        "id": "lZvOK7bPKS42"
      },
      "source": [
        "**OBSERVE AND REFLECT : ** if you succeeded in running the above codes you can see that word2vec model have embedded in it some knowledge about the langauge(tenses), knowledge about the world (capital of countries) by observing the conexts of the words in huge amounts of text.  \n",
        "\n",
        "Why do you think a languge model might act like the following?\n",
        "\n",
        "` model.wv.most_similar(positive=[\"doctor\", \"woman\"], negative=[\"man\"], topn=1) = 'nurse' `\n",
        "\n",
        "Read about [Bias In Language Models](https://towardsdatascience.com/bias-in-natural-language-processing-nlp-a-dangerous-but-fixable-problem-7d01a12cf0f7)\n",
        "\n",
        "\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "tags": [],
        "cell_id": "00079-ea5890bf-108a-43c6-83c3-d123ad9d406a",
        "deepnote_cell_type": "markdown",
        "id": "95jlNXMCKS42"
      },
      "source": [
        "We are planning a separate session on ethics of NLP - stay tuned!"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "tags": [],
        "cell_id": "00080-01970802-79d9-44e0-8fa0-96ae688fec74",
        "deepnote_cell_type": "markdown",
        "id": "WxjCoAeOKS42"
      },
      "source": [
        "The following function in Gensim library finds a word in the list of words which is the most dissimilar to the others:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "tags": [],
        "output_cleared": false,
        "cell_id": "00081-636d1e8c-8dab-4476-84a9-b898f75ecdb0",
        "deepnote_to_be_reexecuted": false,
        "source_hash": "5729ba8f",
        "execution_start": 1610716296365,
        "execution_millis": 43,
        "deepnote_cell_type": "code",
        "id": "NPJUU0N9KS42"
      },
      "source": [
        "print(model.wv.doesnt_match([\"France\",\"Germany\",\"Britain\",\"cheese\"]))\n",
        "\n",
        "print(model.wv.doesnt_match([\"year\",\"book\",\"month\",\"day\"]))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "tags": [],
        "cell_id": "00082-a320400d-7dfc-4a31-9e15-459b7300c6fa",
        "deepnote_cell_type": "markdown",
        "id": "SEiVojZmKS42"
      },
      "source": [
        "## Visualization\n",
        "\n",
        "\n",
        "In order to be able to visualize word embeddings in vector space, we need to use a dimentionality reduction method.\n",
        "\n",
        "Embedding projector visualizes the word2vec and any other uploaded word embedding model.\n",
        "https://projector.tensorflow.org\n",
        "\n",
        "**Exercise** Load a word2vec model look for a word you find interesting and find the 10 words most close to it isolate them and upload an screen-shot in the next cell. The following cell contains an example of the word `watergate` and the top 10 closest words.\n",
        "(You should upload your image) "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "tags": [],
        "cell_id": "00083-669e47be-4c2c-4569-b96c-7ce85556759f",
        "deepnote_cell_type": "markdown",
        "id": "OhnwbKC7KS43"
      },
      "source": [
        "<img src = \"embedding_watergate.png\">"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "tags": [],
        "cell_id": "00085-f86cd173-70a0-40ae-aac4-4146eb66ea67",
        "deepnote_cell_type": "markdown",
        "id": "AWVCBK5lKS43"
      },
      "source": [
        "**IF YOU FANCY** Download one of the gensim models from this repository in your preferred language and run the functions from gensim Word2Vec model class on samples.\n",
        "http://vectors.nlpl.eu/repository/"
      ]
    }
  ]
}