6.WordEmbeddings.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "colab": {
      "name": "WordEmbeddings.ipynb",
      "provenance": []
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "code",
      "metadata": {
        "id": "fAMadjTEylBb",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "!pip install --upgrade grpcio\n",
        "!pip install tensorflow==2.0"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "vBPDXmgYygQI",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "from __future__ import absolute_import, division, print_function, unicode_literals\n",
        "\n",
        "import tensorflow as tf"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "HZqib5wfygQR",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "from tensorflow import keras\n",
        "from tensorflow.keras import layers\n",
        "import tensorflow_datasets as tfds"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Dlj_ShbPygQX",
        "colab_type": "text"
      },
      "source": [
        "# Word Embeddings "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "TG5WTIH2ygQY",
        "colab_type": "text"
      },
      "source": [
        "The embedding layer acts as a lookup table that maps the integer indices to vectors. When we create an embedding layer, the weights are randomly initialized and then adjusted via backpropogation during training."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "CDyLWWRtygQa",
        "colab_type": "text"
      },
      "source": [
        "The parameters to the Embedding Layer are input_dim, output_dim, and input size."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "J4QUq652ygQb",
        "colab_type": "text"
      },
      "source": [
        " *input_dim* is the size of the vocabulary."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "VrB2TPsjygQd",
        "colab_type": "text"
      },
      "source": [
        "*output_dim* is the size of the vector space in which the words are embedded, i.e. the dimension of the vector embeddings."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "IfDO1iwtygQe",
        "colab_type": "text"
      },
      "source": [
        " *input_length* is the length of the input sequence."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "EWHhNtQIygQg",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "embedding_layer = layers.Embedding(1000, 5)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8dd34t3HygQk",
        "colab_type": "text"
      },
      "source": [
        "Passing a sequence of integers to the embedding layer returns the floating point vector which is mapped to the integer sequence. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "_ImVa4NoygQm",
        "colab_type": "code",
        "outputId": "da51d5e3-77e4-4e07-e052-e2cfcb8c8fdb",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 122
        }
      },
      "source": [
        "result = embedding_layer(tf.constant([1, 4, 7, 9]))\n",
        "print(dir(result))\n",
        "\n",
        "# We print the 5-dim vectors for each of the integer in the sequence provided above.\n",
        "print(result.numpy())\n"
      ],
      "execution_count": 6,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "['OVERLOADABLE_OPERATORS', '_USE_EQUALITY', '__abs__', '__add__', '__and__', '__array__', '__array_priority__', '__bool__', '__class__', '__copy__', '__deepcopy__', '__delattr__', '__dict__', '__dir__', '__div__', '__doc__', '__eq__', '__float__', '__floordiv__', '__format__', '__ge__', '__getattribute__', '__getitem__', '__gt__', '__hash__', '__index__', '__init__', '__init_subclass__', '__int__', '__invert__', '__iter__', '__le__', '__len__', '__long__', '__lt__', '__matmul__', '__mod__', '__module__', '__mul__', '__ne__', '__neg__', '__new__', '__nonzero__', '__or__', '__pow__', '__radd__', '__rand__', '__rdiv__', '__reduce__', '__reduce_ex__', '__repr__', '__rfloordiv__', '__rmatmul__', '__rmod__', '__rmul__', '__ror__', '__rpow__', '__rsub__', '__rtruediv__', '__rxor__', '__setattr__', '__sizeof__', '__str__', '__sub__', '__subclasshook__', '__truediv__', '__weakref__', '__xor__', '_add_consumer', '_as_node_def_input', '_as_tf_output', '_c_api_shape', '_copy', '_copy_nograd', '_copy_to_device', '_datatype_enum', '_disallow_bool_casting', '_disallow_in_graph_mode', '_disallow_iteration', '_disallow_when_autograph_disabled', '_disallow_when_autograph_enabled', '_get_input_ops_without_shapes', '_handle_data', '_id', '_keras_mask', '_num_elements', '_numpy', '_override_operator', '_rank', '_shape', '_shape_as_list', '_shape_tuple', '_tensor_shape', '_tf_api_names', '_tf_api_names_v1', 'backing_device', 'consumers', 'cpu', 'device', 'dtype', 'eval', 'experimental_ref', 'get_shape', 'gpu', 'graph', 'name', 'ndim', 'numpy', 'op', 'set_shape', 'shape', 'value_index']\n",
            "[[ 0.03986926 -0.03173438 -0.02799033  0.03823464  0.03856868]\n",
            " [ 0.00484568  0.0084567   0.00654433  0.01273901  0.03975483]\n",
            " [-0.00329332 -0.04582177 -0.02787818  0.03070165  0.02871773]\n",
            " [-0.02884469  0.03908208  0.02034127 -0.03785248 -0.01361613]]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Ukga5ShAygQs",
        "colab_type": "text"
      },
      "source": [
        "We can pass a 2D tensor of integers to the Embedding layer, of shape **(number_of_samples, sequence_length)**. Each entry is a sequence of integers. The output is a 3D floating point tensor, which has shape **(number_of_samples, sequence_length, embedding_dim)**."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "SkTkSjyBygQt",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "result = embedding_layer(tf.constant([[1,2,4], [2,3,4]]))"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "0__vrmBfygQy",
        "colab_type": "code",
        "outputId": "c442219e-1c79-466f-eee8-e5f759a5e929",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 136
        }
      },
      "source": [
        "print(result.numpy())"
      ],
      "execution_count": 8,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[[[ 0.03986926 -0.03173438 -0.02799033  0.03823464  0.03856868]\n",
            "  [-0.02281406  0.02966856  0.02286342  0.03767908  0.03811861]\n",
            "  [ 0.00484568  0.0084567   0.00654433  0.01273901  0.03975483]]\n",
            "\n",
            " [[-0.02281406  0.02966856  0.02286342  0.03767908  0.03811861]\n",
            "  [ 0.02198944  0.02700372  0.03230542  0.02984843 -0.00549867]\n",
            "  [ 0.00484568  0.0084567   0.00654433  0.01273901  0.03975483]]]\n"
          ],
          "name": "stdout"
        }
      ]
    }
  ]
}