NLTK.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Natural Language ToolKit (NLTK)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Importing the NLTK library\n",
    "import nltk"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [],
   "source": [
    "data = '''A man and a young teenage boy checked into a hotel and were shown to their room. The receptionist noted the quiet manner of the guests and the pale appearance of the boy. Later, the man and boy ate dinner in the hotel restaurant.\n",
    "\n",
    "The staff again noticed that the two guests were very quiet and that the boy seemed disinterested in his food.\n",
    "\n",
    "After eating, the boy went to his room and the man went to ask the receptionist to see the manager. The receptionist initially asked if there was a problem with the service or the room, and offered to fix things, but the man said that there was no problem of the sort and repeated his request.\n",
    "\n",
    "When the manager appeared, he took him aside and explained that he was spending the night in the hotel with his fourteen-year-old son, who was seriously ill, probably terminally so. The boy was very soon to undergo therapy, which would cause him to lose his hair. They had come to the hotel to have a break together and also because the boy planned to shave his head, that night, rather than feel that the illness was beating him. The father said that he would be shaving his own head too, in support of his son.\n",
    "\n",
    "He asked that staff be respectful when the two of them came to breakfast with their shaved heads.\n",
    "\n",
    "The manager assured the father that he would inform all staff and that they would behave appropriately.\n",
    "\n",
    "The following morning the father and son entered the restaurant for breakfast. There they saw the four male restaurant staff attending to their duties, perfectly normally, all with shaved heads.\n",
    "\n",
    "No matter what business you are in, you can help people and you can make a difference.'''"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "str"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#type of the data\n",
    "type(data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1638"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#returns total characters\n",
    "len(data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Frequency Distribution"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Tokenizer\n",
    "from nltk.tokenize import sent_tokenize, word_tokenize, TreebankWordTokenizer, wordpunct_tokenize"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [],
   "source": [
    "#splitting the data into tokens\n",
    "tokens = word_tokenize(data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[('the', 34), ('.', 15), (',', 14), ('and', 13), ('to', 12), ('that', 10), ('his', 8), ('boy', 7)]\n"
     ]
    }
   ],
   "source": [
    "from nltk.probability import FreqDist\n",
    "fdist = FreqDist()\n",
    "\n",
    "for i in tokens:\n",
    "    fdist[i.lower()]+=1\n",
    "print(fdist.most_common(8))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('the', 34),\n",
       " ('.', 15),\n",
       " (',', 14),\n",
       " ('and', 13),\n",
       " ('to', 12),\n",
       " ('that', 10),\n",
       " ('his', 8),\n",
       " ('boy', 7),\n",
       " ('a', 6),\n",
       " ('was', 6)]"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#printing top 10 frequent words\n",
    "fdist_top10 = fdist.most_common(10)\n",
    "fdist_top10"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Tokenization"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['Mumbai', 'is', 'one', 'of', 'the', 'most', 'populous', 'city', 'in', 'the', 'world']\n"
     ]
    }
   ],
   "source": [
    "#word_tokenizer\n",
    "from nltk.tokenize import word_tokenize\n",
    "txt1 = 'Mumbai is one of the most populous city in the world'\n",
    "\n",
    "print(word_tokenize(txt1))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['All pencils are pens.', 'Some books are pens.']\n"
     ]
    }
   ],
   "source": [
    "#sentence_Tokenizer\n",
    "from nltk.tokenize import sent_tokenize\n",
    "txt2 = 'All pencils are pens. Some books are pens.'\n",
    "\n",
    "print(sent_tokenize(txt2))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "8"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#Blank_line tokenizer splits the data into paragrahs\n",
    "from nltk.tokenize import blankline_tokenize\n",
    "bl_tokenize = blankline_tokenize(data)\n",
    "len(bl_tokenize)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['Hello', 'John', ',', 'How', 'are', 'you', 'doing', '?']\n"
     ]
    }
   ],
   "source": [
    "#Word Punct Tokenizer\n",
    "from nltk.tokenize import wordpunct_tokenize\n",
    "txt = 'Hello John, How are you doing ?'\n",
    "\n",
    "print(wordpunct_tokenize(txt))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['A', 'man', 'and', 'a', 'young', 'teenage', 'boy', 'checked', 'into', 'a', 'hotel', 'and', 'were', 'shown', 'to', 'their', 'room.', 'The', 'receptionist', 'noted', 'the', 'quiet', 'manner', 'of', 'the', 'guests', 'and', 'the', 'pale', 'appearance', 'of', 'the', 'boy.', 'Later', ',', 'the', 'man', 'and', 'boy', 'ate', 'dinner', 'in', 'the', 'hotel', 'restaurant.', 'The', 'staff', 'again', 'noticed', 'that', 'the', 'two', 'guests', 'were', 'very', 'quiet', 'and', 'that', 'the', 'boy', 'seemed', 'disinterested', 'in', 'his', 'food.', 'After', 'eating', ',', 'the', 'boy', 'went', 'to', 'his', 'room', 'and', 'the', 'man', 'went', 'to', 'ask', 'the', 'receptionist', 'to', 'see', 'the', 'manager.', 'The', 'receptionist', 'initially', 'asked', 'if', 'there', 'was', 'a', 'problem', 'with', 'the', 'service', 'or', 'the', 'room', ',', 'and', 'offered', 'to', 'fix', 'things', ',', 'but', 'the', 'man', 'said', 'that', 'there', 'was', 'no', 'problem', 'of', 'the', 'sort', 'and', 'repeated', 'his', 'request.', 'When', 'the', 'manager', 'appeared', ',', 'he', 'took', 'him', 'aside', 'and', 'explained', 'that', 'he', 'was', 'spending', 'the', 'night', 'in', 'the', 'hotel', 'with', 'his', 'fourteen-year-old', 'son', ',', 'who', 'was', 'seriously', 'ill', ',', 'probably', 'terminally', 'so.', 'The', 'boy', 'was', 'very', 'soon', 'to', 'undergo', 'therapy', ',', 'which', 'would', 'cause', 'him', 'to', 'lose', 'his', 'hair.', 'They', 'had', 'come', 'to', 'the', 'hotel', 'to', 'have', 'a', 'break', 'together', 'and', 'also', 'because', 'the', 'boy', 'planned', 'to', 'shave', 'his', 'head', ',', 'that', 'night', ',', 'rather', 'than', 'feel', 'that', 'the', 'illness', 'was', 'beating', 'him.', 'The', 'father', 'said', 'that', 'he', 'would', 'be', 'shaving', 'his', 'own', 'head', 'too', ',', 'in', 'support', 'of', 'his', 'son.', 'He', 'asked', 'that', 'staff', 'be', 'respectful', 'when', 'the', 'two', 'of', 'them', 'came', 'to', 'breakfast', 'with', 'their', 'shaved', 'heads.', 'The', 'manager', 'assured', 'the', 'father', 'that', 'he', 'would', 'inform', 'all', 'staff', 'and', 'that', 'they', 'would', 'behave', 'appropriately.', 'The', 'following', 'morning', 'the', 'father', 'and', 'son', 'entered', 'the', 'restaurant', 'for', 'breakfast.', 'There', 'they', 'saw', 'the', 'four', 'male', 'restaurant', 'staff', 'attending', 'to', 'their', 'duties', ',', 'perfectly', 'normally', ',', 'all', 'with', 'shaved', 'heads.', 'No', 'matter', 'what', 'business', 'you', 'are', 'in', ',', 'you', 'can', 'help', 'people', 'and', 'you', 'can', 'make', 'a', 'difference', '.']\n"
     ]
    }
   ],
   "source": [
    "treebank_tokenizer = TreebankWordTokenizer()\n",
    "x = treebank_tokenizer.tokenize(data)\n",
    "\n",
    "print(x)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Bigrams, Trigrams and Ngrams"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [],
   "source": [
    "#importing bigrams, trigrams and ngrams\n",
    "from nltk.util import bigrams, trigrams, ngrams"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [],
   "source": [
    "string = 'A man and a young teenage boy checked into a hotel and were shown to their room. The receptionist noted the quiet manner of the guests and the pale appearance of the boy.'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['A',\n",
       " 'man',\n",
       " 'and',\n",
       " 'a',\n",
       " 'young',\n",
       " 'teenage',\n",
       " 'boy',\n",
       " 'checked',\n",
       " 'into',\n",
       " 'a',\n",
       " 'hotel',\n",
       " 'and',\n",
       " 'were',\n",
       " 'shown',\n",
       " 'to',\n",
       " 'their',\n",
       " 'room',\n",
       " '.',\n",
       " 'The',\n",
       " 'receptionist',\n",
       " 'noted',\n",
       " 'the',\n",
       " 'quiet',\n",
       " 'manner',\n",
       " 'of',\n",
       " 'the',\n",
       " 'guests',\n",
       " 'and',\n",
       " 'the',\n",
       " 'pale',\n",
       " 'appearance',\n",
       " 'of',\n",
       " 'the',\n",
       " 'boy',\n",
       " '.']"
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#Unigram or Word Tokenizer\n",
    "string_tokens = nltk.word_tokenize(string)\n",
    "string_tokens"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('A', 'man'),\n",
       " ('man', 'and'),\n",
       " ('and', 'a'),\n",
       " ('a', 'young'),\n",
       " ('young', 'teenage'),\n",
       " ('teenage', 'boy'),\n",
       " ('boy', 'checked'),\n",
       " ('checked', 'into'),\n",
       " ('into', 'a'),\n",
       " ('a', 'hotel'),\n",
       " ('hotel', 'and'),\n",
       " ('and', 'were'),\n",
       " ('were', 'shown'),\n",
       " ('shown', 'to'),\n",
       " ('to', 'their'),\n",
       " ('their', 'room'),\n",
       " ('room', '.'),\n",
       " ('.', 'The'),\n",
       " ('The', 'receptionist'),\n",
       " ('receptionist', 'noted'),\n",
       " ('noted', 'the'),\n",
       " ('the', 'quiet'),\n",
       " ('quiet', 'manner'),\n",
       " ('manner', 'of'),\n",
       " ('of', 'the'),\n",
       " ('the', 'guests'),\n",
       " ('guests', 'and'),\n",
       " ('and', 'the'),\n",
       " ('the', 'pale'),\n",
       " ('pale', 'appearance'),\n",
       " ('appearance', 'of'),\n",
       " ('of', 'the'),\n",
       " ('the', 'boy'),\n",
       " ('boy', '.')]"
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#creating list of tokens with two consecutive words \n",
    "string_bigrams = list(nltk.bigrams(string_tokens))\n",
    "string_bigrams"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('A', 'man', 'and', 'a'),\n",
       " ('man', 'and', 'a', 'young'),\n",
       " ('and', 'a', 'young', 'teenage'),\n",
       " ('a', 'young', 'teenage', 'boy'),\n",
       " ('young', 'teenage', 'boy', 'checked'),\n",
       " ('teenage', 'boy', 'checked', 'into'),\n",
       " ('boy', 'checked', 'into', 'a'),\n",
       " ('checked', 'into', 'a', 'hotel'),\n",
       " ('into', 'a', 'hotel', 'and'),\n",
       " ('a', 'hotel', 'and', 'were'),\n",
       " ('hotel', 'and', 'were', 'shown'),\n",
       " ('and', 'were', 'shown', 'to'),\n",
       " ('were', 'shown', 'to', 'their'),\n",
       " ('shown', 'to', 'their', 'room'),\n",
       " ('to', 'their', 'room', '.'),\n",
       " ('their', 'room', '.', 'The'),\n",
       " ('room', '.', 'The', 'receptionist'),\n",
       " ('.', 'The', 'receptionist', 'noted'),\n",
       " ('The', 'receptionist', 'noted', 'the'),\n",
       " ('receptionist', 'noted', 'the', 'quiet'),\n",
       " ('noted', 'the', 'quiet', 'manner'),\n",
       " ('the', 'quiet', 'manner', 'of'),\n",
       " ('quiet', 'manner', 'of', 'the'),\n",
       " ('manner', 'of', 'the', 'guests'),\n",
       " ('of', 'the', 'guests', 'and'),\n",
       " ('the', 'guests', 'and', 'the'),\n",
       " ('guests', 'and', 'the', 'pale'),\n",
       " ('and', 'the', 'pale', 'appearance'),\n",
       " ('the', 'pale', 'appearance', 'of'),\n",
       " ('pale', 'appearance', 'of', 'the'),\n",
       " ('appearance', 'of', 'the', 'boy'),\n",
       " ('of', 'the', 'boy', '.')]"
      ]
     },
     "execution_count": 49,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#creating list of tokens with four consecutive words\n",
    "string_ngrams = list(nltk.ngrams(string_tokens,4))\n",
    "string_ngrams"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Stemming"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "connecting : connect\n",
      "connected : connect\n",
      "connects : connect\n"
     ]
    }
   ],
   "source": [
    "#stemming the tokens into their root word\n",
    "from nltk.stem.porter import PorterStemmer\n",
    "ps = PorterStemmer()\n",
    "word = ['connecting','connected','connects']\n",
    "\n",
    "for i in word:\n",
    "    print(i ,\":\", ps.stem(i) )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Lemmatization"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "better : good\n",
      "cleverer : clever\n",
      "farest : far\n"
     ]
    }
   ],
   "source": [
    "#Lemmatizing the tokens into their dictionary form\n",
    "from nltk.stem.wordnet import WordNetLemmatizer\n",
    "lem = WordNetLemmatizer()\n",
    "words = ['better','cleverer','farest']\n",
    "\n",
    "for i in words:\n",
    "    print(i ,\":\", lem.lemmatize(i,'a'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "fry\n",
      "fri\n"
     ]
    }
   ],
   "source": [
    "word = 'fries'\n",
    "z = lem.lemmatize(word,'v')\n",
    "y = ps.stem(word)\n",
    "print(z)\n",
    "print(y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### StopWords"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['Hello', 'John', ',', 'How', 'are', 'you', 'doing', '?']\n",
      "['Hello', 'John', ',', 'How', '?']\n"
     ]
    }
   ],
   "source": [
    "#Removing the stopwords\n",
    "from nltk.corpus import stopwords\n",
    "stop_words = set(stopwords.words('english'))\n",
    "\n",
    "text = 'Hello John, How are you doing ?'\n",
    "tokens = word_tokenize(text)\n",
    "\n",
    "filtered_text = [] \n",
    "for w in tokens: \n",
    "    if w not in stop_words: \n",
    "        filtered_text.append(w) \n",
    "  \n",
    "print(tokens) \n",
    "print(filtered_text) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### POS_Tagging"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[('She', 'PRP'), ('likes', 'VBZ'), ('classical', 'JJ'), ('music', 'NN')]\n"
     ]
    }
   ],
   "source": [
    "#POS_tagging\n",
    "from nltk import pos_tag\n",
    "txtt = 'She likes classical music'\n",
    "tokens=nltk.word_tokenize(txtt)\n",
    "\n",
    "print(nltk.pos_tag(tokens))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Named Entity Recognition"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(S\n",
      "  (PERSON Sundar/NNP)\n",
      "  (PERSON Pichai/NNP)\n",
      "  is/VBZ\n",
      "  the/DT\n",
      "  (ORGANIZATION CEO/NN of/IN Google/NNP))\n"
     ]
    }
   ],
   "source": [
    "#Named Entity Recognition\n",
    "from nltk import ne_chunk\n",
    "file = 'Sundar Pichai is the CEO of Google'\n",
    "\n",
    "ne_tokens = nltk.word_tokenize(file)\n",
    "ne_tags = nltk.pos_tag(ne_tokens)\n",
    "ne_ner = ne_chunk(ne_tags)\n",
    "print(ne_ner)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### TFIDF Vectorization"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [],
   "source": [
    "import re"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['A man and a young teenage boy checked into a hotel and were shown to their room.',\n",
       " 'The receptionist noted the quiet manner of the guests and the pale appearance of the boy.',\n",
       " 'Later, the man and boy ate dinner in the hotel restaurant.',\n",
       " 'The staff again noticed that the two guests were very quiet and that the boy seemed disinterested in his food.',\n",
       " 'After eating, the boy went to his room and the man went to ask the receptionist to see the manager.',\n",
       " 'The receptionist initially asked if there was a problem with the service or the room, and offered to fix things, but the man said that there was no problem of the sort and repeated his request.',\n",
       " 'When the manager appeared, he took him aside and explained that he was spending the night in the hotel with his fourteen-year-old son, who was seriously ill, probably terminally so.',\n",
       " 'The boy was very soon to undergo therapy, which would cause him to lose his hair.',\n",
       " 'They had come to the hotel to have a break together and also because the boy planned to shave his head, that night, rather than feel that the illness was beating him.',\n",
       " 'The father said that he would be shaving his own head too, in support of his son.',\n",
       " 'He asked that staff be respectful when the two of them came to breakfast with their shaved heads.',\n",
       " 'The manager assured the father that he would inform all staff and that they would behave appropriately.',\n",
       " 'The following morning the father and son entered the restaurant for breakfast.',\n",
       " 'There they saw the four male restaurant staff attending to their duties, perfectly normally, all with shaved heads.',\n",
       " 'No matter what business you are in, you can help people and you can make a difference.']"
      ]
     },
     "execution_count": 57,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "senten = nltk.sent_tokenize(data)\n",
    "senten"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [],
   "source": [
    "#cleaning the text\n",
    "filtered = []\n",
    "for i in range(len(senten)):\n",
    "    c_txt = re.sub('[^a-zA-Z]',' ', senten[i])           #removing special characters, numbers and punctuations\n",
    "    c_txt = c_txt.lower()                                #converting into lower case\n",
    "    c_txt = c_txt.split()                                #splitting into tokens\n",
    "    c_txt = [lem.lemmatize(word) for word in c_txt if not word in set(stopwords.words('english'))]   #removing stopwords\n",
    "    c_txt = ' '.join(c_txt)                             #rejoining the words\n",
    "    filtered.append(c_txt)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['man young teenage boy checked hotel shown room',\n",
       " 'receptionist noted quiet manner guest pale appearance boy',\n",
       " 'later man boy ate dinner hotel restaurant',\n",
       " 'staff noticed two guest quiet boy seemed disinterested food',\n",
       " 'eating boy went room man went ask receptionist see manager',\n",
       " 'receptionist initially asked problem service room offered fix thing man said problem sort repeated request',\n",
       " 'manager appeared took aside explained spending night hotel fourteen year old son seriously ill probably terminally',\n",
       " 'boy soon undergo therapy would cause lose hair',\n",
       " 'come hotel break together also boy planned shave head night rather feel illness beating',\n",
       " 'father said would shaving head support son',\n",
       " 'asked staff respectful two came breakfast shaved head',\n",
       " 'manager assured father would inform staff would behave appropriately',\n",
       " 'following morning father son entered restaurant breakfast',\n",
       " 'saw four male restaurant staff attending duty perfectly normally shaved head',\n",
       " 'matter business help people make difference']"
      ]
     },
     "execution_count": 59,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "filtered"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.feature_extraction.text import TfidfVectorizer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = TfidfVectorizer()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [],
   "source": [
    "x = model.fit_transform(filtered).toarray()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "        all       are     books   pencils      pens  sketches      some\n",
      "0  0.584483  0.345205  0.000000  0.444514  0.584483  0.000000  0.000000\n",
      "1  0.000000  0.409123  0.526820  0.526820  0.000000  0.000000  0.526820\n",
      "2  0.000000  0.373119  0.480458  0.000000  0.000000  0.631745  0.480458\n"
     ]
    }
   ],
   "source": [
    "documentA = 'All pens are pencils.'\n",
    "documentB = 'some pencils are books'\n",
    "documentC = 'Some books are sketches'\n",
    "\n",
    "vectorizer = TfidfVectorizer()\n",
    "vectors = vectorizer.fit_transform([documentA, documentB, documentC])\n",
    "feature_names = vectorizer.get_feature_names()\n",
    "dense = vectors.todense()\n",
    "denselist = dense.tolist()\n",
    "df = pd.DataFrame(denselist, columns=feature_names)\n",
    "\n",
    "print(df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Count Vectorizer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [],
   "source": [
    "text1 = ['You will be prompted for your password.','The password is your PAN number in upper case.']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.feature_extraction.text import CountVectorizer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {},
   "outputs": [],
   "source": [
    "cv = CountVectorizer()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {},
   "outputs": [],
   "source": [
    "vect = cv.fit_transform(text1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {},
   "outputs": [],
   "source": [
    "cnt_vect = cv.get_feature_names()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>be</th>\n",
       "      <th>case</th>\n",
       "      <th>for</th>\n",
       "      <th>in</th>\n",
       "      <th>is</th>\n",
       "      <th>number</th>\n",
       "      <th>pan</th>\n",
       "      <th>password</th>\n",
       "      <th>prompted</th>\n",
       "      <th>the</th>\n",
       "      <th>upper</th>\n",
       "      <th>will</th>\n",
       "      <th>you</th>\n",
       "      <th>your</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>doc1</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>doc2</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      be  case  for  in  is  number  pan  password  prompted  the  upper  \\\n",
       "doc1   1     0    1   0   0       0    0         1         1    0      0   \n",
       "doc2   0     1    0   1   1       1    1         1         0    1      1   \n",
       "\n",
       "      will  you  your  \n",
       "doc1     1    1     1  \n",
       "doc2     0    0     1  "
      ]
     },
     "execution_count": 69,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.DataFrame(data = vect.toarray(), index = ['doc1','doc2'], columns=cnt_vect)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}