loading_corpora.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Loading Davies Corpora\n",
    "\n",
    "This notebook will explore the corpora in this folder."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import re\n",
    "import zipfile\n",
    "import os\n",
    "import sys"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Loading raw data\n",
    "\n",
    "The following method iterates through the files in the folder, and unzips the files, storing them in a dictionary with each zip file mapping to a list of the texts.\n",
    "\n",
    "```corpus_name``` is a string which contains the directory of the corpus you need to use. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "corpus_name = \"/Users/bhargavvader/Downloads/Movies\"\n",
    "# corpus_name = \"Movies\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "def loadcorpus(corpus_name, corpus_style=\"text\"):\n",
    "    texts_raw = {}\n",
    "    for file in os.listdir(corpus_name + \"/\"):\n",
    "        if corpus_style in file:\n",
    "            print(file)\n",
    "            zfile = zipfile.ZipFile(corpus_name + \"/\" + file)\n",
    "            for file in zfile.namelist():\n",
    "                texts_raw[file] = []\n",
    "                with zfile.open(file) as f:\n",
    "                    for line in f:\n",
    "                        texts_raw[file].append(line)\n",
    "    return texts_raw"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will be using the movies corpus for our purposes, but you can uncomment the code and try out the other corpora too.\n",
    "You might have to make some adjustments in the cleaning for the other corpora; I have tried it for most of them and it works fine."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "text_13_idi.zip\n",
      "text_16_qll.zip\n",
      "text_32_ldf.zip\n",
      "text_19_gvc.zip\n",
      "text_05_nko.zip\n",
      "text_17_arp.zip\n",
      "text_01_ote.zip\n",
      "text_28_rfy.zip\n",
      "text_31_akv.zip\n",
      "text_22_etp.zip\n",
      "text_11_uoy.zip\n",
      "text_09_oii.zip\n",
      "text_06_jfy.zip\n",
      "text_14_lnc.zip\n",
      "text_08_loh.zip\n",
      "text_33_kje.zip\n",
      "text_30_wkp.zip\n",
      "text_07_oma.zip\n",
      "text_03_mnq.zip\n",
      "text_21_fqa.zip\n",
      "text_29_oye.zip\n",
      "text_27_fle.zip\n",
      "text_23_fmh.zip\n",
      "text_12_rcq.zip\n",
      "text_00_myn.zip\n",
      "text_10_aoy.zip\n",
      "text_04_mlq.zip\n",
      "text_20_cde.zip\n",
      "text_02_mqu.zip\n",
      "text_26_ngj.zip\n",
      "text_24_ywo.zip\n",
      "text_18_jfj.zip\n",
      "text_25_byg.zip\n",
      "text_15_guo.zip\n"
     ]
    }
   ],
   "source": [
    "movie_raw = loadcorpus(corpus_name)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "# tv_raw = loadcorpus(\"TV\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "# wiki_raw = loadcorpus(\"Wiki\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "# soap_raw = loadcorpus(\"SOAP\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "# span_raw = loadcorpus(\"SPAN\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "# now_raw = loadcorpus(\"NOW\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "# web_raw = loadcorpus(\"iWeb\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "# glowbe_raw = loadcorpus(\"GloWbe\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "# coha_raw = loadcorpus(\"COHA\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "# coca_raw = loadcorpus(\"COCA\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let us look at what the raw movie data looks like, and what we would like to use."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "b'@@3512517 I \\'m \\' most frightened to death . Sure , after you \\'ve done it eight or nine times , you wo n\\'t even give it a thought . - Gee , Dot , you look swell . - Am I all right ? Lovely . dddd Well , there goes the maiden \\'s prayer . I wonder how I \\'ll act . It \\'s like diving overboard-you never know how the water \\'s going to be till you hit it . - I \\'m so nervous . - Say ... if I could look like you in a wedding gown , I \\'d be a bigamist . Come on . dddd I say , is n\\'t that girl in the bride \\'s outfit a new model ? Why , yes . She \\'s a salesgirl downstairs . We \\'re trying her out . She \\'s got my okay . These guys usually make wisecracks . Do n\\'t let it bother you . I know all the answers- men have been insulting me for years . Say , beautiful . Doing anything tonight ? I \\'m taking my two pet fish out for a drive . There \\'ll be @ @ @ @ @ @ @ @ @ @ Do n\\'t talk back to them . You \\'ll get fired . When they deliver baloney at my door , I always give them a receipt . dddd Honestly , if I ever met a fella ... that did n\\'t try to date me up right off the bat , I think I \\'d fall on his neck . He \\'d probably be too weak to hold you up . Oh , men are all alike . Rich or poor- when it comes to women , they \\'ve only got one idea in their head . Take those fellas out there today- When they did n\\'t try to date you up ... you could tell what they were thinking by the way they looked at ya . And it is n\\'t only those swell society guys either . Ridin \\' the subways during the rush hour , fellas brushing \\' up against you-you know what . Or sit in the theater . Right away some stranger ... takes a seat next to you and that knee business starts . - You said it . - Do n\\'t I know it @ @ @ @ @ @ @ @ @ @ . - Good night . After all , I guess nature made men that way . It \\'s their great weakness . And how we women encourage that weakness . - Well , I don\\'t . - Oh , no . I know you do n\\'t , Dot . - Oh , Mlss Haley . - Yes , Mr . Thompson . - Good night , Dot . Good night . - So long . I just wanted to tell you how awfully well you did this afternoon . - Oh , thank you so much , Mr . Thompson . - You were marvelous . It \\'s been such a hot day- I thought you might like to take a drive . Oh , I \\'d just love to , Mr . Thompson , but , you see ... - my husband \\'s a prizefighter . - Oh . Yes . And he heard about a fella here in the store who tried to make a date with me ... and he \\'s waiting downstairs now to beat him up . - Well - Oh . - Yes . @ @ @ @ @ @ @ @ @ @ take the ride , I had nothing wrong in mind . Oh , no , Mr . Thompson . You see , I never think of things like that . Well , good night . Gee , Dot , that husband prizefighter gag \\'s a pip . I \\'m gon na use that one too . - Do n\\'t men ever think of anything else ? - Well , hardly ever . dddd dddd What are you looking at the river for , baby ? You ai n\\'t expecting anybody , are ya ? Yeah , my grandfather \\'s swimming in from Europe tonight . I got ya . Well ... when the old guy gets here , give him my best regards . You think you can spare any ? Hey , Dot . - Hey , Dot . Come on . I \\'ve got something to show ya . - What ? - There \\'s a fella over here who wo n\\'t flirt . - There ai n\\'t no such animal . - On the level . - His wife must be watching him . No , he @ @ @ @ @ @ @ @ @ @ tumble . A girl ca n\\'t get a tumble ? Wan na bet a quarter ? - That you can make him speak to you ? - Yeah ! - All right . A quarter . - The bet \\'s on . Where is he ? Come on . dd d You want a little sweet Come , come , say pretty please d d Come on , baby and beg for it d d You want a little kiss So pucker up like this d d Coax me nice and gentle , baby d - d Maybe I \\'ll fall dd - Hey . Do you have to do that ? - I \\'m singin \\' . - I had an aunt once who used to make noises like that . She called it \\' \\' neuraliga . \" Neuralgia must run in your family . Yeah ? Well , listen , kid . If you want to prove that something runs in your family , this is the spot . You got a clear deck . - I do n\\'t know whether I won or not . - @ @ @ @ @ @ @ @ @ @ bet her a quarter she could n\\'t get you to talk to her . - Which one of us wins ? - Oh . So you \\'re a couple of them smart janes , huh ? No , we \\'re not . But almost every fella we meet makes a pass at us . When my girlfriend here said you would n\\'t flirt , I just could n\\'t believe it . I suppose you think when you pass by ... guys just naturally drop dead . - No , I don\\'t . - If you do n\\'t want them to flirt with you ... what do you dress like that for ? You wear those clothes for what ? To show off the curves of your body . Look at the waist . What \\'s that so low for ? Yeah . So some guy can get an eyeful . And your dress is deliberately made for the wind to blow up over your knees . Listen , sister . If you do n\\'t want guys to salute ya , take down your flag . You \\'re like all other @ @ @ @ @ @ @ @ @ @ , you wan na call out the police . And if he do n\\'t , you wan na call the army . - You do n\\'t seem to like me . - I think you \\'re a pain in the neck . Well , my name is Dorothy Haley . You ai n\\'t kiddin \\' me a bit . You \\'re like all the other fellas . You just have a new line . That \\'s all . - You \\'ll be making a pass at us yet . - We \\'d make a great team- with my- and your conceit . - Oh , hello , Edna . - Hello there , Jane . How are you ? - Hey . My sister \\'s over here . Come over , will ya ? - Okay . - d Come on , baby d - Hey . Hey . - d You want to dd - Please . Please , will ya ? My nerves . dd Hey . Ai n\\'t there any other tune you can almost sing ? You ai n\\'t told me your name . - Joe @ @ @ @ @ @ @ @ @ @ . He drove a Ford . Did you ever meet him ? Well , I might \\'ve , at that . No . No . The guy I knew drove a Chevrolet , and his name was Fred . No . It could n\\'t have been the same guy . - Oh , you \\'re quite a kidder . - Yeah , you \\'re pretty fresh yourself . Just fresh enough . Say , what makes you such a grouch ? Janes like you . You work all day in some store ... and then you rush home to sleep at night in some stuffy tenement . And you \\'ll be content to do it for the rest of your life ... because you \\'re just a good-time girl . This is where you wan na be , with a lot of saps like that ... yellin \\' their heads off and thinkin \\' they \\'re enjoying themselves . You ai n\\'t got brains enough to realize ... that now is the time you ought a be saving your money ... and trying to make something better of yourself . @ @ @ @ @ @ @ @ @ @ ? - I \\'m here to get fresh air . But you notice I ai n\\'t spendin \\' any dough on you , do n\\'t ya ? Not even an ice cream soda . Say , listen , you big stiff . If you think I \\'m worried ... about you buying me an ice cream soda , you \\'re crazy . I would n\\'t be found dead with ya . Thanks for bringing me home . I had a swell time . Okay . Well , do you want to see me again ? Oh , I should say not . But accidents will happen . Well , I got ta go . I got ta go upstairs now . You see , my mother \\'s dead , and my brother \\'s boss of the house . He gets sore when I stay out late . You know , he \\'s careful for me . But as Edna says , you ca n\\'t watch a girl hard enough ... to keep her good if she do n\\'t want to be . Get 10 cents\\'worth of potato salad too . @ @ @ @ @ @ @ @ @ @ flat all smelled up again . Oh , I know what to get . Gee , I wish she would n\\'t holler down the stairs that way . It do n\\'t give the house no class . My mother always used to holler down the stairs . She did ? Yeah . So \\'s my old man . He sold his coat for a drink and caught pneumonia . He was a terrible lush . Gee . My old lady was nice though . - Hello , Dot . - Hello , Paula . Geez . Everything lives in this house . Nobody \\'ll speak to her anymore but me . It makes my brother sore when I do . But as Edna says , nobody knows whether a person \\'s good or bad but the person themselves . And they wo n\\'t tell . This Edna jane does most of your thinking , do n\\'t she ? Edna \\'s swell . She \\'s got a kid almost seven . She \\'s a widow . Jerome 7 ... 4-7-5-3 . Her mother \\'s been awful sick . Hello ? Esther @ @ @ @ @ @ @ @ @ @ Esther ... you got ta control yourself . 9:15 . I had just looked at the clock . You better stop in at Mr . Levant \\'s . Esther , you got ta get ahold of yourself . There \\'s things to be done . Good-bye . Your mother , Mrs . Gardner ? - Yes . - Oh , I \\'m so sorry for you . Thanks , Miss Haley . Still stays hot out , do n\\'t it ? Do you want me to go up with you , ma\\'am ? Thank you very kindly . I \\'m all right . I \\'m just a little tired , I guess . Good night . - Good night . - Good night . You know , a tenement house like this is awful funny . Here her mother \\'s dead on the fifth floor . And only this morning ... on the second floor ... Mrs . Mills had a new baby . Yeah ? Well , she did n\\'t do that kid no favor . - What do you mean ? - I mean bringing it into the @ @ @ @ @ @ @ @ @ @ take care of it . Oh , Joe , that \\'s a terrible thing to say . What \\'s the kid got to look forward to ? Starvin \\' himself to death in some tenement . Born on the second floor and ... probably die on the fifth . His whole life spent in climbin \\' three flights of stairs . - Oh , Joe , you \\'re terrible hard . - Yeah ? Well , here \\'s one guy \\'s gon na beat this game , see ? No poverty , no pinching \\' , no scrimping \\' for me . I got $580 saved up , see ? In a couple of months , I \\'m gon na have my own radio shop . Yeah . Gee , there are a lot of things in life besides just money . Yeah . Sure there \\'s a lot of things in life beside money . But you got ta have money to find them . Well ... I got ta go now . My brother will lay me out in lavender . I went to Coney Island once and @ @ @ @ @ @ @ @ @ @ . He hit me in the eye . - He did ? - Yeah . - Say , do you want me to go up with you ? - Oh , no . - That \\'d only make it worse . - Yeah ? Well , no guy \\'s gon na hit any jane I go out with , brother or no brother . Oh , he just wants to see that I keep straight . - But as Edna says-- Ah , who cares what Edna says ? Well ... I got ta go now . I told you not to bring any Llmburger cheese In here . I wo n\\'t have the flat all stunk up ! Well , It do n\\'t smell after It \\'s eaten . There \\'s a tenement for ya . A woman dies , a baby \\'s born ... and a guy \\'s wife wo n\\'t let him eat Limburger . Gee , you \\'re a funny fella , Joe . In fact , you \\'re the first guy I ever went out with that did n\\'t try to get fresh . Well @ @ @ @ @ @ @ @ @ @ , you know , I like you an awful lot . Okay . You \\'ll probably be just nutty enough to call me up at the shop . The number \\'s Schuyler 4 ... 6-5-8-3 . Four , 65 , 83 . That \\'s the way I always remember numbers . Well , good night . So long . Hey . Walt a minute . My name ain\\'tJoe . It \\'s Eddie Collins . Just a moment . Oh , Eddie . Oh , Eddie , uh , will you have this gentleman \\'s set finished by tomorrow morning ? Yes , sir . I \\'ll work on it tonight . You \\'ve got a great set . - Mmm . Your wife will enjoy that . - Oh , I have n\\'t any wife . - Oh . - That \\'s why I \\'m able to pay $300 for a radio . Well , there \\'s something to that . Of course , if you keep your radio turned on all the time ... you have the effect of a wife anyway . - Yes . - Oh , @ @ @ @ @ @ @ @ @ @ fella \\'s got the right idea . Eddie , now , you take a lesson from him . Do n\\'t get married . Not a Chinaman \\'s chance . He \\'s been getting a lot of telephone calls lately , Mr . Lathrop . - Ah . - Well , I \\'d hate to see you making any mistake , Eddie . Why , you can have your own little place in six months . - But not if you let a woman get a hold on ya . - Oh . I never even think about it . That \\'s the danger . Do you suppose any man would ever marry if he thought of it ? Why , I ca n\\'t even talk to a girl . You know , it \\'s funny about me that way . I \\'d like to be nice to women . You know , say nice things to them , like fellas can . I ca n\\'t though . I think of nice things to say . But when it comes to putting \\' em into words-I only say something sarcastic and @ @ @ @ @ @ @ @ @ @ ? - That \\'s a laugh . - Well , I hope so . Yeah . Hey , sheikh . Hello ? Hello , stupid . How are ya ? Sure , it \\'s me . Yeah . The same place . In front of Loft \\'s candy store . Listen , unconscious . Be on time . I ai n\\'t waiting \\' for any dizzy janes on a street corner , see ? Yeah . 7:30 . Okay . Uh-uh . Oh-Waiting for me , baby ? - I \\'m waiting for my husband . He \\'s the cop on this beat . - Excuse me . Yeah . - I was just going . - Well , ai n\\'t that swell . Say , you got a lot of nerve letting me stand out there in the pouring rain . Gee , I did n\\'t know it was so late . I got to foolin \\' around with this thing . Ai n\\'t it a pip ? It \\'ll bring in anything from KGO to L20 . - No kidding . - Yeah . Sweet job , huh @ @ @ @ @ @ @ @ @ @ I had to do with it . What do I care what you had to do with it ? I \\'m not Marconi ... or Edison or whoever invented the thing . - I \\'m standing out there-- All right . My mistake . Radio \\'s my job . I \\'m gon na have my own store soon . - I thought you were interested in my work . - I am interested , Eddie . But , gee , I got a right to be sore , standing out in the rain like that . Well , after all , you ca n\\'t saw sawdust . What kind of a crack is that ? \\' \\' You ca n\\'t saw sawdust . \" I mean it \\'s done . All I can do is say that I \\'m sorry . - Do you forgive me ? - Well , I got a right to be sore . All right . You \\'re sore . What are you kicking about ? - Well , I got a right . - All right . - Well ? - Well ? Aw @ @ @ @ @ @ @ @ @ @ , come on . Come on . Come on . Say it . Come on . Come on . Okay . How \\'d you ever find the place ? I remember you said it was on the third floor in the front . Come on . Take your coat off and stay a while . Oh , no , Eddie . Let \\'s go to a movie , huh ? Ah , look . It \\'s raining cats and dogs out . - Well , I ca n\\'t stay here . - What \\'s the matter with this ? My brother would rip me apart if he knew I was alone with a fella in his room . Suppose your landlady found me . Gee , she \\'d think I was a tramp or something . Ah , they do n\\'t expect nothing raw here . This ai n\\'t no $12-a-day hotel . Ah , let \\'s wait downstairs then . It might rain for an hour yet . Let \\'s stay here where they have ashtrays and things . I got ta go . Besides , it ai n\\'t @ @ @ @ @ @ @ @ @ @ you . - Did I ever get fresh with ya ? - No . But I ai n\\'t that kind of a girl . I do n\\'t go to fellas \\' rooms . Listen . I got other things to think about , see ? I \\'m gon na have my own store soon . I was talking to the boss about it only today . Oh , gee , Eddie . That \\'ll be swell . None of that mushy stuff for me . - Give us a kiss , will ya ? - Oh . None of that mushy stuff , huh ? Ah , well , a little of it \\'s all right . - Come on . Take off your coat and stay a while . - Oh , no . I ca n\\'t . Oh , no . - Eddie , now-- Come on . Take it off . Come on . - Take that hat off . - Stay - Oh , no . - Eddie , I \\'m not gon na take my hat off . - Come on . - Oh , @ @ @ @ @ @ @ @ @ @ gee . I did n\\'t know it was as late as that . - Ah , it \\'s early yet . - Well , I got ta go . Stopped raining . Hey , lookit . There \\'s a fella trying to kiss a girl . Where ? - Here . - Oh , Eddie . Oh , Eddie , please do n\\'t . I got ta go . Aw , gee . You ca n\\'t be with me two minutes but you got ta pull that \\' \\' I got ta go \" gag . Aw , no . But you do n\\'t understand . I must go . You \\'d think I was poison or something . - Oh , no , but it \\'s getting late , and I ca n\\'t stay any longer . - I know . You got ta go . Aw , Eddie . Do n\\'t get mad . I \\'m not mad . But you \\'re always talking about \\' \\' I got ta go . \" Well , I do have to go . It \\'s only about 4:00 . Yeah @ @ @ @ @ @ @ @ @ @ in my life . Oh , my brother will kill me . Aw , gee , kid . Please do n\\'t do that . If I could only find Edna . Aw , ca n\\'t you do anything without Edna \\'s help ? My brother \\'s stuck on her . If she was there , he would n\\'t be sore at me . At least he would n\\'t holler . He \\'s got a terrible temper . Well , you want to go over and try Edna \\'s house again ? No . If she was there , she \\'d open the door . She must be sleeping at her aunt\\'s . - Come on . - What are you gon na do ? I \\'m goin \\' up and tell your brother you were with me . No . You ca n\\'t go up . There \\'d only be a fight . Aw , there would n\\'t be any fight . He \\'d make one crack , I \\'d hang that in his jaw , and the fight would be all over . Oh , gee , Eddie . @ @ @ @ @ @ @ @ @ @ - Oh , do n\\'t just say ... - \\' \\' Okay . \" - Aw , gee , kid . I ai n\\'t much , and ... well , you-you know ... words do n\\'t seem to-- Aw , what are you bawling for ? - Oh , Eddie . Aw , gee , kid . I \\'m sorry . It \\'s all my fault . Me staying out till 4:00 in the morning . What \\'ll I do ? I tell ya . You can say we were gon na get married . - Oh , he \\'d find out we weren\\'t . - How ? When we did n\\'t . Well , we would . - We would ? - Would ya ? You mean you\\'re- you \\'re asking me to marry you ? Yeah . Oh . Oh , sure I will , Eddie . Okay . I \\'ll love to marry you , Eddie . You \\'ll never be sorry or anything , honest you won\\'t . - But I never thought you meant we \\'d really get married . - Neither did I . @ @ @ @ @ @ @ @ @ @ to back out , I won\\'t-- Oh . - So you want to make an argument out of it , huh ? - No . Listen . I never back out ever , see ? And let me tell you something . You ai n\\'t putting \\' nothin \\' over on me neither . - I don\\'t-- Not a thing . If I did n\\'t think ... you were the swellest jane in the world , I never would have given you a tumble . For me , nothing but the best , see ? So if you \\'ve got any idea in your nut that I \\'m the kind of guy ... runs around asking any jane to marry him , you \\'re crazy . Oh , gee , Eddie , you \\'re a scream . Why , you can-you can even make a fight out of a proposal of marriage . But you ai n\\'t kiddin \\' me . You try to make out you \\'re hard-boiled ... but you \\'re not really . Why , you \\'ve got a heart as big as anything . - And @ @ @ @ @ @ @ @ @ @ of a fella a girl can depend on . I was in love with you almost the first time we met . Oh , you \\'re fine and decent and tenderhearted . Oh , Eddie . You \\'re just the sweetest guy in all the world . - Okay . - And we can be happy . Lots of people are married and happy . I \\'m sick and tired of It . ... while you stay out till 4 . ? 00 In the morning-I said enough Is enough ! Get out , you pig ! That \\'ll never happen to us . - I should say not . - Well ? - Do you want me to go up with you ? - No . I \\'m not afraid now . I \\'m not afraid of anything now . You know ... it \\'s sort of like not being alone anymore . Yeah . It is like that , ai n\\'t it ? When \\'ll we be married ? In the morning . And I \\'ll take the day off . Say ... I \\'m kind of happy @ @ @ @ @ @ @ @ @ @ that ? Oh , Eddie . Good night , kid . Good night , darling . - Eddie . - What ? Husband . Okay . Hey , cab . Hello , Edna . What are you doing here ? Jim come over to my house about an hour ago to see if you were there . He was worried about you , so I came back here with him . I , um - I could n\\'t remember the name of that hospital ... you said you were going to to visit that girl . She must be worse , huh ? Your staying there until almost 4:00 . You mean well , Edna , but that \\'s a lot of hooey . - Where were ya ? - Where do you think , Edna ? - I \\'m gon na get married . - No . So , it \\'s as bad as that , huh ? What do you mean it \\'s as bad as that ? I thought girls brought their fellas around and showed them off first to their families . They do in families @ @ @ @ @ @ @ @ @ @ Well , not that you \\'d have brains enough to know it ... but I \\'ve been a pretty good brother to you , see ? I \\'ve been a father and a mother to ya . I \\'m supporting this house . And you \\'ve got some nerve comin \\' in at 4:00 in the morning ... tellin \\' me that you \\'re gon na get married . Why do you object ? You do n\\'t even know the fella . No , but I \\'m going to . Bring him around . Let him meet your family , like any decent fella would want to do . And after a while , if I think he \\'s all right ... you can marry him . - After a while ? - Oh , that gets a rise out of you , does it ? I thought it would . One of them rush marriages . - What do you mean ? - You know what I mean . - You do n\\'t know that I did anything wrong . - I can put two and two together . @ @ @ @ @ @ @ @ @ @ You can prove that you \\'re not . You tell me that you \\'ll wait , or I \\'ll know that you got ta marry this guy . And if that \\'s the case , I do n\\'t want you in the house . - Now , what do you think about that ? - Where \\'ll I go ? Go back where you come from , you little tramp . And get out now . You wo n\\'t take any clothes with you either . I paid for every rag you got on your back . Go on . Get out . You deserve everything you \\'re gettin \\' . Sure does . A girl brought up under the influence of a fine brother like you got . You \\'ll probably end up in the streets . That \\'s what you got in your mind , ai n\\'t it , Jim ? And I wo n\\'t be sorry for her either . No , I do n\\'t expect you to be sorry for her , Jim . But let her have her clothes . - I paid for \\' @ @ @ @ @ @ @ @ @ @ . Let her find out there ai n\\'t another man who \\'ll treat her as white as her own brother did . Go on . Let her have her clothes . All right . Get \\' em , and get out . Hey , Dot . Hey . - Where you goin \\' ? - With the kid , you big bully . You called her a tramp , did n\\'t ya ? Just because she stayed out until 4:00 in the morning and you suspect where she was . Well , I stayed out until 4:00 in the morning , and you knew where I was . So I got a pretty good idea what you think of me . I \\'ll see you in the cemetery . Go on , Dot . Wake up , Dotty . Wake up . Hello , Floyd . Ma just hollered across the air shaft ... and said that she was makin \\' breakfast for a new bride . Are you a new bride ? Yes , darling . I \\'m going to be today . - What \\'s an old one @ @ @ @ @ @ @ @ @ @ on . Run along and play now . Aw , I wan na see Dotty get out of bed . You get out of here . I wan na see Dotty get out of bed . And at his age . - What time is it ? - Almost 11:00 . - I phoned the store and said we were both sick . - Has Eddie come yet ? - Nope . - He has n\\'t ? - Not yet . - I wonder why he does n\\'t come . - Maybe it \\'s \\' cause he does n\\'t know I \\'m here . - That \\'d be a good reason . He certainly would n\\'t come to see me . Oh , he does n\\'t really dislike you , Edna . No , not dislike . He hates me . And it is n\\'t just an average , everyday , amateur kind of hatred either . I \\'ll bet that guy spends hours practicing hating me . Aw , that \\'s just his way . You know , he \\'s kind of funny . That would be great . @ @ @ @ @ @ @ @ @ @ , gee , Edna , it was swell the way you came through for me last night . - I \\'m awful grateful to ya . - And I \\'m grateful toJim . - He saved my life . - How ? Just suppose he did n\\'t turn out to be a heel last night . I \\'d have been married to him in three months , and after being nicely cemented ... with a wedding ring , the church service and everything ... I \\'d have found out what he really is . Yes , sir . He saved my life- \\' cause they send you to the chair these days for killing your husband . In the name of goodness , what are you doing ? - I \\'m trying something , but it wo n\\'t work . - What wo n\\'t ? I put some black ink in the white milk ... and the white milk turned black . But when I put the white milk in the black ink , the black wo n\\'t turn white . Put that ink away and clean up that mess @ @ @ @ @ @ @ @ @ @ what time Is It now ? It \\'s a little after 11:00 . - Where \\'d you put my stockings ? - Why , I put \\' em , uh-Is this Mrs . Cook \\'s boardinghouse ? Well , is Mr . Collins there ? Collins . Eddie Collins . He what ? He moved away ? Well-Well , he lived there last night . He moved away this morning ? Well-Well , did he say where he was moving to ? What \\'s the matter ? - He \\'s walked out on me . - Do n\\'t be a goof . He has . Oh . I wish I was dead . We all get that wish someday . Call up his store . Do you know the number ? Schuyler 4-6583 . Schuyler 4-6583 . And hurry it up , deary . There \\'s a fire . Here . Hello ? Hello , Lathrop Radio Shop ? Well , can I speak to Mr . Collins , please ? Eddie Collins . Never mind the wisecracks . He \\'s there , I wan na speak to him . @ @ @ @ @ @ @ @ @ @ to . Fresh guy trying to flirt with me . He what ? He was fired ? What for ? He said he was fired for robbin \\' the cash register . Well , you \\'re all right then . If he \\'s lost his job , it \\'s a cinch he \\'ll be here to marry ya . I ca n\\'t tell whether the fella was kiddin \\' or not . There he is now . See ? All of your troubles for nothing . Oh , gee , and I \\'m not dressed or anything . Oh , give me those stockings . I do n\\'t wan na subscribe to any magazines . I do n\\'t care if you \\'re working your way through the penitentiary . - Was that him , Edna ? - No . A book agent . Oh . What am I gon na do ? Now , stop worrying . If Eddie is only half the guy I think he is , he wo n\\'t let you down . That \\'s the trouble with us women . If we \\'d only realize that-Something @ @ @ @ @ @ @ @ @ @ ? - I \\'m cookin \\' it for ya . - A whole pound of butter . I told you a piece of butter , not a pound of butter . A pound \\'s a piece . I \\'ll go , Edna . Oh , darling , you \\'re driving me crazy . Get out into the hall . You \\'ve done all the damage in here that you can . See if you can ruin the wallpaper . - Shall I do it with a knife ? - Get out of here ! - Was n\\'t him , huh ? - No . A man selling-For goodness \\' sake , take that long face off . This is your wedding day , not your funeral . He \\'ll be here . There \\'s a mistake somewhere . That \\'s all . Yeah , last night- that was the mistake . Do n\\'t be silly . He \\'ll be here . Oh , but I love him so much . Well , you can still love him and have a cup of coffee in your stomach too . I do @ @ @ @ @ @ @ @ @ @ anything again . - Fine . Then have a glass of milk . - I do n\\'t want any milk . Oh , come on . Have a glass of milk . It \\'ll do you good . You know , the first thing that you have to learn is-I \\'m getting hysterical . Open the door , Floyd . And if it \\'s a man selling coffins , tell him we \\'ll take two . Who \\'s dead ? Oh , Eddie ! - What \\'s the matter ? - Nothing . Say , listen . You \\'re going to be a married man . The first thing you have to learn is to get accustomed to your wife \\'s tears . I was over to your house , and they said you was n\\'t there . So I took a chance and come here . I called up your house , and your landlady said that you \\'d moved . I got a bigger room across the street . My landlady was sore . That \\'s why she would n\\'t tell you where I went . And they @ @ @ @ @ @ @ @ @ @ stealing . They were kidding \\' ya . Oh , Eddie , I \\'m so happy . - Okay . - Now maybe you can eat some breakfast . We \\'ll eat downtown . We got ta get goin \\' . Go get your things . Her master \\'s voice . - Do you know how to get there ? - I \\'ll find out . - I \\'ll tell ya . You take the subway to City Hall . - Yeah ? When you get out of the train , you walk straight ahead and turn to the right . There are two exits there . You take the one to the left . Understand ? - Yeah . - Then when you get up on the street ... - you walk straight ahead-- I \\'m gon na take a cab . I \\'ll tell you which cab to take . Hey . Whose wedding is this , yours or mine ? Oh , Edna \\'s just trying to help us , Eddie . You was n\\'t on the other side when they had the war , were ya @ @ @ @ @ @ @ @ @ @ I was wondering how they ever got along without you . I phoned my orders over . Wo n\\'t you come down and see us get married ? No . I hate those things- deaths and funerals and things like that . Oh , gee , Edna . I can never thank you enough for all you \\'ve done for me . Oh , that \\'s all right , kid . I only hope you \\'re gon na be awful happy . If she ai n\\'t , you \\'ll hear from me , you big stiff . Okay . - Good luck . - So long . Good-bye . Oh , darling ! Night , now . - Hello , darling . - Hello . - What \\'s this ? - Oh . Just a couple of hunks of candy . Oh , gee , Eddie . - You know why ? - Sure . Because you love me , do n\\'t ya ? Ah-- Oh , say it . - What ? Say \\' \\' I love you , darling . \" Right out in all the light ? @ @ @ @ @ @ @ @ @ @ . Ten weeks ago today we got hitched . Aw , gee , it was sweet of you to bring me the candy . And I got news for ya . I found a store . The boss is gon na lend me a hundred bucks ... and with the 650 I got saved up , I can swing the deal . He \\'ll let me have all the radios I want on time . - What do you know about that , huh ? - That \\'ll be swell . Of course , we may have to pinch nickels for a while but ... in the end , it \\'ll all be jake . - Gee whiz , I \\'m a happy guy . - Are you , Eddie ? Yeah . I been thinkin \\' about having my own store for over two years now . And while I ai n\\'t much on tossing \\' this baloney ... and all that mush stuff ... a guy could have a worse wife than you . - Ya little squirt . - Oh , Eddie . - I love you @ @ @ @ @ @ @ @ @ @ your hat and coat on , and I \\'ll take you over and show you the place . - Oh , I ca n\\'t now . - Why not ? I \\'m waitin \\' for Edna . She phoned she \\'ll be up in a little while . - Ah , what does she want ? - Well , I wanted to ask her advice about something . - About what ? - About-Well , about the store , see what she thinks of it . Oh , gee . It \\'ll be tough if she do n\\'t like it . Oh , Eddie , that \\'s not a nice thing to say . Here she is . Now , please do n\\'t be mean to her . Come in . - Hello , Dot . - HI , Edna . How \\'s the grouch ? I thought they sent you down to Washington ... so you could tell the president how to run the country . Ooh . They sent the president up to see me . Oh , do n\\'t mind him , Edna . - He \\'s @ @ @ @ @ @ @ @ @ @ looks it . - He \\'s gon na open his own store soon . - Hmm ! I heard Wanamaker \\'s were going out of business . They probably figured the competition would be too much for \\' em . I \\'m going down to the corner and get a paper . Come on . Sit down . I \\'ve got so much to tell you . - Still in love with him ? - Oh , I \\'m just crazy about him . Edna , I \\'m going to have a baby . Oh , Dot . - I \\'m afraid . - Oh , do n\\'t be silly . - My mother had eight of_\\'em . - Yeah . My mother died having me . Yes , but things are different now . You know , the hospitals are better , and the doctors and things . No girl minds it anymore . - What does Eddie say ? - He does n\\'t know . Well , why do n\\'t you tell him ? Well , his heart \\'s so set on having that store . It \\'ll @ @ @ @ @ @ @ @ @ @ do n\\'t know what to do . Edna , I \\'m so worried . You know , if the store \\'s gon na make Eddie happy , I want him to have it . - Sure . - When I think of what happened to my mother-Unless I had everything of the best- you know , the best doctors and everything-- I \\'d be frightened . - You poor kid . Now listen . This fella you \\'re married to is all right . He \\'s 100 ? /o . I know what \\'s the matter with him . He \\'s one of those sensitive eggs . Of course , he may not like me , but what the heck . Napol ? on had a couple of enemies too . Now you tell Eddie tonight . And if I know anything about men ... he \\'ll be stopping strangers on the street tomorrow , bragging about it . - Do you really think so ? - Oh , sure . All men are alike . They puff themselves up . You \\'d think they \\'d done something . - @ @ @ @ @ @ @ @ @ @ . - He will be . I do n\\'t know though . He \\'s always yelling about living in a tenement ... and what a sin it is to have children if you ca n\\'t afford to give \\' em everything . I do n\\'t think I read anything in the papers about him being born in the White House . He was awful poor . That \\'s why he thinks like that about it . Here he is . Do n\\'t say anything . - You still here ? - No . Dld you get your paper , Eddle ? I got down there , and I did n\\'t have any change . You can get one next week . They \\'ll have a dozen new murders for you by then . Here , let me fix that tie . - It looks like a unbent pretzel . - The tie is-You only wear it . Other people have to look at it . I do n\\'t know how you ever picked this jane out for a friend . - She \\'s the dizziest dame I ever seen . @ @ @ @ @ @ @ @ @ @ , Eddie . Look what I got ! Look what I got ! - Where \\'d you get all this stuff ? - Eddie give us 60 cents to buy it . Aw , he \\'s crazy . It was - Sixty-Oh , what did you wan na come up here for anyway ? Go on outside and play . Go on . Go on . Go on now . Get out of here . Thanks a lot , Eddie ! Gim me a slice of that cone down there . Poor kids . I found \\' em downstairs in that dark hallway tryin \\' to play games . They ai n\\'t got a chance in life . Eddie , did you give him the 60 cents ? Listen , I got better things to do with my money ... than throw it away on a lot of kids . Say , you \\'re a swell egg . What are you gon na sell in your radio store ? Never mind . The way you run people \\'s business for them ... if you start butting \\' in , I @ @ @ @ @ @ @ @ @ @ Well , I \\'m going . - Remember now . Tell him . - Tell him what ? Oh , nothing , Eddie . She \\'s got a secret for ya . Oh , she \\'s got a secret for me , but you know it . Geez , you \\'re runnin \\' my house for me , too , I see . - Sure . Bye , darlin \\' . - See you tomorrow . - Bye , grouch . - So long , dizzy . - What \\'s the big secret ? - I do n\\'t know whether you \\'ll like it or not . Hey , listen , kid . I \\'m so happy about the store , I \\'d like anything . Oh . Well ? What is it ? Eddie , I \\'m going back to work . - Back to work ? - Yeah . Hey , listen . No wife of mine is goin \\' to work . - What \\'d be the harm ? - I \\'ll do all the work that \\'s to be done in this family . - What @ @ @ @ @ @ @ @ @ @ Well-Well , it gets kind of lonesome staying \\' here all day all alone . Ca n\\'t you go to picture shows and things ? You just said we \\'d have to pinch nickels . Yeah , but I did n\\'t mean we \\'d have to pinch \\' em that hard . - You do n\\'t like livin \\' in just one room . Is that it ? - Oh , it \\'s all right . But I wish we had a flat of our own or something-you know , our own furniture . Ca n\\'t you see , Eddie ? If I went to work , you could have your store just the same . And with the money I make , we could-Do you think I \\'d take money you worked for ? - Oh , it \\'d be all right . - Not with me , it would n\\'t . My idea of a husband is a guy that looks after his wife and takes care of her ... and sees that she do n\\'t want for nothin \\' . If I ca n\\'t do that , @ @ @ @ @ @ @ @ @ @ money- you work all day- or let you use it for something that I ought to be payin \\' for . Listen . They got a name for guys like that . Oh , Eddie , I did n\\'t mean to make you sore . I ai n\\'t sore . But I ai n\\'t as happy as I was . You can lay that on a line . It was our 10 weeks anniversary . And I had the store all picked out and everything . And , say , when I come in here tonight , I was walkin \\' on air . This kinda crumbs it all up . Gee , I did n\\'t know you was n\\'t satisfied . Oh , Eddie , I am satisfied . It \\'s all right . Let \\'s not talk about it . - I \\'ll bet that Edna jane put that idea in your nut . - No , she did n\\'t . Let \\'s not say any more about it . You \\'re crazy to have the store , and it \\'ll make you happy . Well , that @ @ @ @ @ @ @ @ @ @ . I \\'m the husband , and it \\'s my job to make you happy . If a husband ai n\\'t for that , he ai n\\'t for nothin \\' . Gee , I told the real estate guy I \\'d meet him tonight . I ca n\\'t make up my mind about that sign . \\' \\' Eddie Collins \" or \\' \\' Edward Collins \" ? Course , \\' \\' Eddie Collins \" is all right for a saloon or somethin \\' like that . Nope , it ai n\\'t got class enough for a radio store . I \\'ll be back in 15 or 20 minutes . And I still think that Edna jane put that idea in your nut . I was afraid you was n\\'t comin \\' . What \\'s the idea of phoning \\' me to meet you here ? - Dot and me are movin \\' here . - You \\'re movin \\' here ? Sure . I got new furniture and everything . That \\'s why I asked you to come . I thought maybe you \\'d help me put \\' em @ @ @ @ @ @ @ @ @ @ in anyway , so I figured out you might as well do it in the beginning . You \\'re moving here ? Yeah . Dot told me that secret . What \\'s that got to do with your moving and spending your money ? She wanted to go to work . - You mean that \\'s the secret she told you ? - Yeah . So I put two and two together and figured out why . She wanted a home with nice furnishings and things . - Say , how much did all this cost you ? - I shot the bankroll . - Oh , no . - Yeah . - All you \\'d saved ? - Sure . This is a swell joint . The furniture man is takin \\' charge . I bought everything new- from soup to nuts . When it \\'s finished , it \\'ll look just like a home that somebody \\'s lived in for a long time . Some surprise for the kid , huh ? Yeah , but what about your store ? Well , I figured , why let her @ @ @ @ @ @ @ @ @ @ build up a business ? I might \\'ve made a flop of the store anyhow . Not that I would \\'ve . I \\'d have made a success of it . Do n\\'t get that idea in your nut . And you spent all your savings ? Sure . She \\'s worth it . She \\'s a great kid . You \\'ll have me crying in a minute , Eddie . What for ? You would n\\'t know . Come on . You never saw anything sweller than that parlor set I bought . Come on . Wait till you get a load of the bedroom . It \\'s got sheets and pillowcases and beds and everything in it . And the kitchen- We got a lot of pots and pans . It \\'s gon na be like a party . See , I \\'m gon na tell Dot that Pat and Sue have got a new flat ... and are havin \\' a housewarming and we \\'re invited . I asked a lot of the gang , and they \\'re in on it too . It \\'s gon na @ @ @ @ @ @ @ @ @ @ parties . And the big payoff comes when I say ... \\' \\' Well , kid , how do you like your new home ? \" What do you think of the idea ? - I do n\\'t know . - You did n\\'t think it up . That \\'s why . Hey , wait a minute . Do n\\'t put that great big chair over there by the divan . - That \\'s where the chair belongs . - No , it is n\\'t . It belongs over on this side . Have n\\'t you got any sense of balance ? - You would get your mouth in it some way or another . dd Hey , hey . Wait until they get here . Aw , have a heart . I have n\\'t had any supper . And do n\\'t drink up all the punch . Here they are . Listen , Pat . Remember , this is supposed to be your home . You \\'re the host . - Right . I \\'ll take off my coat and collar . - That ai n\\'t funny . - @ @ @ @ @ @ @ @ @ @ door . Go on . I did n\\'t even know that Pat had moved from 92nd Street . Sure . They got a swell place here . - Hello , Dot . How are you ? - Hello , Sue . - Hello , Eddie . - I did n\\'t know that you \\'d moved . Oh , I \\'ve been here a week . Come in . Hi , everybody . - Hello , Chad . How are you ? - Hello , Dot . - Hello , Pat . How are you ? - Let me take your coat . - Have a schnapps . - Oh , no , thanks . Oh , gee , Sue , this is swell . Pat must have murdered a millionaire or something . Hey , you ai n\\'t seen nothin \\' yet . Come on out here and I \\'ll show you the kitchen . You \\'d think this was his house . Come on . Let \\'s eat . A closet , and it \\'s got glass in it too . And a sink and a washtub . And @ @ @ @ @ @ @ @ @ @ where you keep your salt and pepper and spices and things . In the one marked \\' \\' Salt , \" you keep the salt . In the one marked \\' \\' Pepper , \" you keep the pepper . Gee , Eddie , ai n\\'t it swell ? Look at that darling table . The guy said it was the breakfast nook . Oh , Eddie , you think we \\'ll ever have a place like this ? - Would you like to live here , kid ? - Oh , gee , would I ? - Of course , we could n\\'t afford-- Why not ? Well , we could n\\'t afford the money now . Well , why now ? - Well , you wan na get your store first . - Yeah . Oh , yeah . That \\'s right . Say , wait till you get a load of the bedroom . Come on ! Say , if Eddie can get Dot a home like this , why ca n\\'t you get me one ? I thought that \\'d crop up . - Look at @ @ @ @ @ @ @ @ @ @ , kid . There \\'s a bed . You ca n\\'t buy a better bed than that- 65 bucks . - But how did you-- Oh , Pat told me . Come on . Feel it . Just feel it . Ai n\\'t this a bed ? Not like that old egg crate we sleep in . I \\'ve never seen you so enthusiastic . The house has sure made a hit with you . Come on . Now I \\'m gon na show you somethin \\' . Just grand . There \\'s the bridges over there . And there \\'s all New York to pick from . Ai n\\'t the lights great ? And the East River . And stars . Are n\\'t they beautiful ? Not phony one\\'s either . Gee , a roof like this would be a swell place for ... kids to play . Would n\\'t it , Eddie ? You know , they could get plenty of fresh air ... and their mothers would n\\'t have to worry about streetcars ... and taxis and trucks and things . You know , rich kids could @ @ @ @ @ @ @ @ @ @ it , do n\\'t you , kid ? Oh , I think it \\'s just grand . I \\'m awful glad for Pat and Sue . Okay . We better get back to the bunch now . You said it . Sue , it \\'s the grandest flat I ever saw . - I \\'m awfully glad you like it , Dot . - Oh , it \\'s sweet for anything . Dot , sit down . Here goes . Well , kid , it \\'s yours . - Mine ? What do you mean , mine ? - Sure . Yours . This is where you \\'re gon na live . I bought it for you . - You bought it for me ? - Sure . You know that secret about goin \\' back to work ? Well , I doped it out that you was n\\'t satisfied where you \\'re livin \\' . So this is the answer . - Well , what about your store ? - Thls Is It . You spent all your money for this ? I shot the works . - You @ @ @ @ @ @ @ @ @ @ n\\'t owe a nickel on It . Well , what do ya say ? I do n\\'t know . Do n\\'t seem to make such a hit with ya now . Naturally , Eddie . She \\'s surprised . You ca n\\'t get a thing like this through your head in a minute . - Did you know he was doing it , Edna ? - Yeah , I helped him fi ? x It up . Why \\'d you let him spend all the money ? Well , folks , it looks like a bust . Oh , no , it is n\\'t , Eddie . I think it \\'s just great . I \\'m tickled to death . - You \\'re the swellest husband any girl ever had . - Okay . Dot \\'s only disappointed for you , Eddie . She was so anxious for you to have the store ... she did n\\'t want you to spend the money for anything else . - That \\'s right . - Sure . That \\'s why , dear . You know I think it \\'s just swell . @ @ @ @ @ @ @ @ @ @ , a long time dead . Live while ya can . That \\'s what I say . When I think of the way my folks used to live . They did n\\'t have enough money to support themselves ... so they go and have nine kids . My old lady used to scrub floors in an office building . 4:00 every morning she went to work- sick or well , summer or winter . I can see her goin\\'out when It was 20 below zero ... with a cough that \\'d tear the heart out of ya . And for what ? To bring up a couple ofklds and die- dle before they was ever any comfort to her . Not for Dot . Every nickel I make , she and I are gon na spend on ourselves . - My sister died bringing up-- Oh , stop it ! Ca n\\'t you stop it ! You \\'d think there was nothin \\' else in the world to talk about . All right . There , there , dear . Shh . What \\'s wrong ? Dot \\'s going to @ @ @ @ @ @ @ @ @ @ , Eddle . No , she \\'s just a little hysterical . That \\'s all . We should n\\'t have talked about those things . - That \\'s right . I guess we shouldn\\'t . - Sue told me , but I forgot . Gee , everybody in New York seems to know about it ... but the only guy that \\'s interested . Oh , gee , Eddie , I wish you \\'d let me do that . Do n\\'t be crazy , will ya ? You \\'re goin \\' to the hospital in a few weeks . You know the doc told you not to do any hard work . I do n\\'t think he knows what he \\'s talkin \\' about . I do n\\'t like him . - You want me to get you a different one ? - No . If you ca n\\'t get the best , he \\'s as good as any of the others , I guess . He \\'s all right . I worry . That \\'s all . - About what ? - About everything . You having to do @ @ @ @ @ @ @ @ @ @ and pinching \\' every nickel the way you have . I do n\\'t want a baby any more than you do . All right , all right . Now you told me that a thousand times . Let \\'s not talk about it . Is there any other doctor you got in mind ? There was somethin \\' in the paper about a society woman . The doctor \\'s name was Burgess . His picture was there . Gee , he was awful kind-lookin \\' . Not that we could get him . He \\'s only for millionaires . As long as you ca n\\'t get him , you might as well pick out the best . Whoo-ooh ! Dot ! Now do n\\'t bring her in here . I do n\\'t want her razzin \\' me . - Good morning . - Hello , Edna . - Hello , Eddie . - So long . Floyd made such a racket , I did n\\'t wait for breakfast . - Got a cup of coffee ? - Sure . Hmm . You got a new washerwoman , I see . @ @ @ @ @ @ @ @ @ @ n\\'t got enough bluing in this water . I knew somethin \\' would be wrong with it when you seen it . - Here . - Thanks . Got a cigarette ? No , I cut \\' em out . They was puttin \\' my throat on the bum . Come on , Edna . Drink this in the other room . Yeah , go on . Beat it , will ya ? You give me the willies . That \\'s good . - How do you feel ? - Oh , all right . - How do you feel when you feel bad ? - I worry . Yeah , and you worry other people too . You worry him . Your child \\'ll probably grow up to be a radio broadcaster . Gee , It \\'s 10 minutes to 8 . ? 00 . I got ta blow . Do n\\'t touch those things . I \\'ll finish \\' em when I come home . - Good-bye , darling . - So long . - So long , meddlesome . - Good-bye , grouch . Break a leg @ @ @ @ @ @ @ @ @ @ me so miserable . - What ? - He \\'s so square about everything . Says smoking hurts his throat . He cut it out so he could save the money . It does n\\'t seem fair that he should give up all his fun just because of me . Yeah , it \\'s tough on husbands . Nobody knows what a husband suffers when his wife has a baby ... and a big juicy raspberry . It must be great for a woman when everything is right- you know , when she can afford a doctor like that man Burgess ... and when she knows her husband wants the baby . You want it , do n\\'t you , Dot ? I \\'m crazy about it . That is , I would be ... if it were n\\'t for Eddie and if I was n\\'t afraid . - How do you know he does n\\'t want it ? - The things he says . I can feel it , too , in the way he acts . He does n\\'t like children . Coochie-coo . Coochie-coo . Coochie , @ @ @ @ @ @ @ @ @ @ it ? - Only seven months . Of course , all parents are proud of their children . But honestly , even if this was somebody else \\'s kid ... I \\'d have to admit it \\'s the cleverest youngster I ever saw . - Do n\\'t you think so , Mama ? - I honestly do . Hey , lookit ! The little son of a gun \\'s got ahold of my finger . That \\'s what I mean . Is n\\'t that clever ? - She \\'s a darling . Look at her . - Coochie-coo . Coochie-coo . Coochie , coochie-coo . Coochie-coo . Coochie-coo . Hey , Eddie . Eddie . Eddie . - Has Mr . Randall \\'s set been fixed for him ? - Yes , sir . - Gee , that \\'s a great kid you got . - We think so . - Say , Mr . Lathrop ? - Yes ? Do you think that a woman havin \\' a baby ... you know , that she has n\\'t always wanted ... could-could learn to love it ... you know , @ @ @ @ @ @ @ @ @ @ Your wife , eh ? She \\'s afraid . Say , there ai n\\'t any danger in these things , is there ? Well , lots of women manage to have babies ... without calling out the militia . There \\'s no danger when they have the proper attention . She should be made to have confidence in her doctor . That \\'s just it . My wife ai n\\'t got any . That \\'s not right . - She speaks of a Dr . Burgess . - Burgess ? You \\'d have to be a millionaire to have him . - Goin \\' to lunch , Eddie ? - Nah , I cut lunches out . I was gettin \\' too fat . - This Dr . Burgess is good , huh ? - And expensive . When a king is born , they send for him . I wonder how you could get in to see a guy like that . - Mr . Lathrop , will you take a look at this set , please ? - Oh , yes . Certainly . Hey , Joe . @ @ @ @ @ @ @ @ @ @ Club ? - Yeah . Do you suppose he could get me on in one of them preliminary fights ? Say , some of those boys are pretty good . - What do you wan na take a beatin \\' for ? - Ten bucks a round . Four rounds- 40 bucks . I can use that dough swell . - I can fix it for ya , but I think you \\'re nutty . - Okay . 7:00 . Say , ai n\\'t that job of yours got any hours ? A guy wanted his radio fixed , so I did it on the side . I should think you \\'d try gettin \\' home ... with your wife goin \\' to the hospital and everything . - How is she ? - She \\'s worrying herself sick . That \\'s how she is . But , gee , she ought n\\'t to do that . I do n\\'t think she knows that . You ought to tell her . - You had any supper ? - I ai n\\'t hungry . Gettin \\' your beauty sleep , kid @ @ @ @ @ @ @ @ @ @ would hurt you to hang around once in a while . Okay . That \\' \\' okay \" does me a lot of good , do n\\'t it ? Gee , kid , you know I \\'m for you , do n\\'t ya ? Well , it do n\\'t look like it . You have n\\'t spent an evening in the house for two weeks now . I do n\\'t mean to be a grouch , Eddie , but ... I \\'m frightened . Is it the doctor ? Is that what makes you afraid ? Oh , he \\'s all right , I guess . It \\'s my fault . You know , you have confidence in some people , and others you have n\\'t . There \\'s no reason . You just have . That \\'s all . Sure , I know how it is , kid . Eddie , you \\'re gon na stick around and go to the hospital with me tonight , ai n\\'t ya ? You betcha . No . Wait a minute . - I tell ya , I got ta go out-- @ @ @ @ @ @ @ @ @ @ - Oh , no . Just for a half an hour , kid . - Ca n\\'t even stay with me tonight . I \\'ll be right back . Honest I will . Mmm . Your honey is searching among the \\' \\' bees . \" Hello ? Yeah , this is his secretary . Mm-hmm . Who wants to talk to him ? Joe ? - There \\'s a Joe wants to talk to you . - Hello , Joe . Tonight ? Sure I \\'ll be there . Hey , that \\'s great . It comes just at the right time . Thanks . So long . I \\'ll be a little longer now , but I \\'ll be back at 9:00 . I got another fella to see now . - You pick a swell night to make a date . - You sure do . This ai n\\'t a date . Honest it ain\\'t . - I \\'ll be back as quick as I can . - Give my love toJenny , will ya ? - WhatJenny ? - WhateverJenny happens to be there . Okay @ @ @ @ @ @ @ @ @ @ hospital . But I \\'m sure the doctor you have is competent . It ai n\\'t that . You see , she \\'s just a kid herself , and she \\'s frightened . Her mother died that way . She read about you In the paper , Dr . Burgess . Well , of course , we never thought we could get ya . She was sayin\\'that It would be swell ... If she could get a guy like you to be with her . Did you tell her you were coming to see me ? No , sir . She and me ai n\\'t been gettin \\' along very well lately . I \\'ve been workin \\' nights . I did n\\'t tell her about it . She thinks I \\'ve been chiseling \\' on her , I guess . I ca n\\'t tell things like that . I \\'m kind of a mug that way . Of course , I could n\\'t pay you what you \\'re worth . But I tell ya . I make 45 bucks a week , and pick up 15 workin \\' nights @ @ @ @ @ @ @ @ @ @ fight tonight . And I got 381 saved up , see ? - Are you a prizefighter ? - No , sir . But a friend of mine \\'s gettin \\' me on ... in the preliminaries over at the Empire Club . I get 10 bucks a round , and I do n\\'t mind takin \\' a beating for 40 bucks . Llsten , Dr . Burgess-I always been kinda proud , see ? You know , pay my own way , take nothin \\' from nobody . But I ai n\\'t that way now , Doc . I \\'m beggin \\' you . If she knew she was gon na have you with her ... everything would be all right . But as it is ... she \\'s so sick from worry-I \\'m afraid maybe she \\'ll die . Will you do it , Doc ? I \\'ll give you the $381 I got saved up ... and I \\'ll pay you back dollar for dollar . Please do it , will ya , Doc ? So help me , God , I\\'ll- I \\'ll lay down my @ @ @ @ @ @ @ @ @ @ Thanks . - I guess I \\'m kinda nervous about this child thing . - Mm-hmm . Kinda goofy , huh ? Gee , Doc , I \\'m sorry I have n\\'t got a cigar or somethin \\' to offer ya . I cut out smokin \\' . It was hurtin \\' my throat . This is a swell dump you got here . Oh , I did n\\'t mean-Okay . Flght ! - He said 9:00 . - Do n\\'t worry . You \\'d think the least he could do would be to be with me tonight . Now , now , now . Do n\\'t get yourself all worked up . One , two-Get out . Get out . Hey , Mike , knock that fella out , will ya ? This ai n\\'t a slaughterhouse . I wanted to give him a break , but I \\'ll put him away right at the next gong . Aw , why do n\\'t ya quit ? I know what I \\'m doin \\' . I would n\\'t take a beatin \\' like that for a hundred bucks . Yeah @ @ @ @ @ @ @ @ @ @ me . One , two , three , four ... five , six ... seven , eight , nine-Uh ! Do n\\'t knock me out , will ya , buddy ? I need the cash . - My wife \\'s gon na have a kid . - Well , why did n\\'t you tell me ? I guess I ca n\\'t make it though . Hold on to me . Take a deep breath . Break it up . Thanks . Good-bye . I \\'ve arranged for the hospital . The cab will be here at 11:00 . I never thought Eddie would do this to me . But , darling , he \\'s only an hour late . Some hours are lifetimes . He \\'s walked out on me . - When \\'s the kid expected ? - Any-Anytime now . - I got two of them . - Yeah ? How long before they get to know ya ? Mine knew me inside of two months . - But of course , mine was kinda extra bright . - Yeah . Oh , gee , kid , I @ @ @ @ @ @ @ @ @ @ bad about you . - I could n\\'t help it . Could n\\'t spend a couple of minutes with me , tonight of all nights ? You know how nervous I am . But you could go to some speakeasy with a bunch of fellas ... and get your face all battered up in a drunken brawl . Is that what ya think ? That \\'s what it looks like to me . Okay . Do you want me to go with ya ? You do n\\'t have to bother about me anymore at all , Eddie . I got Burgess for ya ! - How about it , Doctor ? - Congratulations . Twins . - Twins ? - Twins . - Boys or girls ? - Both . - Twins ! - Yes , sir . The baby business is picking up . Sometimes I was here , and I was the only one . - This your first ? - Yes , and my last . I would n\\'t go through this again for all the money in the world . How many kids have you got @ @ @ @ @ @ @ @ @ @ n\\'t know . You do n\\'t know ? I know I got six . What \\'s happening upstairs , I ca n\\'t tell ya . Here I am . You \\'re the father of a fine baby girl , Mr . Vernon . - Oh , well . - Oh ! - Put him over here on the-- It was too much for him . - He \\'ll be all right . - What \\'s the matter with him ? - I \\'ve got some smelling salts . - That \\'s fine . Give him some . - He \\'ll be all right . - He \\'s as white as a sheet . He \\'ll be all right in a minute . People kid about fathers , but they do n\\'t realize ... what a guy has to go through when his wife has a baby . If he had as many as I got , he \\'d stay unconscious . He \\'s coming to now . He \\'s fine . - What happened ? - You just had a baby . I would n\\'t go through this again for @ @ @ @ @ @ @ @ @ @ , or he \\'ll faint again . - It \\'s all right , Mr . Vernon . You come with me . - Hold him up . - Gee , I \\'m gettin \\' nervous myself now . Do n\\'t be nervous . Say , Mr . Linsky , I wan na ask you a question . My wife was n\\'t any too anxious to- to have this kid . Say , in two weeks , she will be crazy about it . - Is that straight ? - They always are . Be careful . Do n\\'t let her think that you are more concerned for the baby ... than what you are for her . Women get jealous under those conditions . You know , be interested , but not anxious . After all , you \\'ve known your wife for years ... and the baby is practically a stranger . Yeah , that \\'s right . Excuse me . I got ta phone my partner . Mr . Collins ? - Yeah ? - A boy . Okay . Your wife will see you in just a @ @ @ @ @ @ @ @ @ @ you know about that ? My kid \\'s 22 hours old already . They get old very quickly when they \\'re young . - Hello , Doctor . How are they ? - Fine . Say , Doc , I \\'ve got 350 bucks here . - And I \\'ll give ya-- Yeah , well , I have your bill here . Now , I have another patient . I must hurry . - Good luck to you , my lad . - Thanks . Gee , he \\'ll never let me speak to him about money . It says paid in full . And here \\'s a check . \\' \\' Pay to the order of Mr . Edward Collins \\'s son- $50 . \" What does that mean ? He \\'s giving your boy $50 to start a bank account . And he would n\\'t let me give him anything . Gee . That \\'s a guy for ya , ai n\\'t it ? Oh , they \\'re just grand , Mrs . Linsky . Every time I have a baby , my husband gives me a new @ @ @ @ @ @ @ @ @ @ n\\'t have any more room . - What do you think of it ? - Oh . Oh , no , this is n\\'t yours . This baby \\'s six days old . She acts like she hates the child . I guess maybe she did n\\'t want it . Do parents hate children they do n\\'t want ? I heard of people like that . How do you like this one ? - Oh , it \\'s darling . - Oh , no . This is young Mussolini . dd Gee , it \\'s funny when you think of it , ai n\\'t it ? Everybody in the world was once just a little bundle like that . - Is n\\'t this one cute ? - Oh , it \\'s lovely . Just look at those little hands . Did you ever see such an adorable nose ? Gee , it \\'s swell . Do n\\'t you want to hold your baby ? Is this mine ? - Honest ? Oh . - Thls Is It . I would n\\'t cry over It , Mrs . Colllns . It @ @ @ @ @ @ @ @ @ @ . No ? Well , it \\'s the first time this baby was ever born . You can come in now . Hello , kid . - How do you feel ? - All right . Thanks for sending me the flowers . Okay . - Is that it ? - Yes , that \\'s it , and it \\'s no \\' \\' it . \" Well , I did n\\'t mean it that way . Let \\'s not talk about it . - That suits me too , kid . - I thought it would . We \\'ll take him back to the nursery now . I asked Dr . Burgess how we happened to get a fine doctor like him . He just laughed and said that you \\'d tell me . Oh , my boss knew him . He did it for him . - He give me a check for the baby \\'s bank too . - He did ? - Was n\\'t that swell of him ? - Yeah . Those big guys do a lot of work for charity . So you think of @ @ @ @ @ @ @ @ @ @ I-Well - Gee , everything I say seems to be wrong . Oh , Eddie . I \\'d give anything if- well , if things were different . Aw , everything \\'s gon na be all right , kid . We ai n\\'t gon na stop living just because we got a baby . You and me is gon na have a good time , the same as always . Why , In six months we \\'ll have forgotten we got a kid . Please go now . I \\'m tired . Sure . - When \\'ll you be home ? - In about two weeks . Well , hurry it up . I \\'ll feel like stepping \\' out a bit . - So long , kid . - So long . I \\'ll be back every day . Oh , Mrs . Linsky , she \\'s getting bigger every day . What are you going to call her ? I \\'m calling her Shelly , after my mother-in-law . Her name was Esther , but it \\'s close enough . Well , how are all the mothers today @ @ @ @ @ @ @ @ @ @ Is the cab downstairs ? - Yes . Are you ready ? Mm-hmm . - Are you leaving now ? - Yes . But I heard you tell your husband that you were going tomorrow . - I want to surprise him . - Oh . Say , listen , kid . - Have you thought this thing over ? - Yep . I could n\\'t live with a man that did n\\'t like his own child . All he thinks about is stepping out . I could n\\'t go back to Eddie for anything . I \\'d get to hate him . Well , it \\'s your funeral , kid . Well , what are you going to do with that ? Well , I thought maybe the kid would like to fool around with it . What ? A two-weeks-old baby ? Well , it \\'s only a little balloon . The doctor \\'s in the ward . You \\'ll have to wait a few minutes . - Dr . Gibson ? - Yes ? Just a minute , please . - Hurry , Nurse Parson . - @ @ @ @ @ @ @ @ @ @ . My baby . Something \\'s happened to my baby ! Somethlng \\'s happening to my baby . No , Mrs . Collins . It \\'s all right . - I want my baby ! I want my baby ! - He \\'s all right . - I want my baby ! - Please listen . I want my baby ! I want my baby ! I want my baby ! - I want my baby ! - What \\'s the matter , kid ? - Something \\'s happened to my baby . - Where ? In there ! It \\'s not your baby . There \\'s no danger . Your husband will get it . Now , everything is all right . - It \\'s all right . - He \\'s all right , kid . Look . Here he is . - He \\'s all right . - It was Mrs . Burns \\'s baby . Nothing serious . Just a little coughing spell . Oh , gee , kid . I did n\\'t know you liked it . And I thought all along that you did n\\'t @ @ @ @ @ @ @ @ @ @ in your nut . The things you said that night in the flat- you know , about your mother and everything . Well , I thought like that then . But gee , it \\'s different when you got your own kid . Hey , lookit . The little son of a gun \\'s got ahold of my finger . - Dr . Burgess said he had a finely shaped head . - He did ? Hey , look out there , buddy . You might have the future president of the United States in this cab . That \\'s what they all say . - Oh , yeah ? - Oh , Eddie . Now do n\\'t , Eddie . Hey . Let me hold him , will ya ? - You really want to , Eddie ? - Sure . Be careful . Put your arm under his head . - Okay . - There you go , baby . Say , Dr . Burgess was right about the kid \\'s head . That \\'s the best-shaped kid \\'s head I ever saw . He has a lovely @ @ @ @ @ @ @ @ @ @ give me the baby . He \\'s dribbling all over you . Dribbling ? That ai n\\'t the half of it . I \\'m wet clean through . But it \\'s okay . dddd dddd dddd dddd \\r\\n'"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "movie_raw['11.txt'][2]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It seems messy, but nothing we can't clean. This basic method replaces some of the issues with the formatting, and prints the errors if any for debugging. Let us clean one of the raw text files. \n",
    "\n",
    "Note: we skip any text data which isn't utf-8 encoded here. I do this to keep things clean; you might want more data or anticipate special characters and not include that restriction."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "def clean_raw_text(raw_texts):\n",
    "    clean_texts = []\n",
    "    for text in raw_texts:\n",
    "        try:\n",
    "            text = text.decode(\"utf-8\")\n",
    "            clean_text = text.replace(\" \\'m\", \"'m\").replace(\" \\'ll\", \"'ll\").replace(\" \\'re\", \"'re\").replace(\" \\'s\", \"'s\").replace(\" \\'re\", \"'re\").replace(\" n\\'t\", \"n't\").replace(\" \\'ve\", \"'ve\").replace(\" /'d\", \"'d\")\n",
    "            clean_texts.append(clean_text)\n",
    "        except AttributeError:\n",
    "            # print(\"ERROR CLEANING\")\n",
    "            # print(text)\n",
    "            continue\n",
    "        except UnicodeDecodeError:\n",
    "            # print(\"Unicode Error, Skip\")\n",
    "            continue\n",
    "    return clean_texts"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "clean_11 = clean_raw_text(movie_raw['11.txt'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Nice. This is looking a lot cleaner. We can now run some of our lucem_illud text cleaning methods we discuss/model in week 4. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "import lucem_illud_2020"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['@@216680',\n",
       " 'Hey',\n",
       " 'I',\n",
       " \"'m\",\n",
       " 'talking',\n",
       " 'to',\n",
       " 'you',\n",
       " 'Give',\n",
       " 'me',\n",
       " '600',\n",
       " 'dollars',\n",
       " 'You',\n",
       " 'wish',\n",
       " 'That',\n",
       " \"'s\",\n",
       " 'all',\n",
       " 'we',\n",
       " \"'ve\",\n",
       " 'left',\n",
       " 'And',\n",
       " 'you',\n",
       " 'still',\n",
       " 'go',\n",
       " 'to',\n",
       " 'gamble',\n",
       " 'Shut',\n",
       " 'up',\n",
       " 'I',\n",
       " 'earn',\n",
       " 'the',\n",
       " 'money',\n",
       " 'Even',\n",
       " 'that',\n",
       " 'you',\n",
       " 'ca',\n",
       " \"n't\",\n",
       " 'take',\n",
       " 'it',\n",
       " 'for',\n",
       " 'gamble',\n",
       " 'Shut',\n",
       " 'up',\n",
       " 'What',\n",
       " \"'re\",\n",
       " 'you',\n",
       " 'doing',\n",
       " 'Bastard',\n",
       " 'I',\n",
       " \"'m\",\n",
       " 'gon',\n",
       " 'na',\n",
       " 'beat',\n",
       " 'you',\n",
       " 'You',\n",
       " 'gambling',\n",
       " 'pig',\n",
       " 'I',\n",
       " \"'ll\",\n",
       " 'beat',\n",
       " 'the',\n",
       " 'shit',\n",
       " 'out',\n",
       " 'of',\n",
       " 'you',\n",
       " 'You',\n",
       " 'bitch',\n",
       " 'I',\n",
       " \"'ll\",\n",
       " 'beat',\n",
       " 'you',\n",
       " 'You',\n",
       " 'dare',\n",
       " 'to',\n",
       " 'hit',\n",
       " 'me',\n",
       " 'with',\n",
       " 'something',\n",
       " 'I',\n",
       " \"'ll\",\n",
       " 'kill',\n",
       " 'you',\n",
       " 'All',\n",
       " 'you',\n",
       " 'know',\n",
       " 'is',\n",
       " 'gambling',\n",
       " 'I',\n",
       " \"'ll\",\n",
       " 'beat',\n",
       " 'you',\n",
       " 'What',\n",
       " \"'re\",\n",
       " 'you',\n",
       " 'doing',\n",
       " 'Let',\n",
       " 'go',\n",
       " 'of',\n",
       " 'me',\n",
       " 'Stop',\n",
       " 'You',\n",
       " \"'ll\",\n",
       " 'kill',\n",
       " 'Mom',\n",
       " 'Mom',\n",
       " 'are',\n",
       " 'you',\n",
       " 'all',\n",
       " 'right',\n",
       " 'Do',\n",
       " \"n't\",\n",
       " 'touch',\n",
       " 'my',\n",
       " 'money',\n",
       " 'Dad',\n",
       " 'where',\n",
       " \"'re\",\n",
       " 'you',\n",
       " 'going',\n",
       " 'Go',\n",
       " 'away',\n",
       " 'Dad',\n",
       " 'do',\n",
       " \"n't\",\n",
       " 'go',\n",
       " 'Get',\n",
       " 'lost',\n",
       " 'Dad',\n",
       " 'come',\n",
       " 'back',\n",
       " 'You',\n",
       " 'take',\n",
       " 'Marble',\n",
       " 'with',\n",
       " 'you',\n",
       " 'The',\n",
       " 'way',\n",
       " 'she',\n",
       " \"'s\",\n",
       " 'now',\n",
       " 'Do',\n",
       " 'you',\n",
       " 'want',\n",
       " 'to',\n",
       " 'give',\n",
       " 'me',\n",
       " 'trouble',\n",
       " 'I',\n",
       " \"'m\",\n",
       " 'in',\n",
       " 'more',\n",
       " 'trouble',\n",
       " 'than',\n",
       " 'you',\n",
       " 'I',\n",
       " 'am',\n",
       " 'a',\n",
       " 'woman',\n",
       " 'I',\n",
       " \"'m\",\n",
       " 'a',\n",
       " 'sailor',\n",
       " 'You',\n",
       " 'take',\n",
       " 'care',\n",
       " 'of',\n",
       " 'Marble',\n",
       " 'Just',\n",
       " 'gamble',\n",
       " 'a',\n",
       " 'bit',\n",
       " 'less',\n",
       " 'is',\n",
       " 'enough',\n",
       " 'to',\n",
       " 'raise',\n",
       " 'the',\n",
       " 'kid',\n",
       " 'No',\n",
       " 'way',\n",
       " 'Raising',\n",
       " 'her',\n",
       " 'would',\n",
       " 'be',\n",
       " 'like',\n",
       " 'raising',\n",
       " 'others',\n",
       " 'take',\n",
       " 'care',\n",
       " 'of',\n",
       " 'her',\n",
       " 'Marry',\n",
       " 'her',\n",
       " 'when',\n",
       " 'she',\n",
       " 'grows',\n",
       " 'up',\n",
       " 'And',\n",
       " 'you',\n",
       " 'tell',\n",
       " 'her',\n",
       " 'to',\n",
       " 'be',\n",
       " 'smart',\n",
       " 'Do',\n",
       " \"n't\",\n",
       " 'marry',\n",
       " 'a',\n",
       " 'gambler',\n",
       " 'Hey',\n",
       " 'you',\n",
       " 'want',\n",
       " 'a',\n",
       " 'quarrel',\n",
       " 'No',\n",
       " 'never',\n",
       " 'This',\n",
       " 'is',\n",
       " 'the',\n",
       " 'last',\n",
       " 'time',\n",
       " 'You',\n",
       " \"'re\",\n",
       " 'SO',\n",
       " 'right',\n",
       " 'Do',\n",
       " \"n't\",\n",
       " 'rush',\n",
       " 'Everyone',\n",
       " 'will',\n",
       " 'have',\n",
       " 'a',\n",
       " 'share',\n",
       " 'Why',\n",
       " 'so',\n",
       " 'crowded',\n",
       " 'What',\n",
       " \"'re\",\n",
       " 'they',\n",
       " 'doing',\n",
       " 'Worshipping',\n",
       " 'ancestors',\n",
       " 'With',\n",
       " 'all',\n",
       " 'those',\n",
       " 'many',\n",
       " 'things',\n",
       " 'Hey',\n",
       " 'Granny',\n",
       " 'Come',\n",
       " 'on',\n",
       " 'Come',\n",
       " 'and',\n",
       " 'take',\n",
       " 'a',\n",
       " 'bite',\n",
       " 'The',\n",
       " 'dishes',\n",
       " 'are',\n",
       " 'good',\n",
       " 'this',\n",
       " 'year',\n",
       " 'Right',\n",
       " 'Marble',\n",
       " 'have',\n",
       " 'some',\n",
       " 'I',\n",
       " \"'ll\",\n",
       " 'go',\n",
       " 'get',\n",
       " 'my',\n",
       " 'share',\n",
       " 'of',\n",
       " 'pork',\n",
       " 'This',\n",
       " 'is',\n",
       " 'a',\n",
       " 'leg',\n",
       " 'washing',\n",
       " 'basin',\n",
       " 'Hey',\n",
       " 'how',\n",
       " 'can',\n",
       " 'you',\n",
       " 'say',\n",
       " 'that',\n",
       " 'Let',\n",
       " \"'s\",\n",
       " 'go',\n",
       " 'get',\n",
       " 'a',\n",
       " 'share',\n",
       " 'of',\n",
       " 'pork',\n",
       " 'Your',\n",
       " 'share',\n",
       " 'You',\n",
       " 'wish',\n",
       " 'That',\n",
       " \"'s\",\n",
       " 'only',\n",
       " 'for',\n",
       " 'boys',\n",
       " 'Yeah',\n",
       " 'I',\n",
       " 'have',\n",
       " 'a',\n",
       " 'wee',\n",
       " 'wee',\n",
       " 'Have',\n",
       " 'you',\n",
       " 'Aunty',\n",
       " 'Chiang',\n",
       " 'Lin',\n",
       " 'one',\n",
       " 'more',\n",
       " 'baby',\n",
       " 'this',\n",
       " 'year',\n",
       " 'Yes',\n",
       " 'You',\n",
       " 'are',\n",
       " 'the',\n",
       " 'best',\n",
       " 'in',\n",
       " 'your',\n",
       " 'family',\n",
       " 'Hey',\n",
       " 'you',\n",
       " 'think',\n",
       " 'it',\n",
       " \"'s\",\n",
       " 'my',\n",
       " 'will',\n",
       " 'I',\n",
       " 'was',\n",
       " 'compelled',\n",
       " 'by',\n",
       " 'my',\n",
       " 'stupid',\n",
       " 'husband',\n",
       " 'Come',\n",
       " 'on',\n",
       " 'Let',\n",
       " \"'s\",\n",
       " 'worship',\n",
       " 'first',\n",
       " 'Eat',\n",
       " 'some',\n",
       " 'roast',\n",
       " 'pork',\n",
       " 'We',\n",
       " 'have',\n",
       " 'a',\n",
       " 'lot',\n",
       " 'of',\n",
       " 'it',\n",
       " 'Honey',\n",
       " 'baby',\n",
       " 'Yes',\n",
       " 'Only',\n",
       " 'with',\n",
       " 'Chinese',\n",
       " 'blood',\n",
       " 'What',\n",
       " 'do',\n",
       " 'you',\n",
       " 'mean',\n",
       " 'It',\n",
       " 'means',\n",
       " 'I',\n",
       " 'give',\n",
       " 'you',\n",
       " 'a',\n",
       " 'green',\n",
       " 'hat',\n",
       " 'Oh',\n",
       " 'that',\n",
       " 'goes',\n",
       " 'with',\n",
       " 'my',\n",
       " 'clothes',\n",
       " 'then',\n",
       " 'Eat',\n",
       " 'no',\n",
       " 'more',\n",
       " 'talk',\n",
       " 'So',\n",
       " 'rude',\n",
       " 'Take',\n",
       " 'it',\n",
       " 'home',\n",
       " 'Get',\n",
       " 'up',\n",
       " 'go',\n",
       " 'We',\n",
       " \"'re\",\n",
       " 'going',\n",
       " 'back',\n",
       " 'home',\n",
       " 'Be',\n",
       " 'quick',\n",
       " 'Honey',\n",
       " 'Hey',\n",
       " 'make',\n",
       " 'a',\n",
       " 'stance',\n",
       " 'Hold',\n",
       " 'it',\n",
       " 'Master',\n",
       " 'every',\n",
       " 'time',\n",
       " 'we',\n",
       " \"'ve\",\n",
       " 'to',\n",
       " 'stand',\n",
       " 'in',\n",
       " 'a',\n",
       " 'pose',\n",
       " 'If',\n",
       " 'you',\n",
       " 'could',\n",
       " 'teach',\n",
       " 'us',\n",
       " 'something',\n",
       " 'else',\n",
       " 'please',\n",
       " 'Okay',\n",
       " 'Shaolin',\n",
       " 'Masters',\n",
       " 'learn',\n",
       " 'poses',\n",
       " 'for',\n",
       " 'years',\n",
       " 'Hold',\n",
       " 'it',\n",
       " 'Okay',\n",
       " 'I',\n",
       " \"'ll\",\n",
       " 'teach',\n",
       " 'you',\n",
       " 'something',\n",
       " 'new',\n",
       " 'Watch',\n",
       " 'me',\n",
       " 'See',\n",
       " 'Get',\n",
       " 'to',\n",
       " 'it',\n",
       " 'Seeing',\n",
       " 'Shanghai',\n",
       " 'Beach',\n",
       " 'in',\n",
       " 'Mainland',\n",
       " 'Here',\n",
       " 'I',\n",
       " \"'m\",\n",
       " 'still',\n",
       " 'watching',\n",
       " 'it',\n",
       " 'everyday',\n",
       " 'I',\n",
       " \"'m\",\n",
       " 'so',\n",
       " 'tired',\n",
       " 'Tired',\n",
       " 'Let',\n",
       " \"'s\",\n",
       " 'go',\n",
       " 'to',\n",
       " 'bed',\n",
       " 'for',\n",
       " 'rest',\n",
       " 'I',\n",
       " 'hate',\n",
       " 'Chau',\n",
       " 'Yun',\n",
       " 'Fa',\n",
       " 'the',\n",
       " 'most',\n",
       " 'He',\n",
       " 'did',\n",
       " \"n't\",\n",
       " 'marry',\n",
       " 'when',\n",
       " 'I',\n",
       " \"'m\",\n",
       " 'in',\n",
       " 'China',\n",
       " 'As',\n",
       " 'soon',\n",
       " 'as',\n",
       " 'I',\n",
       " \"'m\",\n",
       " 'here',\n",
       " 'he',\n",
       " 'got',\n",
       " 'married',\n",
       " 'You',\n",
       " 'ca',\n",
       " \"n't\",\n",
       " 'come',\n",
       " 'to',\n",
       " 'Hong',\n",
       " 'Kong',\n",
       " 'if',\n",
       " 'you',\n",
       " 'do',\n",
       " \"n't\",\n",
       " 'marry',\n",
       " 'me',\n",
       " 'You',\n",
       " 'said',\n",
       " 'you',\n",
       " 'are',\n",
       " 'big',\n",
       " 'boss',\n",
       " 'Having',\n",
       " 'a',\n",
       " 'big',\n",
       " 'business',\n",
       " 'Now',\n",
       " 'that',\n",
       " 'I',\n",
       " \"'m\",\n",
       " 'in',\n",
       " 'Hong',\n",
       " 'Kong',\n",
       " 'I',\n",
       " \"'m\",\n",
       " 'still',\n",
       " 'for',\n",
       " 'myself',\n",
       " 'Then',\n",
       " 'I',\n",
       " 'ought',\n",
       " 'to',\n",
       " 'feel',\n",
       " 'much',\n",
       " 'more',\n",
       " 'sorry',\n",
       " 'You',\n",
       " 'ca',\n",
       " \"n't\",\n",
       " 'even',\n",
       " 'lay',\n",
       " 'an',\n",
       " 'egg',\n",
       " 'This',\n",
       " 'year',\n",
       " 'I',\n",
       " 'can',\n",
       " 'only',\n",
       " 'get',\n",
       " 'one',\n",
       " 'share',\n",
       " 'of',\n",
       " 'pork',\n",
       " 'And',\n",
       " 'you',\n",
       " \"'re\",\n",
       " 'blaming',\n",
       " 'me',\n",
       " 'for',\n",
       " 'it',\n",
       " 'Damn',\n",
       " 'monkey',\n",
       " 'What',\n",
       " \"'s\",\n",
       " 'going',\n",
       " 'on',\n",
       " 'A',\n",
       " 'letter',\n",
       " 'for',\n",
       " 'you',\n",
       " 'Oh',\n",
       " 'it',\n",
       " \"'s\",\n",
       " 'you',\n",
       " 'Your',\n",
       " 'monkey',\n",
       " 'glared',\n",
       " 'at',\n",
       " 'me',\n",
       " 'fiercely',\n",
       " 'Will',\n",
       " 'you',\n",
       " 'open',\n",
       " 'it',\n",
       " 'for',\n",
       " 'me',\n",
       " 'Read',\n",
       " 'it',\n",
       " 'to',\n",
       " 'me',\n",
       " 'I',\n",
       " 'am',\n",
       " 'quite',\n",
       " 'illiterate',\n",
       " 'Stop',\n",
       " 'it',\n",
       " 'Do',\n",
       " \"n't\",\n",
       " 'bother',\n",
       " 'him',\n",
       " 'Let',\n",
       " 'me',\n",
       " 'help',\n",
       " 'you',\n",
       " 'Give',\n",
       " 'me',\n",
       " 'the',\n",
       " 'letter',\n",
       " 'Postman',\n",
       " \"'s\",\n",
       " 'duty',\n",
       " 'is',\n",
       " 'to',\n",
       " 'deliver',\n",
       " 'and',\n",
       " 'read',\n",
       " 'letters',\n",
       " 'He',\n",
       " 'reads',\n",
       " 'for',\n",
       " 'Aunt',\n",
       " 'Jiao',\n",
       " 'How',\n",
       " 'can',\n",
       " 'you',\n",
       " 'compare',\n",
       " 'to',\n",
       " 'her',\n",
       " 'Aunty',\n",
       " 'Jiao',\n",
       " 'is',\n",
       " '83',\n",
       " 'years',\n",
       " 'old',\n",
       " 'Hello',\n",
       " 'dubbing',\n",
       " 'room',\n",
       " 'Martin',\n",
       " 'Martin',\n",
       " 'is',\n",
       " 'working',\n",
       " 'Are',\n",
       " 'you',\n",
       " 'lying',\n",
       " 'to',\n",
       " 'me',\n",
       " 'Why',\n",
       " 'should',\n",
       " 'I',\n",
       " 'lie',\n",
       " 'I',\n",
       " 'know',\n",
       " 'he',\n",
       " \"'s\",\n",
       " 'free',\n",
       " 'Get',\n",
       " 'him',\n",
       " 'You',\n",
       " 'listen',\n",
       " 'yourself',\n",
       " 'then',\n",
       " 'Hello',\n",
       " 'what',\n",
       " \"'s\",\n",
       " 'the',\n",
       " 'noise',\n",
       " 'Is',\n",
       " 'someone',\n",
       " 'being',\n",
       " 'raped',\n",
       " 'Martin',\n",
       " 'your',\n",
       " 'telephone',\n",
       " 'Hello',\n",
       " 'What',\n",
       " 'took',\n",
       " 'you',\n",
       " 'so',\n",
       " 'long',\n",
       " 'Recording',\n",
       " 'is',\n",
       " 'going',\n",
       " 'on',\n",
       " 'Do',\n",
       " \"n't\",\n",
       " 'mess',\n",
       " 'around',\n",
       " 'Yes',\n",
       " 'I',\n",
       " 'know',\n",
       " 'I',\n",
       " 'wo',\n",
       " \"n't\",\n",
       " 'Honey',\n",
       " 'did',\n",
       " 'you',\n",
       " 'spend',\n",
       " 'so',\n",
       " 'much',\n",
       " 'I',\n",
       " 'was',\n",
       " 'late',\n",
       " 'for',\n",
       " 'work',\n",
       " 'so',\n",
       " 'I',\n",
       " 'took',\n",
       " 'a',\n",
       " 'taxi',\n",
       " 'On',\n",
       " 'your',\n",
       " 'way',\n",
       " 'back',\n",
       " 'do',\n",
       " \"n't\",\n",
       " 'forget',\n",
       " 'to',\n",
       " 'buy',\n",
       " 'vegetables',\n",
       " 'Okay',\n",
       " 'And',\n",
       " 'buy',\n",
       " 'a',\n",
       " 'pair',\n",
       " 'of',\n",
       " 'slippers',\n",
       " 'for',\n",
       " 'me',\n",
       " 'Got',\n",
       " 'it',\n",
       " 'honey',\n",
       " 'I',\n",
       " \"'ll\",\n",
       " 'hang',\n",
       " 'up',\n",
       " 'Bye',\n",
       " 'Would',\n",
       " \"n't\",\n",
       " 'marry',\n",
       " 'if',\n",
       " 'you',\n",
       " 'know',\n",
       " 'your',\n",
       " 'wife',\n",
       " 'is',\n",
       " 'so',\n",
       " 'bad',\n",
       " 'Take',\n",
       " 'pity',\n",
       " 'for',\n",
       " 'the',\n",
       " 'rest',\n",
       " 'of',\n",
       " 'my',\n",
       " 'life',\n",
       " 'Hey',\n",
       " 'what',\n",
       " \"'s\",\n",
       " 'on',\n",
       " 'your',\n",
       " 'neck',\n",
       " 'Nothing',\n",
       " 'You',\n",
       " 'look',\n",
       " 'piteous',\n",
       " 'She',\n",
       " \"'s\",\n",
       " 'the',\n",
       " 'piteous',\n",
       " 'one',\n",
       " 'The',\n",
       " 'vampire',\n",
       " 'is',\n",
       " 'here',\n",
       " 'Grandma',\n",
       " 'grandma',\n",
       " 'Someone',\n",
       " 'is',\n",
       " 'fighting',\n",
       " 'Grandma',\n",
       " 'grandma',\n",
       " 'Come',\n",
       " 'over',\n",
       " 'quickly',\n",
       " 'What',\n",
       " 'happened',\n",
       " 'Someone',\n",
       " 'is',\n",
       " 'fighting',\n",
       " 'Fighting',\n",
       " 'Yes',\n",
       " 'Where',\n",
       " 'Over',\n",
       " 'there',\n",
       " 'You',\n",
       " 'see',\n",
       " 'it',\n",
       " 'I',\n",
       " 'spare',\n",
       " 'you',\n",
       " 'this',\n",
       " 'time',\n",
       " 'No',\n",
       " 'next',\n",
       " 'time',\n",
       " 'Sorry',\n",
       " 'let',\n",
       " 'me',\n",
       " 'kiss',\n",
       " 'you',\n",
       " 'Cut',\n",
       " 'the',\n",
       " 'bullshit',\n",
       " 'Go',\n",
       " 'to',\n",
       " 'bed',\n",
       " 'I',\n",
       " 'need',\n",
       " 'to',\n",
       " 'work',\n",
       " 'tomorrow',\n",
       " 'Where',\n",
       " 'at',\n",
       " 'That',\n",
       " 'villa',\n",
       " 'over',\n",
       " 'there',\n",
       " 'Wow',\n",
       " 'it',\n",
       " 'looks',\n",
       " 'beautiful',\n",
       " 'Who',\n",
       " 'lives',\n",
       " 'there',\n",
       " 'Snow',\n",
       " 'White',\n",
       " 'Kidding',\n",
       " 'there',\n",
       " \"'s\",\n",
       " 'no',\n",
       " 'Snow',\n",
       " 'White',\n",
       " 'Oh',\n",
       " 'murder',\n",
       " 'That',\n",
       " \"'s\",\n",
       " 'the',\n",
       " 'pig',\n",
       " 'scream',\n",
       " 'Why',\n",
       " \"'s\",\n",
       " 'the',\n",
       " 'pig',\n",
       " 'screaming',\n",
       " 'so',\n",
       " 'late',\n",
       " 'at',\n",
       " 'night',\n",
       " 'Shorty',\n",
       " \"'s\",\n",
       " 'late',\n",
       " 'So',\n",
       " 'the',\n",
       " 'government',\n",
       " 'wo',\n",
       " \"n't\",\n",
       " 'know',\n",
       " 'But',\n",
       " 'how',\n",
       " 'could',\n",
       " 'a',\n",
       " 'pig',\n",
       " \"'s\",\n",
       " 'scream',\n",
       " 'be',\n",
       " 'so',\n",
       " 'terrible',\n",
       " 'Why',\n",
       " 'Why',\n",
       " 'are',\n",
       " \"n't\",\n",
       " 'you',\n",
       " 'tired',\n",
       " 'Why',\n",
       " 'are',\n",
       " \"n't\",\n",
       " 'you',\n",
       " 'sleepy',\n",
       " 'Why',\n",
       " 'have',\n",
       " \"n't\",\n",
       " 'you',\n",
       " 'gone',\n",
       " 'to',\n",
       " 'bed',\n",
       " 'Why',\n",
       " 'are',\n",
       " 'you',\n",
       " 'so',\n",
       " 'nosy',\n",
       " 'Go',\n",
       " 'to',\n",
       " 'sleep',\n",
       " 'Here',\n",
       " \"'s\",\n",
       " 'the',\n",
       " 'kitchen',\n",
       " 'Oil',\n",
       " 'Salt',\n",
       " 'Sauce',\n",
       " 'Vinegar',\n",
       " 'All',\n",
       " 'there',\n",
       " 'Go',\n",
       " 'tidy',\n",
       " 'up',\n",
       " 'Ask',\n",
       " 'me',\n",
       " 'if',\n",
       " 'you',\n",
       " 'have',\n",
       " 'any',\n",
       " 'question',\n",
       " 'Come',\n",
       " 'dance',\n",
       " 'with',\n",
       " 'me',\n",
       " 'Come',\n",
       " 'on',\n",
       " 'Come',\n",
       " 'Dance',\n",
       " 'I',\n",
       " \"'ll\",\n",
       " 'teach',\n",
       " 'you',\n",
       " 'Do',\n",
       " \"n't\",\n",
       " 'be',\n",
       " 'afraid',\n",
       " 'Damn',\n",
       " 'dog',\n",
       " 'Damn',\n",
       " 'it',\n",
       " 'What',\n",
       " \"'s\",\n",
       " 'such',\n",
       " 'a',\n",
       " 'big',\n",
       " 'deal',\n",
       " 'to',\n",
       " 'be',\n",
       " 'rich',\n",
       " 'Pay',\n",
       " 'for',\n",
       " 'my',\n",
       " 'pants',\n",
       " 'Sorry',\n",
       " 'I',\n",
       " 'do',\n",
       " \"n't\",\n",
       " 'mean',\n",
       " 'to',\n",
       " 'be',\n",
       " 'rude',\n",
       " 'Be',\n",
       " 'careful',\n",
       " 'next',\n",
       " 'time',\n",
       " 'Miss',\n",
       " 'it',\n",
       " 'seems',\n",
       " 'I',\n",
       " \"'ve\",\n",
       " 'seen',\n",
       " 'you',\n",
       " 'somewhere',\n",
       " 'Of',\n",
       " 'course',\n",
       " 'you',\n",
       " 'have',\n",
       " 'I',\n",
       " \"'ll\",\n",
       " 'be',\n",
       " 'the',\n",
       " 'Miss',\n",
       " 'Hong',\n",
       " 'Kong',\n",
       " 'in',\n",
       " 'next',\n",
       " 'contest',\n",
       " 'I',\n",
       " 'have',\n",
       " 'nine',\n",
       " 'posters',\n",
       " 'at',\n",
       " 'the',\n",
       " ...]"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lucem_illud_2020.word_tokenize(clean_11[1])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['@@216680',\n",
       " 'hey',\n",
       " 'talk',\n",
       " 'dollar',\n",
       " 'wish',\n",
       " 'leave',\n",
       " 'gamble',\n",
       " 'shut',\n",
       " 'earn',\n",
       " 'money',\n",
       " 'gamble',\n",
       " 'shut',\n",
       " 'bastard',\n",
       " 'gon',\n",
       " 'na',\n",
       " 'beat',\n",
       " 'gamble',\n",
       " 'pig',\n",
       " 'beat',\n",
       " 'shit',\n",
       " 'bitch',\n",
       " 'beat',\n",
       " 'dare',\n",
       " 'hit',\n",
       " 'kill',\n",
       " 'know',\n",
       " 'gamble',\n",
       " 'beat',\n",
       " 'let',\n",
       " 'stop',\n",
       " 'kill',\n",
       " 'mom',\n",
       " 'mom',\n",
       " 'right',\n",
       " 'touch',\n",
       " 'money',\n",
       " 'dad',\n",
       " 'go',\n",
       " 'away',\n",
       " 'dad',\n",
       " 'lose',\n",
       " 'dad',\n",
       " 'come',\n",
       " 'marble',\n",
       " 'way',\n",
       " 'want',\n",
       " 'trouble',\n",
       " 'trouble',\n",
       " 'woman',\n",
       " 'sailor',\n",
       " 'care',\n",
       " 'marble',\n",
       " 'gamble',\n",
       " 'bit',\n",
       " 'raise',\n",
       " 'kid',\n",
       " 'way',\n",
       " 'raise',\n",
       " 'like',\n",
       " 'raise',\n",
       " 'care',\n",
       " 'marry',\n",
       " 'grow',\n",
       " 'tell',\n",
       " 'smart',\n",
       " 'marry',\n",
       " 'gambler',\n",
       " 'hey',\n",
       " 'want',\n",
       " 'quarrel',\n",
       " 'time',\n",
       " 'right',\n",
       " 'rush',\n",
       " 'share',\n",
       " 'crowded',\n",
       " 'worship',\n",
       " 'ancestor',\n",
       " 'thing',\n",
       " 'hey',\n",
       " 'granny',\n",
       " 'come',\n",
       " 'come',\n",
       " 'bite',\n",
       " 'dish',\n",
       " 'good',\n",
       " 'year',\n",
       " 'right',\n",
       " 'marble',\n",
       " 'share',\n",
       " 'pork',\n",
       " 'leg',\n",
       " 'washing',\n",
       " 'basin',\n",
       " 'hey',\n",
       " 'let',\n",
       " 'share',\n",
       " 'pork',\n",
       " 'share',\n",
       " 'wish',\n",
       " 'boy',\n",
       " 'yeah',\n",
       " 'wee',\n",
       " 'wee',\n",
       " 'aunty',\n",
       " 'chiang',\n",
       " 'lin',\n",
       " 'baby',\n",
       " 'year',\n",
       " 'yes',\n",
       " 'good',\n",
       " 'family',\n",
       " 'hey',\n",
       " 'think',\n",
       " 'compel',\n",
       " 'stupid',\n",
       " 'husband',\n",
       " 'come',\n",
       " 'let',\n",
       " 'worship',\n",
       " 'eat',\n",
       " 'roast',\n",
       " 'pork',\n",
       " 'lot',\n",
       " 'honey',\n",
       " 'baby',\n",
       " 'yes',\n",
       " 'chinese',\n",
       " 'blood',\n",
       " 'mean',\n",
       " 'mean',\n",
       " 'green',\n",
       " 'hat',\n",
       " 'oh',\n",
       " 'go',\n",
       " 'clothe',\n",
       " 'eat',\n",
       " 'talk',\n",
       " 'rude',\n",
       " 'home',\n",
       " 'go',\n",
       " 'home',\n",
       " 'quick',\n",
       " 'honey',\n",
       " 'hey',\n",
       " 'stance',\n",
       " 'hold',\n",
       " 'master',\n",
       " 'time',\n",
       " 'stand',\n",
       " 'pose',\n",
       " 'teach',\n",
       " 'okay',\n",
       " 'shaolin',\n",
       " 'master',\n",
       " 'learn',\n",
       " 'pose',\n",
       " 'year',\n",
       " 'hold',\n",
       " 'okay',\n",
       " 'teach',\n",
       " 'new',\n",
       " 'watch',\n",
       " 'see',\n",
       " 'shanghai',\n",
       " 'beach',\n",
       " 'mainland',\n",
       " 'watch',\n",
       " 'everyday',\n",
       " 'tired',\n",
       " 'tired',\n",
       " 'let',\n",
       " 'bed',\n",
       " 'rest',\n",
       " 'hate',\n",
       " 'chau',\n",
       " 'yun',\n",
       " 'fa',\n",
       " 'marry',\n",
       " 'china',\n",
       " 'soon',\n",
       " 'get',\n",
       " 'marry',\n",
       " 'come',\n",
       " 'hong',\n",
       " 'kong',\n",
       " 'marry',\n",
       " 'say',\n",
       " 'big',\n",
       " 'boss',\n",
       " 'have',\n",
       " 'big',\n",
       " 'business',\n",
       " 'hong',\n",
       " 'kong',\n",
       " 'ought',\n",
       " 'feel',\n",
       " 'sorry',\n",
       " 'lay',\n",
       " 'egg',\n",
       " 'year',\n",
       " 'share',\n",
       " 'pork',\n",
       " 'blame',\n",
       " 'damn',\n",
       " 'monkey',\n",
       " 'go',\n",
       " 'letter',\n",
       " 'oh',\n",
       " 'monkey',\n",
       " 'glare',\n",
       " 'fiercely',\n",
       " 'open',\n",
       " 'read',\n",
       " 'illiterate',\n",
       " 'stop',\n",
       " 'bother',\n",
       " 'let',\n",
       " 'help',\n",
       " 'letter',\n",
       " 'postman',\n",
       " 'duty',\n",
       " 'deliver',\n",
       " 'read',\n",
       " 'letter',\n",
       " 'read',\n",
       " 'aunt',\n",
       " 'jiao',\n",
       " 'compare',\n",
       " 'aunty',\n",
       " 'jiao',\n",
       " 'year',\n",
       " 'old',\n",
       " 'hello',\n",
       " 'dub',\n",
       " 'room',\n",
       " 'martin',\n",
       " 'martin',\n",
       " 'work',\n",
       " 'lie',\n",
       " 'lie',\n",
       " 'know',\n",
       " 'free',\n",
       " 'listen',\n",
       " 'hello',\n",
       " 'noise',\n",
       " 'rape',\n",
       " 'martin',\n",
       " 'telephone',\n",
       " 'hello',\n",
       " 'take',\n",
       " 'long',\n",
       " 'record',\n",
       " 'go',\n",
       " 'mess',\n",
       " 'yes',\n",
       " 'know',\n",
       " 'will',\n",
       " 'honey',\n",
       " 'spend',\n",
       " 'late',\n",
       " 'work',\n",
       " 'take',\n",
       " 'taxi',\n",
       " 'way',\n",
       " 'forget',\n",
       " 'buy',\n",
       " 'vegetable',\n",
       " 'okay',\n",
       " 'buy',\n",
       " 'pair',\n",
       " 'slipper',\n",
       " 'get',\n",
       " 'honey',\n",
       " 'hang',\n",
       " 'bye',\n",
       " 'marry',\n",
       " 'know',\n",
       " 'wife',\n",
       " 'bad',\n",
       " 'pity',\n",
       " 'rest',\n",
       " 'life',\n",
       " 'hey',\n",
       " 'neck',\n",
       " 'look',\n",
       " 'piteous',\n",
       " 'piteous',\n",
       " 'vampire',\n",
       " 'grandma',\n",
       " 'grandma',\n",
       " 'fight',\n",
       " 'grandma',\n",
       " 'grandma',\n",
       " 'come',\n",
       " 'quickly',\n",
       " 'happen',\n",
       " 'fight',\n",
       " 'fight',\n",
       " 'yes',\n",
       " 'spare',\n",
       " 'time',\n",
       " 'time',\n",
       " 'sorry',\n",
       " 'let',\n",
       " 'kiss',\n",
       " 'cut',\n",
       " 'bullshit',\n",
       " 'bed',\n",
       " 'need',\n",
       " 'work',\n",
       " 'tomorrow',\n",
       " 'villa',\n",
       " 'wow',\n",
       " 'look',\n",
       " 'beautiful',\n",
       " 'live',\n",
       " 'snow',\n",
       " 'white',\n",
       " 'kidding',\n",
       " 'snow',\n",
       " 'white',\n",
       " 'oh',\n",
       " 'murder',\n",
       " 'pig',\n",
       " 'scream',\n",
       " 'pig',\n",
       " 'scream',\n",
       " 'late',\n",
       " 'night',\n",
       " 'shorty',\n",
       " 'late',\n",
       " 'government',\n",
       " 'will',\n",
       " 'know',\n",
       " 'pig',\n",
       " 'scream',\n",
       " 'terrible',\n",
       " 'tired',\n",
       " 'sleepy',\n",
       " 'go',\n",
       " 'bed',\n",
       " 'nosy',\n",
       " 'sleep',\n",
       " 'kitchen',\n",
       " 'oil',\n",
       " 'salt',\n",
       " 'sauce',\n",
       " 'vinegar',\n",
       " 'tidy',\n",
       " 'ask',\n",
       " 'question',\n",
       " 'come',\n",
       " 'dance',\n",
       " 'come',\n",
       " 'come',\n",
       " 'dance',\n",
       " 'teach',\n",
       " 'afraid',\n",
       " 'damn',\n",
       " 'dog',\n",
       " 'damn',\n",
       " 'big',\n",
       " 'deal',\n",
       " 'rich',\n",
       " 'pay',\n",
       " 'pant',\n",
       " 'sorry',\n",
       " 'mean',\n",
       " 'rude',\n",
       " 'careful',\n",
       " 'time',\n",
       " 'miss',\n",
       " 'see',\n",
       " 'course',\n",
       " 'miss',\n",
       " 'hong',\n",
       " 'kong',\n",
       " 'contest',\n",
       " 'poster',\n",
       " 'ferry',\n",
       " 'pier',\n",
       " 'hand',\n",
       " 'photograph',\n",
       " 'see',\n",
       " 'rude',\n",
       " 'chang',\n",
       " 'jin',\n",
       " 'girl',\n",
       " 'great',\n",
       " 'live',\n",
       " 'village',\n",
       " 'course',\n",
       " 'ugly',\n",
       " 'live',\n",
       " 'village',\n",
       " 'miss',\n",
       " 'want',\n",
       " 'ball',\n",
       " 'holland',\n",
       " 'shall',\n",
       " 'friendship',\n",
       " 'dad',\n",
       " 'say',\n",
       " 'get',\n",
       " 'marry',\n",
       " 'married',\n",
       " 'way',\n",
       " 'look',\n",
       " 'curve',\n",
       " 'sure',\n",
       " 'know',\n",
       " 'fit',\n",
       " 'maybe',\n",
       " 'take',\n",
       " 'birth',\n",
       " 'control',\n",
       " 'pill',\n",
       " 'hey',\n",
       " 'pal',\n",
       " 'nice',\n",
       " 'girl',\n",
       " 'huh',\n",
       " 'mean',\n",
       " 'think',\n",
       " 'kungfu',\n",
       " 'master',\n",
       " 'ball',\n",
       " 'okay',\n",
       " 'friend',\n",
       " 'face',\n",
       " 'dad',\n",
       " 'sake',\n",
       " 'let',\n",
       " 'fuss',\n",
       " 'alright',\n",
       " 'hit',\n",
       " 'beat',\n",
       " 'stop',\n",
       " 'play',\n",
       " 'stop',\n",
       " 'stop',\n",
       " 'stop',\n",
       " 'let',\n",
       " 'damn',\n",
       " 'kid',\n",
       " 'let',\n",
       " 'kill',\n",
       " 'time',\n",
       " 'let',\n",
       " 'waste',\n",
       " 'strength',\n",
       " 'post',\n",
       " 'boy',\n",
       " 'think',\n",
       " 'good',\n",
       " 'take',\n",
       " 'pity',\n",
       " 'yeah',\n",
       " 'sorry',\n",
       " 'bother',\n",
       " 'sign',\n",
       " 'wang',\n",
       " 'go',\n",
       " 'kowloon',\n",
       " 'business',\n",
       " 'oh',\n",
       " 'bleed',\n",
       " 'hurt',\n",
       " 'let',\n",
       " 'help',\n",
       " 'suppress',\n",
       " 'anger',\n",
       " 'good',\n",
       " 'thing',\n",
       " 'let',\n",
       " 'clean',\n",
       " 'wound',\n",
       " 'okay',\n",
       " 'will',\n",
       " 'touch',\n",
       " 'water',\n",
       " 'deliver',\n",
       " 'day',\n",
       " 'draw',\n",
       " 'late',\n",
       " 'night',\n",
       " 'bed',\n",
       " 'class',\n",
       " 'tomorrow',\n",
       " 'sleep',\n",
       " 'dad',\n",
       " 'go',\n",
       " 'school',\n",
       " 'late',\n",
       " 'okay',\n",
       " 'know',\n",
       " 'shepherd',\n",
       " 'drive',\n",
       " 'sheep',\n",
       " 'hill',\n",
       " 'watch',\n",
       " 'farmer',\n",
       " 'farm',\n",
       " 'jin',\n",
       " 'bao',\n",
       " 'way',\n",
       " 'uncle',\n",
       " 'smuggle',\n",
       " 'mainland',\n",
       " 'exchange',\n",
       " 'secret',\n",
       " 'secret',\n",
       " 'night',\n",
       " 'white',\n",
       " 'man',\n",
       " 'torture',\n",
       " 'wife',\n",
       " 'believe',\n",
       " 'coward',\n",
       " 'wife',\n",
       " 'beat',\n",
       " 'kind',\n",
       " 'shepherd',\n",
       " 'laugh',\n",
       " 'master',\n",
       " 'good',\n",
       " 'morning',\n",
       " 'teacher',\n",
       " 'teach',\n",
       " 'write',\n",
       " 'letter',\n",
       " 'learn',\n",
       " 'write',\n",
       " 'home',\n",
       " 'address',\n",
       " 'help',\n",
       " 'uncle',\n",
       " 'postman',\n",
       " 'deliver',\n",
       " 'mister',\n",
       " 'postman',\n",
       " 'oh',\n",
       " 'yes',\n",
       " 'call',\n",
       " 'uncle',\n",
       " 'old',\n",
       " 'yes',\n",
       " 'come',\n",
       " 'letter',\n",
       " 'wang',\n",
       " 'jia',\n",
       " 'leung',\n",
       " 'wang',\n",
       " 'shi',\n",
       " 'chung',\n",
       " 'shu',\n",
       " 'ker',\n",
       " 'ying',\n",
       " 'cripple',\n",
       " 'laugh',\n",
       " 'wang',\n",
       " 'jia',\n",
       " 'hao',\n",
       " 'wang',\n",
       " 'jie',\n",
       " 'hwa',\n",
       " 'wang',\n",
       " 'wan',\n",
       " 'wen',\n",
       " 'wang',\n",
       " 'tian',\n",
       " 'sheng',\n",
       " 'marble',\n",
       " 'mister',\n",
       " 'letter',\n",
       " 'dad',\n",
       " 'grandma',\n",
       " 'say',\n",
       " 'panama',\n",
       " 'write',\n",
       " 'ask',\n",
       " 'grandma',\n",
       " 'dad',\n",
       " 'dead',\n",
       " 'curse',\n",
       " 'like',\n",
       " 'away',\n",
       " 'away',\n",
       " 'let',\n",
       " 'let',\n",
       " 'damn',\n",
       " 'dog',\n",
       " 'away',\n",
       " 'let',\n",
       " 'damn',\n",
       " 'dog',\n",
       " 'bite',\n",
       " 'see',\n",
       " 'bite',\n",
       " 'deliver',\n",
       " 'letter',\n",
       " 'tell',\n",
       " 'people',\n",
       " 'will',\n",
       " 'say',\n",
       " 'responsible',\n",
       " 'secret',\n",
       " 'let',\n",
       " 'hook',\n",
       " 'pinkie',\n",
       " 'true',\n",
       " 'word',\n",
       " 'wait',\n",
       " 'young',\n",
       " 'like',\n",
       " 'hope',\n",
       " 'love',\n",
       " 'like',\n",
       " 'crazy',\n",
       " 'hope',\n",
       " 'lot',\n",
       " 'money',\n",
       " 'money',\n",
       " 'wish',\n",
       " 'famous',\n",
       " 'let',\n",
       " 'know',\n",
       " 'hope',\n",
       " 'pretty',\n",
       " 'stay',\n",
       " 'pretty',\n",
       " 'longer',\n",
       " 'okay',\n",
       " 'woman',\n",
       " 'tragic',\n",
       " 'poor',\n",
       " 'silly',\n",
       " 'girl',\n",
       " 'stay',\n",
       " 'overnight',\n",
       " 'better',\n",
       " 'find',\n",
       " 'excuse',\n",
       " 'convince',\n",
       " 'wife',\n",
       " 'understand',\n",
       " 'man',\n",
       " 'world',\n",
       " 'wife',\n",
       " 'understand',\n",
       " 'love',\n",
       " 'money',\n",
       " 'think',\n",
       " 'santa',\n",
       " 'claus',\n",
       " 'cheap',\n",
       " 'bed',\n",
       " 'early',\n",
       " 'abuse',\n",
       " 'drug',\n",
       " 'wang',\n",
       " 'red',\n",
       " 'cloth',\n",
       " 'okay',\n",
       " 'lin',\n",
       " 'red',\n",
       " 'cloth',\n",
       " 'big',\n",
       " 'small',\n",
       " 'big',\n",
       " 'course',\n",
       " 'think',\n",
       " 'old',\n",
       " 'wang',\n",
       " 'good',\n",
       " 'hand',\n",
       " 'white',\n",
       " 'believe',\n",
       " 'lin',\n",
       " 'work',\n",
       " 'okay',\n",
       " 'dollar',\n",
       " 'bin',\n",
       " 'come',\n",
       " 'buy',\n",
       " 'need',\n",
       " 'come',\n",
       " 'later',\n",
       " 'know',\n",
       " 'neighbour',\n",
       " 'hey',\n",
       " 'button',\n",
       " 'fasten',\n",
       " 'breath',\n",
       " 'fasten',\n",
       " 'buy',\n",
       " 'new',\n",
       " 'dress',\n",
       " 'okay',\n",
       " 'okay',\n",
       " 'let',\n",
       " 'discount',\n",
       " 'stall',\n",
       " 'discount',\n",
       " 'stall',\n",
       " 'sell',\n",
       " 'cheap',\n",
       " 'stuff',\n",
       " 'want',\n",
       " 'visit',\n",
       " 'fashion',\n",
       " 'shop',\n",
       " 'look',\n",
       " 'month',\n",
       " 'learn',\n",
       " 'learn',\n",
       " 'long',\n",
       " 'come',\n",
       " 'good',\n",
       " 'buy',\n",
       " 'want',\n",
       " 'okay',\n",
       " 'be',\n",
       " 'good',\n",
       " 'good',\n",
       " 'good',\n",
       " 'sir',\n",
       " 'like',\n",
       " 'know',\n",
       " 'new',\n",
       " 'villa',\n",
       " 'real',\n",
       " 'talent',\n",
       " 'mistress',\n",
       " 'husband',\n",
       " 'want',\n",
       " 'sure',\n",
       " 'santa',\n",
       " 'claus',\n",
       " 'think',\n",
       " 'young',\n",
       " 'young',\n",
       " 'hear',\n",
       " 'dancer',\n",
       " 'kind',\n",
       " 'dance',\n",
       " 'good',\n",
       " 'kind',\n",
       " 'take',\n",
       " 'hard',\n",
       " 'work',\n",
       " 'look',\n",
       " 'like',\n",
       " 'mad',\n",
       " 'jumping',\n",
       " 'money',\n",
       " 'work',\n",
       " 'hard',\n",
       " 'money',\n",
       " 'depend',\n",
       " 'person',\n",
       " 'love',\n",
       " 'man',\n",
       " 'use',\n",
       " 'laugh',\n",
       " 'night',\n",
       " 'husband',\n",
       " 'tell',\n",
       " 'wife',\n",
       " 'understand',\n",
       " 'lie',\n",
       " 'fool',\n",
       " 'bad',\n",
       " 'lie',\n",
       " 'way',\n",
       " 'husband',\n",
       " 'sure',\n",
       " 'bad',\n",
       " 'luck',\n",
       " 'mistress',\n",
       " 'like',\n",
       " 'luck',\n",
       " 'actually',\n",
       " 'meet',\n",
       " 'whore',\n",
       " 'great',\n",
       " 'let',\n",
       " 'okay',\n",
       " 'like',\n",
       " 'bit',\n",
       " 'like',\n",
       " 'good',\n",
       " 'oh',\n",
       " 'dare',\n",
       " 'hand',\n",
       " 'come',\n",
       " 'time',\n",
       " 'use',\n",
       " 'hip',\n",
       " 'come',\n",
       " 'ready',\n",
       " 'get',\n",
       " 'weak',\n",
       " 'hip',\n",
       " 'forget',\n",
       " 'porn',\n",
       " 'star',\n",
       " 'need',\n",
       " 'bald',\n",
       " 'head',\n",
       " 'extra',\n",
       " 'forever',\n",
       " 'oh',\n",
       " 'yao',\n",
       " 'ju',\n",
       " 'come',\n",
       " 'come',\n",
       " 'play',\n",
       " 'hold',\n",
       " 'suit',\n",
       " 'wardrobe',\n",
       " 'meet',\n",
       " 'girlfriend',\n",
       " 'want',\n",
       " 'look',\n",
       " 'age',\n",
       " 'girl',\n",
       " 'beautiful',\n",
       " 'nanny',\n",
       " 'yao',\n",
       " 'ju',\n",
       " 'know',\n",
       " 'come',\n",
       " 'thing',\n",
       " 'wait',\n",
       " 'wait',\n",
       " 'long',\n",
       " 'time',\n",
       " 'wait',\n",
       " 'wait',\n",
       " 'long',\n",
       " 'time',\n",
       " 'wrong',\n",
       " 'single',\n",
       " 'mind',\n",
       " 'hit',\n",
       " 'yeah',\n",
       " 'dare',\n",
       " 'kill',\n",
       " 'want',\n",
       " 'dump',\n",
       " 'sex',\n",
       " 'money',\n",
       " 'money',\n",
       " 'love',\n",
       " 'money',\n",
       " 'love',\n",
       " 'crazy',\n",
       " 'run',\n",
       " 'cut',\n",
       " 'piece',\n",
       " 'damn',\n",
       " 'shameless',\n",
       " 'bitch',\n",
       " 'let',\n",
       " 'play',\n",
       " 'rubber',\n",
       " 'band',\n",
       " 'rope',\n",
       " 'come',\n",
       " 'let',\n",
       " 'play',\n",
       " 'rubber',\n",
       " 'band',\n",
       " 'rope',\n",
       " 'come',\n",
       " 'come',\n",
       " 'event',\n",
       " 'let',\n",
       " 'clarify',\n",
       " 'relationship',\n",
       " 'today',\n",
       " 'husband',\n",
       " 'husband',\n",
       " 'use',\n",
       " 'low',\n",
       " 'voice',\n",
       " 'afraid',\n",
       " 'need',\n",
       " 'fear',\n",
       " 'dare',\n",
       " 'short',\n",
       " 'money',\n",
       " 'money',\n",
       " 'yes',\n",
       " 'need',\n",
       " 'lot',\n",
       " 'think',\n",
       " 'afford',\n",
       " 'prepared',\n",
       " 'huh',\n",
       " 'let',\n",
       " 'tell',\n",
       " 'cousin',\n",
       " 'come',\n",
       " 'month',\n",
       " 'need',\n",
       " 'cash',\n",
       " 'permit',\n",
       " 'would',\n",
       " 'card',\n",
       " 'take',\n",
       " 'dollar',\n",
       " 'worth',\n",
       " 'maybe',\n",
       " 'worth',\n",
       " 'spread',\n",
       " 'word',\n",
       " 'woman',\n",
       " 'trouble',\n",
       " 'think',\n",
       " 'know',\n",
       " 'mistress',\n",
       " 'think',\n",
       " 'man',\n",
       " 'know',\n",
       " 'affair',\n",
       " 'bear',\n",
       " 'lose',\n",
       " 'face',\n",
       " 'money',\n",
       " 'hit',\n",
       " 'husband',\n",
       " 'dare',\n",
       " 'hit',\n",
       " 'dare',\n",
       " 'rape',\n",
       " 'help',\n",
       " 'rape',\n",
       " 'help',\n",
       " 'help',\n",
       " 'rape',\n",
       " 'rape',\n",
       " 'stop',\n",
       " 'kill',\n",
       " 'hey',\n",
       " 'carnation',\n",
       " 'like',\n",
       " 'carnation',\n",
       " 'mother',\n",
       " 'mother',\n",
       " 'day',\n",
       " 'mom',\n",
       " 'want',\n",
       " 'want',\n",
       " 'money',\n",
       " 'marry',\n",
       " 'kid',\n",
       " 'teen',\n",
       " 'fat',\n",
       " 'barrel',\n",
       " 'smell',\n",
       " 'bad',\n",
       " 'day',\n",
       " 'cook',\n",
       " 'egg',\n",
       " 'breakfast',\n",
       " 'say',\n",
       " 'good',\n",
       " 'week',\n",
       " 'kowloon',\n",
       " 'tea',\n",
       " 'half',\n",
       " 'year',\n",
       " 'husband',\n",
       " 'okay',\n",
       " 'sure',\n",
       " 'find',\n",
       " 'phone',\n",
       " 'truly',\n",
       " 'lie',\n",
       " 'wife',\n",
       " 'lie',\n",
       " 'sure',\n",
       " 'business',\n",
       " 'leave',\n",
       " 'woman',\n",
       " 'know',\n",
       " 'money',\n",
       " 'care',\n",
       " 'let',\n",
       " 'sit',\n",
       " 'rita',\n",
       " 'miss',\n",
       " 'rita',\n",
       " 'night',\n",
       " 'pass',\n",
       " 'house',\n",
       " 'dog',\n",
       " 'scare',\n",
       " ...]"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lucem_illud_2020.normalizeTokens(clean_11[1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Great! Now let us create a Pandas dataframe with movie names, raw words, tokenized words, and so on.\n",
    "The file \"sources_movies.zip\" has this information. Similar information files are found for the other datasets too, in their respective folders."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "zfile = zipfile.ZipFile(corpus_name + \"/sources_movies.zip\")\n",
    "source = []"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "for file in zfile.namelist():\n",
    "    with zfile.open(file) as f:\n",
    "        for line in f:\n",
    "            source.append(line)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[b'textID\\tfileID\\t#words\\tgenre\\tyear\\tlanguage(s)\\tcountry\\timdb\\ttitle\\r\\n',\n",
       " b'-----\\t-----\\t-----\\t-----\\t-----\\t-----\\t-----\\t-----\\t-----\\r\\n',\n",
       " b'\\r\\n',\n",
       " b'290635\\t3547424\\t4722\\tShort, Musical\\t1930\\tUK\\tEnglish\\t0290635\\tGoodbye to All That\\r\\n',\n",
       " b'21165\\t6332374\\t10220\\tCrime, Mystery, Thriller\\t1930\\tUK\\tEnglish\\t0021165\\tMurder!\\r\\n',\n",
       " b'21191\\t6013789\\t5281\\tDrama, Romance\\t1930\\tUSA\\tEnglish\\t0021191\\tA Notorious Affair\\r\\n',\n",
       " b'20620\\t3660608\\t6724\\tBiography, Drama, History\\t1930\\tUSA\\tEnglish\\t0020620\\tAbraham Lincoln\\r\\n',\n",
       " b'20629\\t60053\\t9552\\tDrama, War\\t1930\\tUSA\\tEnglish, French, German, Latin\\t0020629\\tAll Quiet on the Western Front\\r\\n',\n",
       " b'20640\\t6850720\\t13862\\tComedy, Musical\\t1930\\tUSA\\tEnglish\\t0020640\\tAnimal Crackers\\r\\n',\n",
       " b'20641\\t176501\\t11140\\tDrama, Romance\\t1930\\tUSA\\tEnglish\\t0020641\\tAnna Christie\\r\\n',\n",
       " b'20643\\t3603861\\t1748\\tComedy, Short\\t1930\\tUSA\\tEnglish\\t0020643\\tAnother Fine Mess\\r\\n',\n",
       " b'20670\\t4159455\\t6966\\tComedy, Musical\\t1930\\tUSA\\tEnglish\\t0020670\\tBe Yourself!\\r\\n',\n",
       " b'20702\\t3435788\\t6870\\tCrime, Drama, Film-Noir\\t1930\\tUSA\\tEnglish, Italian, French\\t0020702\\tBorn Reckless\\r\\n',\n",
       " b'20768\\t3128644\\t1782\\tDrama, Romance\\t1930\\tUSA\\tEnglish\\t0020768\\tCity Girl\\r\\n',\n",
       " b'20876\\t105788\\t8463\\tComedy\\t1930\\tUSA\\tEnglish\\t0020876\\tFeet First\\r\\n',\n",
       " b'20945\\t6935753\\t8669\\tComedy, Musical\\t1930\\tUSA\\tEnglish, French\\t0020945\\tHalf Shot at Sunrise\\r\\n',\n",
       " b\"20960\\t5583644\\t7331\\tDrama, War\\t1930\\tUSA\\tEnglish, German, French\\t0020960\\tHell's Angels\\r\\n\",\n",
       " b'21025\\t6922578\\t4495\\tAnimation, Music\\t1930\\tUSA\\tEnglish\\t0021025\\tKing of Jazz\\r\\n',\n",
       " b'21040\\t4056770\\t9201\\tDrama, Romance\\t1930\\tUSA\\tEnglish\\t0021040\\tLadies of Leisure\\r\\n',\n",
       " b'22058\\t3445333\\t919\\tShort, Comedy\\t1930\\tUSA\\tEnglish\\t0022058\\tLaughing Gravy\\r\\n']"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "source[0:20]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It looks dirty because the file is encoded as bytes, but we can certainly see the information in here. The file id is also present in the original raw text data: as the first \"word\". Look back at the normalized/tokenized words to confirm that. We're going to use this to create a dataframe with: Fileid, movie name, genre, year, and country.\n",
    "\n",
    "It is advised that you run a similar check of the source file before you do other extraction.\n",
    "\n",
    "First, let us create a dictionary mapping file-id to all the text. Each movie will be mapped to a list of the tokenized words.\n",
    "\n",
    "In this example, I only use it to load 1000 movies. You can comment this out or increase/decrease the number as inspired."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "movie_texts = {}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "for files in movie_raw:\n",
    "    if len(movie_texts) > 1000:\n",
    "        break\n",
    "    movies = clean_raw_text(movie_raw[files][1:])\n",
    "    for movie in movies:\n",
    "        txts = lucem_illud_2020.word_tokenize(movie)\n",
    "        try:\n",
    "            movie_texts[txts[0][2:]] = txts[1:]\n",
    "        except IndexError:\n",
    "            continue"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "movie_df = pd.DataFrame(columns=[\"Movie Name\", \"Genre\", \"Year\", \"Country\", \"Tokenized Texts\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "for movie in source[3:]:\n",
    "    try:\n",
    "        tid, fileid, total_words, genre, year, lang, country, imdb, title = movie.decode(\"utf-8\").split(\"\\t\")\n",
    "    except UnicodeDecodeError:\n",
    "        continue\n",
    "    try:\n",
    "        movie_df.loc[fileid.strip()] = [title.strip(), genre.strip(), year.strip(), country.strip(), movie_texts[fileid.strip()]]\n",
    "    except KeyError:\n",
    "        continue"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Movie Name</th>\n",
       "      <th>Genre</th>\n",
       "      <th>Year</th>\n",
       "      <th>Country</th>\n",
       "      <th>Tokenized Texts</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>6861982</th>\n",
       "      <td>Blonde Crazy</td>\n",
       "      <td>Comedy, Crime, Drama</td>\n",
       "      <td>1931</td>\n",
       "      <td>English</td>\n",
       "      <td>[Who, cares, for, starlit, skies, when, you, '...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6606107</th>\n",
       "      <td>Five and Ten</td>\n",
       "      <td>Drama, Romance</td>\n",
       "      <td>1931</td>\n",
       "      <td>English</td>\n",
       "      <td>[Subtitles, Lu, s, Filipe, Bernardes, Mr, Rari...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6406611</th>\n",
       "      <td>Five Star Final</td>\n",
       "      <td>Crime, Drama</td>\n",
       "      <td>1931</td>\n",
       "      <td>English</td>\n",
       "      <td>[Extra, Extra, Extra, Five, star, final, Indis...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3251135</th>\n",
       "      <td>The Smiling Lieutenant</td>\n",
       "      <td>Comedy, Romance, Musical</td>\n",
       "      <td>1931</td>\n",
       "      <td>English, French</td>\n",
       "      <td>[Bell_Rings, Bell_Rings, Sighing, Yawns, Yes, ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6909562</th>\n",
       "      <td>Faithless</td>\n",
       "      <td>Drama</td>\n",
       "      <td>1932</td>\n",
       "      <td>English</td>\n",
       "      <td>[But, Carol, this, bank, is, your, guardian, W...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                     Movie Name                     Genre  Year  \\\n",
       "6861982            Blonde Crazy      Comedy, Crime, Drama  1931   \n",
       "6606107            Five and Ten            Drama, Romance  1931   \n",
       "6406611         Five Star Final              Crime, Drama  1931   \n",
       "3251135  The Smiling Lieutenant  Comedy, Romance, Musical  1931   \n",
       "6909562               Faithless                     Drama  1932   \n",
       "\n",
       "                 Country                                    Tokenized Texts  \n",
       "6861982          English  [Who, cares, for, starlit, skies, when, you, '...  \n",
       "6606107          English  [Subtitles, Lu, s, Filipe, Bernardes, Mr, Rari...  \n",
       "6406611          English  [Extra, Extra, Extra, Five, star, final, Indis...  \n",
       "3251135  English, French  [Bell_Rings, Bell_Rings, Sighing, Yawns, Yes, ...  \n",
       "6909562          English  [But, Carol, this, bank, is, your, guardian, W...  "
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "movie_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This dataframe contains information of the name, the genre, the year, the country, and the texts associated with it: all sorts of analysis can be run with this information now.\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You are encouraged to try the similar process and load the other datasets."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}