5_Sparknlp_text_summarization_and_question_answering.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "eKpvOyQjjWJY"
   },
   "source": [
    "# Text Summarization & Question Answering using google's T5 Transformer\n",
    "This notebook uses google's pre-trained T5 Transformer to conduct Text Summarization for Reddit selftext (with average length 800 characters) and Question Answering (because many posts' title is a question)\n",
    "\n",
    "### Reference:\n",
    "\n",
    "Spark NLP documentation and instructions: https://nlp.johnsnowlabs.com/docs/en/quickstart\n",
    "\n",
    "Spark NLP Google T5 Article: https://towardsdatascience.com/hands-on-googles-text-to-text-transfer-transformer-t5-with-spark-nlp-6f7db75cecff\n",
    "\n",
    "Spark NLP annotators: https://nlp.johnsnowlabs.com/docs/en/annotators\n",
    "\n",
    "Spark NLP models: https://nlp.johnsnowlabs.com/models\n",
    "\n",
    "Read CSV correctly in pyspark: https://stackoverflow.com/questions/50751687/spark-incorrectly-reading-csv"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "eCIT5VLxS3I1"
   },
   "source": [
    "## 1. Start the Spark session and read data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "khjM-z9ORFU3"
   },
   "source": [
    "Import dependencies and start Spark session."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "id": "sw-t1zxlHTB7"
   },
   "outputs": [],
   "source": [
    "import json\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "\n",
    "from pyspark.ml import Pipeline\n",
    "from pyspark.sql import SparkSession\n",
    "import pyspark.sql.functions as F\n",
    "from sparknlp.annotator import *\n",
    "from sparknlp.base import *\n",
    "import sparknlp\n",
    "from sparknlp.pretrained import PretrainedPipeline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "id": "SGz1Z4s6hDvC"
   },
   "outputs": [],
   "source": [
    "# need to set sep and escape parameters\n",
    "auto = spark.read.csv('Auto.csv', header=True, sep=',', escape=\"\\\"\", multiLine=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "gTMhYNs4hOZV",
    "outputId": "88a7e388-9a7d-4d84-9742-33669456467d"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+--------------------+--------------------+--------------------+--------------------+--------------------+-----+-----------+\n",
      "|               title|              author|link_flair_css_class|            selftext|                 url|score|created_utc|\n",
      "+--------------------+--------------------+--------------------+--------------------+--------------------+-----+-----------+\n",
      "|Trade in car or k...|  crosstitchchampion|                Auto|So we have a Jeep...|https://www.reddi...|    1| 1584143188|\n",
      "|Wanting to buy fi...|      victorriiaaaaa|                Auto|I'm not sure whic...|https://www.reddi...|    1| 1584140907|\n",
      "|Please help! Youn...|         shanerobles|                Auto|hello, i’m 18 yea...|https://www.reddi...|    1| 1584138407|\n",
      "|Do I let them rep...|      username910975|                Auto|23F. Bought my ca...|https://www.reddi...|    1| 1584136256|\n",
      "|Car Dealership Ne...|             Rewbies|                Auto|Last month I purc...|https://www.reddi...|    1| 1584135468|\n",
      "|Ford Motor Credit...|      ifavordoomsday|                Auto|My apologies if t...|https://www.reddi...|    1| 1584134570|\n",
      "|Finance new car o...|             _dayday|                Auto|Life hit extremel...|https://www.reddi...|    1| 1582564951|\n",
      "|How much did I ov...|         superman_ru|                Auto|I bought a used c...|https://www.reddi...|    1| 1579420625|\n",
      "|Car wasn’t on aut...| ApatheticEnthusiast|                Auto|I am so embarrass...|https://www.reddi...|    1| 1579414587|\n",
      "|Collision claim f...|      BlueLongIsland|                Auto|I have a complica...|https://www.reddi...|    1| 1579403199|\n",
      "|How much should I...|bringbackradarto4077|                Auto|Hi all, my credit...|https://www.reddi...|    1| 1579401035|\n",
      "|Accedently fraude...|             onlyboi|                Auto|So I recently got...|https://www.reddi...|    1| 1580993618|\n",
      "|Trading in a car ...|        IIIumination|                Auto|I need some advic...|https://www.reddi...|    1| 1580987818|\n",
      "|Good or bad deal ...|    throw_away484848|                Auto|As title states, ...|https://www.reddi...|    1| 1585713780|\n",
      "|Should I take my ...|           Muskrat21|                Auto|backstory: Bought...|https://www.reddi...|    2| 1585706500|\n",
      "|TX - Jackass Rear...|           BobbyZ123|                Auto|Guy rear ended me...|https://www.reddi...|    1| 1585705496|\n",
      "|Pay off stuff or ...|            timmy911|                Auto|I’m pretty poor w...|https://www.reddi...|    1| 1587283763|\n",
      "|Have any of you t...|            maiden69|                Auto|I'm attempting to...|https://www.reddi...|    1| 1587276092|\n",
      "|Need help decidin...|           landonmeh|                Auto|I have decided th...|https://www.reddi...|    1| 1587275505|\n",
      "|In desperate need...|            maiden69|                Auto|I bought a new ca...|https://www.reddi...|    1| 1587274038|\n",
      "+--------------------+--------------------+--------------------+--------------------+--------------------+-----+-----------+\n",
      "only showing top 20 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "auto.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "TRhBjhevhOyq",
    "outputId": "571fef02-051b-4ffe-a275-f41c98e02924"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "7324"
      ]
     },
     "execution_count": 7,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "auto.count()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "Y-Ms-nE3zNo4",
    "outputId": "c13ee3be-dd9b-4f54-dcf8-0ca2431c6620"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "root\n",
      " |-- title: string (nullable = true)\n",
      " |-- author: string (nullable = true)\n",
      " |-- link_flair_css_class: string (nullable = true)\n",
      " |-- selftext: string (nullable = true)\n",
      " |-- url: string (nullable = true)\n",
      " |-- score: string (nullable = true)\n",
      " |-- created_utc: string (nullable = true)\n",
      "\n"
     ]
    }
   ],
   "source": [
    "auto.printSchema()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "9RgiqfX5XDqb"
   },
   "source": [
    "## 2. Select the Deep Learning model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "TPqBOgDhnvjB"
   },
   "source": [
    "For complete model list: \n",
    "https://nlp.johnsnowlabs.com/models\n",
    "\n",
    "For `T5` models:\n",
    "https://nlp.johnsnowlabs.com/models?tag=t5"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "jQACwlw5dJT6"
   },
   "source": [
    "## 3. Text Summaization using T5 Transformer"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "XftYgju4XOw_"
   },
   "source": [
    " Define Spark NLP pipeline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "lBggF5P8J1gc",
    "outputId": "e9e7c14b-8aa7-41fb-88d1-26a3966c683b"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "t5_small download started this may take some time.\n",
      "Approximate size to download 139 MB\n",
      "[OK!]\n"
     ]
    }
   ],
   "source": [
    "document_assembler = DocumentAssembler()\\\n",
    ".setInputCol(\"selftext\")\\\n",
    ".setOutputCol(\"documents\")\n",
    "\n",
    "t5 = T5Transformer() \\\n",
    "  .pretrained(\"t5_small\", 'en') \\\n",
    "  .setTask(\"summarize:\")\\\n",
    "  .setMaxOutputLength(100)\\\n",
    "  .setInputCols([\"documents\"]) \\\n",
    "  .setOutputCol(\"summaries\")\n",
    "\n",
    "summarizer_pp = Pipeline(stages=[\n",
    "    document_assembler, t5\n",
    "])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "mv0abcwhXWC-"
   },
   "source": [
    "Run the pipeline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "id": "EYf_9sXDXR4t"
   },
   "outputs": [],
   "source": [
    "empty_df = spark.createDataFrame([['']]).toDF('selftext')\n",
    "pipeline_model = summarizer_pp.fit(empty_df)\n",
    "sum_lmodel = LightPipeline(pipeline_model)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "id": "sMMOt8rNZNqA"
   },
   "outputs": [],
   "source": [
    "result = sum_lmodel.transform(auto)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "JpQdz7nfcAFJ",
    "outputId": "84321569-c0eb-4003-a99f-5fb646b1b971"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+\n",
      "|summaries                                                                                                                                                                                                                                                          |\n",
      "+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+\n",
      "|we have a Jeep Grand Cherokee and we average about 20MPG . we owe about 17.5K on it and our payments are about 420 a month . we talked about trading it in on a smaller car to get substantial fuel gains .                                                        |\n",
      "|i'm in desperate need of a first car asap . i have no credit and i'm not sure if it affects anything . i'm worried about going to a dealership and being sold a piece of junk .                                                                                    |\n",
      "|i'm 18 and trying to by a car entirely by myself . i make 784 a month and am without bills . i have no debt, i don’t pay rent, and on top of everything i get a military allowance of 1248 a month .                                                               |\n",
      "|owes 8k on 2015 kia for 6 years . owes 10k on my 2015 .                                                                                                                                                                                                            |\n",
      "|the extended warranty company sends me the cancellation contract, I fill it out, and send it to the dealership via email . the manager there states the person I worked with has moved to Florida .                                                                |\n",
      "|a couple weeks ago I initiated a lease transfer for a 2018 Ford Fusion from my name into my grandmother's name . this was due to me needing a bigger vehicle and my grandmother needing a car . now, I have two leases in my name, my new vehicle and this Fusion .|\n",
      "|life hit extremely hard the past few years . ended up about 45k in debt between myself and the wife .                                                                                                                                                              |\n",
      "|a used preowned car was listed online for $17,595 . it was listed online for $17,595 . the car was listed online for $17,595 .                                                                                                                                     |\n",
      "|a moron has a loan company that has a filter against unknown phone numbers . a moron has a loan company that has a filter against unknown phone numbers .                                                                                                          |\n",
      "|insurance company has no collision coverage on the car . a check goes out to my grandfather's name . the car is titled and registered under my grandfather's name .                                                                                                |\n",
      "|my plan is to pay back my mom first for some school related expenses . plan is to save up for a down payment on a car, like $3k . plan to save up for a down payment on a car, like $3k .                                                                          |\n",
      "|a dealership called me back and told them they will try to find another lender . they said they got one for 14k but I'll have to cash the rest of it . the dealership said they got one for 14k but they said they got one for 14k .                               |\n",
      "|in September I bought a sports car for $18,000 with 70k miles on it . the car now has $13,000 left on the loan with 81k miles .                                                                                                                                    |\n",
      "|trade in 2016 sentra with 71.5k miles and  ⁇ $7,100 left on loan for 2020 Rogue . dealer pays off current loan .                                                                                                                                                   |\n",
      "|i made my first payment last week . i have the option to defer 3 months . i have more that 6 months saved up of expenses .                                                                                                                                         |\n",
      "|insurance company has already told me the car is totaled . i currently owe $11432 on the vehicle, more than the value of the car . i have virtually no savings for a down payment for another car .                                                                |\n",
      "|a 2011 truck that’s always costing me an arm and a leg is nearly paid off . the only other debt besides mortgage, medical and student loans is some debt from buying furniture .                                                                                   |\n",
      "|i'm attempting to refinance my current car and they've given me the lowest interest rate . I don't know how creditable they are .                                                                                                                                  |\n",
      "|mk 7.5 VW Golf R is a mk 7.5 car with 350k km . avg costs $700/month . avg costs as low as 500 and high as $1000 .                                                                                                                                                 |\n",
      "|my car note is $490 on a 73-month payment plan . I can afford the car note but just wanna pay it off faster and cheaper .                                                                                                                                          |\n",
      "+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+\n",
      "only showing top 20 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "result.select(F.explode(F.arrays_zip('summaries.result'\n",
    "                                )).alias('cols'))\\\n",
    "      .select(F.expr(\"cols['0']\").alias('summaries'),\n",
    "      )\\\n",
    "      .show(truncate=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "czL7-z7Wb_xt",
    "outputId": "dbd6a7c9-4372-4a8a-ab3d-a31ee0a1fe21"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+\n",
      "|summaries                                                                                                                                                                                                                                                          |\n",
      "+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+\n",
      "|we have a Jeep Grand Cherokee and we average about 20MPG . we owe about 17.5K on it and our payments are about 420 a month . we talked about trading it in on a smaller car to get substantial fuel gains .                                                        |\n",
      "|i'm in desperate need of a first car asap . i have no credit and i'm not sure if it affects anything . i'm worried about going to a dealership and being sold a piece of junk .                                                                                    |\n",
      "|i'm 18 and trying to by a car entirely by myself . i make 784 a month and am without bills . i have no debt, i don’t pay rent, and on top of everything i get a military allowance of 1248 a month .                                                               |\n",
      "|owes 8k on 2015 kia for 6 years . owes 10k on my 2015 .                                                                                                                                                                                                            |\n",
      "|the extended warranty company sends me the cancellation contract, I fill it out, and send it to the dealership via email . the manager there states the person I worked with has moved to Florida .                                                                |\n",
      "|a couple weeks ago I initiated a lease transfer for a 2018 Ford Fusion from my name into my grandmother's name . this was due to me needing a bigger vehicle and my grandmother needing a car . now, I have two leases in my name, my new vehicle and this Fusion .|\n",
      "|life hit extremely hard the past few years . ended up about 45k in debt between myself and the wife .                                                                                                                                                              |\n",
      "|a used preowned car was listed online for $17,595 . it was listed online for $17,595 . the car was listed online for $17,595 .                                                                                                                                     |\n",
      "|a moron has a loan company that has a filter against unknown phone numbers . a moron has a loan company that has a filter against unknown phone numbers .                                                                                                          |\n",
      "|insurance company has no collision coverage on the car . a check goes out to my grandfather's name . the car is titled and registered under my grandfather's name .                                                                                                |\n",
      "|my plan is to pay back my mom first for some school related expenses . plan is to save up for a down payment on a car, like $3k . plan to save up for a down payment on a car, like $3k .                                                                          |\n",
      "|a dealership called me back and told them they will try to find another lender . they said they got one for 14k but I'll have to cash the rest of it . the dealership said they got one for 14k but they said they got one for 14k .                               |\n",
      "|in September I bought a sports car for $18,000 with 70k miles on it . the car now has $13,000 left on the loan with 81k miles .                                                                                                                                    |\n",
      "|trade in 2016 sentra with 71.5k miles and  ⁇ $7,100 left on loan for 2020 Rogue . dealer pays off current loan .                                                                                                                                                   |\n",
      "|i made my first payment last week . i have the option to defer 3 months . i have more that 6 months saved up of expenses .                                                                                                                                         |\n",
      "|insurance company has already told me the car is totaled . i currently owe $11432 on the vehicle, more than the value of the car . i have virtually no savings for a down payment for another car .                                                                |\n",
      "|a 2011 truck that’s always costing me an arm and a leg is nearly paid off . the only other debt besides mortgage, medical and student loans is some debt from buying furniture .                                                                                   |\n",
      "|i'm attempting to refinance my current car and they've given me the lowest interest rate . I don't know how creditable they are .                                                                                                                                  |\n",
      "|mk 7.5 VW Golf R is a mk 7.5 car with 350k km . avg costs $700/month . avg costs as low as 500 and high as $1000 .                                                                                                                                                 |\n",
      "|my car note is $490 on a 73-month payment plan . I can afford the car note but just wanna pay it off faster and cheaper .                                                                                                                                          |\n",
      "+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+\n",
      "only showing top 20 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "summary_cols = result.select(F.explode(F.arrays_zip('summaries.result'\n",
    "                                )).alias('cols'))\\\n",
    "      .select(F.expr(\"cols['0']\").alias('summaries'),\n",
    "      )\n",
    "summary_cols.show(truncate=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "2QPs_I1JeaFV"
   },
   "source": [
    "## 4. Question Answering using T5 Transformer"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Oup7_-KNefgz"
   },
   "source": [
    " Define Spark NLP pipeline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "vmIBAkUWeiAO",
    "outputId": "75da34df-2630-42a3-f799-9fdb2925feb7"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "sentence_detector_dl download started this may take some time.\n",
      "Approximate size to download 354.6 KB\n",
      "[OK!]\n",
      "google_t5_small_ssm_nq download started this may take some time.\n",
      "Approximate size to download 139 MB\n",
      "[OK!]\n"
     ]
    }
   ],
   "source": [
    "document_assembler = DocumentAssembler() \\\n",
    "    .setInputCol(\"text\") \\\n",
    "    .setOutputCol(\"documents\")\n",
    "\n",
    "sentence_detector = SentenceDetectorDLModel\\\n",
    "    .pretrained(\"sentence_detector_dl\", \"en\")\\\n",
    "    .setInputCols([\"documents\"])\\\n",
    "    .setOutputCol(\"questions\")\n",
    "\n",
    "t5 = T5Transformer()\\\n",
    "    .pretrained(\"google_t5_small_ssm_nq\", 'en')\\\n",
    "    .setInputCols([\"questions\"])\\\n",
    "    .setOutputCol(\"answers\")\\\n",
    "\n",
    "qa_pp = Pipeline(stages=[\n",
    "    document_assembler, sentence_detector, t5\n",
    "])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "mBRwmr-lfB6T"
   },
   "source": [
    "Run the pipeline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "Pry1KLQExlpG",
    "outputId": "3a52648c-5184-45d2-e758-fe1a63658858"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Question: Do student loans or credit card debt take precedence?\n",
      "Answer:\t Over one million students\n",
      "Question: How does Wageworks work?\n",
      "Answer:\t sales of ten million units\n",
      "Question: What to ask for when buying a used car?\n",
      "Answer:\t car gage\n",
      "Question: How fast does your credit score actually update?\n",
      "Answer:\t until you are interrogated\n",
      "Question: What happens with unused credit card accounts?\n",
      "Answer:\t bankruptcy\n"
     ]
    }
   ],
   "source": [
    "empty_df = spark.createDataFrame([['']]).toDF('text')\n",
    "pipeline_model = qa_pp.fit(empty_df)\n",
    "qa_lmodel = LightPipeline(pipeline_model)\n",
    "\n",
    "questions = [\"Do student loans or credit card debt take precedence?\",\n",
    "             \"How does Wageworks work?\",\n",
    "             \"What to ask for when buying a used car?\",\n",
    "             \"How fast does your credit score actually update?\",\n",
    "             \"What happens with unused credit card accounts?\"\n",
    "]\n",
    "\n",
    "res = qa_lmodel.fullAnnotate(questions)\n",
    "\n",
    "\n",
    "for i, r in enumerate(res):\n",
    "    print (\"Question:\", questions[i])\n",
    "    for sent in r['answers']:\n",
    "        print ('Answer:\\t', sent.result)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "id": "oqefWQpU0FjR"
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "colab": {
   "collapsed_sections": [],
   "name": "T5TRANSFORMER-blackhumorQ&A.ipynb",
   "provenance": [],
   "toc_visible": true
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}