1_Data_Exploration_pt-br.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Dados de texto de plágio\n",
    "\n",
    "Neste projeto, você terá a tarefa de construir um detector de plágio que examina um arquivo de texto e executa a classificação binária; rotular esse arquivo como plagiado ou não, dependendo de quão semelhante o arquivo de texto é quando comparado a um texto fonte fornecido.\n",
    "\n",
    "A primeira etapa para trabalhar com qualquer conjunto de dados é carregar os dados e observar quais informações estão incluídas no conjunto de dados. Essa é uma etapa importante para, eventualmente, trabalhar com esses dados e saber com quais tipos de recursos você deve trabalhar ao transformar e agrupar os dados!\n",
    "\n",
    "Portanto, este bloco de notas trata de explorar os dados e observar padrões sobre os recursos que você recebe e a distribuição dos dados.\n",
    "\n",
    "> Não há exercícios ou perguntas neste caderno, ele serve apenas para exploração. Este bloco de notas será necessário no envio do projeto final.\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Leia nos dados\n",
    "\n",
    "A célula abaixo fará o download dos dados necessários e extrairá os arquivos para a pasta `data /`.\n",
    "\n",
    "Esses dados são uma versão ligeiramente modificada de um conjunto de dados criado por Paul Clough (Estudos de Informação) e Mark Stevenson (Ciência da Computação), na Universidade de Sheffield. Você pode ler tudo sobre a coleta de dados e corpus, em [sua página da universidade](https://ir.shef.ac.uk/cloughie/resources/plagiarism_corpus.html).\n",
    "\n",
    "> **Citação para dados**: Clough, P. e Stevenson, M. Developing A Corpus of Plagiarized Short Answers, Language Resources and Evaluation: Special Issue on Plagiarism and Authorship Analysis, In Press.[Download]\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "--2020-10-13 22:30:18--  https://s3.amazonaws.com/video.udacity-data.com/topher/2019/January/5c4147f9_data/data.zip\n",
      "Resolving s3.amazonaws.com (s3.amazonaws.com)... 52.216.152.150\n",
      "Connecting to s3.amazonaws.com (s3.amazonaws.com)|52.216.152.150|:443... connected.\n",
      "HTTP request sent, awaiting response... 200 OK\n",
      "Length: 113826 (111K) [application/zip]\n",
      "Saving to: ‘data.zip’\n",
      "\n",
      "data.zip            100%[===================>] 111.16K   478KB/s    in 0.2s    \n",
      "\n",
      "2020-10-13 22:30:19 (478 KB/s) - ‘data.zip’ saved [113826/113826]\n",
      "\n",
      "Archive:  data.zip\n",
      "   creating: data/\n",
      "  inflating: data/.DS_Store          \n",
      "   creating: __MACOSX/\n",
      "   creating: __MACOSX/data/\n",
      "  inflating: __MACOSX/data/._.DS_Store  \n",
      "  inflating: data/file_information.csv  \n",
      "  inflating: __MACOSX/data/._file_information.csv  \n",
      "  inflating: data/g0pA_taska.txt     \n",
      "  inflating: __MACOSX/data/._g0pA_taska.txt  \n",
      "  inflating: data/g0pA_taskb.txt     \n",
      "  inflating: __MACOSX/data/._g0pA_taskb.txt  \n",
      "  inflating: data/g0pA_taskc.txt     \n",
      "  inflating: __MACOSX/data/._g0pA_taskc.txt  \n",
      "  inflating: data/g0pA_taskd.txt     \n",
      "  inflating: __MACOSX/data/._g0pA_taskd.txt  \n",
      "  inflating: data/g0pA_taske.txt     \n",
      "  inflating: __MACOSX/data/._g0pA_taske.txt  \n",
      "  inflating: data/g0pB_taska.txt     \n",
      "  inflating: __MACOSX/data/._g0pB_taska.txt  \n",
      "  inflating: data/g0pB_taskb.txt     \n",
      "  inflating: __MACOSX/data/._g0pB_taskb.txt  \n",
      "  inflating: data/g0pB_taskc.txt     \n",
      "  inflating: __MACOSX/data/._g0pB_taskc.txt  \n",
      "  inflating: data/g0pB_taskd.txt     \n",
      "  inflating: __MACOSX/data/._g0pB_taskd.txt  \n",
      "  inflating: data/g0pB_taske.txt     \n",
      "  inflating: __MACOSX/data/._g0pB_taske.txt  \n",
      "  inflating: data/g0pC_taska.txt     \n",
      "  inflating: __MACOSX/data/._g0pC_taska.txt  \n",
      "  inflating: data/g0pC_taskb.txt     \n",
      "  inflating: __MACOSX/data/._g0pC_taskb.txt  \n",
      "  inflating: data/g0pC_taskc.txt     \n",
      "  inflating: __MACOSX/data/._g0pC_taskc.txt  \n",
      "  inflating: data/g0pC_taskd.txt     \n",
      "  inflating: __MACOSX/data/._g0pC_taskd.txt  \n",
      "  inflating: data/g0pC_taske.txt     \n",
      "  inflating: __MACOSX/data/._g0pC_taske.txt  \n",
      "  inflating: data/g0pD_taska.txt     \n",
      "  inflating: __MACOSX/data/._g0pD_taska.txt  \n",
      "  inflating: data/g0pD_taskb.txt     \n",
      "  inflating: __MACOSX/data/._g0pD_taskb.txt  \n",
      "  inflating: data/g0pD_taskc.txt     \n",
      "  inflating: __MACOSX/data/._g0pD_taskc.txt  \n",
      "  inflating: data/g0pD_taskd.txt     \n",
      "  inflating: __MACOSX/data/._g0pD_taskd.txt  \n",
      "  inflating: data/g0pD_taske.txt     \n",
      "  inflating: __MACOSX/data/._g0pD_taske.txt  \n",
      "  inflating: data/g0pE_taska.txt     \n",
      "  inflating: __MACOSX/data/._g0pE_taska.txt  \n",
      "  inflating: data/g0pE_taskb.txt     \n",
      "  inflating: __MACOSX/data/._g0pE_taskb.txt  \n",
      "  inflating: data/g0pE_taskc.txt     \n",
      "  inflating: __MACOSX/data/._g0pE_taskc.txt  \n",
      "  inflating: data/g0pE_taskd.txt     \n",
      "  inflating: __MACOSX/data/._g0pE_taskd.txt  \n",
      "  inflating: data/g0pE_taske.txt     \n",
      "  inflating: __MACOSX/data/._g0pE_taske.txt  \n",
      "  inflating: data/g1pA_taska.txt     \n",
      "  inflating: __MACOSX/data/._g1pA_taska.txt  \n",
      "  inflating: data/g1pA_taskb.txt     \n",
      "  inflating: __MACOSX/data/._g1pA_taskb.txt  \n",
      "  inflating: data/g1pA_taskc.txt     \n",
      "  inflating: __MACOSX/data/._g1pA_taskc.txt  \n",
      "  inflating: data/g1pA_taskd.txt     \n",
      "  inflating: __MACOSX/data/._g1pA_taskd.txt  \n",
      "  inflating: data/g1pA_taske.txt     \n",
      "  inflating: __MACOSX/data/._g1pA_taske.txt  \n",
      "  inflating: data/g1pB_taska.txt     \n",
      "  inflating: __MACOSX/data/._g1pB_taska.txt  \n",
      "  inflating: data/g1pB_taskb.txt     \n",
      "  inflating: __MACOSX/data/._g1pB_taskb.txt  \n",
      "  inflating: data/g1pB_taskc.txt     \n",
      "  inflating: __MACOSX/data/._g1pB_taskc.txt  \n",
      "  inflating: data/g1pB_taskd.txt     \n",
      "  inflating: __MACOSX/data/._g1pB_taskd.txt  \n",
      "  inflating: data/g1pB_taske.txt     \n",
      "  inflating: __MACOSX/data/._g1pB_taske.txt  \n",
      "  inflating: data/g1pD_taska.txt     \n",
      "  inflating: __MACOSX/data/._g1pD_taska.txt  \n",
      "  inflating: data/g1pD_taskb.txt     \n",
      "  inflating: __MACOSX/data/._g1pD_taskb.txt  \n",
      "  inflating: data/g1pD_taskc.txt     \n",
      "  inflating: __MACOSX/data/._g1pD_taskc.txt  \n",
      "  inflating: data/g1pD_taskd.txt     \n",
      "  inflating: __MACOSX/data/._g1pD_taskd.txt  \n",
      "  inflating: data/g1pD_taske.txt     \n",
      "  inflating: __MACOSX/data/._g1pD_taske.txt  \n",
      "  inflating: data/g2pA_taska.txt     \n",
      "  inflating: __MACOSX/data/._g2pA_taska.txt  \n",
      "  inflating: data/g2pA_taskb.txt     \n",
      "  inflating: __MACOSX/data/._g2pA_taskb.txt  \n",
      "  inflating: data/g2pA_taskc.txt     \n",
      "  inflating: __MACOSX/data/._g2pA_taskc.txt  \n",
      "  inflating: data/g2pA_taskd.txt     \n",
      "  inflating: __MACOSX/data/._g2pA_taskd.txt  \n",
      "  inflating: data/g2pA_taske.txt     \n",
      "  inflating: __MACOSX/data/._g2pA_taske.txt  \n",
      "  inflating: data/g2pB_taska.txt     \n",
      "  inflating: __MACOSX/data/._g2pB_taska.txt  \n",
      "  inflating: data/g2pB_taskb.txt     \n",
      "  inflating: __MACOSX/data/._g2pB_taskb.txt  \n",
      "  inflating: data/g2pB_taskc.txt     \n",
      "  inflating: __MACOSX/data/._g2pB_taskc.txt  \n",
      "  inflating: data/g2pB_taskd.txt     \n",
      "  inflating: __MACOSX/data/._g2pB_taskd.txt  \n",
      "  inflating: data/g2pB_taske.txt     \n",
      "  inflating: __MACOSX/data/._g2pB_taske.txt  \n",
      "  inflating: data/g2pC_taska.txt     \n",
      "  inflating: __MACOSX/data/._g2pC_taska.txt  \n",
      "  inflating: data/g2pC_taskb.txt     \n",
      "  inflating: __MACOSX/data/._g2pC_taskb.txt  \n",
      "  inflating: data/g2pC_taskc.txt     \n",
      "  inflating: __MACOSX/data/._g2pC_taskc.txt  \n",
      "  inflating: data/g2pC_taskd.txt     \n",
      "  inflating: __MACOSX/data/._g2pC_taskd.txt  \n",
      "  inflating: data/g2pC_taske.txt     \n",
      "  inflating: __MACOSX/data/._g2pC_taske.txt  \n",
      "  inflating: data/g2pE_taska.txt     \n",
      "  inflating: __MACOSX/data/._g2pE_taska.txt  \n",
      "  inflating: data/g2pE_taskb.txt     \n",
      "  inflating: __MACOSX/data/._g2pE_taskb.txt  \n",
      "  inflating: data/g2pE_taskc.txt     \n",
      "  inflating: __MACOSX/data/._g2pE_taskc.txt  \n",
      "  inflating: data/g2pE_taskd.txt     \n",
      "  inflating: __MACOSX/data/._g2pE_taskd.txt  \n",
      "  inflating: data/g2pE_taske.txt     \n",
      "  inflating: __MACOSX/data/._g2pE_taske.txt  \n",
      "  inflating: data/g3pA_taska.txt     \n",
      "  inflating: __MACOSX/data/._g3pA_taska.txt  \n",
      "  inflating: data/g3pA_taskb.txt     \n",
      "  inflating: __MACOSX/data/._g3pA_taskb.txt  \n",
      "  inflating: data/g3pA_taskc.txt     \n",
      "  inflating: __MACOSX/data/._g3pA_taskc.txt  \n",
      "  inflating: data/g3pA_taskd.txt     \n",
      "  inflating: __MACOSX/data/._g3pA_taskd.txt  \n",
      "  inflating: data/g3pA_taske.txt     \n",
      "  inflating: __MACOSX/data/._g3pA_taske.txt  \n",
      "  inflating: data/g3pB_taska.txt     \n",
      "  inflating: __MACOSX/data/._g3pB_taska.txt  \n",
      "  inflating: data/g3pB_taskb.txt     \n",
      "  inflating: __MACOSX/data/._g3pB_taskb.txt  \n",
      "  inflating: data/g3pB_taskc.txt     \n",
      "  inflating: __MACOSX/data/._g3pB_taskc.txt  \n",
      "  inflating: data/g3pB_taskd.txt     \n",
      "  inflating: __MACOSX/data/._g3pB_taskd.txt  \n",
      "  inflating: data/g3pB_taske.txt     \n",
      "  inflating: __MACOSX/data/._g3pB_taske.txt  \n",
      "  inflating: data/g3pC_taska.txt     \n",
      "  inflating: __MACOSX/data/._g3pC_taska.txt  \n",
      "  inflating: data/g3pC_taskb.txt     \n",
      "  inflating: __MACOSX/data/._g3pC_taskb.txt  \n",
      "  inflating: data/g3pC_taskc.txt     \n",
      "  inflating: __MACOSX/data/._g3pC_taskc.txt  \n",
      "  inflating: data/g3pC_taskd.txt     \n",
      "  inflating: __MACOSX/data/._g3pC_taskd.txt  \n",
      "  inflating: data/g3pC_taske.txt     \n",
      "  inflating: __MACOSX/data/._g3pC_taske.txt  \n",
      "  inflating: data/g4pB_taska.txt     \n",
      "  inflating: __MACOSX/data/._g4pB_taska.txt  \n",
      "  inflating: data/g4pB_taskb.txt     \n",
      "  inflating: __MACOSX/data/._g4pB_taskb.txt  \n",
      "  inflating: data/g4pB_taskc.txt     \n",
      "  inflating: __MACOSX/data/._g4pB_taskc.txt  \n",
      "  inflating: data/g4pB_taskd.txt     \n",
      "  inflating: __MACOSX/data/._g4pB_taskd.txt  \n",
      "  inflating: data/g4pB_taske.txt     \n",
      "  inflating: __MACOSX/data/._g4pB_taske.txt  \n",
      "  inflating: data/g4pC_taska.txt     \n",
      "  inflating: __MACOSX/data/._g4pC_taska.txt  \n",
      "  inflating: data/g4pC_taskb.txt     \n",
      "  inflating: __MACOSX/data/._g4pC_taskb.txt  \n",
      "  inflating: data/g4pC_taskc.txt     \n",
      "  inflating: __MACOSX/data/._g4pC_taskc.txt  \n",
      "  inflating: data/g4pC_taskd.txt     \n",
      "  inflating: __MACOSX/data/._g4pC_taskd.txt  \n",
      "  inflating: data/g4pC_taske.txt     \n",
      "  inflating: __MACOSX/data/._g4pC_taske.txt  \n",
      "  inflating: data/g4pD_taska.txt     \n",
      "  inflating: __MACOSX/data/._g4pD_taska.txt  \n",
      "  inflating: data/g4pD_taskb.txt     \n",
      "  inflating: __MACOSX/data/._g4pD_taskb.txt  \n",
      "  inflating: data/g4pD_taskc.txt     \n",
      "  inflating: __MACOSX/data/._g4pD_taskc.txt  \n",
      "  inflating: data/g4pD_taskd.txt     \n",
      "  inflating: __MACOSX/data/._g4pD_taskd.txt  \n",
      "  inflating: data/g4pD_taske.txt     \n",
      "  inflating: __MACOSX/data/._g4pD_taske.txt  \n",
      "  inflating: data/g4pE_taska.txt     \n",
      "  inflating: __MACOSX/data/._g4pE_taska.txt  \n",
      "  inflating: data/g4pE_taskb.txt     \n",
      "  inflating: __MACOSX/data/._g4pE_taskb.txt  \n",
      "  inflating: data/g4pE_taskc.txt     \n",
      "  inflating: __MACOSX/data/._g4pE_taskc.txt  \n",
      "  inflating: data/g4pE_taskd.txt     \n",
      "  inflating: __MACOSX/data/._g4pE_taskd.txt  \n",
      "  inflating: data/g4pE_taske.txt     \n",
      "  inflating: __MACOSX/data/._g4pE_taske.txt  \n",
      "  inflating: data/orig_taska.txt     \n",
      "  inflating: __MACOSX/data/._orig_taska.txt  \n",
      "  inflating: data/orig_taskb.txt     \n",
      "  inflating: data/orig_taskc.txt     \n",
      "  inflating: __MACOSX/data/._orig_taskc.txt  \n",
      "  inflating: data/orig_taskd.txt     \n",
      "  inflating: __MACOSX/data/._orig_taskd.txt  \n",
      "  inflating: data/orig_taske.txt     \n",
      "  inflating: __MACOSX/data/._orig_taske.txt  \n",
      "  inflating: data/test_info.csv      \n",
      "  inflating: __MACOSX/data/._test_info.csv  \n",
      "  inflating: __MACOSX/._data         \n"
     ]
    }
   ],
   "source": [
    "!wget https://s3.amazonaws.com/video.udacity-data.com/topher/2019/January/5c4147f9_data/data.zip\n",
    "!unzip data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# import libraries\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "import os"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Este conjunto de dados de plágio é composto de vários arquivos de texto; cada um desses arquivos tem características que são resumidas em um arquivo `.csv` chamado` file_information.csv`, que podemos ler usando `pandas`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>File</th>\n",
       "      <th>Task</th>\n",
       "      <th>Category</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>g0pA_taska.txt</td>\n",
       "      <td>a</td>\n",
       "      <td>non</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>g0pA_taskb.txt</td>\n",
       "      <td>b</td>\n",
       "      <td>cut</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>g0pA_taskc.txt</td>\n",
       "      <td>c</td>\n",
       "      <td>light</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>g0pA_taskd.txt</td>\n",
       "      <td>d</td>\n",
       "      <td>heavy</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>g0pA_taske.txt</td>\n",
       "      <td>e</td>\n",
       "      <td>non</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>g0pB_taska.txt</td>\n",
       "      <td>a</td>\n",
       "      <td>non</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>g0pB_taskb.txt</td>\n",
       "      <td>b</td>\n",
       "      <td>non</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>g0pB_taskc.txt</td>\n",
       "      <td>c</td>\n",
       "      <td>cut</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>g0pB_taskd.txt</td>\n",
       "      <td>d</td>\n",
       "      <td>light</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>g0pB_taske.txt</td>\n",
       "      <td>e</td>\n",
       "      <td>heavy</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "             File Task Category\n",
       "0  g0pA_taska.txt    a      non\n",
       "1  g0pA_taskb.txt    b      cut\n",
       "2  g0pA_taskc.txt    c    light\n",
       "3  g0pA_taskd.txt    d    heavy\n",
       "4  g0pA_taske.txt    e      non\n",
       "5  g0pB_taska.txt    a      non\n",
       "6  g0pB_taskb.txt    b      non\n",
       "7  g0pB_taskc.txt    c      cut\n",
       "8  g0pB_taskd.txt    d    light\n",
       "9  g0pB_taske.txt    e    heavy"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "csv_file = 'data/file_information.csv'\n",
    "plagiarism_df = pd.read_csv(csv_file)\n",
    "\n",
    "# print out the first few rows of data info\n",
    "plagiarism_df.head(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Tipos de plágio\n",
    "\n",
    "Cada arquivo de texto está associado a uma **Task** (tarefa A-E) e uma **Categoria** de plágio, que você pode ver no DataFrame acima.\n",
    "\n",
    "### Cinco tipos de tarefas, A-E\n",
    "\n",
    "Cada arquivo de texto contém uma resposta a uma pergunta curta; essas perguntas são rotuladas como tarefas A-E.\n",
    "* Cada tarefa, A-E, é sobre um tópico que pode ser incluído no currículo de Ciência da Computação criado pelos autores deste conjunto de dados.\n",
    "    * Por exemplo, a Tarefa A faz a pergunta: \"O que é herança na programação orientada a objetos?\"\n",
    "\n",
    "### Quatro categorias de plágio\n",
    "\n",
    "Cada arquivo de texto tem um rótulo / categoria de plágio associado:\n",
    "\n",
    "1. `cut`: uma resposta é plagiada; é copiado e colado diretamente do texto-fonte relevante da Wikipedia.\n",
    "2. `light`: uma resposta é plagiada; é baseado no texto fonte da Wikipedia e inclui algumas cópias e parafraseando.\n",
    "3. `heavy`: uma resposta é plagiada; é baseado no texto fonte da Wikipedia, mas expresso em palavras e estruturas diferentes. Como isso não copia diretamente de um texto de origem, esse provavelmente será o tipo de plágio mais desafiador de detectar.\n",
    "4. `non`: uma resposta não é plagiada; o texto-fonte da Wikipedia não é usado para criar esta resposta.\n",
    "5. `orig`: Esta é uma categoria específica para o texto-fonte original da Wikipedia. Usaremos esses arquivos apenas para fins de comparação.\n",
    "\n",
    "> Então, dos arquivos enviados, a única categoria que não contém nenhum plágio é `não`.\n",
    "\n",
    "Na próxima célula, imprima algumas estatísticas sobre os dados."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of files:  100\n",
      "Number of unique tasks/question types (A-E):  5\n",
      "Unique plagiarism categories:  ['non' 'cut' 'light' 'heavy' 'orig']\n"
     ]
    }
   ],
   "source": [
    "# print out some stats about the data\n",
    "print('Number of files: ', plagiarism_df.shape[0])  # .shape[0] gives the rows \n",
    "# .unique() gives unique items in a specified column\n",
    "print('Number of unique tasks/question types (A-E): ', (len(plagiarism_df['Task'].unique())))\n",
    "print('Unique plagiarism categories: ', (plagiarism_df['Category'].unique()))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Você deve ver o número de arquivos de texto no conjunto de dados, bem como algumas características sobre as colunas `Tarefa` e` Categoria`. **Observe que a contagem de arquivos de 100 * inclui * os 5 arquivos wikipedia _original_ para tarefas AE.** Se você der uma olhada nos arquivos no diretório `data`, notará que os textos originais começam com o nome do arquivo ʻorig_` em oposição a `g` para\" grupo \".\n",
    "\n",
    "> Portanto, no total, existem 100 arquivos, 95 dos quais são respostas (enviadas por pessoas) e 5 dos quais são os textos-fonte originais da Wikipedia.\n",
    "\n",
    "Seu objetivo final será usar essas informações para classificar qualquer texto de resposta em uma das duas categorias, plagiado ou não plagiado."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Distribuição de dados\n",
    "\n",
    "A seguir, vamos examinar a distribuição dos dados. Neste curso, falamos sobre características como desequilíbrio de classe que podem informar como você desenvolve um algoritmo. Então, aqui, perguntaremos: ** Quão uniformemente nossos dados são distribuídos entre as diferentes tarefas e níveis de plágio? **\n",
    "\n",
    "Abaixo, você deve observar duas coisas:\n",
    "* Nosso conjunto de dados é bastante pequeno, especialmente no que diz respeito a exemplos de níveis variáveis de plágio.\n",
    "* Os dados são distribuídos de maneira bastante uniforme entre os tipos de tarefa e plágio."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Task:\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Task</th>\n",
       "      <th>Counts</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>a</td>\n",
       "      <td>20</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>b</td>\n",
       "      <td>20</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>c</td>\n",
       "      <td>20</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>d</td>\n",
       "      <td>20</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>e</td>\n",
       "      <td>20</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  Task  Counts\n",
       "0    a      20\n",
       "1    b      20\n",
       "2    c      20\n",
       "3    d      20\n",
       "4    e      20"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Plagiarism Levels:\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Category</th>\n",
       "      <th>Counts</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>cut</td>\n",
       "      <td>19</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>heavy</td>\n",
       "      <td>19</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>light</td>\n",
       "      <td>19</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>non</td>\n",
       "      <td>38</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>orig</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  Category  Counts\n",
       "0      cut      19\n",
       "1    heavy      19\n",
       "2    light      19\n",
       "3      non      38\n",
       "4     orig       5"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Task & Plagiarism Level Combos :\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Task</th>\n",
       "      <th>Category</th>\n",
       "      <th>Counts</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>a</td>\n",
       "      <td>cut</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>a</td>\n",
       "      <td>heavy</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>a</td>\n",
       "      <td>light</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>a</td>\n",
       "      <td>non</td>\n",
       "      <td>9</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>a</td>\n",
       "      <td>orig</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>b</td>\n",
       "      <td>cut</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>b</td>\n",
       "      <td>heavy</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>b</td>\n",
       "      <td>light</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>b</td>\n",
       "      <td>non</td>\n",
       "      <td>9</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>b</td>\n",
       "      <td>orig</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>c</td>\n",
       "      <td>cut</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>c</td>\n",
       "      <td>heavy</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>c</td>\n",
       "      <td>light</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>c</td>\n",
       "      <td>non</td>\n",
       "      <td>7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>c</td>\n",
       "      <td>orig</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>d</td>\n",
       "      <td>cut</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>d</td>\n",
       "      <td>heavy</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>d</td>\n",
       "      <td>light</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>d</td>\n",
       "      <td>non</td>\n",
       "      <td>6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>d</td>\n",
       "      <td>orig</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20</th>\n",
       "      <td>e</td>\n",
       "      <td>cut</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>21</th>\n",
       "      <td>e</td>\n",
       "      <td>heavy</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>22</th>\n",
       "      <td>e</td>\n",
       "      <td>light</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23</th>\n",
       "      <td>e</td>\n",
       "      <td>non</td>\n",
       "      <td>7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>24</th>\n",
       "      <td>e</td>\n",
       "      <td>orig</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Task Category  Counts\n",
       "0     a      cut       4\n",
       "1     a    heavy       3\n",
       "2     a    light       3\n",
       "3     a      non       9\n",
       "4     a     orig       1\n",
       "5     b      cut       3\n",
       "6     b    heavy       4\n",
       "7     b    light       3\n",
       "8     b      non       9\n",
       "9     b     orig       1\n",
       "10    c      cut       3\n",
       "11    c    heavy       5\n",
       "12    c    light       4\n",
       "13    c      non       7\n",
       "14    c     orig       1\n",
       "15    d      cut       4\n",
       "16    d    heavy       4\n",
       "17    d    light       5\n",
       "18    d      non       6\n",
       "19    d     orig       1\n",
       "20    e      cut       5\n",
       "21    e    heavy       3\n",
       "22    e    light       4\n",
       "23    e      non       7\n",
       "24    e     orig       1"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Show counts by different tasks and amounts of plagiarism\n",
    "\n",
    "# group and count by task\n",
    "counts_per_task=plagiarism_df.groupby(['Task']).size().reset_index(name=\"Counts\")\n",
    "print(\"\\nTask:\")\n",
    "display(counts_per_task)\n",
    "\n",
    "# group by plagiarism level\n",
    "counts_per_category=plagiarism_df.groupby(['Category']).size().reset_index(name=\"Counts\")\n",
    "print(\"\\nPlagiarism Levels:\")\n",
    "display(counts_per_category)\n",
    "\n",
    "# group by task AND plagiarism level\n",
    "counts_task_and_plagiarism=plagiarism_df.groupby(['Task', 'Category']).size().reset_index(name=\"Counts\")\n",
    "print(\"\\nTask & Plagiarism Level Combos :\")\n",
    "display(counts_task_and_plagiarism)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Também pode ser útil olhar graficamente este último DataFrame.\n",
    "\n",
    "Abaixo, você pode ver que as contagens seguem um padrão dividido por tarefa. Cada tarefa tem um texto fonte (original) e o maior número em casos `não` plagiados."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<BarContainer object of 25 artists>"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAdoAAAEvCAYAAADiuwAFAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAMDklEQVR4nO3dX4il913H8c/XTIsmrRjJKDV/3FZEKL0wZRA1UkKrolWMgkoCldab9cJqKoJWb9IbQaSWeiGFta1UjC2SRi1StAVb1JvQ2TSYP2u11JimicmWgm28iTVfL+YE183uzpnd892z5+zrBcvOnHnOme/85hne+zznzLPV3QEAZnzDugcAgG0mtAAwSGgBYJDQAsAgoQWAQUILAIN2Jh70hhtu6GPHjk08NABccU6ePPnl7t4918dGQnvs2LHs7+9PPDQAXHGq6t/P9zGnjgFgkNACwCChBYBBQgsAg4QWAAYJLQAMEloAGCS0ADBIaAFgkNACwCChBYBBI9c65qWqjn6f7tXPsemsI7BpHNECwCChBYBBQgsAg4QWAAYJLQAMEloAGCS0ADBIaAFgkNACwCChBYBBQgsAg4QWAAYJLQAMEloAGCS0ADBIaAFgkNACwCChBYBBQgsAg4QWAAYJLQAMEloAGCS0ADBIaAFg0FKhrapfq6pHq+qRqvpwVX3j9GAAsA0ODW1V3ZjkV5PsdffrklyT5M7pwQBgGyx76ngnyTdV1U6Sa5M8NTcSAGyPQ0Pb3V9K8u4kTyR5Osl/dvcnpgcDgG2wzKnj65PckeTVSb4jyXVV9ZZzbHe8qvarav/06dOrnxQANtAyp45/OMm/dffp7v7vJPcn+cGzN+ruE9291917u7u7q54TADbSMqF9Isn3V9W1VVVJ3pTk1OxYALAdlnmO9oEk9yV5MMnDi/ucGJ4LALbCzjIbdfc9Se4ZngUAto4rQwHAIKEFgEFCCwCDhBYABgktAAwSWgAYJLQAMEhoAWCQ0ALAIKEFgEFCCwCDhBYABgktAAwSWgAYJLQAMEhoAWCQ0ALAIKEFgEFCCwCDhBYABgktAAwSWgAYJLQAMGhn3QMAl1/V0e/Tvfo54GrYFx3RAsAgoQWAQUILAIOEFgAGCS0ADBJaABgktAAwSGgBYJDQAsAgoQWAQUILAIOEFgAGCS0ADBJaABgktAAwSGgBYJDQAsAgoQWAQUILAIOEFgAGCS0ADBJaABgktAAwSGgBYNBSoa2qb6mq+6rqn6vqVFX9wPRgALANdpbc7g+S/E13/2xVvTzJtYMzAcDWODS0VfXNSd6Q5G1J0t3PJ3l+diwA2A7LnDp+TZLTSf64qj5bVe+vquuG5wKArbBMaHeSvD7J+7r71iT/leSdZ29UVcerar+q9k+fPr3iMQGuLFUX94erzzKhfTLJk939wOL9+3IQ3v+nu09091537+3u7q5yRgDYWIeGtrv/I8kXq+p7Fje9Kcljo1MBwJZY9lXHv5Lk3sUrjr+Q5BfnRgKA7bFUaLv7oSR7w7MAwNZxZSgAGCS0ADBIaAFgkNACwCChBYBBQgsAg4QWAAYJLQAMEloAGCS0ADBIaAFgkNACwCChBYBBQgsAg4QWAAYJLQAMEloAGCS0ADBIaAFgkNACwCChBYBBQgsAg4QWAAYJLQAM2ln3AHC1qTr6fbpXP8fVbhu+D9vwNVwNHNECwCChBYBBQgsAg4QWAAYJLQAMEloAGCS0ADBIaAFgkNACwCChBYBBQgsAg4QWAAYJLQAMEloAGCS0ADBIaAFgkNACwCChBYBBQgsAg4QWAAYJLQAMEloAGCS0ADBIaAFg0NKhraprquqzVfXXkwMBwDY5yhHt3UlOTQ0CANtoqdBW1U1JfiLJ+2fHAYDtsuwR7XuT/EaSFwZnAYCts3PYBlX1k0me7e6TVXX7BbY7nuR4ktxyyy0rG/DgsY9+n+6VjkB8H64U2/B9WMXXsA3rsA18Hw63zBHtbUl+qqoeT/KRJG+sqj89e6PuPtHde929t7u7u+IxAWAzHRra7v6t7r6pu48luTPJ33X3W8YnA4At4PdoAWDQoc/Rnqm7P53k0yOTAMAWckQLAIOEFgAGCS0ADBJaABgktAAwSGgBYJDQAsAgoQWAQUILAIOEFgAGCS0ADBJaABgktAAwSGgBYJDQAsAgoQWAQUILAIOEFgAGCS0ADBJaABgktAAwSGgBYJDQAsAgoQWAQTvrHuByqDr6fbpX/xjrtu6v4WI+/6pnWIV1ryOskv15niNaABgktAAwSGgBYJDQAsAgoQWAQUILAIOEFgAGCS0ADBJaABgktAAwSGgBYJDQAsAgoQWAQUILAIOEFgAGCS0ADBJaABgktAAwSGgBYJDQAsAgoQWAQUILAIOEFgAGHRraqrq5qj5VVaeq6tGquvtyDAYA22BniW2+nuTXu/vBqnplkpNV9cnufmx4NgDYeIce0Xb309394OLtryU5leTG6cEAYBsc6TnaqjqW5NYkD0wMAwDbZunQVtUrknw0yTu6+6vn+Pjxqtqvqv3Tp0+vckYA2FhLhbaqXpaDyN7b3fefa5vuPtHde929t7u7u8oZAWBjLfOq40rygSSnuvs98yMBwPZY5oj2tiS/kOSNVfXQ4s+bh+cCgK1w6K/3dPc/JqnLMAsAbB1XhgKAQUILAIOEFgAGCS0ADBJaABgktAAwSGgBYJDQAsAgoQWAQUILAIOEFgAGCS0ADBJaABgktAAwSGgBYJDQAsAgoQWAQUILAIOEFgAGCS0ADBJaABgktAAwSGgBYJDQAsCgnXUPwHKqjn6f7tXPsems42pYx9Wwjqtxpa+jI1oAGCS0ADBIaAFgkNACwCChBYBBQgsAg4QWAAYJLQAMEloAGCS0ADBIaAFgkNACwCChBYBBQgsAg4QWAAYJLQAMEloAGCS0ADBIaAFgkNACwCChBYBBQgsAg4QWAAYJLQAMWiq0VfVjVfW5qvp8Vb1zeigA2BaHhraqrknyh0l+PMlrk9xVVa+dHgwAtsEyR7Tfl+Tz3f2F7n4+yUeS3DE7FgBsh2VCe2OSL57x/pOL2wCAQ+wssU2d47Z+yUZVx5McX7z7XFV97lIGW9INSb58rg/UuaY+gku9/5UwwxHubx1Xc/8rdh037PtwznXcsK/hSphha9fxMn8N5/25Pst3nu8Dy4T2ySQ3n/H+TUmeOnuj7j6R5MQSj7cyVbXf3XuX83NuI+u4GtZxNazjaljH1VjFOi5z6vgzSb67ql5dVS9PcmeSj13KJwWAq8WhR7Td/fWqenuSv01yTZIPdvej45MBwBZY5tRxuvvjST4+PMvFuKynqreYdVwN67ga1nE1rONqXPI6VvdLXtcEAKyISzACwKCNDa3LQq5GVT1eVQ9X1UNVtb/ueTZFVX2wqp6tqkfOuO1bq+qTVfWvi7+vX+eMm+A86/iuqvrSYp98qKrevM4Zr3RVdXNVfaqqTlXVo1V19+J2++MRXGAdL3l/3MhTx4vLQv5Lkh/Jwa8ffSbJXd392FoH20BV9XiSve5e5vfEWKiqNyR5LsmfdPfrFrf9XpKvdPfvLv7xd313/+Y657zSnWcd35Xkue5+9zpn2xRV9aokr+ruB6vqlUlOJvnpJG+L/XFpF1jHn88l7o+bekTrspCsVXf/fZKvnHXzHUk+tHj7Qzn4IeUCzrOOHEF3P93dDy7e/lqSUzm4ep/98QgusI6XbFND67KQq9NJPlFVJxdX9+LifXt3P50c/NAm+bY1z7PJ3l5V/7Q4teyU55Kq6liSW5M8EPvjRTtrHZNL3B83NbRLXRaSpdzW3a/Pwf/O9MuLU3mwTu9L8l1JvjfJ00l+f73jbIaqekWSjyZ5R3d/dd3zbKpzrOMl74+bGtqlLgvJ4br7qcXfzyb5ixyclufiPLN4nufF53ueXfM8G6m7n+nu/+nuF5L8UeyTh6qql+UgDvd29/2Lm+2PR3SudVzF/ripoXVZyBWoqusWT/qnqq5L8qNJHrnwvbiAjyV56+Lttyb5qzXOsrFejMPCz8Q+eUFVVUk+kORUd7/njA/ZH4/gfOu4iv1xI191nCSLl1i/N/93WcjfWfNIG6eqXpODo9jk4Cphf2Ydl1NVH05yew7+Z49nktyT5C+T/HmSW5I8keTnutsLfS7gPOt4ew5O03WSx5P80ovPNfJSVfVDSf4hycNJXljc/Ns5eH7R/rikC6zjXbnE/XFjQwsAm2BTTx0DwEYQWgAYJLQAMEhoAWCQ0ALAIKEFgEFCCwCDhBYABv0vBPak0wOCLMgAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 576x360 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "\n",
    "# counts\n",
    "group = ['Task', 'Category']\n",
    "counts = plagiarism_df.groupby(group).size().reset_index(name=\"Counts\")\n",
    "\n",
    "plt.figure(figsize=(8,5))\n",
    "plt.bar(range(len(counts)), counts['Counts'], color = 'blue')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## A seguir\n",
    "\n",
    "Este bloco de notas trata apenas de carregamento e exploração de dados, e você não precisa incluí-lo no envio final do projeto.\n",
    "\n",
    "Nos próximos blocos de notas, você usará esses dados para treinar um classificador de plágio completo. Você terá a tarefa de extrair recursos significativos dos dados do texto, lendo as respostas para diferentes tarefas e comparando-as com o texto-fonte original da Wikipedia. Você criará recursos de similaridade que ajudarão a identificar casos de plágio. Em seguida, você usará esses recursos para treinar e implantar um modelo de classificação em uma instância de notebook SageMaker. "
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "conda_python3",
   "language": "python",
   "name": "conda_python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}