toxic_tutorial.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# [Toxic comment classification challenge](https://www.kaggle.com/c/jigsaw-toxic-comment-classification-challenge)\n",
    "\n",
    "In this competition, you’re challenged to build a multi-headed model that’s capable of detecting different types of of toxicity like threats, obscenity, insults, and identity-based hate better than Perspective’s current models. You’ll be using a dataset of comments from Wikipedia’s talk page edits. Improvements to the current model will hopefully help online discussion become more productive and respectful."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "########################################\n",
    "# Load the packages\n",
    "########################################\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import re\n",
    "\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns\n",
    "\n",
    "from nltk.stem import SnowballStemmer\n",
    "\n",
    "from keras.preprocessing.text import Tokenizer\n",
    "from keras.preprocessing.sequence import pad_sequences\n",
    "from keras.layers import Dense, Input, Embedding, Bidirectional, GRU, Conv1D, GlobalMaxPooling1D, Dropout, TimeDistributed\n",
    "from keras.layers.merge import concatenate\n",
    "from keras.models import Model\n",
    "from keras.callbacks import EarlyStopping, ModelCheckpoint\n",
    "\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "########################################\n",
    "# Define the hyper parameters\n",
    "########################################\n",
    "path = 'Dataset/'\n",
    "TRAIN_DATA_FILE=path + 'train.csv'\n",
    "TEST_DATA_FILE=path + 'test.csv'\n",
    "\n",
    "MAX_SEQUENCE_LENGTH = 100\n",
    "MAX_NB_WORDS = 100000\n",
    "EMBEDDING_DIM = 50"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Prepare the training / testing data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "########################################\n",
    "# Load the training / testing set with pandas csv format\n",
    "########################################\n",
    "train_df = pd.read_csv(TRAIN_DATA_FILE)\n",
    "test_df = pd.read_csv(TEST_DATA_FILE)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Expolary Data Analysis"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"A quick view of training set\")\n",
    "train_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"A quick view of testing set\")\n",
    "test_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Check the labels distribution"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Check the balance of labels\n",
    "\n",
    "We would like to know the positive ratio of training set. Because we do not want the model become a lazy guy, for a less frequent positive case, we may give it more penalty when model targets it wrong."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "'''\n",
    "What's the positive ratio of each class ?\n",
    "'''\n",
    "def get_pos_ratio(data):\n",
    "    pass\n",
    "\n",
    "pos_ratio = []\n",
    "for col in ['toxic', 'severe_toxic', 'obscene', 'threat', 'insult', 'identity_hate']:\n",
    "    pos_ratio.append(get_pos_ratio(train_df[col]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "assert pos_ratio[0] == 0.09584448302009764, \"The answer is not correct.\"\n",
    "print(\"Congrats, you passed the test.\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "x = train_df.iloc[:,2:].sum()\n",
    "\n",
    "plt.figure(figsize=(8,4))\n",
    "ax= sns.barplot(x.index, x.values, alpha=0.8)\n",
    "plt.title(\"# per class\")\n",
    "plt.ylabel('# of Occurrences', fontsize=12)\n",
    "plt.xlabel('Type ', fontsize=12)\n",
    "\n",
    "rects = ax.patches\n",
    "labels = x.values\n",
    "for rect, label in zip(rects, labels):\n",
    "    height = rect.get_height()\n",
    "    ax.text(rect.get_x() + rect.get_width()/2, height + 5, label, ha='center', va='bottom')\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Correlations of labels\n",
    "\n",
    "Because this is a mulit-label classification, we will want to know the relation betweens labels, which helps for feature engineering and model design. For example, if we know that a toxic comment is always a insult comment, when we have a high confident toxic comment, we can also consider it as insult comment."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "corr=train_df.corr()\n",
    "plt.figure(figsize=(10,8))\n",
    "sns.heatmap(corr,\n",
    "            xticklabels=corr.columns.values,\n",
    "            yticklabels=corr.columns.values, annot=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Preprocessing\n",
    "\n",
    "We apply 2 propressing method on texts:\n",
    "\n",
    "1. Make all alphabet lower case:\n",
    "    * It is very important. We do not want the model cosider 'Hello' and 'hello' as different words.\n",
    "2. Remove some special tokens and deal with postfix:\n",
    "    * For example: what's -> what is, aren't -> are not. Giving the same concept as same token helps for regularization.\n",
    "\n",
    "Always remember to do preprocess for NLP task. In many kinds of cases, the performance of model trianed with cleaned text significantly outperforms the model trained with raw data. Knowing your data is always the best policy."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "########################################\n",
    "## Text pre-processing and cleaning\n",
    "########################################\n",
    "print('Processing text dataset')\n",
    "from collections import defaultdict\n",
    "\n",
    "# regex to remove all Non-Alpha Numeric and space\n",
    "special_character_removal=re.compile(r'[^a-z\\d ]',re.IGNORECASE)\n",
    "\n",
    "# regex to replace all numeric\n",
    "replace_numbers=re.compile(r'\\d+',re.IGNORECASE)\n",
    "\n",
    "def clean_text(text, stem_words=False):\n",
    "    # Clean the text, with the option to remove stopwords and to stem words.\n",
    "    text = text.lower()\n",
    "    text = re.sub(r\"what's\", \"what is \", text)\n",
    "    text = re.sub(r\"\\'s\", \" \", text)\n",
    "    text = re.sub(r\"\\'ve\", \" have \", text)\n",
    "    text = re.sub(r\"can't\", \"cannot \", text)\n",
    "    text = re.sub(r\"n't\", \" not \", text)\n",
    "    text = re.sub(r\"i'm\", \"i am \", text)\n",
    "    text = re.sub(r\"i’m\", \"i am\", text)\n",
    "    text = re.sub(r\"\\'re\", \" are \", text)\n",
    "    text = re.sub(r\"\\'d\", \" would \", text)\n",
    "    text = re.sub(r\"\\'ll\", \" will \", text)\n",
    "    text = re.sub(r\",\", \" \", text)\n",
    "    text = re.sub(r\"\\.\", \" \", text)\n",
    "    text = re.sub(r\"'\", \" \", text)\n",
    "    text = re.sub(r\"\\s{2,}\", \" \", text)\n",
    "    text = replace_numbers.sub('', text)\n",
    "    text = special_character_removal.sub('',text)\n",
    "    \n",
    "    return text"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "'''\n",
    "Apply preprocessing and extract the training sentences and testing senteces from pandas dataframe.\n",
    "Note that there are some N/A comment in the train/test set. Fill them up first.\n",
    "'''\n",
    "train_comments = []\n",
    "test_comments = []"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "assert len(train_comments) == 159571 and len(test_comments) == 153164, \"It seems that you lost some data.\"\n",
    "assert 'E' not in train_comments[0], \"It seems you did not preprocess the sentecnes. I found a upper case alphabet in your train set.\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Let's have a comparsion between cleaned text and original one"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for i in range(5):\n",
    "    print(\"Cleaned\\n\", train_comments[i] + '\\n')\n",
    "    print(\"Raw\\n\", train_df.iloc[i]['comment_text'] + '\\n')\n",
    "    print(\"------------------\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Tokenization\n",
    "\n",
    "Tokenization separates a sentence into words by space, for example:\n",
    "\n",
    "* \"Hello world\" -> [\"Hello\", \"world\"]\n",
    "\n",
    "The input of the neural network is a digit not a word. So we have to apply one hot encoding or index encoding on them.\n",
    "![onehot](resources/onehot.png)\n",
    "Now we use keras tokenizer to learn the encoding table."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Create a tokenize, which transforms a sentence to a list of ids\n",
    "tokenizer = Tokenizer(num_words=MAX_NB_WORDS)\n",
    "# Build the relation between words and ids \n",
    "tokenizer.fit_on_texts(train_comments + test_comments)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "tokenizer.word_index # map 'the' to 1, map 'to' to 2,......"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Transform training/testing sentences to training/testing sequences.\n",
    "train_sequences = tokenizer.texts_to_sequences(train_comments)\n",
    "test_sequences = tokenizer.texts_to_sequences(test_comments)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## have a look on transformed sequences"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "for i in range(1):\n",
    "    print(\"Transformed\\n\", str(train_sequences[i]) + '\\n')\n",
    "    print(\"Cleaned\\n\", train_comments[i] + '\\n')\n",
    "    print(\"------------------\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "word_index = tokenizer.word_index\n",
    "print('Found %s unique tokens' % len(word_index))\n",
    "\n",
    "train_data = pad_sequences(train_sequences, maxlen=MAX_SEQUENCE_LENGTH)\n",
    "print('Shape of data tensor:', train_data.shape)\n",
    "print('Shape of label tensor:', train_labels.shape)\n",
    "\n",
    "test_data = pad_sequences(test_sequences, maxlen=MAX_SEQUENCE_LENGTH)\n",
    "print('Shape of test_data tensor:', test_data.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "'''\n",
    "Try to build a tokenzier, which transform [['Hello', 'World'], ['Greeting', 'my', 'friend'], ['Hello', 'have', 'a', 'nice', 'day']]\n",
    "to a list of index sequences. Note that the index should start from 1 because 0 is reserverd for padding token for some framework.\n",
    "'''\n",
    "tests_input_sentences =  [['Hello', 'World'], ['Greeting', 'my', 'friend'], ['Hello', 'have', 'a', 'nice', 'day']]\n",
    "transform_this_sentences = [['Hello', 'my', 'friend']]\n",
    "\n",
    "def index_encoding(sentences, raw_sent):\n",
    "    pass\n",
    "    return results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "transformed = index_encoding(tests_input_sentences, transform_this_sentences)\n",
    "assert transformed == [[1, 4, 5]], \"The answer is not correct.\"\n",
    "print(\"Congrats, you passed the test.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Models"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Text RNN\n",
    "\n",
    "![TextRNN](resources/textrnn.png)\n",
    "\n",
    "Here we present a classical structure, 2 layer bidirectional GRU, for text classification. Instead of adding a fully connected layer after all time steps, here we only select the last hidden unit of sequence. (LB: 50%, AUC: 0.982)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "########################################\n",
    "## Define the text rnn model structure\n",
    "########################################\n",
    "def get_text_rnn():\n",
    "    recurrent_units = 48\n",
    "    dense_units = 32\n",
    "    output_units = 6\n",
    "    \n",
    "    input_layer = Input(shape=(MAX_SEQUENCE_LENGTH,))\n",
    "    embedding_layer = Embedding(MAX_NB_WORDS, EMBEDDING_DIM, input_length=MAX_SEQUENCE_LENGTH)(input_layer)\n",
    "    \n",
    "    x = Bidirectional(GRU(recurrent_units, return_sequences=True))(embedding_layer)\n",
    "    x = Bidirectional(GRU(recurrent_units, return_sequences=False))(x)\n",
    "    \n",
    "    x = Dense(dense_units, activation=\"relu\")(x)\n",
    "    output_layer = Dense(output_units, activation=\"sigmoid\")(x)\n",
    "    \n",
    "    model = Model(inputs=input_layer, outputs=output_layer)\n",
    "    model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])\n",
    "    return model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## TextCNN\n",
    "\n",
    "![TextCNN](resources/textcnn.png)\n",
    "\n",
    "Convolution in natural langauge proceessing can be consider as a special type of ngram. We simply select the kernels with window sizes (2, 3, 4) to extract regional features. (LB: 50%, AUC: 0.982)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "########################################\n",
    "## Define the text cnn model structure\n",
    "########################################\n",
    "def get_text_cnn():\n",
    "    filter_nums = 120\n",
    "    dense_units = 72\n",
    "    output_units = 6\n",
    "    \n",
    "    input_layer = Input(shape=(MAX_SEQUENCE_LENGTH,), dtype='int32')\n",
    "    embedding_layer = Embedding(MAX_NB_WORDS, EMBEDDING_DIM, input_length=MAX_SEQUENCE_LENGTH,)(input_layer)\n",
    "        \n",
    "    conv_0 = Conv1D(filter_nums, 2, kernel_initializer=\"normal\", padding=\"valid\", activation=\"relu\")(embedding_layer)\n",
    "    conv_1 = Conv1D(filter_nums, 3, kernel_initializer=\"normal\", padding=\"valid\", activation=\"relu\")(embedding_layer)\n",
    "    conv_2 = Conv1D(filter_nums, 4, kernel_initializer=\"normal\", padding=\"valid\", activation=\"relu\")(embedding_layer)\n",
    "\n",
    "    maxpool_0 = GlobalMaxPooling1D()(conv_0)\n",
    "    maxpool_1 = GlobalMaxPooling1D()(conv_1)\n",
    "    maxpool_2 = GlobalMaxPooling1D()(conv_2)\n",
    "\n",
    "    merged_tensor = concatenate([maxpool_0, maxpool_1, maxpool_2])\n",
    "    h1 = Dense(units=dense_units, activation=\"relu\")(merged_tensor)\n",
    "    output = Dense(units=output_units, activation='sigmoid')(h1)\n",
    "\n",
    "    model = Model(inputs=input_layer, outputs=output)\n",
    "    model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])\n",
    "    return model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Hybrid Text NN\n",
    "\n",
    "![hybrid](resources/hybrid.png)\n",
    "\n",
    "This structure mixed the feature representation ideas of RNN and CNN. We firstly place the recurrent layer after embedding for building the word's level sequential information and make it connected with a convolution layer to extract the regional features of hiddens. (LB: 30%, AUC: 0.983)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "########################################\n",
    "## Define the text hybrid model structure\n",
    "########################################\n",
    "def get_hybrid_textnn():\n",
    "    recurrent_units = 48\n",
    "    dense_units = 32\n",
    "    filter_nums = 64\n",
    "    output_units = 6\n",
    "\n",
    "    input_layer = Input(shape=(MAX_SEQUENCE_LENGTH,))\n",
    "    embedding_layer = Embedding(MAX_NB_WORDS, EMBEDDING_DIM, input_length=MAX_SEQUENCE_LENGTH)(input_layer)\n",
    "    \n",
    "    x = Bidirectional(GRU(recurrent_units, return_sequences=True))(embedding_layer)\n",
    "    x = Conv1D(filter_nums, 2, kernel_initializer=\"normal\", padding=\"valid\", activation=\"relu\")(x)    \n",
    "    \n",
    "    max_pool = GlobalMaxPooling1D()(x)\n",
    "    max_pool = Dropout(0.5)(max_pool)\n",
    "    \n",
    "    output_layer = Dense(output_units, activation=\"sigmoid\")(max_pool)\n",
    "\n",
    "    model = Model(inputs=input_layer, outputs=output_layer)\n",
    "    model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])\n",
    "    return model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### K-Fold Cross Valiadtion\n",
    "![kfold](resources/kfold.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "########################################\n",
    "## Construct the cross-validation framework\n",
    "########################################\n",
    "def _train_model_by_logloss(model, batch_size, train_x, train_y, val_x, val_y, fold_id):\n",
    "    # set an early stopping checker.\n",
    "    # the training phase would stop when validation log loss decreases continuously for `patience` rounds. \n",
    "    early_stopping = EarlyStopping(monitor='val_loss', patience=5)\n",
    "    bst_model_path = \"ToxicModel\" + str(fold_id) + '.h5'\n",
    "    model_checkpoint = ModelCheckpoint(bst_model_path, save_best_only=True, save_weights_only=True)\n",
    "    \n",
    "    # training on given fold data\n",
    "    hist = model.fit(train_x, train_y,\n",
    "        validation_data=(val_x, val_y),\n",
    "        epochs=50, batch_size=batch_size, shuffle=True,\n",
    "        callbacks=[early_stopping, model_checkpoint])\n",
    "    \n",
    "    # get the minimal validation log loss on this fold\n",
    "    bst_val_score = min(hist.history['val_loss'])\n",
    "\n",
    "    # return the model with best weight, best fold-val score\n",
    "    return model, bst_val_score\n",
    "\n",
    "def train_folds(X, y, fold_count, batch_size, get_model_func):\n",
    "    fold_size = len(X) // fold_count\n",
    "    models = []\n",
    "    score = 0\n",
    "    \n",
    "    # split the whole dataset to `fold_count` fold, and train our model on each fold\n",
    "    for fold_id in range(0, fold_count):\n",
    "        fold_start = fold_size * fold_id\n",
    "        fold_end = fold_start + fold_size\n",
    "\n",
    "        if fold_id == fold_size - 1:\n",
    "            fold_end = len(X)\n",
    "\n",
    "        # Generate the train/val data on fold i\n",
    "        train_x = np.concatenate([X[:fold_start], X[fold_end:]])\n",
    "        train_y = np.concatenate([y[:fold_start], y[fold_end:]])\n",
    "\n",
    "        val_x = X[fold_start:fold_end]\n",
    "        val_y = y[fold_start:fold_end]\n",
    "    \n",
    "        print(\"Training on fold #\", fold_id)\n",
    "        model, bst_val_score = _train_model_by_logloss(get_model_func(), batch_size, train_x, train_y, val_x, val_y, fold_id)\n",
    "        score += bst_val_score\n",
    "        models.append(model)\n",
    "    return models, score / fold_count"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "models, val_loss = train_folds(train_data, train_labels, 2, 256, get_text_cnn)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Practice: Try to beat the baseline: val_loss=0.050"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "your_batch_size = 256\n",
    "\n",
    "def get_your_model():\n",
    "    '''your show time'''\n",
    "    return model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "models, val_loss = train_folds(train_data, train_labels, 2, your_batch_size, get_your_model)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Make the predections"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#test_data = test_df\n",
    "CLASSES = [\"toxic\", \"severe_toxic\", \"obscene\", \"threat\", \"insult\", \"identity_hate\"]\n",
    "submit_path_prefix = \"ToxicNN-\" + str(MAX_SEQUENCE_LENGTH) \n",
    "\n",
    "print(\"Predicting testing results...\")\n",
    "test_predicts_list = []\n",
    "for fold_id, model in enumerate(models):\n",
    "    test_predicts = model.predict(test_data, batch_size=256, verbose=1)\n",
    "    test_predicts_list.append(test_predicts)\n",
    "\n",
    "# merge each folds' predictions by averaging\n",
    "test_predicts = np.zeros(test_predicts_list[0].shape)\n",
    "for fold_predict in test_predicts_list:\n",
    "    test_predicts += fold_predict\n",
    "test_predicts /= len(test_predicts_list)\n",
    "\n",
    "# create the submission file\n",
    "test_ids = test_df[\"id\"].values\n",
    "test_ids = test_ids.reshape((len(test_ids), 1))\n",
    "test_predicts = pd.DataFrame(data=test_predicts, columns=CLASSES)\n",
    "test_predicts[\"id\"] = test_ids\n",
    "test_predicts = test_predicts[[\"id\"] + CLASSES]\n",
    "submit_path = submit_path_prefix + \"-Loss{:4f}.csv\".format(val_loss)\n",
    "test_predicts.to_csv(submit_path, index=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Discussions\n",
    "\n",
    "\n",
    "### Better method to compose a sequence of vectors into a single vector ?\n",
    "\n",
    "Either in CNN or RNN, the outputs are a sequence of vectors, which means a best practice to compose a sequence of vectors into a single one is very important. We have tried to simply select the last one (in RNN) or select the one with max value (in CNN and hybrid NN) to represent a sequence, and there clearly is much room for improvement. For example, how about selecting the top K max vectors? or averaging the whole sequence to get one vector? Furthermore, we can apply **weighted averaging** to the sequence, which is called **Attention** in natural language processing  and it does help a lot on catching information in long sequences."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Jointly train or not ?\n",
    "\n",
    "This is a multilabel classification challenge, so why do we jointly train 6 labels together rather than train them one by one? Indeed, this is a good question. In some cases which are labeled sparsely or not clearly related to other classes (like threat in this dataset), training these labels independently might get a better socre because these cases should build their own unique feature representations. You can give it a try, and find the best combination on training labels."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### The power of unsupervised learing\n",
    "\n",
    "In the above tutorial, we just defined a **random initialized** embedding matrix for text classificaiton. With this method, the embedding matrix will fit well on the training set but it would also be biasd by some special tokens or noise since our dataset is not that large, that cause overfitting.\n",
    "\n",
    "We can deal with this by using some pretrained resources, like:\n",
    "\n",
    "* [GloVe embedding](https://nlp.stanford.edu/projects/glove/)\n",
    "* [Fasttext embedding](https://github.com/facebookresearch/fastText/blob/master/pretrained-vectors.md)\n",
    "* [Word2Vec](https://radimrehurek.com/gensim/models/word2vec.html)\n",
    "\n",
    "You might get a significant boost by replacing the old matrix with vectors pretrained on a big corpus which catch the similarity between words."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "'''\n",
    "Try to load the word embedding as the format\n",
    "{\n",
    "   'word1': embedding,\n",
    "   'word2': embedding,\n",
    "   ...\n",
    "}\n",
    "i.e. A key-value pair whose key is the word and the whose value is the embedding\n",
    "'''\n",
    "GLOVE_EMBEDDING = {}\n",
    "\n",
    "def load_embedding(embeddings_index, embedding_path='glove.6B.50d.txt'):\n",
    "    '''return a dict whose key is word, value is pretrained word embedding'''\n",
    "    return embeddings_index\n",
    "\n",
    "GLOVE_EMBEDDING = load_embedding(GLOVE_EMBEDDING)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "assert len(GLOVE_EMBEDDING) == 400000 , \"Failed. Did you load the whole file ?\"\n",
    "assert 'hello' in GLOVE_EMBEDDING.keys(), \"Oops, it seems that you miss some words\"\n",
    "assert len(GLOVE_EMBEDDING['hello'] == 50), \"You have a wrong dimension. Check it again.\"\n",
    "\n",
    "print(\"Congrats, you passed the test.\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def build_embedding_matrix(embeddings_index, word_index):\n",
    "    embedding_matrix = np.zeros((MAX_NB_WORDS, 50))\n",
    "\n",
    "    for word, i in word_index.items():\n",
    "        embedding_vector = embeddings_index.get(word)\n",
    "        if i >= MAX_NB_WORDS:\n",
    "            continue\n",
    "        if embedding_vector is not None:\n",
    "            embedding_matrix[i] = embedding_vector\n",
    "    print('Null word embeddings: %d' % np.sum(np.sum(embedding_matrix, axis=1) == 0))\n",
    "    return embedding_matrix\n",
    "\n",
    "GLOVE_EMBEDDING = build_embedding_matrix(GLOVE_EMBEDDING, word_index)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Replace the embedding to a pretrained one is easy\n",
    "\n",
    "To freeze or keep train the word embedding depends on your task. Sometimes, we tend to add a WordEncoder among the word embedding.\n",
    "The WordEncoder can be a simple Dense layer or something complicated (for example: Highway network).\n",
    "\n",
    "```\n",
    "embedding_layer = Embedding(MAX_NB_WORDS,\n",
    "                        EMBEDDING_DIM,\n",
    "                        weights=[GLOVE_EMBEDDING],\n",
    "                        input_length=MAX_SEQUENCE_LENGTH,\n",
    "                        trainable=False)(TENSOR)\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "'''\n",
    "Try to create a model with pretrained word embedding which has a single layer dense WordEncoder.\n",
    "Reference: https://keras.io/layers/wrappers/\n",
    "'''\n",
    "def get_your_model_with_pretrained_embedding():\n",
    "    '''your show time'''\n",
    "    return model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "assert type(get_your_model_with_pretrained_embedding().layers[2]) == TimeDistributed, \"Your model do not have a word encoder.\"\n",
    "print(\"Congrats, you passed the test.\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "models, val_loss = train_folds(train_data, train_labels, 2, 256, get_your_model_with_pretrained_embedding)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Reference: [My solution to this challenge](https://github.com/zake7749/DeepToxic)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}