Feature Preparation -Choosing train, val and test set.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Splitting train, validation and test sets"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "-    After collecting dataset images for classes with_mask and without_mask; and before training the YOLO network we need to partition the dataset in order to obtain train, validation and test sets. The first idea would be shuffle the dataset and then assign a random partition to the desired sets. Instead of that, we need to understand that we are going to train our detector using 4 different sources of data (Kaggle,RMDF,Pascal and scraped images). Not all images come from the same distribution, for example, some datasets have a bigger amount of high quality images, taken by professional cameras for example, other datasets have noisy images where people's faces are small, low quality, etc."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "-    The default path for storing train, validation and test sets is: **'data/voc2012_raw/VOCdevkit/VOC2012/ImageSets/Main'**."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- We will partition our data in approximately a 70% train - 30% validation split. The total number of images in our dataset is currently 2749. In order to have better generalization we would need to collect much more labeled images for  our classes."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- The following steps are the ones I performed in order to obtain the final train, test and validation partitions. Different dataset partitions were used, but they resulted in messy inaccurate predictions. Final split files are stored on the data/voc2012_raw/VOCdevkit/VOC2012/ImageSets/Main directory."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Reading dataset samples"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We read the filenames from all samples and store them on different lists depending on their class or dataset:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Path for all images from our dataset\n",
    "my_dataset_and_voc_path = 'data/voc2012_raw/VOCdevkit/VOC2012/JPEGImages'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "mask_samples_internet_and_rmdf = [] #samples from RMDF dataset and scraped data\n",
    "mask_samples_kaggle = [] #samples from kaggle dataset\n",
    "no_mask_samples_pascal =  [] #samples from PASCAL dataset\n",
    "\n",
    "\n",
    "#Reading all image filenames on the my_dataset_and_voc_path directory\n",
    "for filename in os.listdir(my_dataset_and_voc_path):\n",
    "    name = filename.split('.')[0]\n",
    "    if name[:4]=='2020': #Filenames from  RMDF dataset and scraped data\n",
    "        mask_samples_internet_and_rmdf.append(name)\n",
    "    elif name[:5]=='makss': #Filenames from kaggle dataset\n",
    "        mask_samples_kaggle.append(name)\n",
    "    else: #samples from PASCAL dataset starting with 2008 or 2007\n",
    "        no_mask_samples_pascal.append(name)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(1017, 879, 853)"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(no_mask_samples_pascal), len(mask_samples_internet_and_rmdf), len(mask_samples_kaggle)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Randomly shuffle samples on every list:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "random.shuffle(no_mask_samples_pascal)\n",
    "random.shuffle(mask_samples_internet_and_rmdf)\n",
    "random.shuffle(mask_samples_kaggle)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Splitting PASCAL VOC dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import random"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- First, we are going to split our subset of images from the PASCAL VOC dataset contained on the **no_mask_samples_pascal** array. As a reminder, the PASCAL VOC dataset have images for several object classes, but we picked a small subset containing the 'person' class to adapt for our problem. Each class in the PASCAL dataset comes with txt files that contain the partitions for training and validation. \n",
    "\n",
    "\n",
    "- We will first read and store all validation samples from the 'person' class that came on the original dataset, and then intersect this list with our subset. That way we will use the same validation samples indicated by the Pascal dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Extracting images from person validation file"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "person_val_file = 'data/voc2012_raw/VOCdevkit/VOC2012/ImageSets/Main/person_val.txt'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Printing first 5 validation samples:  ['2008_000003', '2008_000026', '2008_000032', '2008_000034', '2008_000051']\n"
     ]
    }
   ],
   "source": [
    "person_val = open(person_val_file,\"r\")\n",
    "lines = person_val.readlines()\n",
    "lines = [ line.split() for line in lines]\n",
    "\n",
    "#Storing all validation samples for our subset as a list\n",
    "val_samples_person = []\n",
    "\n",
    "for line in lines:\n",
    "    if line[1]=='1':\n",
    "        val_samples_person.append(line[0])\n",
    "        \n",
    "print(\"Printing first 5 validation samples: \",val_samples_person[:5])        \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finding intersection between our PASCAL samples and validation samples:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "train = []\n",
    "val = []\n",
    "\n",
    "for sample in no_mask_samples_pascal:\n",
    "    #If a sample in our subset is present on the original validation txt, add it to our list \n",
    "    if sample in val_samples_person:\n",
    "        val.append(sample)\n",
    "    else:\n",
    "        train.append(sample)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(536, 481)"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(val), len(train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Approximately 30% of our PASCAL samples will be used for validation (306), the rest for training(701) and test(10) samples:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "val_pascal = val[:306]\n",
    "train_pascal = val[306:] + train[:-10]\n",
    "test_pascal = train[-10:]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(306, 701, 10)"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(val_pascal), len(train_pascal), len(test_pascal)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Splitting RMDF and scraped dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we will split samples from RMDF and scraped datasets. There are a total of 879 samples. We are using almost 80% samples for training (700) and the rest for validation (165) and test(14):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_internet_and_rmdf = mask_samples_internet_and_rmdf[:700]\n",
    "val_internet_and_rmdf = mask_samples_internet_and_rmdf[700:700+165]\n",
    "test_internet_and_rmdf = mask_samples_internet_and_rmdf[700+165:]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Splitting Kaggle dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For the Kaggle dataset approximately 80%(700) will be used for training and the rest for validation (139) and testing (14):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_kaggle = mask_samples_kaggle[:700]\n",
    "val_kaggle = mask_samples_kaggle[700:700+139]\n",
    "test_kaggle = mask_samples_kaggle[500+338:]\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(701, 700, 500)"
      ]
     },
     "execution_count": 79,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(train_pascal),len(train_internet_and_rmdf),len(train_kaggle)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(306, 165, 338)"
      ]
     },
     "execution_count": 80,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(val_pascal),len(val_internet_and_rmdf),len(val_kaggle)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(10, 14, 15)"
      ]
     },
     "execution_count": 81,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(test_pascal),len(test_internet_and_rmdf),len(test_kaggle)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Final dataset splits"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now are merging training, validation and test samples from every dataset to obtain our final splits:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "metadata": {},
   "outputs": [],
   "source": [
    "samples_train = train_pascal + train_internet_and_rmdf + train_kaggle\n",
    "samples_val = val_pascal + val_internet_and_rmdf + val_kaggle\n",
    "samples_test = test_pascal + test_internet_and_rmdf + test_kaggle"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 84,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['2008_000278', '2008_002042', '2008_002069', '2008_001356', '2008_001275']"
      ]
     },
     "execution_count": 84,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "samples_train[:5]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Randomly shuffle all samples:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 85,
   "metadata": {},
   "outputs": [],
   "source": [
    "random.shuffle(samples_train)\n",
    "random.shuffle(samples_val)\n",
    "random.shuffle(samples_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 86,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['2008_002718',\n",
       " '2020_02_285',\n",
       " '2020_112',\n",
       " 'maksssksksss758',\n",
       " 'maksssksksss389']"
      ]
     },
     "execution_count": 86,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "samples_train[:5]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "These are the final split lengths:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 87,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(1901, 809, 39)"
      ]
     },
     "execution_count": 87,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(samples_train),len(samples_val),len(samples_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, given our splits we will store the corresponding txt files that will be used for training the model. The model doesn't read information directly from XML files, instead it reads information from TF.RECORD files."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Path for storing txt files with the dataset splits\n",
    "main = 'data/voc2012_raw/VOCdevkit/VOC2012/ImageSets/Main/'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 88,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Writing train, validation and test files\n",
    "\n",
    "train = open(main + 'train.txt',\"w\")\n",
    "val = open(main + 'val.txt',\"w\")\n",
    "test = open(main + 'test.txt',\"w\")\n",
    "\n",
    "label = 1\n",
    "for i, name in enumerate(samples_train):\n",
    "    train.write(name+' '+str(label))\n",
    "    if i < (len(samples_train) - 1):\n",
    "        train.write('\\n')\n",
    "    \n",
    "for i, name in enumerate(samples_val):\n",
    "    val.write(name+' '+str(label))\n",
    "    if i < (len(samples_val) - 1):\n",
    "        val.write('\\n')\n",
    "        \n",
    "for i, name in enumerate(samples_test):\n",
    "    test.write(name+' '+str(label))\n",
    "    if i < (len(samples_test) - 1):\n",
    "        test.write('\\n')\n",
    "        \n",
    "train.close() \n",
    "val.close()\n",
    "test.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Generating tf.record train and validation files"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Yolo pre-trained network comes with tools for generating tf.records from our train.txt, val.txt and test.txt files. For generating them, we need to call the following commands:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "python tools/voc2012.py \\\n",
    "  --data_dir './data/voc2012_raw/VOCdevkit/VOC2012' \\\n",
    "  --split train \\\n",
    "  --output_file ./data/voc2012_train.tfrecord"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "python tools/voc2012.py \\\n",
    "  --data_dir './data/voc2012_raw/VOCdevkit/VOC2012' \\\n",
    "  --split val \\\n",
    "  --output_file ./data/voc2012_val.tfrecord"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}