HPSO-LS.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "68138243",
   "metadata": {},
   "source": [
    "# A Hybrid Particle Swarm Optimization for feature subset selection by integrating a novel Local Search strategy (HPSO-LS)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "79c00e53",
   "metadata": {},
   "source": [
    "***BY: Rafe Sharif ***  \n",
    "*Jan 1, 2023*"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "20fb8178",
   "metadata": {},
   "source": [
    "## Imports"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "5e535b32",
   "metadata": {},
   "outputs": [],
   "source": [
    "import random\n",
    "import numpy as np \n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns\n",
    "\n",
    "from sklearn.preprocessing import StandardScaler\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.neighbors import KNeighborsClassifier\n",
    "from sklearn.model_selection import GridSearchCV\n",
    "\n",
    "from sklearn.metrics import accuracy_score\n",
    "\n",
    "import warnings \n",
    "warnings.filterwarnings('ignore')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2fadf86a",
   "metadata": {},
   "source": [
    "## DATA"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c78e77e9",
   "metadata": {},
   "source": [
    "**Dataset Description**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "4b295c3e",
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "NAME: Sonar, Mines vs. Rocks\n",
      "\n",
      "SUMMARY: This is the data set used by Gorman and Sejnowski in their study\n",
      "of the classification of sonar signals using a neural network [1].  The\n",
      "task is to train a network to discriminate between sonar signals bounced\n",
      "off a metal cylinder and those bounced off a roughly cylindrical rock.\n",
      "\n",
      "SOURCE: The data set was contributed to the benchmark collection by Terry\n",
      "Sejnowski, now at the Salk Institute and the University of California at\n",
      "San Deigo.  The data set was developed in collaboration with R. Paul\n",
      "Gorman of Allied-Signal Aerospace Technology Center.\n",
      "\n",
      "MAINTAINER: Scott E. Fahlman\n",
      "\n",
      "PROBLEM DESCRIPTION:\n",
      "\n",
      "The file \"sonar.mines\" contains 111 patterns obtained by bouncing sonar\n",
      "signals off a metal cylinder at various angles and under various\n",
      "conditions.  The file \"sonar.rocks\" contains 97 patterns obtained from\n",
      "rocks under similar conditions.  The transmitted sonar signal is a\n",
      "frequency-modulated chirp, rising in frequency.  The data set contains\n",
      "signals obtained from a variety of different aspect angles, spanning 90\n",
      "degrees for the cylinder and 180 degrees for the rock.\n",
      "\n",
      "Each pattern is a set of 60 numbers in the range 0.0 to 1.0.  Each number\n",
      "represents the energy within a particular frequency band, integrated over\n",
      "a certain period of time.  The integration aperture for higher frequencies\n",
      "occur later in time, since these frequencies are transmitted later during\n",
      "the chirp.\n",
      "\n",
      "The label associated with each record contains the letter \"R\" if the object\n",
      "is a rock and \"M\" if it is a mine (metal cylinder).  The numbers in the\n",
      "labels are in increasing order of aspect angle, but they do not encode the\n",
      "angle directly.\n",
      "\n",
      "METHODOLOGY: \n",
      "\n",
      "This data set can be used in a number of different ways to test learning\n",
      "speed, quality of ultimate learning, ability to generalize, or combinations\n",
      "of these factors.\n",
      "\n",
      "In [1], Gorman and Sejnowski report two series of experiments: an\n",
      "\"aspect-angle independent\" series, in which the whole data set is used\n",
      "without controlling for aspect angle, and an \"aspect-angle dependent\"\n",
      "series in which the training and testing sets were carefully controlled to\n",
      "ensure that each set contained cases from each aspect angle in\n",
      "appropriate proportions.\n",
      "\n",
      "For the aspect-angle independent experiments the combined set of 208 cases\n",
      "is divided randomly into 13 disjoint sets with 16 cases in each.  For each\n",
      "experiment, 12 of these sets are used as training data, while the 13th is\n",
      "reserved for testing.  The experiment is repeated 13 times so that every\n",
      "case appears once as part of a test set.  The reported performance is an\n",
      "average over the entire set of 13 different test sets, each run 10 times.\n",
      "\n",
      "It was observed that this random division of the sample set led to rather\n",
      "uneven performance.  A few of the splits gave poor results, presumably\n",
      "because the test set contains some samples from aspect angles that are\n",
      "under-represented in the corresponding training set.  This motivated Gorman\n",
      "and Sejnowski to devise a different set of experiments in which an attempt\n",
      "was made to balance the training and test sets so that each would have a\n",
      "representative number of samples from all aspect angles.  Since detailed\n",
      "aspect angle information was not present in the data base of samples, the\n",
      "208 samples were first divided into clusters, using a 60-dimensional\n",
      "Euclidian metric; each of these clusters was then divided between the\n",
      "104-member training set and the 104-member test set.  \n",
      "\n",
      "The actual training and testing samples used for the \"aspect angle\n",
      "dependent\" experiments are marked in the data files.  The reported\n",
      "performance is an average over 10 runs with this single division of the\n",
      "data set.\n",
      "\n",
      "A standard back-propagation network was used for all experiments.  The\n",
      "network had 60 inputs and 2 output units, one indicating a cylinder and the\n",
      "other a rock.  Experiments were run with no hidden units (direct\n",
      "connections from each input to each output) and with a single hidden layer\n",
      "with 2, 3, 6, 12, or 24 units.  Each network was trained by 300 epochs over\n",
      "the entire training set.\n",
      "\n",
      "The weight-update formulas used in this study were slightly different from\n",
      "the standard form.  A learning rate of 2.0 and momentum of 0.0 was used.\n",
      "Errors less than 0.2 were treated as zero.  Initial weights were uniform\n",
      "random values in the range -0.3 to +0.3.\n",
      "\n",
      "RESULTS: \n",
      "\n",
      "For the angle independent experiments, Gorman and Sejnowski report the\n",
      "following results for networks with different numbers of hidden units:\n",
      "\n",
      "Hidden\t% Right on\tStd.\t% Right on\tStd.\n",
      "Units\tTraining set\tDev.\tTest Set\tDev.\n",
      "------\t------------\t----\t----------\t----\n",
      "0\t89.4\t\t2.1\t77.1\t\t8.3\n",
      "2\t96.5\t\t0.7\t81.9\t\t6.2\n",
      "3\t98.8\t\t0.4\t82.0\t\t7.3\n",
      "6\t99.7\t\t0.2\t83.5\t\t5.6\n",
      "12\t99.8\t\t0.1\t84.7\t\t5.7\n",
      "24\t99.8\t\t0.1\t84.5\t\t5.7\n",
      "\n",
      "For the angle-dependent experiments Gorman and Sejnowski report the\n",
      "following results:\n",
      "\n",
      "Hidden\t% Right on\tStd.\t% Right on\tStd.\n",
      "Units\tTraining set\tDev.\tTest Set\tDev.\n",
      "------\t------------\t----\t----------\t----\n",
      "0\t79.3\t\t3.4\t73.1\t\t4.8\n",
      "2\t96.2\t\t2.2\t85.7\t\t6.3\n",
      "3\t98.1\t\t1.5\t87.6\t\t3.0\n",
      "6\t99.4\t\t0.9\t89.3\t\t2.4\n",
      "12\t99.8\t\t0.6\t90.4\t\t1.8\n",
      "24     100.0\t\t0.0\t89.2\t\t1.4\n",
      "\n",
      "Not surprisingly, the network's performance on the test set was somewhat\n",
      "better when the aspect angles in the training and test sets were balanced.\n",
      "\n",
      "Gorman and Sejnowski further report that a nearest neighbor classifier on\n",
      "the same data gave an 82.7% probability of correct classification.\n",
      "\n",
      "Three trained human subjects were each tested on 100 signals, chosen at\n",
      "random from the set of 208 returns used to create this data set.  Their\n",
      "responses ranged between 88% and 97% correct.  However, they may have been\n",
      "using information from the raw sonar signal that is not preserved in the\n",
      "processed data sets presented here.\n",
      "\n",
      "REFERENCES: \n",
      "\n",
      "1. Gorman, R. P., and Sejnowski, T. J. (1988).  \"Analysis of Hidden Units\n",
      "in a Layered Network Trained to Classify Sonar Targets\" in Neural Networks,\n",
      "Vol. 1, pp. 75-89.\n",
      "\n"
     ]
    }
   ],
   "source": [
    "with open(\"./sonar/sonar.names\") as f:\n",
    "    print(f.read())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "3378d1d9",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "      <th>5</th>\n",
       "      <th>6</th>\n",
       "      <th>7</th>\n",
       "      <th>8</th>\n",
       "      <th>9</th>\n",
       "      <th>...</th>\n",
       "      <th>50</th>\n",
       "      <th>51</th>\n",
       "      <th>52</th>\n",
       "      <th>53</th>\n",
       "      <th>54</th>\n",
       "      <th>55</th>\n",
       "      <th>56</th>\n",
       "      <th>57</th>\n",
       "      <th>58</th>\n",
       "      <th>59</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>208.000000</td>\n",
       "      <td>208.000000</td>\n",
       "      <td>208.000000</td>\n",
       "      <td>208.000000</td>\n",
       "      <td>208.000000</td>\n",
       "      <td>208.000000</td>\n",
       "      <td>208.000000</td>\n",
       "      <td>208.000000</td>\n",
       "      <td>208.000000</td>\n",
       "      <td>208.000000</td>\n",
       "      <td>...</td>\n",
       "      <td>208.000000</td>\n",
       "      <td>208.000000</td>\n",
       "      <td>208.000000</td>\n",
       "      <td>208.000000</td>\n",
       "      <td>208.000000</td>\n",
       "      <td>208.000000</td>\n",
       "      <td>208.000000</td>\n",
       "      <td>208.000000</td>\n",
       "      <td>208.000000</td>\n",
       "      <td>208.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>0.029164</td>\n",
       "      <td>0.038437</td>\n",
       "      <td>0.043832</td>\n",
       "      <td>0.053892</td>\n",
       "      <td>0.075202</td>\n",
       "      <td>0.104570</td>\n",
       "      <td>0.121747</td>\n",
       "      <td>0.134799</td>\n",
       "      <td>0.178003</td>\n",
       "      <td>0.208259</td>\n",
       "      <td>...</td>\n",
       "      <td>0.016069</td>\n",
       "      <td>0.013420</td>\n",
       "      <td>0.010709</td>\n",
       "      <td>0.010941</td>\n",
       "      <td>0.009290</td>\n",
       "      <td>0.008222</td>\n",
       "      <td>0.007820</td>\n",
       "      <td>0.007949</td>\n",
       "      <td>0.007941</td>\n",
       "      <td>0.006507</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>0.022991</td>\n",
       "      <td>0.032960</td>\n",
       "      <td>0.038428</td>\n",
       "      <td>0.046528</td>\n",
       "      <td>0.055552</td>\n",
       "      <td>0.059105</td>\n",
       "      <td>0.061788</td>\n",
       "      <td>0.085152</td>\n",
       "      <td>0.118387</td>\n",
       "      <td>0.134416</td>\n",
       "      <td>...</td>\n",
       "      <td>0.012008</td>\n",
       "      <td>0.009634</td>\n",
       "      <td>0.007060</td>\n",
       "      <td>0.007301</td>\n",
       "      <td>0.007088</td>\n",
       "      <td>0.005736</td>\n",
       "      <td>0.005785</td>\n",
       "      <td>0.006470</td>\n",
       "      <td>0.006181</td>\n",
       "      <td>0.005031</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>0.001500</td>\n",
       "      <td>0.000600</td>\n",
       "      <td>0.001500</td>\n",
       "      <td>0.005800</td>\n",
       "      <td>0.006700</td>\n",
       "      <td>0.010200</td>\n",
       "      <td>0.003300</td>\n",
       "      <td>0.005500</td>\n",
       "      <td>0.007500</td>\n",
       "      <td>0.011300</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000800</td>\n",
       "      <td>0.000500</td>\n",
       "      <td>0.001000</td>\n",
       "      <td>0.000600</td>\n",
       "      <td>0.000400</td>\n",
       "      <td>0.000300</td>\n",
       "      <td>0.000300</td>\n",
       "      <td>0.000100</td>\n",
       "      <td>0.000600</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25%</th>\n",
       "      <td>0.013350</td>\n",
       "      <td>0.016450</td>\n",
       "      <td>0.018950</td>\n",
       "      <td>0.024375</td>\n",
       "      <td>0.038050</td>\n",
       "      <td>0.067025</td>\n",
       "      <td>0.080900</td>\n",
       "      <td>0.080425</td>\n",
       "      <td>0.097025</td>\n",
       "      <td>0.111275</td>\n",
       "      <td>...</td>\n",
       "      <td>0.008425</td>\n",
       "      <td>0.007275</td>\n",
       "      <td>0.005075</td>\n",
       "      <td>0.005375</td>\n",
       "      <td>0.004150</td>\n",
       "      <td>0.004400</td>\n",
       "      <td>0.003700</td>\n",
       "      <td>0.003600</td>\n",
       "      <td>0.003675</td>\n",
       "      <td>0.003100</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>0.022800</td>\n",
       "      <td>0.030800</td>\n",
       "      <td>0.034300</td>\n",
       "      <td>0.044050</td>\n",
       "      <td>0.062500</td>\n",
       "      <td>0.092150</td>\n",
       "      <td>0.106950</td>\n",
       "      <td>0.112100</td>\n",
       "      <td>0.152250</td>\n",
       "      <td>0.182400</td>\n",
       "      <td>...</td>\n",
       "      <td>0.013900</td>\n",
       "      <td>0.011400</td>\n",
       "      <td>0.009550</td>\n",
       "      <td>0.009300</td>\n",
       "      <td>0.007500</td>\n",
       "      <td>0.006850</td>\n",
       "      <td>0.005950</td>\n",
       "      <td>0.005800</td>\n",
       "      <td>0.006400</td>\n",
       "      <td>0.005300</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75%</th>\n",
       "      <td>0.035550</td>\n",
       "      <td>0.047950</td>\n",
       "      <td>0.057950</td>\n",
       "      <td>0.064500</td>\n",
       "      <td>0.100275</td>\n",
       "      <td>0.134125</td>\n",
       "      <td>0.154000</td>\n",
       "      <td>0.169600</td>\n",
       "      <td>0.233425</td>\n",
       "      <td>0.268700</td>\n",
       "      <td>...</td>\n",
       "      <td>0.020825</td>\n",
       "      <td>0.016725</td>\n",
       "      <td>0.014900</td>\n",
       "      <td>0.014500</td>\n",
       "      <td>0.012100</td>\n",
       "      <td>0.010575</td>\n",
       "      <td>0.010425</td>\n",
       "      <td>0.010350</td>\n",
       "      <td>0.010325</td>\n",
       "      <td>0.008525</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>0.137100</td>\n",
       "      <td>0.233900</td>\n",
       "      <td>0.305900</td>\n",
       "      <td>0.426400</td>\n",
       "      <td>0.401000</td>\n",
       "      <td>0.382300</td>\n",
       "      <td>0.372900</td>\n",
       "      <td>0.459000</td>\n",
       "      <td>0.682800</td>\n",
       "      <td>0.710600</td>\n",
       "      <td>...</td>\n",
       "      <td>0.100400</td>\n",
       "      <td>0.070900</td>\n",
       "      <td>0.039000</td>\n",
       "      <td>0.035200</td>\n",
       "      <td>0.044700</td>\n",
       "      <td>0.039400</td>\n",
       "      <td>0.035500</td>\n",
       "      <td>0.044000</td>\n",
       "      <td>0.036400</td>\n",
       "      <td>0.043900</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>8 rows × 60 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "               0           1           2           3           4           5   \\\n",
       "count  208.000000  208.000000  208.000000  208.000000  208.000000  208.000000   \n",
       "mean     0.029164    0.038437    0.043832    0.053892    0.075202    0.104570   \n",
       "std      0.022991    0.032960    0.038428    0.046528    0.055552    0.059105   \n",
       "min      0.001500    0.000600    0.001500    0.005800    0.006700    0.010200   \n",
       "25%      0.013350    0.016450    0.018950    0.024375    0.038050    0.067025   \n",
       "50%      0.022800    0.030800    0.034300    0.044050    0.062500    0.092150   \n",
       "75%      0.035550    0.047950    0.057950    0.064500    0.100275    0.134125   \n",
       "max      0.137100    0.233900    0.305900    0.426400    0.401000    0.382300   \n",
       "\n",
       "               6           7           8           9   ...          50  \\\n",
       "count  208.000000  208.000000  208.000000  208.000000  ...  208.000000   \n",
       "mean     0.121747    0.134799    0.178003    0.208259  ...    0.016069   \n",
       "std      0.061788    0.085152    0.118387    0.134416  ...    0.012008   \n",
       "min      0.003300    0.005500    0.007500    0.011300  ...    0.000000   \n",
       "25%      0.080900    0.080425    0.097025    0.111275  ...    0.008425   \n",
       "50%      0.106950    0.112100    0.152250    0.182400  ...    0.013900   \n",
       "75%      0.154000    0.169600    0.233425    0.268700  ...    0.020825   \n",
       "max      0.372900    0.459000    0.682800    0.710600  ...    0.100400   \n",
       "\n",
       "               51          52          53          54          55          56  \\\n",
       "count  208.000000  208.000000  208.000000  208.000000  208.000000  208.000000   \n",
       "mean     0.013420    0.010709    0.010941    0.009290    0.008222    0.007820   \n",
       "std      0.009634    0.007060    0.007301    0.007088    0.005736    0.005785   \n",
       "min      0.000800    0.000500    0.001000    0.000600    0.000400    0.000300   \n",
       "25%      0.007275    0.005075    0.005375    0.004150    0.004400    0.003700   \n",
       "50%      0.011400    0.009550    0.009300    0.007500    0.006850    0.005950   \n",
       "75%      0.016725    0.014900    0.014500    0.012100    0.010575    0.010425   \n",
       "max      0.070900    0.039000    0.035200    0.044700    0.039400    0.035500   \n",
       "\n",
       "               57          58          59  \n",
       "count  208.000000  208.000000  208.000000  \n",
       "mean     0.007949    0.007941    0.006507  \n",
       "std      0.006470    0.006181    0.005031  \n",
       "min      0.000300    0.000100    0.000600  \n",
       "25%      0.003600    0.003675    0.003100  \n",
       "50%      0.005800    0.006400    0.005300  \n",
       "75%      0.010350    0.010325    0.008525  \n",
       "max      0.044000    0.036400    0.043900  \n",
       "\n",
       "[8 rows x 60 columns]"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = pd.read_csv(\"./sonar/sonar.all-data\",header=None)\n",
    "df.describe()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "365c7cc2",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAEGCAYAAACKB4k+AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/MnkTPAAAACXBIWXMAAAsTAAALEwEAmpwYAAANh0lEQVR4nO3df6zd9V3H8ecLKgGmW8t6U0sLtjoyQ3ADvEGUxCyrJmzTlSxIIM5VRlL/QMbcotT9IbrEhEV0dtMsaWCszIWBbFq2kBlSQWMyq7eD8KNIaFCgpKV3A8Z0mxN8+8f59rNLbeHQ3nO+pz3PR3Jyz/d7vuec9x+XPvl+v+d7bqoKSZIATuh7AEnS5DAKkqTGKEiSGqMgSWqMgiSpWdL3AEdj+fLltWbNmr7HkKRjys6dO79ZVTOHeuyYjsKaNWuYm5vrewxJOqYkefJwj3n4SJLUGAVJUmMUJEmNUZAkNUZBktQYBUlSYxQkSY1RkCQ1RkGS1BzTVzRLx7OnPv4zfY+gCXTmHzw00td3T0GS1BgFSVJjFCRJjVGQJDVGQZLUGAVJUmMUJEmNUZAkNUZBktQYBUlSYxQkSY1RkCQ1RkGS1BgFSVJjFCRJjVGQJDUji0KSzybZn+ThBetOS3JPkse7n8u69UnyqSS7kzyY5PxRzSVJOrxR7il8Drj4oHWbgO1VdRawvVsGeBdwVnfbCHxmhHNJkg5jZFGoqn8Enjto9Xpga3d/K3DJgvW31sA/A0uTrBzVbJKkQxv3OYUVVbW3u78PWNHdXwU8vWC7Pd26/yfJxiRzSebm5+dHN6kkTaHeTjRXVQF1BM/bUlWzVTU7MzMzgskkaXqNOwrPHjgs1P3c361/BjhjwXaru3WSpDEadxTuAjZ09zcA2xas/0D3KaQLgW8vOMwkSRqTJaN64SS3Ae8AlifZA1wP3ADckeQq4Engsm7zu4F3A7uB7wJXjmouSdLhjSwKVXXFYR5ad4htC7h6VLNIkoYzsigcK372d2/tewRNoJ1/8oG+R5B64ddcSJIaoyBJaoyCJKkxCpKkxihIkhqjIElqjIIkqTEKkqTGKEiSGqMgSWqMgiSpMQqSpMYoSJIaoyBJaoyCJKkxCpKkxihIkhqjIElqjIIkqTEKkqTGKEiSGqMgSWqMgiSpMQqSpMYoSJIaoyBJaoyCJKnpJQpJfifJI0keTnJbkpOTrE2yI8nuJLcnOamP2SRpmo09CklWAR8CZqvqHOBE4HLgE8Anq+otwPPAVeOeTZKmXV+Hj5YApyRZApwK7AXeCdzZPb4VuKSf0SRpeo09ClX1DHAj8BSDGHwb2Am8UFUvdZvtAVaNezZJmnZ9HD5aBqwH1gKnA28ALn4dz9+YZC7J3Pz8/IimlKTp1Mfho18C/r2q5qvqf4AvAxcBS7vDSQCrgWcO9eSq2lJVs1U1OzMzM56JJWlK9BGFp4ALk5yaJMA6YBdwL3Bpt80GYFsPs0nSVOvjnMIOBieUvwE81M2wBbgO+EiS3cCbgZvHPZskTbslr73J4quq64HrD1r9BHBBD+NIkjpe0SxJaoyCJKkxCpKkxihIkhqjIElqjIIkqTEKkqTGKEiSGqMgSWqMgiSpMQqSpMYoSJIaoyBJaoyCJKkxCpKkxihIkhqjIElqjIIkqTEKkqTGKEiSGqMgSWqMgiSpMQqSpMYoSJKaoaKQZPsw6yRJx7Ylr/ZgkpOBU4HlSZYB6R56I7BqxLNJksbsVaMA/BbwYeB0YCc/jMKLwF+MbixJUh9eNQpVtRnYnOSaqvr0mGaSJPXktfYUAKiqTyf5BWDNwudU1a0jmkuS1IOhopDk88BPAQ8AL3erCziiKCRZCtwEnNO9zgeBx4DbGYTnP4DLqur5I3l9SdKRGSoKwCxwdlXVIr3vZuBrVXVpkpMYnMz+GLC9qm5IsgnYBFy3SO8nSRrCsNcpPAz8+GK8YZI3Ab8I3AxQVT+oqheA9cDWbrOtwCWL8X6SpOENu6ewHNiV5F+A/z6wsqreewTvuRaYB25J8nYGn2q6FlhRVXu7bfYBKw715CQbgY0AZ5555hG8vSTpcIaNwh8u8nueD1xTVTuSbGZwqKipqkpyyENVVbUF2AIwOzu7WIezJEkM/+mjf1jE99wD7KmqHd3ynQyi8GySlVW1N8lKYP8ivqckaQjDfs3Fd5K82N2+n+TlJC8eyRtW1T7g6SRv7VatA3YBdwEbunUbgG1H8vqSpCM37J7Cjx24nyQMTgpfeBTvew3whe6TR08AVzII1B1JrgKeBC47iteXJB2BYc8pNN3HUv82yfUcdC7gdbzGAww+5nqwdUfyepKkxTHsxWvvW7B4AoN/0L8/kokkSb0Zdk/hVxfcf4nBFcfrF30aSVKvhj2ncOWoB5Ek9W/YTx+tTvI3SfZ3ty8lWT3q4SRJ4zXs11zcwuAjo6d3t6906yRJx5FhozBTVbdU1Uvd7XPAzAjnkiT1YNgofCvJ+5Oc2N3eD3xrlINJksZv2Ch8kMHFZPuAvcClwG+OaCZJUk+G/Ujqx4ENB/7oTZLTgBsZxEKSdJwYdk/hbQv/ClpVPQecN5qRJEl9GTYKJyRZdmCh21N43V+RIUmabMP+w/6nwNeT/HW3/GvAH49mJElSX4a9ovnWJHPAO7tV76uqXaMbS5LUh6EPAXURMASSdBwb9pyCJGkKGAVJUmMUJEmNUZAkNUZBktQYBUlSYxQkSY1RkCQ1RkGS1BgFSVJjFCRJjVGQJDVGQZLUGAVJUtNbFJKcmOT+JF/tltcm2ZFkd5Lbk5zU12ySNK363FO4Fnh0wfIngE9W1VuA54GreplKkqZYL1FIshp4D3BTtxwGf9Xtzm6TrcAlfcwmSdOsrz2FPwd+D/jfbvnNwAtV9VK3vAdYdagnJtmYZC7J3Pz8/MgHlaRpMvYoJPkVYH9V7TyS51fVlqqararZmZmZRZ5Okqbb0H+jeRFdBLw3ybuBk4E3ApuBpUmWdHsLq4FnephNkqba2PcUqur3q2p1Va0BLgf+vqp+HbgXuLTbbAOwbdyzSdK0m6TrFK4DPpJkN4NzDDf3PI8kTZ0+Dh81VXUfcF93/wnggj7nkaRpN0l7CpKknhkFSVJjFCRJjVGQJDVGQZLUGAVJUmMUJEmNUZAkNUZBktQYBUlSYxQkSY1RkCQ1RkGS1BgFSVJjFCRJjVGQJDVGQZLUGAVJUmMUJEmNUZAkNUZBktQYBUlSYxQkSY1RkCQ1RkGS1BgFSVJjFCRJjVGQJDVjj0KSM5Lcm2RXkkeSXNutPy3JPUke734uG/dskjTt+thTeAn4aFWdDVwIXJ3kbGATsL2qzgK2d8uSpDEaexSqam9VfaO7/x3gUWAVsB7Y2m22Fbhk3LNJ0rTr9ZxCkjXAecAOYEVV7e0e2gesOMxzNiaZSzI3Pz8/nkElaUr0FoUkPwp8CfhwVb248LGqKqAO9byq2lJVs1U1OzMzM4ZJJWl69BKFJD/CIAhfqKovd6ufTbKye3wlsL+P2SRpmvXx6aMANwOPVtWfLXjoLmBDd38DsG3cs0nStFvSw3teBPwG8FCSB7p1HwNuAO5IchXwJHBZD7NJ0lQbexSq6p+AHObhdeOcRZL0Sl7RLElqjIIkqTEKkqTGKEiSGqMgSWqMgiSpMQqSpMYoSJIaoyBJaoyCJKkxCpKkxihIkhqjIElqjIIkqTEKkqTGKEiSGqMgSWqMgiSpMQqSpMYoSJIaoyBJaoyCJKkxCpKkxihIkhqjIElqjIIkqTEKkqTGKEiSGqMgSWomKgpJLk7yWJLdSTb1PY8kTZuJiUKSE4G/BN4FnA1ckeTsfqeSpOkyMVEALgB2V9UTVfUD4IvA+p5nkqSpsqTvARZYBTy9YHkP8HMHb5RkI7CxW/zPJI+NYbZpsRz4Zt9DTILcuKHvEfRK/m4ecH0W41V+4nAPTFIUhlJVW4Atfc9xPEoyV1Wzfc8hHczfzfGZpMNHzwBnLFhe3a2TJI3JJEXhX4GzkqxNchJwOXBXzzNJ0lSZmMNHVfVSkt8G/g44EfhsVT3S81jTxsNymlT+bo5JqqrvGSRJE2KSDh9JknpmFCRJjVGYckleTvJAkoeTfCXJ0r5nkgCSVJK/WrC8JMl8kq/2Odfxzijoe1V1blWdAzwHXN33QFLnv4BzkpzSLf8yfkx95IyCFvo6gyvLpUlxN/Ce7v4VwG09zjIVjIKA9oWE6/DaEE2WLwKXJzkZeBuwo+d5jntGQackeQDYB6wA7ul3HOmHqupBYA2DvYS7+51mOhgFfa+qzmXwBVnBcwqaPHcBN+Kho7EwCgKgqr4LfAj4aJKJudJdAj4L/FFVPdT3INPAKKipqvuBBxnsqksToar2VNWn+p5jWvg1F5Kkxj0FSVJjFCRJjVGQJDVGQZLUGAVJUmMUpEWQZGmSO5P8W5JHk/x8ktOS3JPk8e7nsr7nlF6LUZAWx2bga1X108DbgUeBTcD2qjoL2N4tSxPN6xSko5TkTcADwE/Wgv+gkjwGvKOq9iZZCdxXVW/taUxpKO4pSEdvLTAP3JLk/iQ3JXkDsKKq9nbbHPjCQWmiGQXp6C0Bzgc+U1XnMfjjMK84VNTtQbhbrolnFKSjtwfYU1UHvuv/TgaReLY7bET3c39P80lDMwrSUaqqfcDTSQ6cL1gH7GLwlc8bunUbgG09jCe9Lp5olhZBknOBm4CTgCeAKxn8T9cdwJnAk8BlVfVcXzNKwzAKkqTGw0eSpMYoSJIaoyBJaoyCJKkxCpKkxihIkhqjIElq/g+8KPDoiFbqeAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "sns.countplot(x=df[60]);"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "81567ca5",
   "metadata": {},
   "source": [
    "**Defining X values and Y values**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "d2d77fdc",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "      <th>5</th>\n",
       "      <th>6</th>\n",
       "      <th>7</th>\n",
       "      <th>8</th>\n",
       "      <th>9</th>\n",
       "      <th>...</th>\n",
       "      <th>50</th>\n",
       "      <th>51</th>\n",
       "      <th>52</th>\n",
       "      <th>53</th>\n",
       "      <th>54</th>\n",
       "      <th>55</th>\n",
       "      <th>56</th>\n",
       "      <th>57</th>\n",
       "      <th>58</th>\n",
       "      <th>59</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.0200</td>\n",
       "      <td>0.0371</td>\n",
       "      <td>0.0428</td>\n",
       "      <td>0.0207</td>\n",
       "      <td>0.0954</td>\n",
       "      <td>0.0986</td>\n",
       "      <td>0.1539</td>\n",
       "      <td>0.1601</td>\n",
       "      <td>0.3109</td>\n",
       "      <td>0.2111</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0232</td>\n",
       "      <td>0.0027</td>\n",
       "      <td>0.0065</td>\n",
       "      <td>0.0159</td>\n",
       "      <td>0.0072</td>\n",
       "      <td>0.0167</td>\n",
       "      <td>0.0180</td>\n",
       "      <td>0.0084</td>\n",
       "      <td>0.0090</td>\n",
       "      <td>0.0032</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.0453</td>\n",
       "      <td>0.0523</td>\n",
       "      <td>0.0843</td>\n",
       "      <td>0.0689</td>\n",
       "      <td>0.1183</td>\n",
       "      <td>0.2583</td>\n",
       "      <td>0.2156</td>\n",
       "      <td>0.3481</td>\n",
       "      <td>0.3337</td>\n",
       "      <td>0.2872</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0125</td>\n",
       "      <td>0.0084</td>\n",
       "      <td>0.0089</td>\n",
       "      <td>0.0048</td>\n",
       "      <td>0.0094</td>\n",
       "      <td>0.0191</td>\n",
       "      <td>0.0140</td>\n",
       "      <td>0.0049</td>\n",
       "      <td>0.0052</td>\n",
       "      <td>0.0044</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.0262</td>\n",
       "      <td>0.0582</td>\n",
       "      <td>0.1099</td>\n",
       "      <td>0.1083</td>\n",
       "      <td>0.0974</td>\n",
       "      <td>0.2280</td>\n",
       "      <td>0.2431</td>\n",
       "      <td>0.3771</td>\n",
       "      <td>0.5598</td>\n",
       "      <td>0.6194</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0033</td>\n",
       "      <td>0.0232</td>\n",
       "      <td>0.0166</td>\n",
       "      <td>0.0095</td>\n",
       "      <td>0.0180</td>\n",
       "      <td>0.0244</td>\n",
       "      <td>0.0316</td>\n",
       "      <td>0.0164</td>\n",
       "      <td>0.0095</td>\n",
       "      <td>0.0078</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>3 rows × 60 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "       0       1       2       3       4       5       6       7       8   \\\n",
       "0  0.0200  0.0371  0.0428  0.0207  0.0954  0.0986  0.1539  0.1601  0.3109   \n",
       "1  0.0453  0.0523  0.0843  0.0689  0.1183  0.2583  0.2156  0.3481  0.3337   \n",
       "2  0.0262  0.0582  0.1099  0.1083  0.0974  0.2280  0.2431  0.3771  0.5598   \n",
       "\n",
       "       9   ...      50      51      52      53      54      55      56  \\\n",
       "0  0.2111  ...  0.0232  0.0027  0.0065  0.0159  0.0072  0.0167  0.0180   \n",
       "1  0.2872  ...  0.0125  0.0084  0.0089  0.0048  0.0094  0.0191  0.0140   \n",
       "2  0.6194  ...  0.0033  0.0232  0.0166  0.0095  0.0180  0.0244  0.0316   \n",
       "\n",
       "       57      58      59  \n",
       "0  0.0084  0.0090  0.0032  \n",
       "1  0.0049  0.0052  0.0044  \n",
       "2  0.0164  0.0095  0.0078  \n",
       "\n",
       "[3 rows x 60 columns]"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X = df.drop(60, axis=1)\n",
    "y = df[60]\n",
    "X.head(3)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "47768dcf",
   "metadata": {},
   "source": [
    "**Train|Test Split**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "c707be55",
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.33, random_state=101)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ef04cb3f",
   "metadata": {},
   "source": [
    "**Scaling Data**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "01176feb",
   "metadata": {},
   "outputs": [],
   "source": [
    "scaler = StandardScaler()\n",
    "X_train = scaler.fit_transform(X_train)\n",
    "X_test = scaler.transform(X_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5696bada",
   "metadata": {},
   "source": [
    "### Parameters\n",
    "(Based on the article)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "8b3d2594",
   "metadata": {},
   "outputs": [],
   "source": [
    "max_iter = 50\n",
    "particles_num = 20\n",
    "c1 = 2\n",
    "c2 = 2\n",
    "V_max = 4\n",
    "V_min = -4\n",
    "alpha = 0.65\n",
    "\n",
    "seed = 454\n",
    "features = X.columns\n",
    "features_num = len(features)\n",
    "salient_features = None\n",
    "similar_features = []\n",
    "dissimilar_features = []"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6391b0e6",
   "metadata": {},
   "source": [
    "## Functions "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0ed0ff8a",
   "metadata": {},
   "source": [
    "#### Determining the number of features\n",
    "- The `saliet features` is randomly selected in the range [x, M] where M = ε . f and x generally is set to 3.\n",
    "- `lsf` is the probability value of determining `sf` as an initial number of features\n",
    "$$\n",
    "l_{sf} = \\frac{f - sf}{\\sum_{i=1}^{l = f-sf}(f-i)}\n",
    "$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "a11ef814",
   "metadata": {},
   "outputs": [],
   "source": [
    "def sf_determiner():\n",
    "    # seed\n",
    "    np.random.seed(seed)\n",
    "    random.seed(seed)\n",
    "        \n",
    "    # define M and interval of choosing sf\n",
    "    # based on article epsilon is in range of [0.15, 0.7]\n",
    "    epsilon = (np.random.randint(15,70))/100\n",
    "    M = int(epsilon * features_num)\n",
    "    # if M is less than x in [x,M]\n",
    "    M = 4 if M <= 3 else M\n",
    "    \n",
    "    # the probability value of determining salient features as an initial number of features : l_sf\n",
    "    l_sf = []\n",
    "    \n",
    "    for feature in range(1,features_num):\n",
    "        # nominator\n",
    "        l = features_num - feature\n",
    "        \n",
    "        # denominator\n",
    "        sums = 0\n",
    "        for i in range(l):\n",
    "            sums += features_num - i \n",
    "            \n",
    "        l_sf.append(round(l / sums, 3))\n",
    "    \n",
    "    salient_features = random.choices([i for i in range(3,M)], weights= l_sf[3:M], k=1)[0]\n",
    "    return salient_features    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "96832af4",
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "8"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "salient_features = sf_determiner()\n",
    "salient_features"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "40cd3b80",
   "metadata": {},
   "source": [
    "#### Grouping of the features\n",
    "- In this step, the features are divided into similar and dissimilar groups by Pearson correlation coefficient using equation\n",
    "\n",
    "$$\n",
    "corr_i = \\frac{\\sum_{j=1}^{f}|c_{ij}|}{f - 1 } \\space if \\space  \\space i ≠ j\n",
    "$$\n",
    "\n",
    "- divide the original feature set into two equal groups.\n",
    "- sorts all of the features in ascending order according to their correlation values. Those in the first half of the features have the lowest correlation values and they are put into the dissimilar group,while the rest of the fea-tures have higher correlation values and they are included in the second group called the similar group"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "54769909",
   "metadata": {},
   "outputs": [],
   "source": [
    "def dividing_features():\n",
    "    corrs = []\n",
    "    # find each column correlation \n",
    "    for i in range(1,features_num):\n",
    "        # Pearson Correlation Coefficient\n",
    "        corr_i = X.corrwith(X[i])\n",
    "        # if i = j\n",
    "        corr_i[i] = 0\n",
    "        \n",
    "        corrs.append( (round((sum(abs(i) for i in corr_i) / (features_num - 1) ),3), i))\n",
    "        corrs.sort()\n",
    "    \n",
    "    # dividing into Similar and Dissimilar\n",
    "    # first half => Dissmilar\n",
    "    dissimilar_features = corrs[:len(corrs)//2]\n",
    "    # seconf half => Similar\n",
    "    similar_features = corrs[len(corrs)//2:]\n",
    "    \n",
    "    return dissimilar_features, similar_features"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "91ff6411",
   "metadata": {},
   "outputs": [],
   "source": [
    "dissimilar_features, similar_features = dividing_features()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "a813b314",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[(0.173, 11), (0.184, 5), (0.184, 52), (0.185, 59), (0.186, 56), (0.187, 6), (0.198, 9), (0.2, 55), (0.201, 8), (0.202, 49), (0.202, 50), (0.202, 51), (0.208, 4), (0.21, 53), (0.211, 10), (0.212, 23), (0.215, 36), (0.216, 27), (0.217, 22), (0.222, 7), (0.223, 12), (0.223, 54), (0.224, 24), (0.224, 31), (0.225, 3), (0.225, 29), (0.225, 30), (0.227, 58), (0.228, 35)]\n"
     ]
    }
   ],
   "source": [
    "print(dissimilar_features)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "588812a9",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[(0.229, 28), (0.23, 48), (0.23, 57), (0.232, 46), (0.233, 21), (0.233, 43), (0.235, 2), (0.236, 34), (0.237, 20), (0.238, 25), (0.238, 33), (0.238, 39), (0.241, 32), (0.242, 26), (0.248, 47), (0.249, 42), (0.252, 38), (0.254, 37), (0.255, 19), (0.257, 1), (0.26, 41), (0.261, 16), (0.262, 17), (0.263, 18), (0.265, 45), (0.271, 40), (0.271, 44), (0.272, 13), (0.277, 15), (0.281, 14)]\n"
     ]
    }
   ],
   "source": [
    "print(similar_features)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "aa95adaa",
   "metadata": {},
   "source": [
    "#### Initializing particles\n",
    "\n",
    "- each particle is represented by a binary vector\n",
    "- The length of the vector is equal to the number of the original features\n",
    "- In this type of representation if the value of a cell in the vector is set to 1, it denotes that the corresponding feature is selected and when the value is set to 0, it means that the corresponding feature is not selected\n",
    "- for each particle a velocity vector is generated by using a random float number generator"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "b10e5c6e",
   "metadata": {},
   "outputs": [],
   "source": [
    "def initializing_particles():\n",
    "    # seed\n",
    "    np.random.seed(seed)\n",
    "    \n",
    "    # random initial Velocities\n",
    "    velocities = [\n",
    "        [round(np.random.rand(),3) for i in range(features_num)] for i in range(particles_num)\n",
    "    ]\n",
    "    \n",
    "    # random initial particles\n",
    "    particles = []\n",
    "    salient_features = sf_determiner()\n",
    "    \n",
    "    sample_particle = [0]*(features_num-salient_features) + [1]*salient_features\n",
    "    \n",
    "    for particle in range(particles_num):\n",
    "        np.random.shuffle(sample_particle)\n",
    "        random_particle = sample_particle.copy()\n",
    "        \n",
    "        particles.append(random_particle)\n",
    "    \n",
    "    \n",
    "    return particles , velocities"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "4e247149",
   "metadata": {},
   "outputs": [],
   "source": [
    "Particles, Velocities =  initializing_particles()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "72492c94",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(20, 60)"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.array(Particles).shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "4dcd3830",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(20, 60)"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.array(Velocities).shape"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "944a7f87",
   "metadata": {},
   "source": [
    "#### Updating particle positions\n",
    "- In PSO, each particle changes its position according to its velocityas follows\n",
    "$$ v_{id}(t+1) = v_{id}(t) + c_1 . r_{i,1} . (x^{best}_i −  x_{id}(t)) . v_i(t) + c_2 . r_{i,2} . (x^{best}_g − x_{id}(t)$$\n",
    "\n",
    "- if the sum of accelerations causes the velocity of that dimension to exceed $V_{max}$, then the velocity of that dimension is limited to $V_{max}$\n",
    "\n",
    "- The position of a particle is changed based on the following equations\n",
    "\n",
    "    if $ rand < sigmoid (v_{id}(t+1)) $ then $ x_{id}(t+1) = 1$ else $x_{id}(t+1) =0 $"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "6f4c6ab9",
   "metadata": {},
   "outputs": [],
   "source": [
    "def update_particle_positions(particles, velocities, best_particles, best_global):\n",
    "    # seed\n",
    "    np.random.seed(seed)\n",
    "    \n",
    "    new_velocities = np.zeros_like(velocities)\n",
    "    new_positions = np.zeros_like(particles)\n",
    "    \n",
    "    # Particle changes \n",
    "    for particle in range(particles_num):\n",
    "        for d in range(features_num):\n",
    "            new_velocities[particle][d] = velocities[particle][d] + c1 * np.random.rand() * (best_particles[particle][d] - particles[particle][d]) * velocities[particle][d] + c2 * np.random.rand() * (best_global[d] - particles[particle][d])\n",
    "            \n",
    "            # velocities limitation\n",
    "            if new_velocities[particle][d] > V_max : \n",
    "                new_velocities[particle][d] = 4  \n",
    "            if new_velocities[particle][d] < V_min : \n",
    "                new_velocities[particle][d] = -4 \n",
    "            \n",
    "            # Changing Particle Position \n",
    "            # sigmoid func\n",
    "            sigmoid = lambda x: 1 / (1 + np.exp(-x)) \n",
    "            new_positions[particle][d] = 1 if np.random.rand() < sigmoid(new_velocities[particle][d]) else 0\n",
    "            \n",
    "    return new_positions, new_velocities"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "8f3c34f4",
   "metadata": {},
   "outputs": [],
   "source": [
    "# new_positions, new_velocities = update_particle_positions(Particles, Velocities, ?, ?)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "df9c1d58",
   "metadata": {},
   "source": [
    "#### Local Search \n",
    "- two steps are considered, including (1) feature segmentation, and (2) particle movement\n",
    "- In these steps for a given particle the “Add” and “Delete” operators are employed to improve the local search of a particle\n",
    "- The `Add` operator inserts the dissimilar features into the particle and the similar features are deleted from the particle using the `Delete` operator\n",
    "- **The main idea behind the local search is to select distinct features with the lowest correlation.**\n",
    "- X is array of feature index numbers, each element of X is compared with those of D and then the elements of X are segmented into Xd and Xs. Specifically, Xd includes the dissimilar features that are included in D and their corresponding values in the newly generated particle, while Xs includes the similar features of X which are included in both X and S\n",
    "- The numbers of similar and dissimilar features are given by calculating the values of `ns` and `nd`, respectively\n",
    "- Here, `ns= alpha`  . sf and `nd=(1 − alpha).sf` where `alpha` is a user specific parameter\n",
    "- The number of features should be kept constant during the improvement of a particle by adding the most dissimilar features and deleting the least dissimilar ones\n",
    "- When the number of dissimilar features in the particle is smaller than `nd`, then (nd− Xd) features in (D-Xd) are added to the particle, otherwise (Xd− nd) features in Xd should be deleted from the particle"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "9d6ceb2b",
   "metadata": {},
   "outputs": [],
   "source": [
    "def local_search(particles):\n",
    "    # based on article\n",
    "    N_s = round(alpha * salient_features)\n",
    "    N_d = round((1-alpha) * salient_features)\n",
    "    \n",
    "    # store t+1 particle positions\n",
    "    updated_particles = []\n",
    "    \n",
    "    for particle in range(particles_num):\n",
    "#         print('before:',particles[particle])\n",
    "        # Define X_d and X_s \n",
    "        features_index = [index for index,f in enumerate(particles[particle]) if f == 1]\n",
    "#         print('N_s:', N_s)\n",
    "#         print('N_d:', N_d)\n",
    "        X_s = []\n",
    "        X_d = []\n",
    "        \n",
    "        # Adding features to X_s and X_d\n",
    "        for i in similar_features:\n",
    "            if i[1] in features_index: X_s.append(i)\n",
    "        for i in dissimilar_features:\n",
    "            if i[1] in features_index: X_d.append(i)\n",
    "        \n",
    "        # ADD and DELETE Operators\n",
    "        if (N_s - len(X_s)) > 0:\n",
    "            for i in range(N_s - len(X_s)):\n",
    "                for j in similar_features:\n",
    "                    if j not in X_s:\n",
    "                        X_s.append(j)\n",
    "                        break\n",
    "        elif (N_s - len(X_s)) < 0:\n",
    "            for i in range(abs(N_s - len(X_s))):\n",
    "                for j in X_s[::-1]:\n",
    "                    X_s.pop()\n",
    "                    break\n",
    "        if 0 < (N_d - len(X_d)):\n",
    "            for i in range(N_d - len(X_d)):\n",
    "                for j in dissimilar_features:\n",
    "                    if j not in X_d:\n",
    "                        X_d.append(j)\n",
    "                        break\n",
    "        elif (N_d - len(X_d)) < 0:\n",
    "            for i in range(abs(N_d - len(X_d))):\n",
    "                for j in X_d[::-1]:\n",
    "                    X_d.pop()\n",
    "                    break\n",
    "        \n",
    "        # Only save features\n",
    "        X_s = [i[1] for i in X_s]\n",
    "        X_d = [i[1] for i in X_d]\n",
    "        \n",
    "        # Features \n",
    "        X = X_d + X_s\n",
    "        \n",
    "        # Add features to particle\n",
    "        particle_sample = [0]* features_num\n",
    "        for feature_index in X:\n",
    "            particle_sample[feature_index] = 1\n",
    "        new_partcle = particle_sample.copy()\n",
    "        updated_particles.append(new_partcle)\n",
    "        \n",
    "#         print('after:', new_partcle)\n",
    "        \n",
    "    return updated_particles"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "eb7ce313",
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[0, 1, 0, 1, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]\n"
     ]
    }
   ],
   "source": [
    "print(local_search(Particles)[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d8a55148",
   "metadata": {},
   "source": [
    "#### Calculating fitness\n",
    "- The proposed method, employs the k-NN classifier to evaluate a candidate feature subset solution\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "c5a808b4",
   "metadata": {},
   "outputs": [],
   "source": [
    "def calculate_fitness(particles):\n",
    "    scores = []\n",
    "    for particle in particles:\n",
    "        # Define KNN model with CV = 10 as article suggested\n",
    "        model = KNeighborsClassifier()\n",
    "        params = {'n_neighbors' : [5]}\n",
    "        grid_model = GridSearchCV(model, params, cv=10)\n",
    "\n",
    "        # Select salient features\n",
    "        X_tr = pd.DataFrame(X_train)[[j for i,j in zip(particle,pd.DataFrame(X_train).columns) if i==1]] \n",
    "        X_tst = pd.DataFrame(X_test)[[j for i,j in zip(particle,pd.DataFrame(X_test).columns) if i==1]]\n",
    "        \n",
    "        # fit model \n",
    "        grid_model.fit(X_tr,y_train)\n",
    "        # predict model\n",
    "        y_pred = grid_model.predict(X_tst)\n",
    "        # evaluate model \n",
    "        scores.append(round(accuracy_score(y_test,y_pred),3))\n",
    "        \n",
    "    return scores "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "6b3531fe",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[0.725, 0.609, 0.783, 0.681, 0.652, 0.71, 0.681, 0.696, 0.681, 0.71, 0.681, 0.681, 0.725, 0.667, 0.681, 0.681, 0.71, 0.797, 0.71, 0.696]\n"
     ]
    }
   ],
   "source": [
    "print(calculate_fitness(Particles))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cb55429a",
   "metadata": {},
   "source": [
    "# The HPSO-LS Algorithm"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "8f3ea9a9",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "salient features:  6 \n",
      "\n",
      "Particles Score: ▼\n",
      "0.71\n",
      "0.725\n",
      "0.638\n",
      "0.797\n",
      "0.652\n",
      "0.768\n",
      "0.609\n",
      "0.71\n",
      "0.652\n",
      "0.725\n",
      "Best Global: 000000000001000000001000000000000010000000000010000000000101\n",
      "Best Score: 0.797 \n",
      "\n",
      "Particles Score: ▼\n",
      "0.754\n",
      "0.739\n",
      "0.739\n",
      "0.797\n",
      "0.667\n",
      "0.681\n",
      "0.623\n",
      "0.826\n",
      "0.652\n",
      "0.725\n",
      "Best Global: 001000000001000000001000000000000010000000000000000000000101\n",
      "Best Score: 0.826 \n",
      "\n",
      "Particles Score: ▼\n",
      "0.725\n",
      "0.696\n",
      "0.739\n",
      "0.754\n",
      "0.725\n",
      "0.623\n",
      "0.667\n",
      "0.826\n",
      "0.71\n",
      "0.739\n",
      "Best Global: 001000000001000000001000000000000010000000000000000000000101\n",
      "Best Score: 0.826 \n",
      "\n",
      "Particles Score: ▼\n",
      "0.783\n",
      "0.696\n",
      "0.725\n",
      "0.797\n",
      "0.725\n",
      "0.826\n",
      "0.667\n",
      "0.826\n",
      "0.71\n",
      "0.725\n",
      "Best Global: 001000000001000000001000000000000010000000000000000000000101\n",
      "Best Score: 0.826 \n",
      "\n",
      "Particles Score: ▼\n",
      "0.754\n",
      "0.696\n",
      "0.725\n",
      "0.826\n",
      "0.667\n",
      "0.826\n",
      "0.71\n",
      "0.826\n",
      "0.696\n",
      "0.609\n",
      "Best Global: 001000000001000000001000000000000010000000000000000000000101\n",
      "Best Score: 0.826 \n",
      "\n",
      "Particles Score: ▼\n",
      "0.739\n",
      "0.696\n",
      "0.725\n",
      "0.826\n",
      "0.667\n",
      "0.826\n",
      "0.71\n",
      "0.826\n",
      "0.696\n",
      "0.667\n",
      "Best Global: 001000000001000000001000000000000010000000000000000000000101\n",
      "Best Score: 0.826 \n",
      "\n",
      "Particles Score: ▼\n",
      "0.783\n",
      "0.696\n",
      "0.797\n",
      "0.826\n",
      "0.826\n",
      "0.826\n",
      "0.725\n",
      "0.826\n",
      "0.667\n",
      "0.71\n",
      "Best Global: 001000000001000000001000000000000010000000000000000000000101\n",
      "Best Score: 0.826 \n",
      "\n",
      "Particles Score: ▼\n",
      "0.783\n",
      "0.696\n",
      "0.797\n",
      "0.826\n",
      "0.826\n",
      "0.826\n",
      "0.696\n",
      "0.826\n",
      "0.609\n",
      "0.739\n",
      "Best Global: 001000000001000000001000000000000010000000000000000000000101\n",
      "Best Score: 0.826 \n",
      "\n",
      "Particles Score: ▼\n",
      "0.681\n",
      "0.696\n",
      "0.797\n",
      "0.826\n",
      "0.826\n",
      "0.826\n",
      "0.696\n",
      "0.826\n",
      "0.638\n",
      "0.696\n",
      "Best Global: 001000000001000000001000000000000010000000000000000000000101\n",
      "Best Score: 0.826 \n",
      "\n",
      "Particles Score: ▼\n",
      "0.783\n",
      "0.696\n",
      "0.826\n",
      "0.826\n",
      "0.826\n",
      "0.826\n",
      "0.696\n",
      "0.826\n",
      "0.638\n",
      "0.768\n",
      "Best Global: 001000000001000000001000000000000010000000000000000000000101\n",
      "Best Score: 0.826 \n",
      "\n",
      "Particles Score: ▼\n",
      "0.783\n",
      "0.696\n",
      "0.826\n",
      "0.826\n",
      "0.826\n",
      "0.826\n",
      "0.696\n",
      "0.826\n",
      "0.754\n",
      "0.768\n",
      "Best Global: 001000000001000000001000000000000010000000000000000000000101\n",
      "Best Score: 0.826 \n",
      "\n",
      "Particles Score: ▼\n",
      "0.783\n",
      "0.696\n",
      "0.826\n",
      "0.826\n",
      "0.826\n",
      "0.826\n",
      "0.696\n",
      "0.826\n",
      "0.754\n",
      "0.768\n",
      "Best Global: 001000000001000000001000000000000010000000000000000000000101\n",
      "Best Score: 0.826 \n",
      "\n",
      "Particles Score: ▼\n",
      "0.783\n",
      "0.696\n",
      "0.826\n",
      "0.826\n",
      "0.826\n",
      "0.826\n",
      "0.696\n",
      "0.826\n",
      "0.754\n",
      "0.768\n",
      "Best Global: 001000000001000000001000000000000010000000000000000000000101\n",
      "Best Score: 0.826 \n",
      "\n",
      "Particles Score: ▼\n",
      "0.783\n",
      "0.696\n",
      "0.826\n",
      "0.826\n",
      "0.826\n",
      "0.826\n",
      "0.696\n",
      "0.826\n",
      "0.754\n",
      "0.739\n",
      "Best Global: 001000000001000000001000000000000010000000000000000000000101\n",
      "Best Score: 0.826 \n",
      "\n",
      "Particles Score: ▼\n",
      "0.681\n",
      "0.681\n",
      "0.826\n",
      "0.826\n",
      "0.826\n",
      "0.826\n",
      "0.696\n",
      "0.826\n",
      "0.652\n",
      "0.739\n",
      "Best Global: 001000000001000000001000000000000010000000000000000000000101\n",
      "Best Score: 0.826 \n",
      "\n",
      "Particles Score: ▼\n",
      "0.783\n",
      "0.681\n",
      "0.826\n",
      "0.826\n",
      "0.826\n",
      "0.826\n",
      "0.696\n",
      "0.826\n",
      "0.652\n",
      "0.739\n",
      "Best Global: 001000000001000000001000000000000010000000000000000000000101\n",
      "Best Score: 0.826 \n",
      "\n",
      "Particles Score: ▼\n",
      "0.783\n",
      "0.681\n",
      "0.826\n",
      "0.826\n",
      "0.826\n",
      "0.826\n",
      "0.696\n",
      "0.826\n",
      "0.754\n",
      "0.754\n",
      "Best Global: 001000000001000000001000000000000010000000000000000000000101\n",
      "Best Score: 0.826 \n",
      "\n",
      "Particles Score: ▼\n",
      "0.783\n",
      "0.681\n",
      "0.826\n",
      "0.826\n",
      "0.826\n",
      "0.826\n",
      "0.696\n",
      "0.826\n",
      "0.754\n",
      "0.739\n",
      "Best Global: 001000000001000000001000000000000010000000000000000000000101\n",
      "Best Score: 0.826 \n",
      "\n",
      "Particles Score: ▼\n",
      "0.783\n",
      "0.681\n",
      "0.826\n",
      "0.826\n",
      "0.826\n",
      "0.826\n",
      "0.696\n",
      "0.826\n",
      "0.71\n",
      "0.667\n",
      "Best Global: 001000000001000000001000000000000010000000000000000000000101\n",
      "Best Score: 0.826 \n",
      "\n",
      "Particles Score: ▼\n",
      "0.783\n",
      "0.681\n",
      "0.826\n",
      "0.826\n",
      "0.826\n",
      "0.826\n",
      "0.696\n",
      "0.826\n",
      "0.71\n",
      "0.71\n",
      "Best Global: 001000000001000000001000000000000010000000000000000000000101\n",
      "Best Score: 0.826 \n",
      "\n"
     ]
    }
   ],
   "source": [
    "max_iter = 20\n",
    "particles_num = 10\n",
    "c1 = 2\n",
    "c2 = 2\n",
    "V_max = 4\n",
    "V_min = -4\n",
    "alpha = 0.65\n",
    "\n",
    "seed = 555\n",
    "features = X.columns\n",
    "features_num = len(features)\n",
    "salient_features = None\n",
    "similar_features = []\n",
    "dissimilar_features = []\n",
    "\n",
    "# finding salient features \n",
    "salient_features = sf_determiner()\n",
    "print('salient features: ',salient_features, '\\n')\n",
    "\n",
    "# finding similar and dissmilar features \n",
    "dissimilar_features, similar_features = dividing_features()\n",
    "\n",
    "# initializing \n",
    "Particles, Velocities= initializing_particles()\n",
    "\n",
    "# finding the initial scores\n",
    "scores = calculate_fitness(Particles)\n",
    "\n",
    "# evaluate particles\n",
    "best_particles = [Particles[i] for i in range(len(Particles))]\n",
    "\n",
    "# finding best global \n",
    "best_global = Particles[scores.index(max(scores))]\n",
    "best_global_score = max(scores)\n",
    "\n",
    "\n",
    "for iteration in range(max_iter):\n",
    "    # updating t+1 particle posistions \n",
    "    new_particles, new_velocities = update_particle_positions(Particles, Velocities, best_particles, best_global)\n",
    "    \n",
    "    # finding t+1 particle posistions \n",
    "    new_particles = local_search(new_particles)\n",
    "    \n",
    "    # evaluate the t+1 particles\n",
    "    new_scores = calculate_fitness(new_particles)\n",
    "\n",
    "    # keep better particles \n",
    "    dummy_particles = []\n",
    "    for particle in range(particles_num):\n",
    "        if new_scores[particle] < scores[particle]:\n",
    "            dummy_particles.append(Particles[particle])\n",
    "        else:\n",
    "            dummy_particles.append(new_particles[particle])\n",
    "\n",
    "        # update best_global and best_global_score\n",
    "        if best_global_score < new_scores[particle] :\n",
    "            best_global = new_particles[particle]\n",
    "            best_global_score = new_scores[particle]\n",
    "\n",
    "    Particles = new_particles\n",
    "    Velocities = new_velocities\n",
    "\n",
    "    best_particles = dummy_particles.copy()\n",
    "    scores = new_scores.copy()\n",
    "    \n",
    "    # printing evaluation of particles\n",
    "    print('Particles Score: ▼')\n",
    "    for i, particle in enumerate(best_particles):\n",
    "        print(scores[i])\n",
    "\n",
    "    print('Best Global: ', end='')\n",
    "    for i in range(features_num):\n",
    "        print(best_global[i], end='')\n",
    "    print('\\nBest Score:', best_global_score, '\\n')  \n",
    "        "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "937fa197",
   "metadata": {},
   "source": [
    "---\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2e0e2d1d",
   "metadata": {},
   "source": [
    "# Wine DATA"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "c18d54e9",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "      <th>5</th>\n",
       "      <th>6</th>\n",
       "      <th>7</th>\n",
       "      <th>8</th>\n",
       "      <th>9</th>\n",
       "      <th>10</th>\n",
       "      <th>11</th>\n",
       "      <th>12</th>\n",
       "      <th>13</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>14.23</td>\n",
       "      <td>1.71</td>\n",
       "      <td>2.43</td>\n",
       "      <td>15.6</td>\n",
       "      <td>127</td>\n",
       "      <td>2.80</td>\n",
       "      <td>3.06</td>\n",
       "      <td>0.28</td>\n",
       "      <td>2.29</td>\n",
       "      <td>5.64</td>\n",
       "      <td>1.04</td>\n",
       "      <td>3.92</td>\n",
       "      <td>1065</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>13.20</td>\n",
       "      <td>1.78</td>\n",
       "      <td>2.14</td>\n",
       "      <td>11.2</td>\n",
       "      <td>100</td>\n",
       "      <td>2.65</td>\n",
       "      <td>2.76</td>\n",
       "      <td>0.26</td>\n",
       "      <td>1.28</td>\n",
       "      <td>4.38</td>\n",
       "      <td>1.05</td>\n",
       "      <td>3.40</td>\n",
       "      <td>1050</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1</td>\n",
       "      <td>13.16</td>\n",
       "      <td>2.36</td>\n",
       "      <td>2.67</td>\n",
       "      <td>18.6</td>\n",
       "      <td>101</td>\n",
       "      <td>2.80</td>\n",
       "      <td>3.24</td>\n",
       "      <td>0.30</td>\n",
       "      <td>2.81</td>\n",
       "      <td>5.68</td>\n",
       "      <td>1.03</td>\n",
       "      <td>3.17</td>\n",
       "      <td>1185</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1</td>\n",
       "      <td>14.37</td>\n",
       "      <td>1.95</td>\n",
       "      <td>2.50</td>\n",
       "      <td>16.8</td>\n",
       "      <td>113</td>\n",
       "      <td>3.85</td>\n",
       "      <td>3.49</td>\n",
       "      <td>0.24</td>\n",
       "      <td>2.18</td>\n",
       "      <td>7.80</td>\n",
       "      <td>0.86</td>\n",
       "      <td>3.45</td>\n",
       "      <td>1480</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1</td>\n",
       "      <td>13.24</td>\n",
       "      <td>2.59</td>\n",
       "      <td>2.87</td>\n",
       "      <td>21.0</td>\n",
       "      <td>118</td>\n",
       "      <td>2.80</td>\n",
       "      <td>2.69</td>\n",
       "      <td>0.39</td>\n",
       "      <td>1.82</td>\n",
       "      <td>4.32</td>\n",
       "      <td>1.04</td>\n",
       "      <td>2.93</td>\n",
       "      <td>735</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   0      1     2     3     4    5     6     7     8     9     10    11    12  \\\n",
       "0   1  14.23  1.71  2.43  15.6  127  2.80  3.06  0.28  2.29  5.64  1.04  3.92   \n",
       "1   1  13.20  1.78  2.14  11.2  100  2.65  2.76  0.26  1.28  4.38  1.05  3.40   \n",
       "2   1  13.16  2.36  2.67  18.6  101  2.80  3.24  0.30  2.81  5.68  1.03  3.17   \n",
       "3   1  14.37  1.95  2.50  16.8  113  3.85  3.49  0.24  2.18  7.80  0.86  3.45   \n",
       "4   1  13.24  2.59  2.87  21.0  118  2.80  2.69  0.39  1.82  4.32  1.04  2.93   \n",
       "\n",
       "     13  \n",
       "0  1065  \n",
       "1  1050  \n",
       "2  1185  \n",
       "3  1480  \n",
       "4   735  "
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = pd.read_csv(\"./Wine/wine.data\",header=None)\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "41660be0",
   "metadata": {},
   "outputs": [],
   "source": [
    "X = df.drop(0, axis=1)\n",
    "y = df[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "29ab7482",
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.33, random_state=101)\n",
    "X_train = scaler.fit_transform(X_train)\n",
    "X_test = scaler.transform(X_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9e5f70f1",
   "metadata": {},
   "source": [
    "## HPSO-LS on Wine Dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "ac63b6af",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "salient_features: 3\n",
      "Best Global: 0000000100101\n",
      "Best Score: 0.949 \n",
      "\n"
     ]
    }
   ],
   "source": [
    "# New Parameters \n",
    "max_iter = 50\n",
    "particles_num = 20\n",
    "c1 = 2\n",
    "c2 = 2\n",
    "V_max = 4\n",
    "V_min = -4\n",
    "alpha = 0.65\n",
    "\n",
    "seed = 1345\n",
    "features = X.columns\n",
    "features_num = len(features)\n",
    "salient_features = None\n",
    "similar_features = []\n",
    "dissimilar_features = []\n",
    "    \n",
    "    \n",
    "salient_features = sf_determiner()\n",
    "print('salient_features:', salient_features)\n",
    "\n",
    "dissimilar_features, similar_features = dividing_features()\n",
    "\n",
    "Particles, Velocities= initializing_particles()\n",
    "\n",
    "scores = calculate_fitness(Particles)\n",
    "\n",
    "best_particles = [Particles[i] for i in range(len(Particles))]\n",
    "# best_scores = scores\n",
    "best_global = Particles[scores.index(max(scores))]\n",
    "best_global_score = max(scores)\n",
    "\n",
    "for iteration in range(max_iter):\n",
    "    new_particles, new_velocities = update_particle_positions(Particles, Velocities, best_particles, best_global)\n",
    "\n",
    "    new_particles = local_search(new_particles)\n",
    "\n",
    "    new_scores = calculate_fitness(new_particles)\n",
    "\n",
    "\n",
    "    dummy_particles = []\n",
    "    for particle in range(particles_num):\n",
    "        if new_scores[particle] < scores[particle]:\n",
    "            dummy_particles.append(Particles[particle])\n",
    "        else:\n",
    "            dummy_particles.append(new_particles[particle])\n",
    "\n",
    "        # update best_global and best_global_score\n",
    "        if best_global_score < new_scores[particle] :\n",
    "            best_global = new_particles[particle]\n",
    "            best_global_score = new_scores[particle]\n",
    "\n",
    "    Particles = new_particles\n",
    "    Velocities = new_velocities\n",
    "\n",
    "    best_particles = dummy_particles.copy()\n",
    "    scores = new_scores.copy()\n",
    "\n",
    "#     for i, particle in enumerate(best_particles):\n",
    "#         print(particle.count(1),end=' ')\n",
    "#         print(scores[i])\n",
    "\n",
    "print('Best Global: ', end='')\n",
    "for i in range(features_num):\n",
    "    print(best_global[i], end='')\n",
    "print('\\nBest Score:', best_global_score, '\\n')       "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ee7a5781",
   "metadata": {},
   "source": [
    "---\n",
    "***"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "569c9374",
   "metadata": {},
   "source": [
    "# Heart Dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "id": "fd85b063",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "      <th>5</th>\n",
       "      <th>6</th>\n",
       "      <th>7</th>\n",
       "      <th>8</th>\n",
       "      <th>9</th>\n",
       "      <th>10</th>\n",
       "      <th>11</th>\n",
       "      <th>12</th>\n",
       "      <th>13</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>70.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>130.0</td>\n",
       "      <td>322.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>109.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.4</td>\n",
       "      <td>2.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>67.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>115.0</td>\n",
       "      <td>564.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>160.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.6</td>\n",
       "      <td>2.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>7.0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>57.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>124.0</td>\n",
       "      <td>261.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>141.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.3</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>7.0</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>64.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>128.0</td>\n",
       "      <td>263.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>105.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.2</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>7.0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>74.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>120.0</td>\n",
       "      <td>269.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>121.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.2</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>265</th>\n",
       "      <td>52.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>172.0</td>\n",
       "      <td>199.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>162.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.5</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>7.0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>266</th>\n",
       "      <td>44.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>120.0</td>\n",
       "      <td>263.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>173.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>7.0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>267</th>\n",
       "      <td>56.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>140.0</td>\n",
       "      <td>294.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>153.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.3</td>\n",
       "      <td>2.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>268</th>\n",
       "      <td>57.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>140.0</td>\n",
       "      <td>192.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>148.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.4</td>\n",
       "      <td>2.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>269</th>\n",
       "      <td>67.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>160.0</td>\n",
       "      <td>286.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>108.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.5</td>\n",
       "      <td>2.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>270 rows × 14 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "       0    1    2      3      4    5    6      7    8    9    10   11   12  \\\n",
       "0    70.0  1.0  4.0  130.0  322.0  0.0  2.0  109.0  0.0  2.4  2.0  3.0  3.0   \n",
       "1    67.0  0.0  3.0  115.0  564.0  0.0  2.0  160.0  0.0  1.6  2.0  0.0  7.0   \n",
       "2    57.0  1.0  2.0  124.0  261.0  0.0  0.0  141.0  0.0  0.3  1.0  0.0  7.0   \n",
       "3    64.0  1.0  4.0  128.0  263.0  0.0  0.0  105.0  1.0  0.2  2.0  1.0  7.0   \n",
       "4    74.0  0.0  2.0  120.0  269.0  0.0  2.0  121.0  1.0  0.2  1.0  1.0  3.0   \n",
       "..    ...  ...  ...    ...    ...  ...  ...    ...  ...  ...  ...  ...  ...   \n",
       "265  52.0  1.0  3.0  172.0  199.0  1.0  0.0  162.0  0.0  0.5  1.0  0.0  7.0   \n",
       "266  44.0  1.0  2.0  120.0  263.0  0.0  0.0  173.0  0.0  0.0  1.0  0.0  7.0   \n",
       "267  56.0  0.0  2.0  140.0  294.0  0.0  2.0  153.0  0.0  1.3  2.0  0.0  3.0   \n",
       "268  57.0  1.0  4.0  140.0  192.0  0.0  0.0  148.0  0.0  0.4  2.0  0.0  6.0   \n",
       "269  67.0  1.0  4.0  160.0  286.0  0.0  2.0  108.0  1.0  1.5  2.0  3.0  3.0   \n",
       "\n",
       "     13  \n",
       "0     2  \n",
       "1     1  \n",
       "2     2  \n",
       "3     1  \n",
       "4     1  \n",
       "..   ..  \n",
       "265   1  \n",
       "266   1  \n",
       "267   1  \n",
       "268   1  \n",
       "269   2  \n",
       "\n",
       "[270 rows x 14 columns]"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = pd.read_csv(\"./heart/heart.dat\",header=None, sep= ' ')\n",
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "639754b6",
   "metadata": {},
   "source": [
    "``` This database contains 13 attributes (which have been extracted from\n",
    "a larger set of 75)       \n",
    "  \n",
    "\n",
    "\n",
    "Attribute Information:\n",
    "------------------------\n",
    "      -- 1. age       \n",
    "      -- 2. sex       \n",
    "      -- 3. chest pain type  (4 values)       \n",
    "      -- 4. resting blood pressure  \n",
    "      -- 5. serum cholestoral in mg/dl      \n",
    "      -- 6. fasting blood sugar > 120 mg/dl       \n",
    "      -- 7. resting electrocardiographic results  (values 0,1,2) \n",
    "      -- 8. maximum heart rate achieved  \n",
    "      -- 9. exercise induced angina    \n",
    "      -- 10. oldpeak = ST depression induced by exercise relative to rest   \n",
    "      -- 11. the slope of the peak exercise ST segment     \n",
    "      -- 12. number of major vessels (0-3) colored by flourosopy        \n",
    "      -- 13.  thal: 3 = normal; 6 = fixed defect; 7 = reversable defect     \n",
    "\n",
    "Attributes types\n",
    "-----------------\n",
    "\n",
    "Real: 1,4,5,8,10,12\n",
    "Ordered:11,\n",
    "Binary: 2,6,9\n",
    "Nominal:7,3,13\n",
    "\n",
    "Variable to be predicted\n",
    "------------------------\n",
    "Absence (1) or presence (2) of heart disease\n",
    "\n",
    "Cost Matrix\n",
    "\n",
    "\t abse  pres\n",
    "absence\t  0\t1\n",
    "presence  5\t0\n",
    "\n",
    "where the rows represent the true values and the columns the predicted.\n",
    "\n",
    "No missing values.\n",
    "\n",
    "270 observations\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "id": "9cccb607",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAEGCAYAAACKB4k+AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/MnkTPAAAACXBIWXMAAAsTAAALEwEAmpwYAAAPKklEQVR4nO3df6zddX3H8ecLqs5fDLB3WPtjJdqwdM4FdsPYSIyTZYJzlqgzsClVWToz/DXNFDUZxoVEo9OhmySdIGUhIKKObjGbpMORJYJrEfnR6mxQoE2xVxF/brrqe3+cbz+71nvhcNtzvrf3PB/Jyf1+P9/vOd/XHw0vvr9TVUiSBHBM3wEkSYuHpSBJaiwFSVJjKUiSGktBktQs6zvA4Vi+fHmtXbu27xiSdFTZsWPHN6tqaq5lR3UprF27lu3bt/cdQ5KOKknum2+Zh48kSY2lIElqLAVJUmMpSJIaS0GS1FgKkqTGUpAkNZaCJKmxFCRJzVF9R/OR8Bt/cXXfEbQI7XjfBX1HkHrhnoIkqbEUJEmNpSBJaiwFSVJjKUiSGktBktRYCpKkxlKQJDWWgiSpGVkpJLkyyf4kd8+x7C1JKsnybj5JPpRkd5I7k5w2qlySpPmNck/hKuDsQweTrAZ+D7h/1vA5wLruswm4fIS5JEnzGFkpVNUtwENzLPog8FagZo1tAK6ugVuB45OsGFU2SdLcxnpOIckGYG9VfemQRSuBB2bN7+nGJEljNLanpCZ5EvAOBoeODud3NjE4xMSaNWuOQDJJ0kHj3FN4JnAy8KUkXwdWAbcneTqwF1g9a91V3djPqarNVTVdVdNTU1MjjixJk2VspVBVd1XVL1XV2qpay+AQ0WlV9SCwFbiguwrpDOA7VbVvXNkkSQOjvCT1WuDzwClJ9iS58BFW/wxwL7Ab+Hvgz0aVS5I0v5GdU6iq8x9l+dpZ0wVcNKoskqTheEezJKmxFCRJjaUgSWosBUlSYylIkhpLQZLUWAqSpMZSkCQ1loIkqbEUJEmNpSBJaiwFSVJjKUiSGktBktRYCpKkxlKQJDWWgiSpsRQkSY2lIElqRlYKSa5Msj/J3bPG3pfky0nuTPLpJMfPWvb2JLuTfCXJC0aVS5I0v1HuKVwFnH3I2E3As6vqOcB/AW8HSLIeOA/41e47H0ly7AizSZLmMLJSqKpbgIcOGftsVR3oZm8FVnXTG4DrqupHVfU1YDdw+qiySZLmtqzHbb8G+Hg3vZJBSRy0pxv7OUk2AZsA1qxZM8p8Uq/uf/ev9R1Bi9Cav7xrpL/fy4nmJO8EDgDXPNbvVtXmqpququmpqakjH06SJtjY9xSSvAp4EXBWVVU3vBdYPWu1Vd2YJGmMxrqnkORs4K3Ai6vqh7MWbQXOS/KEJCcD64AvjDObJGmEewpJrgWeByxPsge4hMHVRk8AbkoCcGtVvbaq7klyPbCTwWGli6rqJ6PKJkma28hKoarOn2P4ikdY/1Lg0lHlkSQ9Ou9oliQ1loIkqbEUJEmNpSBJaiwFSVJjKUiSGktBktRYCpKkxlKQJDWWgiSpsRQkSY2lIElqLAVJUmMpSJIaS0GS1FgKkqTGUpAkNZaCJKmxFCRJzchKIcmVSfYnuXvW2IlJbkry1e7vCd14knwoye4kdyY5bVS5JEnzG+WewlXA2YeMXQxsq6p1wLZuHuAcYF332QRcPsJckqR5jKwUquoW4KFDhjcAW7rpLcC5s8avroFbgeOTrBhVNknS3MZ9TuGkqtrXTT8InNRNrwQemLXenm7s5yTZlGR7ku0zMzOjSypJE6i3E81VVUAt4Hubq2q6qqanpqZGkEySJte4S+EbBw8LdX/3d+N7gdWz1lvVjUmSxmjcpbAV2NhNbwRunDV+QXcV0hnAd2YdZpIkjcmyUf1wkmuB5wHLk+wBLgHeA1yf5ELgPuDl3eqfAV4I7AZ+CLx6VLkkSfMbWSlU1fnzLDprjnULuGhUWSRJw/GOZklSYylIkhpLQZLUWAqSpMZSkCQ1loIkqbEUJEmNpSBJaiwFSVJjKUiSGktBktRYCpKkZqhSSLJtmDFJ0tHtEZ+SmuQXgCcxePz1CUC6Rccxz+syJUlHr0d7dPafAm8CngHs4P9L4bvA344uliSpD49YClV1GXBZktdX1YfHlEmS1JOhXrJTVR9O8tvA2tnfqaqrR5RLktSDoUohyT8AzwTuAH7SDRdgKUjSEjLs6zingfXdazMPW5I/B/6EQbHcxeCdzCuA64CnMTh/8cqq+vGR2J4kaTjD3qdwN/D0I7HBJCuBNwDTVfVs4FjgPOC9wAer6lnAt4ELj8T2JEnDG3ZPYTmwM8kXgB8dHKyqFx/Gdp+Y5H8ZXPK6D3g+8Efd8i3Au4DLF/j7kqQFGLYU3nWkNlhVe5O8H7gf+G/gswwOFz1cVQe61fYwz30QSTYBmwDWrFlzpGJJkhj+6qN/P1Ib7G6C2wCcDDwMfAI4e9jvV9VmYDPA9PT0ETnHIUkaGPbqo+8xOCkM8HjgccAPquq4BWzzd4GvVdVM99ufAs4Ejk+yrNtbWAXsXcBvS5IOw1AnmqvqqVV1XFcCTwReCnxkgdu8HzgjyZOSBDgL2AncDLysW2cjcOMCf1+StECP+SmpNfCPwAsWssGqug24AbidweWoxzA4HPQ24M1JdjO4LPWKhfy+JGnhhj189JJZs8cwuG/hfxa60aq6BLjkkOF7gdMX+puSpMM37NVHfzBr+gDwdQYniyVJS8iwVx+9etRBJEn9G/YlO6uSfDrJ/u7zySSrRh1OkjRew55o/hiwlcF7FZ4B/FM3JklaQoYthamq+lhVHeg+VwFTI8wlSerBsKXwrSSvSHJs93kF8K1RBpMkjd+wpfAa4OXAgwweXvcy4FUjyiRJ6smwl6S+G9hYVd8GSHIi8H4GZSFJWiKG3VN4zsFCAKiqh4BTRxNJktSXYUvhmO7ppkDbUxh2L0OSdJQY9j/sfw18Psknuvk/BC4dTSRJUl+GvaP56iTbGbwdDeAlVbVzdLEkSX0Y+hBQVwIWgSQtYY/50dmSpKXLUpAkNZaCJKmxFCRJjaUgSWosBUlS00spJDk+yQ1JvpxkV5LfSnJikpuSfLX7e8Kj/5Ik6Ujqa0/hMuBfqupXgF8HdgEXA9uqah2wrZuXJI3R2EshyS8CzwWuAKiqH1fVw8AGYEu32hbg3HFnk6RJ18eewsnADPCxJF9M8tEkTwZOqqp93ToPAifN9eUkm5JsT7J9ZmZmTJElaTL0UQrLgNOAy6vqVOAHHHKoqKoKqLm+XFWbq2q6qqanpnwjqCQdSX2Uwh5gT1Xd1s3fwKAkvpFkBUD3d38P2SRpoo29FKrqQeCBJKd0Q2cxeNDeVmBjN7YRuHHc2SRp0vX1opzXA9ckeTxwL/BqBgV1fZILgfsYvBNakjRGvZRCVd0BTM+x6KwxR5EkzeIdzZKkxlKQJDWWgiSpsRQkSY2lIElqLAVJUmMpSJIaS0GS1FgKkqTGUpAkNZaCJKmxFCRJjaUgSWosBUlSYylIkhpLQZLUWAqSpMZSkCQ1vZVCkmOTfDHJP3fzJye5LcnuJB/v3t8sSRqjPvcU3gjsmjX/XuCDVfUs4NvAhb2kkqQJ1kspJFkF/D7w0W4+wPOBG7pVtgDn9pFNkiZZX3sKfwO8FfhpN/804OGqOtDN7wFW9pBLkiba2EshyYuA/VW1Y4Hf35Rke5LtMzMzRzidJE22PvYUzgRenOTrwHUMDhtdBhyfZFm3zipg71xfrqrNVTVdVdNTU1PjyCtJE2PspVBVb6+qVVW1FjgP+Leq+mPgZuBl3WobgRvHnU2SJt1iuk/hbcCbk+xmcI7hip7zSNLEWfboq4xOVX0O+Fw3fS9wep95JGnSLaY9BUlSzywFSVJjKUiSGktBktRYCpKkxlKQJDWWgiSpsRQkSY2lIElqLAVJUmMpSJIaS0GS1FgKkqTGUpAkNZaCJKmxFCRJjaUgSWosBUlSYylIkpqxl0KS1UluTrIzyT1J3tiNn5jkpiRf7f6eMO5skjTp+thTOAC8parWA2cAFyVZD1wMbKuqdcC2bl6SNEZjL4Wq2ldVt3fT3wN2ASuBDcCWbrUtwLnjziZJk67XcwpJ1gKnArcBJ1XVvm7Rg8BJ83xnU5LtSbbPzMyMJ6gkTYjeSiHJU4BPAm+qqu/OXlZVBdRc36uqzVU1XVXTU1NTY0gqSZOjl1JI8jgGhXBNVX2qG/5GkhXd8hXA/j6ySdIk6+PqowBXALuq6gOzFm0FNnbTG4Ebx51Nkibdsh62eSbwSuCuJHd0Y+8A3gNcn+RC4D7g5T1kk6SJNvZSqKr/ADLP4rPGmUWS9LO8o1mS1FgKkqTGUpAkNZaCJKmxFCRJjaUgSWosBUlSYylIkhpLQZLUWAqSpMZSkCQ1loIkqbEUJEmNpSBJaiwFSVJjKUiSGktBktRYCpKkxlKQJDWLrhSSnJ3kK0l2J7m47zySNEkWVSkkORb4O+AcYD1wfpL1/aaSpMmxqEoBOB3YXVX3VtWPgeuADT1nkqSJsazvAIdYCTwwa34P8JuzV0iyCdjUzX4/yVfGlG0SLAe+2XeIxSDv39h3BP0s/20edEmOxK/88nwLFlspPKqq2gxs7jvHUpRke1VN951DOpT/NsdnsR0+2gusnjW/qhuTJI3BYiuF/wTWJTk5yeOB84CtPWeSpImxqA4fVdWBJK8D/hU4Friyqu7pOdYk8bCcFiv/bY5JqqrvDJKkRWKxHT6SJPXIUpAkNZaCSHJlkv1J7u47izRbktVJbk6yM8k9Sd7Yd6alznMKIslzge8DV1fVs/vOIx2UZAWwoqpuT/JUYAdwblXt7DnakuWegqiqW4CH+s4hHaqq9lXV7d3094BdDJ58oBGxFCQdFZKsBU4Fbus5ypJmKUha9JI8Bfgk8Kaq+m7feZYyS0HSopbkcQwK4Zqq+lTfeZY6S0HSopUkwBXArqr6QN95JoGlIJJcC3weOCXJniQX9p1J6pwJvBJ4fpI7us8L+w61lHlJqiSpcU9BktRYCpKkxlKQJDWWgiSpsRQkSY2lIB2muZ4ym+SvktzZXUL52STP6DOjNCwvSZUO01xPmU1y3MHHMSR5A7C+ql7bY0xpKO4pSIdprqfMHvJ8nicD/t+XjgrL+g4gLVVJLgUuAL4D/E7PcaShuKcgjUhVvbOqVgPXAK/rO480DEtBGr1rgJf2HUIahqUgjUCSdbNmNwBf7iuL9Fh4TkE6TN1TZp8HLE+yB7gEeGGSU4CfAvcBXnmko4KXpEqSGg8fSZIaS0GS1FgKkqTGUpAkNZaCJKmxFCRJjaUgSWr+D2gHk8yBz7nTAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "sns.countplot(x=df[13]);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "id": "f7f0a62d",
   "metadata": {},
   "outputs": [],
   "source": [
    "X = df.drop(13, axis=1)\n",
    "y = df[13]\n",
    "\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.33, random_state=101)\n",
    "X_train = scaler.fit_transform(X_train)\n",
    "X_test = scaler.transform(X_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cd00f83a",
   "metadata": {},
   "source": [
    "# HPSO-LS on Heart Dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "id": "e206e752",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "salient_features: 3\n",
      "Best Global : 0100000000011\n",
      "Best Score: 0.822 \n",
      "\n",
      "Best Global Features :1 11 12 "
     ]
    }
   ],
   "source": [
    "# New Parameters \n",
    "max_iter = 100\n",
    "particles_num = 20\n",
    "c1 = 2\n",
    "c2 = 2\n",
    "V_max = 4\n",
    "V_min = -4\n",
    "alpha = 0.65\n",
    "\n",
    "seed = 123\n",
    "features = X.columns\n",
    "features_num = len(features)\n",
    "salient_features = None\n",
    "similar_features = []\n",
    "dissimilar_features = []\n",
    "    \n",
    "    \n",
    "salient_features = sf_determiner()\n",
    "print('salient_features:', salient_features)\n",
    "\n",
    "dissimilar_features, similar_features = dividing_features()\n",
    "\n",
    "Particles, Velocities= initializing_particles()\n",
    "\n",
    "scores = calculate_fitness(Particles)\n",
    "\n",
    "best_particles = [Particles[i] for i in range(len(Particles))]\n",
    "# best_scores = scores\n",
    "best_global = Particles[scores.index(max(scores))]\n",
    "best_global_score = max(scores)\n",
    "\n",
    "for iteration in range(max_iter):\n",
    "    new_particles, new_velocities = update_particle_positions(Particles, Velocities, best_particles, best_global)\n",
    "\n",
    "    new_particles = local_search(new_particles)\n",
    "\n",
    "    new_scores = calculate_fitness(new_particles)\n",
    "\n",
    "\n",
    "    dummy_particles = []\n",
    "    for particle in range(particles_num):\n",
    "        if new_scores[particle] < scores[particle]:\n",
    "            dummy_particles.append(Particles[particle])\n",
    "        else:\n",
    "            dummy_particles.append(new_particles[particle])\n",
    "\n",
    "        # update best_global and best_global_score\n",
    "        if best_global_score < new_scores[particle] :\n",
    "            best_global = new_particles[particle]\n",
    "            best_global_score = new_scores[particle]\n",
    "\n",
    "    Particles = new_particles\n",
    "    Velocities = new_velocities\n",
    "\n",
    "    best_particles = dummy_particles.copy()\n",
    "    scores = new_scores.copy()\n",
    "\n",
    "#     for i, particle in enumerate(best_particles):\n",
    "#         print(particle.count(1),end=' ')\n",
    "#         print(scores[i])\n",
    "\n",
    "print('Best Global : ', end='')\n",
    "for i in range(features_num):\n",
    "    print(best_global[i], end='')\n",
    "print('\\nBest Score:', best_global_score, '\\n')\n",
    "print('Best Global Features :', end='')\n",
    "for i in range(features_num):\n",
    "    if best_global[i] == 1 : print(i, end=' ') "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5af0db8d",
   "metadata": {},
   "source": [
    "It seems that it found better features with better accuracy than article! :) "
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}