EGV_Data_exploreer.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "source": [
    "## Eigenvector Metal Etch Data for Fault Detection Evaluation\n",
    "Jaganadh Gopinadhan (Jagan)\n",
    "\n",
    "https://www.linkedin.com/in/jaganadhg/\n",
    "\n",
    "### Introduction\n",
    "\n",
    "This Data Set is one of the oldest Microfabrication/Semiconductor process-related open data. The data was released by Eigenvector, a niche industry analytics company. The information was used in a research study and paper published in 1999[1]. The first author Barry M. Wise is one of the company's founding members.\n",
    "\n",
    "The data was taken from LAM 9600 Metal Etcher[2]. Etching is used in microfabrication to chemically remove layers from the surface of a wafer during manufacturing[3]. The data was released in Matlab format, suitable for analyzing using the PLS Toolbox[9] developed by Eigenvector. There are three .mat files; MACHINE_Data.mat, OES_DATA.mat, and RFM_DATA.mat. A detailed note on the data and attributes available in reference [1] and [2]. Since the data is in Matlab format, we created a python based parser to convert the data in pandas DataFrames. This parser should enable Data Miners and Data Scientists to play around with data using Open Source tools like Python or R. \n",
    "\n",
    "### The Eigenvector Etch Data Parser\n",
    "\n",
    "The Eigenvector Etch Data Parser is developed to read Matlab data files published by Eigenvector[2]. The data is from a LAM 9600 Metal Etching Machine and was collected in 1995's. The parser reads each file and converts the calibration data (sensor data) and test data (sensor data) into a single DataFrame. The parser introduced an additional field in the data 'fault_name', which helps the user identify the normal/calibration wafers and test wafers(with defects). We tested the parser in Python3 environments only; if you are looking for Python2 compatibility, please test and create a bug/pull request as applicable. The source code is released under Apache 2.0 license and is available at https://github.com/jaganadhg/egvsemicon.\n",
    "\n",
    "### Etch Data\n",
    "\n",
    "The Eigenvector Etch Data is provided in three Matlab vector files MACHINE_Data.mat, OES_DATA.mat, and RFM_DATA.mat. The MACHINE_Data.mat file consists of the engineering variables, time, and the etch recipe steps. The variable/feature categories are pressure, gas flow rate, and power (If you wonder why gas flow is here, it is part of the semiconductor chemistry process and a more significant topic beyond this note!). The OES_DATA.mat file consists of the optical emission spectroscopy (OES) of the plasma. OES description is available in Hitachi's reference pge[4]. The file RFM_DATA.mat contains radio-frequency monitoring (RFM) system to monitor the power and phase relationships of the plasma generator. \n",
    "\n",
    "The OES data do not come with sensor names, unlike the Machine Data/Engineering Variable and RFM Data. The data is a field wave_axis which represents wavelengths in nm of peaks. A special note from the data \"Note that this data consists of integrated peak areas at for peaks at 43 wavelengths but looking across the plasma in 3 different locations perpendicular to the overall gas flow in the system.\"[2]\n",
    "\n",
    "The RFM Data has associated information related to the unit of variables (units: [ 71x6  char]  Units of the variables). This project provides the variable and its unit mapping as a CSV file (https://github.com/jaganadhg/egvsemicon/blob/main/rfm_variable_unit_map.csv). There are 35 variables measured with VRMS [5], 20 variables with the degree, 15 variables with IRMS [5], and one variable with the second. The pandas DataFrame from the parser will not be providing this information. The sensor names are masked to protect the process and machine details in this data. It is difficult to determine the sensor name without explicit Etch machine knowledge.However, a smart and experienced Process Engineer would infer some from the units. \n",
    "\n",
    "### Data Exploration\n",
    "Let's look at the data."
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "source": [
    "%matplotlib inline\n",
    "import warnings\n",
    "warnings.filterwarnings('ignore')\n",
    "\n",
    "import pandas as pd\n",
    "import numpy as np \n",
    "import scipy as sp \n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "from egvparser import egienvec_parser"
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "#### Engineering Variables Data"
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "source": [
    "ev_data_path = \"/home/jaganadhg/AI_RND/Semiconductor/eigenvector/MACHINE_Data.mat\"\n",
    "ev_key = \"LAMDATA\"\n",
    "\n",
    "ev_data = egienvec_parser(ev_data_path,\n",
    "                        dkey=ev_key)"
   ],
   "outputs": [
    {
     "output_type": "stream",
     "name": "stderr",
     "text": [
      "2022-01-29 20:58:51,477 :: INFO :: The sensor names are ['Time          ', 'Step Number   ', 'BCl3 Flow     ', 'Cl2 Flow      ', 'RF Btm Pwr    ', 'RF Btm Rfl Pwr', 'Endpt A       ', 'He Press      ', 'Pressure      ', 'RF Tuner      ', 'RF Load       ', 'RF Phase Err  ', 'RF Pwr        ', 'RF Impedance  ', 'TCP Tuner     ', 'TCP Phase Err ', 'TCP Impedance ', 'TCP Top Pwr   ', 'TCP Rfl Pwr   ', 'TCP Load      ', 'Vat Valve     ']\n",
      "2022-01-29 20:58:51,477 :: INFO :: Processing calibration data for LAMDATA\n",
      "2022-01-29 20:58:51,598 :: INFO :: Processed calibration data for LAMDATA\n",
      "2022-01-29 20:58:51,599 :: INFO :: Processing test data for LAMDATA\n",
      "2022-01-29 20:58:51,616 :: INFO :: Processed test data for LAMDATA\n",
      "2022-01-29 20:58:51,619 :: INFO :: Total sensor values in the data LAMDATA is 12829\n"
     ]
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "source": [
    "ev_data.head()"
   ],
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Time</th>\n",
       "      <th>Step Number</th>\n",
       "      <th>BCl3 Flow</th>\n",
       "      <th>Cl2 Flow</th>\n",
       "      <th>RF Btm Pwr</th>\n",
       "      <th>RF Btm Rfl Pwr</th>\n",
       "      <th>Endpt A</th>\n",
       "      <th>He Press</th>\n",
       "      <th>Pressure</th>\n",
       "      <th>RF Tuner</th>\n",
       "      <th>...</th>\n",
       "      <th>RF Impedance</th>\n",
       "      <th>TCP Tuner</th>\n",
       "      <th>TCP Phase Err</th>\n",
       "      <th>TCP Impedance</th>\n",
       "      <th>TCP Top Pwr</th>\n",
       "      <th>TCP Rfl Pwr</th>\n",
       "      <th>TCP Load</th>\n",
       "      <th>Vat Valve</th>\n",
       "      <th>wafer_names</th>\n",
       "      <th>fault_name</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>11.9460</td>\n",
       "      <td>4.0</td>\n",
       "      <td>751.0</td>\n",
       "      <td>753.0</td>\n",
       "      <td>132.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>626.0</td>\n",
       "      <td>100.0</td>\n",
       "      <td>1227.0</td>\n",
       "      <td>9408.0</td>\n",
       "      <td>...</td>\n",
       "      <td>16599.0</td>\n",
       "      <td>20028.0</td>\n",
       "      <td>-296.0</td>\n",
       "      <td>16848.0</td>\n",
       "      <td>360.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>27594.0</td>\n",
       "      <td>49.0</td>\n",
       "      <td>l2901.txm</td>\n",
       "      <td>calibration</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>13.0280</td>\n",
       "      <td>4.0</td>\n",
       "      <td>751.0</td>\n",
       "      <td>753.0</td>\n",
       "      <td>134.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>620.0</td>\n",
       "      <td>99.0</td>\n",
       "      <td>1229.0</td>\n",
       "      <td>9431.0</td>\n",
       "      <td>...</td>\n",
       "      <td>16568.0</td>\n",
       "      <td>20042.0</td>\n",
       "      <td>-676.0</td>\n",
       "      <td>16796.0</td>\n",
       "      <td>350.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>27440.0</td>\n",
       "      <td>49.0</td>\n",
       "      <td>l2901.txm</td>\n",
       "      <td>calibration</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>14.0490</td>\n",
       "      <td>4.0</td>\n",
       "      <td>751.0</td>\n",
       "      <td>755.0</td>\n",
       "      <td>134.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>599.0</td>\n",
       "      <td>102.0</td>\n",
       "      <td>1221.0</td>\n",
       "      <td>9389.0</td>\n",
       "      <td>...</td>\n",
       "      <td>16442.0</td>\n",
       "      <td>20146.0</td>\n",
       "      <td>-291.0</td>\n",
       "      <td>16512.0</td>\n",
       "      <td>344.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>27276.0</td>\n",
       "      <td>49.0</td>\n",
       "      <td>l2901.txm</td>\n",
       "      <td>calibration</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>15.1329</td>\n",
       "      <td>4.0</td>\n",
       "      <td>751.0</td>\n",
       "      <td>753.0</td>\n",
       "      <td>133.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>586.0</td>\n",
       "      <td>100.0</td>\n",
       "      <td>1201.0</td>\n",
       "      <td>9445.0</td>\n",
       "      <td>...</td>\n",
       "      <td>16960.0</td>\n",
       "      <td>20148.0</td>\n",
       "      <td>-262.0</td>\n",
       "      <td>17020.0</td>\n",
       "      <td>352.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>27330.0</td>\n",
       "      <td>50.0</td>\n",
       "      <td>l2901.txm</td>\n",
       "      <td>calibration</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>16.1390</td>\n",
       "      <td>4.0</td>\n",
       "      <td>751.0</td>\n",
       "      <td>754.0</td>\n",
       "      <td>132.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>587.0</td>\n",
       "      <td>102.0</td>\n",
       "      <td>1182.0</td>\n",
       "      <td>9456.0</td>\n",
       "      <td>...</td>\n",
       "      <td>16564.0</td>\n",
       "      <td>20226.0</td>\n",
       "      <td>-547.0</td>\n",
       "      <td>16440.0</td>\n",
       "      <td>346.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>27262.0</td>\n",
       "      <td>50.0</td>\n",
       "      <td>l2901.txm</td>\n",
       "      <td>calibration</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 23 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "      Time  Step Number  BCl3 Flow  Cl2 Flow  RF Btm Pwr  RF Btm Rfl Pwr  \\\n",
       "0  11.9460          4.0      751.0     753.0       132.0             0.0   \n",
       "1  13.0280          4.0      751.0     753.0       134.0             0.0   \n",
       "2  14.0490          4.0      751.0     755.0       134.0             0.0   \n",
       "3  15.1329          4.0      751.0     753.0       133.0             0.0   \n",
       "4  16.1390          4.0      751.0     754.0       132.0             0.0   \n",
       "\n",
       "   Endpt A  He Press  Pressure  RF Tuner  ...  RF Impedance  TCP Tuner  \\\n",
       "0    626.0     100.0    1227.0    9408.0  ...       16599.0    20028.0   \n",
       "1    620.0      99.0    1229.0    9431.0  ...       16568.0    20042.0   \n",
       "2    599.0     102.0    1221.0    9389.0  ...       16442.0    20146.0   \n",
       "3    586.0     100.0    1201.0    9445.0  ...       16960.0    20148.0   \n",
       "4    587.0     102.0    1182.0    9456.0  ...       16564.0    20226.0   \n",
       "\n",
       "   TCP Phase Err  TCP Impedance  TCP Top Pwr  TCP Rfl Pwr  TCP Load  \\\n",
       "0         -296.0        16848.0        360.0          0.0   27594.0   \n",
       "1         -676.0        16796.0        350.0          0.0   27440.0   \n",
       "2         -291.0        16512.0        344.0          0.0   27276.0   \n",
       "3         -262.0        17020.0        352.0          0.0   27330.0   \n",
       "4         -547.0        16440.0        346.0          0.0   27262.0   \n",
       "\n",
       "   Vat Valve  wafer_names   fault_name  \n",
       "0       49.0    l2901.txm  calibration  \n",
       "1       49.0    l2901.txm  calibration  \n",
       "2       49.0    l2901.txm  calibration  \n",
       "3       50.0    l2901.txm  calibration  \n",
       "4       50.0    l2901.txm  calibration  \n",
       "\n",
       "[5 rows x 23 columns]"
      ]
     },
     "metadata": {},
     "execution_count": 3
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "source": [
    "ev_data.columns"
   ],
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": [
       "Index(['Time', 'Step Number', 'BCl3 Flow', 'Cl2 Flow', 'RF Btm Pwr',\n",
       "       'RF Btm Rfl Pwr', 'Endpt A', 'He Press', 'Pressure', 'RF Tuner',\n",
       "       'RF Load', 'RF Phase Err', 'RF Pwr', 'RF Impedance', 'TCP Tuner',\n",
       "       'TCP Phase Err', 'TCP Impedance', 'TCP Top Pwr', 'TCP Rfl Pwr',\n",
       "       'TCP Load', 'Vat Valve', 'wafer_names', 'fault_name'],\n",
       "      dtype='object')"
      ]
     },
     "metadata": {},
     "execution_count": 4
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "The first two variables represent Time (in seconds) and Step Number. The step number should be correlated to the machine's process recipe, which is absent in the data (due to IP reasons, I believe). As mentioned earlier, the rest of the variables represent gas, pressure, and power. The last two variables, wafer_names, refer to a unique wafer name, and  fault_name indicates whether a wafer is a calibration/normal or test wafer/defect (induced defect as per the original paper [1]). If you are interested in further exploration, wafer_names will help you identify individual wafers. More than one record is there for each wafer because the process takes a defined time range to complete a recipe. The data consists of only steps 4 and 5 from the process recipe.  \n",
    "\n",
    "12829 records span across 126 normal/calibration wafers and 20 test (defect induced) wafers. \n",
    "\n",
    "\n",
    "#### OES Data"
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "source": [
    "oes_data_path = \"/home/jaganadhg/AI_RND/Semiconductor/eigenvector/OES_DATA.mat\"\n",
    "oes_key = \"OESDATA\"\n",
    "\n",
    "oes_data = egienvec_parser(oes_data_path,\n",
    "                        dkey=oes_key)"
   ],
   "outputs": [
    {
     "output_type": "stream",
     "name": "stderr",
     "text": [
      "2022-01-29 20:58:51,950 :: INFO :: The sensor names are [array([250.  , 261.8 , 266.6 , 272.2 , 278.3 , 284.6 , 288.25, 308.25,\n",
      "       309.25, 324.8 , 327.5 , 336.98, 364.33, 388.  , 394.4 , 395.8 ,\n",
      "       415.  , 532.6 , 544.2 , 556.7 , 580.  , 611.5 , 613.9 , 616.3 ,\n",
      "       618.5 , 639.7 , 643.2 , 644.9 , 652.8 , 660.  , 669.5 , 670.6 ,\n",
      "       674.  , 725.  , 740.8 , 748.5 , 753.7 , 770.6 , 773.2 , 781.  ,\n",
      "       783.5 , 787.5 , 791.5 , 250.  , 261.8 , 266.6 , 272.2 , 278.3 ,\n",
      "       284.6 , 288.25, 308.25, 309.25, 324.8 , 327.5 , 336.98, 364.33,\n",
      "       388.  , 394.4 , 395.8 , 415.  , 532.6 , 544.2 , 556.7 , 580.  ,\n",
      "       611.5 , 613.9 , 616.3 , 618.5 , 639.7 , 643.2 , 644.9 , 652.8 ,\n",
      "       660.  , 669.5 , 670.6 , 674.  , 725.  , 740.8 , 748.5 , 753.7 ,\n",
      "       770.6 , 773.2 , 781.  , 783.5 , 787.5 , 791.5 , 250.  , 261.8 ,\n",
      "       266.6 , 272.2 , 278.3 , 284.6 , 288.25, 308.25, 309.25, 324.8 ,\n",
      "       327.5 , 336.98, 364.33, 388.  , 394.4 , 395.8 , 415.  , 532.6 ,\n",
      "       544.2 , 556.7 , 580.  , 611.5 , 613.9 , 616.3 , 618.5 , 639.7 ,\n",
      "       643.2 , 644.9 , 652.8 , 660.  , 669.5 , 670.6 , 674.  , 725.  ,\n",
      "       740.8 , 748.5 , 753.7 , 770.6 , 773.2 , 781.  , 783.5 , 787.5 ,\n",
      "       791.5 ])]\n",
      "2022-01-29 20:58:51,951 :: INFO :: Processing calibration data for OESDATA\n",
      "2022-01-29 20:58:52,064 :: INFO :: Processed calibration data for OESDATA\n",
      "2022-01-29 20:58:52,064 :: INFO :: Processing test data for OESDATA\n",
      "2022-01-29 20:58:52,088 :: INFO :: Processed test data for OESDATA\n",
      "2022-01-29 20:58:52,092 :: INFO :: Total sensor values in the data OESDATA is 4786\n"
     ]
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "source": [
    "oes_data.head()"
   ],
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>250.0</th>\n",
       "      <th>261.8</th>\n",
       "      <th>266.6</th>\n",
       "      <th>272.2</th>\n",
       "      <th>278.3</th>\n",
       "      <th>284.6</th>\n",
       "      <th>288.25</th>\n",
       "      <th>308.25</th>\n",
       "      <th>309.25</th>\n",
       "      <th>324.8</th>\n",
       "      <th>...</th>\n",
       "      <th>748.5</th>\n",
       "      <th>753.7</th>\n",
       "      <th>770.6</th>\n",
       "      <th>773.2</th>\n",
       "      <th>781.0</th>\n",
       "      <th>783.5</th>\n",
       "      <th>787.5</th>\n",
       "      <th>791.5</th>\n",
       "      <th>wafer_names</th>\n",
       "      <th>fault_name</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>715.8</td>\n",
       "      <td>11926.3</td>\n",
       "      <td>1242.3</td>\n",
       "      <td>37720.0</td>\n",
       "      <td>3851.3</td>\n",
       "      <td>876.1</td>\n",
       "      <td>107.8</td>\n",
       "      <td>2732.2</td>\n",
       "      <td>2732.2</td>\n",
       "      <td>157.5</td>\n",
       "      <td>...</td>\n",
       "      <td>1791.8</td>\n",
       "      <td>8848.0</td>\n",
       "      <td>6260.1</td>\n",
       "      <td>6417.7</td>\n",
       "      <td>5981.9</td>\n",
       "      <td>5981.9</td>\n",
       "      <td>4692.4</td>\n",
       "      <td>8513.2</td>\n",
       "      <td>s2901.int</td>\n",
       "      <td>calibration</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>756.1</td>\n",
       "      <td>13309.8</td>\n",
       "      <td>1105.2</td>\n",
       "      <td>38144.7</td>\n",
       "      <td>4100.9</td>\n",
       "      <td>996.7</td>\n",
       "      <td>179.5</td>\n",
       "      <td>2532.3</td>\n",
       "      <td>2532.3</td>\n",
       "      <td>186.1</td>\n",
       "      <td>...</td>\n",
       "      <td>1784.5</td>\n",
       "      <td>9094.1</td>\n",
       "      <td>6273.9</td>\n",
       "      <td>4914.2</td>\n",
       "      <td>6054.9</td>\n",
       "      <td>6054.9</td>\n",
       "      <td>4947.9</td>\n",
       "      <td>8452.9</td>\n",
       "      <td>s2901.int</td>\n",
       "      <td>calibration</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>754.5</td>\n",
       "      <td>14061.6</td>\n",
       "      <td>1199.7</td>\n",
       "      <td>38646.1</td>\n",
       "      <td>4163.6</td>\n",
       "      <td>945.2</td>\n",
       "      <td>145.6</td>\n",
       "      <td>2923.5</td>\n",
       "      <td>2923.5</td>\n",
       "      <td>239.7</td>\n",
       "      <td>...</td>\n",
       "      <td>1686.7</td>\n",
       "      <td>9024.4</td>\n",
       "      <td>6409.7</td>\n",
       "      <td>6226.2</td>\n",
       "      <td>5709.4</td>\n",
       "      <td>5709.4</td>\n",
       "      <td>4783.3</td>\n",
       "      <td>8310.1</td>\n",
       "      <td>s2901.int</td>\n",
       "      <td>calibration</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>804.4</td>\n",
       "      <td>13489.3</td>\n",
       "      <td>1312.3</td>\n",
       "      <td>40037.2</td>\n",
       "      <td>4175.4</td>\n",
       "      <td>898.8</td>\n",
       "      <td>83.1</td>\n",
       "      <td>2731.0</td>\n",
       "      <td>2731.0</td>\n",
       "      <td>220.9</td>\n",
       "      <td>...</td>\n",
       "      <td>1539.5</td>\n",
       "      <td>9338.8</td>\n",
       "      <td>6330.3</td>\n",
       "      <td>4896.1</td>\n",
       "      <td>5925.9</td>\n",
       "      <td>5931.6</td>\n",
       "      <td>4770.0</td>\n",
       "      <td>8621.3</td>\n",
       "      <td>s2901.int</td>\n",
       "      <td>calibration</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>800.7</td>\n",
       "      <td>14719.5</td>\n",
       "      <td>1262.9</td>\n",
       "      <td>40158.1</td>\n",
       "      <td>4318.2</td>\n",
       "      <td>962.2</td>\n",
       "      <td>170.8</td>\n",
       "      <td>3247.8</td>\n",
       "      <td>3247.8</td>\n",
       "      <td>235.9</td>\n",
       "      <td>...</td>\n",
       "      <td>1857.0</td>\n",
       "      <td>9633.7</td>\n",
       "      <td>6506.2</td>\n",
       "      <td>4963.8</td>\n",
       "      <td>6205.1</td>\n",
       "      <td>6205.1</td>\n",
       "      <td>5009.1</td>\n",
       "      <td>8941.3</td>\n",
       "      <td>s2901.int</td>\n",
       "      <td>calibration</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 131 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   250.0    261.8   266.6    272.2   278.3  284.6  288.25  308.25  309.25  \\\n",
       "0  715.8  11926.3  1242.3  37720.0  3851.3  876.1   107.8  2732.2  2732.2   \n",
       "1  756.1  13309.8  1105.2  38144.7  4100.9  996.7   179.5  2532.3  2532.3   \n",
       "2  754.5  14061.6  1199.7  38646.1  4163.6  945.2   145.6  2923.5  2923.5   \n",
       "3  804.4  13489.3  1312.3  40037.2  4175.4  898.8    83.1  2731.0  2731.0   \n",
       "4  800.7  14719.5  1262.9  40158.1  4318.2  962.2   170.8  3247.8  3247.8   \n",
       "\n",
       "   324.8  ...   748.5   753.7   770.6   773.2   781.0   783.5   787.5   791.5  \\\n",
       "0  157.5  ...  1791.8  8848.0  6260.1  6417.7  5981.9  5981.9  4692.4  8513.2   \n",
       "1  186.1  ...  1784.5  9094.1  6273.9  4914.2  6054.9  6054.9  4947.9  8452.9   \n",
       "2  239.7  ...  1686.7  9024.4  6409.7  6226.2  5709.4  5709.4  4783.3  8310.1   \n",
       "3  220.9  ...  1539.5  9338.8  6330.3  4896.1  5925.9  5931.6  4770.0  8621.3   \n",
       "4  235.9  ...  1857.0  9633.7  6506.2  4963.8  6205.1  6205.1  5009.1  8941.3   \n",
       "\n",
       "   wafer_names   fault_name  \n",
       "0    s2901.int  calibration  \n",
       "1    s2901.int  calibration  \n",
       "2    s2901.int  calibration  \n",
       "3    s2901.int  calibration  \n",
       "4    s2901.int  calibration  \n",
       "\n",
       "[5 rows x 131 columns]"
      ]
     },
     "metadata": {},
     "execution_count": 6
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "The first 129 variables in this data represent wavelength measured from three different locations. Practically every 43 variables represent measurement from one location only. The last two variables are wafer_names, fault_name represents wafer, and calibration or test wafer indication. \n",
    "\n",
    "#### Radio Frequency Monitoring (RFM) Data"
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "source": [
    "rfm_data_path = \"/home/jaganadhg/AI_RND/Semiconductor/eigenvector/RFM_DATA.mat\"\n",
    "rfm_key = \"RFMDATA\"\n",
    "\n",
    "rfm_data = egienvec_parser(rfm_data_path,\n",
    "                        dkey=rfm_key)"
   ],
   "outputs": [
    {
     "output_type": "stream",
     "name": "stderr",
     "text": [
      "2022-01-29 20:58:55,625 :: INFO :: The sensor names are ['TIME  ', 'S1V1  ', 'S1V2  ', 'S1V3  ', 'S1V4  ', 'S1V5  ', 'S1I1  ', 'S1I2  ', 'S1I3  ', 'S1I4  ', 'S1I5  ', 'S1P1  ', 'S1P2  ', 'S1P3  ', 'S1P4  ', 'S1P5  ', 'S2V1  ', 'S2V2  ', 'S2V3  ', 'S2V4  ', 'S2V5  ', 'S2I1  ', 'S2I2  ', 'S2I3  ', 'S2I4  ', 'S2I5  ', 'S2P1  ', 'S2P2  ', 'S2P3  ', 'S2P4  ', 'S2P5  ', 'S3V1  ', 'S3V2  ', 'S3V3  ', 'S3V4  ', 'S3V5  ', 'S4V1  ', 'S4V2  ', 'S4V3  ', 'S4V4  ', 'S4V5  ', 'S34PV1', 'S34PV2', 'S34PV3', 'S34PV4', 'S34PV5', 'S3I1  ', 'S3I2  ', 'S3I3  ', 'S3I4  ', 'S3I5  ', 'S4I1  ', 'S4I2  ', 'S4I3  ', 'S4I4  ', 'S4I5  ', 'S34PI1', 'S34PI2', 'S34PI3', 'S34PI4', 'S34PI5', 'S34V1 ', 'S34V2 ', 'S34V3 ', 'S34V4 ', 'S34V5 ', 'S34I1 ', 'S34I2 ', 'S34I3 ', 'S34I4 ', 'S34I5 ']\n",
      "2022-01-29 20:58:55,627 :: INFO :: Processing calibration data for RFMDATA\n",
      "2022-01-29 20:58:55,750 :: INFO :: Processed calibration data for RFMDATA\n",
      "2022-01-29 20:58:55,750 :: INFO :: Processing test data for RFMDATA\n",
      "2022-01-29 20:58:55,769 :: INFO :: Processed test data for RFMDATA\n",
      "2022-01-29 20:58:55,771 :: INFO :: Total sensor values in the data RFMDATA is 3519\n"
     ]
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "source": [
    "rfm_data.head()"
   ],
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>TIME</th>\n",
       "      <th>S1V1</th>\n",
       "      <th>S1V2</th>\n",
       "      <th>S1V3</th>\n",
       "      <th>S1V4</th>\n",
       "      <th>S1V5</th>\n",
       "      <th>S1I1</th>\n",
       "      <th>S1I2</th>\n",
       "      <th>S1I3</th>\n",
       "      <th>S1I4</th>\n",
       "      <th>...</th>\n",
       "      <th>S34V3</th>\n",
       "      <th>S34V4</th>\n",
       "      <th>S34V5</th>\n",
       "      <th>S34I1</th>\n",
       "      <th>S34I2</th>\n",
       "      <th>S34I3</th>\n",
       "      <th>S34I4</th>\n",
       "      <th>S34I5</th>\n",
       "      <th>wafer_names</th>\n",
       "      <th>fault_name</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>15.32813</td>\n",
       "      <td>146.2126</td>\n",
       "      <td>7.474548</td>\n",
       "      <td>5.444838</td>\n",
       "      <td>2.468795</td>\n",
       "      <td>1.458764</td>\n",
       "      <td>2.651642</td>\n",
       "      <td>0.003891</td>\n",
       "      <td>0.002745</td>\n",
       "      <td>0.000811</td>\n",
       "      <td>...</td>\n",
       "      <td>5.041583</td>\n",
       "      <td>2.695145</td>\n",
       "      <td>172.2877</td>\n",
       "      <td>14.99166</td>\n",
       "      <td>0.064546</td>\n",
       "      <td>0.018406</td>\n",
       "      <td>0.005021</td>\n",
       "      <td>0.004922</td>\n",
       "      <td>r2901.txt</td>\n",
       "      <td>calibration</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>18.45703</td>\n",
       "      <td>145.8764</td>\n",
       "      <td>7.500408</td>\n",
       "      <td>5.459276</td>\n",
       "      <td>2.465954</td>\n",
       "      <td>1.457085</td>\n",
       "      <td>2.657755</td>\n",
       "      <td>0.003940</td>\n",
       "      <td>0.002761</td>\n",
       "      <td>0.000817</td>\n",
       "      <td>...</td>\n",
       "      <td>5.020921</td>\n",
       "      <td>2.687807</td>\n",
       "      <td>172.2852</td>\n",
       "      <td>14.92634</td>\n",
       "      <td>0.064253</td>\n",
       "      <td>0.017902</td>\n",
       "      <td>0.004927</td>\n",
       "      <td>0.004872</td>\n",
       "      <td>r2901.txt</td>\n",
       "      <td>calibration</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>21.58984</td>\n",
       "      <td>146.8875</td>\n",
       "      <td>7.413699</td>\n",
       "      <td>5.437320</td>\n",
       "      <td>2.437727</td>\n",
       "      <td>1.447055</td>\n",
       "      <td>2.633389</td>\n",
       "      <td>0.004037</td>\n",
       "      <td>0.002748</td>\n",
       "      <td>0.000815</td>\n",
       "      <td>...</td>\n",
       "      <td>5.047616</td>\n",
       "      <td>2.691589</td>\n",
       "      <td>172.2836</td>\n",
       "      <td>14.92634</td>\n",
       "      <td>0.063929</td>\n",
       "      <td>0.017908</td>\n",
       "      <td>0.004853</td>\n",
       "      <td>0.004855</td>\n",
       "      <td>r2901.txt</td>\n",
       "      <td>calibration</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>24.71875</td>\n",
       "      <td>146.8875</td>\n",
       "      <td>7.372842</td>\n",
       "      <td>5.386851</td>\n",
       "      <td>2.432120</td>\n",
       "      <td>1.443727</td>\n",
       "      <td>2.636422</td>\n",
       "      <td>0.003990</td>\n",
       "      <td>0.002748</td>\n",
       "      <td>0.000817</td>\n",
       "      <td>...</td>\n",
       "      <td>5.037209</td>\n",
       "      <td>2.695823</td>\n",
       "      <td>172.2836</td>\n",
       "      <td>14.91610</td>\n",
       "      <td>0.064036</td>\n",
       "      <td>0.017935</td>\n",
       "      <td>0.004829</td>\n",
       "      <td>0.004861</td>\n",
       "      <td>r2901.txt</td>\n",
       "      <td>calibration</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>27.84766</td>\n",
       "      <td>146.5498</td>\n",
       "      <td>7.357578</td>\n",
       "      <td>5.393677</td>\n",
       "      <td>2.423734</td>\n",
       "      <td>1.443727</td>\n",
       "      <td>2.636422</td>\n",
       "      <td>0.003954</td>\n",
       "      <td>0.002739</td>\n",
       "      <td>0.000815</td>\n",
       "      <td>...</td>\n",
       "      <td>5.049484</td>\n",
       "      <td>2.697655</td>\n",
       "      <td>172.1836</td>\n",
       "      <td>14.92634</td>\n",
       "      <td>0.063893</td>\n",
       "      <td>0.017631</td>\n",
       "      <td>0.004860</td>\n",
       "      <td>0.004860</td>\n",
       "      <td>r2901.txt</td>\n",
       "      <td>calibration</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 73 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "       TIME      S1V1      S1V2      S1V3      S1V4      S1V5      S1I1  \\\n",
       "0  15.32813  146.2126  7.474548  5.444838  2.468795  1.458764  2.651642   \n",
       "1  18.45703  145.8764  7.500408  5.459276  2.465954  1.457085  2.657755   \n",
       "2  21.58984  146.8875  7.413699  5.437320  2.437727  1.447055  2.633389   \n",
       "3  24.71875  146.8875  7.372842  5.386851  2.432120  1.443727  2.636422   \n",
       "4  27.84766  146.5498  7.357578  5.393677  2.423734  1.443727  2.636422   \n",
       "\n",
       "       S1I2      S1I3      S1I4  ...     S34V3     S34V4     S34V5     S34I1  \\\n",
       "0  0.003891  0.002745  0.000811  ...  5.041583  2.695145  172.2877  14.99166   \n",
       "1  0.003940  0.002761  0.000817  ...  5.020921  2.687807  172.2852  14.92634   \n",
       "2  0.004037  0.002748  0.000815  ...  5.047616  2.691589  172.2836  14.92634   \n",
       "3  0.003990  0.002748  0.000817  ...  5.037209  2.695823  172.2836  14.91610   \n",
       "4  0.003954  0.002739  0.000815  ...  5.049484  2.697655  172.1836  14.92634   \n",
       "\n",
       "      S34I2     S34I3     S34I4     S34I5  wafer_names   fault_name  \n",
       "0  0.064546  0.018406  0.005021  0.004922    r2901.txt  calibration  \n",
       "1  0.064253  0.017902  0.004927  0.004872    r2901.txt  calibration  \n",
       "2  0.063929  0.017908  0.004853  0.004855    r2901.txt  calibration  \n",
       "3  0.064036  0.017935  0.004829  0.004861    r2901.txt  calibration  \n",
       "4  0.063893  0.017631  0.004860  0.004860    r2901.txt  calibration  \n",
       "\n",
       "[5 rows x 73 columns]"
      ]
     },
     "metadata": {},
     "execution_count": 8
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "If you are interested in reading about RFM, an interesting resource is \"RF Technology in Semiconductor Wafer Processing\" [6]. \n",
    "\n",
    "We have provided the sensor and unit mapping separately [7]. The data sets come with reference to the unit of each sensor value to understand the data better. The actual sensor names are masked, and the last two variables represent the wafer name and indicate test or calibration wafers. \n",
    "\n",
    "#### Combining Data Set\n",
    "\n",
    "The three DataFrames generated have various records and a caveat in identity column wafer_names. The RFM has 3519, OES has 4786, and Engineering Variables has 12829 records. Value in the identity column wafer_names; values are prefixed by l,s, and r for Engineering Variables, OES, and RFM data, respectively. A derived variable can be generated for identity by replacing the leading alphabet in the wafer_names.  \n",
    "\n",
    "**Note : The idea of representing OES data as DataFrame may not be the best. We are working towards a better representation.**\n",
    "\n",
    "\n",
    "\n",
    "#### Data Mining/Data Science and Next Steps\n",
    "\n",
    "We are not venturing into any detailed analytics solution in the scope of current notes—the industry practices simple techniques from univariate analysis to employing Deep Learning to solve the problems. From the data description, one can infer the nature of data preprocessing and feature engineering techniques. In the same domain, understanding or active guidance from field processing engineers may benefit you in starting an exciting project. A good starting point will be the original paper [1]. \n",
    "\n",
    "#### Competing Interests\n",
    "\n",
    "This notebook is intended to introduce the Egionvector Metal Etch Data Parser[8] and the data [2]. The authors declare that no proprietary information related to the authors, affiliated company, or its approach, methodologies, and IPR is discussed in these notes. The authors declare that they have no competing interests."
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "### Reference\n",
    "\n",
    "[1] B.M. Wise, N.B. Gallagher, S.W. Butler, D.D. White, Jr. and G.G. Barna, “A Comparison of Principal Components Analysis, Multi-way Principal Components Analysis, Tri-linear Decomposition and Parallel Factor Analysis for Fault Detection in a Semiconductor Etch Process”, J. Chemometrics, 13, 379­396 (1999)\n",
    "\n",
    "[2] https://eigenvector.com/resources/data-sets/\n",
    "\n",
    "[3] https://en.wikipedia.org/wiki/Etching_(microfabrication) \n",
    "\n",
    "[4] https://hha.hitachi-hightech.com/en/blogs-events/blogs/2017/10/25/optical-emission-spectroscopy-(oes)/\n",
    "\n",
    "[5] IRMS - The current in an alternating current circuit varies continuously in direction and magnitude. ... Diagrams denote this current as \"IRMS,\" with the \"RMS\" in subscript. A constant level of the root means square current dissipates the same amount of heat through a resistor as the alternating current does.\n",
    "VRMS - In electricity: Alternating-current circuits. The root-mean-square (RMS) voltage of a sinusoidal source of electromotive force (Vrms) is used to characterize the source. It is the square root of the time average of the voltage squared. - https://www.quora.com/What-are-VRMS-and-IRMS \n",
    "\n",
    "[6] https://www.microwavejournal.com/articles/21140-rf-technology-in-semiconductor-wafer-processing \n",
    "\n",
    "[7] https://github.com/jaganadhg/egvsemicon/blob/main/rfm_variable_unit_map.csv\n",
    "\n",
    "[8] Jaganadh Gopinadhan, “Eigenvector Metal Etch Data Parser - Python”. Zenodo, Jan. 29, 2022. doi: 10.5281/zenodo.5919197.\n",
    "\n",
    "[9] https://eigenvector.com/software/pls-toolbox/"
   ],
   "metadata": {}
  }
 ],
 "metadata": {
  "orig_nbformat": 4,
  "language_info": {
   "name": "python",
   "version": "3.7.8",
   "mimetype": "text/x-python",
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "pygments_lexer": "ipython3",
   "nbconvert_exporter": "python",
   "file_extension": ".py"
  },
  "kernelspec": {
   "name": "python3",
   "display_name": "Python 3.7.8 64-bit ('wafer': conda)"
  },
  "interpreter": {
   "hash": "d1208ee3017fa12782748204357f63627cc790dec7c166de019be4f55a51cd93"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}