Skip to content

Commit c29d859

Browse files
danielrazavidanielrazavi
committed
2 parents 80c3801 + 4e9d303 commit c29d859

File tree

7 files changed

+1347
-0
lines changed

7 files changed

+1347
-0
lines changed

‎01_materials/slides/Classification_I.md

Lines changed: 238 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,238 @@
1+
---
2+
marp: true
3+
theme: uncover
4+
_class: invert
5+
paginate: true
6+
7+
style: |
8+
section{;
9+
}
10+
---
11+
<style>
12+
p {
13+
text-align: left;
14+
font-size: 30px
15+
}
16+
ul {
17+
margin: 0;
18+
font-size: 30px;
19+
}
20+
table {
21+
font-size: 30px;
22+
}
23+
ol {
24+
margin: 0;
25+
font-size: 30px;
26+
}
27+
blockquote {
28+
border-left: 10px solid #ccc;
29+
margin: 1.5em 10px;
30+
padding: 0.5em 30px;
31+
quotes: "\201C""\201D""\2018""\2019";
32+
}
33+
34+
blockquote:before {
35+
color: #ccc;
36+
content: none;
37+
font-size: 4em;
38+
line-height: 0.1em;
39+
margin-right: 0.25em;
40+
vertical-align: -0.4em;
41+
}
42+
43+
blockquote:after{
44+
content: none;
45+
font-size: 4em
46+
}
47+
48+
img {
49+
width: 50%;
50+
height: auto;
51+
}
52+
</style>
53+
54+
# **Classification I**
55+
```console
56+
Data Sciences Institute
57+
Applying Statistical Concepts
58+
```
59+
60+
---
61+
##### **Acknowledgements**
62+
- Slides created by Julia Gallucci under the supervision of Rohan Alexander.
63+
- Content adapted from: A First Introduction (Python Edition) Tiffany Timbers, Trevor Campbell, Melissa Lee, Joel Ostblom, Lindsey Heagy https://python.datasciencebook.ca/index.html
64+
---
65+
##### **Learning objectives**
66+
67+
- Identify when to use a classifier for predictions.
68+
- Use a training dataset and explore its role in classification.
69+
- Interpret classifier outputs.
70+
- Implement K-nearest neighbors classification in Python with scikit-learn.
71+
- Preprocess data using scikit-learn for centering, scaling, balancing, and imputing.
72+
---
73+
74+
75+
##### **The classification problem**
76+
77+
- Uses one or more variables to predict a categorical class (label) of interest.
78+
79+
- Binary classification involves two classes; multiclass classification involves more than two categories.
80+
81+
- *Examples:*
82+
- Diagnose a patient as diseased or healthy
83+
- Tag an email as spam or not spam
84+
- Predict if a credit card purchase is fraudulent
85+
86+
---
87+
88+
- A classifier assigns a class to new observations based on their similarity to known observations (training set).
89+
90+
- Training sets consist of observations with known classes used to train classifiers.
91+
- K-nearest neighbors is a widely used classification method.
92+
93+
Other methods include decision trees, SVMs, logistic regression, and neural networks.
94+
---
95+
##### **Example dataset**
96+
- ​​Breast Cancer Wisconsin Dataset is a popular dataset in machine learning and data science for classification tasks, specifically for predicting whether breast cancer tumors are malignant or benign.
97+
- *Key features:*
98+
- 569 samples of breast tumors (rows)
99+
- 30 features (eg., radius, texture, area, smoothness…)
100+
- 1 label of interest (Malignant; M or Benign; B)
101+
102+
---
103+
##### **K-nearest neighbors (KNN)**
104+
105+
- KNN finds the "nearest" or "most similar" observations in the training set to predict a new observation's label.
106+
- "Closest" can be measured by calculating the straight-line between points, known as distance.
107+
- K is a predefined number of neighbors considered for classification.
108+
![bg right:50% w:600][KNN_Nearest]
109+
---
110+
111+
To classify a new observation with K-nearest neighbors, follow these steps:
112+
1. Determine the nearest observation to the new observation from the training set.
113+
2. Identify the K observations that are closest.
114+
3. Classify the new observation based on the most common class among these neighbors.
115+
---
116+
##### **Distance metric**
117+
- We determine the "nearest" points to our new observation using the straight-line distance (often just called distance).
118+
- Suppose we have two observations, $A$ and $B$, each with two predictor variables, $x$ and $y$.
119+
![bg right:31% w:410][KNN_distance]
120+
121+
Observation | Variable x | Variable y |
122+
-----|------|:-----:|
123+
$A$ | $xA$ | $yA$ |
124+
$B$ | $xB$ | $yB$ |
125+
126+
The straight-line distance between observations A and B can be computed using the following formula:
127+
$$
128+
\text{Distance} = \sqrt{(x_B - x_A)^2 + (y_B - y_A)^2}
129+
$$
130+
131+
---
132+
##### **Example: K = 1**
133+
- New observation with an unknown diagnosis is a red diamond. The closest point to this new observation is malignant.
134+
- Points close together have similar perimeter and concavity values, suggesting they likely share the same diagnosis.
135+
![bg right:50% w:600][KNN_1]
136+
137+
---
138+
139+
- However, a nearest neighbor of 1 is **sensitive to noise.**
140+
The nearest neighbor to this new point is a benign observation.
141+
Does this seem like the right prediction? *Probably not*, considering the other nearby points.
142+
![bg right:50% w:600][KNN_1_WRONG]
143+
144+
---
145+
##### **Example: K = 3**
146+
- To improve the prediction, consider several neighboring points (e.g., 3) closest to the new observation.
147+
- Here, we are using the majority class among these 3 closest points to predict the new observation's diagnosis.
148+
- 2 of the 3 nearest neighbors to the new observation are malignant.
149+
![bg right:50% w:600][KNN_3]
150+
---
151+
##### **Impact of Variable Scaling on Prediction**
152+
- In K-nearest neighbors classification, the **scale** of each variable affects predictions.
153+
- Variables with larger scales (e.g., salary) have a bigger impact on distance calculations than variables with smaller scales (e.g., years of education).
154+
- This means that variables with large scales might **dominate the prediction process.**
155+
- *Example:*
156+
- a $1000 difference in salary affects distances more than a 10-year difference in education. However, in reality, 10 years of education might be more significant for predicting job type than a $1000 difference in salary.
157+
---
158+
##### **Variable standardization**
159+
- To ensure fair contribution from all variables, we often scale and center data before entering it into the model:
160+
1. Find the mean (μ) and standard deviation (σ) of each variable.
161+
2. Subtract the mean and divide by the standard deviation for each value ($x$).
162+
163+
$$
164+
(x- μ)/ σ
165+
$$
166+
167+
- This process, called *standardization*, adjusts the data so each variable has a mean of 0 and a standard deviation of 1, allowing the model to consider each variable equally based on its relationship to the outcome rather than its scale.
168+
---
169+
##### **Example: unstandardized vs standardized data**
170+
![w:900][KNN_standardization]
171+
- When unstandardized, 'area' has a much larger scale than 'smoothness', and could overpower the model, even if it's not the most significant factor. Standardizing data ensures that no single variable disproportionately influences the model due to its scale.
172+
173+
---
174+
##### **Class imbalance**
175+
- Occurs when one label is much more common than another in a dataset.
176+
- Classifiers like KNN use nearby points' labels to predict the new point's label. If one label is much more common, the classifier is more likely to predict that label, even if the pattern suggests otherwise.
177+
- *Example:*
178+
- Class imbalance is frequent in cases like rare disease diagnosis or malicious email detection, where the important class (e.g., presence of disease, malicious email) is much rarer than the unimportant class (e.g., no disease, normal email).
179+
180+
---
181+
##### **Example**
182+
- Here, K = 7
183+
- With only 3 observations of malignant tumors, the classifier will always predict benign (majority rules)!
184+
185+
![bg right:50% w:600][KNN_imbalance]
186+
187+
---
188+
##### **Missing data**
189+
- Handling missing data is challenging and generally relies on expert knowledge about the data, setting, and collection methods.
190+
- Missing entries can be informative: the fact that an entry is missing may relate to other variable values.
191+
- *Example*: Survey participants from marginalized groups may avoid certain questions due to fear of negative consequences, leading to biased conclusions if their missing data is discarded.
192+
- Ignoring missing data issues can result in misleading analyses and detrimental impacts.
193+
---
194+
##### **Options for missing data**
195+
- KNN requires access to the values of all variables for all observations in the training data.
196+
- Handling missing data in KNN classification:
197+
1. **Remove observations** with missing entries before building the classifier – this will work when there are a few missing entries, but if many rows have missing entries, removing them can result in losing too much data.
198+
2. **Impute missing entries/ fill in missing data with estimated values.** (eg., Mean imputation is a reasonable choice, where missing entries are filled in using the mean of the present entries in each variable)
199+
200+
---
201+
##### **What if we have more than 2 variables?**
202+
- The same K-nearest neighbors algorithm applies when you have a higher number of predictor variables.
203+
- Each predictor variable may give us new information to help create our classifier.
204+
- The only difference is the formula for the distance between points.
205+
---
206+
207+
- Example, 3 variables $x,y,z$
208+
209+
Observation | Variable x | Variable y | Variable z |
210+
-----|------|:-----:|:-----:|
211+
$A$ | $xA$ | $yA$ | $zA$
212+
$B$ | $xB$ | $yB$ | $zB$
213+
214+
$$
215+
\text{Distance} = \sqrt{(x_B - x_A)^2 + (y_B - y_A)^2 + (z_B - z_A)^2}
216+
$$
217+
- This formula still corresponds to a straight-line distance, just in a space with more dimensions!
218+
---
219+
##### **Example**
220+
- Here, K = 5 with 3 predictor variables
221+
222+
![bg right:50% w:1200][KNN_3features]
223+
224+
225+
---
226+
<!--_color: white -->
227+
<!--_backgroundColor: #f4a534 -->
228+
## `Putting it all together`
229+
### `KNN with scikit-learn`
230+
231+
[KNN_distance]: ../DSI%20Certificate%20Coordinator/KNN_distance.jpg
232+
[KNN_Nearest]: ../DSI%20Certificate%20Coordinator/KNN_Nearest.png
233+
[KNN_1]: ../DSI%20Certificate%20Coordinator/KNN_K1.png
234+
[KNN_1_WRONG]: ../DSI%20Certificate%20Coordinator/KNN_K1_WRONG.png
235+
[KNN_3]: ../DSI%20Certificate%20Coordinator/KNN_K3.png
236+
[KNN_standardization]: ../DSI%20Certificate%20Coordinator/KNN_standardization.png
237+
[KNN_imbalance]: ../DSI%20Certificate%20Coordinator/KNN_imbalance.png
238+
[KNN_3features]: ../DSI%20Certificate%20Coordinator/KNN_3features.jpg

0 commit comments

Comments
 (0)