class: bottom, left
background-image: url(assets/g.png)

<h2 class="title_headings_sml">COSC102 - Data Science Studio 1</h2>

<h1 class="title_headings_sml"> Topic 18 - Classification Performance </h1>

<h3 class="title_headings_sml"> Dr. Mitchell Welch </h3>

---

## Reading

* Chapter 3 from ***Hands-on Machine Learning with Scikit-Learn & TensorFlow***

---
## Summary

* Classifier Performance
* Confusion Matrix
* Precision/Recall
---

## Classifier Performance

* Carrying on from the previous topic, we have:
  * Trained a simple classifier.
  * Used the classifier to predict individual values
* Now we will complete a more comprehensive assessment of well the classifier performs. 
* To achieve this, we will be using the Scikit-learn built-in cross validation functions.

---

## Classifier Performance

* One of the easiest ways to assess the performance is with *K-Fold* cross-validation.
* The process involves splitting the dataset into *folds* (subsets)
* All but one fold is used to train the classifier, and the remaining fold is used to test the classifier.
* This process is repeated for all combinations of the folds and performance measures are averaged across tests on all combinations of folds.


---

## Classifier Performance

.center[![Matplotlib](assets/topic_18/cross_val.png)]

---

## Classifier Performance

* This is all built in to Scikit-learn:
* Our fist try:

```python
from sklearn.model_selection import cross_val_score
cross_val_score(sgd_clf, X_train, y_train_5, cv=3, scoring="accuracy")

```

---

## Classifier Performance

* At first glance this looks good, with 90+% accuracy. But this is hiding a very ineffective classifier.
* The problem is that the classifier has been trained and assessed on an unbalanced dataset.
* Only about 10% of the dataset contains '5's.
* It just classifiers everything as in the 'not-5' class to achieve this accuracy.
* We need a better way to assess the performance.

---

## Confusion Matrix

* We will look at using a *Confusion Matrix*
* The general idea is to count the number of times instances of class A are
classified as class B.
* A confusion matrix record:
  * True positives (TP) - correctly identified positive samples
  * True negatives (TN) - correctly identified negative samples
  * False positives (FP) - incorrectly identified positive samples
  * False negatives (FP) - incorrectly identified negative samples


---
## Confusion Matrix

.center[![Matplotlib](assets/topic_18/confusion_example.png)]

---

## Confusion Matrix

* To compute the confusion matrix, you first need to have a set of predictions so that
they can be compared to the actual targets.
* You could make predictions on the test set, but let’s keep it untouched for now (remember that you want to use the test set only at the very end of your project)
* We will use the `cross_val_predict( ... )` to conduct a k-fold cross validation and return the performance for each fold.
* Review the [demo notebook]().

---

## Precision/Recall

* Using the results from the confusion matrix, we can calculate some concise metrics.
* *Precision* - the accuracy of the positive predictions.

$$
precision = TP / (TP + FP)
$$

* *Recall* (Sometimes called *Sensitivity*) - The ratio of positive instances that are correctly detected by the classifier. 

$$
recall = TP /(TP + FN)
$$

---

## Precision/Recall

* Review the demo notebook to see this in action:

```python
from sklearn.metrics import precision_score, recall_score
print(precision_score(y_train_5, y_train_pred)) # == 4096 / (4096 + 1522)
print(recall_score(y_train_5, y_train_pred)) # == 4096 / (4096 + 1325)

```

---

## Precision/Recall

* Precision and recall can be summarized by calculating the *F1* score:

$$
F1 = TP / (TP + ((FN+FP)/2))
$$

* This produces a harmonic mean that is only high if *both* the recall and precision are high. 

---
## Summary

* So far we have:
  * Trained a simple binary classifier (our '5' detector).
  * Tested it with simple predictions.
  * Assessed the performance using a confusion matrix.
  * Calculated the precision, recall and F1 scores.


---
## Summary

* Classifier Performance
* Confusion Matrix
* Precision/Recall
---