class: bottom, left
background-image: url(assets/g.png)

<h2 class="title_headings_sml">COSC102 - Data Science Studio 1</h2>

<h1 class="title_headings_sml"> Topic 19 - The Receiver Operating Characteristic Curve </h1>

<h3 class="title_headings_sml"> Dr. Mitchell Welch </h3>

---

## Reading

* Chapter 3 from ***Hands-on Machine Learning with Scikit-Learn & TensorFlow***

---

## Summary

* Sensitivity-Precision Trade-off
* The ROC Curve

---

## Sensitivity-Precision Trade-off

* Using the results from the confusion matrix, we can calculate some concise metrics.
* *Precision* - the accuracy of the positive predictions.

$$
precision = TP / (TP + FP)
$$

* *Recall* (Sometimes called *Sensitivity*) - The ratio of positive instances that are correctly detected by the classifier. 

$$
recall = TP /(TP + FN)
$$


---
## Sensitivity-Precision Trade-off

* When building a classifier, there is always a level of trade-off between the *recall* and *precision*.
* Unfortunately, when working with classification problems, you can’t have it both ways: increasing precision reduces recall, and vice versa.

---

## Sensitivity-Precision Trade-off

* Our KNN example from topic 1: Remember the coloured regions represent the decision boundaries.
* We can move the decision boundaries to achieve higher/lower recall, but this is done at the cost of precision. 
.center[![Matplotlib](assets/topic_19/tradeoff.png)]


---

## Sensitivity-Precision Trade-off

* In a perfect world we are aiming to train a classifier that classifies the instances in a way such that this trade-off is minimized.
* In the previous example, this means learning decision boundaries that are detailed enough to achieve a high sensitivity, while not scooping up lots of false positives that drive precision down.
* Within the inner workings of most classifier algorithms, the decision to assign an instance to a class is made based on assigning a *score*.
* This score is compared to an arbitrary *threshold* value that is used to decide the classification.

---

## Sensitivity-Precision Trade-off

* This threshold effectively represents the location of the decision boundary. 
* So by adjusting the threshold, you can manipulate the tradeoff between recall and precision and assess the performance of the classifier.
* We can access the scores that the classifier assigns in the cross validation process and plot them understand the tradeoff.

```python
y_scores = cross_val_predict(sgd_clf, X_train, y_train_5, cv=3, method="decision_function")
from sklearn.metrics import precision_recall_curve
precisions, recalls, thresholds = precision_recall_curve(y_train_5, y_scores)
```
---

## The ROC Curve

* The receiver operating characteristic (ROC) curve is another common tool used with binary classifiers.
* The ROC curve plots the *true positive rate* (another name for recall) against the *false positive rate (FPR)* 
* Once again there is a trade-off: the higher the recall (TPR), the more false positive 
(FPR) the classifier produces.
* [Run the demo to produce the ROC curve for our data]().

---

## The ROC Curve

* Interpreting a ROC curve is fairly straight forward:
* The dotted line represents the ROC curve of a purely random classifier
* A good classifier stays as far away from that line as possible (toward the top-left corner).
* We assess this by calculating the *Area under the curve (AUC)*

```python
from sklearn.metrics import roc_auc_score
roc_auc_score(y_train_5, y_scores)
```

---

## The ROC Curve

* We have looked at two curves for assessing performance PR and ROC:
  * As a rule of thumb, you should prefer the PR curve whenever the positive class is rare or when you care more about the false positives than the false negatives.

* For example, looking at the previous ROC curve (and the ROC AUC score), you
may think that the classifier is really good.
  * But this is mostly because there are few positives (5s) compared to the negative (non-5s).

* The PR curve makes it clear that there is room for improvement.
---

## The ROC Curve

* Take away points:
  * Don't rely on accuracy alone as a measure of performance - especially with skewed datasets. 
  * The trade-off between the recall and precision determines the effectiveness of the classifier.
  * PR and ROC curves provide a nice way to visualize this. 

---

## The ROC Curve

* Sensitivity Precision Trade-off
* The ROC Curve
* ROC Implementation

---
## Reading

* Chapter 3 from ***Hands-on Machine Learning with Scikit-Learn & TensorFlow***

---