diff --git a/topic_17/.ipynb_checkpoints/Untitled-checkpoint.ipynb b/topic_17/.ipynb_checkpoints/Untitled-checkpoint.ipynb
new file mode 100644
index 0000000000000000000000000000000000000000..35eac67af2e6e9a9ec924d04c98b18eba5d2988d
--- /dev/null
+++ b/topic_17/.ipynb_checkpoints/Untitled-checkpoint.ipynb
@@ -0,0 +1,32 @@
+{
+ "cells": [
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": []
+ }
+ ],
+ "metadata": {
+ "kernelspec": {
+ "display_name": "Python 3",
+ "language": "python",
+ "name": "python3"
+ },
+ "language_info": {
+ "codemirror_mode": {
+ "name": "ipython",
+ "version": 3
+ },
+ "file_extension": ".py",
+ "mimetype": "text/x-python",
+ "name": "python",
+ "nbconvert_exporter": "python",
+ "pygments_lexer": "ipython3",
+ "version": "3.7.6"
+ }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 4
+}
diff --git a/topic_17/.ipynb_checkpoints/topic_17_classification_demo-checkpoint.ipynb b/topic_17/.ipynb_checkpoints/topic_17_classification_demo-checkpoint.ipynb
new file mode 100644
index 0000000000000000000000000000000000000000..96fe8ccf5adb3ac5bbd73563e8e62561de99e718
--- /dev/null
+++ b/topic_17/.ipynb_checkpoints/topic_17_classification_demo-checkpoint.ipynb
@@ -0,0 +1,222 @@
+{
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "# Topic 17 - Classification Demonstration\n",
+ "\n",
+ "In this demonstration we will train some classifier algorithms on the MNIST data set.\n",
+ "\n",
+ "We will start by loading the dataset from the Scikit-learn library.\n",
+ "\n",
+ "**Note: This may take a long time to execute - there are 70000 images.**\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "from sklearn.datasets import fetch_openml\n",
+ "mnist = fetch_openml('mnist_784', version=1)\n",
+ "mnist.keys()\n",
+ "dict_keys(['data', 'target', 'feature_names', 'DESCR', 'details',\n",
+ "'categories', 'url'])"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 2,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "**Author**: Yann LeCun, Corinna Cortes, Christopher J.C. Burges \n",
+ "**Source**: [MNIST Website](http://yann.lecun.com/exdb/mnist/) - Date unknown \n",
+ "**Please cite**: \n",
+ "\n",
+ "The MNIST database of handwritten digits with 784 features, raw data available at: http://yann.lecun.com/exdb/mnist/. It can be split in a training set of the first 60,000 examples, and a test set of 10,000 examples \n",
+ "\n",
+ "It is a subset of a larger set available from NIST. The digits have been size-normalized and centered in a fixed-size image. It is a good database for people who want to try learning techniques and pattern recognition methods on real-world data while spending minimal efforts on preprocessing and formatting. The original black and white (bilevel) images from NIST were size normalized to fit in a 20x20 pixel box while preserving their aspect ratio. The resulting images contain grey levels as a result of the anti-aliasing technique used by the normalization algorithm. the images were centered in a 28x28 image by computing the center of mass of the pixels, and translating the image so as to position this point at the center of the 28x28 field. \n",
+ "\n",
+ "With some classification methods (particularly template-based methods, such as SVM and K-nearest neighbors), the error rate improves when the digits are centered by bounding box rather than center of mass. If you do this kind of pre-processing, you should report it in your publications. The MNIST database was constructed from NIST's NIST originally designated SD-3 as their training set and SD-1 as their test set. However, SD-3 is much cleaner and easier to recognize than SD-1. The reason for this can be found on the fact that SD-3 was collected among Census Bureau employees, while SD-1 was collected among high-school students. Drawing sensible conclusions from learning experiments requires that the result be independent of the choice of training set and test among the complete set of samples. Therefore it was necessary to build a new database by mixing NIST's datasets. \n",
+ "\n",
+ "The MNIST training set is composed of 30,000 patterns from SD-3 and 30,000 patterns from SD-1. Our test set was composed of 5,000 patterns from SD-3 and 5,000 patterns from SD-1. The 60,000 pattern training set contained examples from approximately 250 writers. We made sure that the sets of writers of the training set and test set were disjoint. SD-1 contains 58,527 digit images written by 500 different writers. In contrast to SD-3, where blocks of data from each writer appeared in sequence, the data in SD-1 is scrambled. Writer identities for SD-1 is available and we used this information to unscramble the writers. We then split SD-1 in two: characters written by the first 250 writers went into our new training set. The remaining 250 writers were placed in our test set. Thus we had two sets with nearly 30,000 examples each. The new training set was completed with enough examples from SD-3, starting at pattern # 0, to make a full set of 60,000 training patterns. Similarly, the new test set was completed with SD-3 examples starting at pattern # 35,000 to make a full set with 60,000 test patterns. Only a subset of 10,000 test images (5,000 from SD-1 and 5,000 from SD-3) is available on this site. The full 60,000 sample training set is available.\n",
+ "\n",
+ "Downloaded from openml.org.\n"
+ ]
+ }
+ ],
+ "source": [
+ "print(mnist.DESCR)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Next we will preview one of the images and its corresponding label."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 4,
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "image/png": "iVBORw0KGgoAAAANSUhEUgAAAOcAAADnCAYAAADl9EEgAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAGaElEQVR4nO3dPUiWfR/G8dveSyprs2gOXHqhcAh6hZqsNRqiJoPKRYnAoTGorWyLpqhFcmgpEmqIIByKXiAHIaKhFrGghiJ81ucBr991Z/Z4XPr5jB6cXSfVtxP6c2rb9PT0P0CeJfN9A8DMxAmhxAmhxAmhxAmhljXZ/Vcu/H1tM33RkxNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCLZvvG+B//fr1q9y/fPnyVz9/aGio4fb9+/fy2vHx8XK/ceNGuQ8MDDTc7t69W167atWqcr948WK5X7p0qdzngycnhBInhBInhBInhBInhBInhBInhHLOOYMPHz6U+48fP8r92bNn5f706dOG29TUVHnt8PBwuc+nLVu2lPv58+fLfWRkpOG2du3a8tpt27aV+759+8o9kScnhBInhBInhBInhBInhBInhGqbnp6u9nJsVS9evCj3gwcPlvvffm0r1dKlS8v91q1b5d7e3j7rz960aVO5b9iwody3bt0668/+P2ib6YuenBBKnBBKnBBKnBBKnBBKnBBKnBBqUZ5zTk5Olnt3d3e5T0xMzOXtzKlm997sPPDx48cNtxUrVpTXLtbz3zngnBNaiTghlDghlDghlDghlDghlDgh1KL81pgbN24s96tXr5b7/fv3y33Hjh3l3tfXV+6V7du3l/vo6Gi5N3un8s2bNw23a9euldcytzw5IZQ4IZQ4IZQ4IZQ4IZQ4IZQ4IdSifJ/zT339+rXcm/24ut7e3obbzZs3y2tv375d7idOnCh3InmfE1qJOCGUOCGUOCGUOCGUOCGUOCHUonyf80+tW7fuj65fv379rK9tdg56/Pjxcl+yxL/HrcKfFIQSJ4QSJ4QSJ4QSJ4QSJ4Tyytg8+PbtW8Otp6envPbJkyfl/uDBg3I/fPhwuTMvvDIGrUScEEqcEEqcEEqcEEqcEEqcEMo5Z5iJiYly37lzZ7l3dHSU+4EDB8p9165dDbezZ8+W17a1zXhcR3POOaGViBNCiRNCiRNCiRNCiRNCiRNCOedsMSMjI+V++vTpcm/24wsrly9fLveTJ0+We2dn56w/e4FzzgmtRJwQSpwQSpwQSpwQSpwQSpwQyjnnAvP69ety7+/vL/fR0dFZf/aZM2fKfXBwsNw3b948689ucc45oZWIE0KJE0KJE0KJE0KJE0KJE0I551xkpqamyv3+/fsNt1OnTpXXNvm79M+hQ4fK/dGjR+W+gDnnhFYiTgglTgglTgglTgglTgjlKIV/beXKleX+8+fPcl++fHm5P3z4sOG2f//+8toW5ygFWok4IZQ4IZQ4IZQ4IZQ4IZQ4IdSy+b4B5tarV6/KfXh4uNzHxsYabs3OMZvp6uoq97179/7Rr7/QeHJCKHFCKHFCKHFCKHFCKHFCKHFCKOecYcbHx8v9+vXr5X7v3r1y//Tp02/f07+1bFn916mzs7PclyzxrPhvfjcglDghlDghlDghlDghlDghlDghlHPOv6DZWeKdO3cabkNDQ+W179+/n80tzYndu3eX++DgYLkfPXp0Lm9nwfPkhFDihFDihFDihFDihFDihFCOUmbw+fPncn/79m25nzt3rtzfvXv32/c0V7q7u8v9woULDbdjx46V13rla2753YRQ4oRQ4oRQ4oRQ4oRQ4oRQ4oRQC/acc3JysuHW29tbXvvy5ctyn5iYmNU9zYU9e/aUe39/f7kfOXKk3FevXv3b98Tf4ckJocQJocQJocQJocQJocQJocQJoWLPOZ8/f17uV65cKfexsbGG28ePH2d1T3NlzZo1Dbe+vr7y2mbffrK9vX1W90QeT04IJU4IJU4IJU4IJU4IJU4IJU4IFXvOOTIy8kf7n+jq6ir3np6ecl+6dGm5DwwMNNw6OjrKa1k8PDkhlDghlDghlDghlDghlDghlDghVNv09HS1lyMwJ9pm+qInJ4QSJ4QSJ4QSJ4QSJ4QSJ4QSJ4QSJ4QSJ4QSJ4QSJ4QSJ4QSJ4QSJ4QSJ4QSJ4QSJ4QSJ4QSJ4QSJ4QSJ4QSJ4Rq9iMAZ/yWfcDf58kJocQJocQJocQJocQJocQJof4DO14Dhyk10VwAAAAASUVORK5CYII=\n",
+ "text/plain": [
+ "<Figure size 432x288 with 1 Axes>"
+ ]
+ },
+ "metadata": {
+ "needs_background": "light"
+ },
+ "output_type": "display_data"
+ }
+ ],
+ "source": [
+ "import matplotlib as mpl\n",
+ "import matplotlib.pyplot as plt\n",
+ "\n",
+ "#Assign the data and the targets\n",
+ "X, y = mnist[\"data\"], mnist[\"target\"]\n",
+ "\n",
+ "#Display the first image\n",
+ "some_digit = X[0]\n",
+ "some_digit_image = some_digit.reshape(28, 28)\n",
+ "plt.imshow(some_digit_image, cmap=\"binary\")\n",
+ "plt.axis(\"off\")\n",
+ "plt.show()"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 5,
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "'5'"
+ ]
+ },
+ "execution_count": 5,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "#The corresponding class label\n",
+ "y[0]"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Next we will partition the data into a training and testing sets. We are going to train a binary classifier that detects the number \"5\", so we need to find all of the 5s and mark them as \"true\" in the train and test sets."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 13,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "import numpy as np\n",
+ "y = y.astype(np.uint8)\n",
+ "X_train, X_test, y_train, y_test = X[:60000], X[60000:], y[:60000], y[60000:]\n",
+ "y_train_5 = (y_train == 5) # True for all 5s, False for all other digits\n",
+ "y_test_5 = (y_test == 5)\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Now its time to train the classifier. We will train a stochastic gradient descent classifier.\n",
+ "\n",
+ "We will set the random state for reproducability.\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 18,
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "SGDClassifier(alpha=0.0001, average=False, class_weight=None,\n",
+ " early_stopping=False, epsilon=0.1, eta0=0.0, fit_intercept=True,\n",
+ " l1_ratio=0.15, learning_rate='optimal', loss='hinge',\n",
+ " max_iter=1000, n_iter_no_change=5, n_jobs=None, penalty='l2',\n",
+ " power_t=0.5, random_state=42, shuffle=True, tol=0.001,\n",
+ " validation_fraction=0.1, verbose=0, warm_start=False)"
+ ]
+ },
+ "execution_count": 18,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "from sklearn.linear_model import SGDClassifier\n",
+ "\n",
+ "sgd_clf = SGDClassifier(random_state=42)\n",
+ "sgd_clf.fit(X_train, y_train_5)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Now we can do a simple test to see if our classifier can classify a an image from the training set. We will look at a full test later.\n",
+ "\n",
+ "The letter in `X[0]` represents a \"5\", hence a result of \"true\" is expected."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 21,
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "array([ True])"
+ ]
+ },
+ "execution_count": 21,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "sgd_clf.predict([X[0]])"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": []
+ }
+ ],
+ "metadata": {
+ "kernelspec": {
+ "display_name": "Python 3",
+ "language": "python",
+ "name": "python3"
+ }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 4
+}
diff --git a/topic_17/Untitled.ipynb b/topic_17/Untitled.ipynb
new file mode 100644
index 0000000000000000000000000000000000000000..35eac67af2e6e9a9ec924d04c98b18eba5d2988d
--- /dev/null
+++ b/topic_17/Untitled.ipynb
@@ -0,0 +1,32 @@
+{
+ "cells": [
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": []
+ }
+ ],
+ "metadata": {
+ "kernelspec": {
+ "display_name": "Python 3",
+ "language": "python",
+ "name": "python3"
+ },
+ "language_info": {
+ "codemirror_mode": {
+ "name": "ipython",
+ "version": 3
+ },
+ "file_extension": ".py",
+ "mimetype": "text/x-python",
+ "name": "python",
+ "nbconvert_exporter": "python",
+ "pygments_lexer": "ipython3",
+ "version": "3.7.6"
+ }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 4
+}
diff --git a/topic_17/topic_17.md b/topic_17/topic_17.md
new file mode 100644
index 0000000000000000000000000000000000000000..24f9546a7ee60e19febcba31c2b437d1f62ee1fe
--- /dev/null
+++ b/topic_17/topic_17.md
@@ -0,0 +1,154 @@
+class: bottom, left
+background-image: url(assets/g.png)
+
+<h2 class="title_headings_sml">COSC102 - Data Science Studio 1</h2>
+
+<h1 class="title_headings_sml"> Topic 17 - Classification using Machine Learning </h1>
+
+<h3 class="title_headings_sml"> Dr. Mitchell Welch </h3>
+
+---
+
+## Reading
+
+* Chapter 3 from ***Hands-on Machine Learning with Scikit-Learn & TensorFlow***
+
+---
+## Summary
+
+* Machine Learning & Classification
+* MNIST Dataset
+* Partitioning the Data Set
+* Training the Classifier
+---
+
+## Machine Learning & Classification
+
+* Recall from the previous topics that the most common supervised learning tasks are:
+ * Regression (predicting values)
+ * Classification (predicting category classes)
+* In machine learning, classification refers to a predictive modeling problem where a class label is predicted for a given example of input data.
+* Examples: Activity recognition from accelerometer sensors, natural language processing, plant species classification.
+---
+
+## Machine Learning & Classification
+
+.center[]
+
+
+---
+## Machine Learning & Classification
+
+* Classification requires a training dataset with **many** examples of inputs and outputs from which to learn.
+* *Binary classification* refers to those classification tasks that have two class labels.
+* *Multi-class classification* refers to those classification tasks that have more than two class labels.
+
+---
+
+## MNIST Dataset
+
+* To continue our discussion on machine learning, we will introduce the *MNIST* dataset.
+* The MNIST dataset contains 70,000 small images of digits handwritten by high school students and employees of the US Census Bureau.
+* Each image is labeled with the digit it represents.
+* The digit images (technically the pixels from the images) are the input features and the ground-truth labels are the output.
+* We will train some classifier algorithms to predict the digit given the image.
+
+
+---
+
+## MNIST Dataset
+
+* A copy of the MNIST dataset is available within Scikit-learn.
+* Review the demonstration notebook to how to load and preview the data.
+* There are 70,000 images, and each image has 784 features.
+* This is because each image is 28 × 28 pixels, and each feature simply represents one pixel’s intensity, from 0 (white) to 255 (black).
+
+---
+## MNIST Dataset
+
+.center[]
+
+---
+
+## Partitioning the Data Set
+
+* Before we move on to train our classifier, we need to divide our data set into a *training* and *testing* set
+* The training set is used to train and tune the classifier.
+* The test set is used to test the predictive ability of the classifier.
+* None of the data items in the test set should be used as part of the training process.
+
+---
+
+## Partitioning the Data Set
+
+* If we look at the MNIST dataset packaged with Scikit-Learn, we can see it has test/train partitions built in:
+ * The first 60000 images make up the training set.
+ * The last 10000 images make up the testing set.
+* This partitioning ensures that there are equal examples of each digit within each set.
+
+```python
+X_train, X_test, y_train, y_test = X[:60000], X[60000:], y[:60000], y[60000:]
+```
+
+---
+
+## Training the Classifier
+
+* Now we can investigate some classifier algorithms.
+* Let’s simplify the problem for now and only try to identify one digit—for example, the number "5"
+* This “5-detector” will be an example of a binary classifier, capable of distinguishing between just two classes, "5" and "not-5"
+
+---
+## Training the Classifier
+
+* We will train a linear classifier using Stochastic Gradient Descent (SGD) algorithm.
+* Strictly speaking, SGD is merely an optimization technique and does not correspond to a specific family of machine learning models.
+* We will be using the default model: the *Support Vector Machine (SVM)*
+* In a later topic we will cover the workings of the SVM.
+---
+
+## Training the Classifier
+
+* The Support Vector Machine (SVM) works by constructing a *hyperplane* - essentially a decision boundary that separates the classes within the data.
+* This algorithm is very efficient and works well on data the is *linearly separable* (classes that can be separated by a straight line within the features)
+* This classifier has the advantage of being capable of handling very large datasets efficiently.
+
+---
+## Training the Classifier
+
+.center[]
+
+---
+## Training the Classifier
+
+* Now its time to actually train the classifier using the training set.
+
+```python
+from sklearn.linear_model import SGDClassifier
+sgd_clf = SGDClassifier(random_state=42)
+sgd_clf.fit(X_train, y_train_5)
+
+# A quick test on our classifier
+sgd_clf.predict([X[0]])
+
+```
+
+---
+## Summary
+
+* Machine Learning & Classification
+* MNIST Dataset
+* Partitioning the Data Set
+* Training the Classifier
+
+---
+
+## Reading
+
+* Chapter 3 from ***Hands-on Machine Learning with Scikit-Learn & TensorFlow***
+
+---
+
+
+
+
diff --git a/topic_17/topic_17_classification_demo.ipynb b/topic_17/topic_17_classification_demo.ipynb
new file mode 100644
index 0000000000000000000000000000000000000000..0292ffa265337bd544f10f731d4e9c4f43726714
--- /dev/null
+++ b/topic_17/topic_17_classification_demo.ipynb
@@ -0,0 +1,222 @@
+{
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "# Topic 17 - Classification Demonstration\n",
+ "\n",
+ "In this demonstration we will train some classifier algorithms on the MNIST data set.\n",
+ "\n",
+ "We will start by loading the dataset from the Scikit-learn library.\n",
+ "\n",
+ "**Note: This may take a long time to execute - there are 70000 images.**\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "from sklearn.datasets import fetch_openml\n",
+ "mnist = fetch_openml('mnist_784', version=1)\n",
+ "mnist.keys()\n",
+ "dict_keys(['data', 'target', 'feature_names', 'DESCR', 'details',\n",
+ "'categories', 'url'])"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 2,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "**Author**: Yann LeCun, Corinna Cortes, Christopher J.C. Burges \n",
+ "**Source**: [MNIST Website](http://yann.lecun.com/exdb/mnist/) - Date unknown \n",
+ "**Please cite**: \n",
+ "\n",
+ "The MNIST database of handwritten digits with 784 features, raw data available at: http://yann.lecun.com/exdb/mnist/. It can be split in a training set of the first 60,000 examples, and a test set of 10,000 examples \n",
+ "\n",
+ "It is a subset of a larger set available from NIST. The digits have been size-normalized and centered in a fixed-size image. It is a good database for people who want to try learning techniques and pattern recognition methods on real-world data while spending minimal efforts on preprocessing and formatting. The original black and white (bilevel) images from NIST were size normalized to fit in a 20x20 pixel box while preserving their aspect ratio. The resulting images contain grey levels as a result of the anti-aliasing technique used by the normalization algorithm. the images were centered in a 28x28 image by computing the center of mass of the pixels, and translating the image so as to position this point at the center of the 28x28 field. \n",
+ "\n",
+ "With some classification methods (particularly template-based methods, such as SVM and K-nearest neighbors), the error rate improves when the digits are centered by bounding box rather than center of mass. If you do this kind of pre-processing, you should report it in your publications. The MNIST database was constructed from NIST's NIST originally designated SD-3 as their training set and SD-1 as their test set. However, SD-3 is much cleaner and easier to recognize than SD-1. The reason for this can be found on the fact that SD-3 was collected among Census Bureau employees, while SD-1 was collected among high-school students. Drawing sensible conclusions from learning experiments requires that the result be independent of the choice of training set and test among the complete set of samples. Therefore it was necessary to build a new database by mixing NIST's datasets. \n",
+ "\n",
+ "The MNIST training set is composed of 30,000 patterns from SD-3 and 30,000 patterns from SD-1. Our test set was composed of 5,000 patterns from SD-3 and 5,000 patterns from SD-1. The 60,000 pattern training set contained examples from approximately 250 writers. We made sure that the sets of writers of the training set and test set were disjoint. SD-1 contains 58,527 digit images written by 500 different writers. In contrast to SD-3, where blocks of data from each writer appeared in sequence, the data in SD-1 is scrambled. Writer identities for SD-1 is available and we used this information to unscramble the writers. We then split SD-1 in two: characters written by the first 250 writers went into our new training set. The remaining 250 writers were placed in our test set. Thus we had two sets with nearly 30,000 examples each. The new training set was completed with enough examples from SD-3, starting at pattern # 0, to make a full set of 60,000 training patterns. Similarly, the new test set was completed with SD-3 examples starting at pattern # 35,000 to make a full set with 60,000 test patterns. Only a subset of 10,000 test images (5,000 from SD-1 and 5,000 from SD-3) is available on this site. The full 60,000 sample training set is available.\n",
+ "\n",
+ "Downloaded from openml.org.\n"
+ ]
+ }
+ ],
+ "source": [
+ "print(mnist.DESCR)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Next we will preview one of the images and its corresponding label."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 32,
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "image/png": "iVBORw0KGgoAAAANSUhEUgAAAOcAAADnCAYAAADl9EEgAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAFPUlEQVR4nO3dPUiVbRzHcc+TREI2NDRGiAiBSxIO7dUgtAbq5qrQWGsvaGJULg0OQktTQXtuDr6Ak1NbNLRk5KKjTT0Qj/ffB092fsc+n7Ef1+lQfLmhi9taBwcHPUCefzr9BYDDiRNCiRNCiRNCiRNC9R6x+6dcOHmtw37RkxNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNC9Xb6C8BPnz59atz29/fLs2/evCn3V69eHes7/TQ2Nta4LS8vt/XZTTw5IZQ4IZQ4IZQ4IZQ4IZQ4IZQ4IZR7Tn6bDx8+lPu7d+/Kvbqr/P79e3m21WqVe7vW1tZO9PMP48kJocQJocQJocQJocQJocQJocQJodxz8oupqanGbXt7uzy7sbHxu7/Ovy5cuFDuExMT5X79+vVyHx8fL/dz586V+0nw5IRQ4oRQ4oRQ4oRQ4oRQ4oRQ4oRQrYODg2ovR/Ls7OyU+4MHD8p9aWmpcbt48WJ5dmBgoNzv379f7sPDw41bX19fefby5cvlHu7Ql1E9OSGUOCGUOCGUOCGUOCGUOCGUOCGUe85T5t69e+W+uLhY7jMzM43bkydPyrPnz58vdxq554RuIk4IJU4IJU4IJU4IJU4I5SqlA/b29hq3p0+flmdfv35d7i9fviz3I/6+e27fvt24deLHQ/4lXKVANxEnhBInhBInhBInhBInhBInhPJfAHbA48ePG7e5ubny7N27d8v91q1b5e6usnt4ckIocUIocUIocUIocUIocUIocUIo73N2QKt16Ot7/8v79+/L/c6dO8f+bDrG+5zQTcQJocQJocQJocQJocQJocQJobzP2QGjo6ON2+bmZnl2enq63Pv6+sr95s2b5U4OT04IJU4IJU4IJU4IJU4IJU4IJU4I5X3OQ6yvr5f7tWvXyv3s2bPl/u3bt8ZtcXGxPPvw4cNy7+/vL/e1tbVyv3r1arlzIrzPCd1EnBBKnBBKnBBKnBBKnBDq1F6lfPnypXEbGxsrz37+/Lncnz9/Xu6Tk5PlXvn69Wu5X7p06dif3dPT07O6ulruN27caOvzORZXKdBNxAmhxAmhxAmhxAmhxAmhxAmhTu2PxhwZGWncdnd3y7Pz8/Pl3s495lFevHjR1vmjfvTl8PBwW5/Pn+PJCaHECaHECaHECaHECaHECaHECaFO7fucs7OzjdujR4/Ks/v7+7/76/xiaGiocfv48WN59sqVK+X+9u3bcq/uf+kY73NCNxEnhBInhBInhBInhBInhBInhDq195yVhYWFct/a2ir3lZWVtn7/6s98dHS0PPvs2bNyHxwcLPczZ86UOx3hnhO6iTghlDghlDghlDghlDgh1F95lQJhXKVANxEnhBInhBInhBInhBInhBInhBInhBInhBInhBInhBInhBInhBInhBInhBInhBInhBInhBInhBInhBInhBInhBInhBInhBInhBInhBInhBInhBInhBInhBInhBInhBInhBInhBInhBInhBInhBInhBInhBInhBInhBInhBInhBInhBInhBInhBInhBInhBInhBInhBInhBInhBInhBInhOo9Ym/9kW8B/IcnJ4QSJ4QSJ4QSJ4QSJ4QSJ4T6AZAorng0D88IAAAAAElFTkSuQmCC\n",
+ "text/plain": [
+ "<Figure size 432x288 with 1 Axes>"
+ ]
+ },
+ "metadata": {
+ "needs_background": "light"
+ },
+ "output_type": "display_data"
+ }
+ ],
+ "source": [
+ "import matplotlib as mpl\n",
+ "import matplotlib.pyplot as plt\n",
+ "\n",
+ "#Assign the data and the targets\n",
+ "X, y = mnist[\"data\"], mnist[\"target\"]\n",
+ "\n",
+ "#Display the first image\n",
+ "some_digit = X[11]\n",
+ "some_digit_image = some_digit.reshape(28, 28)\n",
+ "plt.imshow(some_digit_image, cmap=\"binary\")\n",
+ "plt.axis(\"off\")\n",
+ "plt.show()"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 33,
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "'5'"
+ ]
+ },
+ "execution_count": 33,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "#The corresponding class label\n",
+ "y[11]"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Next we will partition the data into a training and testing sets. We are going to train a binary classifier that detects the number \"5\", so we need to find all of the 5s and mark them as \"true\" in the train and test sets."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 28,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "import numpy as np\n",
+ "y = y.astype(np.uint8)\n",
+ "X_train, X_test, y_train, y_test = X[:60000], X[60000:], y[:60000], y[60000:]\n",
+ "y_train_5 = (y_train == 5) # True for all 5s, False for all other digits\n",
+ "y_test_5 = (y_test == 5)\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Now its time to train the classifier. We will train a stochastic gradient descent classifier.\n",
+ "\n",
+ "We will set the random state for reproducability.\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 18,
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "SGDClassifier(alpha=0.0001, average=False, class_weight=None,\n",
+ " early_stopping=False, epsilon=0.1, eta0=0.0, fit_intercept=True,\n",
+ " l1_ratio=0.15, learning_rate='optimal', loss='hinge',\n",
+ " max_iter=1000, n_iter_no_change=5, n_jobs=None, penalty='l2',\n",
+ " power_t=0.5, random_state=42, shuffle=True, tol=0.001,\n",
+ " validation_fraction=0.1, verbose=0, warm_start=False)"
+ ]
+ },
+ "execution_count": 18,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "from sklearn.linear_model import SGDClassifier\n",
+ "\n",
+ "sgd_clf = SGDClassifier(random_state=42)\n",
+ "sgd_clf.fit(X_train, y_train_5)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Now we can do a simple test to see if our classifier can classify a an image from the training set. We will look at a full test later.\n",
+ "\n",
+ "The letter in `X[0]` represents a \"5\", hence a result of \"true\" is expected."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 34,
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "array([ True])"
+ ]
+ },
+ "execution_count": 34,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "sgd_clf.predict([X[11]])"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": []
+ }
+ ],
+ "metadata": {
+ "kernelspec": {
+ "display_name": "Python 3",
+ "language": "python",
+ "name": "python3"
+ }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 4
+}
diff --git a/topic_18/.ipynb_checkpoints/Untitled-checkpoint.ipynb b/topic_18/.ipynb_checkpoints/Untitled-checkpoint.ipynb
new file mode 100644
index 0000000000000000000000000000000000000000..146acb87f0df56fbe0f239b95d44e36269ea3144
--- /dev/null
+++ b/topic_18/.ipynb_checkpoints/Untitled-checkpoint.ipynb
@@ -0,0 +1,67 @@
+{
+ "cells": [
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": []
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "from sklearn.datasets import fetch_openml\n",
+ "mnist = fetch_openml('mnist_784', version=1)\n",
+ "mnist.keys()\n",
+ "dict_keys(['data', 'target', 'feature_names', 'DESCR', 'details',\n",
+ "'categories', 'url'])\n",
+ "\n",
+ "import matplotlib as mpl\n",
+ "import matplotlib.pyplot as plt\n",
+ "\n",
+ "X, y = mnist[\"data\"], mnist[\"target\"]\n",
+ "\n",
+ "some_digit = X[0]\n",
+ "some_digit_image = some_digit.reshape(28, 28)\n",
+ "plt.imshow(some_digit_image, cmap=\"binary\")\n",
+ "plt.axis(\"off\")\n",
+ "plt.show()\n",
+ "\n",
+ "import numpy as np\n",
+ "y = y.astype(np.uint8)\n",
+ "X_train, X_test, y_train, y_test = X[:60000], X[60000:], y[:60000], y[60000:]\n",
+ "y_train_5 = (y_train == 5) # True for all 5s, False for all other digits\n",
+ "y_test_5 = (y_test == 5)\n",
+ "\n",
+ "from sklearn.linear_model import SGDClassifier\n",
+ "\n",
+ "sgd_clf = SGDClassifier(random_state=42)\n",
+ "sgd_clf.fit(X_train, y_train_5)"
+ ]
+ }
+ ],
+ "metadata": {
+ "kernelspec": {
+ "display_name": "Python 3",
+ "language": "python",
+ "name": "python3"
+ },
+ "language_info": {
+ "codemirror_mode": {
+ "name": "ipython",
+ "version": 3
+ },
+ "file_extension": ".py",
+ "mimetype": "text/x-python",
+ "name": "python",
+ "nbconvert_exporter": "python",
+ "pygments_lexer": "ipython3",
+ "version": "3.7.6"
+ }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 4
+}
diff --git a/topic_18/.ipynb_checkpoints/performance_demo-checkpoint.ipynb b/topic_18/.ipynb_checkpoints/performance_demo-checkpoint.ipynb
new file mode 100644
index 0000000000000000000000000000000000000000..143ea6e97e9cfb97731a7cfffd6ae228f7f877c5
--- /dev/null
+++ b/topic_18/.ipynb_checkpoints/performance_demo-checkpoint.ipynb
@@ -0,0 +1,100 @@
+{
+ "cells": [
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": []
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 2,
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "image/png": "iVBORw0KGgoAAAANSUhEUgAAAOcAAADnCAYAAADl9EEgAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAGaElEQVR4nO3dPUiWfR/G8dveSyprs2gOXHqhcAh6hZqsNRqiJoPKRYnAoTGorWyLpqhFcmgpEmqIIByKXiAHIaKhFrGghiJ81ucBr991Z/Z4XPr5jB6cXSfVtxP6c2rb9PT0P0CeJfN9A8DMxAmhxAmhxAmhxAmhljXZ/Vcu/H1tM33RkxNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCLZvvG+B//fr1q9y/fPnyVz9/aGio4fb9+/fy2vHx8XK/ceNGuQ8MDDTc7t69W167atWqcr948WK5X7p0qdzngycnhBInhBInhBInhBInhBInhBInhHLOOYMPHz6U+48fP8r92bNn5f706dOG29TUVHnt8PBwuc+nLVu2lPv58+fLfWRkpOG2du3a8tpt27aV+759+8o9kScnhBInhBInhBInhBInhBInhGqbnp6u9nJsVS9evCj3gwcPlvvffm0r1dKlS8v91q1b5d7e3j7rz960aVO5b9iwody3bt0668/+P2ib6YuenBBKnBBKnBBKnBBKnBBKnBBKnBBqUZ5zTk5Olnt3d3e5T0xMzOXtzKlm997sPPDx48cNtxUrVpTXLtbz3zngnBNaiTghlDghlDghlDghlDghlDgh1KL81pgbN24s96tXr5b7/fv3y33Hjh3l3tfXV+6V7du3l/vo6Gi5N3un8s2bNw23a9euldcytzw5IZQ4IZQ4IZQ4IZQ4IZQ4IZQ4IdSifJ/zT339+rXcm/24ut7e3obbzZs3y2tv375d7idOnCh3InmfE1qJOCGUOCGUOCGUOCGUOCGUOCHUonyf80+tW7fuj65fv379rK9tdg56/Pjxcl+yxL/HrcKfFIQSJ4QSJ4QSJ4QSJ4QSJ4Tyytg8+PbtW8Otp6envPbJkyfl/uDBg3I/fPhwuTMvvDIGrUScEEqcEEqcEEqcEEqcEEqcEMo5Z5iJiYly37lzZ7l3dHSU+4EDB8p9165dDbezZ8+W17a1zXhcR3POOaGViBNCiRNCiRNCiRNCiRNCiRNCOedsMSMjI+V++vTpcm/24wsrly9fLveTJ0+We2dn56w/e4FzzgmtRJwQSpwQSpwQSpwQSpwQSpwQyjnnAvP69ety7+/vL/fR0dFZf/aZM2fKfXBwsNw3b948689ucc45oZWIE0KJE0KJE0KJE0KJE0KJE0I551xkpqamyv3+/fsNt1OnTpXXNvm79M+hQ4fK/dGjR+W+gDnnhFYiTgglTgglTgglTgglTgjlKIV/beXKleX+8+fPcl++fHm5P3z4sOG2f//+8toW5ygFWok4IZQ4IZQ4IZQ4IZQ4IZQ4IdSy+b4B5tarV6/KfXh4uNzHxsYabs3OMZvp6uoq97179/7Rr7/QeHJCKHFCKHFCKHFCKHFCKHFCKHFCKOecYcbHx8v9+vXr5X7v3r1y//Tp02/f07+1bFn916mzs7PclyzxrPhvfjcglDghlDghlDghlDghlDghlDghlHPOv6DZWeKdO3cabkNDQ+W179+/n80tzYndu3eX++DgYLkfPXp0Lm9nwfPkhFDihFDihFDihFDihFDihFCOUmbw+fPncn/79m25nzt3rtzfvXv32/c0V7q7u8v9woULDbdjx46V13rla2753YRQ4oRQ4oRQ4oRQ4oRQ4oRQ4oRQC/acc3JysuHW29tbXvvy5ctyn5iYmNU9zYU9e/aUe39/f7kfOXKk3FevXv3b98Tf4ckJocQJocQJocQJocQJocQJocQJoWLPOZ8/f17uV65cKfexsbGG28ePH2d1T3NlzZo1Dbe+vr7y2mbffrK9vX1W90QeT04IJU4IJU4IJU4IJU4IJU4IJU4IFXvOOTIy8kf7n+jq6ir3np6ecl+6dGm5DwwMNNw6OjrKa1k8PDkhlDghlDghlDghlDghlDghlDghVNv09HS1lyMwJ9pm+qInJ4QSJ4QSJ4QSJ4QSJ4QSJ4QSJ4QSJ4QSJ4QSJ4QSJ4QSJ4QSJ4QSJ4QSJ4QSJ4QSJ4QSJ4QSJ4QSJ4QSJ4QSJ4Rq9iMAZ/yWfcDf58kJocQJocQJocQJocQJocQJof4DO14Dhyk10VwAAAAASUVORK5CYII=\n",
+ "text/plain": [
+ "<Figure size 432x288 with 1 Axes>"
+ ]
+ },
+ "metadata": {
+ "needs_background": "light"
+ },
+ "output_type": "display_data"
+ },
+ {
+ "data": {
+ "text/plain": [
+ "SGDClassifier(alpha=0.0001, average=False, class_weight=None,\n",
+ " early_stopping=False, epsilon=0.1, eta0=0.0, fit_intercept=True,\n",
+ " l1_ratio=0.15, learning_rate='optimal', loss='hinge',\n",
+ " max_iter=1000, n_iter_no_change=5, n_jobs=None, penalty='l2',\n",
+ " power_t=0.5, random_state=42, shuffle=True, tol=0.001,\n",
+ " validation_fraction=0.1, verbose=0, warm_start=False)"
+ ]
+ },
+ "execution_count": 2,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "from sklearn.datasets import fetch_openml\n",
+ "mnist = fetch_openml('mnist_784', version=1)\n",
+ "mnist.keys()\n",
+ "\n",
+ "import matplotlib as mpl\n",
+ "import matplotlib.pyplot as plt\n",
+ "\n",
+ "X, y = mnist[\"data\"], mnist[\"target\"]\n",
+ "\n",
+ "some_digit = X[0]\n",
+ "some_digit_image = some_digit.reshape(28, 28)\n",
+ "plt.imshow(some_digit_image, cmap=\"binary\")\n",
+ "plt.axis(\"off\")\n",
+ "plt.show()\n",
+ "\n",
+ "import numpy as np\n",
+ "y = y.astype(np.uint8)\n",
+ "X_train, X_test, y_train, y_test = X[:60000], X[60000:], y[:60000], y[60000:]\n",
+ "y_train_5 = (y_train == 5) # True for all 5s, False for all other digits\n",
+ "y_test_5 = (y_test == 5)\n",
+ "\n",
+ "from sklearn.linear_model import SGDClassifier\n",
+ "\n",
+ "sgd_clf = SGDClassifier(random_state=42)\n",
+ "sgd_clf.fit(X_train, y_train_5)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": []
+ }
+ ],
+ "metadata": {
+ "kernelspec": {
+ "display_name": "Python 3",
+ "language": "python",
+ "name": "python3"
+ },
+ "language_info": {
+ "codemirror_mode": {
+ "name": "ipython",
+ "version": 3
+ },
+ "file_extension": ".py",
+ "mimetype": "text/x-python",
+ "name": "python",
+ "nbconvert_exporter": "python",
+ "pygments_lexer": "ipython3",
+ "version": "3.7.6"
+ }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 4
+}
diff --git a/topic_18/Untitled.ipynb b/topic_18/Untitled.ipynb
new file mode 100644
index 0000000000000000000000000000000000000000..146acb87f0df56fbe0f239b95d44e36269ea3144
--- /dev/null
+++ b/topic_18/Untitled.ipynb
@@ -0,0 +1,67 @@
+{
+ "cells": [
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": []
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "from sklearn.datasets import fetch_openml\n",
+ "mnist = fetch_openml('mnist_784', version=1)\n",
+ "mnist.keys()\n",
+ "dict_keys(['data', 'target', 'feature_names', 'DESCR', 'details',\n",
+ "'categories', 'url'])\n",
+ "\n",
+ "import matplotlib as mpl\n",
+ "import matplotlib.pyplot as plt\n",
+ "\n",
+ "X, y = mnist[\"data\"], mnist[\"target\"]\n",
+ "\n",
+ "some_digit = X[0]\n",
+ "some_digit_image = some_digit.reshape(28, 28)\n",
+ "plt.imshow(some_digit_image, cmap=\"binary\")\n",
+ "plt.axis(\"off\")\n",
+ "plt.show()\n",
+ "\n",
+ "import numpy as np\n",
+ "y = y.astype(np.uint8)\n",
+ "X_train, X_test, y_train, y_test = X[:60000], X[60000:], y[:60000], y[60000:]\n",
+ "y_train_5 = (y_train == 5) # True for all 5s, False for all other digits\n",
+ "y_test_5 = (y_test == 5)\n",
+ "\n",
+ "from sklearn.linear_model import SGDClassifier\n",
+ "\n",
+ "sgd_clf = SGDClassifier(random_state=42)\n",
+ "sgd_clf.fit(X_train, y_train_5)"
+ ]
+ }
+ ],
+ "metadata": {
+ "kernelspec": {
+ "display_name": "Python 3",
+ "language": "python",
+ "name": "python3"
+ },
+ "language_info": {
+ "codemirror_mode": {
+ "name": "ipython",
+ "version": 3
+ },
+ "file_extension": ".py",
+ "mimetype": "text/x-python",
+ "name": "python",
+ "nbconvert_exporter": "python",
+ "pygments_lexer": "ipython3",
+ "version": "3.7.6"
+ }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 4
+}
diff --git a/topic_18/topic_18.md b/topic_18/topic_18.md
new file mode 100644
index 0000000000000000000000000000000000000000..23c1d80081068ad8b5d375b640d61d0144efdfd7
--- /dev/null
+++ b/topic_18/topic_18.md
@@ -0,0 +1,158 @@
+class: bottom, left
+background-image: url(assets/g.png)
+
+<h2 class="title_headings_sml">COSC102 - Data Science Studio 1</h2>
+
+<h1 class="title_headings_sml"> Topic 18 - Classification Performance </h1>
+
+<h3 class="title_headings_sml"> Dr. Mitchell Welch </h3>
+
+---
+
+## Reading
+
+* Chapter 3 from ***Hands-on Machine Learning with Scikit-Learn & TensorFlow***
+
+---
+## Summary
+
+* Classifier Performance
+* Confusion Matrix
+* Precision/Recall
+---
+
+## Classifier Performance
+
+* Carrying on from the previous topic, we have:
+ * Trained a simple classifier.
+ * Used the classifier to predict individual values
+* Now we will complete a more comprehensive assessment of well the classifier performs.
+* To achieve this, we will be using the Scikit-learn built-in cross validation functions.
+
+---
+
+## Classifier Performance
+
+* One of the easiest ways to assess the performance is with *K-Fold* cross-validation.
+* The process involves splitting the dataset into *folds* (subsets)
+* All but one fold is used to train the classifier, and the remaining fold is used to test the classifier.
+* This process is repeated for all combinations of the folds and performance measures are averaged across tests on all combinations of folds.
+
+
+---
+
+## Classifier Performance
+
+.center[]
+
+---
+
+## Classifier Performance
+
+* This is all built in to Scikit-learn:
+* Our fist try:
+
+```python
+from sklearn.model_selection import cross_val_score
+cross_val_score(sgd_clf, X_train, y_train_5, cv=3, scoring="accuracy")
+
+```
+
+---
+
+## Classifier Performance
+
+* At first glance this looks good, with 90+% accuracy. But this is hiding a very ineffective classifier.
+* The problem is that the classifier has been trained and assessed on an unbalanced dataset.
+* Only about 10% of the dataset contains '5's.
+* It just classifiers everything as in the 'not-5' class to achieve this accuracy.
+* We need a better way to assess the performance.
+
+---
+
+## Confusion Matrix
+
+* We will look at using a *Confusion Matrix*
+* The general idea is to count the number of times instances of class A are
+classified as class B.
+* A confusion matrix record:
+ * True positives (TP) - correctly identified positive samples
+ * True negatives (TN) - correctly identified negative samples
+ * False positives (FP) - incorrectly identified positive samples
+ * False negatives (FP) - incorrectly identified negative samples
+
+
+---
+## Confusion Matrix
+
+.center[]
+
+---
+
+## Confusion Matrix
+
+* To compute the confusion matrix, you first need to have a set of predictions so that
+they can be compared to the actual targets.
+* You could make predictions on the test set, but let’s keep it untouched for now (remember that you want to use the test set only at the very end of your project)
+* We will use the `cross_val_predict( ... )` to conduct a k-fold cross validation and return the performance for each fold.
+* Review the [demo notebook]().
+
+---
+
+## Precision/Recall
+
+* Using the results from the confusion matrix, we can calculate some concise metrics.
+* *Precision* - the accuracy of the positive predictions.
+
+$$
+precision = TP / (TP + FP)
+$$
+
+* *Recall* (Sometimes called *Sensitivity*) - The ratio of positive instances that are correctly detected by the classifier.
+
+$$
+recall = TP /(TP + FN)
+$$
+
+---
+
+## Precision/Recall
+
+* Review the demo notebook to see this in action:
+
+```python
+from sklearn.metrics import precision_score, recall_score
+print(precision_score(y_train_5, y_train_pred)) # == 4096 / (4096 + 1522)
+print(recall_score(y_train_5, y_train_pred)) # == 4096 / (4096 + 1325)
+
+```
+
+---
+
+## Precision/Recall
+
+* Precision and recall can be summarized by calculating the *F1* score:
+
+$$
+F1 = TP / (TP + ((FN+FP)/2))
+$$
+
+* This produces a harmonic mean that is only high if *both* the recall and precision are high.
+
+---
+## Summary
+
+* So far we have:
+ * Trained a simple binary classifier (our '5' detector).
+ * Tested it with simple predictions.
+ * Assessed the performance using a confusion matrix.
+ * Calculated the precision, recall and F1 scores.
+
+
+---
+## Summary
+
+* Classifier Performance
+* Confusion Matrix
+* Precision/Recall
+---
\ No newline at end of file
diff --git a/topic_18/topic_18_performance_demo.ipynb b/topic_18/topic_18_performance_demo.ipynb
new file mode 100644
index 0000000000000000000000000000000000000000..5781e71a1c0df34be8f8e3b9cfca218747952b35
--- /dev/null
+++ b/topic_18/topic_18_performance_demo.ipynb
@@ -0,0 +1,400 @@
+{
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "# Topic 18 - Classifier Performance\n",
+ "\n",
+ "In this demonstration we will continue on from where we started in topic 17. First we will train a classifier algorithm on the MNIST data set."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 1,
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "image/png": "iVBORw0KGgoAAAANSUhEUgAAAOcAAADnCAYAAADl9EEgAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAGaElEQVR4nO3dPUiWfR/G8dveSyprs2gOXHqhcAh6hZqsNRqiJoPKRYnAoTGorWyLpqhFcmgpEmqIIByKXiAHIaKhFrGghiJ81ucBr991Z/Z4XPr5jB6cXSfVtxP6c2rb9PT0P0CeJfN9A8DMxAmhxAmhxAmhxAmhljXZ/Vcu/H1tM33RkxNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCLZvvG+B//fr1q9y/fPnyVz9/aGio4fb9+/fy2vHx8XK/ceNGuQ8MDDTc7t69W167atWqcr948WK5X7p0qdzngycnhBInhBInhBInhBInhBInhBInhHLOOYMPHz6U+48fP8r92bNn5f706dOG29TUVHnt8PBwuc+nLVu2lPv58+fLfWRkpOG2du3a8tpt27aV+759+8o9kScnhBInhBInhBInhBInhBInhGqbnp6u9nJsVS9evCj3gwcPlvvffm0r1dKlS8v91q1b5d7e3j7rz960aVO5b9iwody3bt0668/+P2ib6YuenBBKnBBKnBBKnBBKnBBKnBBKnBBqUZ5zTk5Olnt3d3e5T0xMzOXtzKlm997sPPDx48cNtxUrVpTXLtbz3zngnBNaiTghlDghlDghlDghlDghlDgh1KL81pgbN24s96tXr5b7/fv3y33Hjh3l3tfXV+6V7du3l/vo6Gi5N3un8s2bNw23a9euldcytzw5IZQ4IZQ4IZQ4IZQ4IZQ4IZQ4IdSifJ/zT339+rXcm/24ut7e3obbzZs3y2tv375d7idOnCh3InmfE1qJOCGUOCGUOCGUOCGUOCGUOCHUonyf80+tW7fuj65fv379rK9tdg56/Pjxcl+yxL/HrcKfFIQSJ4QSJ4QSJ4QSJ4QSJ4Tyytg8+PbtW8Otp6envPbJkyfl/uDBg3I/fPhwuTMvvDIGrUScEEqcEEqcEEqcEEqcEEqcEMo5Z5iJiYly37lzZ7l3dHSU+4EDB8p9165dDbezZ8+W17a1zXhcR3POOaGViBNCiRNCiRNCiRNCiRNCiRNCOedsMSMjI+V++vTpcm/24wsrly9fLveTJ0+We2dn56w/e4FzzgmtRJwQSpwQSpwQSpwQSpwQSpwQyjnnAvP69ety7+/vL/fR0dFZf/aZM2fKfXBwsNw3b948689ucc45oZWIE0KJE0KJE0KJE0KJE0KJE0I551xkpqamyv3+/fsNt1OnTpXXNvm79M+hQ4fK/dGjR+W+gDnnhFYiTgglTgglTgglTgglTgjlKIV/beXKleX+8+fPcl++fHm5P3z4sOG2f//+8toW5ygFWok4IZQ4IZQ4IZQ4IZQ4IZQ4IdSy+b4B5tarV6/KfXh4uNzHxsYabs3OMZvp6uoq97179/7Rr7/QeHJCKHFCKHFCKHFCKHFCKHFCKHFCKOecYcbHx8v9+vXr5X7v3r1y//Tp02/f07+1bFn916mzs7PclyzxrPhvfjcglDghlDghlDghlDghlDghlDghlHPOv6DZWeKdO3cabkNDQ+W179+/n80tzYndu3eX++DgYLkfPXp0Lm9nwfPkhFDihFDihFDihFDihFDihFCOUmbw+fPncn/79m25nzt3rtzfvXv32/c0V7q7u8v9woULDbdjx46V13rla2753YRQ4oRQ4oRQ4oRQ4oRQ4oRQ4oRQC/acc3JysuHW29tbXvvy5ctyn5iYmNU9zYU9e/aUe39/f7kfOXKk3FevXv3b98Tf4ckJocQJocQJocQJocQJocQJocQJoWLPOZ8/f17uV65cKfexsbGG28ePH2d1T3NlzZo1Dbe+vr7y2mbffrK9vX1W90QeT04IJU4IJU4IJU4IJU4IJU4IJU4IFXvOOTIy8kf7n+jq6ir3np6ecl+6dGm5DwwMNNw6OjrKa1k8PDkhlDghlDghlDghlDghlDghlDghVNv09HS1lyMwJ9pm+qInJ4QSJ4QSJ4QSJ4QSJ4QSJ4QSJ4QSJ4QSJ4QSJ4QSJ4QSJ4QSJ4QSJ4QSJ4QSJ4QSJ4QSJ4QSJ4QSJ4QSJ4QSJ4Rq9iMAZ/yWfcDf58kJocQJocQJocQJocQJocQJof4DO14Dhyk10VwAAAAASUVORK5CYII=\n",
+ "text/plain": [
+ "<Figure size 432x288 with 1 Axes>"
+ ]
+ },
+ "metadata": {
+ "needs_background": "light"
+ },
+ "output_type": "display_data"
+ },
+ {
+ "data": {
+ "text/plain": [
+ "SGDClassifier(alpha=0.0001, average=False, class_weight=None,\n",
+ " early_stopping=False, epsilon=0.1, eta0=0.0, fit_intercept=True,\n",
+ " l1_ratio=0.15, learning_rate='optimal', loss='hinge',\n",
+ " max_iter=1000, n_iter_no_change=5, n_jobs=None, penalty='l2',\n",
+ " power_t=0.5, random_state=42, shuffle=True, tol=0.001,\n",
+ " validation_fraction=0.1, verbose=0, warm_start=False)"
+ ]
+ },
+ "execution_count": 1,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "from sklearn.datasets import fetch_openml\n",
+ "mnist = fetch_openml('mnist_784', version=1)\n",
+ "mnist.keys()\n",
+ "\n",
+ "import matplotlib as mpl\n",
+ "import matplotlib.pyplot as plt\n",
+ "\n",
+ "X, y = mnist[\"data\"], mnist[\"target\"]\n",
+ "\n",
+ "some_digit = X[0]\n",
+ "some_digit_image = some_digit.reshape(28, 28)\n",
+ "plt.imshow(some_digit_image, cmap=\"binary\")\n",
+ "plt.axis(\"off\")\n",
+ "plt.show()\n",
+ "\n",
+ "import numpy as np\n",
+ "y = y.astype(np.uint8)\n",
+ "X_train, X_test, y_train, y_test = X[:60000], X[60000:], y[:60000], y[60000:]\n",
+ "y_train_5 = (y_train == 5) # True for all 5s, False for all other digits\n",
+ "y_test_5 = (y_test == 5)\n",
+ "\n",
+ "from sklearn.linear_model import SGDClassifier\n",
+ "\n",
+ "sgd_clf = SGDClassifier(random_state=42)\n",
+ "sgd_clf.fit(X_train, y_train_5)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Here we will run a 3-fold cross validation on the classifier that we have trained using the training set. "
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 22,
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "array([0.95035, 0.96035, 0.9604 ])"
+ ]
+ },
+ "execution_count": 22,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "from sklearn.model_selection import cross_val_score\n",
+ "cross_val_score(sgd_clf, X_train, y_train_5, cv=3, scoring=\"accuracy\")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "At first glance this looks good, with 90+% accuracy. This is hiding a very ineffective classifier. The problem is that the classifier has been trained and assessed on an unbalanced dataset. \n",
+ "\n",
+ "**Only about 10% of the dataset contains '5's.**\n",
+ "\n",
+ "Lets construct a confusion matrix to see what is really happening. We will start by using `cross_val_predict( ... ) ` to test the classifier. This will return a list of the predictions made by the classifier.\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 19,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "from sklearn.model_selection import cross_val_predict\n",
+ "y_train_pred = cross_val_predict(sgd_clf, X_train, y_train_5, cv=3)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Now we can calculate a confusion matrix using the `confusion_matrix( ... )` function."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 20,
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "array([[53892, 687],\n",
+ " [ 1891, 3530]], dtype=int64)"
+ ]
+ },
+ "execution_count": 20,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "from sklearn.metrics import confusion_matrix\n",
+ "confusion_matrix(y_train_5, y_train_pred)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "From the confuction matrix, it is evident that there are actually many mis-classifications.\n",
+ "\n",
+ "Using the results from the confusion matrix, we can calculate some more concise metrics.\n",
+ "* *Precision* - the accuracy of the positive predictions.\n",
+ "\n",
+ "$$\n",
+ "precision = TP / (TP + FP)\n",
+ "$$\n",
+ "\n",
+ "* *Recall* (Sometimes called *Sensitivity*) - The ratio of positive instances that are correctly detected by the classifier. \n",
+ "\n",
+ "$$\n",
+ "recall = TP /(TP + FN)\n",
+ "$$"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 5,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "0.8370879772350012\n",
+ "0.6511713705958311\n"
+ ]
+ }
+ ],
+ "source": [
+ "from sklearn.metrics import precision_score, recall_score\n",
+ "print(precision_score(y_train_5, y_train_pred)) # == 3530 / (3530 + 1891)\n",
+ "print(recall_score(y_train_5, y_train_pred)) # == 3530 / (3530 + 687)\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "When the classifier claims a value is a 5, it is only correct 83% of the time (Precision).\n",
+ "The classifier only detects 65% of the 5s in the dataset \n",
+ "\n",
+ "Precision and recall can be summarized by calculating the *F1* score:\n",
+ "\n",
+ "$$\n",
+ "F1 = TP / (TP + ((FN+FP)/2))\n",
+ "$$"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 6,
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "0.7325171197343846"
+ ]
+ },
+ "execution_count": 6,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "from sklearn.metrics import f1_score\n",
+ "f1_score(y_train_5, y_train_pred)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "The F1 score summarises the recall and precision. This produces a harmonic mean that is only high if *both* the recall and precision are high. "
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 9,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "y_scores = cross_val_predict(sgd_clf, X_train, y_train_5, cv=3,\n",
+ "method=\"decision_function\")\n",
+ "\n",
+ "from sklearn.metrics import precision_recall_curve\n",
+ "precisions, recalls, thresholds = precision_recall_curve(y_train_5, y_scores)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 12,
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "image/png": "iVBORw0KGgoAAAANSUhEUgAAAfYAAAEPCAYAAACwduZtAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAgAElEQVR4nO3dd3xUVdrA8d+TSiD0DqGDQKQTEJBmoapgF+xrwbLoqsjq6r4ra1/dtayIihUUFVYXl3UBUUFQpIUuTar03klIPe8fZyaZhEkygZm5M5Pny+d+bjtzz3Nzhzy57RwxxqCUUkqpyBDldABKKaWU8h9N7EoppVQE0cSulFJKRRBN7EoppVQE0cSulFJKRRBN7EoppVQEKTGxi8gHIrJfRH4pYr2IyD9FZJOIrBKRTv4PUymllFK+8OWM/SNgYDHrBwEtXMMI4K1zD0sppZRSZ6PExG6MmQccLqbIUGCisRYCVUSkrr8CVEoppZTvYvywjfrADo/5na5lewoXFJER2LN6SKAzVfxQu4pMAtESTXRUNFESRbREIyIIgoi4itjp2KhYYqNjbXl3ORGixX5WELuNKLu9aIl2eOdUJNmxA/bv974uPh7atMmfX74ccnO9l01Kgtq17fThw7B1a8H1iYnQsuW5x6uctXTp0oPGmJqBrMMfiV28LPPaTq0xZjwwHqBRciPzxMQnPD5gCpcttMHSrffHNsKljozsDLJyszDGkGtyMbjGHvNFrUvLTsubd5dz1+G5zHPsjsmXMqeyTpGRnUFGTgYnMk6QlZtFrsktdsjKySIjJ4Mc1z9/i42KpWpCVcrFlCMuOo7EuEQqxlWkXEw5qpSrQkJsAgkxCVQtV5XEuESqJlSlQmwFqpSrQmJcIvUq1qNVjVZ5f2CosmfJEti0CVJS4NgxWL/ee7lKlWDIkPz5zz6DnCK+0p07Q+vWdnrrVpg/v+D6WrWgf/9zj105S0R+C3Qd/kjsO4EGHvNJwO6SPlSzfE3uSbnHD9WrSJSRncGR00c4kXGC9Ox0jmccJysni+zcbHJMDjm5OWTnZnM6+zS7TuziUNohDqUf4nT2aTJyMkjLSuNU5ikyczLJyMng2OljnMw8mVdm/6kiTrF8lBiXSM3yNWlVoxVtarWhUnwlmlZtSrva7WhVoxUxUf74r6VC1ccfwxtvwKuvwkMP2QTvi+HDfSvXpIkdlDob/vjtMw0YKSKfAxcAx4wxZ1yGV6o04mPiqZNYhzqJdfy+7bSsNI6dPlYg6adnp3My8yTHM46TkZ3B8YzjnMg8wbHTxziWcYxTWac4nH6YtKw01h1Yx75T+ziZeZKtR7cyY9OMAtuPi46jVY1WdKvfjSZVm9CyeksGNh9IQmyC3/dFOePECTuuWNHZOJTypsTELiKfAX2BGiKyE3gKiAUwxrwNTAcGA5uANOB3gQpWKX8oH1ue8rHlz/rzxhiOnD7CjmM7WHtgLduObmP/qf1sPrKZJbuXsPfkXlbtW8WqfavyPhMXHceQlkPoULsDA5sPpF3tdsRGx/pjd5QDNLGrUCZOdduakpJiUlNTHalbqUA6kn6ElftWsnrfajYf2cy3W75l7YG1BcokxiUy+drJDG4x2KEoQ9OJEzBtGvTuDQ0alFzeKQMGwKxZMH06DBrkdDQqnIjIUmOMjzdvzo7eCFTKz6omVKVv4770bdw3b9mmw5uYs3UOP+/8mY9WfMTJzJNc9ull9GzYk0lXT6Jh5YbOBRxgR45AZmb+E9+eMjIgLs6Op06FZ56Bdevsuo8+gttuC2qoPtMzdhXKtElZpYKgebXm3N35bj4c+iFpT6TxSLdHiJIoftr+E63fbM0/F/2TU5mnnA7Tb06fhpgYEIFq1aBOHZuo582zy665BqKioFw5GDUK/vxnuPHG/KQOcPHFcOoUXHklvP++Y7vilSZ2FcpC+lL8sWPHOHjwIJmZmUGKSoWKuLg4atSoQeXKlZ0OJWC2HtnKXf+9i9lbZwNQLaEa/+j/D27vcLuzgZ2jvXuhrpcmqh54wL7G9fXXZ6775BO4+eb8+c2boWlT+OEHuOgiu+yxx+D55+0fBE5r0gS2bbOvvDVr5nQ0KpwE41J8yCb206dPs337dpKSkkhISNB3hssQYwzp6ens3LmThg0bUq5cOadDCqjPVn/GE7OfYNvRbQCM6DSCty5/iygJgQxWSunpsHs3dOoEx4/nL1+9Gs47zybDwo2sLF1qy6el2fvW/ftDeY9nGydMgDvvtO9/X3IJ/Pvf9v1wJ2VlwcmTNo5obe9IlUKZTuw7duwgMTGRqlWrBjEqFUoOHz7MqVOnaBDKT1H5iTGG1xa+xh+/+yPZudk82v1RXu7/stNhlcovv0DbtjB0qG2UpUcPaNWq6PLbt9vW1nw5A585E4YNs43BtGxpH1pr2tR/sSsVLMFI7CF7SnD69GkSExOdDkM5qGLFipw+fdrpMIJCRHi4+8NMvHIiAH9f8Hfu+/q+M1odDEXPPGPvm7dta+f/8x/o16/4pA7QsKHvl9UHDoSFC6FRI9iwwZ7V79tXdPOsSpVlIZvYs7OziYnRh/bLspiYGLKzs50OI6iGtx3Ou1e8S7RE8/bSt3l67tNOh1QsY2Ds2ILL/vjHwLyq1qoVrFxpL9s3amRbfhs0CH791f91FefoUbjgArj66uDWq5SvQjaxA3pfvYwrq8f/rk53MfnayQCMmTuG8UvHOxxRQVOn2jN0EbjnHpvc3TZvhr/9LXB1V64M334LM2bYhD5rlm3SNZgXNo4ehcWLQZvhUKEqpBO7UmXVNcnX8OxFzwJwz9f38P4y59/3SkuzydzzTPXdd+39b2PsEIz73tWq2XffX3oJKlSwSf7ttwNfr5v7VTe9U6hClSZ2pULUE72e4MGuDwJw13/vYv3BIroQC5L//e/MZU89ZS+NO6FxY3jrLTt9//3BO4M+edKO9R12Fao0sQfRRx99lNdXuIhQsWJF2rdvz9ixY4N6L3nMmDGlvszdt29f+vbtG5iAlFciwmsDX+OK864A4IrPruBExomgx1Gnjj1T7949f1lOjj1DHzMm6OEUcMstcN11dvrpID2OoI3TqFCnid0B//rXv1iwYAFffvklXbt25YEHHuDpYP1WAu666y4WLFhQqs+MGzeOcePGBSgiVRQR4YOhH9C8WnM2Hd7EH7/9Y9DqXrXKJvR9++x8dnb+JfdQaCTG7fXX7fi772zTtYGmiV2FuhD671l2dOjQgW7dutG/f3/effdd+vbty2uvvea1rDHG7y3vJSUl0a1bt1J9Jjk5meTkZL/GoXxTo3wNvrz+SwTh3WXvFug1LlB69YL27fPnK1Wyl75DUd260K2bbaHu4MHA16eJXYU6TewhoEuXLpw4cYL9+/fTuHFjbr75Zj744ANatWpFXFwc/3Pd3ExLS+Oxxx6jSZMmxMXF0aRJE5577jlyC73Me+DAAe6//34aNGhAfHw8DRo04JZbbiEjIwPwfin+9ddfp3Xr1iQkJFC1alVSUlKYOnVq3npvl+I3bNjAVVddRZUqVUhISKBbt27MnDmzQBl3XRs3buSyyy4jMTGRRo0a8fTTT58Rtypau9rtuKfzPeSYHG7/6vaAtyu/Y0fB+WPHAlrdOfv5Z/sMQL16ga+rSRO49Vb7x49SoSgsE7v7VRtvw3iPN4PGjy++rKfOnYsuN2JEfrmlS/2/P1u3biU6OjqvQZ45c+bwyiuv8NRTTzFz5kzatWtHdnY2AwYM4L333uMPf/gDM2bM4K677uKZZ55h9OjReds6cuQIPXr0YPLkyTzyyCNMnz6dl156iaysrCLP/CdNmsSoUaMYPnw406dPZ9KkSVx77bUcPny4yJh3795Nz549WblyJWPHjmXKlClUqVKFyy67jBkzZpxR/qqrruLiiy/mq6++4sorr+Spp55iwoQJ5/iTK1uev+R5mlVtxvK9y3kr9a2A1OF+beyvf7XjgweD+yrZ2Qrmm5F9+thmbu++O3h1KlUqxhhHhs6dO5virF27tsh1+Xf6zhzeeSe/3DvvFF/WU6dORZe7++78cqmpxYZdrA8//NAAZv369SYrK8scPnzYvP322yYqKsoMHTrUGGNMo0aNTEJCgtmzZ0+Bz06cONEAZu7cuQWWP/vssyY2Ntbs27fPGGPM//3f/5moqCizbNmyIuN46qmnDB4/gN///vemY8eOxcbep08f06dPn7z5UaNGmejoaLNx48a8ZdnZ2ea8884rsC13XR988EGB7bVp08b069ev2DqNKf57UBb9Z/1/DGMwNV+qadIy0/y67ZwcY1q1MubZZ41JT/frpoNi3Tpjxo0z5vvvnY5EqaIBqSbA+TUsz9iLS9eeZ9cjRhRf1tPSpUWX87wK0LnzucffqlUrYmNjqVatGvfffz833XQTH3zwQd76bt26UadOnQKfmTlzJo0aNaJHjx5kZ2fnDf379ycrK4uFCxcCMGvWLLp06ULHjh19jqdLly6sWLGCBx54gO+++460tLQSPzNv3jy6detG8+bN85ZFR0czfPhwVqxYwXHPHkCAyy67rMB8mzZt2L59u88xKuvy8y6nVY1WHEg7wJfrvvTrtqOjYf1624VqTo5fNx0UP/xgX3ubNCmw9WzbZtvFL/QVVypkhGViD3dTp05lyZIlrF+/nlOnTjFx4kSqVauWt76ulz4v9+/fz2+//UZsbGyBoWvXrgAcOnQob5yUlFSqeG699VbeeustFi1axIABA6hWrRpXX30127ZtK/Izhw8f9hpnnTp1MMZw5MiRAss99w8gPj6+zLQD709REsX9KfcD8OGKD/22XfeT72AbmalQwW+bDhp396lr1wa2njFjbLv4X3wR2HqUOlvaGLsD2rRpU+BMtzBv75hXr16dJk2aMGXKFK+faex6ZLlGjRrs2rWrVPGICPfccw/33HMPR44cYdasWYwaNYobbriBRYsWef1MtWrV2Lt37xnL9+7di4ickciV/9zc7mYe++4xZm+dzbebv6Vfs37ntL3Dh+276m6uiz9hx/0U/6JF9mw6UF276lPxKtTpGXuYGDhwYF5XtikpKWcMNWrUAKB///4sXryYlStXnlU9VatW5YYbbuD666/nl19+KbJcnz59WLhwYYGz+pycHCZPnkzHjh2pqL/1AqZqQlX+cMEfAHj8+8fPuQc4z7s2I0ZAzZrntDnH1Kple4wzxvYAFyia2FWo08QeJm666SZ69OjBJZdcwiuvvML333/PjBkzGDt2LP3798+7L/7www/TtGlTLr30Ul5//XVmz57NlClTuOmmmzhxwnurZSNGjGDUqFF88cUXzJs3j/fee4+PP/6Y/v37FxnPww8/TJUqVejXrx+ffvopX3/9NVdccQW//vorzz33XEB+Birf//X5PyrEVmDZnmX8tP2nc9rWK6/Y8XXXwTvv+CE4Bw0YYMfPPlvy0/zp6b5v9+uvbf/xoIldhT5N7GEiNjaWb775hrvvvpvx48czePBgbrrpJiZMmECPHj2Ii4sDoEqVKsyfP5+rrrqKF198kYEDBzJq1ChiYmLyyhR24YUXsnTpUu6//3769evHc889x80331zs62j16tXjp59+4vzzz+e+++7Lez3uf//7HwMHDgzIz0DlKx9bnpFdRwLw5pI3z3o7EyfahGUMFHGXJ6w8+ijEx8O0abYHNrcvv4QHHshvme7oUWjZEh57rOQEv3w5XHGF7SoWtBMYFQYC/dh9UcO5vO6myg79HhRty+EthjGY+GfizdH0o6X+/M8/57/7cfp0AAJ0yOjRxjz2mH19z829n88+a+cnTzZGxC4bPdou27XLmMcfN+bgwYLbmzkz//Pr1hnTqJGd3rw5KLujIgz6uptSqihNqjahd6PeZORkMHnN5FJ/vkeP/On4eD8G5rAXX7SDt/bs3U+yX389vPqqnd640Y4fecR+zvWiSZ4Yj0eMExP1UrwKfZrYlQpjd3a8E4B3l71bqs95dnE6fLg/I3KeZ0LfsQMWLAB3a8grVsD8+fDEE3DkiC371VcweTIcOGDLbNlScHvud/ovvRSSkmD2bJg7F6pWDfiuKHVWNLErFcauS76OSvGVSN2dyroD63z+XJcu+dOBbtDFKcuWQatWcOONtue3li3t8i+/hBdegK1b8x8cfPjh/H7ln3224HbciT062o7bt4fevQueySsVSjSxKxXGEmITuKrVVQBMWePb02/z5uVPT5sW3HbWg6ldO0hLsy3F3XMPJCdDlSqwe7ddHxcHDz4I550He/bArFnet5OYaLdVvrztIjZS/xBSkUMTu1Jh7trkawH47JfPSv1O+xVXBCKi0OB5Rv3++/CnP8H+/fm9ssXF2T9qHnrIvr/vbkgxJ8fea3ffS+/VC1auhDvusGVvvhmeeSa4+6JUaWhiVyrMDWg2gGoJ1dhwaAMbD28ssXyvXvZec3Z2EIILIV27wm+/5b/y5n7787777Pv77kvxTz1l/wi48caCbeZfckn+9EsvBSdmpc6GJnalwlxsdCwXNb4IgM9/+bzYsu+8Yy+/t2+ff884kr3+esH5qKj8tuQTEgque/hh2wnOwoVQrZptlOaPf8xfn5CQ39/7yZOBi1mpc6WJXakIcG/KvYDtGKaoy/HZ2XDvvXDllQWfio9kDz5oL7+7ffUVvPeenfbszmDXLttAT61acMEF9gG7mBj7cN2TT9rpm2+29+oBXC04KxWSNLErFQEuanwRtSvUZtvRbSzbs8xrmZ9/zp/2R/fD4cKz7XvPxhdr186fvu02GDnSnqWDfT3ujjvs9OTJ9pK8MTbJv/IKfP99wMNW6qxpYlcqAkRHRTOoxSAAZm32/nh3nz52/MgjkfskfFGWLoX//Md2SQswaBDcckv++qvsiwXcequ9VeG5bPNmO46OtsPDD9un5JUKVZrYg+ijjz5CRPKGuLg4mjVrxhNPPOF43+SNGzfm9ttvz5t3x1pcn+wqtAxoZntAmb1t9hnrjh3Ln/Y4zGVGp04wZEj+Gbv7ATq3oUPzp5cuteOLLy7Y9WtZeCZBRQZtYsEB//rXv0hKSuLEiRNMnTqVF154gRMnTvDGG284HZoKYxc3uRhBmPfbPNKy0igfWz5vnedT3G3bOhBciHA3nXvqVMHlSUn50+7OY+Li4LXX4Jtv7OV4TewqXPh0xi4iA0Vkg4hsEpHHvaxvKCJzRGS5iKwSkcH+DzVydOjQgW7dutGvXz/GjRvHpZdeyvvvv09ubq7ToakwVqtCLTrU6UBmTiYLdizIW24MPP+8nX7sMYeCCxHnnWfHCxfaRmk8jRtnxw8/nL/sd7/Lb45WE7sKFyUmdhGJBt4EBgHJwHARSS5U7M/AFGNMR2AYMM7fgUayTp06kZ6ezsGDB/OWbd26lZtuuomaNWsSHx9Phw4dmDp16hmfXblyJVdddRXVq1cnISGBli1b8sILL+StnzVrFoMHD6Zu3bqUL1+eNm3a8I9//IMczxd0VcRwv/Y2Z9ucvGW5ufb+8i23wHPPORVZaKhdGxo2tE+1V69ecN1990FGBvTvX3B54SZllQp1vlyK7wpsMsZsARCRz4GhwFqPMgZw342qDOz2Z5Ce5K+h8dSPeap0LXwVZ9u2bVSuXJnqrt80O3bs4IILLqBWrVq8+uqr1KxZk8mTJ3PNNdfw1VdfMWTIEAAWL15M3759ad68Oa+++ipJSUls3LiRVatW5W17y5YtXHLJJTzwwAOUK1eO1NRUxowZw4EDB3jxxRf9tg8qNFzU5CJeWfhKgcQeHW1f4erVS5MTwIYN9h675xPybt6WuRvyycgIbFxK+Ysvib0+sMNjfidwQaEyY4BZIvIAUAG41NuGRGQEMAKgYcOGpY01YuTk5JCdnZ13j/3LL7/ktddeI9r1W3fMmDEYY5g7d25esh8wYAA7duzgL3/5S15if/TRR6levToLFy6kfHl7P/Xiiy8uUNe9996bN22MoVevXmRmZvL3v/+d559/nihvfVuqsNWrYS+iJIrFuxZzMvMkFWITSU8v+GpXWVeunB18tXevHTdrFph4lPI3XxK7t1Pkwqerw4GPjDH/EJHuwMci0sYYU+CmsTFmPDAeICUl5axOef15puyUVq1aFZi///77GTlyZN78zJkzGTx4MJUrVybbo93PAQMGMHr0aI4fP05MTAzz589n9OjReUndmz179jBmzBhmzpzJ7t27C2xv//791KlTx497ppxWuVxlOtftzJLdS5i/fT4HFw3g5pvtulI2I69c3Fc59BEYFS58OV3bCTTwmE/izEvtdwJTAIwxC4BygLbNVISpU6eyZMkSpk+fzqWXXsq4ceOYOHFi3vr9+/czceJEYmNjCwyjR48G4NChQxw5coTc3FySPB/nLSQ3N5chQ4bw9ddf8+c//5nZs2ezZMkSnnzySQDHX7FTgeF5n92d1NXZc9/Zcj8tr1So8+WMfQnQQkSaALuwD8fdWKjMduAS4CMRaY1N7Af8GWgkadOmDc2bNwfspfN27doxevRorrnmGipUqED16tXp1asXjxXxCHO9evXIyckhKiqKXbt2FVnP5s2bSU1N5eOPP+Zmj9/w//3vf/27Qyqk9G3cl5d+fonvt2rzaP4wY4Yd638bFS5KPGM3xmQDI4FvgHXYp9/XiMjTIjLEVWwUcLeIrAQ+A243pe0/soyKj4/n5ZdfZv/+/YxzvW8zcOBAVq1axfnnn09KSsoZQ3x8POXLl6dnz5588sknpKene912WloaALGxsXnLsrKymKQdSke0Xo16ERsVS+ruVEg4DMCECQ4HFcbat7fj1q2djUMpX/nUQI0xZjowvdCyv3hMrwUu9G9oZceQIUPo0qULf//73xk5ciRPP/00Xbt2pXfv3owcOZLGjRtz5MgRfvnlF7Zs2cIHH3wAwN///nf69OlD9+7dGTVqFElJSWzZsoUVK1bwxhtv0Lp1axo1asSTTz5JdHQ0sbGxvPrqqw7vrQq0xLhEutbvyvwd8yFpAWy8jGHDnI4qfP3737aBn0cfdToSpXyjj0SHiGeffZb9+/fz9ttv07BhQ1JTU2nfvj1PPPEE/fr147777mPu3LkFnnrv0qUL8+fPp0GDBjzwwAMMHjyYl19+Oe++e1xcHF999RV16tTh1ltv5fe//z29e/fm8cfPaGNIRZjejXrbiWbfAt5f41K+adgQxo6Fxo2djkQp34hTV8xTUlJMajF9R65bt47Weu2rzNPvwdmZsXEGgz8dDDu6w/s/6xPxSoUIEVlqjEkJZB16xq5UBOpSv4udqLuMF/6WXXxhpVRE0cSuVASS9BpwrAHEZHDZzZudDkcpFUSa2JWKQHPmAHs6AZC672dng1FKBZUmdqUi0HXXAVvtg5bTN00vvrBSKqKEdGLXV+HLNj3+52jjIAB+2PYDuUbbQ1WqrAjZxB4bG1tkwyuqbEhPTy/QuI7yzcmTronDzakZX5+DaQdZs3+NozEppYInZBN7rVq12LVrF2lpaXrmVsYYY0hLS2PXrl3UqlXL6XDCzqxZ7ilhQMsz+2dXSkU2n1qec0KlSrZ79927d5OVleVwNCrYYmNjqV27dt73QPmud287DB8OcY0v4pNVnzBn2xwevOBBp0NTSgVByCZ2sMldf7ErVTo//QQPPwx9+8Jh0xeAudvmkmtyiZKQvUinlPIT/V+uVIR56im46ipYtw6aVGlCw8oNOXL6CKv2rXI6NKVUEGhiVyqCZGTA2rUgAm3bgojk98++Ve+zK1UWaGJXKoKsXQvZ2dCiBSQm2mV5iV0foFOqTNDErlQEWbrUjjt1yl92UROb2Of9No+c3BwHolJKBZMmdqUiyLJlduyZ2BtWbkjTqk05lnGMpXuWOhOYUipoNLErFUHeesuOPRM7QN9GfQH4eYe2G69UpNPErlQEat++4HynujbT65PxSkW+kH6PXSlVOkePwunTUKNGweVta7cF4Jf9vzgQlVIqmPSMXakIUrky1K595vK2tWxiX7lvJaezTwc5KqVUMGliVypC7N8PRbW+XDWhKufXPJ/MnExSd6cGNzClVFBpYlcqQtSuDXFx8N133tf3aNADgKW79cl4pSKZJnalIkCOx+vpzZt7L9O+tn2ibuW+lUGISCnlFE3sSkWAFSvypxs39l6mXe12gCZ2pSKdJnalIsA339jxtdcWXaZd7XZESRSr9q3SB+iUimCa2JWKANOn27FI0WUql6tMcs1ksnOzWbxrcXACU0oFnSZ2pSLAokV2fNllxZfr3bA3AAt2LAhwREopp2hiVyoCZGfbceEW5wrT++xKRT5teU6pCHD8OKxcCe3aFV+ua/2uACzbsywIUSmlnKBn7EpFgIoVoWdPiCrhf3RyzWSiJZqNhzeSlpUWnOCUUkGliV2pMiQ+Jp7za51Prsllxd4VJX9AKRV2NLErFebuuAMGDLCX4n3RuW5nQFugUypSaWJXKszNmQOzZkFsrG/l3R3CrD2wNoBRKaWcooldqTB27Bhs22anzzvPt8+0rtkagDUH1gQmKKWUo3xK7CIyUEQ2iMgmEXm8iDLXi8haEVkjIp/6N0yllDfuFucAYnx8x6Vz3c4IwuJdi8nMyQxMYEopx5SY2EUkGngTGAQkA8NFJLlQmRbAn4ALjTHnAw8FIFalVCFrzuKku2aFmjSv1pyMnAzWHVjn/6CUUo7y5Yy9K7DJGLPFGJMJfA4MLVTmbuBNY8wRAGPMfv+GqZTy5oMP7Lh799J9rkOdDgAs37vczxEppZzmS2KvD+zwmN/pWubpPOA8EZkvIgtFZKC3DYnICBFJFZHUAwcOnF3ESqk8O3fa8YgRpftcxzodAfSVN6UikC+J3Vu3EqbQfAzQAugLDAfeE5EqZ3zImPHGmBRjTErNmjVLG6tSqpArrrDjgV7/lC6anrErFbl8edxmJ9DAYz4J2O2lzEJjTBawVUQ2YBP9Er9EqZTyatq0s/tcx7r5Z+zGGKS4buGUUmHFlzP2JUALEWkiInHAMKDwr5OvgIsARKQG9tL8Fn8GqpTynzqJdahdoTbHM46z9ehWp8NRSvlRiYndGJMNjAS+AdYBU4wxa0TkaREZ4iNdFmMAACAASURBVCr2DXBIRNYCc4DRxphDgQpaKQX/+x/885/w669n93n3WfvyPXo5XqlI4tObr8aY6cD0Qsv+4jFtgEdcg1IqCH73OzhwAN57z/fGaTydX/N8Zm6ayYZDG/wfnFLKMdrynFJhyv1iydkkdYCW1VsC8Mv+X/wUkVIqFGhiVyoMpXn0uNqly9ltI+9SvD4Zr1RE0cSuVBha52owLjkZypU7u220qdWGaIlmw8ENnMo85b/glFKO0sSuVBhyNyV7/vlnv41yMeVIrpmMwbBq3yr/BKaUcpwmdqXC0Pr1dty69bltRy/HKxV5NLErFYYqVoSkJGjR4ty206lOJ0BfeVMqkmhiVyoM/elPsGMH3HTTuW0npV4KAD/v/NkPUSmlQoEmdqXC2Lm2BJtSL4UKsRVYe2At+07u809QSilHaWJXKsycPAnHj/tnW/Ex8bSv0x6AlftW+mejSilHaWJXKsxMmQKVK8MDD/hne53rdgZg6e6l/tmgUspRmtiVCjOrXG+m1avnn+2577Mv3aOJXalIoIldqTCzerUdt2vnn+11qWebrpv721xyTa5/NqqUcowmdqXCiDGw0nUr3F+JvVWNViRVSuJg2kHWHVjnn40qpRyjiV2pMLJ3Lxw6BFWq2PfY/UFEuKD+BQAs3rXYPxtVSjlGE7tSYcR9f71du3N/1c1Tr4a9AJizbY7/NqqUcoQmdqXCiGdi96fuDboDsGT3Ev9uWCkVdDFOB6CU8t2tt9oe3fz1RLxbxzodSYhJYP3B9RxJP0LVhKr+rUApFTR6xq5UGKldGy67DDp29O92Y6Nj8xqqWbZnmX83rpQKKk3sSikAUura99n1ATqlwpsmdqXCxK+/wi23wIcfBmb7PRr0AGDe9nmBqUApFRSa2JUKE0uWwCefwH//G5jt92pkn4xfsGOBNlSjVBjTxK5UmFju6jLd30/EuyVVSqJh5YYcyzjGmv1rAlOJUirgNLErFSaWuN5E69IlcHW4L8f/vEP7Z1cqXGliVyoM5OTAMtfD6ikpgavnwgYXAjB/x/zAVaKUCihN7EqFgc2bbT/s9evbV94CRRO7UuFPE7tSYSBQLc4V1rZ2WxLjEtlyZAt7T+4NbGVKqYDQxK5UGKhSxTZMc/HFga0nJiqGbkndAJi/Xc/alQpHmtiVCgOXXgpffw2PPhr4utwdwny75dvAV6aU8jtN7EqpAi4/73IApm+c7nAkSqmzoYldqRB38iTMnw8nTgSnvg51OlA+tjw7ju/gSPqR4FSqlPIbTexKhbjFi6FnT+jXLzj1RUkUyTWTAVi+d3lwKlVK+Y0mdqVC3NKlduzvHt2Kc0H9CwBI3Z0avEqVUn6hiV2pEOduca5r1+DV2bGO/Stixd4VwatUKeUXmtiVCnHuM/ZAtjhXWIc6HQC9FK9UONLErlQIO3YMtmyBcuWgdevg1dumVhsSYhJYf3A9B04dCF7FSqlz5lNiF5GBIrJBRDaJyOPFlLtWRIyIBPHcQqnItcbVyVqrVhATE7x642PiSaln/xsv3rU4eBUrpc5ZiYldRKKBN4FBQDIwXESSvZSrCDwILPJ3kEqVVStX2nGnTsGvu2t9e1N/0S79L61UOPHlHKArsMkYswVARD4HhgJrC5V7BngJCELbWEqVDffeC5dfbnt3CzZ3Yl+6Z2nwK1dKnTVfLsXXB3Z4zO90LcsjIh2BBsaYr4vbkIiMEJFUEUk9cEDv2ylVEhFo0AAaNw5+3e7EvnDnQowxwQ9AKXVWfEns4mVZ3v9yEYkCXgVGlbQhY8x4Y0yKMSalZs2avkeplAq6RpUbUSexDofTD7P5yGanw1FK+ciXxL4TaOAxnwTs9pivCLQBfhCRbUA3YJo+QKfUuZk7F9q0gWefdaZ+EaFz3c4ALNqp99mVChe+JPYlQAsRaSIiccAwYJp7pTHmmDGmhjGmsTGmMbAQGGKM0SarlDoHixbZp+J37y65bKD0adQHgO+2fudcEEqpUikxsRtjsoGRwDfAOmCKMWaNiDwtIkMCHaBSZZW7xbkuXZyLoV8z20D93G1znQtCKVUqPr0Za4yZDkwvtOwvRZTte+5hKaVCIbG3qdWGcjHl2Hp0K0fSj1A1oapzwSilfKItzykVgvbuhd9+g8TE4LY4V1hMVAzta7cHtKEapcKFJnalQtCCBXbctStERzsbS8+GPQH4cfuPzgailPKJJnalQtAPP9hxz56OhgFA38Z9AZi9dbazgSilfBLE1qeVUr667jrb8cvQoU5HAr0b9SZaolm8azHHM45TKb6S0yEppYqhZ+xKhaCePeFvf3OmjfjCKsVXomv9ruSYHH78TS/HKxXqNLErpUrkvhw/Z9scZwNRSpVIE7tSIebjj+HNN51tmKawfk3t++zfbvnW4UiUUiXRe+xKhZjXXoNly6BlS6hXz+lorB4NepAQk8CqfavYd3IftRNrOx2SUqoIesauVAg5dgxWrICYGOje3elo8sXHxHNhwwsBfe1NqVCniV2pEPLTT5Cba1ubq1DB6WgKuqjxRQBMWj3J4UiUUsXRxK5UCLn2Wjvu2tXZOLy5rf1tAMzaPIuM7AyHo1FKFUUTu1Ih5PRpO05OdjYOb+pXqk+72u1Iy0pj3m/znA5HKVUETexKhYhdu/Knb7vNuTiKM7j5YABmbJrhcCRKqaJoYlcqRBw/DtdcA7fcAvHxTkfj3aAWgwBN7EqFMn3dTakQ0bo1fPGF01EUr3tSdyrFV2L9wfVsPbKVJlWbOB2SUqoQPWNXSvksNjqW/s36AzB943SHo1FKeaOJXakQ8OOPULMmzAmDFlsvb3E5AJ+s/sThSJRS3mhiVyoEjB0LBw/Cyy87HUnJrkm+hrjoOBbtXMShtENOh6OUKkQTu1IOy821DdMAPP20s7H4IjEukR4NemAw/LT9J6fDUUoVooldKYctWWI7fGnQADp3djoa33Sr3w3Q5mWVCkWa2JVy2Pvv2/GVV4KIs7H4akDzAQD8e92/McY4HI1SypMmdqUcZAy8+66dHjLE2VhKo2fDntQoX4OtR7eydM9Sp8NRSnnQxK6Ug5Z65MS+fR0Lo9RiomK4Pvl6AGZumulwNEopT9pAjVIOOnYM2ra1ST0mzP43dqzbEYC1B9Y6HIlSylOY/SpRKrJccgmsWgWZmU5HUnrdk2yH8d9u+Zbs3GxiovTXiVKhQC/FKxUC4uKcjqD0kmsm07J6Sw6mHeSHbT84HY5SykUTu1IOmTABNm1yOoqzJyJcf769zz5lzRSHo1FKuWliV8oBmzbB7bfb++tHjzodzdlzJ/ap66fqa29KhQhN7Eo54K237HjYMKhSxdlYzsX5Nc+nbmJdDqYdZNGuRU6Ho5RCE7tSQXfsWP676yNHOhvLuRIRbmx7IwATV050OBqlFGhiVyro3n0XTpyAiy4KnyZkizOszTAAvlr/Fbkm1+FolFKa2JUKosxMeO01O/3oo87G4i+d63amcZXG7Dm5h593/Ox0OEqVeZrYlQqiyZNh1y5IToZBg5yOxj9EhCHn2fZwp2+c7nA0SilN7EoFUdeucP31MHp0+HT44ourW18NwPil4zmVecrhaJQq23xK7CIyUEQ2iMgmEXncy/pHRGStiKwSke9FpJH/Q1Uq/LVsac/ab7/d6Uj8q3ej3nSt35VD6Yd4K/Utp8NRqkwrMbGLSDTwJjAISAaGi0hyoWLLgRRjTDvgC+AlfweqVDjLyoLcCH6uTER47MLHAH06Ximn+XLG3hXYZIzZYozJBD4HhnoWMMbMMcakuWYXAkn+DVOp8Pa3v0H37rB8udORBM5lLS6jSrkqrN6/mhV7VzgdjlJlli+JvT6ww2N+p2tZUe4EZnhbISIjRCRVRFIPHDjge5RKhbHt2+GFF2Dx4vBuZa4k8THx3NLuFgDeXfquw9EoVXb5kti9PeLjte1IEbkZSAFe9rbeGDPeGJNijEmpWbOm71EqFaaMgbvugrQ0uO46++56JLur010ATFo9ifSsdIejUaps8iWx7wQaeMwnAbsLFxKRS4EngSHGmAz/hKdUeHv/ffj2W6hWDd54w+loAq9d7Xak1EvhWMYxpq6f6nQ4SpVJviT2JUALEWkiInHAMGCaZwER6Qi8g03q+/0fplLhZ8cOGDXKTo8dC7VrOxtPsNzZ8U4APlzxocORKFU2lZjYjTHZwEjgG2AdMMUYs0ZEnhaRIa5iLwOJwL9EZIWITCtic0qVCTk5MHw4HD8OQ4bYzl7KiuuSr6NcTDm+2/Idy/Ysczocpcocn95jN8ZMN8acZ4xpZox5zrXsL8aYaa7pS40xtY0xHVzDkOK3qFRki46GO+6Apk3hvfciqzGaklQvX507OtwBwKsLX3U4GqXKHm15TqkAueMOWLsWyuJzoo90fwSAf6/7NyczTzocjVJliyZ2pfzo009h4cL8+fh452JxUrNqzejRoAdpWWlMWDHB6XCUKlM0sSvlJ2++CTfdZDt32bPH6Wic92DXBwF4e+nbGOP1DVmlVABoYlfqHBkDzz0HI0fa+SeegLp1nY0pFFzZ6krqJtbll/2/MGGlnrUrFSya2JU6Bzk58PDD8Oc/2wfk3n3X9tymbEt0j/e0fUb96fs/kZWT5XBESpUNmtiVOkt790K/fvD66xAbC59/bluZU/ke6PoAyTWT2XtyL5+s+sTpcJQqEzSxK3WWdu6EH3+0Dc98953tZ10VJCKM6m5b6RmXOs7haJQqGzSxK1UK27fbe+oAKSkwcaLtsa13b2fjCmXD2wynarmqpO5O1QZrlAoCTexK+WDVKrjlFmjWDKZ6NIE+fLg+KFeShNgEbmt/GwBjF491OBqlIp8mdqWKYAzMng0DB0L79vDJJ3ZZJPepHij3ptyLIExYOYGDaQedDkepiKaJXSkvZs2CLl3gkkvgm2+gfHn4wx9g82Z45hmnows/LWu05JKml5Brcnlv2XtOh6NURNPErso8Y2DdOti6NX/Zr7/C0qW2OdhnnrH31l97DRo1ci7OcPdwt4cB+Ovcv7LnhLbgo1SgaGJXZdK+fTBpEtx+OzRoAMnJ8M9/5q+/4QZ45x347Tf7jnr16o6FGjEGNR9E38Z9OZ19mhd/etHpcJSKWJrYVZny+uv2fnmdOnDzzTBhAuzaZV9Zq1Ahv1zNmjBiBCQkOBdrpBERXu73MgBvpb7F+oPrHY5IqcgU43QASp2rnBzYscO+V75tG2zaZO+Fb95sp2fNgg4dbNlNm+wT7gkJ9hW1fv3s0LZt2epa1Skp9VK4pd0tfLzqY2788kZ+vvNnysWUczospSKKJnbluKwsOHECjh+H+vVtK24A8+bBhg12eeHh/PNhzBhbbtcuaNKk6O2vW5ef2O+6C669Frp1K7s9rzntn4P+yY/bf2T53uW88OML/PWivzodklIRRZzqdSklJcWkpqY6Unc4MyZ/iIrKP8vMyIDMTMjNPXOIicm/R2yMPas1xnvZ+vWhalVbdvdue9ablQXZ2QXHxsB11+XH9emnsH+/jSEjww6nT9txnz42mQKkpsK99xZM0unp+dvZuBGaN7fTw4bB5Mnefw4XXgg//WSnMzPt++VJSdCwoZ12D82bQ7169melQsecrXO4eOLFxEbF8vOdP5NSL8XpkJQKChFZaowJ6BfesTP21avtL1733xXucWysfSLZbcAAe8bl+feHe/r22+HZZ+10aioMGXLm9tzjOXPsA1IADz4In33mfZspKfb1JrAJo1Yt79sDGDfO3qcF+6DVqFHe64+NtQnMLSXF7r+3+u+7L/8hrgULbAIrXK/bqlX2EjLAPffY+8XedOtmtwU2KTdt6r0cwIcf2p8rwBdf2Fe8vImNLZjYn38e1qwpervuxJ6dbZ829xQVBZUq2SEzM3/5xRdDxYr56zyHevXyy8XF2UvxKnxc1OQiHuj6AG8sfoOR00ey4M4FiN4LUcovHEvsmZmwZcuZy92XYd327Cn6l/bRo/nTWVnF94GdnZ0/feIEHCyijQzPBAxw7FjR28zKKjh96pT3cjk5BeczMwsmME+5uQXni7qgEhVVsGy5cpCYaJcXHtxn4O7PNWrkvVxUFFSunF82Kcn+YREbmz/ExNhxXFzBeG66yXaKEhdnL3GXK5c/bt8+v1ybNrB4ccEkXb689/vbI0bYQUWm5y5+jkmrJ7Fo1yKmbZjG0FZDnQ5JqYjg2KX4tm1TzFdf2Uvx7l/q7rHn/dI9e/ITqOcvfxH7FLM7aWVm5ifrwtsTgWrV8v9oOH7cXiL2ts2YmPzkZkzBxF54u+XK5Sc4z2TtrX7Pp6vddXurPyoKoqPz6/c8PIW3q1S4e+HHF3hi9hPUKF+D1fetpk5iHadDUiqggnEpXu+xK6Uck52bzYBPBjB762wuP+9ypg2bppfkVUQLRmLXR4qUUo6JiYphwpUTqFKuCl//+jXjlmjXrkqdK03sSilHJVVK4pmLbAP8I2eMZMqaKQ5HpFR408SulHLcyK4j+VPPPwFw57Q7+WHbD84GpFQY08SulAoJz1z0DMPbDOdk5kkGfjKQaRumOR2SUmFJE7tSKiRER0XzydWfcH/K/WTkZDDsi2Gk7tYHbJUqLU3sSqmQESVRjB08llvb30p6djq9P+zNl2u/dDospcKKJnalVEgREcZfPp7bO9xOenY61/7rWl5Z8IrTYSkVNjSxK6VCTnxMPB8M+YCXLn0JQXh01qOMWzIOp9rdUCqcaGJXSoUkEWH0haN54ZIXMBh+P/33DP50MDuOaccAShVHE7tSKqQ91vMxPr36UyrHV2bmppk0+2czHpr5ECczTzodmlIhSRO7UirkDW87nNX3rWZIyyFk5Wbx+qLXaTm2Je8te08vzytViCZ2pVRYaFC5Af8Z9h8W3bWIVjVasfvEbu7+7910Ht+Zqeumkp2bXfJGlCoDNLErpcJK1/pdWXXvKj4a+hHx0fEs37ucq6dcTa8Pe7HuwDqnw1PKcdq7m1IqbB1KO8SElRP4x4J/sPvEbgCuOO8KBrcYTItqLWhRvQVJlZKIEj2HUaEhZLptFZGBwOtANPCeMebFQuvjgYlAZ+AQcIMxZltx29TErpTyl90ndvPM3Gd4f/n7ZOVmnbG+b+O+dK7bmQ51OtC2VlsaV2lMhbgKxETFOBCtKstCIrGLSDTwK9AP2AksAYYbY9Z6lLkfaGeMuVdEhgFXGWNuKG67mtiVUv62+8RupqyZwsp9K9l4aCPzd8wvtnx8dDwV4iqQGJdItYRqeUNCTAIxUTHERsUSExVTYIiNLrisNGXcfc0L4tM02Nf+3NNVylWhfZ32AfnZqeAIRmL35c/VrsAmY8wWV1CfA0OBtR5lhgJjXNNfAGNFRIw+rqqUCqJ6FevxULeH8uaNMew8vpMVe1ewYu8Klu9dzur9q9l7ci+nMk+RkZNBRnoGh9MPs/3Ydgcj903vRr2Ze/tcp8NQIc6XxF4f8GwRYidwQVFljDHZInIMqA4c9CwkIiOAEa7ZDBH55WyCDhM1KLT/EUb3L3xF8r5BBO/fPOYhv5OI3T+XSN+/loGuwJfELl6WFT4T96UMxpjxwHgAEUkN9OUIJ+n+hbdI3r9I3jfQ/Qt3ZWH/Al2HL4+K7gQaeMwnAbuLKiMiMUBl4LA/AlRKKaWU73xJ7EuAFiLSRETigGHAtEJlpgG3uaavBWbr/XWllFIq+Eq8FO+6Zz4S+Ab7utsHxpg1IvI0kGqMmQa8D3wsIpuwZ+rDfKh7/DnEHQ50/8JbJO9fJO8b6P6FO92/c+RYAzVKKaWU8j9tjkkppZSKIJrYlVJKqQhyToldRK4TkTUikisiKYXW/UlENonIBhEZ4LF8oGvZJhF53GN5ExFZJCIbRWSy60E9RCTeNb/Jtb5xSXUEgiuGFa5hm4iscC1vLCLpHuve9vhMZxFZ7Yrxn+JqUkpEqonIt659/VZEqrqWi6vcJhFZJSKdArlPhfZvjIjs8tiPwR7rAn4sg7B/L4vIetfPdaqIVHEtj4jj56uijlmoEZEGIjJHRNa5fsf8wbU84N/TIO7jNtf3a4W4XoE6m++WiNzmKr9RRG7zWO71+xukfWvpcYxWiMhxEXkonI+fiHwgIvvFo/2VYByvouooljHmrAegNfZl+x+AFI/lycBKIB5oAmzGPngX7ZpuCsS5yiS7PjMFGOaafhu4zzV9P/C2a3oYMLm4Os5lf0qx3/8A/uKabgz8UkS5xUB37Hv+M4BBruUvAY+7ph8H/uaaHuwqJ0A3YFEw9sdV9xjgUS/LA34sg7R//YEY1/TfPH7mEXH8fPwZFHnMQm0A6gKdXNMVsc1aJwfjexrEfdwG1Ci0rFTfLaAasMU1ruqarlrc99eh791eoFE4Hz+gN9DJ8/dFMI5XUXUUN5zTGbsxZp0xZoOXVUOBz40xGcaYrcAmbNO0ec3TGmMygc+Boa6/TC7GNkcLMAG40mNbE1zTXwCXuMoXVUdAueq+HvishHJ1gUrGmAXGHpGJeN+nwvs60VgLgSqu7TgpGMcy4Iwxs4wx7g67F2LbYyhSBB0/T16PmcMxeWWM2WOMWeaaPgGsw7ZwWRR/fk+dVNrv1gDgW2PMYWPMEeBbYGAJ399guwTYbIz5rZgyIX/8jDHzOLN9lmAcr6LqKFKg7rF7a4a2fjHLqwNHPX7xupcX2JZrvbu52qK2FWi9gH3GmI0ey5qIyHIRmSsivVzL6rti8hZfbWPMHrC/wIBaHp9xYp/cRrouG33gcbknGMcy2O7A/kXsFinHryShHp9XYm/ZdAQWuRYF+nsaLAaYJSJLxTa3DaX/bhW3vKjvb7ANo+CJUKQcPwjO8SqqjiKVmNhF5DsR+cXLUNxf+kU1MVva5WezrbPm474Op+CXdA/Q0BjTEXgE+FREKp1lfH7fpwIbL37/3gKaAR2w+/SPEmLy57H0C1+On4g8CWQDk1yLwub4+UGox3cGEUkEvgQeMsYcJzjf02C50BjTCRgE/F5EehdTNhz3D9d97yHAv1yLIun4FcfR/fGlgZpLz2K7xTVD6235QeylihjXX2Ce5d3b2ikFm6v1panbUilpX131X43td979mQwgwzW9VEQ2A+e54vO83OsZ3z4RqWuM2eO6BLPftdzv++TJ12MpIu8CX/sQk7+OpV/4cPxuAy4HLnFd7gqr4+cHoR5fASISi03qk4wx/wYwxuzzWB+o72lQGGN2u8b7RWQq9rJzab9bO4G+hZb/QPHf32AaBCxzH7dIOn4uwTheRdVRpEBdip8GDBP7FHQToAX2wQCvzdO6fsnOwTZHC7Z52v94bMtbc7VF1RFIlwLrjTF5l0xEpKbYPusRkaauOLa4LpmcEJFurvtBtxaxT4X39VaxugHH3JdgAq3QveCrAPeTn8E4lgEnIgOBx4Ahxpg0j+URcfx85Evz0CHB9TN/H1hnjHnFY3kwvqcBJyIVRKSiexr7cOcvlP679Q3QX0Squi5r9we+KeH7G0wFrnBGyvHzEIzjVVQdRTPn9pTgVdi/NDKAfa4A3euexD7NuAGPpzGxTwv+6lr3pMfyptgDuQl72Sbetbyca36Ta33TkuoI1AB8BNxbaNk1wBrs05rLgCs81qVgv7ibgbHkt/RXHfge2OgaV3MtF+BNV/nVeLxpEIR9+9hV5yrXF6luMI9lEPZvE/be1grX4H46PyKOXyl+Dl6PWagNQE/spchVHsdscDC+p0Hav6au79xK1/fvybP9bmGfGdnkGn5X0vc3iPtYHjgEVPZYFrbHD/sHyh4gC5v37gzG8SqqjuIGbVJWKaWUiiDa8pxSSikVQTSxK6WUUhFEE7tSSikVQTSxK6WUUhFEE7tSSikVQTSxKxUAImJ8GLa5yn4kIjtL2GRQiO19y7gaEPLb9nwo19dVb19/1KtUWeaX/7xKqTN0LzQ/FfvO8hiPZRlBi0YpVWZoYlcqAIzt0SmPiGQABwsvP1ciEm9ss7hKKQXopXilQoaIdBSRH0UkTUQ2isi9hdbf7rpc3VtE/iUiR8nv8QwR6SMi34vICRE5JSLfiEibQtsYICLzReSYiJwUkQ0i8hcv4TQRkf+5yvwmIn8RkahC22opIlNF5KiIpIvIQlfTvSXtZ00R+VREjrs+OxGoUqofllKqSJrYlQoNlYBPgU+w/S8vAd4SkYu8lJ0EbMW2k/04gIhchm1u8iRwM3AjUBH4UUQauMo0xTbjuQ24Advr1itABS91TAVmY/t+/gr4K/ntVSMi9YCfgPbASOB64CjwPxEZVMK+/hvbGc8TrjiygTdK+IxSykd6KV6p0FARuN8YMwdAROZhO4gYju3swtMXxpg/Flr2OjDXGOPZRe0cYAswCngI6ATEAfcZ2wUq2OTtzT+MMR+6pr8TkYtdsbiXPQJUBbobYza56psOrAWeo2B/93lEpB+2HfjhxpjPXYu/EZEZFOzdSil1lvSMXanQkOZO6pDXnexGoKGXslM9Z0SkBbaP60kiEuMegDRgAeDu53sFtgOLz0XkWhGpVUw8/ys0/0uhWHoDC91J3RVzDrajjA5i+7T3pjuQg+2O1dPnXsoqpc6CJnalQsMRL8sysD3iFVa4K1h3gn4fm7g9h8uxvUPhSsIDsP/vPwb2isgiEenjpY7DJcRSzUscAHuxPVtV9bIOoC5wxBiTVWj5Pm+FlVKlp5filQo/hd8LP+Qa/wn4zkv5zLwP2qsCc0QkHrgQeBp7X7yxMeZgKWI4DNTxsryOK77Cfxi47QGqikhsoeReuxR1K6WKoYldqfC3AftA3PnGmBd9+YDrUv9sEUkE/gM0AUqT2OcCD7n+INgGICLR2IfhlhtjThTxuQVANHANBS+/DytF3UqpYmhiVyrMGWOMiPwe+I+IxAFTsEm6NtAD2G6MecX1+lxvYDqwA6iBPcvfjb2HXhqvArcD34rIU8Bx4H7g/blKnQAAALdJREFUPOCyYmL9VkR+At4RkRrY5whuANoU9RmlVOnoPXalIoAxZjo2aVcA3gO+AV7CXhpf4Cq20rX+BWAWMBb72tzFxpj0Uta3G/t0+xrgLeAL7H33y4wxM0v4+NXYPy5eACZjTzBGlqZ+pVTRxJgSm3FWSimlVJjQM3allFIqgmhiV0oppSKIJnallFIqgmhiV0oppSKIJnallFIqgmhiV0oppSKIJnallFIqgmhiV0oppSLI/wPH10gOO5LhFgAAAABJRU5ErkJggg==\n",
+ "text/plain": [
+ "<Figure size 576x288 with 1 Axes>"
+ ]
+ },
+ "metadata": {
+ "needs_background": "light"
+ },
+ "output_type": "display_data"
+ }
+ ],
+ "source": [
+ "def plot_precision_recall_vs_threshold(precisions, recalls, thresholds):\n",
+ " plt.plot(thresholds, precisions[:-1], \"b--\", label=\"Precision\", linewidth=2)\n",
+ " plt.plot(thresholds, recalls[:-1], \"g-\", label=\"Recall\", linewidth=2)\n",
+ " plt.xlabel(\"Threshold\", fontsize=16)\n",
+ " plt.legend(loc=\"upper left\", fontsize=16)\n",
+ " plt.ylim([0, 1])\n",
+ "\n",
+ "plt.figure(figsize=(8, 4))\n",
+ "plot_precision_recall_vs_threshold(precisions, recalls, thresholds)\n",
+ "plt.xlim([-100000, 100000])\n",
+ "plt.show()\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 14,
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "image/png": "iVBORw0KGgoAAAANSUhEUgAAAf4AAAF8CAYAAAAuF9n2AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAgAElEQVR4nO3deZgU1bnH8d/LsAzgKMpyUUARRQVR0Iy4oEHjhksguIJ6FUUJrhFR44YLGjXifl1Q4xKNC+7B3biiCAZQISKigNsACqiAsi/n/nF6Uj1DD3TPdFd1d30/z9NPnaqurnqnYnirzjl1jjnnBAAA4qFe1AEAAIDwkPgBAIgREj8AADFC4gcAIEZI/AAAxAiJHwCAGAk18ZvZA2Y2z8w+reF7M7PbzWyGmU0xs13DjA8AgGIX9hP/Q5J6ref7QyR1THwGSbo7hJgAAIiNUBO/c26MpJ/Ws0sfSQ87b7ykZma2eTjRAQBQ/PKtjb+NpO+S1isS2wAAQBbUjzqAaizFtpRjCpvZIPnmAEnNf7PFFu21OXUDAIAYmDRp0gLnXMva/DbfEn+FpHZJ620lzUm1o3PuXkn3SpJZuRs8eKKGDct9gAAARM3Mvqntb/Otqn+0pBMTvfv3kLTIOTc36qAAACgWoT7xm9njkvaV1MLMKiRdIamBJDnnRkp6WdKhkmZIWirp5DDjAwCg2IWa+J1z/TfwvZN0ZkjhAAAQO/lW1Q8AAHKIxA8AQIyQ+DO0dq3kUr5gCABA/iPxp2nKFOnCC6WttpJatJDmzUu9308/SWvWhBsbAADpIvGvx6pV0mOPSbvtJnXtKo0YIVVU+OR+xBHSihX+6X/SJOnyy6WddpKaN/ffTZ8urV4tTZggvfJKOLUEixdLU6f68wIAkEq+DeATqaVLpTvu8E/1ixZJV1/tE70kbbaZdOyx0iefSOPGSWPHSqWl0pZbSt9+W/U4o0f7T3WnnSZtvbX05ZfSyJFSw4Z++7x5/vg//SQ1auSbE1atkpo0kSZP9uf44Qfpo4+Cz4QJ/rdlZdIvv0ht2waxVnriCalzZ39DMG2a/12XLlL9+j4WSzVOIgCgqJkrggZrs3I3fHjdRu574w3puOOk+fOrbu/USRoyRDrhBKlxY588f/ObqvtssYXUp490wAHSgAE+Eadj8GDptdekr76qfdx1se220owZ0sknS2edJa1c6WssVqzwNyItW0oNGkQTGwCgZmY2yTlXXqvfxj3xr1kjDRsmXXdd1e3bbCNde6101FFSvWoNIr/8Il1xhbRkidSvn9SzZ7DPokXSwoX+BqFTJ6ljR+nNN6WDD05dO5CuTp2kXXf1n513lr75xj/ll5RIv/7qn+w7dJDmzJFOOcWfU/K1Ep07+1h//rnmvgk1Wb3ax9y6tb/xAQBEj8Rfy8S/ZIlP7K++6hP3VVf55NakiTRwYFAVn00//yz16iU1bSrtv7//NGvmq+M7dPBJdtttfS1Ax44+hhUr/P614VzVKv21a6VRo/xxf/hBOjMxXFK9ev679fn4Y6lbt9rFAQDIHhJ/LRL/4sXSYYdJ77/vq7RHjZL22y93MRaCyv8U1q71/QBSOeccqbxc2ntv318BABC+uiT+WHbuW7lS+sMffNJv21Z66y3/dB13lTUDJSX+JmDuXGmjjXxHxRNO8N/dfnuwf8eO0pgxvhZjyy1rXysBAAhPrBL/qlU+mV19tfT2277deswYnlxrsvnmfnn88dLGG0u9e/sxDBYs8Nu//DLYp9L48b5D4MYb+30bNPDNF1Om+OaEbt24yQKAKMUm8Tsnde/uX8eTfFv+6NEk/XT9/vdBU8DHH/tOhqnssUd6xzvlFN+xsmlT6eyzff+GVauoNQCAXIvNAD7PPhskfUm65RY/MA8yt8su/iZgzRrf67+yw6CUemyAzTZbd9sDD0h//7t0113+jYVGjXyzgpnfXjn6oXMMkQwA2RSLxL9qlXT++cF6s2bSoEHRxVMs6tXz/QHMpGOO8TcBy5b5RD1vnl9fvlz68ccggT/zjP/tpptKrVqlPu6AAb5zYY8evrmgXj1/jsrPsGG+n4ZzvpPmzz/7JoYZM4LzzJ7tm3FmzMj8FUYAKGaxqOp/7DHp6699uWdP6R//YNS6XCgp8R/JvylRuS3ZEUdUfYJfs8bXGCxZIp16anBjIEkffJD6PNdc4z+Z+vOf/U3EJpv4OOhrACCOiv51PuekHXf0Q9Y++KB/mkR+e/556YsvfD+CTp38E/u0af7ti/vvX3f/xo19TUNtNGjga4Qq3Xeff4Nh5kzfQbFdu9odFwByiff415P4331X2ndfP6zuV1/lZlAehGvJEt+E4JxvtqkcNXHmTP8036CBb0qo9OijPpl37ix99lntztmli3Trrb4fQtu2/m2G6iM6AkBYeI9/PR580C8HDCDpF4umTVP3/q+p6v744/0n2axZfgbFlSv9TcSYMb6jYU0+/dTPxZBKjx6+OalZM19Tsdlmvo/CZpv5GwUAyCdF/cS/dKnvQLZkiX/nfNtto4kPhaGyU2CrVj5x//vf0lNPSTffXLfj9u7tJ0JasEA66CDffEAfEwB1UZcn/qKurHzzTZ/0y8tJ+tgwM1+N37Chr8bfYw/pppuCNwUqP2vX+kmYjj46veOOHi317eunQt5qq+AthSOP9MvOnaWHH/a1BkuXBudZvdrXSABANhV1Vf8//+mXffpEGweKi5kfy+DJJ4NtixZJZWVBu79zvk/JgQf6ZoVUnn3WL6dNk046af3n7NvX3whssYW/mR00SNpnn7r/LQDip2gTv3PSSy/5cu/e0caC4rfJJlXXzfxohDNnVt2+apXvS/DNN9KkSb5vQTqee67q+j/+kXq/k0/2N7xnnx28ssh0ygCSFW0b/xdfSNtv79trv/+eNlUUhpkzpebN/SuMrVv75oZGjaSKCqm01C9ro2lT6bjjfE1Bly7+WAAKF736U6h8kvrtb0n6KBzbbOOXzZr5ZU2vHy5c6F9VXbLE1zYMHOibFWqqCViyxI9RcN99wbYDD/RTUQ8dyhsvQJwUbee+ysRPOyiKUbNmvu/KccdJhx3ma7UeeSR1J8THH5d69Vr3GP/6l3TJJb5GoXI45LIy6dprpR9+CP9vAhCOok3848b55d57RxsHEJXKToj9+kmvvBLcEIwdK91xR+rf/PqrdOmlvpnBTHrjDW4CgGJTlIn/l1/85CwNG/r2TACBvfaSzjzT3wSsWiWtWCFNmSI99JAfkTDZgQf6m4DSUmnkyEjCBZBlRZn4p0zxyx13pO0SWJ/69f3/R3bayb9SOGeOvyG48caq+61YIZ1+uq8F6N07eGMGQOEpysT/ySd+2a1btHEAhWroUH8DsGyZdNttVb974QXp8MP9TUD79tL//q/0xBN+ToS1ayMJF0AGijLxT57sl127RhsHUOhKS6VzzvE3AZMn+2aCZN98498k6N/fT4RUUuJvCIYNk955x9cMvPKKNH9+JOEDSKEoE//06X65447RxgEUk5139h0DnfPjDRx6qN/eo8e6+15zjX9V8PDD/X6tWvkbglNOkZ5+et2BjQCEpygT/5df+mVNs7UBqJsOHfzTvHPS++8HzQIXXFB1v623rrr+4IN+joNttw1eIXz+eeYkAMJUdIn/11+luXN9h6W2baOOBoiP0lLphhuqjiUwa5ZfVr4R0Lr1ur/r29ePJbDDDtLLL4cbMxBHRTNkrzRRY8f6oUm7dZM6dap51DMA0Zo71084tCGlpdIf/yj99a/+5gCAx7S8CT16+HeRJabhBfLZ5psHtQI//SSdf37q/ZYv928VlJZKEyaEGyNQrIoq8UvSrbf6ZfW2RQD5adNNpREj/E3A4sXS7Nm+yv+ii/yERZW6d/d9Ao4/vvaTFQEowsRfqV27qCMAkKmyMt8EcMgh0nXXSQsW+FEGkz32mP//t5mfgdPMv8Hz4YfRxAwUmqJN/G3aRB0BgGy44w5fG3D33et+98UXfvnZZ34K48o3Bf7yl3BjBApJ0SZ+evQDxWXw4KBfwPffS08+Kd1yi28CqO6yy4KbgI4d/X6rV4cfM5CPiqpXf7KZM/27xgDiYdYs6e9/l4YPT2//+fOlFi1yGxOQK/TqTyGdV4UAFI8OHaSrrvI1AhUV0p13Sj171rx/y5a+RuCww6QLL2T6YcRH0T7xF8GfBSBLFi6Uxo3zDwQbmrxr2DDpxBN5JRj5rS5P/CR+ALHjnHT22dKSJcHYH6nUqyc9+6zUp09ooQFpIfGT+AHUgXPSxRf7mQZnz069z7hx0m67+RkIgajRxg8AdWAmXX+97xtQOZpg9QmH9txTql8/eFvgtdekFSuiiReoC574AaAGzkm77ip98sn69zv/fKm8XDroID8SIZBrPPFX8+qrUUcAoBiYSR9/7G8AVq6Uxo9Pvd+NN0r9+kmbbeabBIB8VpSJ/4ADoo4AQLFp0EDaffdgEKG1a6VTT5U22qjqfnvtFTQHMM0w8lFRVvUXwZ8EoMC89JJ0+OGpv1u71t8IANlCVT8AROyww/xDx+rV0oABVb+rV0965JFIwgLWUXSJv3LSDgCIQkmJ9OCD/iZgn32C7SeeGDQBmPk+AUAUii7xMz4/gHwxZoz03nupvxs1yt8A7LKL9Pnn4caFeCu6Nv4i+HMAFKEvv/SDA02a5F//S+Wnn3gdEOmhjR8A8lzHjtK++0pDh0rLlklnnLFuDeVmm/lagCZNpIEDpU8/jSRUFDme+AEgQmvW+BEB1+err6T27UMJBwWioJ74zayXmU03sxlmdlGK77c0s7fN7GMzm2Jmh4YdIwCEpaQkeBvgrbf80MA771x1n6239jUB998fTYwoLqE+8ZtZiaQvJB0oqULSBEn9nXOfJe1zr6SPnXN3m1lnSS8759qv/7g88QMoLs5Jf/qT9H//V3X7rrv6fgKIt0J64u8uaYZzbpZzbqWkJyRVn/DSSdo4Ud5E0px0Dz53blZiBIDImUm33+5vAJLb+j/6yH83cGB0saGwhZ3420j6Lmm9IrEt2ZWSTjCzCkkvSzo73YO3bl3X8AAg/+y4o28K2G67YNsDD/gbgKVLo4sLhSnsxJ9q0MrqlfP9JT3knGsr6VBJj5jZOnGa2SAzm2hmE6t/BwDFpqREmj5dWry46vamTf0NwCWXRBMXCk/Yib9CUruk9bZatyp/oKQnJck5N05SqaQW1Q/knLvXOVde2zYOAChEZWW++n/w4Krbr7uO+QCQnrAT/wRJHc1sazNrKKmfpNHV9vlW0v6SZGad5BP//FCjBIA8d/fd/gbgueeqbq8cB2DKlGjiQv4LNfE751ZLOkvSa5KmSXrSOTfVzIabWe/EbkMlnWZmkyU9LmmAK4bBBgAgB/7wB38D0Capt9SyZVLXrtQAILWiGsCnCP4UAKi1igrphBOkd9+tup1pgYtPIb3OBwDIkbZtpXfe8TUAyW851avHREAIkPgBoAjNnSs1bhysd+rkn/qpGUXRJP4LLog6AgDIL0uXSg8/XHXbkCHRxIL8URRt/LvuWu4++ojX+QGgJslt/KtX+3EBULhi38Zfryj+CgDInU8+Ccr166/bARDxQcoEgBjo2lXq0CFY33df6dxzIwsHESLxA0BMzJzpe/1Xuu026dBD6fAXNyR+AIiRnj2lL74I1l95xTeXfvttdDEhXCR+AIiZjh396H5t2wbbttrKdwBcsya6uBAOEj8AxFBpqfTdd9JNN1XdXr++dPnlVP8XMxI/AMTYeef5IX332CPYdvXVvvr/7bejiwu5Q+IHgJgzk8aNk559tur23/2OWf6KEYkfACBJ6tvXV/E//XSwrWtX6ZproosJ2UfiBwBUceSR0i23BOvDhkllZdKCBdHFhOwh8QMA1nHuuVUT/a+/Si1bSt26RRcTsoPEDwBIqXlzX/U/bFiwbfJk6ZlnoosJdUfiBwCs1/Dh/om/0lFH+Q6BFRXRxYTaI/EDADaoadN1E327dnT8K0QkfgBAWtq08VX/l18ebBs2zD/9L1kSXVzIDIkfAJCRq66Spk2rum2jjaKJBZkj8QMAMrbDDv7p/5hjgm3Jo/8hf5H4AQC1NmpUUP7wQ2mvvaKLBekh8QMA6mTt2qA8bpy06abRxYINI/EDAOrETFq6NFhfuNBvQ34i8QMA6qxxY9/m36JFsO3gg6OLBzUj8QMAsmb+/KD8+uvShRdGFwtSI/EDALIquc1/xAiptFRasya6eFAViR8AkFVm0s8/B+srVkj160cXD6oi8QMAsq5ZM//kf8QRwbbvv48uHgRI/ACAnDCTnn46WN98c+mkk6KLBx6JHwCQM2bS3/4WrD/8sDRhQnTxgMQPAMixgQOlxYuD9e7dpblzo4sn7kj8AICcKyuTHnssWN9iC+nGG6OLJ85I/ACAUPTvL911V7B+wQW+6h/hIvEDAEJz+unSO+8E6yedJN16a2ThxBKJHwAQqp49pYqKYH3IED+zH8JB4gcAhK5NG2natGB9jz38QD/IPRI/ACASO+wgvfBCsF5aKv3wQ3TxxAWJHwAQmcMPrzqoT+vWVcf6R/aR+AEAkXrooaod/EpK/BS/yA0SPwAgcn/6kzRoULB+xhnRxVLsSPwAgLxwzz1Sw4a+PHJktLEUMxI/ACBv/Oc/Qfn116OLo5iR+AEAeWO77YLywQdLU6dGF0uxIvEDAPLKAw8E5S5dpKFDo4ulGJH4AQB55eSTpcGDg/Wbb5buvz+6eIoNiR8AkHfuvlt6//1g/dRTo4ul2JD4AQB5qUcP6euvg/V+/SILpaiQ+AEAeWurraSddvLlUaOkN96INp5iQOIHAOS1SZOC8oEHSiNGRBdLMSDxAwDyWoMGVafxvfBChvStCxI/ACDvtWlTNfnXI3vVGpcOAFAQ2rSRunYN1s8+O7pYChmJHwBQMD75RCot9eU77pB+/TXaeAoRiR8AUFDmzQvKZWXRxVGoSPwAgIJSViYdd1yw3qpVdLEUIhI/AKDgPPpoUJ4/n17+mQg98ZtZLzObbmYzzOyiGvY5xsw+M7OpZvZY2DECAPLf2rVB+ZZboouj0ISa+M2sRNKdkg6R1FlSfzPrXG2fjpIultTDObejpHPDjBEAUBjMpI4dfXnoUGnZsmjjKRRhP/F3lzTDOTfLObdS0hOS+lTb5zRJdzrnfpYk59w8AQCQwnvvBeXWraOLo5CEnfjbSPouab0isS3ZdpK2M7OxZjbezHqlOpCZDTKziWY2cf78+TkKFwCQz/7nf6SmTX158WJpzz2jjacQZJz4zewkM3s10QY/q9pn5oZ+nmJb9S4Z9SV1lLSvpP6S/mZmzdb5kXP3OufKnXPlLVu2zPTPAAAUieTX+8aPlx5/PLpYCkH9THY2s2GSrpL0qaRPJK3I8HwVktolrbeVNCfFPuOdc6skfWVm0+VvBCZkeC4AQAw0aSItXx4M7HPccdI++0ht20YbV77KKPFLGijpNufckFqeb4Kkjma2taTZkvpJOq7aPs/LP+k/ZGYt5Kv+Z9XyfACAGGjUSPrhB1/1L0nt2vGKX00yrepvLumF2p7MObda0lmSXpM0TdKTzrmpZjbczHondntN0o9m9pmktyVd4Jz7sbbnBADEQ6tW0mmnBevvvhtdLPnMXAa3RGb2oqQ3nXN59cZkeXm5mzhxYtRhAAAitny51LhxsF6sT/1mNsk5V16b32b6xH+upJPN7EQza2Fm9ap/ahMEAADZUFoqvf9+sD5qVHSx5KtMn/grx0mq6UfOOZdpv4E644kfAJCsXr3gab8Yn/rr8sSfaZIerpqTPgAAeeHLL6Vtt/Xlbt38dL7wMkr8zrkrcxQHAABZs802/jNzpjR5sn/qt1QjycRQrdvkzWwjM2tnZk2zGRAAANnwxhtBuU31MWJjrDYj9x1sZhMlLZT0taRFZvZvMzsw28EBAFBb7dsH1f1z50pjxkQaTt7IKPGb2cGSXpK0kaSrJZ0h6RpJZZJeJvkDAPLJtGlBuWfP6OLIJ5k+8V8p6XVJnZ1zVznn7km0++8o6V/yw/kCAJAX6teXLr00WC/GHv6ZyjTxd5WfMndt8sbE+l2SumUrMAAAsmH48KD8m99EF0e+yDTxr5C0cQ3flSnzSXsAAMipevWk7bf35f/8J9pY8kGmif8dSVcnJtn5LzPbUr4Z4O3shAUAQPbcdptfrl4dbRz5INPE/2dJm0iabmZjzGyUmb0r6UtJzRLfAwCQV8qTxrhLfs0vjjJK/M65LyTtLOl2SY0k7SqpVNJtkro5577MeoQAANRR8+ZB+cAD4/3kn/F7/M65uc65851zuzvnOiaWFzrn5uYiQAAAsuHFF4Py7bdHF0fUmE0PABALhx0mbbKJLw8dKv36a7TxRGWDY/Wb2VuSznDOfZ4or49zzu2fndAAAMiu99+XdtrJl8vK4vlefzpP/MnTGtRLrNf0oQYBAJC3unSRhgwJ1lfE8CV0c0Vwu1NeXu4mTpwYdRgAgALgnH+3v9KqVX6Ev0JiZpOcc+Ub3nNdPKEDAGLFTNpzz2C9QYPoYolCppP09DGzk5PWtzKzcWb2i5k9bWYbZT9EAACy64MPpNatg/UlS6KLJWyZPvFfJqll0vrNktpKulfSb+VH7wMAIO/NmROUN4rRY2umiX8bSVMkycwaSzpU0nnOuaGSLpHUN7vhAQCQG2bS1VcH6zfdFF0sYco08ZdKWpYo7yX/OuDrifXpkrbIUlwAAORc8pS9558fXRxhyjTxfy1p70S5j6RJzrlFifVWkhal+hEAAPnITJo1K1gfNy66WMKS6QsM90i60cz6Suom6fSk7/aU9Fm2AgMAIAxbJ803u9de0tq1/oagWGU6Sc9tkgZIGifpFOfcfUlfl0l6MHuhAQAQjgceCMpdu0YXRxgYwAcAAPkZ/H76yZdXrszv9/sZwAcAgDqaPTso33tvdHHk2gYTv5mtMbPuifLaxHpNnxjPcAwAKGSlpVKnTr581lnRxpJL6XTuGy6pIqlc+G0DAACkcO650h//GHUUuUUbPwAACcuXS40b+/K0adIOO0QbT01Ca+M3swZm1rSG75qaWR53hQAAYP1KS4PymWdGF0cuZdq5735J99Xw3T2JDwAABeuAA/zyrbeijSNXMk38+0r6Zw3fjZa0f52iAQAgYvclPd5+9VV0ceRKpom/laR5NXw3X9L/1C0cAACi1b59UN5998jCyJlME/88STvV8N1Okn6sWzgAAETv6KP9cv58aebMaGPJtkwT/4uShpnZzskbzWwnSZdKeiFbgQEAEJUHkwagP/zw6OLIhUwT/+WSFkqaZGYfmNmTZjZW0kfyM/Ndlu0AAQAIW9Om0tln+/Lnn0tF8Ob7f2U6Sc8CSbtJuk6Syc/QZ5L+Imm3xPcAABS8K64IyueeG10c2cYAPgAA1CB5et4lS6QmTaKLJVnok/SYWQszO9zMTjKzzRLbSs2MSX8AAEVjxoygPGhQdHFkU6Yj95mZjZAfu3+0pAcktU98/U/5Dn4AABSFbbaRdtvNlx99NNpYsiXTJ/SLJZ0lP1nP7vLt+5VekFRkfR8BAHH39NNB+ZtvoosjWzJN/KdKGu6cu1a+J3+yGZK2yUpUAADkiS23DMrJr/kVqkwTfxtJ42v4bqWklBP4AABQyDbZxC+vuiraOLIh08Q/W1KXGr7rKqkIRzUGAMTdc88F5Zdeii6ObMg08T8l6XIz65G0zZnZdpKGSnoia5EBAJAn9tsvKB95ZHRxZEOmif9KSZ9LGiPpy8S2pyT9J7F+fdYiAwAgj9x/v1+uWBFtHHWV6ch9y+Sn5h0g6QNJb0iaIGmQpAOdcyuzHB8AAHnh2GOD8rJl0cVRV/XT3dHMGkg6VNIU59wjkh7JWVQAAOSZpknd1++5p3CH8U37id85t0rSkwoG7AEAIFZKS/1yyJBo46iLTNv4Z0lqlYtAAADIdyNHBuVFi6KLoy4yTfw3SLrUzFrmIhgAAPJZcjt/s2bRxVEXabfxJ/xO0maSvjKz8ZLmSkqe3s85507KVnAAAOST0lLpkkuka6/168uWSY0bRxtTpjKaltfMvlbVRF+dc851qGtQmWJaXgBAmCqn6333Xem3v43i/LWfljfTJ/5ySb8655bX5mQAABSDHj2ksWOlnj2lDJ6f88IG2/jNrMTMrjSzhZJ+kLTYzJ4xs1q1bphZLzObbmYzzOyi9ex3lJk5M6vVHQ0AALmyzz5BedWq6OKojXQ69w2WdLn8bHw3SvqnpD6Sbsn0ZGZWIulOSYdI6iypv5l1TrFfmaRzJH2Y6TkAAMi1yy4Lyg89FFkYtZJO4j9N0n3Oud855/7snDta0pmSTjCzhhmer7ukGc65WYlR/p6Qv4mo7mr5NwhoUgAA5J2mTaXdd/flK66INpZMpZP4O8iPx59slKQSSVtleL42kr5LWq9IbPsvM9tFUjvn3IvrO5CZDTKziWY2cf78+RmGAQBA3Rx1lF/OnRttHJlKJ/FvJGlxtW2/JJZlGZ7PUmz7b7cIM6sn34QwdEMHcs7d65wrd86Vt2zJsAIAgHCdeGLUEdROur3625hZ8mt6JUnbFybv6JybtZ7jVEhql7TeVtKcpPUySV0kvWP+XYnWkkabWW/nHO/rAQDyRvIz5+zZUps2Ne+bT9JN/E/XsP35FNtKUmyrNEFSRzPbWtJsSf0kHVf5pXNukaQWletm9o6k80n6AIB8Y0l12LfdJt1wQ3SxZCKdxH9ytk7mnFttZmdJek3+BuEB59xUMxsuaaJzbnS2zgUAQFhGjCicxJ/RyH35ipH7AABReOIJqX9/X16wQGrePJzz1mXkvkwn6QEAAAn9+gXl/fePLo5MkPgBAKiD00/3y8mTC2MUPxI/AAB1cOedQfmRR6KLI10kfgAA6sBMatTIlwcOjDaWdJD4AQCoo5tvjjqC9JH4AQCoowEDgvKUKZGFkRYSPwAAddSkSVDu2jW6ONJB4gcAIAuSB/DJ57njSPwAAGTB+ecH5XyewIfEDwBAFphJJycGuX/11WhjWR8SPwAAWXLllUE5X6v7SfwAAGTJllsG5VGjootjfUj8AABk0d57++XIkfQC1CMAAA4BSURBVNHGURMSPwAAWbT99n45dWq0cdSExA8AQBZddllQfuWV6OKoCYkfAIAsat8+KN99d2Rh1IjEDwBAllW+0z95crRxpELiBwAgyyrH7v/2W2nOnEhDWQeJHwCALOvUKSi3aRNdHKmQ+AEAyLJ69aTbbgvWx46NLpbqSPwAAOTAOecE5UMOiS6O6kj8AADkyDPP+OUvv0irVkUbSyUSPwAAOdK3b1D+29+iiyMZiR8AgBwxk7p18+Uzzog2lkokfgAAcui886KOoCoSPwAAOXTMMUH5iy+ii6MSiR8AgBxq1Ehq2tSXL7442lgkEj8AADnXr59fPvustHZttLGQ+AEAyLHkwXwmTIguDonEDwBAzjVtKpWU+PLzz0cbC4kfAIAQ7LefX15/fbRxkPgBAAjB8OFRR+CR+AEACEH37kF52bLo4iDxAwAQgso2fkmaNy+6OEj8AACEZIcd/HLOnOhiIPEDABCSBQv88qyzoouBxA8AQEguvdQvP/pIci6aGEj8AACEZPDgoPzYY9HEQOIHACAkpaXSTjv58jXXRBMDiR8AgBAdcYRfzp4dzflJ/AAAhOjYY/3yl1/8J2wkfgAAQrTDDpKZLz/3XPjnJ/EDABAiM+moo3z5pJPCPz+JHwCAkFVW90vS66+He24SPwAAITvySKllS18++OBwz03iBwAgAjffHJTDHMyHxA8AQAROOCEoP/NMeOcl8QMAELGxY8M7F4kfAICI9O/vl7feGt45SfwAAERkwICgvHBhOOck8QMAEJGDDgrKm24azjlJ/AAARCh5xr4lS3J/PhI/AAARuvvuoPz557k/H4kfAICIVVbzk/gBAIiBrbbyy4suyv25SPwAAESsVy+/rKjI/blCT/xm1svMppvZDDNb597GzM4zs8/MbIqZvWlmW4UdIwAAYTrxxKCc6+F7Q038ZlYi6U5Jh0jqLKm/mXWuttvHksqdcztLelrSDWHGCABA2HbYISgvW5bbc4X9xN9d0gzn3Czn3EpJT0jqk7yDc+5t59zSxOp4SW1DjhEAgFCZBbP15bq6P+zE30bSd0nrFYltNRko6ZWcRgQAQB6YP98vR47M7XnCTvyWYlvK1gwzO0FSuaQRNXw/yMwmmtnE+ZVXCwCAAtW9u1/ecktuzxN24q+Q1C5pva2kOdV3MrMDJF0qqbdzbkWqAznn7nXOlTvnyltW1o8AAFCg/vrXoLx8ee7OE3binyCpo5ltbWYNJfWTNDp5BzPbRdI98kl/XsjxAQAQib33DsoTJuTuPKEmfufcaklnSXpN0jRJTzrnpprZcDPrndhthKSNJD1lZp+Y2egaDgcAQNGoXz9I/kcdlcPz5O7QqTnnXpb0crVtlyeVDwg7JgAA8sFuu0nvvy/Ny2F9NyP3AQCQJypn6mvYMHfnIPEDAJAn2iW6v69cKa1enZtzkPgBAMgTpaVBedq03JyDxA8AQJ4wk5o29eXHH8/NOUj8AADkkc6JGWw++CA3xyfxAwCQR04/3S/ffTc3xyfxAwCQR3r0CMq//JL945P4AQDII9ttF5QHDsz+8Un8AADkmSZN/HLq1Owfm8QPAECeuesuv/zss+wfm8QPAECeOewwvzSTXMrJ62uPxA8AQJ5p0UIqKfFJf+HC7B6bxA8AQB5as8Yvs93Bj8QPAEAeqqzuf+657B6XxA8AQB66/vqgvHZt9o5L4gcAIA8lv8//xz9m77gkfgAA8lDDhtLee/vyW29l77gkfgAA8tTgwX45a1b2jkniBwAgT3Xrlv1jkvgBAMhT224blBcvzs4xSfwAAOSpRo2C8lNPZeeYJH4AAPJYmzZ+OW5cdo5H4gcAII/9/vd+ef/92TkeiR8AgDx29NHZPR6JHwCAPLbHHkF5yJC6H4/EDwBAHmvSJCjfemvdj0fiBwAgz734YlB+/vm6HYvEDwBAnjvsMKl5c1/u27duxyLxAwBQAN54IzvHIfEDAFAAunWTTj+97sch8QMAUCBuvrnuxyDxAwBQIEpLpWXL6nYMEj8AAAWktLRuvyfxAwAQIyR+AABihMQPAECMkPgBAIgREj8AADFC4gcAIEZI/AAAxAiJHwCAGCHxAwAQIyR+AABihMQPAECMkPgBAIgREj8AADFC4gcAIEZI/AAAxAiJHwCAGCHxAwAQIyR+AABihMQPAECMkPgBAIgREj8AADFC4gcAIEZI/AAAxEjoid/MepnZdDObYWYXpfi+kZmNSnz/oZm1DztGAACKVaiJ38xKJN0p6RBJnSX1N7PO1XYbKOln59y2km6R9NcwYwQAoJiF/cTfXdIM59ws59xKSU9I6lNtnz6S/p4oPy1pfzOzEGMEAKBohZ3420j6Lmm9IrEt5T7OudWSFklqHkp0AAAUufohny/Vk7urxT4ys0GSBiVWV5jZp3WMDevXQtKCqIOIAa5z7nGNc49rnHvb1/aHYSf+CkntktbbSppTwz4VZlZf0iaSfqp+IOfcvZLulSQzm+icK89JxJDENQ4L1zn3uMa5xzXOPTObWNvfhl3VP0FSRzPb2swaSuonaXS1fUZLOilRPkrSW865dZ74AQBA5kJ94nfOrTazsyS9JqlE0gPOualmNlzSROfcaEn3S3rEzGbIP+n3CzNGAACKWdhV/XLOvSzp5WrbLk8qL5d0dIaHvTcLoWH9uMbh4DrnHtc497jGuVfra2zUogMAEB8M2QsAQIwUVOJnuN/cS+Man2dmn5nZFDN708y2iiLOQraha5y031Fm5syM3tG1kM51NrNjEv89TzWzx8KOsdCl8e/Flmb2tpl9nPg349Ao4ixkZvaAmc2r6ZV1825P/G8wxcx23eBBnXMF8ZHvDDhTUgdJDSVNltS52j5nSBqZKPeTNCrquAvpk+Y13k9Sk0T5dK5x9q9xYr8ySWMkjZdUHnXchfZJ87/ljpI+lrRpYr1V1HEX0ifNa3yvpNMT5c6Svo467kL7SPqtpF0lfVrD94dKekV+DJw9JH24oWMW0hM/w/3m3gavsXPubefc0sTqePmxGJC+dP47lqSrJd0gaXmYwRWRdK7zaZLudM79LEnOuXkhx1jo0rnGTtLGifImWnfcFmyAc26MUoxlk6SPpIedN15SMzPbfH3HLKTEz3C/uZfONU42UP5OE+nb4DU2s10ktXPOvRhmYEUmnf+Wt5O0nZmNNbPxZtYrtOiKQzrX+EpJJ5hZhfzbXGeHE1qsZPrvdviv89VB1ob7RY3Svn5mdoKkckk9cxpR8VnvNTazevKzUg4IK6Ailc5/y/Xlq/v3la+5es/MujjnFuY4tmKRzjXuL+kh59xNZran/BgtXZxza3MfXmxknPcK6Yk/k+F+tb7hflGjdK6xzOwASZdK6u2cWxFSbMViQ9e4TFIXSe+Y2dfybXaj6eCXsXT/vfinc26Vc+4rSdPlbwSQnnSu8UBJT0qSc26cpFL5cfyRPWn9u52skBI/w/3m3gavcaIa+h75pE+baObWe42dc4uccy2cc+2dc+3l+1H0ds7VelzumErn34vn5TurysxayFf9zwo1ysKWzjX+VtL+kmRmneQT//xQoyx+oyWdmOjdv4ekRc65uev7QcFU9TuG+825NK/xCEkbSXoq0W/yW+dc78iCLjBpXmPUUZrX+TVJB5nZZ5LWSLrAOfdjdFEXljSv8VBJ95nZEPnq5wE8jGXGzB6Xb45qkegrcYWkBpLknBsp33fiUEkzJC2VdPIGj8n/BgAAxEchVfUDAIA6IvEDABAjJH4AAGKExA8AQIyQ+AEAiBESPxADZjYgMdNf5Welmc00s2vNrDTi2L42s4eS1itjbR9ZUEARK5j3+AFkxdHyI32VSeor6eJEmTHUgZgg8QPx8olzbkai/C8z6yhpoJn9ifHTgXigqh+It48kNVbS+OmJIVgfNbP5ZrbCzD4xs77Vf2hmXc3sOTP70cyWmdl0M7s46fuDzOxlM5trZkvN7FMzG2pmJeH8aQBS4YkfiLf28tNX/yhJZtZO0oeS5kkaIj+u+rGSnjGzP1QOKWxm3SW9Iz9M6BD55oOOknZOOnYHSW9K+j9Jy+Vnc7xSUktJF+X0rwJQIxI/EC8liZkrK9v4j5R0rnNuTeL7K+Wn+eyZNG79a4kbguEKJmG5Uf5mYQ/n3NLEtreST5QYR1ySZH5ih/ckNZR0vpldQtMCEA0SPxAvn1dbv8s5d0fSei/5ST8WJW4QKr0maYSZbSxptaQekkYkJf11mNnm8jcSvSRtoar/3rSS9H1t/wgAtUfiB+Klr3y1fEtJ50k6w8w+dM49nPi+laQTE59UmktaKd8/qKKmk5hZPfnagS3kk//nkpZJ+oOkS+WnZwUQARI/EC+fVvbqN7O3JE2Rf5J/xjm3RL76/j1Jf63h93Pkp2BdK6nNes6zjXyb/v865/5RudHMfl/3PwFAXdCrH4gp59wKSRfIP+Wfkdj8qnwHvanOuYkpPisS1fvvSzrBzBrXcPgmieWqyg1m1kDS8Tn5YwCkjSd+IMacc6PNbIJ8h7s7JF0u6d+SxiTWv5a0qaQukjo4505J/PR8Se9KGmdmN8lX+3eQ1M05d7akaZK+kfQXM1sjfwMwJLy/DEBNeOIHcJn8U/9g59y38lX0kyVdK+lfku6W1FNJvfadcxPkO/h9J/+63svytQcVie9Xyrfnfy/pYUl3Shoj6fpQ/iIANTLnXNQxAACAkPDEDwBAjJD4AQCIERI/AAAxQuIHACBGSPwAAMQIiR8AgBgh8QMAECMkfgAAYoTEDwBAjPw/pgUhopn/96sAAAAASUVORK5CYII=\n",
+ "text/plain": [
+ "<Figure size 576x432 with 1 Axes>"
+ ]
+ },
+ "metadata": {
+ "needs_background": "light"
+ },
+ "output_type": "display_data"
+ }
+ ],
+ "source": [
+ "def plot_precision_vs_recall(precisions, recalls):\n",
+ " plt.plot(recalls, precisions, \"b-\", linewidth=2)\n",
+ " plt.xlabel(\"Recall\", fontsize=16)\n",
+ " plt.ylabel(\"Precision\", fontsize=16)\n",
+ " plt.axis([0, 1, 0, 1])\n",
+ "\n",
+ "plt.figure(figsize=(8, 6))\n",
+ "plot_precision_vs_recall(precisions, recalls)\n",
+ "plt.show()"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 15,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "from sklearn.metrics import roc_curve\n",
+ "\n",
+ "fpr, tpr, thresholds = roc_curve(y_train_5, y_scores)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 16,
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "image/png": "iVBORw0KGgoAAAANSUhEUgAAAf4AAAF8CAYAAAAuF9n2AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAgAElEQVR4nOzdd3iUVeL28e+ZSU8IhN6kIwoqoSxFBLuLKE2QYtm1UQQRVuxtdVdRsfCDFRUsr+7aBSki4NoVBKQFVFaR3msIAdJnzvvHTDAiCROY5JnM3J9ruZiWmRsXcuec5zznMdZaREREJDK4nA4gIiIi5UfFLyIiEkFU/CIiIhFExS8iIhJBVPwiIiIRRMUvIiISQcq1+I0xrxlj9hhjfizmeWOMmWSMWWeMWW2MaVue+URERMJdeY/4Xwe6l/D85UBz/6+hwIvlkElERCRilGvxW2u/AdJLeElv4N/WZzFQxRhTp3zSiYiIhL9QO8ZfD9ha5P42/2MiIiISBFFOBziGOc5jx91T2BgzFN/hABITE9udccYZZZlLRESCyFrwWovHa3/3Td63i7zvkaLP/ba7vO+xAo8lz+P9XWnYY25YIDffg8tlyMn3YC3ke71EuXxj3nyPN+h/rmAzQLTbhTHgLcgja98ObEEewD5rbY2Tec9QK/5twGlF7tcHdhzvhdbaqcBUgPbt29tly5aVfToRkTLi9VoO5xVgvb5C9P0Ca30Fl5ld4HudtUdLs/DXroO5RLuNv0hh24Es4qLdgO/rLb8VLUVuFz6eV+Bl/d7D1Kkcd/Rzj/2ctK0ZNKyaSIHXy66DOew7nEeNSrH+9/C91mLxevndYzkFHjbvz6J6UiwerxeP15KZUxCU/2aBFFi0//eEUryvy4DLGIwBYwx5BV6qJ8WQFBvFpv1ZnFY1nnPqVcFiqRQbTY1KsRR4LfVS4n/7Wn7/Hl6vJSHWTUpCDNWTYjHG9zlgcPlf4zJg8H1NXLSblIRooty+H1KWL19Ot27dqJ5SmTfeeIMePXpsLtV/rCJCrfhnA7cZY94FOgIHrbU7Hc4kIhWQtZZ8jyUjK488j5f0I3nke7wUeCz7DucdLcGjBVqkcNftOUxKYgx7MnPZeziXdbsPUz8lHk9hKXp9r1uyMZ3mNZOKvI+/XP2/53ss2zOyqRQXdfQxW+S1hY8VeCvGxdJ+3J75u/vbM7ID/tp9h3P/8FhctIucfC8NqyXg9peky5ijhekyhp93ZdK5abWjj/9WyoadB7OpUSmWVnWTiY1yHy1Qc0z5HsjK4/RalbBY6qckULNSLNFuF1EuQ2yUm8RY9+9KOpRYazHG0Lp1a0aMGMEdd9xBnTqntvTNlOfV+Ywx7wAXANWB3cDf8f9AZq19yfj+iz+Pb+V/FnCjtfaEQ3mN+EWcZf3l5fH/ysrzsO1AFtl5Hjz+6VyvtWw/kE1stBuv1/f6tbsPUSUhBo/Xy+ptB6lbOf535Yjvf0dHn4W3s3ILWLv7MA2qJlDg9bJq20GqJsSwKzMH8I2kKkiX/kFyXBQu128lVzgS3J2ZS9Maib6Cc/1Wfi7jG7FnZOXTqm4yUW7f127cd4S2DVJ8ZcZvhWiOjjD9j/HbqDbP46FB1YTfvXdhDozhSG4Bjaol4Ha5KPB4qZoYQ0yU6+j7FJYnFB3t+u6nJMQQ5TJEuVy4XJAQE4XbFVolG2qWLFnCHXfcwYwZM6hZs+bvnjPGLLfWtj+Z9y3XEb+1dvAJnrfAyHKKIxIWCku3wGM5mJ3/23HTY6aDt6RnAeDxcrSId2RkExPlwloo8Fq8Xsv/dmay42A2NSrF4fX+Vtr7j+SxIyObWslxRz/jpx2ZRLsN+R5nWrboiLOw9OG30o9yGQq8lpSEaOKj3ew5lEubBlWIcrlYv/cw7RqmEOV24TL4R5y+kvNYS2Z2PmfWSSYn30PVxFiqJvqmdAtHnm7Xb0VaJSG6SFEXlrW/MIHkuGjc7t/KsWgpmiJTvSpCAfB6vYwfP56HHnqIevXqsWvXrj8U/6kItal+kYiSlVdAZnYBGdl5bN6fRYHHUuD1ciingP2H88jzePD4p6DX7znM5z/voXpSLNFuw86DOSf+gDJw7OcWLf2YKN/0qdsYDuUWYAyc65+mdftHsVvSs2jXIAW32/e6XZk5nFOvMlFuFxlZeTSpkfi7kairSEGaIiPgAo8lPsZNlfhootwGt8tF9aQYEmOiSI6PPlqmIhXJrl27uP766/nss8+4+uqrmTp1KlWqVAnqZ6j4RY5h/aPZAq/lSG4BWXmeo9PY+R4v+w/ncTi3gB0Z2ezIyCYxNoq1uw8RH+NbTLV0UzqNqiUeHSl7vZDr8bJqawaNqiVQ4LVsOxD4sdFjHe9YaZTLV6y5BV5SEqJJiIk6Oh1cOLrMzfeS7/FyTv3KvyvijfuO0LahbxTs9r9PZnY+9VPiaVgt8ejrXMY3U1AlIYb4GDdu/3vERbuoXimW5Ljo46QVkdK49957WbhwIVOnTuWWW24pkx9ey/UYf1nRMX4pLWstGVn5pG3LYN3uw8REuUjbmsHyzQeOTomXp/op8Ww74Fuo1KlJNaJdhoRYNzWS4vyjWd/oODvfQ9MaSaQ2qEK025AcF02s/xiriFRMeXl5ZGRkULNmTfbt28eePXto2bJliV9TYY7xi5Slg1n57MrM4fOfd3Mop8A33ZyTz97DuazaepD4GDdRLsOGvUfIC+D83Zgo37HfoquOXS7D3kO51KkcR0yUi6qJMVRLjKVJjUQOHPGtHI6NdpHvsdROjjs66i48dlspLopE/6KmSnFRJMVGqbRFItj69esZPHgwbrebhQsXUr16dapXr16mn6nilwrD+heY7TqYw7YDWXy/8QCHcvJZuG4fO07heHdKQjRn1knmzDrJZOV5uKxVLS44vYYKWUTK1Ntvv83w4cNxu9288soruFzls5muil9CQoHHy67MHH7dfZhfdh9i7a5DxMW4eXvJFs6oXYmfdx0K+L0aVksg/Uge/dvVp1ZyHEmxUdSpHIcxUCs5jhi3i7hoN/WqxOPSKmoRKWdHjhzhtttu4/XXX6dLly689dZbNGzYsNw+X8UvZeZIbgHr9hzmpx2Z7PCfdrViywHqVI4np8DDis0HqFkpllXbDpb4Pscr/cbVE3EZqJYUS4+zanNWvcqcWSeZxFj9lRaR0GatZcmSJTz00EM8/PDDREWV7/ctfZeUU3IwK59fdh9i24Esth/I5oPl20q1OO54p6Q1qpZA69Oq0Lh6InUrxxPlNrQ+rQrVEmOoHB+tKXgRqXCstbz++usMHDiQpKQkVqxYQVxcnCNZVPwSEI/XsnpbBl+v3cviDftZvKGkqyv/0SVn1qJ+Sjw1KsWSV+BbLOcy/lPBkmKpXTmO+iml2U1bRKRi2LdvHzfddBMfffQRWVlZjBw50rHSBxW/lGBPZg7/mLOGOatPfLmERtUSqJcST5PqSZxTvzKXnFmLKgkanYtIZPvqq6+49tpr2bdvHxMnTmTEiBFOR1Lxi28KavP+LN5ZuoWdGTms33uYfYdz2Z35x41iqifFUis5lsvPqs0FLWrSqm6yyl1E5DheffVVhgwZQvPmzZkzZw5t2rRxOhKg4o84B47k8fnPe/jvT7vIzMkPaMr+qrb16N+2Pp2bVlPJi4gE6Pzzz2fo0KE888wzJCUlOR3nKBV/mLPWsml/FhM/W8vMtB0nfH2DqgkM6dqYPzWuSuX4aGonx6nsRUQCNGPGDD7++GNefvllmjVrxksvveR0pD9Q8YcZay3/XbObd7/fwpe/7C32dXUqx3H+6TXo0qw6DaslcFbdyjqnXUTkJGVnZzN27FhefPFF2rdvz8GDB4N+cZ1gUfGHgbwCL/N+3MmET9eyaX/xp9K1rl+ZF69rR90q8eWYTkQkvK1Zs4ZBgwbxww8/MHbsWMaNG0dMTIzTsYql4q9gCjxeZqXt4Ku1e/n2171kZOUf93XtGqZQOzmOQR1Oo0vT6hrNi4iUgfz8fK644gqOHDnC3Llzufzyy52OdEIq/gpgT2YOry3cxMc/7GBrevGXc60UF8U1HRsw6qLmJGkHOxGRMpOZmUliYiLR0dG8/fbbNGrUiDp16jgdKyBqhxC2ZMN+/r1oMx//8Mfz6C8/qzYt6yTT7fQaNK6RqGuhi4iUk8WLFzN48GBuvvlmHnzwQTp37ux0pFJR8YeYj1btYMJna9mw98gfnut2eg0ua1mLazs20Ep7EZFy5vV6GT9+PA8++CCnnXYal1xyidORToqKPwT8uvsQD8z8ke83/vGc+kqxUdzYpRFDz2+q6XsREYfs3LmT66+/ns8//5wBAwYwZcqUkF21fyJqEgftPZTLyLdW8P2m3xe+22V4+5aOtGmQQkxU+VyfWUREird582aWLl3Kyy+/zM0331yhZ11V/A44klvARc9+9bstcd0uw/WdGjLiwqbUrOTcxRtERMQnLy+PefPm0bt3bzp16sTmzZsr7Ci/KBV/Odp/OJc7P1j1h4117u7eghEXNHMolYiIHGvdunUMHjyYZcuWsXr1as4+++ywKH1Q8Ze5wp30hv1n+R+eu6ZjA/7RqxVRbk3ni4iEirfeeovhw4cTHR3Nhx9+yNlnn+10pKBS8ZehI7kFtP3np+QWeH/3+J8apfD6jR1I1GI9EZGQMnLkSF544QXOO+883nrrLRo0aOB0pKBT85SR95du5e7pq4/eP//0Gvylc0MuPrOWg6lERKQkbdu25eGHH+ahhx4iKio8KzI8/1QOe/Gr9Tw1/+ej9+/pfga3XtDUwUQiInI81lomTZpEtWrVuO6667j55pudjlTmdHA5yG59c/nvSv+TMd1U+iIiIWjfvn306tWLMWPGMHfuXKfjlBuN+IPEWsttb69k3o+7ALiwRQ2m/qU90Vq4JyIScr766iuuvfZa9u3bx6RJk7jtttucjlRuVPxBkFvgocWD84/eb1GrEv/vxg4OJhIRkeL8/PPPXHzxxTRv3pyPP/6Y1NRUpyOVKw1HT9HCdft+V/oXnVGTT/7WzcFEIiJyPNnZvqubnnHGGbzxxhssW7Ys4kofVPwnLSffw/lPf8m1ryw5+liHxlV57YY/OZhKRESO58MPP6Rx48YsX+7bU+W6664jKSnJ4VTOUPGfBK/X0u6fn7J5fxYA8dFu/jW4De8Pq1iXZhQRCXfZ2dmMGDGCfv36cdppp4XN7nunQsf4S8nrtfR8fgFH8jwA3H5xc+649HSHU4mIyLF++uknBg0axI8//sidd97J448/TkxMjNOxHKfiLwWP19LiwXkUeC0Aj/c9i2s7NnQ4lYiIHM8HH3zA7t27mTdvHt27d3c6TsjQVH8pDH558dHS/2fvVip9EZEQk5GRwcqVKwF48MEH+eGHH1T6x9CIP0CvLtjI9xvTAbjv8jO4vnMjZwOJiMjvLFq0iMGDB+P1elm3bh0xMTHUqqVt0o+lEX8A9h7K5Z9z1gAwrFsThp2vnfhEREKF1+vliSeeoGvXrhhj+OCDD3QsvwQa8Z/AzoPZdH7iCwDcLsM93c9wOJGIiBQ6dOgQV111FZ999hkDBw5kypQpVK5c2elYIU3FfwIDpiw6env2bV1wuYyDaUREpKikpCSqVq3KK6+8wk033YQx+h59IprqL8Gz//2Frem+nZ7G9T2bVnX1U6SIiNPy8vK4//772bRpE8YY3nvvPW6++WaVfoBU/MX4YdtB/vXFOgBu7NKIazo2cDiRiIisW7eOc889lyeeeILZs2c7HadC0lR/MYa/ufzo7YevbOlgEhERAXjzzTe59dZbiY6OZsaMGfTp08fpSBWSRvzHMWf1DrZn+Kb437ipg6aPREQc9uqrr3L99dfTpk0bVq1apdI/BRrxHyPf42X8/F8AaNugCuefXsPhRCIikcvj8eB2uxk4cCCHDh3itttuIypK1XUqNOI/xqy0HWxJzyLabfh/N3ZwOo6ISESy1jJx4kQ6dOhAVlYWSUlJjBkzRqUfBCr+Iqy13PnBKgBGX9ycyvHRDicSEYk8e/fupWfPnowZM4Z69eqRm5vrdKSwouIvYuh/flvQ17/daQ4mERGJTF9++SWtW7fm008/ZdKkScyaNYuUlBSnY4UVzZn4bdx3hE/X7Abgnu5nULtynMOJREQii7WWe++9l+TkZObOnUtqaqrTkcKSit/v8Y//B0CVhGhuvUB78YuIlJfNmzeTnJxMSkoK06dPJyUlhcTERKdjhS1N9eO7CM9n//ON9h/t1crhNCIikWP69OmkpqYyevRoAOrXr6/SL2MqfmDK1+uP3u7Vuq6DSUREIkN2djbDhw+nf//+NGvWjL///e9OR4oYKn5gZtoOAO76cwtt1iMiUsZ+/fVXOnTowJQpU7jzzjtZuHAhTZvqEGt5ifhj/N9vTGffYd+pIjd2aeRsGBGRCJCUlATAvHnz6N69u8NpIk/Ej/jHzfUt6qtRKZaEmIj/OUhEpExkZGTw2GOP4fF4qFOnDqtWrVLpOySii/9QTj5pWzMAeKrf2Q6nEREJT4sWLSI1NZVHH32U77//HgCXK6Lrx1Hl/l/eGNPdGPOLMWadMebe4zzfwBjzpTFmpTFmtTGmR1llefa/awGoWzmOC1vULKuPERGJSB6Ph3HjxtG1a1dcLhcLFiygc+fOTseKeOVa/MYYNzAZuBxoCQw2xhx7zdsHgfettW2AQcALZZVnycZ0AHqcXUeL+kREgmzIkCE88MADXH311axcuZKOHTs6HUko/8V9HYB11toNAMaYd4HewJoir7FAsv92ZWBHWQT5385M/rczE4Ch3ZqUxUeIiEQkay3GGIYMGcJ5553HjTfeqMFVCCnv4q8HbC1yfxtw7I+AjwD/NcaMAhKBS8oiyMerdwLQuHoiNZO1Pa+IyKnKzc3lvvvuA+C5556jc+fOmtoPQeV9jP94P/LZY+4PBl631tYHegD/Mcb8IacxZqgxZpkxZtnevXtLHaRwp74hXTXaFxE5Vb/++ivnnnsuEyZMID8/H2uP/dYuoaK8i38bUPSyd/X541T+zcD7ANbaRUAcUP3YN7LWTrXWtrfWtq9Ro0apQqRtzeDnXYeIi3bRp4126hMRORX/+c9/aNu2LRs3bmTGjBn861//0tR+CCvv4l8KNDfGNDbGxOBbvDf7mNdsAS4GMMacia/4Sz+kL8H05dsAOLdpdZ27LyJyCrZt28awYcNo06YNq1atok+fPk5HkhMo19az1hYYY24DPgHcwGvW2p+MMf8AlllrZwNjgZeNMX/DdxjgBhvkOaNp/uLv2bpOMN9WRCRibNy4kcaNG1O/fn2++eYbUlNTiYrSQKoiKPfz+K21c621p1trm1prH/c/9rC/9LHWrrHWdrHWtrbWplpr/xvMzz+SW0CB1wugc/dFRErJWsuECRNo0aIF77zzDgDt27dX6VcgEff/1Peb0sn3WJrVTKJKQozTcUREKoy9e/dyww03MHfuXHr27Mlll13mdCQ5CRG3Z+K3a/cBcG7Tag4nERGpOL766itat27NZ599xqRJk5g1axbVqun7aEUUcSP+r9buAeDCMzTNLyISqPT0dJKTk5k7dy6pqalOx5FTEFEj/twCDxv2HgGgdf0qDqcREQltmzdv5v333wfgqquuYvXq1Sr9MBBRxf/lz76zAmOjXFRN1PF9EZHiTJ8+ndTUVEaOHMmhQ4cAiInR981wEFHF/8JX6wDok1rP4SQiIqEpOzub4cOH079/f04//XSWLFlCpUqVnI4lQRRRx/j3ZOYC0KRGosNJRERCT25uLp06dWL16tXcfffd/POf/9QoPwxFTPF7vZZdmTkAnNf8DzsAi4hEvNjYWP7yl79w9tln61S9MBYxU/3bM7KP3j6jdnIJrxQRiRwHDhxg4MCBfP755wCMHTtWpR/mIqb4l25KB6B6Ugxuly4eISLy3XffkZqayocffsi6deucjiPlJGKKf+G6/QC0aZDicBIREWd5PB7GjRtHt27diIqKYuHChQwbNszpWFJOIqb41+89DEDnJtppSkQi24cffsgDDzzA1VdfzYoVK+jQoYPTkaQcRcTivpx8D2lbMwDo2bquw2lERJyxd+9eatSoQf/+/Zk3bx5//vOfMUaHPiNNRIz4P1i2FYDG1ROpUSnW4TQiIuUrNzeXMWPG0KJFC7Zs2YIxhu7du6v0I1REjPi/+sW3Y1/LOlrNLyKRZe3atQwaNIiVK1dy2223UbOmrlMS6SKi+LekZwFwWataDicRESk///nPf7j11luJjY1l5syZ9O7d2+lIEgLCvvhzCzz8use3sK9r8xoOpxERKT+fffYZ7dq148033+S0005zOo6EiLAv/p92ZB69rQvziEi4W758OXFxcbRq1YqXXnqJ6OhooqLC/lu9lELYL+7bdsC3Y1+zmkkOJxERKTter5fnnnuOzp07M3bsWADi4+NV+vIHYf83Yu7qnQA0q6HiF5HwtGfPHm688Ubmzp1L7969efXVV52OJCEs7It/wz7f8f3k+LD/o4pIBPr555+56KKLSE9P5/nnn2fEiBE6TU9KVKo2NMa0ALoC1YDXrbW7jTGnAfuttVllEfBUrd3tK/4/t6rtcBIRkeBr0qQJF154IXfffTetW7d2Oo5UAAEd4zfGRBtj/gOsAaYC44B6/qefBx4sm3jB07xmJacjiIgExaZNmxg4cCAHDhwgJiaGt956S6UvAQt0cd8/gV7AEKAhUHQeaS7w5yDnCooDR/KO3q6XEu9gEhGR4Jg2bRqpqanMnz+fH3/80ek4UgEFWvzXAg9Za18Ddhzz3AagcVBTBUnh8f0WtSrpUrwiUqFlZWUxbNgwrr76alq0aMHKlSvp2rWr07GkAgq0+GsAJf1oGReELEG3aL3vUrwtamuaX0QqtjvuuIOpU6dyzz33sGDBApo0aeJ0JKmgAl3ctxn4E/DFcZ5rD/watERBlJXnASBKo30RqYCstRw5coSkpCT+/ve/069fPy699FKnY0kFF2jxvwk8YIxZB3zkf8waYzoDd+Bb7BdyvvnVd3GeTk2rOZxERKR0Dhw4wJAhQ0hPT+fTTz+lTp061KlTx+lYEgYCnep/Avgc+ADY53/sS2AB8DXwf8GPdurW7zkCQJX4aIeTiIgEbuHChaSmpjJr1iwuv/xynZcvQRXQiN9aWwD0NcZcim8Ff01gPzDfWvtJGeY7JQVeLwBNtGufiFQAHo+HJ554gkceeYSGDRuycOFCOnTo4HQsCTMBFb8xpia+TXo+BT495jkXUN1au6cM8p2SfI8FoEqCRvwiEvoOHz7MK6+8woABA3jppZdITk52OpKEoUCP8e8EOgPfH+e5Nv7H3cEKFQy5BZ6jt1MSdFU+EQldX3zxBV26dKFy5cp8//331KhRQ9P7UmYCPcZf0t/AKMAbhCxBVXhVPkDn8ItISMrNzWXMmDFcfPHFTJw4EYCaNWuq9KVMFTviN8YkAUXnmaobY+oe87J44BpgdxlkOyXr9/g272nXMMXhJCIif7R27VoGDRrEypUrGTVqFLfffrvTkSRClDTVPxZ42H/b8ttpfMcywOPBDBUMv/qLv+iUv4hIKJg1axbXXnstsbGxzJo1i169ejkdSSJIScU/B9iFr9hfAMYDG495TS6wxlp7vGP/jtp2wHexwOpJsQ4nERH5vWbNmtG1a1defvll6tev73QciTDFFr+1djmwHMAYY4Hp1tp9xb0+1CTG+P5oDaomOJxERASWL1/OjBkzeOyxx2jVqhXz5s1zOpJEqIAW91lrp1Sk0gdI25oBaJ9+EXGW1+vlueeeo3Pnzrzxxhvs3bvX6UgS4QI9nQ9jzOnAjUAL/nhRHmutvSKYwU5VUpzvjxbtCvTEBRGR4NqzZw833HAD8+bNo0+fPrz66qtUrVrV6VgS4QLdwKcd8C2+1fsNgF+Aqvh28NsBbCmrgCdr8QbflflSG1RxOImIRCKPx8OFF17I+vXrmTx5MrfeeqtO05OQEOiI/0ngY2AwkAdcZ61dYYzpAbwC3FNG+U6Kx2vJyfdtLdCwmo7xi0j5yc/Px+1243a7efbZZ6lbty7nnHOO07FEjgp0Hrw18Dq/bdTjBrDWzsV3Zb7xQU92Cg5m5x+9HRsVUhsKikgY27RpE926dWPSpEkAdO/eXaUvISfQ4o8FDllrvUA6UKvIc2uAkPqbvWm/76p8zWrq4jwiUj4++OADUlNTWbNmDXXrHrvXmUjoCLT4NwCFf5N/Am4o8tx1QEhdoCf9cB4AGVn5J3iliMipycrKYujQoQwYMIAzzjiDtLQ0BgwY4HQskWIFWvzzgEv9t58Aehtj0o0xe4C/ApPKItzJysr37dbXRgv7RKSMrVixgtdee4177rmHb7/9lsaNGzsdSaREAS3us9beX+T2fGNMV6A/kADMt9bOLqN8J+Vglm/EXy1RV+UTkeCz1rJ06VI6dOjAeeedx9q1a2nSpInTsUQCclInuVtrF1tr77TWjgi10gf4YftBAKqq+EUkyNLT0+nXrx+dOnVi+fLlACp9qVBOeXcbY0xLY8w7wQgTLHsP5QKQVxByVwsWkQpswYIFpKam8tFHH/H000/Tpk0bpyOJlFqJxW98zjHGXGmMOfOY5842xnwArAauLMuQpZUQ6zuCUbvysRsMioicnKeeeorzzz+f6OhovvvuO8aOHYtLO4NKBVTs31pjTG1gIbASmAX8aIx5wxgTZYx53v/4lfiu3NesPMIG6kf/VH/DaokOJxGRcBEbG8ugQYNYuXIlf/rTn5yOI3LSSlrc9ySQCjwOrAAaA3cDXwOdgfeAu6y128o6ZGntzMgBID5am/eIyMmbM2cOXq+XXr16MXr0aABtuysVXknFfynwD2vtk4UPGGN+BD4BXrLWjijrcCcrz+M7tl8/Jd7hJCJSEeXm5nLPPfcwceJELrjgAnr27KnCl7BR0gGqmvim+osqvB9Si/mKstYevV0rWcf4RaR01q5dS+fOnZk4cSK333478+bNU+lLWClpxO8Gco95rPD+kbKJc+qy/Zv3AMTHaKpfRAK3adMm2rZtS1xcHLNnz6Znz55ORxIJuhNt4Px+KxcAACAASURBVHOZMabowj0XYIHuxpgzir7QWvt2sMOdjPQjvs179AO6iATK6/Xicrlo1KgRf//737nmmmuoV6+e07FEysSJiv8fxTz+2DH3LRASxb8707ewr0p8tMNJRKQiWLZsGTfddBPvvPMOrVq14q677nI6kkiZKqn4zyzhuZCVk+9b2GdP8DoRiWxer5cJEyZw3333Ubt2bQ4fPux0JJFyUWzxW2t/KYsPNMZ0BybiW0PwStGzBoq8ZgDwCL7+XmWtvSbQ99+RkQ1A2wYpwYgrImFoz549/PWvf2X+/Pn06dOHV199lapVqzodS6RcBHSRnmAxxriByfhOFdwGLDXGzLbWrinymubAfUAXa+0BY0zN0nxG4TH+/YePXZcoIuLz/PPP8+WXXzJ58mRuvfVWrdqXiFLe+012ANZZazdYa/OAd4Hex7xmCDDZWnsAwFq7pzQfULg/f/2UhFNPKyJhIz8/nw0bNgDwwAMPsGLFCkaMGKHSl4hT3sVfD9ha5P42/2NFnQ6cboxZaIxZ7D808AfGmKHGmGXGmGV79+49+ni+f/OeBtVU/CLis3HjRrp168ZFF11EVlYWsbGxtGzZ0ulYIo4o7+I/3o/Wx67DiwKaAxcAg4FXjDFV/vBF1k611ra31ravUaPG0ccP5/rO409J0Kp+EYH333+f1NRU1qxZw/jx40lI0KBAIlt5F/824LQi9+sDO47zmlnW2nxr7UbgF3w/CAQkMycfgEpxKn6RSJaTk8OQIUMYOHAgLVu2JC0tjQEDBjgdS8RxpS5+Y0wzY0xHY8zJ/Ni8FGhujGlsjIkBBgGzj3nNTOBC/2dVxzf1vyHQDyg8jz8xtlzXLYpIiImOjmbTpk3cd999fPPNNzRu3NjpSCIhIeDiN8bcbIzZhm8E/h1whv/xacaY4YG8h7W2ALgN34V+/ge8b639yRjzD2NML//LPgH2G2PWAF/iuwLg/kBzFp7OF+3Sgh2RSGOt5eWXX2bHjh243W7mzZvHuHHjiI7WDKBIoYCK3xhzAzAV+AL4K78/Vr8EGBjoB1pr51prT7fWNrXWPu5/7GFr7Wz/bWutvcNa29Jae7a19t1A3xtgu7/4qyTElObLRKSCS09Pp1+/fgwdOpQXX3wRgKgozfyJHCvQEf9dwERr7V/445X5/od/9B8KKvu36q2aqOIXiRQLFiwgNTWVOXPm8Mwzz/Doo486HUkkZAX643BT4ONinjsEhMw2ebszfRv3VEtS8YtEgg8//JCrr76axo0b891339G+fXunI4mEtEBH/On8fjV+UacDO4MT59RYa3H7j+1X1kV6RCLChRdeyO23386KFStU+iIBCLT4PwYeNMYULX/rP79+DDAr6MlOQr7H4vFaolyGaHd5n6koIuXlo48+onv37uTl5ZGSksKECRNITk52OpZIhRBoOz7gf+0aYA6+TXee8d+PBkLigFpWXgEAXqtr84mEo9zcXEaPHk2vXr3YvXs3+/btczqSSIUTUPH798tvC0wCagDbgarAG0DHwn31nXYop7D4HQ4iIkH3yy+/0KlTJyZNmsTo0aNZvHgxdevWdTqWSIUT8Lku1toMfCP/B8ouzqnZ578iX+3kOIeTiEgwWWu54YYb2Lp1Kx999BFXXnml05FEKqyAit8YMw74t7X25zLOc0oKz+GPidLxfZFwkJmZicvlIikpiddff52kpCTq1Tv2ul4iUhqBNuQo4Cf/1fBGGWNqnPArHLD9gK/446PdDicRkVO1dOlS2rZty6hRowBo0aKFSl8kCAIt/prAX4C9wHPAdmPMHGPM1caY2DJLV0qFK/lrVAqZSCJSSl6vl2eeeYZzzz2X/Px8brnlFqcjiYSVQBf3ZVtr37LWXo7vinr3AnWA94DdxpiXyzBjwHILvAC0qqfTekQqoj179tCjRw/uuusuevXqRVpaGl26dHE6lkhYKfXBcGvtbmvtc9badsDF+HbuuynoyU5Ctv90vtgoTfWLVERZWVmsXr2aF198kWnTppGSEjKbgoqEjVJfwcI/td8HuA64DN8Fe4rbzrdcZWTnA5AUq+IXqSjy8/N56623+Otf/0qjRo1Yv3498fHxTscSCVuluSzvBcaYV4Hd+C7UUwu4E6hrre1V4heXk1VbMwColqhj/CIVwcaNG+natSs33ngjn3/+OYBKX6SMBXo63xagHrAVmIzv1L5fyjLYyaiW5Ct8j3bwEQl57733HkOHDsUYw3vvvccll1zidCSRiBDoVP+n+Mr+67IMc6pWb/ON+OunaMQgEsruv/9+nnjiCTp16sQ777xDo0aNnI4kEjECKn5r7c1lHSQYqifFsu9wHtHawEckpBWO7h999FGio3UlTZHyVGzxG2M6AD9aa7P8t0tkrf0+qMlOQp7/dL6UBH0jEQkl1lpeeOEFDhw4wIMPPshFF13ERRdd5HQskYhU0oh/MdAJ+N5/u7gD58b/nONL6bPzPQDEaec+kZCRnp7OzTffzMyZM7niiivweDy43fo3KuKUkor/cuB//ts9KL74Q8aRXN95/EmxpT5LUUTKwIIFC7jmmmvYtWsXzz77LGPGjMHl0qE4EScV25DW2k+K3J5fPnFOTZ7HN9Wvi/SIOG/v3r1cdtll1K1bl++++4727ds7HUlECPA8fmPMGmPM2cU819IYsya4sU5OTr6/+N0qfhGnHDx4EIAaNWrw4YcfsmLFCpW+SAgJtCHPAIo7Ry4BaBGcOCevwD/adxmIUvGLOGL27Nk0bdqU6dOnA9C9e3eSk3XtDJFQUpqGLO4Y/znAwSBkOSVHcn0L+7R3j0j5y8nJ4fbbb6d37940aNCAs88+7gShiISAkk7nGwWM8t+1wDRjTO4xL4sH6gLTyiZe4ApX9NfUJXlFytXPP//MoEGDWLVqFWPGjOHJJ58kNlb/DkVCVUnL33cAy/23mwG/APuPeU0usAZ4MfjRSiczx3eBHq/VkF+kPC1dupTt27czZ84crrjiCqfjiMgJlLSqfzowHcAYA/CAtXZDOeUqtcLNe7LyPA4nEQl/mZmZLFu2jIsuuojrr7+eK6+8UpfQFakgAt2yd3BZBzlVB7LyADizjhYSiZSlpUuXMmjQIPbu3cvmzZtJSUlR6YtUICUd478b34V5dvlvl8Raa58ObrTSKRzp7zqY42QMkbDl9Xp59tlnuf/++6lbty7z5s1T4YtUQCWN+J8EvgJ2+W+XxAKOFv/BLN8x/pZ1NeIXCbaCggJ69uzJ/Pnzueqqq3jllVdU+iIVVEnFH2+tLVzFH/LXuS2c6ncZh4OIhKGoqCjatGlD7969GTZsWOG6HxGpgEpa3Jd7vNuhyu1vfF2gRyQ48vPzeeihh+jduzedO3dm3LhxTkcSkSAIaHGfMaYJkGytTfPfjwXuBc4CPrHWvlJ2EQNTeIy/fkrIT06IhLwNGzYwePBgvv/+e2JjY+ncubPTkUQkSAK9jN0L+M7XT/Pf/yfwN2At0NcY47bWTimDfAHLzPYd468UF+1kDJEK77333mPo0KEYY3j//fe5+uqrnY4kIkEU6Ja9qcA3AMZ3cO8G4H5rbSt8C/+Gl0m6UjiU47skb7KKX+SkzZkzh0GDBtGqVSvS0tJU+iJhKNDirwLs899OBaoB7/vvfwo0DXKuUjuc6yv+SnGBTmKISKGcHN9psJdffjlTp07l66+/plGjRs6GEpEyEWjx7wGa+G9fCmy01m72308EHN8uL7fAFyE2SlfmEwmUtZbJkydz+umns3PnTtxuN0OGDCE6WjNnIuEq0OHxHOBxY8zpwFDgtSLPtQI2BjtYaW3P8I1YYlT8IgFJT0/n5ptvZubMmfTo0YOoKM2WiUSCQP+l3wtUAgYCnwGPFXluAPBFkHOV2hH/VL+InNi3337LNddcw+7du3nuuecYPXo0Lpd+aBaJBIHu1Z8JXF/Mc38KaqKTlBzv+6NUjtcUpciJTJ48mbi4OBYtWkS7du2cjiMi5ahUc3vGmEpAB6Aqvkv0LrXWHiqLYKW1dvdhABJiNF0pcjzbtm0jPz+fxo0bM2XKFFwuF5UqVXI6loiUs4Dn9owxDwI7gf8C7+Gb8t9pjHmgjLKVSqzb90eJdmsrUZFjzZo1i9atW3PTTTcBULlyZZW+SIQKqPiNMSOBfwAzgB5AG+By//1/GGNuLbOEASpc1Jek0/lEjsrJyWHUqFH06dOHhg0bMmWKo/tsiUgICLQlbwNesNbeVuSxVcAnxpiDwCjgxWCHK439R3wX6YnXXv0iAGzdupWePXuyatUqxowZw5NPPklsbKzTsUTEYYEWfxPg9mKemwXcEpw4p04X6RHxqVatGikpKcyZM4crrrjC6TgiEiICPcafDrQo5rkW/udDQrRbpyRJ5MrMzOSuu+7i8OHDJCQk8MUXX6j0ReR3Am3Jmfg28LnaFLkQtzGmL74L9swsi3CBstb3e4xKXyLY999/T5s2bZgwYQJffvklAEX+uYqIAIEX/73Az/hW82cZYzYbY7KAacAv/ucdY/E1f5RW9EsE8nq9PP3003Tp0oWCggK++eYbevbs6XQsEQlRgW7gc9AYcy7QF+iK7zz+dOBrYJa11tG9+gtH/FEuFb9Envvuu4/x48fTr18/Xn75ZVJSUpyOJCIhLOBz3/zlPs3/K6QcLX5N9UsE8Xg8uN1ubr31Vpo1a8Ytt9yiqX0ROaESm9IYM8gYs9gYs88Ys84Y87gxJuROlC+c6k/3n9InEs7y8vK455576Nu3L9ZaGjVqxJAhQ1T6IhKQYovfGHM18DZQG1gIZOE7lv9YcV/jlMIR/2lV450NIlLGNmzYQNeuXRk/fjx169YlPz/f6UgiUsGUNOK/A/gYaG6t7W2tPQd4ChhljAmpOXV/7xOtq4tJGHv33Xdp06YNa9euZdq0abz00kvExMQ4HUtEKpiSmrIF8KK1tuiQYhIQDzQs01SlZK1W9Ut4O3ToEHfccQdnnXUWaWlp9OvXz+lIIlJBlXS8vgqw75jH9vp/TwE2lkmik/Dbqn6N+CW8/PzzzzRr1oxKlSrx9ddf07hxY6KiQm6ZjYhUICdqSlvKxx3h8Tf//iO5DicRCQ5rLc8//zypqak8/fTTADRv3lylLyKn7ETFv9AYk1f4C8j2P76k6OPGmIAb1xjT3Rjzi/8sgWI3/jHG9DfGWGNM+xO+p//3pFh9U5SKb//+/fTt25dRo0Zx8cUXc8stIXMpDBEJAyU15VPB/jBjjBuYDFwKbAOWGmNmW2vXHPO6SvguCrQkkPctnH6oXTkuiGlFyt+iRYsYMGAAu3fv5rnnnmPMmDE6TU9EgqrY4rfW3lcGn9cBWGet3QBgjHkX6A2sOeZ1/wTGA3cG8qY6xi/hIiYmhsqVKzNz5kzatWvndBwRCUPl3ZT1gK1F7m/zP3aUMaYNcJq1dk5Jb2SMGWqMWWaMWXbgYCYA0VrVLxXQtm3b+Ne//gVAu3btWL16tUpfRMpMeRf/8Zr56EJB//4AE4CxJ3oja+1Ua217a2375EqVANi470iwcoqUi1mzZtG6dWvuv/9+tm/fDoBLM1ciUobK+zvMNuC0IvfrAzuK3K8EnAV8ZYzZBHQCZp94gZ/vZ4dWdSsHL6lIGcrJyWHUqFH06dOHRo0asWLFCurVq3fiLxQROUXlvQx+KdDcGNMY2A4MAq4pfNJaexCoXnjfGPMVcKe1dllJb1p4jD82SiMlCX3WWi655BIWLlzImDFjePLJJ4mNjXU6lohEiHJtSmttAXAb8AnwP+B9a+1Pxph/GGN6nfT7+n/X1fkklFlrsdZijGHUqFHMmTOHCRMmqPRFpFyV+4nv1tq5wNxjHnu4mNdeEMh75hZ4iQZitLhPQtTBgwcZNmwYF198MUOGDGHgwIFORxKRCBXwENkYU8sYM84Ys8AYs8YY09L/+IhANtkpS1EuX+Fvz8g+wStFyt+SJUto06YN06ZNIzMz0+k4IhLhAip+Y8wZwA/Arfguz9sCKNwtpwUwpkzSBahwqv/0WpWcjCHyO16vl6eeeorzzjsPr9fLt99+y9ixJzxhRUSkTAU64n8G30V5GgM9+P1peQuBzkHOVSqFV+eL1jF+CSGLFy/m3nvvpW/fvqSlpdG5s6P/TEREgMCP8Z8PXGetzfBvu1vULqBOcGOVTuGIP0ar+iUEbNmyhQYNGnDuuefy3Xff0alTJ227KyIhozRN6Snm8Wr8dvEeR+Tm+6K59M1VHJSXl8fdd99Ns2bNWLbMdwZq586dVfoiElICHfEvA64HjreNbj9gcdASnYTCKf6MrDwnY0gE27BhA4MGDWLp0qUMHz6cVq1aOR1JROS4Ai3+x4H5xpiPgLfwza53M8YMAwYAF5ZRvoAUTvXXr5rgZAyJUO+++y5Dhw7F7XYzbdo0+vXr53QkEZFiBVT81trPjDEDgP8DrvA//By+7XYHWGsXllG+gBTu3Bft0pSqlL9169Zx9tln8/bbb9OwYUOn44iIlMgUrogP6MW+g5WtgJrAfuAHa623jLIFrFbTljb+6qd55urW9G9X3+k4EgFWrVrFgQMHuOCCC/B4PFhriYoq9/2wRCRCGWOWW2tPag+dUn2nsr6fEn48mQ8qSzn5XuLRZXml7FlrmTx5MmPHjqVly5asWLECt/vYE11EREJXQMXvn+YvkbX2/VOPc3IKt+o9klvciQcip27//v3cdNNNzJ49mx49evD6669rxb6IVDiBjvjfLebxoscJHCv+whC1knWxEykbO3bsoEOHDuzZs4cJEyYwevRolb6IVEiBFv+Zx3msGnAl0B/4a9ASnYTCZQq6Op+UlTp16jBw4ECuvfZa2rZt63QcEZGTFuiq/l+Keeo7Y4wH3x7+i4KWqpSOrurXMX4Joq1btzJ8+HAmTpxIs2bNePbZZ52OJCJyyoIxRP4S6BWE9zlp2f6d+7RXvwTLzJkzad26Nd988w2//FLcz70iIhVPMJqyPb4r9jmmcI9+rzfwUxNFjicnJ4eRI0fSt29fmjRpwooVK7jiiitO/IUiIhVEoKv67z7OwzHAWUBf4OVghjpZSXE6j1pOzfjx43nhhRe44447eOKJJ4iJiXE6kohIUAXalE8e5zEPsB2YADwatEQnoXAToiiXpvql9Ky1pKenU61aNe688066dOnCxRdf7HQsEZEyEWjxxx/nsfxQ2LWvKLe27JVSOnjwIMOGDWPlypWsWLGCxMRElb6IhLUTDpGNMTHAI8BZ1trcIr9CpvSPns6n4pdSWLJkCW3atGHatGnccMMNxMXFOR1JRKTMnbD4rbV5wGggsezjnJx8j+9nkCidzicB8Hq9PPXUU5x33nl4vV6+/fZb7rvvPm29KyIRIdCD4quAlmUZ5FQUruV3aSc1CYDH42HWrFn07duXtLQ0Onfu7HQkEZFyE+gx/ruBfxtj1llrPyvLQCej8Nh+4Wl9Isfz6aef0qZNG6pXr878+fOpVKmStt0VkYgTaFO+BlQBPjHGHDLG/GqMWVvkl7M7nOgYv5QgLy+Pu+66i8suu4zHHnsMgOTkZJW+iESkQEf8y/n9BXlCytGpfhW/HGP9+vUMHjyYpUuXMnz4cJ544gmnI4mIOCrQvfoHlXWQYHBrBCdFfPHFF/Tp0we32820adPo16+f05FERBxX7FS/MWaDMaZ1eYY5WYUb+Og8finqrLPO4tJLLyUtLU2lLyLiV9Ix/kZAhbjAfeFUv4pf0tLSuPHGGykoKKBmzZpMnz6dhg0bOh1LRCRkhNUyeE31Ry5rLZMmTaJjx47897//ZdOmTU5HEhEJSScq/pBd0Hc8WtwXmfbt20fv3r0ZPXo0l112GatWraJZs2ZOxxIRCUknWtz3qDFmXwDvY621fw1GIJHS6t+/P4sWLWLixImMGjVKp+mJiJTgRMWfCuQG8D6OzwwkxGi71UhSUFCAx+MhNjaWCRMmANCmTRuHU4mIhL4TFX8fa+335ZLkFOn4fuTYunUr11xzDeeccw6TJ09W4YuIlELYLO5z6wI9EWHmzJm0bt2atLQ0zj33XKfjiIhUOGFT/NquN7xlZ2czcuRI+vbtS5MmTVi5ciXXXnut07FERCqcsCl+ncMf3rZv386///1vxo4dy3fffadV+yIiJ6nYY/zW2gr1Q4GO8Ycfay2ff/45F198Mc2aNWPdunXUqlXL6VgiIhVahSr3kuw4mON0BAmigwcPMnjwYC699FLmzJkDoNIXEQmCQK/OF/Ka1kh0OoIEyeLFixk8eDBbt25l3LhxXHHFFU5HEhEJG2Ez4o9yhc0fJaJNnjyZrl27Yq3l22+/5b777sOl/29FRIImbL6janFfeGjYsCFXXXUVaWlpdO7c2ek4IiJhJ2yKP0rn8VdY8+fPZ/LkyQBceeWVvPfee1SpUsXhVCIi4Slsil8j/oonLy+PO++8k8svv5xXX32V/Px8pyOJiIS9sCn+g1kqjYpk3bp1dOnShWeffZYRI0awcOFCoqOjnY4lIhL2wmZVv1QcGRkZdOjQAWstH374IX379nU6kohIxAib4q9bJd7pCHICBQUFREVFUaVKFSZNmkS3bt1o0KCB07FERCJK2Ez1u3SMP6SlpaVx9tlnM3/+fACuu+46lb6IiAPCpvi1qD80WWuZNGkSHTt2JDMzk4SEBKcjiYhEtLApfpf26g85+/bto3fv3owePZo///nPrFq1im7dujkdS0QkooVP8WuqP+TMnj2bTz75hIkTJzJr1iyqV6/udCQRkYgXNov7dHW+0FBQUMBPP/1E69atufHGG+nWrZsuoSsiEkLCZsSvDXyct2XLFi688EK6du3Knj17MMao9EVEQkzYFL+m+p01Y8YMUlNTWbVqFS+++CI1a9Z0OpKIiBxH2BT/xn2HnY4QkbxeLyNGjOCqq66iadOmrFy5kmuvvdbpWCIiUoywKf6WdZKdjhCRXC4XBQUFjB07loULF9K0aVOnI4mISAnCZnFflDtsfoYJedZaXnnlFdq3b0+bNm2YMmUKRosrRUQqhHJvS2NMd2PML8aYdcaYe4/z/B3GmDXGmNXGmM+NMQ0Ded9oHeMvFxkZGQwcOJChQ4cyZcoUAJW+iEgFUq7Fb4xxA5OBy4GWwGBjTMtjXrYSaG+tPQeYBowP5L3dLo34y9qiRYtITU1lxowZPPnkk7zwwgtORxIRkVIq76n+DsA6a+0GAGPMu0BvYE3hC6y1XxZ5/WLgukDeOEp79papr776iksuuYTTTjuNb7/9lk6dOjkdSURETkJ5D5PrAVuL3N/mf6w4NwPzAnnjHRnZpxBLimOtBaBLly488MADrFy5UqUvIlKBlXfxH29Ybo/7QmOuA9oDTxfz/FBjzDJjzDKAZjWTghZSfObNm0e7du3Yv38/0dHRPProo1SpUsXpWCIicgrKu/i3AacVuV8f2HHsi4wxlwAPAL2stbnHeyNr7VRrbXtrbXuAKC3uC5q8vDzGjh1Ljx49KCgoICMjw+lIIiISJOVd/EuB5saYxsaYGGAQMLvoC4wxbYAp+Ep/T6BvrMV9wbFu3Tq6dOnCc889x4gRI1iyZInOzRcRCSPlurjPWltgjLkN+ARwA69Za38yxvwDWGatnY1vaj8J+MB/mtgWa22vE723RvzBcf/997N+/Xo+/PBD+vbt63QcEREJsnLfwMdaOxeYe8xjDxe5fcnJvK8u0nPyDh8+zOHDh6lduzbPP/88OTk5NGjQwOlYIiJSBsJmfjzf43U6QoW0cuVK2rVrx6BBg7DWUrNmTZW+iEgYU/FHKGstEydOpFOnThw5coRHH31UO/CJiESAsNmrv1pSrNMRKoz09HRuuOEGPvroI3r27Mlrr71G9erVnY4lIiLlIGxG/DrEH7ioqCjWr1/PpEmTmDVrlkpfRCSChM2IX9PUJSsoKGDy5MkMGzaM5ORk0tLSiI6OdjqWiIiUszAa8av4i7N582bOP/98xowZw7Rp0wBU+iIiESqMit/pBKFp+vTppKam8sMPP/D2229z3XUBXfNIRETCVNgUvwb8f/T000/Tv39/mjdvzsqVKxk8eLDTkURExGFhc4xfU/1/1KtXLw4cOMAjjzxCTEyM03FERCQEhNGIX8VvrWXq1KncdNNNWGtp0aIF48aNU+mLiMhRYVP8kX6MPyMjg4EDBzJs2DC2bt1Kdna205FERCQEhVHxR27zL1q0iNTUVGbMmMGTTz7JJ598QkJCgtOxREQkBIXRMX6nEzgjOzubvn37kpCQwIIFC+jYsaPTkUREJISFTfFH2jH+vXv3Uq1aNeLj45k9ezYtWrSgcuXKTscSEZEQFzZT/ZFk3rx5tGrVimeffRaADh06qPRFRCQgYVP8OfkepyOUuby8PMaOHUuPHj2oU6cOV155pdORRESkggmbqf7kuPDegnbdunUMGjSI5cuXM3LkSJ555hni4uKcjiUiIhVM2BQ/YX6If+fOnWzZsoUZM2bQp08fp+OIiEgFFTZT/eHY+4cPH+b9998HoGvXrmzcuFGlLyIipyRsij/czuNfsWIFbdu25ZprrmHDhg0AJCYmOpxKREQqurAp/nDpfWst//d//0enTp3Iysri888/p0mTJk7HEhGRMBE2x/hNGEz2W2sZMGAA06ZNo1evXrz22mtUq1bN6VgiIhJGwqf4K37vY4zhsssu4/zzz2fkyJERtymRiIiUvfApfqcDnKSCggIeeeQRWrVqxeDBgxkyZIjTkUREJIyFzTH+itj8mzdv5vzzz+fxxx9n8eLFTscREZEIEDYj/oq2qn/69OnccssteDwe3n77bQYPHux0JBERiQBhM+KvSLW/fPly+vfvT/PmzVm5cqVKX0REyk34FH8FGPEfOnQIgHbt2vHBBx+wYMECmjZt6nAqERGJJGFU/E4nKJ61lqlTp9KwYUNWrVoFQP/+/YmJxlbJagAAEIFJREFUiXE4mYiIRJrwKX6nAxQjIyODAQMGMGzYMNq3b0+tWrWcjiQiIhEsfIo/BJt/0aJFpKamMnPmTJ566inmz59P7dq1nY4lIiIRLGxW9YfiMf7Zs2fjcrlYsGABHTt2dDqOiIgIxlrrdIZTFlunuV285HvaNEhxOgo7duxg586dtGvXjvz8fLKysqhcubLTsUREJIwYY5Zba9ufzNeG0VS/8yP+jz/+mNatW3PNNdfg8XiIjo5W6YuISEgJn+J38LNzc3P529/+xpVXXkndunWZNWsWbrfbwUQiIiLHF0bH+J353P3793PZZZexYsUKRo0axfjx44mLi3MmjIiIyAmE0YjfmeZPSUnhzDPPZObMmUyaNEmlLyIiIS1sir88HTp0iBEjRrB161ZcLhdvvvkmvXv3djqWiIjICan4S2n58uW0bduWKVOm8OWXXzodR0REpFTCpvhjosr2j2KtZcKECXTu3Pn/t3f30VJV5x3Hvz9BxSAgylssUWKRWGKJUrSYUojVZZBWqUarJOILgjUJrhBdaXFhlSi1Fd9Tk4g2iaIFlRgLxaBWQ3xJxUQDUjAxCxWjorEi4hsvAk//2PvqMN47d+5lZi537u+z1lkzc84+5zyzZ9Y8c/bZZx82bNjA4sWLOf3006u6TzMzs0qrm8Rf7c5911xzDeeffz5jxozh6aefZuTIkdXdoZmZWRXUTa/+XaqU+Ddv3sxuu+3GpEmT6NmzJ2edddZOMWaAmZlZa9TNEX+lr+TfsmUL06ZNY/jw4WzcuJHu3bszYcIEJ30zM2vX6ibxVzIfv/jii4waNYrLL7+cQw89lG3btlVu42ZmZm2obpr6K5X37777biZOnMjWrVuZM2cO48aNq9CWzczM2l79JP4KHPJv2bKFGTNmMGjQIObOncsBBxxQgcjMzMx2HnWT+Hekc9/KlSvp378/PXr04N5776V3797suuuulQvOzMxsJ1E/5/hb0dgfEcyaNYthw4YxdepUAPbdd18nfTMzq1v1k/hbmPfXrVvHySefzLnnnsvIkSOZPn16VeIyMzPbmdRN4m+JpUuXcsghhzB//nxmzpzJokWL6Nu3b1uHZWZmVnX1c46/BSf5e/fuTb9+/Zg3bx6HH354FaMyMzPbudTNEX9zaX/NmjVMmzaNbdu20b9/f5YsWeKkb2ZmHU79JP4SmX/hwoUMGTKE6667jhUrVuTyHoHPzMw6nvpJ/I0c82/atIkpU6Zw3HHH0b9/f5566imGDBnSBtGZmZntHOrmHH9jB/CnnHIK8+fP57zzzmPmzJl06dKl9oGZmZntRBQRbR3DDtv9kwfGS7/7X/p0S4l927Zt7LLLLjz22GOsXbuWsWPHtnGEZmZmlSPpqYgY1pp1a97UL2m0pGclrZI0tZHlu0u6My9/QtKAsraLeOeddxg/fjwXXnghACNGjHDSNzMzK1DTxC+pE/Bd4FhgMDBO0uCiYmcD6yJiIHAtcEU5216+7NcMHTqUOXPm0LVr10qGbWZmVjdqfY7/cGBVRDwPIOkOYCzwTEGZscD0/PzHwA2SFCXOSWx97y2OPWoUffv2ZfHixYwcObI60ZuZmbVztW7q/yPgpYLXL+d5jZaJiC3AemCfUhvd+s5ajv7iaJYtW+akb2ZmVkKtj/gbu3i++Ei+nDJIOgc4J7/ctGjhf63o1avXDoZnJfQC3mjrIDoA13P1uY6rz3VcfZ9p7Yq1TvwvA58qeN0fWNNEmZcldQZ6AG8WbygibgJuApD0ZGt7N1p5XMe14XquPtdx9bmOq0/Sk61dt9ZN/b8CDpT0aUm7AacCC4rKLADOyM9PAn5W6vy+mZmZla+mR/wRsUXSZOB+oBPww4hYKelS4MmIWAD8ALhN0irSkf6ptYzRzMysntV85L6I+Cnw06J5Fxc83wic3MLN3lSB0Kw013FtuJ6rz3Vcfa7j6mt1HdfFyH1mZmZWnrq5SY+ZmZk1r10l/moN92sfKaOOz5f0jKTlkh6StH9bxNmeNVfHBeVOkhSS3Du6FcqpZ0l/l7/PKyXNqXWM7V0Zvxf7SVosaWn+zRjTFnG2Z5J+KOl1SSuaWC5J38mfwXJJQ5vdaES0i4nUGfA54ABgN+BpYHBRma8BN+bnpwJ3tnXc7Wkqs46PBD6Rn3/VdVz5Os7lugGPAEuAYW0dd3ubyvwuHwgsBXrm133aOu72NJVZxzcBX83PBwOr2zru9jYBI4GhwIomlo8BFpHGwBkOPNHcNtvTEf+Hw/1GxGagYbjfQmOBW/PzHwNHSY3dsNea0GwdR8TiiHg/v1xCGovBylfO9xjgMmAmsLGWwdWRcup5EvDdiFgHEBGv1zjG9q6cOg6ge37eg4+P22LNiIhHaGQsmwJjgdmRLAH2kvTJUttsT4m/KsP92nbKqeNCZ5P+aVr5mq1jSYcCn4qIhbUMrM6U810eBAyS9AtJSySNrll09aGcOp4OnCbpZdLVXOfVJrQOpaW/27W/nG8HVGy4X2tS2fUn6TRgGDCqqhHVn5J1LGkX0l0pz6xVQHWqnO9yZ1Jz/xdILVePSjo4It6qcmz1opw6HgfcEhFXSzqCNEbLwRGxrfrhdRgtznvt6Yi/JcP9Umq4X2tSOXWMpKOBacDxEbGpRrHVi+bquBtwMPBzSatJ5+wWuINfi5X7ezE/Ij6IiBeAZ0l/BKw85dTx2cBdABHxONCFNI6/VU5Zv9uF2lPi93C/1ddsHedm6FmkpO9zoi1Xso4jYn1E9IqIARExgNSP4viIaPW43B1UOb8X/0nqrIqkXqSm/+drGmX7Vk4d/x44CkDSn5AS///VNMr6twA4PffuHw6sj4hXS63Qbpr6w8P9Vl2ZdXwlsCcwL/eb/H1EHN9mQbczZdax7aAy6/l+4BhJzwBbgW9FxNq2i7p9KbOOLwBulvRNUvPzmT4YaxlJc0mno3rlvhKXALsCRMSNpL4TY4BVwPvAWc1u05+BmZlZx9GemvrNzMxsBznxm5mZdSBO/GZmZh2IE7+ZmVkH4sRvZmbWgTjxmzVC0pn5zniNTUe3cFsT83o1ua+BpBlF8a7Ld6us+OWtkjrnfVxUMO9ESVMaKXt0Ljui0nGUiG9gUV1slfSqpNsklRzWtMQ2h0qaLmmvSsdrVgvt5jp+szZyMmlkrELPtEUgrXBEftwH+HtgrqTdImJ2pXaQr+U+gu3HCj8RGAFcV1T8lzmmlZXafwvMAO4Fds8xXAwcJOmIfF+PlhhKupb6FsDD+1q748RvVtqyiFjV1kG0Rr5TFwCSHiANSTsFqFjiL95PM+XeJo1E2BaeK4jzYUm7k24gcwjgURGtQ3FTv1krSdpD0vWSVkp6LzchL5D0mTLWHS9pWV5vvaTlkiYWlTlS0s8kvZunRZIGtybWiPgAWAYMLNh+D0nfy3FvlvSspG8UxdBd0g2SXpK0SdIfJP23pEF5+XZN/ZJuB74C7F/QvL4qL9uuqV/STZLWSOpUtM8uuU6uKpjXR9KsXH6zpN9IOrs1dZH9Oj/uV7TvGZKWSnpb0huSHpJ0eMHyicDN+eULBe+xf0F9TMt1uUnSK5KuzH80zHYKPuI3K62T0g2fGkREbM3P98jTpcBrpCb1rwOPSzqoqXsZSBoF3EpqCr+ANNzpYKBnQZmxwN2kcbi/TPqTPpV0B7khEfFKK97Lp8lN0znZLgKGAP9Ean4/HrhO0j4RcXFe53pgNOmmTKtIN1gZQboBVmMuyWU+B5yQ521souxsYBJpLPcHCuaPJd3D/bYc617AL0jDlF4MrCYNUXpzPnXx/bLe/fYG5MfniubvC1xNOr2zJ+neH49KGhoRK4H5wAHAhaRTGg1jojd81nOBY4F/JbVufJb0/dgPOKUVcZpVXkR48uSpaCLdFjcamR4rsU4noCtpvOzzCuZPzOv2z6+nAq+X2I5Iye3+ovl7ke5BcVUzsc/I++ucp77AZXneVbnM3+bXpxWtewspUe+dX/8WmFliX53zdi4qmHc7sLqRskfnsiMK3ufzwG1F5RYCywtefxvYAPxxUbkfAX8AOpWIb2De54Qca1fSH401wB3N1GMn0p+N54CrG/k8BxSVPzLP/3LR/DPy/D9t6++1J08R4aZ+s2acABxWMG3XvCzpVEm/lLQe2AK8S2oFKNXc/yugt6TZkv5aUvHR80HA/sB/5KbjzrnV4V3gCWBkmbF/kKfXgG8B15CO3Mnb2ALcUbTO7aQOcH9eEOvZkqZK+jNJFfvNiIjI+ztBUlcASb2BL7J9P4TRwP8ALxbVx/1AH0rXdYMfkOriXeBB0hH9GcWFJB0j6eeS1pLqZzPpCL+cfYwm/Wm6pyjOhtaMvyxjG2ZV58RvVtqKiHiyYHq2YYGkE0hNuyuAcaRkeRjpqLxLUxuMiIdIzb4DSLeGfUPSA5IOzkX65Mdb+Sh5N0yjSacUytHwZ2Ug0C0iLoiITXnZ3sAb8fEe7a8VLAf4Gumc9iRSJ7jXJV0taY8yY2jObNJR+In59TjS79KcgjJ9gL/i43UxNy8vpz6+TaqLLwDfz8//rbCApMNIPf/Xk1oIhudyKyjxeRbF2YXU4lMYZ8O90cv93Myqyuf4zVrvVOC3ETGhYYakLqQm+ZIi4i7gLkl7kpLaFcAiSfsBDbeG/QdgcSOrb2pkXmP7KNVb/U3SbT47FyX/fvlxbd7GO6RTE1MlDSBd3vgvpCPbaeygiFglaQlwGumc/mnAQxGxpqDYWtLlguc3sZlnm5hfaHVBfTwsqTswUdKNEdHQ0e8k0vv6UmGdSNqbdEqhOWtJSX9UE8vXNDHfrKac+M1a7xOk5uBCp9OClrSIeBdYIGkgqVNZT9I4AS8BgyPiygrFWuxh4JvAl4A7C+Z/hZT8nmgk1tXAlZLGAwcXLy+wiXS6o1y3Ad+RdCTpCHt80fL7SOMQrI6IN1qw3VL+kfTeLyF1JoSPPs8P71Uu6RhSh7/fFKzb8Mer+D3eR+qs2TUiHq5QnGYV58Rv1nr3ATfky84WkZLW14G3S60k6Z9Jzb6LSb3C9wMmA09GxJu5zGTgJ7kFYR7paLIf8Hng+Yi4fgdjXwg8TuoZ34+U2P6G1KnxsohYl+N4AvgJqbn7PVIHts8Cs0ps+xlggqRzgKXAhohYUaL8HcC1pD8A7wH3FC2/itTS8Kika4HfAd1IfSE+HxEn0EIR8YqkG4Epkg6JiGWkz3My8CNJt+btX8THj9QbBnCanC9f/AB4OiIelDSPdI7/GtKARZBO6YwBLoiI4qsIzGqvrXsXevK0M0581Kt/YIkynYDLSYnhfVIi/xyp49i/F5Qr7tV/PKnD16uko8eXSOfR+xVt/y9I55zXkY7CXyCd1x7eTOwzyH3nminXA/hejmMzqcn8G0VlriIl7/WkjnHLgckFyxvr1d+N1IqwLi9bledv16u/aD/35GWzm4h1b9KlhatzrK8Dj1Bw9UQT6zX06j+zkWV98nu6u2DelLyPDaTEfSTwGPBg0bqX5s99a9Fn24nUkrI8f2ZvkcZPuALo3tbfa0+eIgJFfNiqZWZmZnXOvfrNzMw6ECd+MzOzDsSJ38zMrANx4jczM+tAnPjNzMw6ECd+MzOzDsSJ38zMrANx4jczM+tAnPjNzMw6kP8HHmudbz3XHFAAAAAASUVORK5CYII=\n",
+ "text/plain": [
+ "<Figure size 576x432 with 1 Axes>"
+ ]
+ },
+ "metadata": {
+ "needs_background": "light"
+ },
+ "output_type": "display_data"
+ }
+ ],
+ "source": [
+ "def plot_roc_curve(fpr, tpr, label=None):\n",
+ " plt.plot(fpr, tpr, linewidth=2, label=label)\n",
+ " plt.plot([0, 1], [0, 1], 'k--')\n",
+ " plt.axis([0, 1, 0, 1])\n",
+ " plt.xlabel('False Positive Rate', fontsize=16)\n",
+ " plt.ylabel('True Positive Rate', fontsize=16)\n",
+ "\n",
+ "plt.figure(figsize=(8, 6))\n",
+ "plot_roc_curve(fpr, tpr)\n",
+ "plt.show()"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 21,
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "0.9604938554008616"
+ ]
+ },
+ "execution_count": 21,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "from sklearn.metrics import roc_auc_score\n",
+ "roc_auc_score(y_train_5, y_scores)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": []
+ }
+ ],
+ "metadata": {
+ "kernelspec": {
+ "display_name": "Python 3",
+ "language": "python",
+ "name": "python3"
+ },
+ "language_info": {
+ "codemirror_mode": {
+ "name": "ipython",
+ "version": 3
+ },
+ "file_extension": ".py",
+ "mimetype": "text/x-python",
+ "name": "python",
+ "nbconvert_exporter": "python",
+ "pygments_lexer": "ipython3",
+ "version": "3.7.6"
+ }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 4
+}
diff --git a/topic_19/.ipynb_checkpoints/Untitled-checkpoint.ipynb b/topic_19/.ipynb_checkpoints/Untitled-checkpoint.ipynb
new file mode 100644
index 0000000000000000000000000000000000000000..7fec51502cbc3200b3d0ffc6bbba1fe85e197f3d
--- /dev/null
+++ b/topic_19/.ipynb_checkpoints/Untitled-checkpoint.ipynb
@@ -0,0 +1,6 @@
+{
+ "cells": [],
+ "metadata": {},
+ "nbformat": 4,
+ "nbformat_minor": 4
+}
diff --git a/topic_19/.ipynb_checkpoints/roc_demo-checkpoint.ipynb b/topic_19/.ipynb_checkpoints/roc_demo-checkpoint.ipynb
new file mode 100644
index 0000000000000000000000000000000000000000..c108758f94a1a9f26651e0068219422e0c0dd62d
--- /dev/null
+++ b/topic_19/.ipynb_checkpoints/roc_demo-checkpoint.ipynb
@@ -0,0 +1,287 @@
+{
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "# Topic 19 -ROC Performance Demo\n",
+ "\n",
+ "In this demonstration we will continue on from where we started in topic 17. First we will train a classifier algorithm on the MNIST data set. Then we will calulate or performance metrics and analyse these using a ROC curve."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 9,
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "image/png": "iVBORw0KGgoAAAANSUhEUgAAAOcAAADnCAYAAADl9EEgAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAGaElEQVR4nO3dPUiWfR/G8dveSyprs2gOXHqhcAh6hZqsNRqiJoPKRYnAoTGorWyLpqhFcmgpEmqIIByKXiAHIaKhFrGghiJ81ucBr991Z/Z4XPr5jB6cXSfVtxP6c2rb9PT0P0CeJfN9A8DMxAmhxAmhxAmhxAmhljXZ/Vcu/H1tM33RkxNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCLZvvG+B//fr1q9y/fPnyVz9/aGio4fb9+/fy2vHx8XK/ceNGuQ8MDDTc7t69W167atWqcr948WK5X7p0qdzngycnhBInhBInhBInhBInhBInhBInhHLOOYMPHz6U+48fP8r92bNn5f706dOG29TUVHnt8PBwuc+nLVu2lPv58+fLfWRkpOG2du3a8tpt27aV+759+8o9kScnhBInhBInhBInhBInhBInhGqbnp6u9nJsVS9evCj3gwcPlvvffm0r1dKlS8v91q1b5d7e3j7rz960aVO5b9iwody3bt0668/+P2ib6YuenBBKnBBKnBBKnBBKnBBKnBBKnBBqUZ5zTk5Olnt3d3e5T0xMzOXtzKlm997sPPDx48cNtxUrVpTXLtbz3zngnBNaiTghlDghlDghlDghlDghlDgh1KL81pgbN24s96tXr5b7/fv3y33Hjh3l3tfXV+6V7du3l/vo6Gi5N3un8s2bNw23a9euldcytzw5IZQ4IZQ4IZQ4IZQ4IZQ4IZQ4IdSifJ/zT339+rXcm/24ut7e3obbzZs3y2tv375d7idOnCh3InmfE1qJOCGUOCGUOCGUOCGUOCGUOCHUonyf80+tW7fuj65fv379rK9tdg56/Pjxcl+yxL/HrcKfFIQSJ4QSJ4QSJ4QSJ4QSJ4Tyytg8+PbtW8Otp6envPbJkyfl/uDBg3I/fPhwuTMvvDIGrUScEEqcEEqcEEqcEEqcEEqcEMo5Z5iJiYly37lzZ7l3dHSU+4EDB8p9165dDbezZ8+W17a1zXhcR3POOaGViBNCiRNCiRNCiRNCiRNCiRNCOedsMSMjI+V++vTpcm/24wsrly9fLveTJ0+We2dn56w/e4FzzgmtRJwQSpwQSpwQSpwQSpwQSpwQyjnnAvP69ety7+/vL/fR0dFZf/aZM2fKfXBwsNw3b948689ucc45oZWIE0KJE0KJE0KJE0KJE0KJE0I551xkpqamyv3+/fsNt1OnTpXXNvm79M+hQ4fK/dGjR+W+gDnnhFYiTgglTgglTgglTgglTgjlKIV/beXKleX+8+fPcl++fHm5P3z4sOG2f//+8toW5ygFWok4IZQ4IZQ4IZQ4IZQ4IZQ4IdSy+b4B5tarV6/KfXh4uNzHxsYabs3OMZvp6uoq97179/7Rr7/QeHJCKHFCKHFCKHFCKHFCKHFCKHFCKOecYcbHx8v9+vXr5X7v3r1y//Tp02/f07+1bFn916mzs7PclyzxrPhvfjcglDghlDghlDghlDghlDghlDghlHPOv6DZWeKdO3cabkNDQ+W179+/n80tzYndu3eX++DgYLkfPXp0Lm9nwfPkhFDihFDihFDihFDihFDihFCOUmbw+fPncn/79m25nzt3rtzfvXv32/c0V7q7u8v9woULDbdjx46V13rla2753YRQ4oRQ4oRQ4oRQ4oRQ4oRQ4oRQC/acc3JysuHW29tbXvvy5ctyn5iYmNU9zYU9e/aUe39/f7kfOXKk3FevXv3b98Tf4ckJocQJocQJocQJocQJocQJocQJoWLPOZ8/f17uV65cKfexsbGG28ePH2d1T3NlzZo1Dbe+vr7y2mbffrK9vX1W90QeT04IJU4IJU4IJU4IJU4IJU4IJU4IFXvOOTIy8kf7n+jq6ir3np6ecl+6dGm5DwwMNNw6OjrKa1k8PDkhlDghlDghlDghlDghlDghlDghVNv09HS1lyMwJ9pm+qInJ4QSJ4QSJ4QSJ4QSJ4QSJ4QSJ4QSJ4QSJ4QSJ4QSJ4QSJ4QSJ4QSJ4QSJ4QSJ4QSJ4QSJ4QSJ4QSJ4QSJ4QSJ4Rq9iMAZ/yWfcDf58kJocQJocQJocQJocQJocQJof4DO14Dhyk10VwAAAAASUVORK5CYII=\n",
+ "text/plain": [
+ "<Figure size 432x288 with 1 Axes>"
+ ]
+ },
+ "metadata": {
+ "needs_background": "light"
+ },
+ "output_type": "display_data"
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "Precision: 0.8370879772350012\n",
+ "Recall: 0.6511713705958311\n",
+ "F1 Score: 0.7325171197343846\n"
+ ]
+ }
+ ],
+ "source": [
+ "from sklearn.datasets import fetch_openml\n",
+ "mnist = fetch_openml('mnist_784', version=1)\n",
+ "mnist.keys()\n",
+ "\n",
+ "import matplotlib as mpl\n",
+ "import matplotlib.pyplot as plt\n",
+ "\n",
+ "X, y = mnist[\"data\"], mnist[\"target\"]\n",
+ "\n",
+ "some_digit = X[0]\n",
+ "some_digit_image = some_digit.reshape(28, 28)\n",
+ "plt.imshow(some_digit_image, cmap=\"binary\")\n",
+ "plt.axis(\"off\")\n",
+ "plt.show()\n",
+ "\n",
+ "import numpy as np\n",
+ "y = y.astype(np.uint8)\n",
+ "X_train, X_test, y_train, y_test = X[:60000], X[60000:], y[:60000], y[60000:]\n",
+ "y_train_5 = (y_train == 5) # True for all 5s, False for all other digits\n",
+ "y_test_5 = (y_test == 5)\n",
+ "\n",
+ "from sklearn.linear_model import SGDClassifier\n",
+ "\n",
+ "sgd_clf = SGDClassifier(random_state=42)\n",
+ "sgd_clf.fit(X_train, y_train_5)\n",
+ "\n",
+ "from sklearn.model_selection import cross_val_score\n",
+ "cross_val_score(sgd_clf, X_train, y_train_5, cv=3, scoring=\"accuracy\")\n",
+ "\n",
+ "from sklearn.model_selection import cross_val_predict\n",
+ "y_train_pred = cross_val_predict(sgd_clf, X_train, y_train_5, cv=3)\n",
+ "\n",
+ "from sklearn.metrics import confusion_matrix\n",
+ "confusion_matrix(y_train_5, y_train_pred)\n",
+ "\n",
+ "from sklearn.metrics import precision_score, recall_score\n",
+ "print(\"Precision:\",precision_score(y_train_5, y_train_pred)) # == 3530 / (3530 + 1891)\n",
+ "print(\"Recall:\", recall_score(y_train_5, y_train_pred)) # == 3530 / (3530 + 687)\n",
+ "\n",
+ "from sklearn.metrics import f1_score\n",
+ "print(\"F1 Score:\",f1_score(y_train_5, y_train_pred))\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Now we can calculate the scores used to analyse the recall-precision trade-off. We will done by setting the method to \"decision_function\". The scores can then be used to calulate the precision and recalls for the range of thresholds used by the classifier."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 3,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "y_scores = cross_val_predict(sgd_clf, X_train, y_train_5, cv=3,\n",
+ "method=\"decision_function\")\n",
+ "\n",
+ "from sklearn.metrics import precision_recall_curve\n",
+ "precisions, recalls, thresholds = precision_recall_curve(y_train_5, y_scores)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Now we can plot the recalls and precisions against threhold values to understand the nature of the trade-off. We will define a function that will print the two data series. "
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 4,
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "image/png": "iVBORw0KGgoAAAANSUhEUgAAAfYAAAEPCAYAAACwduZtAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAgAElEQVR4nO3dd3xUVdrA8d+TSiD0DqGDQKQTEJBmoapgF+xrwbLoqsjq6r4ra1/dtayIihUUFVYXl3UBUUFQpIUuTar03klIPe8fZyaZhEkygZm5M5Pny+d+bjtzz3Nzhzy57RwxxqCUUkqpyBDldABKKaWU8h9N7EoppVQE0cSulFJKRRBN7EoppVQE0cSulFJKRRBN7EoppVQEKTGxi8gHIrJfRH4pYr2IyD9FZJOIrBKRTv4PUymllFK+8OWM/SNgYDHrBwEtXMMI4K1zD0sppZRSZ6PExG6MmQccLqbIUGCisRYCVUSkrr8CVEoppZTvYvywjfrADo/5na5lewoXFJER2LN6SKAzVfxQu4pMAtESTXRUNFESRbREIyIIgoi4itjp2KhYYqNjbXl3ORGixX5WELuNKLu9aIl2eOdUJNmxA/bv974uPh7atMmfX74ccnO9l01Kgtq17fThw7B1a8H1iYnQsuW5x6uctXTp0oPGmJqBrMMfiV28LPPaTq0xZjwwHqBRciPzxMQnPD5gCpcttMHSrffHNsKljozsDLJyszDGkGtyMbjGHvNFrUvLTsubd5dz1+G5zHPsjsmXMqeyTpGRnUFGTgYnMk6QlZtFrsktdsjKySIjJ4Mc1z9/i42KpWpCVcrFlCMuOo7EuEQqxlWkXEw5qpSrQkJsAgkxCVQtV5XEuESqJlSlQmwFqpSrQmJcIvUq1qNVjVZ5f2CosmfJEti0CVJS4NgxWL/ee7lKlWDIkPz5zz6DnCK+0p07Q+vWdnrrVpg/v+D6WrWgf/9zj105S0R+C3Qd/kjsO4EGHvNJwO6SPlSzfE3uSbnHD9WrSJSRncGR00c4kXGC9Ox0jmccJysni+zcbHJMDjm5OWTnZnM6+zS7TuziUNohDqUf4nT2aTJyMkjLSuNU5ikyczLJyMng2OljnMw8mVdm/6kiTrF8lBiXSM3yNWlVoxVtarWhUnwlmlZtSrva7WhVoxUxUf74r6VC1ccfwxtvwKuvwkMP2QTvi+HDfSvXpIkdlDob/vjtMw0YKSKfAxcAx4wxZ1yGV6o04mPiqZNYhzqJdfy+7bSsNI6dPlYg6adnp3My8yTHM46TkZ3B8YzjnMg8wbHTxziWcYxTWac4nH6YtKw01h1Yx75T+ziZeZKtR7cyY9OMAtuPi46jVY1WdKvfjSZVm9CyeksGNh9IQmyC3/dFOePECTuuWNHZOJTypsTELiKfAX2BGiKyE3gKiAUwxrwNTAcGA5uANOB3gQpWKX8oH1ue8rHlz/rzxhiOnD7CjmM7WHtgLduObmP/qf1sPrKZJbuXsPfkXlbtW8WqfavyPhMXHceQlkPoULsDA5sPpF3tdsRGx/pjd5QDNLGrUCZOdduakpJiUlNTHalbqUA6kn6ElftWsnrfajYf2cy3W75l7YG1BcokxiUy+drJDG4x2KEoQ9OJEzBtGvTuDQ0alFzeKQMGwKxZMH06DBrkdDQqnIjIUmOMjzdvzo7eCFTKz6omVKVv4770bdw3b9mmw5uYs3UOP+/8mY9WfMTJzJNc9ull9GzYk0lXT6Jh5YbOBRxgR45AZmb+E9+eMjIgLs6Op06FZ56Bdevsuo8+gttuC2qoPtMzdhXKtElZpYKgebXm3N35bj4c+iFpT6TxSLdHiJIoftr+E63fbM0/F/2TU5mnnA7Tb06fhpgYEIFq1aBOHZuo582zy665BqKioFw5GDUK/vxnuPHG/KQOcPHFcOoUXHklvP++Y7vilSZ2FcpC+lL8sWPHOHjwIJmZmUGKSoWKuLg4atSoQeXKlZ0OJWC2HtnKXf+9i9lbZwNQLaEa/+j/D27vcLuzgZ2jvXuhrpcmqh54wL7G9fXXZ6775BO4+eb8+c2boWlT+OEHuOgiu+yxx+D55+0fBE5r0gS2bbOvvDVr5nQ0KpwE41J8yCb206dPs337dpKSkkhISNB3hssQYwzp6ens3LmThg0bUq5cOadDCqjPVn/GE7OfYNvRbQCM6DSCty5/iygJgQxWSunpsHs3dOoEx4/nL1+9Gs47zybDwo2sLF1qy6el2fvW/ftDeY9nGydMgDvvtO9/X3IJ/Pvf9v1wJ2VlwcmTNo5obe9IlUKZTuw7duwgMTGRqlWrBjEqFUoOHz7MqVOnaBDKT1H5iTGG1xa+xh+/+yPZudk82v1RXu7/stNhlcovv0DbtjB0qG2UpUcPaNWq6PLbt9vW1nw5A585E4YNs43BtGxpH1pr2tR/sSsVLMFI7CF7SnD69GkSExOdDkM5qGLFipw+fdrpMIJCRHi4+8NMvHIiAH9f8Hfu+/q+M1odDEXPPGPvm7dta+f/8x/o16/4pA7QsKHvl9UHDoSFC6FRI9iwwZ7V79tXdPOsSpVlIZvYs7OziYnRh/bLspiYGLKzs50OI6iGtx3Ou1e8S7RE8/bSt3l67tNOh1QsY2Ds2ILL/vjHwLyq1qoVrFxpL9s3amRbfhs0CH791f91FefoUbjgArj66uDWq5SvQjaxA3pfvYwrq8f/rk53MfnayQCMmTuG8UvHOxxRQVOn2jN0EbjnHpvc3TZvhr/9LXB1V64M334LM2bYhD5rlm3SNZgXNo4ehcWLQZvhUKEqpBO7UmXVNcnX8OxFzwJwz9f38P4y59/3SkuzydzzTPXdd+39b2PsEIz73tWq2XffX3oJKlSwSf7ttwNfr5v7VTe9U6hClSZ2pULUE72e4MGuDwJw13/vYv3BIroQC5L//e/MZU89ZS+NO6FxY3jrLTt9//3BO4M+edKO9R12Fao0sQfRRx99lNdXuIhQsWJF2rdvz9ixY4N6L3nMmDGlvszdt29f+vbtG5iAlFciwmsDX+OK864A4IrPruBExomgx1Gnjj1T7949f1lOjj1DHzMm6OEUcMstcN11dvrpID2OoI3TqFCnid0B//rXv1iwYAFffvklXbt25YEHHuDpYP1WAu666y4WLFhQqs+MGzeOcePGBSgiVRQR4YOhH9C8WnM2Hd7EH7/9Y9DqXrXKJvR9++x8dnb+JfdQaCTG7fXX7fi772zTtYGmiV2FuhD671l2dOjQgW7dutG/f3/effdd+vbty2uvvea1rDHG7y3vJSUl0a1bt1J9Jjk5meTkZL/GoXxTo3wNvrz+SwTh3WXvFug1LlB69YL27fPnK1Wyl75DUd260K2bbaHu4MHA16eJXYU6TewhoEuXLpw4cYL9+/fTuHFjbr75Zj744ANatWpFXFwc/3Pd3ExLS+Oxxx6jSZMmxMXF0aRJE5577jlyC73Me+DAAe6//34aNGhAfHw8DRo04JZbbiEjIwPwfin+9ddfp3Xr1iQkJFC1alVSUlKYOnVq3npvl+I3bNjAVVddRZUqVUhISKBbt27MnDmzQBl3XRs3buSyyy4jMTGRRo0a8fTTT58Rtypau9rtuKfzPeSYHG7/6vaAtyu/Y0fB+WPHAlrdOfv5Z/sMQL16ga+rSRO49Vb7x49SoSgsE7v7VRtvw3iPN4PGjy++rKfOnYsuN2JEfrmlS/2/P1u3biU6OjqvQZ45c+bwyiuv8NRTTzFz5kzatWtHdnY2AwYM4L333uMPf/gDM2bM4K677uKZZ55h9OjReds6cuQIPXr0YPLkyTzyyCNMnz6dl156iaysrCLP/CdNmsSoUaMYPnw406dPZ9KkSVx77bUcPny4yJh3795Nz549WblyJWPHjmXKlClUqVKFyy67jBkzZpxR/qqrruLiiy/mq6++4sorr+Spp55iwoQJ5/iTK1uev+R5mlVtxvK9y3kr9a2A1OF+beyvf7XjgweD+yrZ2Qrmm5F9+thmbu++O3h1KlUqxhhHhs6dO5virF27tsh1+Xf6zhzeeSe/3DvvFF/WU6dORZe7++78cqmpxYZdrA8//NAAZv369SYrK8scPnzYvP322yYqKsoMHTrUGGNMo0aNTEJCgtmzZ0+Bz06cONEAZu7cuQWWP/vssyY2Ntbs27fPGGPM//3f/5moqCizbNmyIuN46qmnDB4/gN///vemY8eOxcbep08f06dPn7z5UaNGmejoaLNx48a8ZdnZ2ea8884rsC13XR988EGB7bVp08b069ev2DqNKf57UBb9Z/1/DGMwNV+qadIy0/y67ZwcY1q1MubZZ41JT/frpoNi3Tpjxo0z5vvvnY5EqaIBqSbA+TUsz9iLS9eeZ9cjRhRf1tPSpUWX87wK0LnzucffqlUrYmNjqVatGvfffz833XQTH3zwQd76bt26UadOnQKfmTlzJo0aNaJHjx5kZ2fnDf379ycrK4uFCxcCMGvWLLp06ULHjh19jqdLly6sWLGCBx54gO+++460tLQSPzNv3jy6detG8+bN85ZFR0czfPhwVqxYwXHPHkCAyy67rMB8mzZt2L59u88xKuvy8y6nVY1WHEg7wJfrvvTrtqOjYf1624VqTo5fNx0UP/xgX3ubNCmw9WzbZtvFL/QVVypkhGViD3dTp05lyZIlrF+/nlOnTjFx4kSqVauWt76ulz4v9+/fz2+//UZsbGyBoWvXrgAcOnQob5yUlFSqeG699VbeeustFi1axIABA6hWrRpXX30127ZtK/Izhw8f9hpnnTp1MMZw5MiRAss99w8gPj6+zLQD709REsX9KfcD8OGKD/22XfeT72AbmalQwW+bDhp396lr1wa2njFjbLv4X3wR2HqUOlvaGLsD2rRpU+BMtzBv75hXr16dJk2aMGXKFK+faex6ZLlGjRrs2rWrVPGICPfccw/33HMPR44cYdasWYwaNYobbriBRYsWef1MtWrV2Lt37xnL9+7di4ickciV/9zc7mYe++4xZm+dzbebv6Vfs37ntL3Dh+276m6uiz9hx/0U/6JF9mw6UF276lPxKtTpGXuYGDhwYF5XtikpKWcMNWrUAKB///4sXryYlStXnlU9VatW5YYbbuD666/nl19+KbJcnz59WLhwYYGz+pycHCZPnkzHjh2pqL/1AqZqQlX+cMEfAHj8+8fPuQc4z7s2I0ZAzZrntDnH1Kple4wzxvYAFyia2FWo08QeJm666SZ69OjBJZdcwiuvvML333/PjBkzGDt2LP3798+7L/7www/TtGlTLr30Ul5//XVmz57NlClTuOmmmzhxwnurZSNGjGDUqFF88cUXzJs3j/fee4+PP/6Y/v37FxnPww8/TJUqVejXrx+ffvopX3/9NVdccQW//vorzz33XEB+Birf//X5PyrEVmDZnmX8tP2nc9rWK6/Y8XXXwTvv+CE4Bw0YYMfPPlvy0/zp6b5v9+uvbf/xoIldhT5N7GEiNjaWb775hrvvvpvx48czePBgbrrpJiZMmECPHj2Ii4sDoEqVKsyfP5+rrrqKF198kYEDBzJq1ChiYmLyyhR24YUXsnTpUu6//3769evHc889x80331zs62j16tXjp59+4vzzz+e+++7Lez3uf//7HwMHDgzIz0DlKx9bnpFdRwLw5pI3z3o7EyfahGUMFHGXJ6w8+ijEx8O0abYHNrcvv4QHHshvme7oUWjZEh57rOQEv3w5XHGF7SoWtBMYFQYC/dh9UcO5vO6myg79HhRty+EthjGY+GfizdH0o6X+/M8/57/7cfp0AAJ0yOjRxjz2mH19z829n88+a+cnTzZGxC4bPdou27XLmMcfN+bgwYLbmzkz//Pr1hnTqJGd3rw5KLujIgz6uptSqihNqjahd6PeZORkMHnN5FJ/vkeP/On4eD8G5rAXX7SDt/bs3U+yX389vPqqnd640Y4fecR+zvWiSZ4Yj0eMExP1UrwKfZrYlQpjd3a8E4B3l71bqs95dnE6fLg/I3KeZ0LfsQMWLAB3a8grVsD8+fDEE3DkiC371VcweTIcOGDLbNlScHvud/ovvRSSkmD2bJg7F6pWDfiuKHVWNLErFcauS76OSvGVSN2dyroD63z+XJcu+dOBbtDFKcuWQatWcOONtue3li3t8i+/hBdegK1b8x8cfPjh/H7ln3224HbciT062o7bt4fevQueySsVSjSxKxXGEmITuKrVVQBMWePb02/z5uVPT5sW3HbWg6ldO0hLsy3F3XMPJCdDlSqwe7ddHxcHDz4I550He/bArFnet5OYaLdVvrztIjZS/xBSkUMTu1Jh7trkawH47JfPSv1O+xVXBCKi0OB5Rv3++/CnP8H+/fm9ssXF2T9qHnrIvr/vbkgxJ8fea3ffS+/VC1auhDvusGVvvhmeeSa4+6JUaWhiVyrMDWg2gGoJ1dhwaAMbD28ssXyvXvZec3Z2EIILIV27wm+/5b/y5n7787777Pv77kvxTz1l/wi48caCbeZfckn+9EsvBSdmpc6GJnalwlxsdCwXNb4IgM9/+bzYsu+8Yy+/t2+ff884kr3+esH5qKj8tuQTEgque/hh2wnOwoVQrZptlOaPf8xfn5CQ39/7yZOBi1mpc6WJXakIcG/KvYDtGKaoy/HZ2XDvvXDllQWfio9kDz5oL7+7ffUVvPeenfbszmDXLttAT61acMEF9gG7mBj7cN2TT9rpm2+29+oBXC04KxWSNLErFQEuanwRtSvUZtvRbSzbs8xrmZ9/zp/2R/fD4cKz7XvPxhdr186fvu02GDnSnqWDfT3ujjvs9OTJ9pK8MTbJv/IKfP99wMNW6qxpYlcqAkRHRTOoxSAAZm32/nh3nz52/MgjkfskfFGWLoX//Md2SQswaBDcckv++qvsiwXcequ9VeG5bPNmO46OtsPDD9un5JUKVZrYg+ijjz5CRPKGuLg4mjVrxhNPPOF43+SNGzfm9ttvz5t3x1pcn+wqtAxoZntAmb1t9hnrjh3Ln/Y4zGVGp04wZEj+Gbv7ATq3oUPzp5cuteOLLy7Y9WtZeCZBRQZtYsEB//rXv0hKSuLEiRNMnTqVF154gRMnTvDGG284HZoKYxc3uRhBmPfbPNKy0igfWz5vnedT3G3bOhBciHA3nXvqVMHlSUn50+7OY+Li4LXX4Jtv7OV4TewqXPh0xi4iA0Vkg4hsEpHHvaxvKCJzRGS5iKwSkcH+DzVydOjQgW7dutGvXz/GjRvHpZdeyvvvv09ubq7ToakwVqtCLTrU6UBmTiYLdizIW24MPP+8nX7sMYeCCxHnnWfHCxfaRmk8jRtnxw8/nL/sd7/Lb45WE7sKFyUmdhGJBt4EBgHJwHARSS5U7M/AFGNMR2AYMM7fgUayTp06kZ6ezsGDB/OWbd26lZtuuomaNWsSHx9Phw4dmDp16hmfXblyJVdddRXVq1cnISGBli1b8sILL+StnzVrFoMHD6Zu3bqUL1+eNm3a8I9//IMczxd0VcRwv/Y2Z9ucvGW5ufb+8i23wHPPORVZaKhdGxo2tE+1V69ecN1990FGBvTvX3B54SZllQp1vlyK7wpsMsZsARCRz4GhwFqPMgZw342qDOz2Z5Ce5K+h8dSPeap0LXwVZ9u2bVSuXJnqrt80O3bs4IILLqBWrVq8+uqr1KxZk8mTJ3PNNdfw1VdfMWTIEAAWL15M3759ad68Oa+++ipJSUls3LiRVatW5W17y5YtXHLJJTzwwAOUK1eO1NRUxowZw4EDB3jxxRf9tg8qNFzU5CJeWfhKgcQeHW1f4erVS5MTwIYN9h675xPybt6WuRvyycgIbFxK+Ysvib0+sMNjfidwQaEyY4BZIvIAUAG41NuGRGQEMAKgYcOGpY01YuTk5JCdnZ13j/3LL7/ktddeI9r1W3fMmDEYY5g7d25esh8wYAA7duzgL3/5S15if/TRR6levToLFy6kfHl7P/Xiiy8uUNe9996bN22MoVevXmRmZvL3v/+d559/nihvfVuqsNWrYS+iJIrFuxZzMvMkFWITSU8v+GpXWVeunB18tXevHTdrFph4lPI3XxK7t1Pkwqerw4GPjDH/EJHuwMci0sYYU+CmsTFmPDAeICUl5axOef15puyUVq1aFZi///77GTlyZN78zJkzGTx4MJUrVybbo93PAQMGMHr0aI4fP05MTAzz589n9OjReUndmz179jBmzBhmzpzJ7t27C2xv//791KlTx497ppxWuVxlOtftzJLdS5i/fT4HFw3g5pvtulI2I69c3Fc59BEYFS58OV3bCTTwmE/izEvtdwJTAIwxC4BygLbNVISpU6eyZMkSpk+fzqWXXsq4ceOYOHFi3vr9+/czceJEYmNjCwyjR48G4NChQxw5coTc3FySPB/nLSQ3N5chQ4bw9ddf8+c//5nZs2ezZMkSnnzySQDHX7FTgeF5n92d1NXZc9/Zcj8tr1So8+WMfQnQQkSaALuwD8fdWKjMduAS4CMRaY1N7Af8GWgkadOmDc2bNwfspfN27doxevRorrnmGipUqED16tXp1asXjxXxCHO9evXIyckhKiqKXbt2FVnP5s2bSU1N5eOPP+Zmj9/w//3vf/27Qyqk9G3cl5d+fonvt2rzaP4wY4Yd638bFS5KPGM3xmQDI4FvgHXYp9/XiMjTIjLEVWwUcLeIrAQ+A243pe0/soyKj4/n5ZdfZv/+/YxzvW8zcOBAVq1axfnnn09KSsoZQ3x8POXLl6dnz5588sknpKene912WloaALGxsXnLsrKymKQdSke0Xo16ERsVS+ruVEg4DMCECQ4HFcbat7fj1q2djUMpX/nUQI0xZjowvdCyv3hMrwUu9G9oZceQIUPo0qULf//73xk5ciRPP/00Xbt2pXfv3owcOZLGjRtz5MgRfvnlF7Zs2cIHH3wAwN///nf69OlD9+7dGTVqFElJSWzZsoUVK1bwxhtv0Lp1axo1asSTTz5JdHQ0sbGxvPrqqw7vrQq0xLhEutbvyvwd8yFpAWy8jGHDnI4qfP3737aBn0cfdToSpXyjj0SHiGeffZb9+/fz9ttv07BhQ1JTU2nfvj1PPPEE/fr147777mPu3LkFnnrv0qUL8+fPp0GDBjzwwAMMHjyYl19+Oe++e1xcHF999RV16tTh1ltv5fe//z29e/fm8cfPaGNIRZjejXrbiWbfAt5f41K+adgQxo6Fxo2djkQp34hTV8xTUlJMajF9R65bt47Weu2rzNPvwdmZsXEGgz8dDDu6w/s/6xPxSoUIEVlqjEkJZB16xq5UBOpSv4udqLuMF/6WXXxhpVRE0cSuVASS9BpwrAHEZHDZzZudDkcpFUSa2JWKQHPmAHs6AZC672dng1FKBZUmdqUi0HXXAVvtg5bTN00vvrBSKqKEdGLXV+HLNj3+52jjIAB+2PYDuUbbQ1WqrAjZxB4bG1tkwyuqbEhPTy/QuI7yzcmTronDzakZX5+DaQdZs3+NozEppYInZBN7rVq12LVrF2lpaXrmVsYYY0hLS2PXrl3UqlXL6XDCzqxZ7ilhQMsz+2dXSkU2n1qec0KlSrZ79927d5OVleVwNCrYYmNjqV27dt73QPmud287DB8OcY0v4pNVnzBn2xwevOBBp0NTSgVByCZ2sMldf7ErVTo//QQPPwx9+8Jh0xeAudvmkmtyiZKQvUinlPIT/V+uVIR56im46ipYtw6aVGlCw8oNOXL6CKv2rXI6NKVUEGhiVyqCZGTA2rUgAm3bgojk98++Ve+zK1UWaGJXKoKsXQvZ2dCiBSQm2mV5iV0foFOqTNDErlQEWbrUjjt1yl92UROb2Of9No+c3BwHolJKBZMmdqUiyLJlduyZ2BtWbkjTqk05lnGMpXuWOhOYUipoNLErFUHeesuOPRM7QN9GfQH4eYe2G69UpNPErlQEat++4HynujbT65PxSkW+kH6PXSlVOkePwunTUKNGweVta7cF4Jf9vzgQlVIqmPSMXakIUrky1K595vK2tWxiX7lvJaezTwc5KqVUMGliVypC7N8PRbW+XDWhKufXPJ/MnExSd6cGNzClVFBpYlcqQtSuDXFx8N133tf3aNADgKW79cl4pSKZJnalIkCOx+vpzZt7L9O+tn2ibuW+lUGISCnlFE3sSkWAFSvypxs39l6mXe12gCZ2pSKdJnalIsA339jxtdcWXaZd7XZESRSr9q3SB+iUimCa2JWKANOn27FI0WUql6tMcs1ksnOzWbxrcXACU0oFnSZ2pSLAokV2fNllxZfr3bA3AAt2LAhwREopp2hiVyoCZGfbceEW5wrT++xKRT5teU6pCHD8OKxcCe3aFV+ua/2uACzbsywIUSmlnKBn7EpFgIoVoWdPiCrhf3RyzWSiJZqNhzeSlpUWnOCUUkGliV2pMiQ+Jp7za51Prsllxd4VJX9AKRV2NLErFebuuAMGDLCX4n3RuW5nQFugUypSaWJXKszNmQOzZkFsrG/l3R3CrD2wNoBRKaWcooldqTB27Bhs22anzzvPt8+0rtkagDUH1gQmKKWUo3xK7CIyUEQ2iMgmEXm8iDLXi8haEVkjIp/6N0yllDfuFucAYnx8x6Vz3c4IwuJdi8nMyQxMYEopx5SY2EUkGngTGAQkA8NFJLlQmRbAn4ALjTHnAw8FIFalVCFrzuKku2aFmjSv1pyMnAzWHVjn/6CUUo7y5Yy9K7DJGLPFGJMJfA4MLVTmbuBNY8wRAGPMfv+GqZTy5oMP7Lh799J9rkOdDgAs37vczxEppZzmS2KvD+zwmN/pWubpPOA8EZkvIgtFZKC3DYnICBFJFZHUAwcOnF3ESqk8O3fa8YgRpftcxzodAfSVN6UikC+J3Vu3EqbQfAzQAugLDAfeE5EqZ3zImPHGmBRjTErNmjVLG6tSqpArrrDjgV7/lC6anrErFbl8edxmJ9DAYz4J2O2lzEJjTBawVUQ2YBP9Er9EqZTyatq0s/tcx7r5Z+zGGKS4buGUUmHFlzP2JUALEWkiInHAMKDwr5OvgIsARKQG9tL8Fn8GqpTynzqJdahdoTbHM46z9ehWp8NRSvlRiYndGJMNjAS+AdYBU4wxa0TkaREZ4iNdFmMAACAASURBVCr2DXBIRNYCc4DRxphDgQpaKQX/+x/885/w669n93n3WfvyPXo5XqlI4tObr8aY6cD0Qsv+4jFtgEdcg1IqCH73OzhwAN57z/fGaTydX/N8Zm6ayYZDG/wfnFLKMdrynFJhyv1iydkkdYCW1VsC8Mv+X/wUkVIqFGhiVyoMpXn0uNqly9ltI+9SvD4Zr1RE0cSuVBha52owLjkZypU7u220qdWGaIlmw8ENnMo85b/glFKO0sSuVBhyNyV7/vlnv41yMeVIrpmMwbBq3yr/BKaUcpwmdqXC0Pr1dty69bltRy/HKxV5NLErFYYqVoSkJGjR4ty206lOJ0BfeVMqkmhiVyoM/elPsGMH3HTTuW0npV4KAD/v/NkPUSmlQoEmdqXC2Lm2BJtSL4UKsRVYe2At+07u809QSilHaWJXKsycPAnHj/tnW/Ex8bSv0x6AlftW+mejSilHaWJXKsxMmQKVK8MDD/hne53rdgZg6e6l/tmgUspRmtiVCjOrXG+m1avnn+2577Mv3aOJXalIoIldqTCzerUdt2vnn+11qWebrpv721xyTa5/NqqUcowmdqXCiDGw0nUr3F+JvVWNViRVSuJg2kHWHVjnn40qpRyjiV2pMLJ3Lxw6BFWq2PfY/UFEuKD+BQAs3rXYPxtVSjlGE7tSYcR9f71du3N/1c1Tr4a9AJizbY7/NqqUcoQmdqXCiGdi96fuDboDsGT3Ev9uWCkVdDFOB6CU8t2tt9oe3fz1RLxbxzodSYhJYP3B9RxJP0LVhKr+rUApFTR6xq5UGKldGy67DDp29O92Y6Nj8xqqWbZnmX83rpQKKk3sSikAUura99n1ATqlwpsmdqXCxK+/wi23wIcfBmb7PRr0AGDe9nmBqUApFRSa2JUKE0uWwCefwH//G5jt92pkn4xfsGOBNlSjVBjTxK5UmFju6jLd30/EuyVVSqJh5YYcyzjGmv1rAlOJUirgNLErFSaWuN5E69IlcHW4L8f/vEP7Z1cqXGliVyoM5OTAMtfD6ikpgavnwgYXAjB/x/zAVaKUCihN7EqFgc2bbT/s9evbV94CRRO7UuFPE7tSYSBQLc4V1rZ2WxLjEtlyZAt7T+4NbGVKqYDQxK5UGKhSxTZMc/HFga0nJiqGbkndAJi/Xc/alQpHmtiVCgOXXgpffw2PPhr4utwdwny75dvAV6aU8jtN7EqpAi4/73IApm+c7nAkSqmzoYldqRB38iTMnw8nTgSnvg51OlA+tjw7ju/gSPqR4FSqlPIbTexKhbjFi6FnT+jXLzj1RUkUyTWTAVi+d3lwKlVK+Y0mdqVC3NKlduzvHt2Kc0H9CwBI3Z0avEqVUn6hiV2pEOduca5r1+DV2bGO/Stixd4VwatUKeUXmtiVCnHuM/ZAtjhXWIc6HQC9FK9UONLErlQIO3YMtmyBcuWgdevg1dumVhsSYhJYf3A9B04dCF7FSqlz5lNiF5GBIrJBRDaJyOPFlLtWRIyIBPHcQqnItcbVyVqrVhATE7x642PiSaln/xsv3rU4eBUrpc5ZiYldRKKBN4FBQDIwXESSvZSrCDwILPJ3kEqVVStX2nGnTsGvu2t9e1N/0S79L61UOPHlHKArsMkYswVARD4HhgJrC5V7BngJCELbWEqVDffeC5dfbnt3CzZ3Yl+6Z2nwK1dKnTVfLsXXB3Z4zO90LcsjIh2BBsaYr4vbkIiMEJFUEUk9cEDv2ylVEhFo0AAaNw5+3e7EvnDnQowxwQ9AKXVWfEns4mVZ3v9yEYkCXgVGlbQhY8x4Y0yKMSalZs2avkeplAq6RpUbUSexDofTD7P5yGanw1FK+ciXxL4TaOAxnwTs9pivCLQBfhCRbUA3YJo+QKfUuZk7F9q0gWefdaZ+EaFz3c4ALNqp99mVChe+JPYlQAsRaSIiccAwYJp7pTHmmDGmhjGmsTGmMbAQGGKM0SarlDoHixbZp+J37y65bKD0adQHgO+2fudcEEqpUikxsRtjsoGRwDfAOmCKMWaNiDwtIkMCHaBSZZW7xbkuXZyLoV8z20D93G1znQtCKVUqPr0Za4yZDkwvtOwvRZTte+5hKaVCIbG3qdWGcjHl2Hp0K0fSj1A1oapzwSilfKItzykVgvbuhd9+g8TE4LY4V1hMVAzta7cHtKEapcKFJnalQtCCBXbctStERzsbS8+GPQH4cfuPzgailPKJJnalQtAPP9hxz56OhgFA38Z9AZi9dbazgSilfBLE1qeVUr667jrb8cvQoU5HAr0b9SZaolm8azHHM45TKb6S0yEppYqhZ+xKhaCePeFvf3OmjfjCKsVXomv9ruSYHH78TS/HKxXqNLErpUrkvhw/Z9scZwNRSpVIE7tSIebjj+HNN51tmKawfk3t++zfbvnW4UiUUiXRe+xKhZjXXoNly6BlS6hXz+lorB4NepAQk8CqfavYd3IftRNrOx2SUqoIesauVAg5dgxWrICYGOje3elo8sXHxHNhwwsBfe1NqVCniV2pEPLTT5Cba1ubq1DB6WgKuqjxRQBMWj3J4UiUUsXRxK5UCLn2Wjvu2tXZOLy5rf1tAMzaPIuM7AyHo1FKFUUTu1Ih5PRpO05OdjYOb+pXqk+72u1Iy0pj3m/znA5HKVUETexKhYhdu/Knb7vNuTiKM7j5YABmbJrhcCRKqaJoYlcqRBw/DtdcA7fcAvHxTkfj3aAWgwBN7EqFMn3dTakQ0bo1fPGF01EUr3tSdyrFV2L9wfVsPbKVJlWbOB2SUqoQPWNXSvksNjqW/s36AzB943SHo1FKeaOJXakQ8OOPULMmzAmDFlsvb3E5AJ+s/sThSJRS3mhiVyoEjB0LBw/Cyy87HUnJrkm+hrjoOBbtXMShtENOh6OUKkQTu1IOy821DdMAPP20s7H4IjEukR4NemAw/LT9J6fDUUoVooldKYctWWI7fGnQADp3djoa33Sr3w3Q5mWVCkWa2JVy2Pvv2/GVV4KIs7H4akDzAQD8e92/McY4HI1SypMmdqUcZAy8+66dHjLE2VhKo2fDntQoX4OtR7eydM9Sp8NRSnnQxK6Ug5Z65MS+fR0Lo9RiomK4Pvl6AGZumulwNEopT9pAjVIOOnYM2ra1ST0mzP43dqzbEYC1B9Y6HIlSylOY/SpRKrJccgmsWgWZmU5HUnrdk2yH8d9u+Zbs3GxiovTXiVKhQC/FKxUC4uKcjqD0kmsm07J6Sw6mHeSHbT84HY5SykUTu1IOmTABNm1yOoqzJyJcf769zz5lzRSHo1FKuWliV8oBmzbB7bfb++tHjzodzdlzJ/ap66fqa29KhQhN7Eo54K237HjYMKhSxdlYzsX5Nc+nbmJdDqYdZNGuRU6Ho5RCE7tSQXfsWP676yNHOhvLuRIRbmx7IwATV050OBqlFGhiVyro3n0XTpyAiy4KnyZkizOszTAAvlr/Fbkm1+FolFKa2JUKosxMeO01O/3oo87G4i+d63amcZXG7Dm5h593/Ox0OEqVeZrYlQqiyZNh1y5IToZBg5yOxj9EhCHn2fZwp2+c7nA0SilN7EoFUdeucP31MHp0+HT44ourW18NwPil4zmVecrhaJQq23xK7CIyUEQ2iMgmEXncy/pHRGStiKwSke9FpJH/Q1Uq/LVsac/ab7/d6Uj8q3ej3nSt35VD6Yd4K/Utp8NRqkwrMbGLSDTwJjAISAaGi0hyoWLLgRRjTDvgC+AlfweqVDjLyoLcCH6uTER47MLHAH06Ximn+XLG3hXYZIzZYozJBD4HhnoWMMbMMcakuWYXAkn+DVOp8Pa3v0H37rB8udORBM5lLS6jSrkqrN6/mhV7VzgdjlJlli+JvT6ww2N+p2tZUe4EZnhbISIjRCRVRFIPHDjge5RKhbHt2+GFF2Dx4vBuZa4k8THx3NLuFgDeXfquw9EoVXb5kti9PeLjte1IEbkZSAFe9rbeGDPeGJNijEmpWbOm71EqFaaMgbvugrQ0uO46++56JLur010ATFo9ifSsdIejUaps8iWx7wQaeMwnAbsLFxKRS4EngSHGmAz/hKdUeHv/ffj2W6hWDd54w+loAq9d7Xak1EvhWMYxpq6f6nQ4SpVJviT2JUALEWkiInHAMGCaZwER6Qi8g03q+/0fplLhZ8cOGDXKTo8dC7VrOxtPsNzZ8U4APlzxocORKFU2lZjYjTHZwEjgG2AdMMUYs0ZEnhaRIa5iLwOJwL9EZIWITCtic0qVCTk5MHw4HD8OQ4bYzl7KiuuSr6NcTDm+2/Idy/Ysczocpcocn95jN8ZMN8acZ4xpZox5zrXsL8aYaa7pS40xtY0xHVzDkOK3qFRki46GO+6Apk3hvfciqzGaklQvX507OtwBwKsLX3U4GqXKHm15TqkAueMOWLsWyuJzoo90fwSAf6/7NyczTzocjVJliyZ2pfzo009h4cL8+fh452JxUrNqzejRoAdpWWlMWDHB6XCUKlM0sSvlJ2++CTfdZDt32bPH6Wic92DXBwF4e+nbGOP1DVmlVABoYlfqHBkDzz0HI0fa+SeegLp1nY0pFFzZ6krqJtbll/2/MGGlnrUrFSya2JU6Bzk58PDD8Oc/2wfk3n3X9tymbEt0j/e0fUb96fs/kZWT5XBESpUNmtiVOkt790K/fvD66xAbC59/bluZU/ke6PoAyTWT2XtyL5+s+sTpcJQqEzSxK3WWdu6EH3+0Dc98953tZ10VJCKM6m5b6RmXOs7haJQqGzSxK1UK27fbe+oAKSkwcaLtsa13b2fjCmXD2wynarmqpO5O1QZrlAoCTexK+WDVKrjlFmjWDKZ6NIE+fLg+KFeShNgEbmt/GwBjF491OBqlIp8mdqWKYAzMng0DB0L79vDJJ3ZZJPepHij3ptyLIExYOYGDaQedDkepiKaJXSkvZs2CLl3gkkvgm2+gfHn4wx9g82Z45hmnows/LWu05JKml5Brcnlv2XtOh6NURNPErso8Y2DdOti6NX/Zr7/C0qW2OdhnnrH31l97DRo1ci7OcPdwt4cB+Ovcv7LnhLbgo1SgaGJXZdK+fTBpEtx+OzRoAMnJ8M9/5q+/4QZ45x347Tf7jnr16o6FGjEGNR9E38Z9OZ19mhd/etHpcJSKWJrYVZny+uv2fnmdOnDzzTBhAuzaZV9Zq1Ahv1zNmjBiBCQkOBdrpBERXu73MgBvpb7F+oPrHY5IqcgU43QASp2rnBzYscO+V75tG2zaZO+Fb95sp2fNgg4dbNlNm+wT7gkJ9hW1fv3s0LZt2epa1Skp9VK4pd0tfLzqY2788kZ+vvNnysWUczospSKKJnbluKwsOHECjh+H+vVtK24A8+bBhg12eeHh/PNhzBhbbtcuaNKk6O2vW5ef2O+6C669Frp1K7s9rzntn4P+yY/bf2T53uW88OML/PWivzodklIRRZzqdSklJcWkpqY6Unc4MyZ/iIrKP8vMyIDMTMjNPXOIicm/R2yMPas1xnvZ+vWhalVbdvdue9ablQXZ2QXHxsB11+XH9emnsH+/jSEjww6nT9txnz42mQKkpsK99xZM0unp+dvZuBGaN7fTw4bB5Mnefw4XXgg//WSnMzPt++VJSdCwoZ12D82bQ7169melQsecrXO4eOLFxEbF8vOdP5NSL8XpkJQKChFZaowJ6BfesTP21avtL1733xXucWysfSLZbcAAe8bl+feHe/r22+HZZ+10aioMGXLm9tzjOXPsA1IADz4In33mfZspKfb1JrAJo1Yt79sDGDfO3qcF+6DVqFHe64+NtQnMLSXF7r+3+u+7L/8hrgULbAIrXK/bqlX2EjLAPffY+8XedOtmtwU2KTdt6r0cwIcf2p8rwBdf2Fe8vImNLZjYn38e1qwpervuxJ6dbZ829xQVBZUq2SEzM3/5xRdDxYr56zyHevXyy8XF2UvxKnxc1OQiHuj6AG8sfoOR00ey4M4FiN4LUcovHEvsmZmwZcuZy92XYd327Cn6l/bRo/nTWVnF94GdnZ0/feIEHCyijQzPBAxw7FjR28zKKjh96pT3cjk5BeczMwsmME+5uQXni7qgEhVVsGy5cpCYaJcXHtxn4O7PNWrkvVxUFFSunF82Kcn+YREbmz/ExNhxXFzBeG66yXaKEhdnL3GXK5c/bt8+v1ybNrB4ccEkXb689/vbI0bYQUWm5y5+jkmrJ7Fo1yKmbZjG0FZDnQ5JqYjg2KX4tm1TzFdf2Uvx7l/q7rHn/dI9e/ITqOcvfxH7FLM7aWVm5ifrwtsTgWrV8v9oOH7cXiL2ts2YmPzkZkzBxF54u+XK5Sc4z2TtrX7Pp6vddXurPyoKoqPz6/c8PIW3q1S4e+HHF3hi9hPUKF+D1fetpk5iHadDUiqggnEpXu+xK6Uck52bzYBPBjB762wuP+9ypg2bppfkVUQLRmLXR4qUUo6JiYphwpUTqFKuCl//+jXjlmjXrkqdK03sSilHJVVK4pmLbAP8I2eMZMqaKQ5HpFR408SulHLcyK4j+VPPPwFw57Q7+WHbD84GpFQY08SulAoJz1z0DMPbDOdk5kkGfjKQaRumOR2SUmFJE7tSKiRER0XzydWfcH/K/WTkZDDsi2Gk7tYHbJUqLU3sSqmQESVRjB08llvb30p6djq9P+zNl2u/dDospcKKJnalVEgREcZfPp7bO9xOenY61/7rWl5Z8IrTYSkVNjSxK6VCTnxMPB8M+YCXLn0JQXh01qOMWzIOp9rdUCqcaGJXSoUkEWH0haN54ZIXMBh+P/33DP50MDuOaccAShVHE7tSKqQ91vMxPr36UyrHV2bmppk0+2czHpr5ECczTzodmlIhSRO7UirkDW87nNX3rWZIyyFk5Wbx+qLXaTm2Je8te08vzytViCZ2pVRYaFC5Af8Z9h8W3bWIVjVasfvEbu7+7910Ht+Zqeumkp2bXfJGlCoDNLErpcJK1/pdWXXvKj4a+hHx0fEs37ucq6dcTa8Pe7HuwDqnw1PKcdq7m1IqbB1KO8SElRP4x4J/sPvEbgCuOO8KBrcYTItqLWhRvQVJlZKIEj2HUaEhZLptFZGBwOtANPCeMebFQuvjgYlAZ+AQcIMxZltx29TErpTyl90ndvPM3Gd4f/n7ZOVmnbG+b+O+dK7bmQ51OtC2VlsaV2lMhbgKxETFOBCtKstCIrGLSDTwK9AP2AksAYYbY9Z6lLkfaGeMuVdEhgFXGWNuKG67mtiVUv62+8RupqyZwsp9K9l4aCPzd8wvtnx8dDwV4iqQGJdItYRqeUNCTAIxUTHERsUSExVTYIiNLrisNGXcfc0L4tM02Nf+3NNVylWhfZ32AfnZqeAIRmL35c/VrsAmY8wWV1CfA0OBtR5lhgJjXNNfAGNFRIw+rqqUCqJ6FevxULeH8uaNMew8vpMVe1ewYu8Klu9dzur9q9l7ci+nMk+RkZNBRnoGh9MPs/3Ydgcj903vRr2Ze/tcp8NQIc6XxF4f8GwRYidwQVFljDHZInIMqA4c9CwkIiOAEa7ZDBH55WyCDhM1KLT/EUb3L3xF8r5BBO/fPOYhv5OI3T+XSN+/loGuwJfELl6WFT4T96UMxpjxwHgAEUkN9OUIJ+n+hbdI3r9I3jfQ/Qt3ZWH/Al2HL4+K7gQaeMwnAbuLKiMiMUBl4LA/AlRKKaWU73xJ7EuAFiLSRETigGHAtEJlpgG3uaavBWbr/XWllFIq+Eq8FO+6Zz4S+Ab7utsHxpg1IvI0kGqMmQa8D3wsIpuwZ+rDfKh7/DnEHQ50/8JbJO9fJO8b6P6FO92/c+RYAzVKKaWU8j9tjkkppZSKIJrYlVJKqQhyToldRK4TkTUikisiKYXW/UlENonIBhEZ4LF8oGvZJhF53GN5ExFZJCIbRWSy60E9RCTeNb/Jtb5xSXUEgiuGFa5hm4iscC1vLCLpHuve9vhMZxFZ7Yrxn+JqUkpEqonIt659/VZEqrqWi6vcJhFZJSKdArlPhfZvjIjs8tiPwR7rAn4sg7B/L4vIetfPdaqIVHEtj4jj56uijlmoEZEGIjJHRNa5fsf8wbU84N/TIO7jNtf3a4W4XoE6m++WiNzmKr9RRG7zWO71+xukfWvpcYxWiMhxEXkonI+fiHwgIvvFo/2VYByvouooljHmrAegNfZl+x+AFI/lycBKIB5oAmzGPngX7ZpuCsS5yiS7PjMFGOaafhu4zzV9P/C2a3oYMLm4Os5lf0qx3/8A/uKabgz8UkS5xUB37Hv+M4BBruUvAY+7ph8H/uaaHuwqJ0A3YFEw9sdV9xjgUS/LA34sg7R//YEY1/TfPH7mEXH8fPwZFHnMQm0A6gKdXNMVsc1aJwfjexrEfdwG1Ci0rFTfLaAasMU1ruqarlrc99eh791eoFE4Hz+gN9DJ8/dFMI5XUXUUN5zTGbsxZp0xZoOXVUOBz40xGcaYrcAmbNO0ec3TGmMygc+Boa6/TC7GNkcLMAG40mNbE1zTXwCXuMoXVUdAueq+HvishHJ1gUrGmAXGHpGJeN+nwvs60VgLgSqu7TgpGMcy4Iwxs4wx7g67F2LbYyhSBB0/T16PmcMxeWWM2WOMWeaaPgGsw7ZwWRR/fk+dVNrv1gDgW2PMYWPMEeBbYGAJ399guwTYbIz5rZgyIX/8jDHzOLN9lmAcr6LqKFKg7rF7a4a2fjHLqwNHPX7xupcX2JZrvbu52qK2FWi9gH3GmI0ey5qIyHIRmSsivVzL6rti8hZfbWPMHrC/wIBaHp9xYp/cRrouG33gcbknGMcy2O7A/kXsFinHryShHp9XYm/ZdAQWuRYF+nsaLAaYJSJLxTa3DaX/bhW3vKjvb7ANo+CJUKQcPwjO8SqqjiKVmNhF5DsR+cXLUNxf+kU1MVva5WezrbPm474Op+CXdA/Q0BjTEXgE+FREKp1lfH7fpwIbL37/3gKaAR2w+/SPEmLy57H0C1+On4g8CWQDk1yLwub4+UGox3cGEUkEvgQeMsYcJzjf02C50BjTCRgE/F5EehdTNhz3D9d97yHAv1yLIun4FcfR/fGlgZpLz2K7xTVD6235QeylihjXX2Ce5d3b2ikFm6v1panbUilpX131X43td979mQwgwzW9VEQ2A+e54vO83OsZ3z4RqWuM2eO6BLPftdzv++TJ12MpIu8CX/sQk7+OpV/4cPxuAy4HLnFd7gqr4+cHoR5fASISi03qk4wx/wYwxuzzWB+o72lQGGN2u8b7RWQq9rJzab9bO4G+hZb/QPHf32AaBCxzH7dIOn4uwTheRdVRpEBdip8GDBP7FHQToAX2wQCvzdO6fsnOwTZHC7Z52v94bMtbc7VF1RFIlwLrjTF5l0xEpKbYPusRkaauOLa4LpmcEJFurvtBtxaxT4X39VaxugHH3JdgAq3QveCrAPeTn8E4lgEnIgOBx4Ahxpg0j+URcfx85Evz0CHB9TN/H1hnjHnFY3kwvqcBJyIVRKSiexr7cOcvlP679Q3QX0Squi5r9we+KeH7G0wFrnBGyvHzEIzjVVQdRTPn9pTgVdi/NDKAfa4A3euexD7NuAGPpzGxTwv+6lr3pMfyptgDuQl72Sbetbyca36Ta33TkuoI1AB8BNxbaNk1wBrs05rLgCs81qVgv7ibgbHkt/RXHfge2OgaV3MtF+BNV/nVeLxpEIR9+9hV5yrXF6luMI9lEPZvE/be1grX4H46PyKOXyl+Dl6PWagNQE/spchVHsdscDC+p0Hav6au79xK1/fvybP9bmGfGdnkGn5X0vc3iPtYHjgEVPZYFrbHD/sHyh4gC5v37gzG8SqqjuIGbVJWKaWUiiDa8pxSSikVQTSxK6WUUhFEE7tSSikVQTSxK6WUUhFEE7tSSikVQTSxKxUAImJ8GLa5yn4kIjtL2GRQiO19y7gaEPLb9nwo19dVb19/1KtUWeaX/7xKqTN0LzQ/FfvO8hiPZRlBi0YpVWZoYlcqAIzt0SmPiGQABwsvP1ciEm9ss7hKKQXopXilQoaIdBSRH0UkTUQ2isi9hdbf7rpc3VtE/iUiR8nv8QwR6SMi34vICRE5JSLfiEibQtsYICLzReSYiJwUkQ0i8hcv4TQRkf+5yvwmIn8RkahC22opIlNF5KiIpIvIQlfTvSXtZ00R+VREjrs+OxGoUqofllKqSJrYlQoNlYBPgU+w/S8vAd4SkYu8lJ0EbMW2k/04gIhchm1u8iRwM3AjUBH4UUQauMo0xTbjuQ24Advr1itABS91TAVmY/t+/gr4K/ntVSMi9YCfgPbASOB64CjwPxEZVMK+/hvbGc8TrjiygTdK+IxSykd6KV6p0FARuN8YMwdAROZhO4gYju3swtMXxpg/Flr2OjDXGOPZRe0cYAswCngI6ATEAfcZ2wUq2OTtzT+MMR+6pr8TkYtdsbiXPQJUBbobYza56psOrAWeo2B/93lEpB+2HfjhxpjPXYu/EZEZFOzdSil1lvSMXanQkOZO6pDXnexGoKGXslM9Z0SkBbaP60kiEuMegDRgAeDu53sFtgOLz0XkWhGpVUw8/ys0/0uhWHoDC91J3RVzDrajjA5i+7T3pjuQg+2O1dPnXsoqpc6CJnalQsMRL8sysD3iFVa4K1h3gn4fm7g9h8uxvUPhSsIDsP/vPwb2isgiEenjpY7DJcRSzUscAHuxPVtV9bIOoC5wxBiTVWj5Pm+FlVKlp5filQo/hd8LP+Qa/wn4zkv5zLwP2qsCc0QkHrgQeBp7X7yxMeZgKWI4DNTxsryOK77Cfxi47QGqikhsoeReuxR1K6WKoYldqfC3AftA3PnGmBd9+YDrUv9sEUkE/gM0AUqT2OcCD7n+INgGICLR2IfhlhtjThTxuQVANHANBS+/DytF3UqpYmhiVyrMGWOMiPwe+I+IxAFTsEm6NtAD2G6MecX1+lxvYDqwA6iBPcvfjb2HXhqvArcD34rIU8Bx4H7g/blKnQAAALdJREFUPOCyYmL9VkR+At4RkRrY5whuANoU9RmlVOnoPXalIoAxZjo2aVcA3gO+AV7CXhpf4Cq20rX+BWAWMBb72tzFxpj0Uta3G/t0+xrgLeAL7H33y4wxM0v4+NXYPy5eACZjTzBGlqZ+pVTRxJgSm3FWSimlVJjQM3allFIqgmhiV0oppSKIJnallFIqgmhiV0oppSKIJnallFIqgmhiV0oppSKIJnallFIqgmhiV0oppSLI/wPH10gOO5LhFgAAAABJRU5ErkJggg==\n",
+ "text/plain": [
+ "<Figure size 576x288 with 1 Axes>"
+ ]
+ },
+ "metadata": {
+ "needs_background": "light"
+ },
+ "output_type": "display_data"
+ }
+ ],
+ "source": [
+ "def plot_precision_recall_vs_threshold(precisions, recalls, thresholds):\n",
+ " plt.plot(thresholds, precisions[:-1], \"b--\", label=\"Precision\", linewidth=2)\n",
+ " plt.plot(thresholds, recalls[:-1], \"g-\", label=\"Recall\", linewidth=2)\n",
+ " plt.xlabel(\"Threshold\", fontsize=16)\n",
+ " plt.legend(loc=\"upper left\", fontsize=16)\n",
+ " plt.ylim([0, 1])\n",
+ "\n",
+ "plt.figure(figsize=(8, 4))\n",
+ "plot_precision_recall_vs_threshold(precisions, recalls, thresholds)\n",
+ "plt.xlim([-100000, 100000])\n",
+ "plt.show()"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Likewise, we can plot the precision against the recall. The area under this curve indicates the performance. A good classifier will have a curve that sits in the top-right corner.\n",
+ "\n",
+ "We have defined a function to plot the curve."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 5,
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "image/png": "iVBORw0KGgoAAAANSUhEUgAAAf4AAAF8CAYAAAAuF9n2AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAgAElEQVR4nO3deZgU1bnH8d/LsAzgKMpyUUARRQVR0Iy4oEHjhksguIJ6FUUJrhFR44YLGjXifl1Q4xKNC+7B3biiCAZQISKigNsACqiAsi/n/nF6Uj1DD3TPdFd1d30/z9NPnaqurnqnYnirzjl1jjnnBAAA4qFe1AEAAIDwkPgBAIgREj8AADFC4gcAIEZI/AAAxAiJHwCAGAk18ZvZA2Y2z8w+reF7M7PbzWyGmU0xs13DjA8AgGIX9hP/Q5J6ref7QyR1THwGSbo7hJgAAIiNUBO/c26MpJ/Ws0sfSQ87b7ykZma2eTjRAQBQ/PKtjb+NpO+S1isS2wAAQBbUjzqAaizFtpRjCpvZIPnmAEnNf7PFFu21OXUDAIAYmDRp0gLnXMva/DbfEn+FpHZJ620lzUm1o3PuXkn3SpJZuRs8eKKGDct9gAAARM3Mvqntb/Otqn+0pBMTvfv3kLTIOTc36qAAACgWoT7xm9njkvaV1MLMKiRdIamBJDnnRkp6WdKhkmZIWirp5DDjAwCg2IWa+J1z/TfwvZN0ZkjhAAAQO/lW1Q8AAHKIxA8AQIyQ+DO0dq3kUr5gCABA/iPxp2nKFOnCC6WttpJatJDmzUu9308/SWvWhBsbAADpIvGvx6pV0mOPSbvtJnXtKo0YIVVU+OR+xBHSihX+6X/SJOnyy6WddpKaN/ffTZ8urV4tTZggvfJKOLUEixdLU6f68wIAkEq+DeATqaVLpTvu8E/1ixZJV1/tE70kbbaZdOyx0iefSOPGSWPHSqWl0pZbSt9+W/U4o0f7T3WnnSZtvbX05ZfSyJFSw4Z++7x5/vg//SQ1auSbE1atkpo0kSZP9uf44Qfpo4+Cz4QJ/rdlZdIvv0ht2waxVnriCalzZ39DMG2a/12XLlL9+j4WSzVOIgCgqJkrggZrs3I3fHjdRu574w3puOOk+fOrbu/USRoyRDrhBKlxY588f/ObqvtssYXUp490wAHSgAE+Eadj8GDptdekr76qfdx1se220owZ0sknS2edJa1c6WssVqzwNyItW0oNGkQTGwCgZmY2yTlXXqvfxj3xr1kjDRsmXXdd1e3bbCNde6101FFSvWoNIr/8Il1xhbRkidSvn9SzZ7DPokXSwoX+BqFTJ6ljR+nNN6WDD05dO5CuTp2kXXf1n513lr75xj/ll5RIv/7qn+w7dJDmzJFOOcWfU/K1Ep07+1h//rnmvgk1Wb3ax9y6tb/xAQBEj8Rfy8S/ZIlP7K++6hP3VVf55NakiTRwYFAVn00//yz16iU1bSrtv7//NGvmq+M7dPBJdtttfS1Ax44+hhUr/P614VzVKv21a6VRo/xxf/hBOjMxXFK9ev679fn4Y6lbt9rFAQDIHhJ/LRL/4sXSYYdJ77/vq7RHjZL22y93MRaCyv8U1q71/QBSOeccqbxc2ntv318BABC+uiT+WHbuW7lS+sMffNJv21Z66y3/dB13lTUDJSX+JmDuXGmjjXxHxRNO8N/dfnuwf8eO0pgxvhZjyy1rXysBAAhPrBL/qlU+mV19tfT2277deswYnlxrsvnmfnn88dLGG0u9e/sxDBYs8Nu//DLYp9L48b5D4MYb+30bNPDNF1Om+OaEbt24yQKAKMUm8Tsnde/uX8eTfFv+6NEk/XT9/vdBU8DHH/tOhqnssUd6xzvlFN+xsmlT6eyzff+GVauoNQCAXIvNAD7PPhskfUm65RY/MA8yt8su/iZgzRrf67+yw6CUemyAzTZbd9sDD0h//7t0113+jYVGjXyzgpnfXjn6oXMMkQwA2RSLxL9qlXT++cF6s2bSoEHRxVMs6tXz/QHMpGOO8TcBy5b5RD1vnl9fvlz68ccggT/zjP/tpptKrVqlPu6AAb5zYY8evrmgXj1/jsrPsGG+n4ZzvpPmzz/7JoYZM4LzzJ7tm3FmzMj8FUYAKGaxqOp/7DHp6699uWdP6R//YNS6XCgp8R/JvylRuS3ZEUdUfYJfs8bXGCxZIp16anBjIEkffJD6PNdc4z+Z+vOf/U3EJpv4OOhrACCOiv51PuekHXf0Q9Y++KB/mkR+e/556YsvfD+CTp38E/u0af7ti/vvX3f/xo19TUNtNGjga4Qq3Xeff4Nh5kzfQbFdu9odFwByiff415P4331X2ndfP6zuV1/lZlAehGvJEt+E4JxvtqkcNXHmTP8036CBb0qo9OijPpl37ix99lntztmli3Trrb4fQtu2/m2G6iM6AkBYeI9/PR580C8HDCDpF4umTVP3/q+p6v744/0n2axZfgbFlSv9TcSYMb6jYU0+/dTPxZBKjx6+OalZM19Tsdlmvo/CZpv5GwUAyCdF/cS/dKnvQLZkiX/nfNtto4kPhaGyU2CrVj5x//vf0lNPSTffXLfj9u7tJ0JasEA66CDffEAfEwB1UZcn/qKurHzzTZ/0y8tJ+tgwM1+N37Chr8bfYw/pppuCNwUqP2vX+kmYjj46veOOHi317eunQt5qq+AthSOP9MvOnaWHH/a1BkuXBudZvdrXSABANhV1Vf8//+mXffpEGweKi5kfy+DJJ4NtixZJZWVBu79zvk/JgQf6ZoVUnn3WL6dNk046af3n7NvX3whssYW/mR00SNpnn7r/LQDip2gTv3PSSy/5cu/e0caC4rfJJlXXzfxohDNnVt2+apXvS/DNN9KkSb5vQTqee67q+j/+kXq/k0/2N7xnnx28ssh0ygCSFW0b/xdfSNtv79trv/+eNlUUhpkzpebN/SuMrVv75oZGjaSKCqm01C9ro2lT6bjjfE1Bly7+WAAKF736U6h8kvrtb0n6KBzbbOOXzZr5ZU2vHy5c6F9VXbLE1zYMHOibFWqqCViyxI9RcN99wbYDD/RTUQ8dyhsvQJwUbee+ysRPOyiKUbNmvu/KccdJhx3ma7UeeSR1J8THH5d69Vr3GP/6l3TJJb5GoXI45LIy6dprpR9+CP9vAhCOok3848b55d57RxsHEJXKToj9+kmvvBLcEIwdK91xR+rf/PqrdOmlvpnBTHrjDW4CgGJTlIn/l1/85CwNG/r2TACBvfaSzjzT3wSsWiWtWCFNmSI99JAfkTDZgQf6m4DSUmnkyEjCBZBlRZn4p0zxyx13pO0SWJ/69f3/R3bayb9SOGeOvyG48caq+61YIZ1+uq8F6N07eGMGQOEpysT/ySd+2a1btHEAhWroUH8DsGyZdNttVb974QXp8MP9TUD79tL//q/0xBN+ToS1ayMJF0AGijLxT57sl127RhsHUOhKS6VzzvE3AZMn+2aCZN98498k6N/fT4RUUuJvCIYNk955x9cMvPKKNH9+JOEDSKEoE//06X65447RxgEUk5139h0DnfPjDRx6qN/eo8e6+15zjX9V8PDD/X6tWvkbglNOkZ5+et2BjQCEpygT/5df+mVNs7UBqJsOHfzTvHPS++8HzQIXXFB1v623rrr+4IN+joNttw1eIXz+eeYkAMJUdIn/11+luXN9h6W2baOOBoiP0lLphhuqjiUwa5ZfVr4R0Lr1ur/r29ePJbDDDtLLL4cbMxBHRTNkrzRRY8f6oUm7dZM6dap51DMA0Zo71084tCGlpdIf/yj99a/+5gCAx7S8CT16+HeRJabhBfLZ5psHtQI//SSdf37q/ZYv928VlJZKEyaEGyNQrIoq8UvSrbf6ZfW2RQD5adNNpREj/E3A4sXS7Nm+yv+ii/yERZW6d/d9Ao4/vvaTFQEowsRfqV27qCMAkKmyMt8EcMgh0nXXSQsW+FEGkz32mP//t5mfgdPMv8Hz4YfRxAwUmqJN/G3aRB0BgGy44w5fG3D33et+98UXfvnZZ34K48o3Bf7yl3BjBApJ0SZ+evQDxWXw4KBfwPffS08+Kd1yi28CqO6yy4KbgI4d/X6rV4cfM5CPiqpXf7KZM/27xgDiYdYs6e9/l4YPT2//+fOlFi1yGxOQK/TqTyGdV4UAFI8OHaSrrvI1AhUV0p13Sj171rx/y5a+RuCww6QLL2T6YcRH0T7xF8GfBSBLFi6Uxo3zDwQbmrxr2DDpxBN5JRj5rS5P/CR+ALHjnHT22dKSJcHYH6nUqyc9+6zUp09ooQFpIfGT+AHUgXPSxRf7mQZnz069z7hx0m67+RkIgajRxg8AdWAmXX+97xtQOZpg9QmH9txTql8/eFvgtdekFSuiiReoC574AaAGzkm77ip98sn69zv/fKm8XDroID8SIZBrPPFX8+qrUUcAoBiYSR9/7G8AVq6Uxo9Pvd+NN0r9+kmbbeabBIB8VpSJ/4ADoo4AQLFp0EDaffdgEKG1a6VTT5U22qjqfnvtFTQHMM0w8lFRVvUXwZ8EoMC89JJ0+OGpv1u71t8IANlCVT8AROyww/xDx+rV0oABVb+rV0965JFIwgLWUXSJv3LSDgCIQkmJ9OCD/iZgn32C7SeeGDQBmPk+AUAUii7xMz4/gHwxZoz03nupvxs1yt8A7LKL9Pnn4caFeCu6Nv4i+HMAFKEvv/SDA02a5F//S+Wnn3gdEOmhjR8A8lzHjtK++0pDh0rLlklnnLFuDeVmm/lagCZNpIEDpU8/jSRUFDme+AEgQmvW+BEB1+err6T27UMJBwWioJ74zayXmU03sxlmdlGK77c0s7fN7GMzm2Jmh4YdIwCEpaQkeBvgrbf80MA771x1n6239jUB998fTYwoLqE+8ZtZiaQvJB0oqULSBEn9nXOfJe1zr6SPnXN3m1lnSS8759qv/7g88QMoLs5Jf/qT9H//V3X7rrv6fgKIt0J64u8uaYZzbpZzbqWkJyRVn/DSSdo4Ud5E0px0Dz53blZiBIDImUm33+5vAJLb+j/6yH83cGB0saGwhZ3420j6Lmm9IrEt2ZWSTjCzCkkvSzo73YO3bl3X8AAg/+y4o28K2G67YNsDD/gbgKVLo4sLhSnsxJ9q0MrqlfP9JT3knGsr6VBJj5jZOnGa2SAzm2hmE6t/BwDFpqREmj5dWry46vamTf0NwCWXRBMXCk/Yib9CUruk9bZatyp/oKQnJck5N05SqaQW1Q/knLvXOVde2zYOAChEZWW++n/w4Krbr7uO+QCQnrAT/wRJHc1sazNrKKmfpNHV9vlW0v6SZGad5BP//FCjBIA8d/fd/gbgueeqbq8cB2DKlGjiQv4LNfE751ZLOkvSa5KmSXrSOTfVzIabWe/EbkMlnWZmkyU9LmmAK4bBBgAgB/7wB38D0Capt9SyZVLXrtQAILWiGsCnCP4UAKi1igrphBOkd9+tup1pgYtPIb3OBwDIkbZtpXfe8TUAyW851avHREAIkPgBoAjNnSs1bhysd+rkn/qpGUXRJP4LLog6AgDIL0uXSg8/XHXbkCHRxIL8URRt/LvuWu4++ojX+QGgJslt/KtX+3EBULhi38Zfryj+CgDInU8+Ccr166/bARDxQcoEgBjo2lXq0CFY33df6dxzIwsHESLxA0BMzJzpe/1Xuu026dBD6fAXNyR+AIiRnj2lL74I1l95xTeXfvttdDEhXCR+AIiZjh396H5t2wbbttrKdwBcsya6uBAOEj8AxFBpqfTdd9JNN1XdXr++dPnlVP8XMxI/AMTYeef5IX332CPYdvXVvvr/7bejiwu5Q+IHgJgzk8aNk559tur23/2OWf6KEYkfACBJ6tvXV/E//XSwrWtX6ZproosJ2UfiBwBUceSR0i23BOvDhkllZdKCBdHFhOwh8QMA1nHuuVUT/a+/Si1bSt26RRcTsoPEDwBIqXlzX/U/bFiwbfJk6ZlnoosJdUfiBwCs1/Dh/om/0lFH+Q6BFRXRxYTaI/EDADaoadN1E327dnT8K0QkfgBAWtq08VX/l18ebBs2zD/9L1kSXVzIDIkfAJCRq66Spk2rum2jjaKJBZkj8QMAMrbDDv7p/5hjgm3Jo/8hf5H4AQC1NmpUUP7wQ2mvvaKLBekh8QMA6mTt2qA8bpy06abRxYINI/EDAOrETFq6NFhfuNBvQ34i8QMA6qxxY9/m36JFsO3gg6OLBzUj8QMAsmb+/KD8+uvShRdGFwtSI/EDALIquc1/xAiptFRasya6eFAViR8AkFVm0s8/B+srVkj160cXD6oi8QMAsq5ZM//kf8QRwbbvv48uHgRI/ACAnDCTnn46WN98c+mkk6KLBx6JHwCQM2bS3/4WrD/8sDRhQnTxgMQPAMixgQOlxYuD9e7dpblzo4sn7kj8AICcKyuTHnssWN9iC+nGG6OLJ85I/ACAUPTvL911V7B+wQW+6h/hIvEDAEJz+unSO+8E6yedJN16a2ThxBKJHwAQqp49pYqKYH3IED+zH8JB4gcAhK5NG2natGB9jz38QD/IPRI/ACASO+wgvfBCsF5aKv3wQ3TxxAWJHwAQmcMPrzqoT+vWVcf6R/aR+AEAkXrooaod/EpK/BS/yA0SPwAgcn/6kzRoULB+xhnRxVLsSPwAgLxwzz1Sw4a+PHJktLEUMxI/ACBv/Oc/Qfn116OLo5iR+AEAeWO77YLywQdLU6dGF0uxIvEDAPLKAw8E5S5dpKFDo4ulGJH4AQB55eSTpcGDg/Wbb5buvz+6eIoNiR8AkHfuvlt6//1g/dRTo4ul2JD4AQB5qUcP6euvg/V+/SILpaiQ+AEAeWurraSddvLlUaOkN96INp5iQOIHAOS1SZOC8oEHSiNGRBdLMSDxAwDyWoMGVafxvfBChvStCxI/ACDvtWlTNfnXI3vVGpcOAFAQ2rSRunYN1s8+O7pYChmJHwBQMD75RCot9eU77pB+/TXaeAoRiR8AUFDmzQvKZWXRxVGoSPwAgIJSViYdd1yw3qpVdLEUIhI/AKDgPPpoUJ4/n17+mQg98ZtZLzObbmYzzOyiGvY5xsw+M7OpZvZY2DECAPLf2rVB+ZZboouj0ISa+M2sRNKdkg6R1FlSfzPrXG2fjpIultTDObejpHPDjBEAUBjMpI4dfXnoUGnZsmjjKRRhP/F3lzTDOTfLObdS0hOS+lTb5zRJdzrnfpYk59w8AQCQwnvvBeXWraOLo5CEnfjbSPouab0isS3ZdpK2M7OxZjbezHqlOpCZDTKziWY2cf78+TkKFwCQz/7nf6SmTX158WJpzz2jjacQZJz4zewkM3s10QY/q9pn5oZ+nmJb9S4Z9SV1lLSvpP6S/mZmzdb5kXP3OufKnXPlLVu2zPTPAAAUieTX+8aPlx5/PLpYCkH9THY2s2GSrpL0qaRPJK3I8HwVktolrbeVNCfFPuOdc6skfWVm0+VvBCZkeC4AQAw0aSItXx4M7HPccdI++0ht20YbV77KKPFLGijpNufckFqeb4Kkjma2taTZkvpJOq7aPs/LP+k/ZGYt5Kv+Z9XyfACAGGjUSPrhB1/1L0nt2vGKX00yrepvLumF2p7MObda0lmSXpM0TdKTzrmpZjbczHondntN0o9m9pmktyVd4Jz7sbbnBADEQ6tW0mmnBevvvhtdLPnMXAa3RGb2oqQ3nXN59cZkeXm5mzhxYtRhAAAitny51LhxsF6sT/1mNsk5V16b32b6xH+upJPN7EQza2Fm9ap/ahMEAADZUFoqvf9+sD5qVHSx5KtMn/grx0mq6UfOOZdpv4E644kfAJCsXr3gab8Yn/rr8sSfaZIerpqTPgAAeeHLL6Vtt/Xlbt38dL7wMkr8zrkrcxQHAABZs802/jNzpjR5sn/qt1QjycRQrdvkzWwjM2tnZk2zGRAAANnwxhtBuU31MWJjrDYj9x1sZhMlLZT0taRFZvZvMzsw28EBAFBb7dsH1f1z50pjxkQaTt7IKPGb2cGSXpK0kaSrJZ0h6RpJZZJeJvkDAPLJtGlBuWfP6OLIJ5k+8V8p6XVJnZ1zVznn7km0++8o6V/yw/kCAJAX6teXLr00WC/GHv6ZyjTxd5WfMndt8sbE+l2SumUrMAAAsmH48KD8m99EF0e+yDTxr5C0cQ3flSnzSXsAAMipevWk7bf35f/8J9pY8kGmif8dSVcnJtn5LzPbUr4Z4O3shAUAQPbcdptfrl4dbRz5INPE/2dJm0iabmZjzGyUmb0r6UtJzRLfAwCQV8qTxrhLfs0vjjJK/M65LyTtLOl2SY0k7SqpVNJtkro5577MeoQAANRR8+ZB+cAD4/3kn/F7/M65uc65851zuzvnOiaWFzrn5uYiQAAAsuHFF4Py7bdHF0fUmE0PABALhx0mbbKJLw8dKv36a7TxRGWDY/Wb2VuSznDOfZ4or49zzu2fndAAAMiu99+XdtrJl8vK4vlefzpP/MnTGtRLrNf0oQYBAJC3unSRhgwJ1lfE8CV0c0Vwu1NeXu4mTpwYdRgAgALgnH+3v9KqVX6Ev0JiZpOcc+Ub3nNdPKEDAGLFTNpzz2C9QYPoYolCppP09DGzk5PWtzKzcWb2i5k9bWYbZT9EAACy64MPpNatg/UlS6KLJWyZPvFfJqll0vrNktpKulfSb+VH7wMAIO/NmROUN4rRY2umiX8bSVMkycwaSzpU0nnOuaGSLpHUN7vhAQCQG2bS1VcH6zfdFF0sYco08ZdKWpYo7yX/OuDrifXpkrbIUlwAAORc8pS9558fXRxhyjTxfy1p70S5j6RJzrlFifVWkhal+hEAAPnITJo1K1gfNy66WMKS6QsM90i60cz6Suom6fSk7/aU9Fm2AgMAIAxbJ803u9de0tq1/oagWGU6Sc9tkgZIGifpFOfcfUlfl0l6MHuhAQAQjgceCMpdu0YXRxgYwAcAAPkZ/H76yZdXrszv9/sZwAcAgDqaPTso33tvdHHk2gYTv5mtMbPuifLaxHpNnxjPcAwAKGSlpVKnTr581lnRxpJL6XTuGy6pIqlc+G0DAACkcO650h//GHUUuUUbPwAACcuXS40b+/K0adIOO0QbT01Ca+M3swZm1rSG75qaWR53hQAAYP1KS4PymWdGF0cuZdq5735J99Xw3T2JDwAABeuAA/zyrbeijSNXMk38+0r6Zw3fjZa0f52iAQAgYvclPd5+9VV0ceRKpom/laR5NXw3X9L/1C0cAACi1b59UN5998jCyJlME/88STvV8N1Okn6sWzgAAETv6KP9cv58aebMaGPJtkwT/4uShpnZzskbzWwnSZdKeiFbgQEAEJUHkwagP/zw6OLIhUwT/+WSFkqaZGYfmNmTZjZW0kfyM/Ndlu0AAQAIW9Om0tln+/Lnn0tF8Ob7f2U6Sc8CSbtJuk6Syc/QZ5L+Imm3xPcAABS8K64IyueeG10c2cYAPgAA1CB5et4lS6QmTaKLJVnok/SYWQszO9zMTjKzzRLbSs2MSX8AAEVjxoygPGhQdHFkU6Yj95mZjZAfu3+0pAcktU98/U/5Dn4AABSFbbaRdtvNlx99NNpYsiXTJ/SLJZ0lP1nP7vLt+5VekFRkfR8BAHH39NNB+ZtvoosjWzJN/KdKGu6cu1a+J3+yGZK2yUpUAADkiS23DMrJr/kVqkwTfxtJ42v4bqWklBP4AABQyDbZxC+vuiraOLIh08Q/W1KXGr7rKqkIRzUGAMTdc88F5Zdeii6ObMg08T8l6XIz65G0zZnZdpKGSnoia5EBAJAn9tsvKB95ZHRxZEOmif9KSZ9LGiPpy8S2pyT9J7F+fdYiAwAgj9x/v1+uWBFtHHWV6ch9y+Sn5h0g6QNJb0iaIGmQpAOdcyuzHB8AAHnh2GOD8rJl0cVRV/XT3dHMGkg6VNIU59wjkh7JWVQAAOSZpknd1++5p3CH8U37id85t0rSkwoG7AEAIFZKS/1yyJBo46iLTNv4Z0lqlYtAAADIdyNHBuVFi6KLoy4yTfw3SLrUzFrmIhgAAPJZcjt/s2bRxVEXabfxJ/xO0maSvjKz8ZLmSkqe3s85507KVnAAAOST0lLpkkuka6/168uWSY0bRxtTpjKaltfMvlbVRF+dc851qGtQmWJaXgBAmCqn6333Xem3v43i/LWfljfTJ/5ySb8655bX5mQAABSDHj2ksWOlnj2lDJ6f88IG2/jNrMTMrjSzhZJ+kLTYzJ4xs1q1bphZLzObbmYzzOyi9ex3lJk5M6vVHQ0AALmyzz5BedWq6OKojXQ69w2WdLn8bHw3SvqnpD6Sbsn0ZGZWIulOSYdI6iypv5l1TrFfmaRzJH2Y6TkAAMi1yy4Lyg89FFkYtZJO4j9N0n3Oud855/7snDta0pmSTjCzhhmer7ukGc65WYlR/p6Qv4mo7mr5NwhoUgAA5J2mTaXdd/flK66INpZMpZP4O8iPx59slKQSSVtleL42kr5LWq9IbPsvM9tFUjvn3IvrO5CZDTKziWY2cf78+RmGAQBA3Rx1lF/OnRttHJlKJ/FvJGlxtW2/JJZlGZ7PUmz7b7cIM6sn34QwdEMHcs7d65wrd86Vt2zJsAIAgHCdeGLUEdROur3625hZ8mt6JUnbFybv6JybtZ7jVEhql7TeVtKcpPUySV0kvWP+XYnWkkabWW/nHO/rAQDyRvIz5+zZUps2Ne+bT9JN/E/XsP35FNtKUmyrNEFSRzPbWtJsSf0kHVf5pXNukaQWletm9o6k80n6AIB8Y0l12LfdJt1wQ3SxZCKdxH9ytk7mnFttZmdJek3+BuEB59xUMxsuaaJzbnS2zgUAQFhGjCicxJ/RyH35ipH7AABReOIJqX9/X16wQGrePJzz1mXkvkwn6QEAAAn9+gXl/fePLo5MkPgBAKiD00/3y8mTC2MUPxI/AAB1cOedQfmRR6KLI10kfgAA6sBMatTIlwcOjDaWdJD4AQCoo5tvjjqC9JH4AQCoowEDgvKUKZGFkRYSPwAAddSkSVDu2jW6ONJB4gcAIAuSB/DJ57njSPwAAGTB+ecH5XyewIfEDwBAFphJJycGuX/11WhjWR8SPwAAWXLllUE5X6v7SfwAAGTJllsG5VGjootjfUj8AABk0d57++XIkfQC1CMAAA4BSURBVNHGURMSPwAAWbT99n45dWq0cdSExA8AQBZddllQfuWV6OKoCYkfAIAsat8+KN99d2Rh1IjEDwBAllW+0z95crRxpELiBwAgyyrH7v/2W2nOnEhDWQeJHwCALOvUKSi3aRNdHKmQ+AEAyLJ69aTbbgvWx46NLpbqSPwAAOTAOecE5UMOiS6O6kj8AADkyDPP+OUvv0irVkUbSyUSPwAAOdK3b1D+29+iiyMZiR8AgBwxk7p18+Uzzog2lkokfgAAcui886KOoCoSPwAAOXTMMUH5iy+ii6MSiR8AgBxq1Ehq2tSXL7442lgkEj8AADnXr59fPvustHZttLGQ+AEAyLHkwXwmTIguDonEDwBAzjVtKpWU+PLzz0cbC4kfAIAQ7LefX15/fbRxkPgBAAjB8OFRR+CR+AEACEH37kF52bLo4iDxAwAQgso2fkmaNy+6OEj8AACEZIcd/HLOnOhiIPEDABCSBQv88qyzoouBxA8AQEguvdQvP/pIci6aGEj8AACEZPDgoPzYY9HEQOIHACAkpaXSTjv58jXXRBMDiR8AgBAdcYRfzp4dzflJ/AAAhOjYY/3yl1/8J2wkfgAAQrTDDpKZLz/3XPjnJ/EDABAiM+moo3z5pJPCPz+JHwCAkFVW90vS66+He24SPwAAITvySKllS18++OBwz03iBwAgAjffHJTDHMyHxA8AQAROOCEoP/NMeOcl8QMAELGxY8M7F4kfAICI9O/vl7feGt45SfwAAERkwICgvHBhOOck8QMAEJGDDgrKm24azjlJ/AAARCh5xr4lS3J/PhI/AAARuvvuoPz557k/H4kfAICIVVbzk/gBAIiBrbbyy4suyv25SPwAAESsVy+/rKjI/blCT/xm1svMppvZDDNb597GzM4zs8/MbIqZvWlmW4UdIwAAYTrxxKCc6+F7Q038ZlYi6U5Jh0jqLKm/mXWuttvHksqdcztLelrSDWHGCABA2HbYISgvW5bbc4X9xN9d0gzn3Czn3EpJT0jqk7yDc+5t59zSxOp4SW1DjhEAgFCZBbP15bq6P+zE30bSd0nrFYltNRko6ZWcRgQAQB6YP98vR47M7XnCTvyWYlvK1gwzO0FSuaQRNXw/yMwmmtnE+ZVXCwCAAtW9u1/ecktuzxN24q+Q1C5pva2kOdV3MrMDJF0qqbdzbkWqAznn7nXOlTvnyltW1o8AAFCg/vrXoLx8ee7OE3binyCpo5ltbWYNJfWTNDp5BzPbRdI98kl/XsjxAQAQib33DsoTJuTuPKEmfufcaklnSXpN0jRJTzrnpprZcDPrndhthKSNJD1lZp+Y2egaDgcAQNGoXz9I/kcdlcPz5O7QqTnnXpb0crVtlyeVDwg7JgAA8sFuu0nvvy/Ny2F9NyP3AQCQJypn6mvYMHfnIPEDAJAn2iW6v69cKa1enZtzkPgBAMgTpaVBedq03JyDxA8AQJ4wk5o29eXHH8/NOUj8AADkkc6JGWw++CA3xyfxAwCQR04/3S/ffTc3xyfxAwCQR3r0CMq//JL945P4AQDII9ttF5QHDsz+8Un8AADkmSZN/HLq1Owfm8QPAECeuesuv/zss+wfm8QPAECeOewwvzSTXMrJ62uPxA8AQJ5p0UIqKfFJf+HC7B6bxA8AQB5as8Yvs93Bj8QPAEAeqqzuf+657B6XxA8AQB66/vqgvHZt9o5L4gcAIA8lv8//xz9m77gkfgAA8lDDhtLee/vyW29l77gkfgAA8tTgwX45a1b2jkniBwAgT3Xrlv1jkvgBAMhT224blBcvzs4xSfwAAOSpRo2C8lNPZeeYJH4AAPJYmzZ+OW5cdo5H4gcAII/9/vd+ef/92TkeiR8AgDx29NHZPR6JHwCAPLbHHkF5yJC6H4/EDwBAHmvSJCjfemvdj0fiBwAgz734YlB+/vm6HYvEDwBAnjvsMKl5c1/u27duxyLxAwBQAN54IzvHIfEDAFAAunWTTj+97sch8QMAUCBuvrnuxyDxAwBQIEpLpWXL6nYMEj8AAAWktLRuvyfxAwAQIyR+AABihMQPAECMkPgBAIgREj8AADFC4gcAIEZI/AAAxAiJHwCAGCHxAwAQIyR+AABihMQPAECMkPgBAIgREj8AADFC4gcAIEZI/AAAxAiJHwCAGCHxAwAQIyR+AABihMQPAECMkPgBAIgREj8AADFC4gcAIEZI/AAAxEjoid/MepnZdDObYWYXpfi+kZmNSnz/oZm1DztGAACKVaiJ38xKJN0p6RBJnSX1N7PO1XYbKOln59y2km6R9NcwYwQAoJiF/cTfXdIM59ws59xKSU9I6lNtnz6S/p4oPy1pfzOzEGMEAKBohZ3420j6Lmm9IrEt5T7OudWSFklqHkp0AAAUufohny/Vk7urxT4ys0GSBiVWV5jZp3WMDevXQtKCqIOIAa5z7nGNc49rnHvb1/aHYSf+CkntktbbSppTwz4VZlZf0iaSfqp+IOfcvZLulSQzm+icK89JxJDENQ4L1zn3uMa5xzXOPTObWNvfhl3VP0FSRzPb2swaSuonaXS1fUZLOilRPkrSW865dZ74AQBA5kJ94nfOrTazsyS9JqlE0gPOualmNlzSROfcaEn3S3rEzGbIP+n3CzNGAACKWdhV/XLOvSzp5WrbLk8qL5d0dIaHvTcLoWH9uMbh4DrnHtc497jGuVfra2zUogMAEB8M2QsAQIwUVOJnuN/cS+Man2dmn5nZFDN708y2iiLOQraha5y031Fm5syM3tG1kM51NrNjEv89TzWzx8KOsdCl8e/Flmb2tpl9nPg349Ao4ixkZvaAmc2r6ZV1825P/G8wxcx23eBBnXMF8ZHvDDhTUgdJDSVNltS52j5nSBqZKPeTNCrquAvpk+Y13k9Sk0T5dK5x9q9xYr8ySWMkjZdUHnXchfZJ87/ljpI+lrRpYr1V1HEX0ifNa3yvpNMT5c6Svo467kL7SPqtpF0lfVrD94dKekV+DJw9JH24oWMW0hM/w/3m3gavsXPubefc0sTqePmxGJC+dP47lqSrJd0gaXmYwRWRdK7zaZLudM79LEnOuXkhx1jo0rnGTtLGifImWnfcFmyAc26MUoxlk6SPpIedN15SMzPbfH3HLKTEz3C/uZfONU42UP5OE+nb4DU2s10ktXPOvRhmYEUmnf+Wt5O0nZmNNbPxZtYrtOiKQzrX+EpJJ5hZhfzbXGeHE1qsZPrvdviv89VB1ob7RY3Svn5mdoKkckk9cxpR8VnvNTazevKzUg4IK6Ailc5/y/Xlq/v3la+5es/MujjnFuY4tmKRzjXuL+kh59xNZran/BgtXZxza3MfXmxknPcK6Yk/k+F+tb7hflGjdK6xzOwASZdK6u2cWxFSbMViQ9e4TFIXSe+Y2dfybXaj6eCXsXT/vfinc26Vc+4rSdPlbwSQnnSu8UBJT0qSc26cpFL5cfyRPWn9u52skBI/w/3m3gavcaIa+h75pE+baObWe42dc4uccy2cc+2dc+3l+1H0ds7VelzumErn34vn5TurysxayFf9zwo1ysKWzjX+VtL+kmRmneQT//xQoyx+oyWdmOjdv4ekRc65uev7QcFU9TuG+825NK/xCEkbSXoq0W/yW+dc78iCLjBpXmPUUZrX+TVJB5nZZ5LWSLrAOfdjdFEXljSv8VBJ95nZEPnq5wE8jGXGzB6Xb45qkegrcYWkBpLknBsp33fiUEkzJC2VdPIGj8n/BgAAxEchVfUDAIA6IvEDABAjJH4AAGKExA8AQIyQ+AEAiBESPxADZjYgMdNf5Welmc00s2vNrDTi2L42s4eS1itjbR9ZUEARK5j3+AFkxdHyI32VSeor6eJEmTHUgZgg8QPx8olzbkai/C8z6yhpoJn9ifHTgXigqh+It48kNVbS+OmJIVgfNbP5ZrbCzD4xs77Vf2hmXc3sOTP70cyWmdl0M7s46fuDzOxlM5trZkvN7FMzG2pmJeH8aQBS4YkfiLf28tNX/yhJZtZO0oeS5kkaIj+u+rGSnjGzP1QOKWxm3SW9Iz9M6BD55oOOknZOOnYHSW9K+j9Jy+Vnc7xSUktJF+X0rwJQIxI/EC8liZkrK9v4j5R0rnNuTeL7K+Wn+eyZNG79a4kbguEKJmG5Uf5mYQ/n3NLEtreST5QYR1ySZH5ih/ckNZR0vpldQtMCEA0SPxAvn1dbv8s5d0fSei/5ST8WJW4QKr0maYSZbSxptaQekkYkJf11mNnm8jcSvSRtoar/3rSS9H1t/wgAtUfiB+Klr3y1fEtJ50k6w8w+dM49nPi+laQTE59UmktaKd8/qKKmk5hZPfnagS3kk//nkpZJ+oOkS+WnZwUQARI/EC+fVvbqN7O3JE2Rf5J/xjm3RL76/j1Jf63h93Pkp2BdK6nNes6zjXyb/v865/5RudHMfl/3PwFAXdCrH4gp59wKSRfIP+Wfkdj8qnwHvanOuYkpPisS1fvvSzrBzBrXcPgmieWqyg1m1kDS8Tn5YwCkjSd+IMacc6PNbIJ8h7s7JF0u6d+SxiTWv5a0qaQukjo4505J/PR8Se9KGmdmN8lX+3eQ1M05d7akaZK+kfQXM1sjfwMwJLy/DEBNeOIHcJn8U/9g59y38lX0kyVdK+lfku6W1FNJvfadcxPkO/h9J/+63svytQcVie9Xyrfnfy/pYUl3Shoj6fpQ/iIANTLnXNQxAACAkPDEDwBAjJD4AQCIERI/AAAxQuIHACBGSPwAAMQIiR8AgBgh8QMAECMkfgAAYoTEDwBAjPw/pgUhopn/96sAAAAASUVORK5CYII=\n",
+ "text/plain": [
+ "<Figure size 576x432 with 1 Axes>"
+ ]
+ },
+ "metadata": {
+ "needs_background": "light"
+ },
+ "output_type": "display_data"
+ }
+ ],
+ "source": [
+ "def plot_precision_vs_recall(precisions, recalls):\n",
+ " plt.plot(recalls, precisions, \"b-\", linewidth=2)\n",
+ " plt.xlabel(\"Recall\", fontsize=16)\n",
+ " plt.ylabel(\"Precision\", fontsize=16)\n",
+ " plt.axis([0, 1, 0, 1])\n",
+ "\n",
+ "plt.figure(figsize=(8, 6))\n",
+ "plot_precision_vs_recall(precisions, recalls)\n",
+ "plt.show()"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Finally we will plot the ROC curve. \n",
+ "\n",
+ "The ROC curve plots the *true positive rate* (another name for recall) against the *false positive rate (FPR)* \n",
+ "\n",
+ "Once again there is a trade-off: the higher the recall (TPR), the more false positive \n",
+ "(FPR) the classifier produces for the set of thresholds.\n",
+ "\n",
+ "We will define a function that plots the ROC curve."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 7,
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "image/png": "iVBORw0KGgoAAAANSUhEUgAAAf4AAAF8CAYAAAAuF9n2AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAgAElEQVR4nOzdd3iUVeL28e+ZSU8IhN6kIwoqoSxFBLuLKE2QYtm1UQQRVuxtdVdRsfCDFRUsr+7aBSki4NoVBKQFVFaR3msIAdJnzvvHTDAiCROY5JnM3J9ruZiWmRsXcuec5zznMdZaREREJDK4nA4gIiIi5UfFLyIiEkFU/CIiIhFExS8iIhJBVPwiIiIRRMUvIiISQcq1+I0xrxlj9hhjfizmeWOMmWSMWWeMWW2MaVue+URERMJdeY/4Xwe6l/D85UBz/6+hwIvlkElERCRilGvxW2u/AdJLeElv4N/WZzFQxRhTp3zSiYiIhL9QO8ZfD9ha5P42/2MiIiISBFFOBziGOc5jx91T2BgzFN/hABITE9udccYZZZlLRESCyFrwWovHa3/3Td63i7zvkaLP/ba7vO+xAo8lz+P9XWnYY25YIDffg8tlyMn3YC3ke71EuXxj3nyPN+h/rmAzQLTbhTHgLcgja98ObEEewD5rbY2Tec9QK/5twGlF7tcHdhzvhdbaqcBUgPbt29tly5aVfToRkTLi9VoO5xVgvb5C9P0Ca30Fl5ld4HudtUdLs/DXroO5RLuNv0hh24Es4qLdgO/rLb8VLUVuFz6eV+Bl/d7D1Kkcd/Rzj/2ctK0ZNKyaSIHXy66DOew7nEeNSrH+9/C91mLxevndYzkFHjbvz6J6UiwerxeP15KZUxCU/2aBFFi0//eEUryvy4DLGIwBYwx5BV6qJ8WQFBvFpv1ZnFY1nnPqVcFiqRQbTY1KsRR4LfVS4n/7Wn7/Hl6vJSHWTUpCDNWTYjHG9zlgcPlf4zJg8H1NXLSblIRooty+H1KWL19Ot27dqJ5SmTfeeIMePXpsLtV/rCJCrfhnA7cZY94FOgIHrbU7Hc4kIhWQtZZ8jyUjK488j5f0I3nke7wUeCz7DucdLcGjBVqkcNftOUxKYgx7MnPZeziXdbsPUz8lHk9hKXp9r1uyMZ3mNZOKvI+/XP2/53ss2zOyqRQXdfQxW+S1hY8VeCvGxdJ+3J75u/vbM7ID/tp9h3P/8FhctIucfC8NqyXg9peky5ijhekyhp93ZdK5abWjj/9WyoadB7OpUSmWVnWTiY1yHy1Qc0z5HsjK4/RalbBY6qckULNSLNFuF1EuQ2yUm8RY9+9KOpRYazHG0Lp1a0aMGMEdd9xBnTqntvTNlOfV+Ywx7wAXANWB3cDf8f9AZq19yfj+iz+Pb+V/FnCjtfaEQ3mN+EWcZf3l5fH/ysrzsO1AFtl5Hjz+6VyvtWw/kE1stBuv1/f6tbsPUSUhBo/Xy+ptB6lbOf535Yjvf0dHn4W3s3ILWLv7MA2qJlDg9bJq20GqJsSwKzMH8I2kKkiX/kFyXBQu128lVzgS3J2ZS9Maib6Cc/1Wfi7jG7FnZOXTqm4yUW7f127cd4S2DVJ8ZcZvhWiOjjD9j/HbqDbP46FB1YTfvXdhDozhSG4Bjaol4Ha5KPB4qZoYQ0yU6+j7FJYnFB3t+u6nJMQQ5TJEuVy4XJAQE4XbFVolG2qWLFnCHXfcwYwZM6hZs+bvnjPGLLfWtj+Z9y3XEb+1dvAJnrfAyHKKIxIWCku3wGM5mJ3/23HTY6aDt6RnAeDxcrSId2RkExPlwloo8Fq8Xsv/dmay42A2NSrF4fX+Vtr7j+SxIyObWslxRz/jpx2ZRLsN+R5nWrboiLOw9OG30o9yGQq8lpSEaOKj3ew5lEubBlWIcrlYv/cw7RqmEOV24TL4R5y+kvNYS2Z2PmfWSSYn30PVxFiqJvqmdAtHnm7Xb0VaJSG6SFEXlrW/MIHkuGjc7t/KsWgpmiJTvSpCAfB6vYwfP56HHnqIevXqsWvXrj8U/6kItal+kYiSlVdAZnYBGdl5bN6fRYHHUuD1ciingP2H88jzePD4p6DX7znM5z/voXpSLNFuw86DOSf+gDJw7OcWLf2YKN/0qdsYDuUWYAyc65+mdftHsVvSs2jXIAW32/e6XZk5nFOvMlFuFxlZeTSpkfi7kairSEGaIiPgAo8lPsZNlfhootwGt8tF9aQYEmOiSI6PPlqmIhXJrl27uP766/nss8+4+uqrmTp1KlWqVAnqZ6j4RY5h/aPZAq/lSG4BWXmeo9PY+R4v+w/ncTi3gB0Z2ezIyCYxNoq1uw8RH+NbTLV0UzqNqiUeHSl7vZDr8bJqawaNqiVQ4LVsOxD4sdFjHe9YaZTLV6y5BV5SEqJJiIk6Oh1cOLrMzfeS7/FyTv3KvyvijfuO0LahbxTs9r9PZnY+9VPiaVgt8ejrXMY3U1AlIYb4GDdu/3vERbuoXimW5Ljo46QVkdK49957WbhwIVOnTuWWW24pkx9ey/UYf1nRMX4pLWstGVn5pG3LYN3uw8REuUjbmsHyzQeOTomXp/op8Ww74Fuo1KlJNaJdhoRYNzWS4vyjWd/oODvfQ9MaSaQ2qEK025AcF02s/xiriFRMeXl5ZGRkULNmTfbt28eePXto2bJliV9TYY7xi5Slg1n57MrM4fOfd3Mop8A33ZyTz97DuazaepD4GDdRLsOGvUfIC+D83Zgo37HfoquOXS7D3kO51KkcR0yUi6qJMVRLjKVJjUQOHPGtHI6NdpHvsdROjjs66i48dlspLopE/6KmSnFRJMVGqbRFItj69esZPHgwbrebhQsXUr16dapXr16mn6nilwrD+heY7TqYw7YDWXy/8QCHcvJZuG4fO07heHdKQjRn1knmzDrJZOV5uKxVLS44vYYKWUTK1Ntvv83w4cNxu9288soruFzls5muil9CQoHHy67MHH7dfZhfdh9i7a5DxMW4eXvJFs6oXYmfdx0K+L0aVksg/Uge/dvVp1ZyHEmxUdSpHIcxUCs5jhi3i7hoN/WqxOPSKmoRKWdHjhzhtttu4/XXX6dLly689dZbNGzYsNw+X8UvZeZIbgHr9hzmpx2Z7PCfdrViywHqVI4np8DDis0HqFkpllXbDpb4Pscr/cbVE3EZqJYUS4+zanNWvcqcWSeZxFj9lRaR0GatZcmSJTz00EM8/PDDREWV7/ctfZeUU3IwK59fdh9i24Esth/I5oPl20q1OO54p6Q1qpZA69Oq0Lh6InUrxxPlNrQ+rQrVEmOoHB+tKXgRqXCstbz++usMHDiQpKQkVqxYQVxcnCNZVPwSEI/XsnpbBl+v3cviDftZvKGkqyv/0SVn1qJ+Sjw1KsWSV+BbLOcy/lPBkmKpXTmO+iml2U1bRKRi2LdvHzfddBMfffQRWVlZjBw50rHSBxW/lGBPZg7/mLOGOatPfLmERtUSqJcST5PqSZxTvzKXnFmLKgkanYtIZPvqq6+49tpr2bdvHxMnTmTEiBFOR1Lxi28KavP+LN5ZuoWdGTms33uYfYdz2Z35x41iqifFUis5lsvPqs0FLWrSqm6yyl1E5DheffVVhgwZQvPmzZkzZw5t2rRxOhKg4o84B47k8fnPe/jvT7vIzMkPaMr+qrb16N+2Pp2bVlPJi4gE6Pzzz2fo0KE888wzJCUlOR3nKBV/mLPWsml/FhM/W8vMtB0nfH2DqgkM6dqYPzWuSuX4aGonx6nsRUQCNGPGDD7++GNefvllmjVrxksvveR0pD9Q8YcZay3/XbObd7/fwpe/7C32dXUqx3H+6TXo0qw6DaslcFbdyjqnXUTkJGVnZzN27FhefPFF2rdvz8GDB4N+cZ1gUfGHgbwCL/N+3MmET9eyaX/xp9K1rl+ZF69rR90q8eWYTkQkvK1Zs4ZBgwbxww8/MHbsWMaNG0dMTIzTsYql4q9gCjxeZqXt4Ku1e/n2171kZOUf93XtGqZQOzmOQR1Oo0vT6hrNi4iUgfz8fK644gqOHDnC3Llzufzyy52OdEIq/gpgT2YOry3cxMc/7GBrevGXc60UF8U1HRsw6qLmJGkHOxGRMpOZmUliYiLR0dG8/fbbNGrUiDp16jgdKyBqhxC2ZMN+/r1oMx//8Mfz6C8/qzYt6yTT7fQaNK6RqGuhi4iUk8WLFzN48GBuvvlmHnzwQTp37ux0pFJR8YeYj1btYMJna9mw98gfnut2eg0ua1mLazs20Ep7EZFy5vV6GT9+PA8++CCnnXYal1xyidORToqKPwT8uvsQD8z8ke83/vGc+kqxUdzYpRFDz2+q6XsREYfs3LmT66+/ns8//5wBAwYwZcqUkF21fyJqEgftPZTLyLdW8P2m3xe+22V4+5aOtGmQQkxU+VyfWUREird582aWLl3Kyy+/zM0331yhZ11V/A44klvARc9+9bstcd0uw/WdGjLiwqbUrOTcxRtERMQnLy+PefPm0bt3bzp16sTmzZsr7Ci/KBV/Odp/OJc7P1j1h4117u7eghEXNHMolYiIHGvdunUMHjyYZcuWsXr1as4+++ywKH1Q8Ze5wp30hv1n+R+eu6ZjA/7RqxVRbk3ni4iEirfeeovhw4cTHR3Nhx9+yNlnn+10pKBS8ZehI7kFtP3np+QWeH/3+J8apfD6jR1I1GI9EZGQMnLkSF544QXOO+883nrrLRo0aOB0pKBT85SR95du5e7pq4/eP//0Gvylc0MuPrOWg6lERKQkbdu25eGHH+ahhx4iKio8KzI8/1QOe/Gr9Tw1/+ej9+/pfga3XtDUwUQiInI81lomTZpEtWrVuO6667j55pudjlTmdHA5yG59c/nvSv+TMd1U+iIiIWjfvn306tWLMWPGMHfuXKfjlBuN+IPEWsttb69k3o+7ALiwRQ2m/qU90Vq4JyIScr766iuuvfZa9u3bx6RJk7jtttucjlRuVPxBkFvgocWD84/eb1GrEv/vxg4OJhIRkeL8/PPPXHzxxTRv3pyPP/6Y1NRUpyOVKw1HT9HCdft+V/oXnVGTT/7WzcFEIiJyPNnZvqubnnHGGbzxxhssW7Ys4kofVPwnLSffw/lPf8m1ryw5+liHxlV57YY/OZhKRESO58MPP6Rx48YsX+7bU+W6664jKSnJ4VTOUPGfBK/X0u6fn7J5fxYA8dFu/jW4De8Pq1iXZhQRCXfZ2dmMGDGCfv36cdppp4XN7nunQsf4S8nrtfR8fgFH8jwA3H5xc+649HSHU4mIyLF++uknBg0axI8//sidd97J448/TkxMjNOxHKfiLwWP19LiwXkUeC0Aj/c9i2s7NnQ4lYiIHM8HH3zA7t27mTdvHt27d3c6TsjQVH8pDH558dHS/2fvVip9EZEQk5GRwcqVKwF48MEH+eGHH1T6x9CIP0CvLtjI9xvTAbjv8jO4vnMjZwOJiMjvLFq0iMGDB+P1elm3bh0xMTHUqqVt0o+lEX8A9h7K5Z9z1gAwrFsThp2vnfhEREKF1+vliSeeoGvXrhhj+OCDD3QsvwQa8Z/AzoPZdH7iCwDcLsM93c9wOJGIiBQ6dOgQV111FZ999hkDBw5kypQpVK5c2elYIU3FfwIDpiw6env2bV1wuYyDaUREpKikpCSqVq3KK6+8wk033YQx+h59IprqL8Gz//2Frem+nZ7G9T2bVnX1U6SIiNPy8vK4//772bRpE8YY3nvvPW6++WaVfoBU/MX4YdtB/vXFOgBu7NKIazo2cDiRiIisW7eOc889lyeeeILZs2c7HadC0lR/MYa/ufzo7YevbOlgEhERAXjzzTe59dZbiY6OZsaMGfTp08fpSBWSRvzHMWf1DrZn+Kb437ipg6aPREQc9uqrr3L99dfTpk0bVq1apdI/BRrxHyPf42X8/F8AaNugCuefXsPhRCIikcvj8eB2uxk4cCCHDh3itttuIypK1XUqNOI/xqy0HWxJzyLabfh/N3ZwOo6ISESy1jJx4kQ6dOhAVlYWSUlJjBkzRqUfBCr+Iqy13PnBKgBGX9ycyvHRDicSEYk8e/fupWfPnowZM4Z69eqRm5vrdKSwouIvYuh/flvQ17/daQ4mERGJTF9++SWtW7fm008/ZdKkScyaNYuUlBSnY4UVzZn4bdx3hE/X7Abgnu5nULtynMOJREQii7WWe++9l+TkZObOnUtqaqrTkcKSit/v8Y//B0CVhGhuvUB78YuIlJfNmzeTnJxMSkoK06dPJyUlhcTERKdjhS1N9eO7CM9n//ON9h/t1crhNCIikWP69OmkpqYyevRoAOrXr6/SL2MqfmDK1+uP3u7Vuq6DSUREIkN2djbDhw+nf//+NGvWjL///e9OR4oYKn5gZtoOAO76cwtt1iMiUsZ+/fVXOnTowJQpU7jzzjtZuHAhTZvqEGt5ifhj/N9vTGffYd+pIjd2aeRsGBGRCJCUlATAvHnz6N69u8NpIk/Ej/jHzfUt6qtRKZaEmIj/OUhEpExkZGTw2GOP4fF4qFOnDqtWrVLpOySii/9QTj5pWzMAeKrf2Q6nEREJT4sWLSI1NZVHH32U77//HgCXK6Lrx1Hl/l/eGNPdGPOLMWadMebe4zzfwBjzpTFmpTFmtTGmR1llefa/awGoWzmOC1vULKuPERGJSB6Ph3HjxtG1a1dcLhcLFiygc+fOTseKeOVa/MYYNzAZuBxoCQw2xhx7zdsHgfettW2AQcALZZVnycZ0AHqcXUeL+kREgmzIkCE88MADXH311axcuZKOHTs6HUko/8V9HYB11toNAMaYd4HewJoir7FAsv92ZWBHWQT5385M/rczE4Ch3ZqUxUeIiEQkay3GGIYMGcJ5553HjTfeqMFVCCnv4q8HbC1yfxtw7I+AjwD/NcaMAhKBS8oiyMerdwLQuHoiNZO1Pa+IyKnKzc3lvvvuA+C5556jc+fOmtoPQeV9jP94P/LZY+4PBl631tYHegD/Mcb8IacxZqgxZpkxZtnevXtLHaRwp74hXTXaFxE5Vb/++ivnnnsuEyZMID8/H2uP/dYuoaK8i38bUPSyd/X541T+zcD7ANbaRUAcUP3YN7LWTrXWtrfWtq9Ro0apQqRtzeDnXYeIi3bRp4126hMRORX/+c9/aNu2LRs3bmTGjBn861//0tR+CCvv4l8KNDfGNDbGxOBbvDf7mNdsAS4GMMacia/4Sz+kL8H05dsAOLdpdZ27LyJyCrZt28awYcNo06YNq1atok+fPk5HkhMo19az1hYYY24DPgHcwGvW2p+MMf8AlllrZwNjgZeNMX/DdxjgBhvkOaNp/uLv2bpOMN9WRCRibNy4kcaNG1O/fn2++eYbUlNTiYrSQKoiKPfz+K21c621p1trm1prH/c/9rC/9LHWrrHWdrHWtrbWplpr/xvMzz+SW0CB1wugc/dFRErJWsuECRNo0aIF77zzDgDt27dX6VcgEff/1Peb0sn3WJrVTKJKQozTcUREKoy9e/dyww03MHfuXHr27Mlll13mdCQ5CRG3Z+K3a/cBcG7Tag4nERGpOL766itat27NZ599xqRJk5g1axbVqun7aEUUcSP+r9buAeDCMzTNLyISqPT0dJKTk5k7dy6pqalOx5FTEFEj/twCDxv2HgGgdf0qDqcREQltmzdv5v333wfgqquuYvXq1Sr9MBBRxf/lz76zAmOjXFRN1PF9EZHiTJ8+ndTUVEaOHMmhQ4cAiInR981wEFHF/8JX6wDok1rP4SQiIqEpOzub4cOH079/f04//XSWLFlCpUqVnI4lQRRRx/j3ZOYC0KRGosNJRERCT25uLp06dWL16tXcfffd/POf/9QoPwxFTPF7vZZdmTkAnNf8DzsAi4hEvNjYWP7yl79w9tln61S9MBYxU/3bM7KP3j6jdnIJrxQRiRwHDhxg4MCBfP755wCMHTtWpR/mIqb4l25KB6B6Ugxuly4eISLy3XffkZqayocffsi6deucjiPlJGKKf+G6/QC0aZDicBIREWd5PB7GjRtHt27diIqKYuHChQwbNszpWFJOIqb41+89DEDnJtppSkQi24cffsgDDzzA1VdfzYoVK+jQoYPTkaQcRcTivpx8D2lbMwDo2bquw2lERJyxd+9eatSoQf/+/Zk3bx5//vOfMUaHPiNNRIz4P1i2FYDG1ROpUSnW4TQiIuUrNzeXMWPG0KJFC7Zs2YIxhu7du6v0I1REjPi/+sW3Y1/LOlrNLyKRZe3atQwaNIiVK1dy2223UbOmrlMS6SKi+LekZwFwWataDicRESk///nPf7j11luJjY1l5syZ9O7d2+lIEgLCvvhzCzz8use3sK9r8xoOpxERKT+fffYZ7dq148033+S0005zOo6EiLAv/p92ZB69rQvziEi4W758OXFxcbRq1YqXXnqJ6OhooqLC/lu9lELYL+7bdsC3Y1+zmkkOJxERKTter5fnnnuOzp07M3bsWADi4+NV+vIHYf83Yu7qnQA0q6HiF5HwtGfPHm688Ubmzp1L7969efXVV52OJCEs7It/wz7f8f3k+LD/o4pIBPr555+56KKLSE9P5/nnn2fEiBE6TU9KVKo2NMa0ALoC1YDXrbW7jTGnAfuttVllEfBUrd3tK/4/t6rtcBIRkeBr0qQJF154IXfffTetW7d2Oo5UAAEd4zfGRBtj/gOsAaYC44B6/qefBx4sm3jB07xmJacjiIgExaZNmxg4cCAHDhwgJiaGt956S6UvAQt0cd8/gV7AEKAhUHQeaS7w5yDnCooDR/KO3q6XEu9gEhGR4Jg2bRqpqanMnz+fH3/80ek4UgEFWvzXAg9Za18Ddhzz3AagcVBTBUnh8f0WtSrpUrwiUqFlZWUxbNgwrr76alq0aMHKlSvp2rWr07GkAgq0+GsAJf1oGReELEG3aL3vUrwtamuaX0QqtjvuuIOpU6dyzz33sGDBApo0aeJ0JKmgAl3ctxn4E/DFcZ5rD/watERBlJXnASBKo30RqYCstRw5coSkpCT+/ve/069fPy699FKnY0kFF2jxvwk8YIxZB3zkf8waYzoDd+Bb7BdyvvnVd3GeTk2rOZxERKR0Dhw4wJAhQ0hPT+fTTz+lTp061KlTx+lYEgYCnep/Avgc+ADY53/sS2AB8DXwf8GPdurW7zkCQJX4aIeTiIgEbuHChaSmpjJr1iwuv/xynZcvQRXQiN9aWwD0NcZcim8Ff01gPzDfWvtJGeY7JQVeLwBNtGufiFQAHo+HJ554gkceeYSGDRuycOFCOnTo4HQsCTMBFb8xpia+TXo+BT495jkXUN1au6cM8p2SfI8FoEqCRvwiEvoOHz7MK6+8woABA3jppZdITk52OpKEoUCP8e8EOgPfH+e5Nv7H3cEKFQy5BZ6jt1MSdFU+EQldX3zxBV26dKFy5cp8//331KhRQ9P7UmYCPcZf0t/AKMAbhCxBVXhVPkDn8ItISMrNzWXMmDFcfPHFTJw4EYCaNWuq9KVMFTviN8YkAUXnmaobY+oe87J44BpgdxlkOyXr9/g272nXMMXhJCIif7R27VoGDRrEypUrGTVqFLfffrvTkSRClDTVPxZ42H/b8ttpfMcywOPBDBUMv/qLv+iUv4hIKJg1axbXXnstsbGxzJo1i169ejkdSSJIScU/B9iFr9hfAMYDG495TS6wxlp7vGP/jtp2wHexwOpJsQ4nERH5vWbNmtG1a1defvll6tev73QciTDFFr+1djmwHMAYY4Hp1tp9xb0+1CTG+P5oDaomOJxERASWL1/OjBkzeOyxx2jVqhXz5s1zOpJEqIAW91lrp1Sk0gdI25oBaJ9+EXGW1+vlueeeo3Pnzrzxxhvs3bvX6UgS4QI9nQ9jzOnAjUAL/nhRHmutvSKYwU5VUpzvjxbtCvTEBRGR4NqzZw833HAD8+bNo0+fPrz66qtUrVrV6VgS4QLdwKcd8C2+1fsNgF+Aqvh28NsBbCmrgCdr8QbflflSG1RxOImIRCKPx8OFF17I+vXrmTx5MrfeeqtO05OQEOiI/0ngY2AwkAdcZ61dYYzpAbwC3FNG+U6Kx2vJyfdtLdCwmo7xi0j5yc/Px+1243a7efbZZ6lbty7nnHOO07FEjgp0Hrw18Dq/bdTjBrDWzsV3Zb7xQU92Cg5m5x+9HRsVUhsKikgY27RpE926dWPSpEkAdO/eXaUvISfQ4o8FDllrvUA6UKvIc2uAkPqbvWm/76p8zWrq4jwiUj4++OADUlNTWbNmDXXrHrvXmUjoCLT4NwCFf5N/Am4o8tx1QEhdoCf9cB4AGVn5J3iliMipycrKYujQoQwYMIAzzjiDtLQ0BgwY4HQskWIFWvzzgEv9t58Aehtj0o0xe4C/ApPKItzJysr37dbXRgv7RKSMrVixgtdee4177rmHb7/9lsaNGzsdSaREAS3us9beX+T2fGNMV6A/kADMt9bOLqN8J+Vglm/EXy1RV+UTkeCz1rJ06VI6dOjAeeedx9q1a2nSpInTsUQCclInuVtrF1tr77TWjgi10gf4YftBAKqq+EUkyNLT0+nXrx+dOnVi+fLlACp9qVBOeXcbY0xLY8w7wQgTLHsP5QKQVxByVwsWkQpswYIFpKam8tFHH/H000/Tpk0bpyOJlFqJxW98zjHGXGmMOfOY5842xnwArAauLMuQpZUQ6zuCUbvysRsMioicnKeeeorzzz+f6OhovvvuO8aOHYtLO4NKBVTs31pjTG1gIbASmAX8aIx5wxgTZYx53v/4lfiu3NesPMIG6kf/VH/DaokOJxGRcBEbG8ugQYNYuXIlf/rTn5yOI3LSSlrc9ySQCjwOrAAaA3cDXwOdgfeAu6y128o6ZGntzMgBID5am/eIyMmbM2cOXq+XXr16MXr0aABtuysVXknFfynwD2vtk4UPGGN+BD4BXrLWjijrcCcrz+M7tl8/Jd7hJCJSEeXm5nLPPfcwceJELrjgAnr27KnCl7BR0gGqmvim+osqvB9Si/mKstYevV0rWcf4RaR01q5dS+fOnZk4cSK333478+bNU+lLWClpxO8Gco95rPD+kbKJc+qy/Zv3AMTHaKpfRAK3adMm2rZtS1xcHLNnz6Znz55ORxIJuhNt4Px+KxcAACAASURBVHOZMabowj0XYIHuxpgzir7QWvt2sMOdjPQjvs179AO6iATK6/Xicrlo1KgRf//737nmmmuoV6+e07FEysSJiv8fxTz+2DH3LRASxb8707ewr0p8tMNJRKQiWLZsGTfddBPvvPMOrVq14q677nI6kkiZKqn4zyzhuZCVk+9b2GdP8DoRiWxer5cJEyZw3333Ubt2bQ4fPux0JJFyUWzxW2t/KYsPNMZ0BybiW0PwStGzBoq8ZgDwCL7+XmWtvSbQ99+RkQ1A2wYpwYgrImFoz549/PWvf2X+/Pn06dOHV199lapVqzodS6RcBHSRnmAxxriByfhOFdwGLDXGzLbWrinymubAfUAXa+0BY0zN0nxG4TH+/YePXZcoIuLz/PPP8+WXXzJ58mRuvfVWrdqXiFLe+012ANZZazdYa/OAd4Hex7xmCDDZWnsAwFq7pzQfULg/f/2UhFNPKyJhIz8/nw0bNgDwwAMPsGLFCkaMGKHSl4hT3sVfD9ha5P42/2NFnQ6cboxZaIxZ7D808AfGmKHGmGXGmGV79+49+ni+f/OeBtVU/CLis3HjRrp168ZFF11EVlYWsbGxtGzZ0ulYIo4o7+I/3o/Wx67DiwKaAxcAg4FXjDFV/vBF1k611ra31ravUaPG0ccP5/rO409J0Kp+EYH333+f1NRU1qxZw/jx40lI0KBAIlt5F/824LQi9+sDO47zmlnW2nxr7UbgF3w/CAQkMycfgEpxKn6RSJaTk8OQIUMYOHAgLVu2JC0tjQEDBjgdS8RxpS5+Y0wzY0xHY8zJ/Ni8FGhujGlsjIkBBgGzj3nNTOBC/2dVxzf1vyHQDyg8jz8xtlzXLYpIiImOjmbTpk3cd999fPPNNzRu3NjpSCIhIeDiN8bcbIzZhm8E/h1whv/xacaY4YG8h7W2ALgN34V+/ge8b639yRjzD2NML//LPgH2G2PWAF/iuwLg/kBzFp7OF+3Sgh2RSGOt5eWXX2bHjh243W7mzZvHuHHjiI7WDKBIoYCK3xhzAzAV+AL4K78/Vr8EGBjoB1pr51prT7fWNrXWPu5/7GFr7Wz/bWutvcNa29Jae7a19t1A3xtgu7/4qyTElObLRKSCS09Pp1+/fgwdOpQXX3wRgKgozfyJHCvQEf9dwERr7V/445X5/od/9B8KKvu36q2aqOIXiRQLFiwgNTWVOXPm8Mwzz/Doo486HUkkZAX643BT4ONinjsEhMw2ebszfRv3VEtS8YtEgg8//JCrr76axo0b891339G+fXunI4mEtEBH/On8fjV+UacDO4MT59RYa3H7j+1X1kV6RCLChRdeyO23386KFStU+iIBCLT4PwYeNMYULX/rP79+DDAr6MlOQr7H4vFaolyGaHd5n6koIuXlo48+onv37uTl5ZGSksKECRNITk52OpZIhRBoOz7gf+0aYA6+TXee8d+PBkLigFpWXgEAXqtr84mEo9zcXEaPHk2vXr3YvXs3+/btczqSSIUTUPH798tvC0wCagDbgarAG0DHwn31nXYop7D4HQ4iIkH3yy+/0KlTJyZNmsTo0aNZvHgxdevWdTqWSIUT8Lku1toMfCP/B8ouzqnZ578iX+3kOIeTiEgwWWu54YYb2Lp1Kx999BFXXnml05FEKqyAit8YMw74t7X25zLOc0oKz+GPidLxfZFwkJmZicvlIikpiddff52kpCTq1Tv2ul4iUhqBNuQo4Cf/1fBGGWNqnPArHLD9gK/446PdDicRkVO1dOlS2rZty6hRowBo0aKFSl8kCAIt/prAX4C9wHPAdmPMHGPM1caY2DJLV0qFK/lrVAqZSCJSSl6vl2eeeYZzzz2X/Px8brnlFqcjiYSVQBf3ZVtr37LWXo7vinr3AnWA94DdxpiXyzBjwHILvAC0qqfTekQqoj179tCjRw/uuusuevXqRVpaGl26dHE6lkhYKfXBcGvtbmvtc9badsDF+HbuuynoyU5Ctv90vtgoTfWLVERZWVmsXr2aF198kWnTppGSEjKbgoqEjVJfwcI/td8HuA64DN8Fe4rbzrdcZWTnA5AUq+IXqSjy8/N56623+Otf/0qjRo1Yv3498fHxTscSCVuluSzvBcaYV4Hd+C7UUwu4E6hrre1V4heXk1VbMwColqhj/CIVwcaNG+natSs33ngjn3/+OYBKX6SMBXo63xagHrAVmIzv1L5fyjLYyaiW5Ct8j3bwEQl57733HkOHDsUYw3vvvccll1zidCSRiBDoVP+n+Mr+67IMc6pWb/ON+OunaMQgEsruv/9+nnjiCTp16sQ777xDo0aNnI4kEjECKn5r7c1lHSQYqifFsu9wHtHawEckpBWO7h999FGio3UlTZHyVGzxG2M6AD9aa7P8t0tkrf0+qMlOQp7/dL6UBH0jEQkl1lpeeOEFDhw4wIMPPshFF13ERRdd5HQskYhU0oh/MdAJ+N5/u7gD58b/nONL6bPzPQDEaec+kZCRnp7OzTffzMyZM7niiivweDy43fo3KuKUkor/cuB//ts9KL74Q8aRXN95/EmxpT5LUUTKwIIFC7jmmmvYtWsXzz77LGPGjMHl0qE4EScV25DW2k+K3J5fPnFOTZ7HN9Wvi/SIOG/v3r1cdtll1K1bl++++4727ds7HUlECPA8fmPMGmPM2cU819IYsya4sU5OTr6/+N0qfhGnHDx4EIAaNWrw4YcfsmLFCpW+SAgJtCHPAIo7Ry4BaBGcOCevwD/adxmIUvGLOGL27Nk0bdqU6dOnA9C9e3eSk3XtDJFQUpqGLO4Y/znAwSBkOSVHcn0L+7R3j0j5y8nJ4fbbb6d37940aNCAs88+7gShiISAkk7nGwWM8t+1wDRjTO4xL4sH6gLTyiZe4ApX9NfUJXlFytXPP//MoEGDWLVqFWPGjOHJJ58kNlb/DkVCVUnL33cAy/23mwG/APuPeU0usAZ4MfjRSiczx3eBHq/VkF+kPC1dupTt27czZ84crrjiCqfjiMgJlLSqfzowHcAYA/CAtXZDOeUqtcLNe7LyPA4nEQl/mZmZLFu2jIsuuojrr7+eK6+8UpfQFakgAt2yd3BZBzlVB7LyADizjhYSiZSlpUuXMmjQIPbu3cvmzZtJSUlR6YtUICUd478b34V5dvlvl8Raa58ObrTSKRzp7zqY42QMkbDl9Xp59tlnuf/++6lbty7z5s1T4YtUQCWN+J8EvgJ2+W+XxAKOFv/BLN8x/pZ1NeIXCbaCggJ69uzJ/Pnzueqqq3jllVdU+iIVVEnFH2+tLVzFH/LXuS2c6ncZh4OIhKGoqCjatGlD7969GTZsWOG6HxGpgEpa3Jd7vNuhyu1vfF2gRyQ48vPzeeihh+jduzedO3dm3LhxTkcSkSAIaHGfMaYJkGytTfPfjwXuBc4CPrHWvlJ2EQNTeIy/fkrIT06IhLwNGzYwePBgvv/+e2JjY+ncubPTkUQkSAK9jN0L+M7XT/Pf/yfwN2At0NcY47bWTimDfAHLzPYd468UF+1kDJEK77333mPo0KEYY3j//fe5+uqrnY4kIkEU6Ja9qcA3AMZ3cO8G4H5rbSt8C/+Gl0m6UjiU47skb7KKX+SkzZkzh0GDBtGqVSvS0tJU+iJhKNDirwLs899OBaoB7/vvfwo0DXKuUjuc6yv+SnGBTmKISKGcHN9psJdffjlTp07l66+/plGjRs6GEpEyEWjx7wGa+G9fCmy01m72308EHN8uL7fAFyE2SlfmEwmUtZbJkydz+umns3PnTtxuN0OGDCE6WjNnIuEq0OHxHOBxY8zpwFDgtSLPtQI2BjtYaW3P8I1YYlT8IgFJT0/n5ptvZubMmfTo0YOoKM2WiUSCQP+l3wtUAgYCnwGPFXluAPBFkHOV2hH/VL+InNi3337LNddcw+7du3nuuecYPXo0Lpd+aBaJBIHu1Z8JXF/Mc38KaqKTlBzv+6NUjtcUpciJTJ48mbi4OBYtWkS7du2cjiMi5ahUc3vGmEpAB6Aqvkv0LrXWHiqLYKW1dvdhABJiNF0pcjzbtm0jPz+fxo0bM2XKFFwuF5UqVXI6loiUs4Dn9owxDwI7gf8C7+Gb8t9pjHmgjLKVSqzb90eJdmsrUZFjzZo1i9atW3PTTTcBULlyZZW+SIQKqPiNMSOBfwAzgB5AG+By//1/GGNuLbOEASpc1Jek0/lEjsrJyWHUqFH06dOHhg0bMmWKo/tsiUgICLQlbwNesNbeVuSxVcAnxpiDwCjgxWCHK439R3wX6YnXXv0iAGzdupWePXuyatUqxowZw5NPPklsbKzTsUTEYYEWfxPg9mKemwXcEpw4p04X6RHxqVatGikpKcyZM4crrrjC6TgiEiICPcafDrQo5rkW/udDQrRbpyRJ5MrMzOSuu+7i8OHDJCQk8MUXX6j0ReR3Am3Jmfg28LnaFLkQtzGmL74L9swsi3CBstb3e4xKXyLY999/T5s2bZgwYQJffvklAEX+uYqIAIEX/73Az/hW82cZYzYbY7KAacAv/ucdY/E1f5RW9EsE8nq9PP3003Tp0oWCggK++eYbevbs6XQsEQlRgW7gc9AYcy7QF+iK7zz+dOBrYJa11tG9+gtH/FEuFb9Envvuu4/x48fTr18/Xn75ZVJSUpyOJCIhLOBz3/zlPs3/K6QcLX5N9UsE8Xg8uN1ubr31Vpo1a8Ytt9yiqX0ROaESm9IYM8gYs9gYs88Ys84Y87gxJuROlC+c6k/3n9InEs7y8vK455576Nu3L9ZaGjVqxJAhQ1T6IhKQYovfGHM18DZQG1gIZOE7lv9YcV/jlMIR/2lV450NIlLGNmzYQNeuXRk/fjx169YlPz/f6UgiUsGUNOK/A/gYaG6t7W2tPQd4ChhljAmpOXV/7xOtq4tJGHv33Xdp06YNa9euZdq0abz00kvExMQ4HUtEKpiSmrIF8KK1tuiQYhIQDzQs01SlZK1W9Ut4O3ToEHfccQdnnXUWaWlp9OvXz+lIIlJBlXS8vgqw75jH9vp/TwE2lkmik/Dbqn6N+CW8/PzzzzRr1oxKlSrx9ddf07hxY6KiQm6ZjYhUICdqSlvKxx3h8Tf//iO5DicRCQ5rLc8//zypqak8/fTTADRv3lylLyKn7ETFv9AYk1f4C8j2P76k6OPGmIAb1xjT3Rjzi/8sgWI3/jHG9DfGWGNM+xO+p//3pFh9U5SKb//+/fTt25dRo0Zx8cUXc8stIXMpDBEJAyU15VPB/jBjjBuYDFwKbAOWGmNmW2vXHPO6SvguCrQkkPctnH6oXTkuiGlFyt+iRYsYMGAAu3fv5rnnnmPMmDE6TU9EgqrY4rfW3lcGn9cBWGet3QBgjHkX6A2sOeZ1/wTGA3cG8qY6xi/hIiYmhsqVKzNz5kzatWvndBwRCUPl3ZT1gK1F7m/zP3aUMaYNcJq1dk5Jb2SMGWqMWWaMWXbgYCYA0VrVLxXQtm3b+Ne//gVAu3btWL16tUpfRMpMeRf/8Zr56EJB//4AE4CxJ3oja+1Ua217a2375EqVANi470iwcoqUi1mzZtG6dWvuv/9+tm/fDoBLM1ciUobK+zvMNuC0IvfrAzuK3K8EnAV8ZYzZBHQCZp94gZ/vZ4dWdSsHL6lIGcrJyWHUqFH06dOHRo0asWLFCurVq3fiLxQROUXlvQx+KdDcGNMY2A4MAq4pfNJaexCoXnjfGPMVcKe1dllJb1p4jD82SiMlCX3WWi655BIWLlzImDFjePLJJ4mNjXU6lohEiHJtSmttAXAb8AnwP+B9a+1Pxph/GGN6nfT7+n/X1fkklFlrsdZijGHUqFHMmTOHCRMmqPRFpFyV+4nv1tq5wNxjHnu4mNdeEMh75hZ4iQZitLhPQtTBgwcZNmwYF198MUOGDGHgwIFORxKRCBXwENkYU8sYM84Ys8AYs8YY09L/+IhANtkpS1EuX+Fvz8g+wStFyt+SJUto06YN06ZNIzMz0+k4IhLhAip+Y8wZwA/Arfguz9sCKNwtpwUwpkzSBahwqv/0WpWcjCHyO16vl6eeeorzzjsPr9fLt99+y9ixJzxhRUSkTAU64n8G30V5GgM9+P1peQuBzkHOVSqFV+eL1jF+CSGLFy/m3nvvpW/fvqSlpdG5s6P/TEREgMCP8Z8PXGetzfBvu1vULqBOcGOVTuGIP0ar+iUEbNmyhQYNGnDuuefy3Xff0alTJ227KyIhozRN6Snm8Wr8dvEeR+Tm+6K59M1VHJSXl8fdd99Ns2bNWLbMdwZq586dVfoiElICHfEvA64HjreNbj9gcdASnYTCKf6MrDwnY0gE27BhA4MGDWLp0qUMHz6cVq1aOR1JROS4Ai3+x4H5xpiPgLfwza53M8YMAwYAF5ZRvoAUTvXXr5rgZAyJUO+++y5Dhw7F7XYzbdo0+vXr53QkEZFiBVT81trPjDEDgP8DrvA//By+7XYHWGsXllG+gBTu3Bft0pSqlL9169Zx9tln8/bbb9OwYUOn44iIlMgUrogP6MW+g5WtgJrAfuAHa623jLIFrFbTljb+6qd55urW9G9X3+k4EgFWrVrFgQMHuOCCC/B4PFhriYoq9/2wRCRCGWOWW2tPag+dUn2nsr6fEn48mQ8qSzn5XuLRZXml7FlrmTx5MmPHjqVly5asWLECt/vYE11EREJXQMXvn+YvkbX2/VOPc3IKt+o9klvciQcip27//v3cdNNNzJ49mx49evD6669rxb6IVDiBjvjfLebxoscJHCv+whC1knWxEykbO3bsoEOHDuzZs4cJEyYwevRolb6IVEiBFv+Zx3msGnAl0B/4a9ASnYTCZQq6Op+UlTp16jBw4ECuvfZa2rZt63QcEZGTFuiq/l+Keeo7Y4wH3x7+i4KWqpSOrurXMX4Joq1btzJ8+HAmTpxIs2bNePbZZ52OJCJyyoIxRP4S6BWE9zlp2f6d+7RXvwTLzJkzad26Nd988w2//FLcz70iIhVPMJqyPb4r9jmmcI9+rzfwUxNFjicnJ4eRI0fSt29fmjRpwooVK7jiiitO/IUiIhVEoKv67z7OwzHAWUBf4OVghjpZSXE6j1pOzfjx43nhhRe44447eOKJJ4iJiXE6kohIUAXalE8e5zEPsB2YADwatEQnoXAToiiXpvql9Ky1pKenU61aNe688066dOnCxRdf7HQsEZEyEWjxxx/nsfxQ2LWvKLe27JVSOnjwIMOGDWPlypWsWLGCxMRElb6IhLUTDpGNMTHAI8BZ1trcIr9CpvSPns6n4pdSWLJkCW3atGHatGnccMMNxMXFOR1JRKTMnbD4rbV5wGggsezjnJx8j+9nkCidzicB8Hq9PPXUU5x33nl4vV6+/fZb7rvvPm29KyIRIdCD4quAlmUZ5FQUruV3aSc1CYDH42HWrFn07duXtLQ0Onfu7HQkEZFyE+gx/ruBfxtj1llrPyvLQCej8Nh+4Wl9Isfz6aef0qZNG6pXr878+fOpVKmStt0VkYgTaFO+BlQBPjHGHDLG/GqMWVvkl7M7nOgYv5QgLy+Pu+66i8suu4zHHnsMgOTkZJW+iESkQEf8y/n9BXlCytGpfhW/HGP9+vUMHjyYpUuXMnz4cJ544gmnI4mIOCrQvfoHlXWQYHBrBCdFfPHFF/Tp0we32820adPo16+f05FERBxX7FS/MWaDMaZ1eYY5WYUb+Og8finqrLPO4tJLLyUtLU2lLyLiV9Ix/kZAhbjAfeFUv4pf0tLSuPHGGykoKKBmzZpMnz6dhg0bOh1LRCRkhNUyeE31Ry5rLZMmTaJjx47897//ZdOmTU5HEhEJSScq/pBd0Hc8WtwXmfbt20fv3r0ZPXo0l112GatWraJZs2ZOxxIRCUknWtz3qDFmXwDvY621fw1GIJHS6t+/P4sWLWLixImMGjVKp+mJiJTgRMWfCuQG8D6OzwwkxGi71UhSUFCAx+MhNjaWCRMmANCmTRuHU4mIhL4TFX8fa+335ZLkFOn4fuTYunUr11xzDeeccw6TJ09W4YuIlELYLO5z6wI9EWHmzJm0bt2atLQ0zj33XKfjiIhUOGFT/NquN7xlZ2czcuRI+vbtS5MmTVi5ciXXXnut07FERCqcsCl+ncMf3rZv386///1vxo4dy3fffadV+yIiJ6nYY/zW2gr1Q4GO8Ycfay2ff/45F198Mc2aNWPdunXUqlXL6VgiIhVahSr3kuw4mON0BAmigwcPMnjwYC699FLmzJkDoNIXEQmCQK/OF/Ka1kh0OoIEyeLFixk8eDBbt25l3LhxXHHFFU5HEhEJG2Ez4o9yhc0fJaJNnjyZrl27Yq3l22+/5b777sOl/29FRIImbL6janFfeGjYsCFXXXUVaWlpdO7c2ek4IiJhJ2yKP0rn8VdY8+fPZ/LkyQBceeWVvPfee1SpUsXhVCIi4Slsil8j/oonLy+PO++8k8svv5xXX32V/Px8pyOJiIS9sCn+g1kqjYpk3bp1dOnShWeffZYRI0awcOFCoqOjnY4lIhL2wmZVv1QcGRkZdOjQAWstH374IX379nU6kohIxAib4q9bJd7pCHICBQUFREVFUaVKFSZNmkS3bt1o0KCB07FERCJK2Ez1u3SMP6SlpaVx9tlnM3/+fACuu+46lb6IiAPCpvi1qD80WWuZNGkSHTt2JDMzk4SEBKcjiYhEtLApfpf26g85+/bto3fv3owePZo///nPrFq1im7dujkdS0QkooVP8WuqP+TMnj2bTz75hIkTJzJr1iyqV6/udCQRkYgXNov7dHW+0FBQUMBPP/1E69atufHGG+nWrZsuoSsiEkLCZsSvDXyct2XLFi688EK6du3Knj17MMao9EVEQkzYFL+m+p01Y8YMUlNTWbVqFS+++CI1a9Z0OpKIiBxH2BT/xn2HnY4QkbxeLyNGjOCqq66iadOmrFy5kmuvvdbpWCIiUoywKf6WdZKdjhCRXC4XBQUFjB07loULF9K0aVOnI4mISAnCZnFflDtsfoYJedZaXnnlFdq3b0+bNm2YMmUKRosrRUQqhHJvS2NMd2PML8aYdcaYe4/z/B3GmDXGmNXGmM+NMQ0Ded9oHeMvFxkZGQwcOJChQ4cyZcoUAJW+iEgFUq7Fb4xxA5OBy4GWwGBjTMtjXrYSaG+tPQeYBowP5L3dLo34y9qiRYtITU1lxowZPPnkk7zwwgtORxIRkVIq76n+DsA6a+0GAGPMu0BvYE3hC6y1XxZ5/WLgukDeOEp79papr776iksuuYTTTjuNb7/9lk6dOjkdSURETkJ5D5PrAVuL3N/mf6w4NwPzAnnjHRnZpxBLimOtBaBLly488MADrFy5UqUvIlKBlXfxH29Ybo/7QmOuA9oDTxfz/FBjzDJjzDKAZjWTghZSfObNm0e7du3Yv38/0dHRPProo1SpUsXpWCIicgrKu/i3AacVuV8f2HHsi4wxlwAPAL2stbnHeyNr7VRrbXtrbXuAKC3uC5q8vDzGjh1Ljx49KCgoICMjw+lIIiISJOVd/EuB5saYxsaYGGAQMLvoC4wxbYAp+Ep/T6BvrMV9wbFu3Tq6dOnCc889x4gRI1iyZInOzRcRCSPlurjPWltgjLkN+ARwA69Za38yxvwDWGatnY1vaj8J+MB/mtgWa22vE723RvzBcf/997N+/Xo+/PBD+vbt63QcEREJsnLfwMdaOxeYe8xjDxe5fcnJvK8u0nPyDh8+zOHDh6lduzbPP/88OTk5NGjQwOlYIiJSBsJmfjzf43U6QoW0cuVK2rVrx6BBg7DWUrNmTZW+iEgYU/FHKGstEydOpFOnThw5coRHH31UO/CJiESAsNmrv1pSrNMRKoz09HRuuOEGPvroI3r27Mlrr71G9erVnY4lIiLlIGxG/DrEH7ioqCjWr1/PpEmTmDVrlkpfRCSChM2IX9PUJSsoKGDy5MkMGzaM5ORk0tLSiI6OdjqWiIiUszAa8av4i7N582bOP/98xowZw7Rp0wBU+iIiESqMit/pBKFp+vTppKam8sMPP/D2229z3XUBXfNIRETCVNgUvwb8f/T000/Tv39/mjdvzsqVKxk8eLDTkURExGFhc4xfU/1/1KtXLw4cOMAjjzxCTEyM03FERCQEhNGIX8VvrWXq1KncdNNNWGtp0aIF48aNU+mLiMhRYVP8kX6MPyMjg4EDBzJs2DC2bt1Kdna205FERCQEhVHxR27zL1q0iNTUVGbMmMGTTz7JJ598QkJCgtOxREQkBIXRMX6nEzgjOzubvn37kpCQwIIFC+jYsaPTkUREJISFTfFH2jH+vXv3Uq1aNeLj45k9ezYtWrSgcuXKTscSEZEQFzZT/ZFk3rx5tGrVimeffRaADh06qPRFRCQgYVP8OfkepyOUuby8PMaOHUuPHj2oU6cOV155pdORRESkggmbqf7kuPDegnbdunUMGjSI5cuXM3LkSJ555hni4uKcjiUiIhVM2BQ/YX6If+fOnWzZsoUZM2bQp08fp+OIiEgFFTZT/eHY+4cPH+b9998HoGvXrmzcuFGlLyIipyRsij/czuNfsWIFbdu25ZprrmHDhg0AJCYmOpxKREQqurAp/nDpfWst//d//0enTp3Iysri888/p0mTJk7HEhGRMBE2x/hNGEz2W2sZMGAA06ZNo1evXrz22mtUq1bN6VgiIhJGwqf4K37vY4zhsssu4/zzz2fkyJERtymRiIiUvfApfqcDnKSCggIeeeQRWrVqxeDBgxkyZIjTkUREJIyFzTH+itj8mzdv5vzzz+fxxx9n8eLFTscREZEIEDYj/oq2qn/69OnccssteDwe3n77bQYPHux0JBERiQBhM+KvSLW/fPly+vfvT/PmzVm5cqVKX0REyk34FH8FGPEfOnQIgHbt2vHBBx+wYMECmjZt6nAqERGJJGFU/E4nKJ61lqlTp9KwYUNWrVoFQP/+/YmJxlbJagAAEIFJREFUiXE4mYiIRJrwKX6nAxQjIyODAQMGMGzYMNq3b0+tWrWcjiQiIhEsfIo/BJt/0aJFpKamMnPmTJ566inmz59P7dq1nY4lIiIRLGxW9YfiMf7Zs2fjcrlYsGABHTt2dDqOiIgIxlrrdIZTFlunuV285HvaNEhxOgo7duxg586dtGvXjvz8fLKysqhcubLTsUREJIwYY5Zba9ufzNeG0VS/8yP+jz/+mNatW3PNNdfg8XiIjo5W6YuISEgJn+J38LNzc3P529/+xpVXXkndunWZNWsWbrfbwUQiIiLHF0bH+J353P3793PZZZexYsUKRo0axfjx44mLi3MmjIiIyAmE0YjfmeZPSUnhzDPPZObMmUyaNEmlLyIiIS1sir88HTp0iBEjRrB161ZcLhdvvvkmvXv3djqWiIjICan4S2n58uW0bduWKVOm8OWXXzodR0REpFTCpvhjosr2j2KtZcKECXTu3Pn/t3f30VJV5x3Hvz9BxSAgylssUWKRWGKJUrSYUojVZZBWqUarJOILgjUJrhBdaXFhlSi1Fd9Tk4g2iaIFlRgLxaBWQ3xJxUQDUjAxCxWjorEi4hsvAk//2PvqMN47d+5lZi537u+z1lkzc84+5zyzZ9Y8c/bZZx82bNjA4sWLOf3006u6TzMzs0qrm8Rf7c5911xzDeeffz5jxozh6aefZuTIkdXdoZmZWRXUTa/+XaqU+Ddv3sxuu+3GpEmT6NmzJ2edddZOMWaAmZlZa9TNEX+lr+TfsmUL06ZNY/jw4WzcuJHu3bszYcIEJ30zM2vX6ibxVzIfv/jii4waNYrLL7+cQw89lG3btlVu42ZmZm2obpr6K5X37777biZOnMjWrVuZM2cO48aNq9CWzczM2l79JP4KHPJv2bKFGTNmMGjQIObOncsBBxxQgcjMzMx2HnWT+Hekc9/KlSvp378/PXr04N5776V3797suuuulQvOzMxsJ1E/5/hb0dgfEcyaNYthw4YxdepUAPbdd18nfTMzq1v1k/hbmPfXrVvHySefzLnnnsvIkSOZPn16VeIyMzPbmdRN4m+JpUuXcsghhzB//nxmzpzJokWL6Nu3b1uHZWZmVnX1c46/BSf5e/fuTb9+/Zg3bx6HH354FaMyMzPbudTNEX9zaX/NmjVMmzaNbdu20b9/f5YsWeKkb2ZmHU79JP4SmX/hwoUMGTKE6667jhUrVuTyHoHPzMw6nvpJ/I0c82/atIkpU6Zw3HHH0b9/f5566imGDBnSBtGZmZntHOrmHH9jB/CnnHIK8+fP57zzzmPmzJl06dKl9oGZmZntRBQRbR3DDtv9kwfGS7/7X/p0S4l927Zt7LLLLjz22GOsXbuWsWPHtnGEZmZmlSPpqYgY1pp1a97UL2m0pGclrZI0tZHlu0u6My9/QtKAsraLeOeddxg/fjwXXnghACNGjHDSNzMzK1DTxC+pE/Bd4FhgMDBO0uCiYmcD6yJiIHAtcEU5216+7NcMHTqUOXPm0LVr10qGbWZmVjdqfY7/cGBVRDwPIOkOYCzwTEGZscD0/PzHwA2SFCXOSWx97y2OPWoUffv2ZfHixYwcObI60ZuZmbVztW7q/yPgpYLXL+d5jZaJiC3AemCfUhvd+s5ajv7iaJYtW+akb2ZmVkKtj/gbu3i++Ei+nDJIOgc4J7/ctGjhf63o1avXDoZnJfQC3mjrIDoA13P1uY6rz3VcfZ9p7Yq1TvwvA58qeN0fWNNEmZcldQZ6AG8WbygibgJuApD0ZGt7N1p5XMe14XquPtdx9bmOq0/Sk61dt9ZN/b8CDpT0aUm7AacCC4rKLADOyM9PAn5W6vy+mZmZla+mR/wRsUXSZOB+oBPww4hYKelS4MmIWAD8ALhN0irSkf6ptYzRzMysntV85L6I+Cnw06J5Fxc83wic3MLN3lSB0Kw013FtuJ6rz3Vcfa7j6mt1HdfFyH1mZmZWnrq5SY+ZmZk1r10l/moN92sfKaOOz5f0jKTlkh6StH9bxNmeNVfHBeVOkhSS3Du6FcqpZ0l/l7/PKyXNqXWM7V0Zvxf7SVosaWn+zRjTFnG2Z5J+KOl1SSuaWC5J38mfwXJJQ5vdaES0i4nUGfA54ABgN+BpYHBRma8BN+bnpwJ3tnXc7Wkqs46PBD6Rn3/VdVz5Os7lugGPAEuAYW0dd3ubyvwuHwgsBXrm133aOu72NJVZxzcBX83PBwOr2zru9jYBI4GhwIomlo8BFpHGwBkOPNHcNtvTEf+Hw/1GxGagYbjfQmOBW/PzHwNHSY3dsNea0GwdR8TiiHg/v1xCGovBylfO9xjgMmAmsLGWwdWRcup5EvDdiFgHEBGv1zjG9q6cOg6ge37eg4+P22LNiIhHaGQsmwJjgdmRLAH2kvTJUttsT4m/KsP92nbKqeNCZ5P+aVr5mq1jSYcCn4qIhbUMrM6U810eBAyS9AtJSySNrll09aGcOp4OnCbpZdLVXOfVJrQOpaW/27W/nG8HVGy4X2tS2fUn6TRgGDCqqhHVn5J1LGkX0l0pz6xVQHWqnO9yZ1Jz/xdILVePSjo4It6qcmz1opw6HgfcEhFXSzqCNEbLwRGxrfrhdRgtznvt6Yi/JcP9Umq4X2tSOXWMpKOBacDxEbGpRrHVi+bquBtwMPBzSatJ5+wWuINfi5X7ezE/Ij6IiBeAZ0l/BKw85dTx2cBdABHxONCFNI6/VU5Zv9uF2lPi93C/1ddsHedm6FmkpO9zoi1Xso4jYn1E9IqIARExgNSP4viIaPW43B1UOb8X/0nqrIqkXqSm/+drGmX7Vk4d/x44CkDSn5AS///VNMr6twA4PffuHw6sj4hXS63Qbpr6w8P9Vl2ZdXwlsCcwL/eb/H1EHN9mQbczZdax7aAy6/l+4BhJzwBbgW9FxNq2i7p9KbOOLwBulvRNUvPzmT4YaxlJc0mno3rlvhKXALsCRMSNpL4TY4BVwPvAWc1u05+BmZlZx9GemvrNzMxsBznxm5mZdSBO/GZmZh2IE7+ZmVkH4sRvZmbWgTjxmzVC0pn5zniNTUe3cFsT83o1ua+BpBlF8a7Ld6us+OWtkjrnfVxUMO9ESVMaKXt0Ljui0nGUiG9gUV1slfSqpNsklRzWtMQ2h0qaLmmvSsdrVgvt5jp+szZyMmlkrELPtEUgrXBEftwH+HtgrqTdImJ2pXaQr+U+gu3HCj8RGAFcV1T8lzmmlZXafwvMAO4Fds8xXAwcJOmIfF+PlhhKupb6FsDD+1q748RvVtqyiFjV1kG0Rr5TFwCSHiANSTsFqFjiL95PM+XeJo1E2BaeK4jzYUm7k24gcwjgURGtQ3FTv1krSdpD0vWSVkp6LzchL5D0mTLWHS9pWV5vvaTlkiYWlTlS0s8kvZunRZIGtybWiPgAWAYMLNh+D0nfy3FvlvSspG8UxdBd0g2SXpK0SdIfJP23pEF5+XZN/ZJuB74C7F/QvL4qL9uuqV/STZLWSOpUtM8uuU6uKpjXR9KsXH6zpN9IOrs1dZH9Oj/uV7TvGZKWSnpb0huSHpJ0eMHyicDN+eULBe+xf0F9TMt1uUnSK5KuzH80zHYKPuI3K62T0g2fGkREbM3P98jTpcBrpCb1rwOPSzqoqXsZSBoF3EpqCr+ANNzpYKBnQZmxwN2kcbi/TPqTPpV0B7khEfFKK97Lp8lN0znZLgKGAP9Ean4/HrhO0j4RcXFe53pgNOmmTKtIN1gZQboBVmMuyWU+B5yQ521souxsYBJpLPcHCuaPJd3D/bYc617AL0jDlF4MrCYNUXpzPnXx/bLe/fYG5MfniubvC1xNOr2zJ+neH49KGhoRK4H5wAHAhaRTGg1jojd81nOBY4F/JbVufJb0/dgPOKUVcZpVXkR48uSpaCLdFjcamR4rsU4noCtpvOzzCuZPzOv2z6+nAq+X2I5Iye3+ovl7ke5BcVUzsc/I++ucp77AZXneVbnM3+bXpxWtewspUe+dX/8WmFliX53zdi4qmHc7sLqRskfnsiMK3ufzwG1F5RYCywtefxvYAPxxUbkfAX8AOpWIb2De54Qca1fSH401wB3N1GMn0p+N54CrG/k8BxSVPzLP/3LR/DPy/D9t6++1J08R4aZ+s2acABxWMG3XvCzpVEm/lLQe2AK8S2oFKNXc/yugt6TZkv5aUvHR80HA/sB/5KbjzrnV4V3gCWBkmbF/kKfXgG8B15CO3Mnb2ALcUbTO7aQOcH9eEOvZkqZK+jNJFfvNiIjI+ztBUlcASb2BL7J9P4TRwP8ALxbVx/1AH0rXdYMfkOriXeBB0hH9GcWFJB0j6eeS1pLqZzPpCL+cfYwm/Wm6pyjOhtaMvyxjG2ZV58RvVtqKiHiyYHq2YYGkE0hNuyuAcaRkeRjpqLxLUxuMiIdIzb4DSLeGfUPSA5IOzkX65Mdb+Sh5N0yjSacUytHwZ2Ug0C0iLoiITXnZ3sAb8fEe7a8VLAf4Gumc9iRSJ7jXJV0taY8yY2jObNJR+In59TjS79KcgjJ9gL/i43UxNy8vpz6+TaqLLwDfz8//rbCApMNIPf/Xk1oIhudyKyjxeRbF2YXU4lMYZ8O90cv93Myqyuf4zVrvVOC3ETGhYYakLqQm+ZIi4i7gLkl7kpLaFcAiSfsBDbeG/QdgcSOrb2pkXmP7KNVb/U3SbT47FyX/fvlxbd7GO6RTE1MlDSBd3vgvpCPbaeygiFglaQlwGumc/mnAQxGxpqDYWtLlguc3sZlnm5hfaHVBfTwsqTswUdKNEdHQ0e8k0vv6UmGdSNqbdEqhOWtJSX9UE8vXNDHfrKac+M1a7xOk5uBCp9OClrSIeBdYIGkgqVNZT9I4AS8BgyPiygrFWuxh4JvAl4A7C+Z/hZT8nmgk1tXAlZLGAwcXLy+wiXS6o1y3Ad+RdCTpCHt80fL7SOMQrI6IN1qw3VL+kfTeLyF1JoSPPs8P71Uu6RhSh7/fFKzb8Mer+D3eR+qs2TUiHq5QnGYV58Rv1nr3ATfky84WkZLW14G3S60k6Z9Jzb6LSb3C9wMmA09GxJu5zGTgJ7kFYR7paLIf8Hng+Yi4fgdjXwg8TuoZ34+U2P6G1KnxsohYl+N4AvgJqbn7PVIHts8Cs0ps+xlggqRzgKXAhohYUaL8HcC1pD8A7wH3FC2/itTS8Kika4HfAd1IfSE+HxEn0EIR8YqkG4Epkg6JiGWkz3My8CNJt+btX8THj9QbBnCanC9f/AB4OiIelDSPdI7/GtKARZBO6YwBLoiI4qsIzGqvrXsXevK0M0581Kt/YIkynYDLSYnhfVIi/xyp49i/F5Qr7tV/PKnD16uko8eXSOfR+xVt/y9I55zXkY7CXyCd1x7eTOwzyH3nminXA/hejmMzqcn8G0VlriIl7/WkjnHLgckFyxvr1d+N1IqwLi9bledv16u/aD/35GWzm4h1b9KlhatzrK8Dj1Bw9UQT6zX06j+zkWV98nu6u2DelLyPDaTEfSTwGPBg0bqX5s99a9Fn24nUkrI8f2ZvkcZPuALo3tbfa0+eIgJFfNiqZWZmZnXOvfrNzMw6ECd+MzOzDsSJ38zMrANx4jczM+tAnPjNzMw6ECd+MzOzDsSJ38zMrANx4jczM+tAnPjNzMw6kP8HHmudbz3XHFAAAAAASUVORK5CYII=\n",
+ "text/plain": [
+ "<Figure size 576x432 with 1 Axes>"
+ ]
+ },
+ "metadata": {
+ "needs_background": "light"
+ },
+ "output_type": "display_data"
+ }
+ ],
+ "source": [
+ "from sklearn.metrics import roc_curve\n",
+ "#We need to calculate hte fpt,tpr and retrieve the thresholds\n",
+ "fpr, tpr, thresholds = roc_curve(y_train_5, y_scores) \n",
+ "\n",
+ "def plot_roc_curve(fpr, tpr, label=None):\n",
+ " plt.plot(fpr, tpr, linewidth=2, label=label)\n",
+ " plt.plot([0, 1], [0, 1], 'k--')\n",
+ " plt.axis([0, 1, 0, 1])\n",
+ " plt.xlabel('False Positive Rate', fontsize=16)\n",
+ " plt.ylabel('True Positive Rate', fontsize=16)\n",
+ "\n",
+ "plt.figure(figsize=(8, 6))\n",
+ "plot_roc_curve(fpr, tpr)\n",
+ "plt.show()"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "We can calculate the area under the curve (AUC) value using the builtin python function."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 8,
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "0.9604938554008616"
+ ]
+ },
+ "execution_count": 8,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "from sklearn.metrics import roc_auc_score\n",
+ "roc_auc_score(y_train_5, y_scores)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": []
+ }
+ ],
+ "metadata": {
+ "kernelspec": {
+ "display_name": "Python 3",
+ "language": "python",
+ "name": "python3"
+ },
+ "language_info": {
+ "codemirror_mode": {
+ "name": "ipython",
+ "version": 3
+ },
+ "file_extension": ".py",
+ "mimetype": "text/x-python",
+ "name": "python",
+ "nbconvert_exporter": "python",
+ "pygments_lexer": "ipython3",
+ "version": "3.7.6"
+ }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 4
+}
diff --git a/topic_19/Untitled.ipynb b/topic_19/Untitled.ipynb
new file mode 100644
index 0000000000000000000000000000000000000000..7fec51502cbc3200b3d0ffc6bbba1fe85e197f3d
--- /dev/null
+++ b/topic_19/Untitled.ipynb
@@ -0,0 +1,6 @@
+{
+ "cells": [],
+ "metadata": {},
+ "nbformat": 4,
+ "nbformat_minor": 4
+}
diff --git a/topic_19/topic_19.md b/topic_19/topic_19.md
new file mode 100644
index 0000000000000000000000000000000000000000..cca4ff4b560dc750d1e733618590b07613b56f22
--- /dev/null
+++ b/topic_19/topic_19.md
@@ -0,0 +1,136 @@
+class: bottom, left
+background-image: url(assets/g.png)
+
+<h2 class="title_headings_sml">COSC102 - Data Science Studio 1</h2>
+
+<h1 class="title_headings_sml"> Topic 19 - The Receiver Operating Characteristic Curve </h1>
+
+<h3 class="title_headings_sml"> Dr. Mitchell Welch </h3>
+
+---
+
+## Reading
+
+* Chapter 3 from ***Hands-on Machine Learning with Scikit-Learn & TensorFlow***
+
+---
+
+## Summary
+
+* Sensitivity-Precision Trade-off
+* The ROC Curve
+
+---
+
+## Sensitivity-Precision Trade-off
+
+* Using the results from the confusion matrix, we can calculate some concise metrics.
+* *Precision* - the accuracy of the positive predictions.
+
+$$
+precision = TP / (TP + FP)
+$$
+
+* *Recall* (Sometimes called *Sensitivity*) - The ratio of positive instances that are correctly detected by the classifier.
+
+$$
+recall = TP /(TP + FN)
+$$
+
+
+---
+## Sensitivity-Precision Trade-off
+
+* When building a classifier, there is always a level of trade-off between the *recall* and *precision*.
+* Unfortunately, when working with classification problems, you can’t have it both ways: increasing precision reduces recall, and vice versa.
+
+---
+
+## Sensitivity-Precision Trade-off
+
+* Our KNN example from topic 1: Remember the coloured regions represent the decision boundaries.
+* We can move the decision boundaries to achieve higher/lower recall, but this is done at the cost of precision.
+.center[]
+
+
+---
+
+## Sensitivity-Precision Trade-off
+
+* In a perfect world we are aiming to train a classifier that classifies the instances in a way such that this trade-off is minimized.
+* In the previous example, this means learning decision boundaries that are detailed enough to achieve a high sensitivity, while not scooping up lots of false positives that drive precision down.
+* Within the inner workings of most classifier algorithms, the decision to assign an instance to a class is made based on assigning a *score*.
+* This score is compared to an arbitrary *threshold* value that is used to decide the classification.
+
+---
+
+## Sensitivity-Precision Trade-off
+
+* This threshold effectively represents the location of the decision boundary.
+* So by adjusting the threshold, you can manipulate the tradeoff between recall and precision and assess the performance of the classifier.
+* We can access the scores that the classifier assigns in the cross validation process and plot them understand the tradeoff.
+
+```python
+y_scores = cross_val_predict(sgd_clf, X_train, y_train_5, cv=3, method="decision_function")
+from sklearn.metrics import precision_recall_curve
+precisions, recalls, thresholds = precision_recall_curve(y_train_5, y_scores)
+```
+---
+
+## The ROC Curve
+
+* The receiver operating characteristic (ROC) curve is another common tool used with binary classifiers.
+* The ROC curve plots the *true positive rate* (another name for recall) against the *false positive rate (FPR)*
+* Once again there is a trade-off: the higher the recall (TPR), the more false positive
+(FPR) the classifier produces.
+* [Run the demo to produce the ROC curve for our data]().
+
+---
+
+## The ROC Curve
+
+* Interpreting a ROC curve is fairly straight forward:
+* The dotted line represents the ROC curve of a purely random classifier
+* A good classifier stays as far away from that line as possible (toward the top-left corner).
+* We assess this by calculating the *Area under the curve (AUC)*
+
+```python
+from sklearn.metrics import roc_auc_score
+roc_auc_score(y_train_5, y_scores)
+```
+
+---
+
+## The ROC Curve
+
+* We have looked at two curves for assessing performance PR and ROC:
+ * As a rule of thumb, you should prefer the PR curve whenever the positive class is rare or when you care more about the false positives than the false negatives.
+
+* For example, looking at the previous ROC curve (and the ROC AUC score), you
+may think that the classifier is really good.
+ * But this is mostly because there are few positives (5s) compared to the negative (non-5s).
+
+* The PR curve makes it clear that there is room for improvement.
+---
+
+## The ROC Curve
+
+* Take away points:
+ * Don't rely on accuracy alone as a measure of performance - especially with skewed datasets.
+ * The trade-off between the recall and precision determines the effectiveness of the classifier.
+ * PR and ROC curves provide a nice way to visualize this.
+
+---
+
+## The ROC Curve
+
+* Sensitivity Precision Trade-off
+* The ROC Curve
+* ROC Implementation
+
+---
+## Reading
+
+* Chapter 3 from ***Hands-on Machine Learning with Scikit-Learn & TensorFlow***
+
+---
\ No newline at end of file
diff --git a/topic_19/topic_19_roc_demo.ipynb b/topic_19/topic_19_roc_demo.ipynb
new file mode 100644
index 0000000000000000000000000000000000000000..7577e76aaf614eda1b3e28b0d2b47da8bc99502c
--- /dev/null
+++ b/topic_19/topic_19_roc_demo.ipynb
@@ -0,0 +1,287 @@
+{
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "# Topic 19 -ROC Performance Demo\n",
+ "\n",
+ "In this demonstration we will continue on from where we started in topic 17. First we will train a classifier algorithm on the MNIST data set. Then we will calulate or performance metrics and analyse these using a ROC curve."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 9,
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "image/png": "iVBORw0KGgoAAAANSUhEUgAAAOcAAADnCAYAAADl9EEgAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAGaElEQVR4nO3dPUiWfR/G8dveSyprs2gOXHqhcAh6hZqsNRqiJoPKRYnAoTGorWyLpqhFcmgpEmqIIByKXiAHIaKhFrGghiJ81ucBr991Z/Z4XPr5jB6cXSfVtxP6c2rb9PT0P0CeJfN9A8DMxAmhxAmhxAmhxAmhljXZ/Vcu/H1tM33RkxNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCLZvvG+B//fr1q9y/fPnyVz9/aGio4fb9+/fy2vHx8XK/ceNGuQ8MDDTc7t69W167atWqcr948WK5X7p0qdzngycnhBInhBInhBInhBInhBInhBInhHLOOYMPHz6U+48fP8r92bNn5f706dOG29TUVHnt8PBwuc+nLVu2lPv58+fLfWRkpOG2du3a8tpt27aV+759+8o9kScnhBInhBInhBInhBInhBInhGqbnp6u9nJsVS9evCj3gwcPlvvffm0r1dKlS8v91q1b5d7e3j7rz960aVO5b9iwody3bt0668/+P2ib6YuenBBKnBBKnBBKnBBKnBBKnBBKnBBqUZ5zTk5Olnt3d3e5T0xMzOXtzKlm997sPPDx48cNtxUrVpTXLtbz3zngnBNaiTghlDghlDghlDghlDghlDgh1KL81pgbN24s96tXr5b7/fv3y33Hjh3l3tfXV+6V7du3l/vo6Gi5N3un8s2bNw23a9euldcytzw5IZQ4IZQ4IZQ4IZQ4IZQ4IZQ4IdSifJ/zT339+rXcm/24ut7e3obbzZs3y2tv375d7idOnCh3InmfE1qJOCGUOCGUOCGUOCGUOCGUOCHUonyf80+tW7fuj65fv379rK9tdg56/Pjxcl+yxL/HrcKfFIQSJ4QSJ4QSJ4QSJ4QSJ4Tyytg8+PbtW8Otp6envPbJkyfl/uDBg3I/fPhwuTMvvDIGrUScEEqcEEqcEEqcEEqcEEqcEMo5Z5iJiYly37lzZ7l3dHSU+4EDB8p9165dDbezZ8+W17a1zXhcR3POOaGViBNCiRNCiRNCiRNCiRNCiRNCOedsMSMjI+V++vTpcm/24wsrly9fLveTJ0+We2dn56w/e4FzzgmtRJwQSpwQSpwQSpwQSpwQSpwQyjnnAvP69ety7+/vL/fR0dFZf/aZM2fKfXBwsNw3b948689ucc45oZWIE0KJE0KJE0KJE0KJE0KJE0I551xkpqamyv3+/fsNt1OnTpXXNvm79M+hQ4fK/dGjR+W+gDnnhFYiTgglTgglTgglTgglTgjlKIV/beXKleX+8+fPcl++fHm5P3z4sOG2f//+8toW5ygFWok4IZQ4IZQ4IZQ4IZQ4IZQ4IdSy+b4B5tarV6/KfXh4uNzHxsYabs3OMZvp6uoq97179/7Rr7/QeHJCKHFCKHFCKHFCKHFCKHFCKHFCKOecYcbHx8v9+vXr5X7v3r1y//Tp02/f07+1bFn916mzs7PclyzxrPhvfjcglDghlDghlDghlDghlDghlDghlHPOv6DZWeKdO3cabkNDQ+W179+/n80tzYndu3eX++DgYLkfPXp0Lm9nwfPkhFDihFDihFDihFDihFDihFCOUmbw+fPncn/79m25nzt3rtzfvXv32/c0V7q7u8v9woULDbdjx46V13rla2753YRQ4oRQ4oRQ4oRQ4oRQ4oRQ4oRQC/acc3JysuHW29tbXvvy5ctyn5iYmNU9zYU9e/aUe39/f7kfOXKk3FevXv3b98Tf4ckJocQJocQJocQJocQJocQJocQJoWLPOZ8/f17uV65cKfexsbGG28ePH2d1T3NlzZo1Dbe+vr7y2mbffrK9vX1W90QeT04IJU4IJU4IJU4IJU4IJU4IJU4IFXvOOTIy8kf7n+jq6ir3np6ecl+6dGm5DwwMNNw6OjrKa1k8PDkhlDghlDghlDghlDghlDghlDghVNv09HS1lyMwJ9pm+qInJ4QSJ4QSJ4QSJ4QSJ4QSJ4QSJ4QSJ4QSJ4QSJ4QSJ4QSJ4QSJ4QSJ4QSJ4QSJ4QSJ4QSJ4QSJ4QSJ4QSJ4QSJ4Rq9iMAZ/yWfcDf58kJocQJocQJocQJocQJocQJof4DO14Dhyk10VwAAAAASUVORK5CYII=\n",
+ "text/plain": [
+ "<Figure size 432x288 with 1 Axes>"
+ ]
+ },
+ "metadata": {
+ "needs_background": "light"
+ },
+ "output_type": "display_data"
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "Precision: 0.8370879772350012\n",
+ "Recall: 0.6511713705958311\n",
+ "F1 Score: 0.7325171197343846\n"
+ ]
+ }
+ ],
+ "source": [
+ "from sklearn.datasets import fetch_openml\n",
+ "mnist = fetch_openml('mnist_784', version=1)\n",
+ "mnist.keys()\n",
+ "\n",
+ "import matplotlib as mpl\n",
+ "import matplotlib.pyplot as plt\n",
+ "\n",
+ "X, y = mnist[\"data\"], mnist[\"target\"]\n",
+ "\n",
+ "some_digit = X[0]\n",
+ "some_digit_image = some_digit.reshape(28, 28)\n",
+ "plt.imshow(some_digit_image, cmap=\"binary\")\n",
+ "plt.axis(\"off\")\n",
+ "plt.show()\n",
+ "\n",
+ "import numpy as np\n",
+ "y = y.astype(np.uint8)\n",
+ "X_train, X_test, y_train, y_test = X[:60000], X[60000:], y[:60000], y[60000:]\n",
+ "y_train_5 = (y_train == 5) # True for all 5s, False for all other digits\n",
+ "y_test_5 = (y_test == 5)\n",
+ "\n",
+ "from sklearn.linear_model import SGDClassifier\n",
+ "\n",
+ "sgd_clf = SGDClassifier(random_state=42)\n",
+ "sgd_clf.fit(X_train, y_train_5)\n",
+ "\n",
+ "from sklearn.model_selection import cross_val_score\n",
+ "cross_val_score(sgd_clf, X_train, y_train_5, cv=3, scoring=\"accuracy\")\n",
+ "\n",
+ "from sklearn.model_selection import cross_val_predict\n",
+ "y_train_pred = cross_val_predict(sgd_clf, X_train, y_train_5, cv=3)\n",
+ "\n",
+ "from sklearn.metrics import confusion_matrix\n",
+ "confusion_matrix(y_train_5, y_train_pred)\n",
+ "\n",
+ "from sklearn.metrics import precision_score, recall_score\n",
+ "print(\"Precision:\",precision_score(y_train_5, y_train_pred)) # == 3530 / (3530 + 1891)\n",
+ "print(\"Recall:\", recall_score(y_train_5, y_train_pred)) # == 3530 / (3530 + 687)\n",
+ "\n",
+ "from sklearn.metrics import f1_score\n",
+ "print(\"F1 Score:\",f1_score(y_train_5, y_train_pred))\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Now we can calculate the scores used to analyse the recall-precision trade-off. We will done by setting the method to \"decision_function\". The scores can then be used to calulate the precision and recalls for the range of thresholds used by the classifier."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 10,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "y_scores = cross_val_predict(sgd_clf, X_train, y_train_5, cv=3,\n",
+ "method=\"decision_function\")\n",
+ "\n",
+ "from sklearn.metrics import precision_recall_curve\n",
+ "precisions, recalls, thresholds = precision_recall_curve(y_train_5, y_scores)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Now we can plot the recalls and precisions against threhold values to understand the nature of the trade-off. We will define a function that will print the two data series. "
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 4,
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "image/png": "iVBORw0KGgoAAAANSUhEUgAAAfYAAAEPCAYAAACwduZtAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAgAElEQVR4nO3dd3xUVdrA8d+TSiD0DqGDQKQTEJBmoapgF+xrwbLoqsjq6r4ra1/dtayIihUUFVYXl3UBUUFQpIUuTar03klIPe8fZyaZhEkygZm5M5Pny+d+bjtzz3Nzhzy57RwxxqCUUkqpyBDldABKKaWU8h9N7EoppVQE0cSulFJKRRBN7EoppVQE0cSulFJKRRBN7EoppVQEKTGxi8gHIrJfRH4pYr2IyD9FZJOIrBKRTv4PUymllFK+8OWM/SNgYDHrBwEtXMMI4K1zD0sppZRSZ6PExG6MmQccLqbIUGCisRYCVUSkrr8CVEoppZTvYvywjfrADo/5na5lewoXFJER2LN6SKAzVfxQu4pMAtESTXRUNFESRbREIyIIgoi4itjp2KhYYqNjbXl3ORGixX5WELuNKLu9aIl2eOdUJNmxA/bv974uPh7atMmfX74ccnO9l01Kgtq17fThw7B1a8H1iYnQsuW5x6uctXTp0oPGmJqBrMMfiV28LPPaTq0xZjwwHqBRciPzxMQnPD5gCpcttMHSrffHNsKljozsDLJyszDGkGtyMbjGHvNFrUvLTsubd5dz1+G5zHPsjsmXMqeyTpGRnUFGTgYnMk6QlZtFrsktdsjKySIjJ4Mc1z9/i42KpWpCVcrFlCMuOo7EuEQqxlWkXEw5qpSrQkJsAgkxCVQtV5XEuESqJlSlQmwFqpSrQmJcIvUq1qNVjVZ5f2CosmfJEti0CVJS4NgxWL/ee7lKlWDIkPz5zz6DnCK+0p07Q+vWdnrrVpg/v+D6WrWgf/9zj105S0R+C3Qd/kjsO4EGHvNJwO6SPlSzfE3uSbnHD9WrSJSRncGR00c4kXGC9Ox0jmccJysni+zcbHJMDjm5OWTnZnM6+zS7TuziUNohDqUf4nT2aTJyMkjLSuNU5ikyczLJyMng2OljnMw8mVdm/6kiTrF8lBiXSM3yNWlVoxVtarWhUnwlmlZtSrva7WhVoxUxUf74r6VC1ccfwxtvwKuvwkMP2QTvi+HDfSvXpIkdlDob/vjtMw0YKSKfAxcAx4wxZ1yGV6o04mPiqZNYhzqJdfy+7bSsNI6dPlYg6adnp3My8yTHM46TkZ3B8YzjnMg8wbHTxziWcYxTWac4nH6YtKw01h1Yx75T+ziZeZKtR7cyY9OMAtuPi46jVY1WdKvfjSZVm9CyeksGNh9IQmyC3/dFOePECTuuWNHZOJTypsTELiKfAX2BGiKyE3gKiAUwxrwNTAcGA5uANOB3gQpWKX8oH1ue8rHlz/rzxhiOnD7CjmM7WHtgLduObmP/qf1sPrKZJbuXsPfkXlbtW8WqfavyPhMXHceQlkPoULsDA5sPpF3tdsRGx/pjd5QDNLGrUCZOdduakpJiUlNTHalbqUA6kn6ElftWsnrfajYf2cy3W75l7YG1BcokxiUy+drJDG4x2KEoQ9OJEzBtGvTuDQ0alFzeKQMGwKxZMH06DBrkdDQqnIjIUmOMjzdvzo7eCFTKz6omVKVv4770bdw3b9mmw5uYs3UOP+/8mY9WfMTJzJNc9ull9GzYk0lXT6Jh5YbOBRxgR45AZmb+E9+eMjIgLs6Op06FZ56Bdevsuo8+gttuC2qoPtMzdhXKtElZpYKgebXm3N35bj4c+iFpT6TxSLdHiJIoftr+E63fbM0/F/2TU5mnnA7Tb06fhpgYEIFq1aBOHZuo582zy665BqKioFw5GDUK/vxnuPHG/KQOcPHFcOoUXHklvP++Y7vilSZ2FcpC+lL8sWPHOHjwIJmZmUGKSoWKuLg4atSoQeXKlZ0OJWC2HtnKXf+9i9lbZwNQLaEa/+j/D27vcLuzgZ2jvXuhrpcmqh54wL7G9fXXZ6775BO4+eb8+c2boWlT+OEHuOgiu+yxx+D55+0fBE5r0gS2bbOvvDVr5nQ0KpwE41J8yCb206dPs337dpKSkkhISNB3hssQYwzp6ens3LmThg0bUq5cOadDCqjPVn/GE7OfYNvRbQCM6DSCty5/iygJgQxWSunpsHs3dOoEx4/nL1+9Gs47zybDwo2sLF1qy6el2fvW/ftDeY9nGydMgDvvtO9/X3IJ/Pvf9v1wJ2VlwcmTNo5obe9IlUKZTuw7duwgMTGRqlWrBjEqFUoOHz7MqVOnaBDKT1H5iTGG1xa+xh+/+yPZudk82v1RXu7/stNhlcovv0DbtjB0qG2UpUcPaNWq6PLbt9vW1nw5A585E4YNs43BtGxpH1pr2tR/sSsVLMFI7CF7SnD69GkSExOdDkM5qGLFipw+fdrpMIJCRHi4+8NMvHIiAH9f8Hfu+/q+M1odDEXPPGPvm7dta+f/8x/o16/4pA7QsKHvl9UHDoSFC6FRI9iwwZ7V79tXdPOsSpVlIZvYs7OziYnRh/bLspiYGLKzs50OI6iGtx3Ou1e8S7RE8/bSt3l67tNOh1QsY2Ds2ILL/vjHwLyq1qoVrFxpL9s3amRbfhs0CH791f91FefoUbjgArj66uDWq5SvQjaxA3pfvYwrq8f/rk53MfnayQCMmTuG8UvHOxxRQVOn2jN0EbjnHpvc3TZvhr/9LXB1V64M334LM2bYhD5rlm3SNZgXNo4ehcWLQZvhUKEqpBO7UmXVNcnX8OxFzwJwz9f38P4y59/3SkuzydzzTPXdd+39b2PsEIz73tWq2XffX3oJKlSwSf7ttwNfr5v7VTe9U6hClSZ2pULUE72e4MGuDwJw13/vYv3BIroQC5L//e/MZU89ZS+NO6FxY3jrLTt9//3BO4M+edKO9R12Fao0sQfRRx99lNdXuIhQsWJF2rdvz9ixY4N6L3nMmDGlvszdt29f+vbtG5iAlFciwmsDX+OK864A4IrPruBExomgx1Gnjj1T7949f1lOjj1DHzMm6OEUcMstcN11dvrpID2OoI3TqFCnid0B//rXv1iwYAFffvklXbt25YEHHuDpYP1WAu666y4WLFhQqs+MGzeOcePGBSgiVRQR4YOhH9C8WnM2Hd7EH7/9Y9DqXrXKJvR9++x8dnb+JfdQaCTG7fXX7fi772zTtYGmiV2FuhD671l2dOjQgW7dutG/f3/effdd+vbty2uvvea1rDHG7y3vJSUl0a1bt1J9Jjk5meTkZL/GoXxTo3wNvrz+SwTh3WXvFug1LlB69YL27fPnK1Wyl75DUd260K2bbaHu4MHA16eJXYU6TewhoEuXLpw4cYL9+/fTuHFjbr75Zj744ANatWpFXFwc/3Pd3ExLS+Oxxx6jSZMmxMXF0aRJE5577jlyC73Me+DAAe6//34aNGhAfHw8DRo04JZbbiEjIwPwfin+9ddfp3Xr1iQkJFC1alVSUlKYOnVq3npvl+I3bNjAVVddRZUqVUhISKBbt27MnDmzQBl3XRs3buSyyy4jMTGRRo0a8fTTT58Rtypau9rtuKfzPeSYHG7/6vaAtyu/Y0fB+WPHAlrdOfv5Z/sMQL16ga+rSRO49Vb7x49SoSgsE7v7VRtvw3iPN4PGjy++rKfOnYsuN2JEfrmlS/2/P1u3biU6OjqvQZ45c+bwyiuv8NRTTzFz5kzatWtHdnY2AwYM4L333uMPf/gDM2bM4K677uKZZ55h9OjReds6cuQIPXr0YPLkyTzyyCNMnz6dl156iaysrCLP/CdNmsSoUaMYPnw406dPZ9KkSVx77bUcPny4yJh3795Nz549WblyJWPHjmXKlClUqVKFyy67jBkzZpxR/qqrruLiiy/mq6++4sorr+Spp55iwoQJ5/iTK1uev+R5mlVtxvK9y3kr9a2A1OF+beyvf7XjgweD+yrZ2Qrmm5F9+thmbu++O3h1KlUqxhhHhs6dO5virF27tsh1+Xf6zhzeeSe/3DvvFF/WU6dORZe7++78cqmpxYZdrA8//NAAZv369SYrK8scPnzYvP322yYqKsoMHTrUGGNMo0aNTEJCgtmzZ0+Bz06cONEAZu7cuQWWP/vssyY2Ntbs27fPGGPM//3f/5moqCizbNmyIuN46qmnDB4/gN///vemY8eOxcbep08f06dPn7z5UaNGmejoaLNx48a8ZdnZ2ea8884rsC13XR988EGB7bVp08b069ev2DqNKf57UBb9Z/1/DGMwNV+qadIy0/y67ZwcY1q1MubZZ41JT/frpoNi3Tpjxo0z5vvvnY5EqaIBqSbA+TUsz9iLS9eeZ9cjRhRf1tPSpUWX87wK0LnzucffqlUrYmNjqVatGvfffz833XQTH3zwQd76bt26UadOnQKfmTlzJo0aNaJHjx5kZ2fnDf379ycrK4uFCxcCMGvWLLp06ULHjh19jqdLly6sWLGCBx54gO+++460tLQSPzNv3jy6detG8+bN85ZFR0czfPhwVqxYwXHPHkCAyy67rMB8mzZt2L59u88xKuvy8y6nVY1WHEg7wJfrvvTrtqOjYf1624VqTo5fNx0UP/xgX3ubNCmw9WzbZtvFL/QVVypkhGViD3dTp05lyZIlrF+/nlOnTjFx4kSqVauWt76ulz4v9+/fz2+//UZsbGyBoWvXrgAcOnQob5yUlFSqeG699VbeeustFi1axIABA6hWrRpXX30127ZtK/Izhw8f9hpnnTp1MMZw5MiRAss99w8gPj6+zLQD709REsX9KfcD8OGKD/22XfeT72AbmalQwW+bDhp396lr1wa2njFjbLv4X3wR2HqUOlvaGLsD2rRpU+BMtzBv75hXr16dJk2aMGXKFK+faex6ZLlGjRrs2rWrVPGICPfccw/33HMPR44cYdasWYwaNYobbriBRYsWef1MtWrV2Lt37xnL9+7di4ickciV/9zc7mYe++4xZm+dzbebv6Vfs37ntL3Dh+276m6uiz9hx/0U/6JF9mw6UF276lPxKtTpGXuYGDhwYF5XtikpKWcMNWrUAKB///4sXryYlStXnlU9VatW5YYbbuD666/nl19+KbJcnz59WLhwYYGz+pycHCZPnkzHjh2pqL/1AqZqQlX+cMEfAHj8+8fPuQc4z7s2I0ZAzZrntDnH1Kple4wzxvYAFyia2FWo08QeJm666SZ69OjBJZdcwiuvvML333/PjBkzGDt2LP3798+7L/7www/TtGlTLr30Ul5//XVmz57NlClTuOmmmzhxwnurZSNGjGDUqFF88cUXzJs3j/fee4+PP/6Y/v37FxnPww8/TJUqVejXrx+ffvopX3/9NVdccQW//vorzz33XEB+Birf//X5PyrEVmDZnmX8tP2nc9rWK6/Y8XXXwTvv+CE4Bw0YYMfPPlvy0/zp6b5v9+uvbf/xoIldhT5N7GEiNjaWb775hrvvvpvx48czePBgbrrpJiZMmECPHj2Ii4sDoEqVKsyfP5+rrrqKF198kYEDBzJq1ChiYmLyyhR24YUXsnTpUu6//3769evHc889x80331zs62j16tXjp59+4vzzz+e+++7Lez3uf//7HwMHDgzIz0DlKx9bnpFdRwLw5pI3z3o7EyfahGUMFHGXJ6w8+ijEx8O0abYHNrcvv4QHHshvme7oUWjZEh57rOQEv3w5XHGF7SoWtBMYFQYC/dh9UcO5vO6myg79HhRty+EthjGY+GfizdH0o6X+/M8/57/7cfp0AAJ0yOjRxjz2mH19z829n88+a+cnTzZGxC4bPdou27XLmMcfN+bgwYLbmzkz//Pr1hnTqJGd3rw5KLujIgz6uptSqihNqjahd6PeZORkMHnN5FJ/vkeP/On4eD8G5rAXX7SDt/bs3U+yX389vPqqnd640Y4fecR+zvWiSZ4Yj0eMExP1UrwKfZrYlQpjd3a8E4B3l71bqs95dnE6fLg/I3KeZ0LfsQMWLAB3a8grVsD8+fDEE3DkiC371VcweTIcOGDLbNlScHvud/ovvRSSkmD2bJg7F6pWDfiuKHVWNLErFcauS76OSvGVSN2dyroD63z+XJcu+dOBbtDFKcuWQatWcOONtue3li3t8i+/hBdegK1b8x8cfPjh/H7ln3224HbciT062o7bt4fevQueySsVSjSxKxXGEmITuKrVVQBMWePb02/z5uVPT5sW3HbWg6ldO0hLsy3F3XMPJCdDlSqwe7ddHxcHDz4I550He/bArFnet5OYaLdVvrztIjZS/xBSkUMTu1Jh7trkawH47JfPSv1O+xVXBCKi0OB5Rv3++/CnP8H+/fm9ssXF2T9qHnrIvr/vbkgxJ8fea3ffS+/VC1auhDvusGVvvhmeeSa4+6JUaWhiVyrMDWg2gGoJ1dhwaAMbD28ssXyvXvZec3Z2EIILIV27wm+/5b/y5n7787777Pv77kvxTz1l/wi48caCbeZfckn+9EsvBSdmpc6GJnalwlxsdCwXNb4IgM9/+bzYsu+8Yy+/t2+ff884kr3+esH5qKj8tuQTEgque/hh2wnOwoVQrZptlOaPf8xfn5CQ39/7yZOBi1mpc6WJXakIcG/KvYDtGKaoy/HZ2XDvvXDllQWfio9kDz5oL7+7ffUVvPeenfbszmDXLttAT61acMEF9gG7mBj7cN2TT9rpm2+29+oBXC04KxWSNLErFQEuanwRtSvUZtvRbSzbs8xrmZ9/zp/2R/fD4cKz7XvPxhdr186fvu02GDnSnqWDfT3ujjvs9OTJ9pK8MTbJv/IKfP99wMNW6qxpYlcqAkRHRTOoxSAAZm32/nh3nz52/MgjkfskfFGWLoX//Md2SQswaBDcckv++qvsiwXcequ9VeG5bPNmO46OtsPDD9un5JUKVZrYg+ijjz5CRPKGuLg4mjVrxhNPPOF43+SNGzfm9ttvz5t3x1pcn+wqtAxoZntAmb1t9hnrjh3Ln/Y4zGVGp04wZEj+Gbv7ATq3oUPzp5cuteOLLy7Y9WtZeCZBRQZtYsEB//rXv0hKSuLEiRNMnTqVF154gRMnTvDGG284HZoKYxc3uRhBmPfbPNKy0igfWz5vnedT3G3bOhBciHA3nXvqVMHlSUn50+7OY+Li4LXX4Jtv7OV4TewqXPh0xi4iA0Vkg4hsEpHHvaxvKCJzRGS5iKwSkcH+DzVydOjQgW7dutGvXz/GjRvHpZdeyvvvv09ubq7ToakwVqtCLTrU6UBmTiYLdizIW24MPP+8nX7sMYeCCxHnnWfHCxfaRmk8jRtnxw8/nL/sd7/Lb45WE7sKFyUmdhGJBt4EBgHJwHARSS5U7M/AFGNMR2AYMM7fgUayTp06kZ6ezsGDB/OWbd26lZtuuomaNWsSHx9Phw4dmDp16hmfXblyJVdddRXVq1cnISGBli1b8sILL+StnzVrFoMHD6Zu3bqUL1+eNm3a8I9//IMczxd0VcRwv/Y2Z9ucvGW5ufb+8i23wHPPORVZaKhdGxo2tE+1V69ecN1990FGBvTvX3B54SZllQp1vlyK7wpsMsZsARCRz4GhwFqPMgZw342qDOz2Z5Ce5K+h8dSPeap0LXwVZ9u2bVSuXJnqrt80O3bs4IILLqBWrVq8+uqr1KxZk8mTJ3PNNdfw1VdfMWTIEAAWL15M3759ad68Oa+++ipJSUls3LiRVatW5W17y5YtXHLJJTzwwAOUK1eO1NRUxowZw4EDB3jxxRf9tg8qNFzU5CJeWfhKgcQeHW1f4erVS5MTwIYN9h675xPybt6WuRvyycgIbFxK+Ysvib0+sMNjfidwQaEyY4BZIvIAUAG41NuGRGQEMAKgYcOGpY01YuTk5JCdnZ13j/3LL7/ktddeI9r1W3fMmDEYY5g7d25esh8wYAA7duzgL3/5S15if/TRR6levToLFy6kfHl7P/Xiiy8uUNe9996bN22MoVevXmRmZvL3v/+d559/nihvfVuqsNWrYS+iJIrFuxZzMvMkFWITSU8v+GpXWVeunB18tXevHTdrFph4lPI3XxK7t1Pkwqerw4GPjDH/EJHuwMci0sYYU+CmsTFmPDAeICUl5axOef15puyUVq1aFZi///77GTlyZN78zJkzGTx4MJUrVybbo93PAQMGMHr0aI4fP05MTAzz589n9OjReUndmz179jBmzBhmzpzJ7t27C2xv//791KlTx497ppxWuVxlOtftzJLdS5i/fT4HFw3g5pvtulI2I69c3Fc59BEYFS58OV3bCTTwmE/izEvtdwJTAIwxC4BygLbNVISpU6eyZMkSpk+fzqWXXsq4ceOYOHFi3vr9+/czceJEYmNjCwyjR48G4NChQxw5coTc3FySPB/nLSQ3N5chQ4bw9ddf8+c//5nZs2ezZMkSnnzySQDHX7FTgeF5n92d1NXZc9/Zcj8tr1So8+WMfQnQQkSaALuwD8fdWKjMduAS4CMRaY1N7Af8GWgkadOmDc2bNwfspfN27doxevRorrnmGipUqED16tXp1asXjxXxCHO9evXIyckhKiqKXbt2FVnP5s2bSU1N5eOPP+Zmj9/w//3vf/27Qyqk9G3cl5d+fonvt2rzaP4wY4Yd638bFS5KPGM3xmQDI4FvgHXYp9/XiMjTIjLEVWwUcLeIrAQ+A243pe0/soyKj4/n5ZdfZv/+/YxzvW8zcOBAVq1axfnnn09KSsoZQ3x8POXLl6dnz5588sknpKene912WloaALGxsXnLsrKymKQdSke0Xo16ERsVS+ruVEg4DMCECQ4HFcbat7fj1q2djUMpX/nUQI0xZjowvdCyv3hMrwUu9G9oZceQIUPo0qULf//73xk5ciRPP/00Xbt2pXfv3owcOZLGjRtz5MgRfvnlF7Zs2cIHH3wAwN///nf69OlD9+7dGTVqFElJSWzZsoUVK1bwxhtv0Lp1axo1asSTTz5JdHQ0sbGxvPrqqw7vrQq0xLhEutbvyvwd8yFpAWy8jGHDnI4qfP3737aBn0cfdToSpXyjj0SHiGeffZb9+/fz9ttv07BhQ1JTU2nfvj1PPPEE/fr147777mPu3LkFnnrv0qUL8+fPp0GDBjzwwAMMHjyYl19+Oe++e1xcHF999RV16tTh1ltv5fe//z29e/fm8cfPaGNIRZjejXrbiWbfAt5f41K+adgQxo6Fxo2djkQp34hTV8xTUlJMajF9R65bt47Weu2rzNPvwdmZsXEGgz8dDDu6w/s/6xPxSoUIEVlqjEkJZB16xq5UBOpSv4udqLuMF/6WXXxhpVRE0cSuVASS9BpwrAHEZHDZzZudDkcpFUSa2JWKQHPmAHs6AZC672dng1FKBZUmdqUi0HXXAVvtg5bTN00vvrBSKqKEdGLXV+HLNj3+52jjIAB+2PYDuUbbQ1WqrAjZxB4bG1tkwyuqbEhPTy/QuI7yzcmTronDzakZX5+DaQdZs3+NozEppYInZBN7rVq12LVrF2lpaXrmVsYYY0hLS2PXrl3UqlXL6XDCzqxZ7ilhQMsz+2dXSkU2n1qec0KlSrZ79927d5OVleVwNCrYYmNjqV27dt73QPmud287DB8OcY0v4pNVnzBn2xwevOBBp0NTSgVByCZ2sMldf7ErVTo//QQPPwx9+8Jh0xeAudvmkmtyiZKQvUinlPIT/V+uVIR56im46ipYtw6aVGlCw8oNOXL6CKv2rXI6NKVUEGhiVyqCZGTA2rUgAm3bgojk98++Ve+zK1UWaGJXKoKsXQvZ2dCiBSQm2mV5iV0foFOqTNDErlQEWbrUjjt1yl92UROb2Of9No+c3BwHolJKBZMmdqUiyLJlduyZ2BtWbkjTqk05lnGMpXuWOhOYUipoNLErFUHeesuOPRM7QN9GfQH4eYe2G69UpNPErlQEat++4HynujbT65PxSkW+kH6PXSlVOkePwunTUKNGweVta7cF4Jf9vzgQlVIqmPSMXakIUrky1K595vK2tWxiX7lvJaezTwc5KqVUMGliVypC7N8PRbW+XDWhKufXPJ/MnExSd6cGNzClVFBpYlcqQtSuDXFx8N133tf3aNADgKW79cl4pSKZJnalIkCOx+vpzZt7L9O+tn2ibuW+lUGISCnlFE3sSkWAFSvypxs39l6mXe12gCZ2pSKdJnalIsA339jxtdcWXaZd7XZESRSr9q3SB+iUimCa2JWKANOn27FI0WUql6tMcs1ksnOzWbxrcXACU0oFnSZ2pSLAokV2fNllxZfr3bA3AAt2LAhwREopp2hiVyoCZGfbceEW5wrT++xKRT5teU6pCHD8OKxcCe3aFV+ua/2uACzbsywIUSmlnKBn7EpFgIoVoWdPiCrhf3RyzWSiJZqNhzeSlpUWnOCUUkGliV2pMiQ+Jp7za51Prsllxd4VJX9AKRV2NLErFebuuAMGDLCX4n3RuW5nQFugUypSaWJXKszNmQOzZkFsrG/l3R3CrD2wNoBRKaWcooldqTB27Bhs22anzzvPt8+0rtkagDUH1gQmKKWUo3xK7CIyUEQ2iMgmEXm8iDLXi8haEVkjIp/6N0yllDfuFucAYnx8x6Vz3c4IwuJdi8nMyQxMYEopx5SY2EUkGngTGAQkA8NFJLlQmRbAn4ALjTHnAw8FIFalVCFrzuKku2aFmjSv1pyMnAzWHVjn/6CUUo7y5Yy9K7DJGLPFGJMJfA4MLVTmbuBNY8wRAGPMfv+GqZTy5oMP7Lh799J9rkOdDgAs37vczxEppZzmS2KvD+zwmN/pWubpPOA8EZkvIgtFZKC3DYnICBFJFZHUAwcOnF3ESqk8O3fa8YgRpftcxzodAfSVN6UikC+J3Vu3EqbQfAzQAugLDAfeE5EqZ3zImPHGmBRjTErNmjVLG6tSqpArrrDjgV7/lC6anrErFbl8edxmJ9DAYz4J2O2lzEJjTBawVUQ2YBP9Er9EqZTyatq0s/tcx7r5Z+zGGKS4buGUUmHFlzP2JUALEWkiInHAMKDwr5OvgIsARKQG9tL8Fn8GqpTynzqJdahdoTbHM46z9ehWp8NRSvlRiYndGJMNjAS+AdYBU4wxa0TkaREZ4iNdFmMAACAASURBVCr2DXBIRNYCc4DRxphDgQpaKQX/+x/885/w669n93n3WfvyPXo5XqlI4tObr8aY6cD0Qsv+4jFtgEdcg1IqCH73OzhwAN57z/fGaTydX/N8Zm6ayYZDG/wfnFLKMdrynFJhyv1iydkkdYCW1VsC8Mv+X/wUkVIqFGhiVyoMpXn0uNqly9ltI+9SvD4Zr1RE0cSuVBha52owLjkZypU7u220qdWGaIlmw8ENnMo85b/glFKO0sSuVBhyNyV7/vlnv41yMeVIrpmMwbBq3yr/BKaUcpwmdqXC0Pr1dty69bltRy/HKxV5NLErFYYqVoSkJGjR4ty206lOJ0BfeVMqkmhiVyoM/elPsGMH3HTTuW0npV4KAD/v/NkPUSmlQoEmdqXC2Lm2BJtSL4UKsRVYe2At+07u809QSilHaWJXKsycPAnHj/tnW/Ex8bSv0x6AlftW+mejSilHaWJXKsxMmQKVK8MDD/hne53rdgZg6e6l/tmgUspRmtiVCjOrXG+m1avnn+2577Mv3aOJXalIoIldqTCzerUdt2vnn+11qWebrpv721xyTa5/NqqUcowmdqXCiDGw0nUr3F+JvVWNViRVSuJg2kHWHVjnn40qpRyjiV2pMLJ3Lxw6BFWq2PfY/UFEuKD+BQAs3rXYPxtVSjlGE7tSYcR9f71du3N/1c1Tr4a9AJizbY7/NqqUcoQmdqXCiGdi96fuDboDsGT3Ev9uWCkVdDFOB6CU8t2tt9oe3fz1RLxbxzodSYhJYP3B9RxJP0LVhKr+rUApFTR6xq5UGKldGy67DDp29O92Y6Nj8xqqWbZnmX83rpQKKk3sSikAUura99n1ATqlwpsmdqXCxK+/wi23wIcfBmb7PRr0AGDe9nmBqUApFRSa2JUKE0uWwCefwH//G5jt92pkn4xfsGOBNlSjVBjTxK5UmFju6jLd30/EuyVVSqJh5YYcyzjGmv1rAlOJUirgNLErFSaWuN5E69IlcHW4L8f/vEP7Z1cqXGliVyoM5OTAMtfD6ikpgavnwgYXAjB/x/zAVaKUCihN7EqFgc2bbT/s9evbV94CRRO7UuFPE7tSYSBQLc4V1rZ2WxLjEtlyZAt7T+4NbGVKqYDQxK5UGKhSxTZMc/HFga0nJiqGbkndAJi/Xc/alQpHmtiVCgOXXgpffw2PPhr4utwdwny75dvAV6aU8jtN7EqpAi4/73IApm+c7nAkSqmzoYldqRB38iTMnw8nTgSnvg51OlA+tjw7ju/gSPqR4FSqlPIbTexKhbjFi6FnT+jXLzj1RUkUyTWTAVi+d3lwKlVK+Y0mdqVC3NKlduzvHt2Kc0H9CwBI3Z0avEqVUn6hiV2pEOduca5r1+DV2bGO/Stixd4VwatUKeUXmtiVCnHuM/ZAtjhXWIc6HQC9FK9UONLErlQIO3YMtmyBcuWgdevg1dumVhsSYhJYf3A9B04dCF7FSqlz5lNiF5GBIrJBRDaJyOPFlLtWRIyIBPHcQqnItcbVyVqrVhATE7x642PiSaln/xsv3rU4eBUrpc5ZiYldRKKBN4FBQDIwXESSvZSrCDwILPJ3kEqVVStX2nGnTsGvu2t9e1N/0S79L61UOPHlHKArsMkYswVARD4HhgJrC5V7BngJCELbWEqVDffeC5dfbnt3CzZ3Yl+6Z2nwK1dKnTVfLsXXB3Z4zO90LcsjIh2BBsaYr4vbkIiMEJFUEUk9cEDv2ylVEhFo0AAaNw5+3e7EvnDnQowxwQ9AKXVWfEns4mVZ3v9yEYkCXgVGlbQhY8x4Y0yKMSalZs2avkeplAq6RpUbUSexDofTD7P5yGanw1FK+ciXxL4TaOAxnwTs9pivCLQBfhCRbUA3YJo+QKfUuZk7F9q0gWefdaZ+EaFz3c4ALNqp99mVChe+JPYlQAsRaSIiccAwYJp7pTHmmDGmhjGmsTGmMbAQGGKM0SarlDoHixbZp+J37y65bKD0adQHgO+2fudcEEqpUikxsRtjsoGRwDfAOmCKMWaNiDwtIkMCHaBSZZW7xbkuXZyLoV8z20D93G1znQtCKVUqPr0Za4yZDkwvtOwvRZTte+5hKaVCIbG3qdWGcjHl2Hp0K0fSj1A1oapzwSilfKItzykVgvbuhd9+g8TE4LY4V1hMVAzta7cHtKEapcKFJnalQtCCBXbctStERzsbS8+GPQH4cfuPzgailPKJJnalQtAPP9hxz56OhgFA38Z9AZi9dbazgSilfBLE1qeVUr667jrb8cvQoU5HAr0b9SZaolm8azHHM45TKb6S0yEppYqhZ+xKhaCePeFvf3OmjfjCKsVXomv9ruSYHH78TS/HKxXqNLErpUrkvhw/Z9scZwNRSpVIE7tSIebjj+HNN51tmKawfk3t++zfbvnW4UiUUiXRe+xKhZjXXoNly6BlS6hXz+lorB4NepAQk8CqfavYd3IftRNrOx2SUqoIesauVAg5dgxWrICYGOje3elo8sXHxHNhwwsBfe1NqVCniV2pEPLTT5Cba1ubq1DB6WgKuqjxRQBMWj3J4UiUUsXRxK5UCLn2Wjvu2tXZOLy5rf1tAMzaPIuM7AyHo1FKFUUTu1Ih5PRpO05OdjYOb+pXqk+72u1Iy0pj3m/znA5HKVUETexKhYhdu/Knb7vNuTiKM7j5YABmbJrhcCRKqaJoYlcqRBw/DtdcA7fcAvHxTkfj3aAWgwBN7EqFMn3dTakQ0bo1fPGF01EUr3tSdyrFV2L9wfVsPbKVJlWbOB2SUqoQPWNXSvksNjqW/s36AzB943SHo1FKeaOJXakQ8OOPULMmzAmDFlsvb3E5AJ+s/sThSJRS3mhiVyoEjB0LBw/Cyy87HUnJrkm+hrjoOBbtXMShtENOh6OUKkQTu1IOy821DdMAPP20s7H4IjEukR4NemAw/LT9J6fDUUoVooldKYctWWI7fGnQADp3djoa33Sr3w3Q5mWVCkWa2JVy2Pvv2/GVV4KIs7H4akDzAQD8e92/McY4HI1SypMmdqUcZAy8+66dHjLE2VhKo2fDntQoX4OtR7eydM9Sp8NRSnnQxK6Ug5Z65MS+fR0Lo9RiomK4Pvl6AGZumulwNEopT9pAjVIOOnYM2ra1ST0mzP43dqzbEYC1B9Y6HIlSylOY/SpRKrJccgmsWgWZmU5HUnrdk2yH8d9u+Zbs3GxiovTXiVKhQC/FKxUC4uKcjqD0kmsm07J6Sw6mHeSHbT84HY5SykUTu1IOmTABNm1yOoqzJyJcf769zz5lzRSHo1FKuWliV8oBmzbB7bfb++tHjzodzdlzJ/ap66fqa29KhQhN7Eo54K237HjYMKhSxdlYzsX5Nc+nbmJdDqYdZNGuRU6Ho5RCE7tSQXfsWP676yNHOhvLuRIRbmx7IwATV050OBqlFGhiVyro3n0XTpyAiy4KnyZkizOszTAAvlr/Fbkm1+FolFKa2JUKosxMeO01O/3oo87G4i+d63amcZXG7Dm5h593/Ox0OEqVeZrYlQqiyZNh1y5IToZBg5yOxj9EhCHn2fZwp2+c7nA0SilN7EoFUdeucP31MHp0+HT44ourW18NwPil4zmVecrhaJQq23xK7CIyUEQ2iMgmEXncy/pHRGStiKwSke9FpJH/Q1Uq/LVsac/ab7/d6Uj8q3ej3nSt35VD6Yd4K/Utp8NRqkwrMbGLSDTwJjAISAaGi0hyoWLLgRRjTDvgC+AlfweqVDjLyoLcCH6uTER47MLHAH06Ximn+XLG3hXYZIzZYozJBD4HhnoWMMbMMcakuWYXAkn+DVOp8Pa3v0H37rB8udORBM5lLS6jSrkqrN6/mhV7VzgdjlJlli+JvT6ww2N+p2tZUe4EZnhbISIjRCRVRFIPHDjge5RKhbHt2+GFF2Dx4vBuZa4k8THx3NLuFgDeXfquw9EoVXb5kti9PeLjte1IEbkZSAFe9rbeGDPeGJNijEmpWbOm71EqFaaMgbvugrQ0uO46++56JLur010ATFo9ifSsdIejUaps8iWx7wQaeMwnAbsLFxKRS4EngSHGmAz/hKdUeHv/ffj2W6hWDd54w+loAq9d7Xak1EvhWMYxpq6f6nQ4SpVJviT2JUALEWkiInHAMGCaZwER6Qi8g03q+/0fplLhZ8cOGDXKTo8dC7VrOxtPsNzZ8U4APlzxocORKFU2lZjYjTHZwEjgG2AdMMUYs0ZEnhaRIa5iLwOJwL9EZIWITCtic0qVCTk5MHw4HD8OQ4bYzl7KiuuSr6NcTDm+2/Idy/Ysczocpcocn95jN8ZMN8acZ4xpZox5zrXsL8aYaa7pS40xtY0xHVzDkOK3qFRki46GO+6Apk3hvfciqzGaklQvX507OtwBwKsLX3U4GqXKHm15TqkAueMOWLsWyuJzoo90fwSAf6/7NyczTzocjVJliyZ2pfzo009h4cL8+fh452JxUrNqzejRoAdpWWlMWDHB6XCUKlM0sSvlJ2++CTfdZDt32bPH6Wic92DXBwF4e+nbGOP1DVmlVABoYlfqHBkDzz0HI0fa+SeegLp1nY0pFFzZ6krqJtbll/2/MGGlnrUrFSya2JU6Bzk58PDD8Oc/2wfk3n3X9tymbEt0j/e0fUb96fs/kZWT5XBESpUNmtiVOkt790K/fvD66xAbC59/bluZU/ke6PoAyTWT2XtyL5+s+sTpcJQqEzSxK3WWdu6EH3+0Dc98953tZ10VJCKM6m5b6RmXOs7haJQqGzSxK1UK27fbe+oAKSkwcaLtsa13b2fjCmXD2wynarmqpO5O1QZrlAoCTexK+WDVKrjlFmjWDKZ6NIE+fLg+KFeShNgEbmt/GwBjF491OBqlIp8mdqWKYAzMng0DB0L79vDJJ3ZZJPepHij3ptyLIExYOYGDaQedDkepiKaJXSkvZs2CLl3gkkvgm2+gfHn4wx9g82Z45hmnows/LWu05JKml5Brcnlv2XtOh6NURNPErso8Y2DdOti6NX/Zr7/C0qW2OdhnnrH31l97DRo1ci7OcPdwt4cB+Ovcv7LnhLbgo1SgaGJXZdK+fTBpEtx+OzRoAMnJ8M9/5q+/4QZ45x347Tf7jnr16o6FGjEGNR9E38Z9OZ19mhd/etHpcJSKWJrYVZny+uv2fnmdOnDzzTBhAuzaZV9Zq1Ahv1zNmjBiBCQkOBdrpBERXu73MgBvpb7F+oPrHY5IqcgU43QASp2rnBzYscO+V75tG2zaZO+Fb95sp2fNgg4dbNlNm+wT7gkJ9hW1fv3s0LZt2epa1Skp9VK4pd0tfLzqY2788kZ+vvNnysWUczospSKKJnbluKwsOHECjh+H+vVtK24A8+bBhg12eeHh/PNhzBhbbtcuaNKk6O2vW5ef2O+6C669Frp1K7s9rzntn4P+yY/bf2T53uW88OML/PWivzodklIRRZzqdSklJcWkpqY6Unc4MyZ/iIrKP8vMyIDMTMjNPXOIicm/R2yMPas1xnvZ+vWhalVbdvdue9ablQXZ2QXHxsB11+XH9emnsH+/jSEjww6nT9txnz42mQKkpsK99xZM0unp+dvZuBGaN7fTw4bB5Mnefw4XXgg//WSnMzPt++VJSdCwoZ12D82bQ7169melQsecrXO4eOLFxEbF8vOdP5NSL8XpkJQKChFZaowJ6BfesTP21avtL1733xXucWysfSLZbcAAe8bl+feHe/r22+HZZ+10aioMGXLm9tzjOXPsA1IADz4In33mfZspKfb1JrAJo1Yt79sDGDfO3qcF+6DVqFHe64+NtQnMLSXF7r+3+u+7L/8hrgULbAIrXK/bqlX2EjLAPffY+8XedOtmtwU2KTdt6r0cwIcf2p8rwBdf2Fe8vImNLZjYn38e1qwpervuxJ6dbZ829xQVBZUq2SEzM3/5xRdDxYr56zyHevXyy8XF2UvxKnxc1OQiHuj6AG8sfoOR00ey4M4FiN4LUcovHEvsmZmwZcuZy92XYd327Cn6l/bRo/nTWVnF94GdnZ0/feIEHCyijQzPBAxw7FjR28zKKjh96pT3cjk5BeczMwsmME+5uQXni7qgEhVVsGy5cpCYaJcXHtxn4O7PNWrkvVxUFFSunF82Kcn+YREbmz/ExNhxXFzBeG66yXaKEhdnL3GXK5c/bt8+v1ybNrB4ccEkXb689/vbI0bYQUWm5y5+jkmrJ7Fo1yKmbZjG0FZDnQ5JqYjg2KX4tm1TzFdf2Uvx7l/q7rHn/dI9e/ITqOcvfxH7FLM7aWVm5ifrwtsTgWrV8v9oOH7cXiL2ts2YmPzkZkzBxF54u+XK5Sc4z2TtrX7Pp6vddXurPyoKoqPz6/c8PIW3q1S4e+HHF3hi9hPUKF+D1fetpk5iHadDUiqggnEpXu+xK6Uck52bzYBPBjB762wuP+9ypg2bppfkVUQLRmLXR4qUUo6JiYphwpUTqFKuCl//+jXjlmjXrkqdK03sSilHJVVK4pmLbAP8I2eMZMqaKQ5HpFR408SulHLcyK4j+VPPPwFw57Q7+WHbD84GpFQY08SulAoJz1z0DMPbDOdk5kkGfjKQaRumOR2SUmFJE7tSKiRER0XzydWfcH/K/WTkZDDsi2Gk7tYHbJUqLU3sSqmQESVRjB08llvb30p6djq9P+zNl2u/dDospcKKJnalVEgREcZfPp7bO9xOenY61/7rWl5Z8IrTYSkVNjSxK6VCTnxMPB8M+YCXLn0JQXh01qOMWzIOp9rdUCqcaGJXSoUkEWH0haN54ZIXMBh+P/33DP50MDuOaccAShVHE7tSKqQ91vMxPr36UyrHV2bmppk0+2czHpr5ECczTzodmlIhSRO7UirkDW87nNX3rWZIyyFk5Wbx+qLXaTm2Je8te08vzytViCZ2pVRYaFC5Af8Z9h8W3bWIVjVasfvEbu7+7910Ht+Zqeumkp2bXfJGlCoDNLErpcJK1/pdWXXvKj4a+hHx0fEs37ucq6dcTa8Pe7HuwDqnw1PKcdq7m1IqbB1KO8SElRP4x4J/sPvEbgCuOO8KBrcYTItqLWhRvQVJlZKIEj2HUaEhZLptFZGBwOtANPCeMebFQuvjgYlAZ+AQcIMxZltx29TErpTyl90ndvPM3Gd4f/n7ZOVmnbG+b+O+dK7bmQ51OtC2VlsaV2lMhbgKxETFOBCtKstCIrGLSDTwK9AP2AksAYYbY9Z6lLkfaGeMuVdEhgFXGWNuKG67mtiVUv62+8RupqyZwsp9K9l4aCPzd8wvtnx8dDwV4iqQGJdItYRqeUNCTAIxUTHERsUSExVTYIiNLrisNGXcfc0L4tM02Nf+3NNVylWhfZ32AfnZqeAIRmL35c/VrsAmY8wWV1CfA0OBtR5lhgJjXNNfAGNFRIw+rqqUCqJ6FevxULeH8uaNMew8vpMVe1ewYu8Klu9dzur9q9l7ci+nMk+RkZNBRnoGh9MPs/3Ydgcj903vRr2Ze/tcp8NQIc6XxF4f8GwRYidwQVFljDHZInIMqA4c9CwkIiOAEa7ZDBH55WyCDhM1KLT/EUb3L3xF8r5BBO/fPOYhv5OI3T+XSN+/loGuwJfELl6WFT4T96UMxpjxwHgAEUkN9OUIJ+n+hbdI3r9I3jfQ/Qt3ZWH/Al2HL4+K7gQaeMwnAbuLKiMiMUBl4LA/AlRKKaWU73xJ7EuAFiLSRETigGHAtEJlpgG3uaavBWbr/XWllFIq+Eq8FO+6Zz4S+Ab7utsHxpg1IvI0kGqMmQa8D3wsIpuwZ+rDfKh7/DnEHQ50/8JbJO9fJO8b6P6FO92/c+RYAzVKKaWU8j9tjkkppZSKIJrYlVJKqQhyToldRK4TkTUikisiKYXW/UlENonIBhEZ4LF8oGvZJhF53GN5ExFZJCIbRWSy60E9RCTeNb/Jtb5xSXUEgiuGFa5hm4iscC1vLCLpHuve9vhMZxFZ7Yrxn+JqUkpEqonIt659/VZEqrqWi6vcJhFZJSKdArlPhfZvjIjs8tiPwR7rAn4sg7B/L4vIetfPdaqIVHEtj4jj56uijlmoEZEGIjJHRNa5fsf8wbU84N/TIO7jNtf3a4W4XoE6m++WiNzmKr9RRG7zWO71+xukfWvpcYxWiMhxEXkonI+fiHwgIvvFo/2VYByvouooljHmrAegNfZl+x+AFI/lycBKIB5oAmzGPngX7ZpuCsS5yiS7PjMFGOaafhu4zzV9P/C2a3oYMLm4Os5lf0qx3/8A/uKabgz8UkS5xUB37Hv+M4BBruUvAY+7ph8H/uaaHuwqJ0A3YFEw9sdV9xjgUS/LA34sg7R//YEY1/TfPH7mEXH8fPwZFHnMQm0A6gKdXNMVsc1aJwfjexrEfdwG1Ci0rFTfLaAasMU1ruqarlrc99eh791eoFE4Hz+gN9DJ8/dFMI5XUXUUN5zTGbsxZp0xZoOXVUOBz40xGcaYrcAmbNO0ec3TGmMygc+Boa6/TC7GNkcLMAG40mNbE1zTXwCXuMoXVUdAueq+HvishHJ1gUrGmAXGHpGJeN+nwvs60VgLgSqu7TgpGMcy4Iwxs4wx7g67F2LbYyhSBB0/T16PmcMxeWWM2WOMWeaaPgGsw7ZwWRR/fk+dVNrv1gDgW2PMYWPMEeBbYGAJ399guwTYbIz5rZgyIX/8jDHzOLN9lmAcr6LqKFKg7rF7a4a2fjHLqwNHPX7xupcX2JZrvbu52qK2FWi9gH3GmI0ey5qIyHIRmSsivVzL6rti8hZfbWPMHrC/wIBaHp9xYp/cRrouG33gcbknGMcy2O7A/kXsFinHryShHp9XYm/ZdAQWuRYF+nsaLAaYJSJLxTa3DaX/bhW3vKjvb7ANo+CJUKQcPwjO8SqqjiKVmNhF5DsR+cXLUNxf+kU1MVva5WezrbPm474Op+CXdA/Q0BjTEXgE+FREKp1lfH7fpwIbL37/3gKaAR2w+/SPEmLy57H0C1+On4g8CWQDk1yLwub4+UGox3cGEUkEvgQeMsYcJzjf02C50BjTCRgE/F5EehdTNhz3D9d97yHAv1yLIun4FcfR/fGlgZpLz2K7xTVD6235QeylihjXX2Ce5d3b2ikFm6v1panbUilpX131X43td979mQwgwzW9VEQ2A+e54vO83OsZ3z4RqWuM2eO6BLPftdzv++TJ12MpIu8CX/sQk7+OpV/4cPxuAy4HLnFd7gqr4+cHoR5fASISi03qk4wx/wYwxuzzWB+o72lQGGN2u8b7RWQq9rJzab9bO4G+hZb/QPHf32AaBCxzH7dIOn4uwTheRdVRpEBdip8GDBP7FHQToAX2wQCvzdO6fsnOwTZHC7Z52v94bMtbc7VF1RFIlwLrjTF5l0xEpKbYPusRkaauOLa4LpmcEJFurvtBtxaxT4X39VaxugHH3JdgAq3QveCrAPeTn8E4lgEnIgOBx4Ahxpg0j+URcfx85Evz0CHB9TN/H1hnjHnFY3kwvqcBJyIVRKSiexr7cOcvlP679Q3QX0Squi5r9we+KeH7G0wFrnBGyvHzEIzjVVQdRTPn9pTgVdi/NDKAfa4A3euexD7NuAGPpzGxTwv+6lr3pMfyptgDuQl72Sbetbyca36Ta33TkuoI1AB8BNxbaNk1wBrs05rLgCs81qVgv7ibgbHkt/RXHfge2OgaV3MtF+BNV/nVeLxpEIR9+9hV5yrXF6luMI9lEPZvE/be1grX4H46PyKOXyl+Dl6PWagNQE/spchVHsdscDC+p0Hav6au79xK1/fvybP9bmGfGdnkGn5X0vc3iPtYHjgEVPZYFrbHD/sHyh4gC5v37gzG8SqqjuIGbVJWKaWUiiDa8pxSSikVQTSxK6WUUhFEE7tSSikVQTSxK6WUUhFEE7tSSikVQTSxKxUAImJ8GLa5yn4kIjtL2GRQiO19y7gaEPLb9nwo19dVb19/1KtUWeaX/7xKqTN0LzQ/FfvO8hiPZRlBi0YpVWZoYlcqAIzt0SmPiGQABwsvP1ciEm9ss7hKKQXopXilQoaIdBSRH0UkTUQ2isi9hdbf7rpc3VtE/iUiR8nv8QwR6SMi34vICRE5JSLfiEibQtsYICLzReSYiJwUkQ0i8hcv4TQRkf+5yvwmIn8RkahC22opIlNF5KiIpIvIQlfTvSXtZ00R+VREjrs+OxGoUqofllKqSJrYlQoNlYBPgU+w/S8vAd4SkYu8lJ0EbMW2k/04gIhchm1u8iRwM3AjUBH4UUQauMo0xTbjuQ24Advr1itABS91TAVmY/t+/gr4K/ntVSMi9YCfgPbASOB64CjwPxEZVMK+/hvbGc8TrjiygTdK+IxSykd6KV6p0FARuN8YMwdAROZhO4gYju3swtMXxpg/Flr2OjDXGOPZRe0cYAswCngI6ATEAfcZ2wUq2OTtzT+MMR+6pr8TkYtdsbiXPQJUBbobYza56psOrAWeo2B/93lEpB+2HfjhxpjPXYu/EZEZFOzdSil1lvSMXanQkOZO6pDXnexGoKGXslM9Z0SkBbaP60kiEuMegDRgAeDu53sFtgOLz0XkWhGpVUw8/ys0/0uhWHoDC91J3RVzDrajjA5i+7T3pjuQg+2O1dPnXsoqpc6CJnalQsMRL8sysD3iFVa4K1h3gn4fm7g9h8uxvUPhSsIDsP/vPwb2isgiEenjpY7DJcRSzUscAHuxPVtV9bIOoC5wxBiTVWj5Pm+FlVKlp5filQo/hd8LP+Qa/wn4zkv5zLwP2qsCc0QkHrgQeBp7X7yxMeZgKWI4DNTxsryOK77Cfxi47QGqikhsoeReuxR1K6WKoYldqfC3AftA3PnGmBd9+YDrUv9sEUkE/gM0AUqT2OcCD7n+INgGICLR2IfhlhtjThTxuQVANHANBS+/DytF3UqpYmhiVyrMGWOMiPwe+I+IxAFTsEm6NtAD2G6MecX1+lxvYDqwA6iBPcvfjb2HXhqvArcD34rIU8Bx4H7g/blKnQAAALdJREFUPOCyYmL9VkR+At4RkRrY5whuANoU9RmlVOnoPXalIoAxZjo2aVcA3gO+AV7CXhpf4Cq20rX+BWAWMBb72tzFxpj0Uta3G/t0+xrgLeAL7H33y4wxM0v4+NXYPy5eACZjTzBGlqZ+pVTRxJgSm3FWSimlVJjQM3allFIqgmhiV0oppSKIJnallFIqgmhiV0oppSKIJnallFIqgmhiV0oppSKIJnallFIqgmhiV0oppSLI/wPH10gOO5LhFgAAAABJRU5ErkJggg==\n",
+ "text/plain": [
+ "<Figure size 576x288 with 1 Axes>"
+ ]
+ },
+ "metadata": {
+ "needs_background": "light"
+ },
+ "output_type": "display_data"
+ }
+ ],
+ "source": [
+ "def plot_precision_recall_vs_threshold(precisions, recalls, thresholds):\n",
+ " plt.plot(thresholds, precisions[:-1], \"b--\", label=\"Precision\", linewidth=2)\n",
+ " plt.plot(thresholds, recalls[:-1], \"g-\", label=\"Recall\", linewidth=2)\n",
+ " plt.xlabel(\"Threshold\", fontsize=16)\n",
+ " plt.legend(loc=\"upper left\", fontsize=16)\n",
+ " plt.ylim([0, 1])\n",
+ "\n",
+ "plt.figure(figsize=(8, 4))\n",
+ "plot_precision_recall_vs_threshold(precisions, recalls, thresholds)\n",
+ "plt.xlim([-100000, 100000])\n",
+ "plt.show()"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Likewise, we can plot the precision against the recall. The area under this curve indicates the performance. A good classifier will have a curve that sits in the top-right corner.\n",
+ "\n",
+ "We have defined a function to plot the curve."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 5,
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "image/png": "iVBORw0KGgoAAAANSUhEUgAAAf4AAAF8CAYAAAAuF9n2AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAgAElEQVR4nO3deZgU1bnH8d/LsAzgKMpyUUARRQVR0Iy4oEHjhksguIJ6FUUJrhFR44YLGjXifl1Q4xKNC+7B3biiCAZQISKigNsACqiAsi/n/nF6Uj1DD3TPdFd1d30/z9NPnaqurnqnYnirzjl1jjnnBAAA4qFe1AEAAIDwkPgBAIgREj8AADFC4gcAIEZI/AAAxAiJHwCAGAk18ZvZA2Y2z8w+reF7M7PbzWyGmU0xs13DjA8AgGIX9hP/Q5J6ref7QyR1THwGSbo7hJgAAIiNUBO/c26MpJ/Ws0sfSQ87b7ykZma2eTjRAQBQ/PKtjb+NpO+S1isS2wAAQBbUjzqAaizFtpRjCpvZIPnmAEnNf7PFFu21OXUDAIAYmDRp0gLnXMva/DbfEn+FpHZJ620lzUm1o3PuXkn3SpJZuRs8eKKGDct9gAAARM3Mvqntb/Otqn+0pBMTvfv3kLTIOTc36qAAACgWoT7xm9njkvaV1MLMKiRdIamBJDnnRkp6WdKhkmZIWirp5DDjAwCg2IWa+J1z/TfwvZN0ZkjhAAAQO/lW1Q8AAHKIxA8AQIyQ+DO0dq3kUr5gCABA/iPxp2nKFOnCC6WttpJatJDmzUu9308/SWvWhBsbAADpIvGvx6pV0mOPSbvtJnXtKo0YIVVU+OR+xBHSihX+6X/SJOnyy6WddpKaN/ffTZ8urV4tTZggvfJKOLUEixdLU6f68wIAkEq+DeATqaVLpTvu8E/1ixZJV1/tE70kbbaZdOyx0iefSOPGSWPHSqWl0pZbSt9+W/U4o0f7T3WnnSZtvbX05ZfSyJFSw4Z++7x5/vg//SQ1auSbE1atkpo0kSZP9uf44Qfpo4+Cz4QJ/rdlZdIvv0ht2waxVnriCalzZ39DMG2a/12XLlL9+j4WSzVOIgCgqJkrggZrs3I3fHjdRu574w3puOOk+fOrbu/USRoyRDrhBKlxY588f/ObqvtssYXUp490wAHSgAE+Eadj8GDptdekr76qfdx1se220owZ0sknS2edJa1c6WssVqzwNyItW0oNGkQTGwCgZmY2yTlXXqvfxj3xr1kjDRsmXXdd1e3bbCNde6101FFSvWoNIr/8Il1xhbRkidSvn9SzZ7DPokXSwoX+BqFTJ6ljR+nNN6WDD05dO5CuTp2kXXf1n513lr75xj/ll5RIv/7qn+w7dJDmzJFOOcWfU/K1Ep07+1h//rnmvgk1Wb3ax9y6tb/xAQBEj8Rfy8S/ZIlP7K++6hP3VVf55NakiTRwYFAVn00//yz16iU1bSrtv7//NGvmq+M7dPBJdtttfS1Ax44+hhUr/P614VzVKv21a6VRo/xxf/hBOjMxXFK9ev679fn4Y6lbt9rFAQDIHhJ/LRL/4sXSYYdJ77/vq7RHjZL22y93MRaCyv8U1q71/QBSOeccqbxc2ntv318BABC+uiT+WHbuW7lS+sMffNJv21Z66y3/dB13lTUDJSX+JmDuXGmjjXxHxRNO8N/dfnuwf8eO0pgxvhZjyy1rXysBAAhPrBL/qlU+mV19tfT2277deswYnlxrsvnmfnn88dLGG0u9e/sxDBYs8Nu//DLYp9L48b5D4MYb+30bNPDNF1Om+OaEbt24yQKAKMUm8Tsnde/uX8eTfFv+6NEk/XT9/vdBU8DHH/tOhqnssUd6xzvlFN+xsmlT6eyzff+GVauoNQCAXIvNAD7PPhskfUm65RY/MA8yt8su/iZgzRrf67+yw6CUemyAzTZbd9sDD0h//7t0113+jYVGjXyzgpnfXjn6oXMMkQwA2RSLxL9qlXT++cF6s2bSoEHRxVMs6tXz/QHMpGOO8TcBy5b5RD1vnl9fvlz68ccggT/zjP/tpptKrVqlPu6AAb5zYY8evrmgXj1/jsrPsGG+n4ZzvpPmzz/7JoYZM4LzzJ7tm3FmzMj8FUYAKGaxqOp/7DHp6699uWdP6R//YNS6XCgp8R/JvylRuS3ZEUdUfYJfs8bXGCxZIp16anBjIEkffJD6PNdc4z+Z+vOf/U3EJpv4OOhrACCOiv51PuekHXf0Q9Y++KB/mkR+e/556YsvfD+CTp38E/u0af7ti/vvX3f/xo19TUNtNGjga4Qq3Xeff4Nh5kzfQbFdu9odFwByiff415P4331X2ndfP6zuV1/lZlAehGvJEt+E4JxvtqkcNXHmTP8036CBb0qo9OijPpl37ix99lntztmli3Trrb4fQtu2/m2G6iM6AkBYeI9/PR580C8HDCDpF4umTVP3/q+p6v744/0n2axZfgbFlSv9TcSYMb6jYU0+/dTPxZBKjx6+OalZM19Tsdlmvo/CZpv5GwUAyCdF/cS/dKnvQLZkiX/nfNtto4kPhaGyU2CrVj5x//vf0lNPSTffXLfj9u7tJ0JasEA66CDffEAfEwB1UZcn/qKurHzzTZ/0y8tJ+tgwM1+N37Chr8bfYw/pppuCNwUqP2vX+kmYjj46veOOHi317eunQt5qq+AthSOP9MvOnaWHH/a1BkuXBudZvdrXSABANhV1Vf8//+mXffpEGweKi5kfy+DJJ4NtixZJZWVBu79zvk/JgQf6ZoVUnn3WL6dNk046af3n7NvX3whssYW/mR00SNpnn7r/LQDip2gTv3PSSy/5cu/e0caC4rfJJlXXzfxohDNnVt2+apXvS/DNN9KkSb5vQTqee67q+j/+kXq/k0/2N7xnnx28ssh0ygCSFW0b/xdfSNtv79trv/+eNlUUhpkzpebN/SuMrVv75oZGjaSKCqm01C9ro2lT6bjjfE1Bly7+WAAKF736U6h8kvrtb0n6KBzbbOOXzZr5ZU2vHy5c6F9VXbLE1zYMHOibFWqqCViyxI9RcN99wbYDD/RTUQ8dyhsvQJwUbee+ysRPOyiKUbNmvu/KccdJhx3ma7UeeSR1J8THH5d69Vr3GP/6l3TJJb5GoXI45LIy6dprpR9+CP9vAhCOok3848b55d57RxsHEJXKToj9+kmvvBLcEIwdK91xR+rf/PqrdOmlvpnBTHrjDW4CgGJTlIn/l1/85CwNG/r2TACBvfaSzjzT3wSsWiWtWCFNmSI99JAfkTDZgQf6m4DSUmnkyEjCBZBlRZn4p0zxyx13pO0SWJ/69f3/R3bayb9SOGeOvyG48caq+61YIZ1+uq8F6N07eGMGQOEpysT/ySd+2a1btHEAhWroUH8DsGyZdNttVb974QXp8MP9TUD79tL//q/0xBN+ToS1ayMJF0AGijLxT57sl127RhsHUOhKS6VzzvE3AZMn+2aCZN98498k6N/fT4RUUuJvCIYNk955x9cMvPKKNH9+JOEDSKEoE//06X65447RxgEUk5139h0DnfPjDRx6qN/eo8e6+15zjX9V8PDD/X6tWvkbglNOkZ5+et2BjQCEpygT/5df+mVNs7UBqJsOHfzTvHPS++8HzQIXXFB1v623rrr+4IN+joNttw1eIXz+eeYkAMJUdIn/11+luXN9h6W2baOOBoiP0lLphhuqjiUwa5ZfVr4R0Lr1ur/r29ePJbDDDtLLL4cbMxBHRTNkrzRRY8f6oUm7dZM6dap51DMA0Zo71084tCGlpdIf/yj99a/+5gCAx7S8CT16+HeRJabhBfLZ5psHtQI//SSdf37q/ZYv928VlJZKEyaEGyNQrIoq8UvSrbf6ZfW2RQD5adNNpREj/E3A4sXS7Nm+yv+ii/yERZW6d/d9Ao4/vvaTFQEowsRfqV27qCMAkKmyMt8EcMgh0nXXSQsW+FEGkz32mP//t5mfgdPMv8Hz4YfRxAwUmqJN/G3aRB0BgGy44w5fG3D33et+98UXfvnZZ34K48o3Bf7yl3BjBApJ0SZ+evQDxWXw4KBfwPffS08+Kd1yi28CqO6yy4KbgI4d/X6rV4cfM5CPiqpXf7KZM/27xgDiYdYs6e9/l4YPT2//+fOlFi1yGxOQK/TqTyGdV4UAFI8OHaSrrvI1AhUV0p13Sj171rx/y5a+RuCww6QLL2T6YcRH0T7xF8GfBSBLFi6Uxo3zDwQbmrxr2DDpxBN5JRj5rS5P/CR+ALHjnHT22dKSJcHYH6nUqyc9+6zUp09ooQFpIfGT+AHUgXPSxRf7mQZnz069z7hx0m67+RkIgajRxg8AdWAmXX+97xtQOZpg9QmH9txTql8/eFvgtdekFSuiiReoC574AaAGzkm77ip98sn69zv/fKm8XDroID8SIZBrPPFX8+qrUUcAoBiYSR9/7G8AVq6Uxo9Pvd+NN0r9+kmbbeabBIB8VpSJ/4ADoo4AQLFp0EDaffdgEKG1a6VTT5U22qjqfnvtFTQHMM0w8lFRVvUXwZ8EoMC89JJ0+OGpv1u71t8IANlCVT8AROyww/xDx+rV0oABVb+rV0965JFIwgLWUXSJv3LSDgCIQkmJ9OCD/iZgn32C7SeeGDQBmPk+AUAUii7xMz4/gHwxZoz03nupvxs1yt8A7LKL9Pnn4caFeCu6Nv4i+HMAFKEvv/SDA02a5F//S+Wnn3gdEOmhjR8A8lzHjtK++0pDh0rLlklnnLFuDeVmm/lagCZNpIEDpU8/jSRUFDme+AEgQmvW+BEB1+err6T27UMJBwWioJ74zayXmU03sxlmdlGK77c0s7fN7GMzm2Jmh4YdIwCEpaQkeBvgrbf80MA771x1n6239jUB998fTYwoLqE+8ZtZiaQvJB0oqULSBEn9nXOfJe1zr6SPnXN3m1lnSS8759qv/7g88QMoLs5Jf/qT9H//V3X7rrv6fgKIt0J64u8uaYZzbpZzbqWkJyRVn/DSSdo4Ud5E0px0Dz53blZiBIDImUm33+5vAJLb+j/6yH83cGB0saGwhZ3420j6Lmm9IrEt2ZWSTjCzCkkvSzo73YO3bl3X8AAg/+y4o28K2G67YNsDD/gbgKVLo4sLhSnsxJ9q0MrqlfP9JT3knGsr6VBJj5jZOnGa2SAzm2hmE6t/BwDFpqREmj5dWry46vamTf0NwCWXRBMXCk/Yib9CUruk9bZatyp/oKQnJck5N05SqaQW1Q/knLvXOVde2zYOAChEZWW++n/w4Krbr7uO+QCQnrAT/wRJHc1sazNrKKmfpNHV9vlW0v6SZGad5BP//FCjBIA8d/fd/gbgueeqbq8cB2DKlGjiQv4LNfE751ZLOkvSa5KmSXrSOTfVzIabWe/EbkMlnWZmkyU9LmmAK4bBBgAgB/7wB38D0Capt9SyZVLXrtQAILWiGsCnCP4UAKi1igrphBOkd9+tup1pgYtPIb3OBwDIkbZtpXfe8TUAyW851avHREAIkPgBoAjNnSs1bhysd+rkn/qpGUXRJP4LLog6AgDIL0uXSg8/XHXbkCHRxIL8URRt/LvuWu4++ojX+QGgJslt/KtX+3EBULhi38Zfryj+CgDInU8+Ccr166/bARDxQcoEgBjo2lXq0CFY33df6dxzIwsHESLxA0BMzJzpe/1Xuu026dBD6fAXNyR+AIiRnj2lL74I1l95xTeXfvttdDEhXCR+AIiZjh396H5t2wbbttrKdwBcsya6uBAOEj8AxFBpqfTdd9JNN1XdXr++dPnlVP8XMxI/AMTYeef5IX332CPYdvXVvvr/7bejiwu5Q+IHgJgzk8aNk559tur23/2OWf6KEYkfACBJ6tvXV/E//XSwrWtX6ZproosJ2UfiBwBUceSR0i23BOvDhkllZdKCBdHFhOwh8QMA1nHuuVUT/a+/Si1bSt26RRcTsoPEDwBIqXlzX/U/bFiwbfJk6ZlnoosJdUfiBwCs1/Dh/om/0lFH+Q6BFRXRxYTaI/EDADaoadN1E327dnT8K0QkfgBAWtq08VX/l18ebBs2zD/9L1kSXVzIDIkfAJCRq66Spk2rum2jjaKJBZkj8QMAMrbDDv7p/5hjgm3Jo/8hf5H4AQC1NmpUUP7wQ2mvvaKLBekh8QMA6mTt2qA8bpy06abRxYINI/EDAOrETFq6NFhfuNBvQ34i8QMA6qxxY9/m36JFsO3gg6OLBzUj8QMAsmb+/KD8+uvShRdGFwtSI/EDALIquc1/xAiptFRasya6eFAViR8AkFVm0s8/B+srVkj160cXD6oi8QMAsq5ZM//kf8QRwbbvv48uHgRI/ACAnDCTnn46WN98c+mkk6KLBx6JHwCQM2bS3/4WrD/8sDRhQnTxgMQPAMixgQOlxYuD9e7dpblzo4sn7kj8AICcKyuTHnssWN9iC+nGG6OLJ85I/ACAUPTvL911V7B+wQW+6h/hIvEDAEJz+unSO+8E6yedJN16a2ThxBKJHwAQqp49pYqKYH3IED+zH8JB4gcAhK5NG2natGB9jz38QD/IPRI/ACASO+wgvfBCsF5aKv3wQ3TxxAWJHwAQmcMPrzqoT+vWVcf6R/aR+AEAkXrooaod/EpK/BS/yA0SPwAgcn/6kzRoULB+xhnRxVLsSPwAgLxwzz1Sw4a+PHJktLEUMxI/ACBv/Oc/Qfn116OLo5iR+AEAeWO77YLywQdLU6dGF0uxIvEDAPLKAw8E5S5dpKFDo4ulGJH4AQB55eSTpcGDg/Wbb5buvz+6eIoNiR8AkHfuvlt6//1g/dRTo4ul2JD4AQB5qUcP6euvg/V+/SILpaiQ+AEAeWurraSddvLlUaOkN96INp5iQOIHAOS1SZOC8oEHSiNGRBdLMSDxAwDyWoMGVafxvfBChvStCxI/ACDvtWlTNfnXI3vVGpcOAFAQ2rSRunYN1s8+O7pYChmJHwBQMD75RCot9eU77pB+/TXaeAoRiR8AUFDmzQvKZWXRxVGoSPwAgIJSViYdd1yw3qpVdLEUIhI/AKDgPPpoUJ4/n17+mQg98ZtZLzObbmYzzOyiGvY5xsw+M7OpZvZY2DECAPLf2rVB+ZZboouj0ISa+M2sRNKdkg6R1FlSfzPrXG2fjpIultTDObejpHPDjBEAUBjMpI4dfXnoUGnZsmjjKRRhP/F3lzTDOTfLObdS0hOS+lTb5zRJdzrnfpYk59w8AQCQwnvvBeXWraOLo5CEnfjbSPouab0isS3ZdpK2M7OxZjbezHqlOpCZDTKziWY2cf78+TkKFwCQz/7nf6SmTX158WJpzz2jjacQZJz4zewkM3s10QY/q9pn5oZ+nmJb9S4Z9SV1lLSvpP6S/mZmzdb5kXP3OufKnXPlLVu2zPTPAAAUieTX+8aPlx5/PLpYCkH9THY2s2GSrpL0qaRPJK3I8HwVktolrbeVNCfFPuOdc6skfWVm0+VvBCZkeC4AQAw0aSItXx4M7HPccdI++0ht20YbV77KKPFLGijpNufckFqeb4Kkjma2taTZkvpJOq7aPs/LP+k/ZGYt5Kv+Z9XyfACAGGjUSPrhB1/1L0nt2vGKX00yrepvLumF2p7MObda0lmSXpM0TdKTzrmpZjbczHondntN0o9m9pmktyVd4Jz7sbbnBADEQ6tW0mmnBevvvhtdLPnMXAa3RGb2oqQ3nXN59cZkeXm5mzhxYtRhAAAitny51LhxsF6sT/1mNsk5V16b32b6xH+upJPN7EQza2Fm9ap/ahMEAADZUFoqvf9+sD5qVHSx5KtMn/grx0mq6UfOOZdpv4E644kfAJCsXr3gab8Yn/rr8sSfaZIerpqTPgAAeeHLL6Vtt/Xlbt38dL7wMkr8zrkrcxQHAABZs802/jNzpjR5sn/qt1QjycRQrdvkzWwjM2tnZk2zGRAAANnwxhtBuU31MWJjrDYj9x1sZhMlLZT0taRFZvZvMzsw28EBAFBb7dsH1f1z50pjxkQaTt7IKPGb2cGSXpK0kaSrJZ0h6RpJZZJeJvkDAPLJtGlBuWfP6OLIJ5k+8V8p6XVJnZ1zVznn7km0++8o6V/yw/kCAJAX6teXLr00WC/GHv6ZyjTxd5WfMndt8sbE+l2SumUrMAAAsmH48KD8m99EF0e+yDTxr5C0cQ3flSnzSXsAAMipevWk7bf35f/8J9pY8kGmif8dSVcnJtn5LzPbUr4Z4O3shAUAQPbcdptfrl4dbRz5INPE/2dJm0iabmZjzGyUmb0r6UtJzRLfAwCQV8qTxrhLfs0vjjJK/M65LyTtLOl2SY0k7SqpVNJtkro5577MeoQAANRR8+ZB+cAD4/3kn/F7/M65uc65851zuzvnOiaWFzrn5uYiQAAAsuHFF4Py7bdHF0fUmE0PABALhx0mbbKJLw8dKv36a7TxRGWDY/Wb2VuSznDOfZ4or49zzu2fndAAAMiu99+XdtrJl8vK4vlefzpP/MnTGtRLrNf0oQYBAJC3unSRhgwJ1lfE8CV0c0Vwu1NeXu4mTpwYdRgAgALgnH+3v9KqVX6Ev0JiZpOcc+Ub3nNdPKEDAGLFTNpzz2C9QYPoYolCppP09DGzk5PWtzKzcWb2i5k9bWYbZT9EAACy64MPpNatg/UlS6KLJWyZPvFfJqll0vrNktpKulfSb+VH7wMAIO/NmROUN4rRY2umiX8bSVMkycwaSzpU0nnOuaGSLpHUN7vhAQCQG2bS1VcH6zfdFF0sYco08ZdKWpYo7yX/OuDrifXpkrbIUlwAAORc8pS9558fXRxhyjTxfy1p70S5j6RJzrlFifVWkhal+hEAAPnITJo1K1gfNy66WMKS6QsM90i60cz6Suom6fSk7/aU9Fm2AgMAIAxbJ803u9de0tq1/oagWGU6Sc9tkgZIGifpFOfcfUlfl0l6MHuhAQAQjgceCMpdu0YXRxgYwAcAAPkZ/H76yZdXrszv9/sZwAcAgDqaPTso33tvdHHk2gYTv5mtMbPuifLaxHpNnxjPcAwAKGSlpVKnTr581lnRxpJL6XTuGy6pIqlc+G0DAACkcO650h//GHUUuUUbPwAACcuXS40b+/K0adIOO0QbT01Ca+M3swZm1rSG75qaWR53hQAAYP1KS4PymWdGF0cuZdq5735J99Xw3T2JDwAABeuAA/zyrbeijSNXMk38+0r6Zw3fjZa0f52iAQAgYvclPd5+9VV0ceRKpom/laR5NXw3X9L/1C0cAACi1b59UN5998jCyJlME/88STvV8N1Okn6sWzgAAETv6KP9cv58aebMaGPJtkwT/4uShpnZzskbzWwnSZdKeiFbgQEAEJUHkwagP/zw6OLIhUwT/+WSFkqaZGYfmNmTZjZW0kfyM/Ndlu0AAQAIW9Om0tln+/Lnn0tF8Ob7f2U6Sc8CSbtJuk6Syc/QZ5L+Imm3xPcAABS8K64IyueeG10c2cYAPgAA1CB5et4lS6QmTaKLJVnok/SYWQszO9zMTjKzzRLbSs2MSX8AAEVjxoygPGhQdHFkU6Yj95mZjZAfu3+0pAcktU98/U/5Dn4AABSFbbaRdtvNlx99NNpYsiXTJ/SLJZ0lP1nP7vLt+5VekFRkfR8BAHH39NNB+ZtvoosjWzJN/KdKGu6cu1a+J3+yGZK2yUpUAADkiS23DMrJr/kVqkwTfxtJ42v4bqWklBP4AABQyDbZxC+vuiraOLIh08Q/W1KXGr7rKqkIRzUGAMTdc88F5Zdeii6ObMg08T8l6XIz65G0zZnZdpKGSnoia5EBAJAn9tsvKB95ZHRxZEOmif9KSZ9LGiPpy8S2pyT9J7F+fdYiAwAgj9x/v1+uWBFtHHWV6ch9y+Sn5h0g6QNJb0iaIGmQpAOdcyuzHB8AAHnh2GOD8rJl0cVRV/XT3dHMGkg6VNIU59wjkh7JWVQAAOSZpknd1++5p3CH8U37id85t0rSkwoG7AEAIFZKS/1yyJBo46iLTNv4Z0lqlYtAAADIdyNHBuVFi6KLoy4yTfw3SLrUzFrmIhgAAPJZcjt/s2bRxVEXabfxJ/xO0maSvjKz8ZLmSkqe3s85507KVnAAAOST0lLpkkuka6/168uWSY0bRxtTpjKaltfMvlbVRF+dc851qGtQmWJaXgBAmCqn6333Xem3v43i/LWfljfTJ/5ySb8655bX5mQAABSDHj2ksWOlnj2lDJ6f88IG2/jNrMTMrjSzhZJ+kLTYzJ4xs1q1bphZLzObbmYzzOyi9ex3lJk5M6vVHQ0AALmyzz5BedWq6OKojXQ69w2WdLn8bHw3SvqnpD6Sbsn0ZGZWIulOSYdI6iypv5l1TrFfmaRzJH2Y6TkAAMi1yy4Lyg89FFkYtZJO4j9N0n3Oud855/7snDta0pmSTjCzhhmer7ukGc65WYlR/p6Qv4mo7mr5NwhoUgAA5J2mTaXdd/flK66INpZMpZP4O8iPx59slKQSSVtleL42kr5LWq9IbPsvM9tFUjvn3IvrO5CZDTKziWY2cf78+RmGAQBA3Rx1lF/OnRttHJlKJ/FvJGlxtW2/JJZlGZ7PUmz7b7cIM6sn34QwdEMHcs7d65wrd86Vt2zJsAIAgHCdeGLUEdROur3625hZ8mt6JUnbFybv6JybtZ7jVEhql7TeVtKcpPUySV0kvWP+XYnWkkabWW/nHO/rAQDyRvIz5+zZUps2Ne+bT9JN/E/XsP35FNtKUmyrNEFSRzPbWtJsSf0kHVf5pXNukaQWletm9o6k80n6AIB8Y0l12LfdJt1wQ3SxZCKdxH9ytk7mnFttZmdJek3+BuEB59xUMxsuaaJzbnS2zgUAQFhGjCicxJ/RyH35ipH7AABReOIJqX9/X16wQGrePJzz1mXkvkwn6QEAAAn9+gXl/fePLo5MkPgBAKiD00/3y8mTC2MUPxI/AAB1cOedQfmRR6KLI10kfgAA6sBMatTIlwcOjDaWdJD4AQCoo5tvjjqC9JH4AQCoowEDgvKUKZGFkRYSPwAAddSkSVDu2jW6ONJB4gcAIAuSB/DJ57njSPwAAGTB+ecH5XyewIfEDwBAFphJJycGuX/11WhjWR8SPwAAWXLllUE5X6v7SfwAAGTJllsG5VGjootjfUj8AABk0d57++XIkfQC1CMAAA4BSURBVNHGURMSPwAAWbT99n45dWq0cdSExA8AQBZddllQfuWV6OKoCYkfAIAsat8+KN99d2Rh1IjEDwBAllW+0z95crRxpELiBwAgyyrH7v/2W2nOnEhDWQeJHwCALOvUKSi3aRNdHKmQ+AEAyLJ69aTbbgvWx46NLpbqSPwAAOTAOecE5UMOiS6O6kj8AADkyDPP+OUvv0irVkUbSyUSPwAAOdK3b1D+29+iiyMZiR8AgBwxk7p18+Uzzog2lkokfgAAcui886KOoCoSPwAAOXTMMUH5iy+ii6MSiR8AgBxq1Ehq2tSXL7442lgkEj8AADnXr59fPvustHZttLGQ+AEAyLHkwXwmTIguDonEDwBAzjVtKpWU+PLzz0cbC4kfAIAQ7LefX15/fbRxkPgBAAjB8OFRR+CR+AEACEH37kF52bLo4iDxAwAQgso2fkmaNy+6OEj8AACEZIcd/HLOnOhiIPEDABCSBQv88qyzoouBxA8AQEguvdQvP/pIci6aGEj8AACEZPDgoPzYY9HEQOIHACAkpaXSTjv58jXXRBMDiR8AgBAdcYRfzp4dzflJ/AAAhOjYY/3yl1/8J2wkfgAAQrTDDpKZLz/3XPjnJ/EDABAiM+moo3z5pJPCPz+JHwCAkFVW90vS66+He24SPwAAITvySKllS18++OBwz03iBwAgAjffHJTDHMyHxA8AQAROOCEoP/NMeOcl8QMAELGxY8M7F4kfAICI9O/vl7feGt45SfwAAERkwICgvHBhOOck8QMAEJGDDgrKm24azjlJ/AAARCh5xr4lS3J/PhI/AAARuvvuoPz557k/H4kfAICIVVbzk/gBAIiBrbbyy4suyv25SPwAAESsVy+/rKjI/blCT/xm1svMppvZDDNb597GzM4zs8/MbIqZvWlmW4UdIwAAYTrxxKCc6+F7Q038ZlYi6U5Jh0jqLKm/mXWuttvHksqdcztLelrSDWHGCABA2HbYISgvW5bbc4X9xN9d0gzn3Czn3EpJT0jqk7yDc+5t59zSxOp4SW1DjhEAgFCZBbP15bq6P+zE30bSd0nrFYltNRko6ZWcRgQAQB6YP98vR47M7XnCTvyWYlvK1gwzO0FSuaQRNXw/yMwmmtnE+ZVXCwCAAtW9u1/ecktuzxN24q+Q1C5pva2kOdV3MrMDJF0qqbdzbkWqAznn7nXOlTvnyltW1o8AAFCg/vrXoLx8ee7OE3binyCpo5ltbWYNJfWTNDp5BzPbRdI98kl/XsjxAQAQib33DsoTJuTuPKEmfufcaklnSXpN0jRJTzrnpprZcDPrndhthKSNJD1lZp+Y2egaDgcAQNGoXz9I/kcdlcPz5O7QqTnnXpb0crVtlyeVDwg7JgAA8sFuu0nvvy/Ny2F9NyP3AQCQJypn6mvYMHfnIPEDAJAn2iW6v69cKa1enZtzkPgBAMgTpaVBedq03JyDxA8AQJ4wk5o29eXHH8/NOUj8AADkkc6JGWw++CA3xyfxAwCQR04/3S/ffTc3xyfxAwCQR3r0CMq//JL945P4AQDII9ttF5QHDsz+8Un8AADkmSZN/HLq1Owfm8QPAECeuesuv/zss+wfm8QPAECeOewwvzSTXMrJ62uPxA8AQJ5p0UIqKfFJf+HC7B6bxA8AQB5as8Yvs93Bj8QPAEAeqqzuf+657B6XxA8AQB66/vqgvHZt9o5L4gcAIA8lv8//xz9m77gkfgAA8lDDhtLee/vyW29l77gkfgAA8tTgwX45a1b2jkniBwAgT3Xrlv1jkvgBAMhT224blBcvzs4xSfwAAOSpRo2C8lNPZeeYJH4AAPJYmzZ+OW5cdo5H4gcAII/9/vd+ef/92TkeiR8AgDx29NHZPR6JHwCAPLbHHkF5yJC6H4/EDwBAHmvSJCjfemvdj0fiBwAgz734YlB+/vm6HYvEDwBAnjvsMKl5c1/u27duxyLxAwBQAN54IzvHIfEDAFAAunWTTj+97sch8QMAUCBuvrnuxyDxAwBQIEpLpWXL6nYMEj8AAAWktLRuvyfxAwAQIyR+AABihMQPAECMkPgBAIgREj8AADFC4gcAIEZI/AAAxAiJHwCAGCHxAwAQIyR+AABihMQPAECMkPgBAIgREj8AADFC4gcAIEZI/AAAxAiJHwCAGCHxAwAQIyR+AABihMQPAECMkPgBAIgREj8AADFC4gcAIEZI/AAAxEjoid/MepnZdDObYWYXpfi+kZmNSnz/oZm1DztGAACKVaiJ38xKJN0p6RBJnSX1N7PO1XYbKOln59y2km6R9NcwYwQAoJiF/cTfXdIM59ws59xKSU9I6lNtnz6S/p4oPy1pfzOzEGMEAKBohZ3420j6Lmm9IrEt5T7OudWSFklqHkp0AAAUufohny/Vk7urxT4ys0GSBiVWV5jZp3WMDevXQtKCqIOIAa5z7nGNc49rnHvb1/aHYSf+CkntktbbSppTwz4VZlZf0iaSfqp+IOfcvZLulSQzm+icK89JxJDENQ4L1zn3uMa5xzXOPTObWNvfhl3VP0FSRzPb2swaSuonaXS1fUZLOilRPkrSW865dZ74AQBA5kJ94nfOrTazsyS9JqlE0gPOualmNlzSROfcaEn3S3rEzGbIP+n3CzNGAACKWdhV/XLOvSzp5WrbLk8qL5d0dIaHvTcLoWH9uMbh4DrnHtc497jGuVfra2zUogMAEB8M2QsAQIwUVOJnuN/cS+Man2dmn5nZFDN708y2iiLOQraha5y031Fm5syM3tG1kM51NrNjEv89TzWzx8KOsdCl8e/Flmb2tpl9nPg349Ao4ixkZvaAmc2r6ZV1825P/G8wxcx23eBBnXMF8ZHvDDhTUgdJDSVNltS52j5nSBqZKPeTNCrquAvpk+Y13k9Sk0T5dK5x9q9xYr8ySWMkjZdUHnXchfZJ87/ljpI+lrRpYr1V1HEX0ifNa3yvpNMT5c6Svo467kL7SPqtpF0lfVrD94dKekV+DJw9JH24oWMW0hM/w/3m3gavsXPubefc0sTqePmxGJC+dP47lqSrJd0gaXmYwRWRdK7zaZLudM79LEnOuXkhx1jo0rnGTtLGifImWnfcFmyAc26MUoxlk6SPpIedN15SMzPbfH3HLKTEz3C/uZfONU42UP5OE+nb4DU2s10ktXPOvRhmYEUmnf+Wt5O0nZmNNbPxZtYrtOiKQzrX+EpJJ5hZhfzbXGeHE1qsZPrvdviv89VB1ob7RY3Svn5mdoKkckk9cxpR8VnvNTazevKzUg4IK6Ailc5/y/Xlq/v3la+5es/MujjnFuY4tmKRzjXuL+kh59xNZran/BgtXZxza3MfXmxknPcK6Yk/k+F+tb7hflGjdK6xzOwASZdK6u2cWxFSbMViQ9e4TFIXSe+Y2dfybXaj6eCXsXT/vfinc26Vc+4rSdPlbwSQnnSu8UBJT0qSc26cpFL5cfyRPWn9u52skBI/w/3m3gavcaIa+h75pE+baObWe42dc4uccy2cc+2dc+3l+1H0ds7VelzumErn34vn5TurysxayFf9zwo1ysKWzjX+VtL+kmRmneQT//xQoyx+oyWdmOjdv4ekRc65uev7QcFU9TuG+825NK/xCEkbSXoq0W/yW+dc78iCLjBpXmPUUZrX+TVJB5nZZ5LWSLrAOfdjdFEXljSv8VBJ95nZEPnq5wE8jGXGzB6Xb45qkegrcYWkBpLknBsp33fiUEkzJC2VdPIGj8n/BgAAxEchVfUDAIA6IvEDABAjJH4AAGKExA8AQIyQ+AEAiBESPxADZjYgMdNf5Welmc00s2vNrDTi2L42s4eS1itjbR9ZUEARK5j3+AFkxdHyI32VSeor6eJEmTHUgZgg8QPx8olzbkai/C8z6yhpoJn9ifHTgXigqh+It48kNVbS+OmJIVgfNbP5ZrbCzD4xs77Vf2hmXc3sOTP70cyWmdl0M7s46fuDzOxlM5trZkvN7FMzG2pmJeH8aQBS4YkfiLf28tNX/yhJZtZO0oeS5kkaIj+u+rGSnjGzP1QOKWxm3SW9Iz9M6BD55oOOknZOOnYHSW9K+j9Jy+Vnc7xSUktJF+X0rwJQIxI/EC8liZkrK9v4j5R0rnNuTeL7K+Wn+eyZNG79a4kbguEKJmG5Uf5mYQ/n3NLEtreST5QYR1ySZH5ih/ckNZR0vpldQtMCEA0SPxAvn1dbv8s5d0fSei/5ST8WJW4QKr0maYSZbSxptaQekkYkJf11mNnm8jcSvSRtoar/3rSS9H1t/wgAtUfiB+Klr3y1fEtJ50k6w8w+dM49nPi+laQTE59UmktaKd8/qKKmk5hZPfnagS3kk//nkpZJ+oOkS+WnZwUQARI/EC+fVvbqN7O3JE2Rf5J/xjm3RL76/j1Jf63h93Pkp2BdK6nNes6zjXyb/v865/5RudHMfl/3PwFAXdCrH4gp59wKSRfIP+Wfkdj8qnwHvanOuYkpPisS1fvvSzrBzBrXcPgmieWqyg1m1kDS8Tn5YwCkjSd+IMacc6PNbIJ8h7s7JF0u6d+SxiTWv5a0qaQukjo4505J/PR8Se9KGmdmN8lX+3eQ1M05d7akaZK+kfQXM1sjfwMwJLy/DEBNeOIHcJn8U/9g59y38lX0kyVdK+lfku6W1FNJvfadcxPkO/h9J/+63svytQcVie9Xyrfnfy/pYUl3Shoj6fpQ/iIANTLnXNQxAACAkPDEDwBAjJD4AQCIERI/AAAxQuIHACBGSPwAAMQIiR8AgBgh8QMAECMkfgAAYoTEDwBAjPw/pgUhopn/96sAAAAASUVORK5CYII=\n",
+ "text/plain": [
+ "<Figure size 576x432 with 1 Axes>"
+ ]
+ },
+ "metadata": {
+ "needs_background": "light"
+ },
+ "output_type": "display_data"
+ }
+ ],
+ "source": [
+ "def plot_precision_vs_recall(precisions, recalls):\n",
+ " plt.plot(recalls, precisions, \"b-\", linewidth=2)\n",
+ " plt.xlabel(\"Recall\", fontsize=16)\n",
+ " plt.ylabel(\"Precision\", fontsize=16)\n",
+ " plt.axis([0, 1, 0, 1])\n",
+ "\n",
+ "plt.figure(figsize=(8, 6))\n",
+ "plot_precision_vs_recall(precisions, recalls)\n",
+ "plt.show()"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Finally we will plot the ROC curve. \n",
+ "\n",
+ "The ROC curve plots the *true positive rate* (another name for recall) against the *false positive rate (FPR)* \n",
+ "\n",
+ "Once again there is a trade-off: the higher the recall (TPR), the more false positive \n",
+ "(FPR) the classifier produces for the set of thresholds.\n",
+ "\n",
+ "We will define a function that plots the ROC curve."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 7,
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "image/png": "iVBORw0KGgoAAAANSUhEUgAAAf4AAAF8CAYAAAAuF9n2AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAgAElEQVR4nOzdd3iUVeL28e+ZSU8IhN6kIwoqoSxFBLuLKE2QYtm1UQQRVuxtdVdRsfCDFRUsr+7aBSki4NoVBKQFVFaR3msIAdJnzvvHTDAiCROY5JnM3J9ruZiWmRsXcuec5zznMdZaREREJDK4nA4gIiIi5UfFLyIiEkFU/CIiIhFExS8iIhJBVPwiIiIRRMUvIiISQcq1+I0xrxlj9hhjfizmeWOMmWSMWWeMWW2MaVue+URERMJdeY/4Xwe6l/D85UBz/6+hwIvlkElERCRilGvxW2u/AdJLeElv4N/WZzFQxRhTp3zSiYiIhL9QO8ZfD9ha5P42/2MiIiISBFFOBziGOc5jx91T2BgzFN/hABITE9udccYZZZlLRESCyFrwWovHa3/3Td63i7zvkaLP/ba7vO+xAo8lz+P9XWnYY25YIDffg8tlyMn3YC3ke71EuXxj3nyPN+h/rmAzQLTbhTHgLcgja98ObEEewD5rbY2Tec9QK/5twGlF7tcHdhzvhdbaqcBUgPbt29tly5aVfToRkTLi9VoO5xVgvb5C9P0Ca30Fl5ld4HudtUdLs/DXroO5RLuNv0hh24Es4qLdgO/rLb8VLUVuFz6eV+Bl/d7D1Kkcd/Rzj/2ctK0ZNKyaSIHXy66DOew7nEeNSrH+9/C91mLxevndYzkFHjbvz6J6UiwerxeP15KZUxCU/2aBFFi0//eEUryvy4DLGIwBYwx5BV6qJ8WQFBvFpv1ZnFY1nnPqVcFiqRQbTY1KsRR4LfVS4n/7Wn7/Hl6vJSHWTUpCDNWTYjHG9zlgcPlf4zJg8H1NXLSblIRooty+H1KWL19Ot27dqJ5SmTfeeIMePXpsLtV/rCJCrfhnA7cZY94FOgIHrbU7Hc4kIhWQtZZ8jyUjK488j5f0I3nke7wUeCz7DucdLcGjBVqkcNftOUxKYgx7MnPZeziXdbsPUz8lHk9hKXp9r1uyMZ3mNZOKvI+/XP2/53ss2zOyqRQXdfQxW+S1hY8VeCvGxdJ+3J75u/vbM7ID/tp9h3P/8FhctIucfC8NqyXg9peky5ijhekyhp93ZdK5abWjj/9WyoadB7OpUSmWVnWTiY1yHy1Qc0z5HsjK4/RalbBY6qckULNSLNFuF1EuQ2yUm8RY9+9KOpRYazHG0Lp1a0aMGMEdd9xBnTqntvTNlOfV+Ywx7wAXANWB3cDf8f9AZq19yfj+iz+Pb+V/FnCjtfaEQ3mN+EWcZf3l5fH/ysrzsO1AFtl5Hjz+6VyvtWw/kE1stBuv1/f6tbsPUSUhBo/Xy+ptB6lbOf535Yjvf0dHn4W3s3ILWLv7MA2qJlDg9bJq20GqJsSwKzMH8I2kKkiX/kFyXBQu128lVzgS3J2ZS9Maib6Cc/1Wfi7jG7FnZOXTqm4yUW7f127cd4S2DVJ8ZcZvhWiOjjD9j/HbqDbP46FB1YTfvXdhDozhSG4Bjaol4Ha5KPB4qZoYQ0yU6+j7FJYnFB3t+u6nJMQQ5TJEuVy4XJAQE4XbFVolG2qWLFnCHXfcwYwZM6hZs+bvnjPGLLfWtj+Z9y3XEb+1dvAJnrfAyHKKIxIWCku3wGM5mJ3/23HTY6aDt6RnAeDxcrSId2RkExPlwloo8Fq8Xsv/dmay42A2NSrF4fX+Vtr7j+SxIyObWslxRz/jpx2ZRLsN+R5nWrboiLOw9OG30o9yGQq8lpSEaOKj3ew5lEubBlWIcrlYv/cw7RqmEOV24TL4R5y+kvNYS2Z2PmfWSSYn30PVxFiqJvqmdAtHnm7Xb0VaJSG6SFEXlrW/MIHkuGjc7t/KsWgpmiJTvSpCAfB6vYwfP56HHnqIevXqsWvXrj8U/6kItal+kYiSlVdAZnYBGdl5bN6fRYHHUuD1ciingP2H88jzePD4p6DX7znM5z/voXpSLNFuw86DOSf+gDJw7OcWLf2YKN/0qdsYDuUWYAyc65+mdftHsVvSs2jXIAW32/e6XZk5nFOvMlFuFxlZeTSpkfi7kairSEGaIiPgAo8lPsZNlfhootwGt8tF9aQYEmOiSI6PPlqmIhXJrl27uP766/nss8+4+uqrmTp1KlWqVAnqZ6j4RY5h/aPZAq/lSG4BWXmeo9PY+R4v+w/ncTi3gB0Z2ezIyCYxNoq1uw8RH+NbTLV0UzqNqiUeHSl7vZDr8bJqawaNqiVQ4LVsOxD4sdFjHe9YaZTLV6y5BV5SEqJJiIk6Oh1cOLrMzfeS7/FyTv3KvyvijfuO0LahbxTs9r9PZnY+9VPiaVgt8ejrXMY3U1AlIYb4GDdu/3vERbuoXimW5Ljo46QVkdK49957WbhwIVOnTuWWW24pkx9ey/UYf1nRMX4pLWstGVn5pG3LYN3uw8REuUjbmsHyzQeOTomXp/op8Ww74Fuo1KlJNaJdhoRYNzWS4vyjWd/oODvfQ9MaSaQ2qEK025AcF02s/xiriFRMeXl5ZGRkULNmTfbt28eePXto2bJliV9TYY7xi5Slg1n57MrM4fOfd3Mop8A33ZyTz97DuazaepD4GDdRLsOGvUfIC+D83Zgo37HfoquOXS7D3kO51KkcR0yUi6qJMVRLjKVJjUQOHPGtHI6NdpHvsdROjjs66i48dlspLopE/6KmSnFRJMVGqbRFItj69esZPHgwbrebhQsXUr16dapXr16mn6nilwrD+heY7TqYw7YDWXy/8QCHcvJZuG4fO07heHdKQjRn1knmzDrJZOV5uKxVLS44vYYKWUTK1Ntvv83w4cNxu9288soruFzls5muil9CQoHHy67MHH7dfZhfdh9i7a5DxMW4eXvJFs6oXYmfdx0K+L0aVksg/Uge/dvVp1ZyHEmxUdSpHIcxUCs5jhi3i7hoN/WqxOPSKmoRKWdHjhzhtttu4/XXX6dLly689dZbNGzYsNw+X8UvZeZIbgHr9hzmpx2Z7PCfdrViywHqVI4np8DDis0HqFkpllXbDpb4Pscr/cbVE3EZqJYUS4+zanNWvcqcWSeZxFj9lRaR0GatZcmSJTz00EM8/PDDREWV7/ctfZeUU3IwK59fdh9i24Esth/I5oPl20q1OO54p6Q1qpZA69Oq0Lh6InUrxxPlNrQ+rQrVEmOoHB+tKXgRqXCstbz++usMHDiQpKQkVqxYQVxcnCNZVPwSEI/XsnpbBl+v3cviDftZvKGkqyv/0SVn1qJ+Sjw1KsWSV+BbLOcy/lPBkmKpXTmO+iml2U1bRKRi2LdvHzfddBMfffQRWVlZjBw50rHSBxW/lGBPZg7/mLOGOatPfLmERtUSqJcST5PqSZxTvzKXnFmLKgkanYtIZPvqq6+49tpr2bdvHxMnTmTEiBFOR1Lxi28KavP+LN5ZuoWdGTms33uYfYdz2Z35x41iqifFUis5lsvPqs0FLWrSqm6yyl1E5DheffVVhgwZQvPmzZkzZw5t2rRxOhKg4o84B47k8fnPe/jvT7vIzMkPaMr+qrb16N+2Pp2bVlPJi4gE6Pzzz2fo0KE888wzJCUlOR3nKBV/mLPWsml/FhM/W8vMtB0nfH2DqgkM6dqYPzWuSuX4aGonx6nsRUQCNGPGDD7++GNefvllmjVrxksvveR0pD9Q8YcZay3/XbObd7/fwpe/7C32dXUqx3H+6TXo0qw6DaslcFbdyjqnXUTkJGVnZzN27FhefPFF2rdvz8GDB4N+cZ1gUfGHgbwCL/N+3MmET9eyaX/xp9K1rl+ZF69rR90q8eWYTkQkvK1Zs4ZBgwbxww8/MHbsWMaNG0dMTIzTsYql4q9gCjxeZqXt4Ku1e/n2171kZOUf93XtGqZQOzmOQR1Oo0vT6hrNi4iUgfz8fK644gqOHDnC3Llzufzyy52OdEIq/gpgT2YOry3cxMc/7GBrevGXc60UF8U1HRsw6qLmJGkHOxGRMpOZmUliYiLR0dG8/fbbNGrUiDp16jgdKyBqhxC2ZMN+/r1oMx//8Mfz6C8/qzYt6yTT7fQaNK6RqGuhi4iUk8WLFzN48GBuvvlmHnzwQTp37ux0pFJR8YeYj1btYMJna9mw98gfnut2eg0ua1mLazs20Ep7EZFy5vV6GT9+PA8++CCnnXYal1xyidORToqKPwT8uvsQD8z8ke83/vGc+kqxUdzYpRFDz2+q6XsREYfs3LmT66+/ns8//5wBAwYwZcqUkF21fyJqEgftPZTLyLdW8P2m3xe+22V4+5aOtGmQQkxU+VyfWUREird582aWLl3Kyy+/zM0331yhZ11V/A44klvARc9+9bstcd0uw/WdGjLiwqbUrOTcxRtERMQnLy+PefPm0bt3bzp16sTmzZsr7Ci/KBV/Odp/OJc7P1j1h4117u7eghEXNHMolYiIHGvdunUMHjyYZcuWsXr1as4+++ywKH1Q8Ze5wp30hv1n+R+eu6ZjA/7RqxVRbk3ni4iEirfeeovhw4cTHR3Nhx9+yNlnn+10pKBS8ZehI7kFtP3np+QWeH/3+J8apfD6jR1I1GI9EZGQMnLkSF544QXOO+883nrrLRo0aOB0pKBT85SR95du5e7pq4/eP//0Gvylc0MuPrOWg6lERKQkbdu25eGHH+ahhx4iKio8KzI8/1QOe/Gr9Tw1/+ej9+/pfga3XtDUwUQiInI81lomTZpEtWrVuO6667j55pudjlTmdHA5yG59c/nvSv+TMd1U+iIiIWjfvn306tWLMWPGMHfuXKfjlBuN+IPEWsttb69k3o+7ALiwRQ2m/qU90Vq4JyIScr766iuuvfZa9u3bx6RJk7jtttucjlRuVPxBkFvgocWD84/eb1GrEv/vxg4OJhIRkeL8/PPPXHzxxTRv3pyPP/6Y1NRUpyOVKw1HT9HCdft+V/oXnVGTT/7WzcFEIiJyPNnZvqubnnHGGbzxxhssW7Ys4kofVPwnLSffw/lPf8m1ryw5+liHxlV57YY/OZhKRESO58MPP6Rx48YsX+7bU+W6664jKSnJ4VTOUPGfBK/X0u6fn7J5fxYA8dFu/jW4De8Pq1iXZhQRCXfZ2dmMGDGCfv36cdppp4XN7nunQsf4S8nrtfR8fgFH8jwA3H5xc+649HSHU4mIyLF++uknBg0axI8//sidd97J448/TkxMjNOxHKfiLwWP19LiwXkUeC0Aj/c9i2s7NnQ4lYiIHM8HH3zA7t27mTdvHt27d3c6TsjQVH8pDH558dHS/2fvVip9EZEQk5GRwcqVKwF48MEH+eGHH1T6x9CIP0CvLtjI9xvTAbjv8jO4vnMjZwOJiMjvLFq0iMGDB+P1elm3bh0xMTHUqqVt0o+lEX8A9h7K5Z9z1gAwrFsThp2vnfhEREKF1+vliSeeoGvXrhhj+OCDD3QsvwQa8Z/AzoPZdH7iCwDcLsM93c9wOJGIiBQ6dOgQV111FZ999hkDBw5kypQpVK5c2elYIU3FfwIDpiw6env2bV1wuYyDaUREpKikpCSqVq3KK6+8wk033YQx+h59IprqL8Gz//2Frem+nZ7G9T2bVnX1U6SIiNPy8vK4//772bRpE8YY3nvvPW6++WaVfoBU/MX4YdtB/vXFOgBu7NKIazo2cDiRiIisW7eOc889lyeeeILZs2c7HadC0lR/MYa/ufzo7YevbOlgEhERAXjzzTe59dZbiY6OZsaMGfTp08fpSBWSRvzHMWf1DrZn+Kb437ipg6aPREQc9uqrr3L99dfTpk0bVq1apdI/BRrxHyPf42X8/F8AaNugCuefXsPhRCIikcvj8eB2uxk4cCCHDh3itttuIypK1XUqNOI/xqy0HWxJzyLabfh/N3ZwOo6ISESy1jJx4kQ6dOhAVlYWSUlJjBkzRqUfBCr+Iqy13PnBKgBGX9ycyvHRDicSEYk8e/fupWfPnowZM4Z69eqRm5vrdKSwouIvYuh/flvQ17/daQ4mERGJTF9++SWtW7fm008/ZdKkScyaNYuUlBSnY4UVzZn4bdx3hE/X7Abgnu5nULtynMOJREQii7WWe++9l+TkZObOnUtqaqrTkcKSit/v8Y//B0CVhGhuvUB78YuIlJfNmzeTnJxMSkoK06dPJyUlhcTERKdjhS1N9eO7CM9n//ON9h/t1crhNCIikWP69OmkpqYyevRoAOrXr6/SL2MqfmDK1+uP3u7Vuq6DSUREIkN2djbDhw+nf//+NGvWjL///e9OR4oYKn5gZtoOAO76cwtt1iMiUsZ+/fVXOnTowJQpU7jzzjtZuHAhTZvqEGt5ifhj/N9vTGffYd+pIjd2aeRsGBGRCJCUlATAvHnz6N69u8NpIk/Ej/jHzfUt6qtRKZaEmIj/OUhEpExkZGTw2GOP4fF4qFOnDqtWrVLpOySii/9QTj5pWzMAeKrf2Q6nEREJT4sWLSI1NZVHH32U77//HgCXK6Lrx1Hl/l/eGNPdGPOLMWadMebe4zzfwBjzpTFmpTFmtTGmR1llefa/awGoWzmOC1vULKuPERGJSB6Ph3HjxtG1a1dcLhcLFiygc+fOTseKeOVa/MYYNzAZuBxoCQw2xhx7zdsHgfettW2AQcALZZVnycZ0AHqcXUeL+kREgmzIkCE88MADXH311axcuZKOHTs6HUko/8V9HYB11toNAMaYd4HewJoir7FAsv92ZWBHWQT5385M/rczE4Ch3ZqUxUeIiEQkay3GGIYMGcJ5553HjTfeqMFVCCnv4q8HbC1yfxtw7I+AjwD/NcaMAhKBS8oiyMerdwLQuHoiNZO1Pa+IyKnKzc3lvvvuA+C5556jc+fOmtoPQeV9jP94P/LZY+4PBl631tYHegD/Mcb8IacxZqgxZpkxZtnevXtLHaRwp74hXTXaFxE5Vb/++ivnnnsuEyZMID8/H2uP/dYuoaK8i38bUPSyd/X541T+zcD7ANbaRUAcUP3YN7LWTrXWtrfWtq9Ro0apQqRtzeDnXYeIi3bRp4126hMRORX/+c9/aNu2LRs3bmTGjBn861//0tR+CCvv4l8KNDfGNDbGxOBbvDf7mNdsAS4GMMacia/4Sz+kL8H05dsAOLdpdZ27LyJyCrZt28awYcNo06YNq1atok+fPk5HkhMo19az1hYYY24DPgHcwGvW2p+MMf8AlllrZwNjgZeNMX/DdxjgBhvkOaNp/uLv2bpOMN9WRCRibNy4kcaNG1O/fn2++eYbUlNTiYrSQKoiKPfz+K21c621p1trm1prH/c/9rC/9LHWrrHWdrHWtrbWplpr/xvMzz+SW0CB1wugc/dFRErJWsuECRNo0aIF77zzDgDt27dX6VcgEff/1Peb0sn3WJrVTKJKQozTcUREKoy9e/dyww03MHfuXHr27Mlll13mdCQ5CRG3Z+K3a/cBcG7Tag4nERGpOL766itat27NZ599xqRJk5g1axbVqun7aEUUcSP+r9buAeDCMzTNLyISqPT0dJKTk5k7dy6pqalOx5FTEFEj/twCDxv2HgGgdf0qDqcREQltmzdv5v333wfgqquuYvXq1Sr9MBBRxf/lz76zAmOjXFRN1PF9EZHiTJ8+ndTUVEaOHMmhQ4cAiInR981wEFHF/8JX6wDok1rP4SQiIqEpOzub4cOH079/f04//XSWLFlCpUqVnI4lQRRRx/j3ZOYC0KRGosNJRERCT25uLp06dWL16tXcfffd/POf/9QoPwxFTPF7vZZdmTkAnNf8DzsAi4hEvNjYWP7yl79w9tln61S9MBYxU/3bM7KP3j6jdnIJrxQRiRwHDhxg4MCBfP755wCMHTtWpR/mIqb4l25KB6B6Ugxuly4eISLy3XffkZqayocffsi6deucjiPlJGKKf+G6/QC0aZDicBIREWd5PB7GjRtHt27diIqKYuHChQwbNszpWFJOIqb41+89DEDnJtppSkQi24cffsgDDzzA1VdfzYoVK+jQoYPTkaQcRcTivpx8D2lbMwDo2bquw2lERJyxd+9eatSoQf/+/Zk3bx5//vOfMUaHPiNNRIz4P1i2FYDG1ROpUSnW4TQiIuUrNzeXMWPG0KJFC7Zs2YIxhu7du6v0I1REjPi/+sW3Y1/LOlrNLyKRZe3atQwaNIiVK1dy2223UbOmrlMS6SKi+LekZwFwWataDicRESk///nPf7j11luJjY1l5syZ9O7d2+lIEgLCvvhzCzz8use3sK9r8xoOpxERKT+fffYZ7dq148033+S0005zOo6EiLAv/p92ZB69rQvziEi4W758OXFxcbRq1YqXXnqJ6OhooqLC/lu9lELYL+7bdsC3Y1+zmkkOJxERKTter5fnnnuOzp07M3bsWADi4+NV+vIHYf83Yu7qnQA0q6HiF5HwtGfPHm688Ubmzp1L7969efXVV52OJCEs7It/wz7f8f3k+LD/o4pIBPr555+56KKLSE9P5/nnn2fEiBE6TU9KVKo2NMa0ALoC1YDXrbW7jTGnAfuttVllEfBUrd3tK/4/t6rtcBIRkeBr0qQJF154IXfffTetW7d2Oo5UAAEd4zfGRBtj/gOsAaYC44B6/qefBx4sm3jB07xmJacjiIgExaZNmxg4cCAHDhwgJiaGt956S6UvAQt0cd8/gV7AEKAhUHQeaS7w5yDnCooDR/KO3q6XEu9gEhGR4Jg2bRqpqanMnz+fH3/80ek4UgEFWvzXAg9Za18Ddhzz3AagcVBTBUnh8f0WtSrpUrwiUqFlZWUxbNgwrr76alq0aMHKlSvp2rWr07GkAgq0+GsAJf1oGReELEG3aL3vUrwtamuaX0QqtjvuuIOpU6dyzz33sGDBApo0aeJ0JKmgAl3ctxn4E/DFcZ5rD/watERBlJXnASBKo30RqYCstRw5coSkpCT+/ve/069fPy699FKnY0kFF2jxvwk8YIxZB3zkf8waYzoDd+Bb7BdyvvnVd3GeTk2rOZxERKR0Dhw4wJAhQ0hPT+fTTz+lTp061KlTx+lYEgYCnep/Avgc+ADY53/sS2AB8DXwf8GPdurW7zkCQJX4aIeTiIgEbuHChaSmpjJr1iwuv/xynZcvQRXQiN9aWwD0NcZcim8Ff01gPzDfWvtJGeY7JQVeLwBNtGufiFQAHo+HJ554gkceeYSGDRuycOFCOnTo4HQsCTMBFb8xpia+TXo+BT495jkXUN1au6cM8p2SfI8FoEqCRvwiEvoOHz7MK6+8woABA3jppZdITk52OpKEoUCP8e8EOgPfH+e5Nv7H3cEKFQy5BZ6jt1MSdFU+EQldX3zxBV26dKFy5cp8//331KhRQ9P7UmYCPcZf0t/AKMAbhCxBVXhVPkDn8ItISMrNzWXMmDFcfPHFTJw4EYCaNWuq9KVMFTviN8YkAUXnmaobY+oe87J44BpgdxlkOyXr9/g272nXMMXhJCIif7R27VoGDRrEypUrGTVqFLfffrvTkSRClDTVPxZ42H/b8ttpfMcywOPBDBUMv/qLv+iUv4hIKJg1axbXXnstsbGxzJo1i169ejkdSSJIScU/B9iFr9hfAMYDG495TS6wxlp7vGP/jtp2wHexwOpJsQ4nERH5vWbNmtG1a1defvll6tev73QciTDFFr+1djmwHMAYY4Hp1tp9xb0+1CTG+P5oDaomOJxERASWL1/OjBkzeOyxx2jVqhXz5s1zOpJEqIAW91lrp1Sk0gdI25oBaJ9+EXGW1+vlueeeo3Pnzrzxxhvs3bvX6UgS4QI9nQ9jzOnAjUAL/nhRHmutvSKYwU5VUpzvjxbtCvTEBRGR4NqzZw833HAD8+bNo0+fPrz66qtUrVrV6VgS4QLdwKcd8C2+1fsNgF+Aqvh28NsBbCmrgCdr8QbflflSG1RxOImIRCKPx8OFF17I+vXrmTx5MrfeeqtO05OQEOiI/0ngY2AwkAdcZ61dYYzpAbwC3FNG+U6Kx2vJyfdtLdCwmo7xi0j5yc/Px+1243a7efbZZ6lbty7nnHOO07FEjgp0Hrw18Dq/bdTjBrDWzsV3Zb7xQU92Cg5m5x+9HRsVUhsKikgY27RpE926dWPSpEkAdO/eXaUvISfQ4o8FDllrvUA6UKvIc2uAkPqbvWm/76p8zWrq4jwiUj4++OADUlNTWbNmDXXrHrvXmUjoCLT4NwCFf5N/Am4o8tx1QEhdoCf9cB4AGVn5J3iliMipycrKYujQoQwYMIAzzjiDtLQ0BgwY4HQskWIFWvzzgEv9t58Aehtj0o0xe4C/ApPKItzJysr37dbXRgv7RKSMrVixgtdee4177rmHb7/9lsaNGzsdSaREAS3us9beX+T2fGNMV6A/kADMt9bOLqN8J+Vglm/EXy1RV+UTkeCz1rJ06VI6dOjAeeedx9q1a2nSpInTsUQCclInuVtrF1tr77TWjgi10gf4YftBAKqq+EUkyNLT0+nXrx+dOnVi+fLlACp9qVBOeXcbY0xLY8w7wQgTLHsP5QKQVxByVwsWkQpswYIFpKam8tFHH/H000/Tpk0bpyOJlFqJxW98zjHGXGmMOfOY5842xnwArAauLMuQpZUQ6zuCUbvysRsMioicnKeeeorzzz+f6OhovvvuO8aOHYtLO4NKBVTs31pjTG1gIbASmAX8aIx5wxgTZYx53v/4lfiu3NesPMIG6kf/VH/DaokOJxGRcBEbG8ugQYNYuXIlf/rTn5yOI3LSSlrc9ySQCjwOrAAaA3cDXwOdgfeAu6y128o6ZGntzMgBID5am/eIyMmbM2cOXq+XXr16MXr0aABtuysVXknFfynwD2vtk4UPGGN+BD4BXrLWjijrcCcrz+M7tl8/Jd7hJCJSEeXm5nLPPfcwceJELrjgAnr27KnCl7BR0gGqmvim+osqvB9Si/mKstYevV0rWcf4RaR01q5dS+fOnZk4cSK333478+bNU+lLWClpxO8Gco95rPD+kbKJc+qy/Zv3AMTHaKpfRAK3adMm2rZtS1xcHLNnz6Znz55ORxIJuhNt4Px+KxcAACAASURBVHOZMabowj0XYIHuxpgzir7QWvt2sMOdjPQjvs179AO6iATK6/Xicrlo1KgRf//737nmmmuoV6+e07FEysSJiv8fxTz+2DH3LRASxb8707ewr0p8tMNJRKQiWLZsGTfddBPvvPMOrVq14q677nI6kkiZKqn4zyzhuZCVk+9b2GdP8DoRiWxer5cJEyZw3333Ubt2bQ4fPux0JJFyUWzxW2t/KYsPNMZ0BybiW0PwStGzBoq8ZgDwCL7+XmWtvSbQ99+RkQ1A2wYpwYgrImFoz549/PWvf2X+/Pn06dOHV199lapVqzodS6RcBHSRnmAxxriByfhOFdwGLDXGzLbWrinymubAfUAXa+0BY0zN0nxG4TH+/YePXZcoIuLz/PPP8+WXXzJ58mRuvfVWrdqXiFLe+012ANZZazdYa/OAd4Hex7xmCDDZWnsAwFq7pzQfULg/f/2UhFNPKyJhIz8/nw0bNgDwwAMPsGLFCkaMGKHSl4hT3sVfD9ha5P42/2NFnQ6cboxZaIxZ7D808AfGmKHGmGXGmGV79+49+ni+f/OeBtVU/CLis3HjRrp168ZFF11EVlYWsbGxtGzZ0ulYIo4o7+I/3o/Wx67DiwKaAxcAg4FXjDFV/vBF1k611ra31ravUaPG0ccP5/rO409J0Kp+EYH333+f1NRU1qxZw/jx40lI0KBAIlt5F/824LQi9+sDO47zmlnW2nxr7UbgF3w/CAQkMycfgEpxKn6RSJaTk8OQIUMYOHAgLVu2JC0tjQEDBjgdS8RxpS5+Y0wzY0xHY8zJ/Ni8FGhujGlsjIkBBgGzj3nNTOBC/2dVxzf1vyHQDyg8jz8xtlzXLYpIiImOjmbTpk3cd999fPPNNzRu3NjpSCIhIeDiN8bcbIzZhm8E/h1whv/xacaY4YG8h7W2ALgN34V+/ge8b639yRjzD2NML//LPgH2G2PWAF/iuwLg/kBzFp7OF+3Sgh2RSGOt5eWXX2bHjh243W7mzZvHuHHjiI7WDKBIoYCK3xhzAzAV+AL4K78/Vr8EGBjoB1pr51prT7fWNrXWPu5/7GFr7Wz/bWutvcNa29Jae7a19t1A3xtgu7/4qyTElObLRKSCS09Pp1+/fgwdOpQXX3wRgKgozfyJHCvQEf9dwERr7V/445X5/od/9B8KKvu36q2aqOIXiRQLFiwgNTWVOXPm8Mwzz/Doo486HUkkZAX643BT4ONinjsEhMw2ebszfRv3VEtS8YtEgg8//JCrr76axo0b891339G+fXunI4mEtEBH/On8fjV+UacDO4MT59RYa3H7j+1X1kV6RCLChRdeyO23386KFStU+iIBCLT4PwYeNMYULX/rP79+DDAr6MlOQr7H4vFaolyGaHd5n6koIuXlo48+onv37uTl5ZGSksKECRNITk52OpZIhRBoOz7gf+0aYA6+TXee8d+PBkLigFpWXgEAXqtr84mEo9zcXEaPHk2vXr3YvXs3+/btczqSSIUTUPH798tvC0wCagDbgarAG0DHwn31nXYop7D4HQ4iIkH3yy+/0KlTJyZNmsTo0aNZvHgxdevWdTqWSIUT8Lku1toMfCP/B8ouzqnZ578iX+3kOIeTiEgwWWu54YYb2Lp1Kx999BFXXnml05FEKqyAit8YMw74t7X25zLOc0oKz+GPidLxfZFwkJmZicvlIikpiddff52kpCTq1Tv2ul4iUhqBNuQo4Cf/1fBGGWNqnPArHLD9gK/446PdDicRkVO1dOlS2rZty6hRowBo0aKFSl8kCAIt/prAX4C9wHPAdmPMHGPM1caY2DJLV0qFK/lrVAqZSCJSSl6vl2eeeYZzzz2X/Px8brnlFqcjiYSVQBf3ZVtr37LWXo7vinr3AnWA94DdxpiXyzBjwHILvAC0qqfTekQqoj179tCjRw/uuusuevXqRVpaGl26dHE6lkhYKfXBcGvtbmvtc9badsDF+HbuuynoyU5Ctv90vtgoTfWLVERZWVmsXr2aF198kWnTppGSEjKbgoqEjVJfwcI/td8HuA64DN8Fe4rbzrdcZWTnA5AUq+IXqSjy8/N56623+Otf/0qjRo1Yv3498fHxTscSCVuluSzvBcaYV4Hd+C7UUwu4E6hrre1V4heXk1VbMwColqhj/CIVwcaNG+natSs33ngjn3/+OYBKX6SMBXo63xagHrAVmIzv1L5fyjLYyaiW5Ct8j3bwEQl57733HkOHDsUYw3vvvccll1zidCSRiBDoVP+n+Mr+67IMc6pWb/ON+OunaMQgEsruv/9+nnjiCTp16sQ777xDo0aNnI4kEjECKn5r7c1lHSQYqifFsu9wHtHawEckpBWO7h999FGio3UlTZHyVGzxG2M6AD9aa7P8t0tkrf0+qMlOQp7/dL6UBH0jEQkl1lpeeOEFDhw4wIMPPshFF13ERRdd5HQskYhU0oh/MdAJ+N5/u7gD58b/nONL6bPzPQDEaec+kZCRnp7OzTffzMyZM7niiivweDy43fo3KuKUkor/cuB//ts9KL74Q8aRXN95/EmxpT5LUUTKwIIFC7jmmmvYtWsXzz77LGPGjMHl0qE4EScV25DW2k+K3J5fPnFOTZ7HN9Wvi/SIOG/v3r1cdtll1K1bl++++4727ds7HUlECPA8fmPMGmPM2cU819IYsya4sU5OTr6/+N0qfhGnHDx4EIAaNWrw4YcfsmLFCpW+SAgJtCHPAIo7Ry4BaBGcOCevwD/adxmIUvGLOGL27Nk0bdqU6dOnA9C9e3eSk3XtDJFQUpqGLO4Y/znAwSBkOSVHcn0L+7R3j0j5y8nJ4fbbb6d37940aNCAs88+7gShiISAkk7nGwWM8t+1wDRjTO4xL4sH6gLTyiZe4ApX9NfUJXlFytXPP//MoEGDWLVqFWPGjOHJJ58kNlb/DkVCVUnL33cAy/23mwG/APuPeU0usAZ4MfjRSiczx3eBHq/VkF+kPC1dupTt27czZ84crrjiCqfjiMgJlLSqfzowHcAYA/CAtXZDOeUqtcLNe7LyPA4nEQl/mZmZLFu2jIsuuojrr7+eK6+8UpfQFakgAt2yd3BZBzlVB7LyADizjhYSiZSlpUuXMmjQIPbu3cvmzZtJSUlR6YtUICUd478b34V5dvlvl8Raa58ObrTSKRzp7zqY42QMkbDl9Xp59tlnuf/++6lbty7z5s1T4YtUQCWN+J8EvgJ2+W+XxAKOFv/BLN8x/pZ1NeIXCbaCggJ69uzJ/Pnzueqqq3jllVdU+iIVVEnFH2+tLVzFH/LXuS2c6ncZh4OIhKGoqCjatGlD7969GTZsWOG6HxGpgEpa3Jd7vNuhyu1vfF2gRyQ48vPzeeihh+jduzedO3dm3LhxTkcSkSAIaHGfMaYJkGytTfPfjwXuBc4CPrHWvlJ2EQNTeIy/fkrIT06IhLwNGzYwePBgvv/+e2JjY+ncubPTkUQkSAK9jN0L+M7XT/Pf/yfwN2At0NcY47bWTimDfAHLzPYd468UF+1kDJEK77333mPo0KEYY3j//fe5+uqrnY4kIkEU6Ja9qcA3AMZ3cO8G4H5rbSt8C/+Gl0m6UjiU47skb7KKX+SkzZkzh0GDBtGqVSvS0tJU+iJhKNDirwLs899OBaoB7/vvfwo0DXKuUjuc6yv+SnGBTmKISKGcHN9psJdffjlTp07l66+/plGjRs6GEpEyEWjx7wGa+G9fCmy01m72308EHN8uL7fAFyE2SlfmEwmUtZbJkydz+umns3PnTtxuN0OGDCE6WjNnIuEq0OHxHOBxY8zpwFDgtSLPtQI2BjtYaW3P8I1YYlT8IgFJT0/n5ptvZubMmfTo0YOoKM2WiUSCQP+l3wtUAgYCnwGPFXluAPBFkHOV2hH/VL+InNi3337LNddcw+7du3nuuecYPXo0Lpd+aBaJBIHu1Z8JXF/Mc38KaqKTlBzv+6NUjtcUpciJTJ48mbi4OBYtWkS7du2cjiMi5ahUc3vGmEpAB6Aqvkv0LrXWHiqLYKW1dvdhABJiNF0pcjzbtm0jPz+fxo0bM2XKFFwuF5UqVXI6loiUs4Dn9owxDwI7gf8C7+Gb8t9pjHmgjLKVSqzb90eJdmsrUZFjzZo1i9atW3PTTTcBULlyZZW+SIQKqPiNMSOBfwAzgB5AG+By//1/GGNuLbOEASpc1Jek0/lEjsrJyWHUqFH06dOHhg0bMmWKo/tsiUgICLQlbwNesNbeVuSxVcAnxpiDwCjgxWCHK439R3wX6YnXXv0iAGzdupWePXuyatUqxowZw5NPPklsbKzTsUTEYYEWfxPg9mKemwXcEpw4p04X6RHxqVatGikpKcyZM4crrrjC6TgiEiICPcafDrQo5rkW/udDQrRbpyRJ5MrMzOSuu+7i8OHDJCQk8MUXX6j0ReR3Am3Jmfg28LnaFLkQtzGmL74L9swsi3CBstb3e4xKXyLY999/T5s2bZgwYQJffvklAEX+uYqIAIEX/73Az/hW82cZYzYbY7KAacAv/ucdY/E1f5RW9EsE8nq9PP3003Tp0oWCggK++eYbevbs6XQsEQlRgW7gc9AYcy7QF+iK7zz+dOBrYJa11tG9+gtH/FEuFb9Envvuu4/x48fTr18/Xn75ZVJSUpyOJCIhLOBz3/zlPs3/K6QcLX5N9UsE8Xg8uN1ubr31Vpo1a8Ytt9yiqX0ROaESm9IYM8gYs9gYs88Ys84Y87gxJuROlC+c6k/3n9InEs7y8vK455576Nu3L9ZaGjVqxJAhQ1T6IhKQYovfGHM18DZQG1gIZOE7lv9YcV/jlMIR/2lV450NIlLGNmzYQNeuXRk/fjx169YlPz/f6UgiUsGUNOK/A/gYaG6t7W2tPQd4ChhljAmpOXV/7xOtq4tJGHv33Xdp06YNa9euZdq0abz00kvExMQ4HUtEKpiSmrIF8KK1tuiQYhIQDzQs01SlZK1W9Ut4O3ToEHfccQdnnXUWaWlp9OvXz+lIIlJBlXS8vgqw75jH9vp/TwE2lkmik/Dbqn6N+CW8/PzzzzRr1oxKlSrx9ddf07hxY6KiQm6ZjYhUICdqSlvKxx3h8Tf//iO5DicRCQ5rLc8//zypqak8/fTTADRv3lylLyKn7ETFv9AYk1f4C8j2P76k6OPGmIAb1xjT3Rjzi/8sgWI3/jHG9DfGWGNM+xO+p//3pFh9U5SKb//+/fTt25dRo0Zx8cUXc8stIXMpDBEJAyU15VPB/jBjjBuYDFwKbAOWGmNmW2vXHPO6SvguCrQkkPctnH6oXTkuiGlFyt+iRYsYMGAAu3fv5rnnnmPMmDE6TU9EgqrY4rfW3lcGn9cBWGet3QBgjHkX6A2sOeZ1/wTGA3cG8qY6xi/hIiYmhsqVKzNz5kzatWvndBwRCUPl3ZT1gK1F7m/zP3aUMaYNcJq1dk5Jb2SMGWqMWWaMWXbgYCYA0VrVLxXQtm3b+Ne//gVAu3btWL16tUpfRMpMeRf/8Zr56EJB//4AE4CxJ3oja+1Ua217a2375EqVANi470iwcoqUi1mzZtG6dWvuv/9+tm/fDoBLM1ciUobK+zvMNuC0IvfrAzuK3K8EnAV8ZYzZBHQCZp94gZ/vZ4dWdSsHL6lIGcrJyWHUqFH06dOHRo0asWLFCurVq3fiLxQROUXlvQx+KdDcGNMY2A4MAq4pfNJaexCoXnjfGPMVcKe1dllJb1p4jD82SiMlCX3WWi655BIWLlzImDFjePLJJ4mNjXU6lohEiHJtSmttAXAb8AnwP+B9a+1Pxph/GGN6nfT7+n/X1fkklFlrsdZijGHUqFHMmTOHCRMmqPRFpFyV+4nv1tq5wNxjHnu4mNdeEMh75hZ4iQZitLhPQtTBgwcZNmwYF198MUOGDGHgwIFORxKRCBXwENkYU8sYM84Ys8AYs8YY09L/+IhANtkpS1EuX+Fvz8g+wStFyt+SJUto06YN06ZNIzMz0+k4IhLhAip+Y8wZwA/Arfguz9sCKNwtpwUwpkzSBahwqv/0WpWcjCHyO16vl6eeeorzzjsPr9fLt99+y9ixJzxhRUSkTAU64n8G30V5GgM9+P1peQuBzkHOVSqFV+eL1jF+CSGLFy/m3nvvpW/fvqSlpdG5s6P/TEREgMCP8Z8PXGetzfBvu1vULqBOcGOVTuGIP0ar+iUEbNmyhQYNGnDuuefy3Xff0alTJ227KyIhozRN6Snm8Wr8dvEeR+Tm+6K59M1VHJSXl8fdd99Ns2bNWLbMdwZq586dVfoiElICHfEvA64HjreNbj9gcdASnYTCKf6MrDwnY0gE27BhA4MGDWLp0qUMHz6cVq1aOR1JROS4Ai3+x4H5xpiPgLfwza53M8YMAwYAF5ZRvoAUTvXXr5rgZAyJUO+++y5Dhw7F7XYzbdo0+vXr53QkEZFiBVT81trPjDEDgP8DrvA//By+7XYHWGsXllG+gBTu3Bft0pSqlL9169Zx9tln8/bbb9OwYUOn44iIlMgUrogP6MW+g5WtgJrAfuAHa623jLIFrFbTljb+6qd55urW9G9X3+k4EgFWrVrFgQMHuOCCC/B4PFhriYoq9/2wRCRCGWOWW2tPag+dUn2nsr6fEn48mQ8qSzn5XuLRZXml7FlrmTx5MmPHjqVly5asWLECt/vYE11EREJXQMXvn+YvkbX2/VOPc3IKt+o9klvciQcip27//v3cdNNNzJ49mx49evD6669rxb6IVDiBjvjfLebxoscJHCv+whC1knWxEykbO3bsoEOHDuzZs4cJEyYwevRolb6IVEiBFv+Zx3msGnAl0B/4a9ASnYTCZQq6Op+UlTp16jBw4ECuvfZa2rZt63QcEZGTFuiq/l+Keeo7Y4wH3x7+i4KWqpSOrurXMX4Joq1btzJ8+HAmTpxIs2bNePbZZ52OJCJyyoIxRP4S6BWE9zlp2f6d+7RXvwTLzJkzad26Nd988w2//FLcz70iIhVPMJqyPb4r9jmmcI9+rzfwUxNFjicnJ4eRI0fSt29fmjRpwooVK7jiiitO/IUiIhVEoKv67z7OwzHAWUBf4OVghjpZSXE6j1pOzfjx43nhhRe44447eOKJJ4iJiXE6kohIUAXalE8e5zEPsB2YADwatEQnoXAToiiXpvql9Ky1pKenU61aNe688066dOnCxRdf7HQsEZEyEWjxxx/nsfxQ2LWvKLe27JVSOnjwIMOGDWPlypWsWLGCxMRElb6IhLUTDpGNMTHAI8BZ1trcIr9CpvSPns6n4pdSWLJkCW3atGHatGnccMMNxMXFOR1JRKTMnbD4rbV5wGggsezjnJx8j+9nkCidzicB8Hq9PPXUU5x33nl4vV6+/fZb7rvvPm29KyIRIdCD4quAlmUZ5FQUruV3aSc1CYDH42HWrFn07duXtLQ0Onfu7HQkEZFyE+gx/ruBfxtj1llrPyvLQCej8Nh+4Wl9Isfz6aef0qZNG6pXr878+fOpVKmStt0VkYgTaFO+BlQBPjHGHDLG/GqMWVvkl7M7nOgYv5QgLy+Pu+66i8suu4zHHnsMgOTkZJW+iESkQEf8y/n9BXlCytGpfhW/HGP9+vUMHjyYpUuXMnz4cJ544gmnI4mIOCrQvfoHlXWQYHBrBCdFfPHFF/Tp0we32820adPo16+f05FERBxX7FS/MWaDMaZ1eYY5WYUb+Og8finqrLPO4tJLLyUtLU2lLyLiV9Ix/kZAhbjAfeFUv4pf0tLSuPHGGykoKKBmzZpMnz6dhg0bOh1LRCRkhNUyeE31Ry5rLZMmTaJjx47897//ZdOmTU5HEhEJSScq/pBd0Hc8WtwXmfbt20fv3r0ZPXo0l112GatWraJZs2ZOxxIRCUknWtz3qDFmXwDvY621fw1GIJHS6t+/P4sWLWLixImMGjVKp+mJiJTgRMWfCuQG8D6OzwwkxGi71UhSUFCAx+MhNjaWCRMmANCmTRuHU4mIhL4TFX8fa+335ZLkFOn4fuTYunUr11xzDeeccw6TJ09W4YuIlELYLO5z6wI9EWHmzJm0bt2atLQ0zj33XKfjiIhUOGFT/NquN7xlZ2czcuRI+vbtS5MmTVi5ciXXXnut07FERCqcsCl+ncMf3rZv386///1vxo4dy3fffadV+yIiJ6nYY/zW2gr1Q4GO8Ycfay2ff/45F198Mc2aNWPdunXUqlXL6VgiIhVahSr3kuw4mON0BAmigwcPMnjwYC699FLmzJkDoNIXEQmCQK/OF/Ka1kh0OoIEyeLFixk8eDBbt25l3LhxXHHFFU5HEhEJG2Ez4o9yhc0fJaJNnjyZrl27Yq3l22+/5b777sOl/29FRIImbL6janFfeGjYsCFXXXUVaWlpdO7c2ek4IiJhJ2yKP0rn8VdY8+fPZ/LkyQBceeWVvPfee1SpUsXhVCIi4Slsil8j/oonLy+PO++8k8svv5xXX32V/Px8pyOJiIS9sCn+g1kqjYpk3bp1dOnShWeffZYRI0awcOFCoqOjnY4lIhL2wmZVv1QcGRkZdOjQAWstH374IX379nU6kohIxAib4q9bJd7pCHICBQUFREVFUaVKFSZNmkS3bt1o0KCB07FERCJK2Ez1u3SMP6SlpaVx9tlnM3/+fACuu+46lb6IiAPCpvi1qD80WWuZNGkSHTt2JDMzk4SEBKcjiYhEtLApfpf26g85+/bto3fv3owePZo///nPrFq1im7dujkdS0QkooVP8WuqP+TMnj2bTz75hIkTJzJr1iyqV6/udCQRkYgXNov7dHW+0FBQUMBPP/1E69atufHGG+nWrZsuoSsiEkLCZsSvDXyct2XLFi688EK6du3Knj17MMao9EVEQkzYFL+m+p01Y8YMUlNTWbVqFS+++CI1a9Z0OpKIiBxH2BT/xn2HnY4QkbxeLyNGjOCqq66iadOmrFy5kmuvvdbpWCIiUoywKf6WdZKdjhCRXC4XBQUFjB07loULF9K0aVOnI4mISAnCZnFflDtsfoYJedZaXnnlFdq3b0+bNm2YMmUKRosrRUQqhHJvS2NMd2PML8aYdcaYe4/z/B3GmDXGmNXGmM+NMQ0Ded9oHeMvFxkZGQwcOJChQ4cyZcoUAJW+iEgFUq7Fb4xxA5OBy4GWwGBjTMtjXrYSaG+tPQeYBowP5L3dLo34y9qiRYtITU1lxowZPPnkk7zwwgtORxIRkVIq76n+DsA6a+0GAGPMu0BvYE3hC6y1XxZ5/WLgukDeOEp79papr776iksuuYTTTjuNb7/9lk6dOjkdSURETkJ5D5PrAVuL3N/mf6w4NwPzAnnjHRnZpxBLimOtBaBLly488MADrFy5UqUvIlKBlXfxH29Ybo/7QmOuA9oDTxfz/FBjzDJjzDKAZjWTghZSfObNm0e7du3Yv38/0dHRPProo1SpUsXpWCIicgrKu/i3AacVuV8f2HHsi4wxlwAPAL2stbnHeyNr7VRrbXtrbXuAKC3uC5q8vDzGjh1Ljx49KCgoICMjw+lIIiISJOVd/EuB5saYxsaYGGAQMLvoC4wxbYAp+Ep/T6BvrMV9wbFu3Tq6dOnCc889x4gRI1iyZInOzRcRCSPlurjPWltgjLkN+ARwA69Za38yxvwDWGatnY1vaj8J+MB/mtgWa22vE723RvzBcf/997N+/Xo+/PBD+vbt63QcEREJsnLfwMdaOxeYe8xjDxe5fcnJvK8u0nPyDh8+zOHDh6lduzbPP/88OTk5NGjQwOlYIiJSBsJmfjzf43U6QoW0cuVK2rVrx6BBg7DWUrNmTZW+iEgYU/FHKGstEydOpFOnThw5coRHH31UO/CJiESAsNmrv1pSrNMRKoz09HRuuOEGPvroI3r27Mlrr71G9erVnY4lIiLlIGxG/DrEH7ioqCjWr1/PpEmTmDVrlkpfRCSChM2IX9PUJSsoKGDy5MkMGzaM5ORk0tLSiI6OdjqWiIiUszAa8av4i7N582bOP/98xowZw7Rp0wBU+iIiESqMit/pBKFp+vTppKam8sMPP/D2229z3XUBXfNIRETCVNgUvwb8f/T000/Tv39/mjdvzsqVKxk8eLDTkURExGFhc4xfU/1/1KtXLw4cOMAjjzxCTEyM03FERCQEhNGIX8VvrWXq1KncdNNNWGtp0aIF48aNU+mLiMhRYVP8kX6MPyMjg4EDBzJs2DC2bt1Kdna205FERCQEhVHxR27zL1q0iNTUVGbMmMGTTz7JJ598QkJCgtOxREQkBIXRMX6nEzgjOzubvn37kpCQwIIFC+jYsaPTkUREJISFTfFH2jH+vXv3Uq1aNeLj45k9ezYtWrSgcuXKTscSEZEQFzZT/ZFk3rx5tGrVimeffRaADh06qPRFRCQgYVP8OfkepyOUuby8PMaOHUuPHj2oU6cOV155pdORRESkggmbqf7kuPDegnbdunUMGjSI5cuXM3LkSJ555hni4uKcjiUiIhVM2BQ/YX6If+fOnWzZsoUZM2bQp08fp+OIiEgFFTZT/eHY+4cPH+b9998HoGvXrmzcuFGlLyIipyRsij/czuNfsWIFbdu25ZprrmHDhg0AJCYmOpxKREQqurAp/nDpfWst//d//0enTp3Iysri888/p0mTJk7HEhGRMBE2x/hNGEz2W2sZMGAA06ZNo1evXrz22mtUq1bN6VgiIhJGwqf4K37vY4zhsssu4/zzz2fkyJERtymRiIiUvfApfqcDnKSCggIeeeQRWrVqxeDBgxkyZIjTkUREJIyFzTH+itj8mzdv5vzzz+fxxx9n8eLFTscREZEIEDYj/oq2qn/69OnccssteDwe3n77bQYPHux0JBERiQBhM+KvSLW/fPly+vfvT/PmzVm5cqVKX0REyk34FH8FGPEfOnQIgHbt2vHBBx+wYMECmjZt6nAqERGJJGFU/E4nKJ61lqlTp9KwYUNWrVoFQP/+/YmJxlbJagAAEIFJREFUiXE4mYiIRJrwKX6nAxQjIyODAQMGMGzYMNq3b0+tWrWcjiQiIhEsfIo/BJt/0aJFpKamMnPmTJ566inmz59P7dq1nY4lIiIRLGxW9YfiMf7Zs2fjcrlYsGABHTt2dDqOiIgIxlrrdIZTFlunuV285HvaNEhxOgo7duxg586dtGvXjvz8fLKysqhcubLTsUREJIwYY5Zba9ufzNeG0VS/8yP+jz/+mNatW3PNNdfg8XiIjo5W6YuISEgJn+J38LNzc3P529/+xpVXXkndunWZNWsWbrfbwUQiIiLHF0bH+J353P3793PZZZexYsUKRo0axfjx44mLi3MmjIiIyAmE0YjfmeZPSUnhzDPPZObMmUyaNEmlLyIiIS1sir88HTp0iBEjRrB161ZcLhdvvvkmvXv3djqWiIjICan4S2n58uW0bduWKVOm8OWXXzodR0REpFTCpvhjosr2j2KtZcKECXTu3Pn/t3f30VJV5x3Hvz9BxSAgylssUWKRWGKJUrSYUojVZZBWqUarJOILgjUJrhBdaXFhlSi1Fd9Tk4g2iaIFlRgLxaBWQ3xJxUQDUjAxCxWjorEi4hsvAk//2PvqMN47d+5lZi537u+z1lkzc84+5zyzZ9Y8c/bZZx82bNjA4sWLOf3006u6TzMzs0qrm8Rf7c5911xzDeeffz5jxozh6aefZuTIkdXdoZmZWRXUTa/+XaqU+Ddv3sxuu+3GpEmT6NmzJ2edddZOMWaAmZlZa9TNEX+lr+TfsmUL06ZNY/jw4WzcuJHu3bszYcIEJ30zM2vX6ibxVzIfv/jii4waNYrLL7+cQw89lG3btlVu42ZmZm2obpr6K5X37777biZOnMjWrVuZM2cO48aNq9CWzczM2l79JP4KHPJv2bKFGTNmMGjQIObOncsBBxxQgcjMzMx2HnWT+Hekc9/KlSvp378/PXr04N5776V3797suuuulQvOzMxsJ1E/5/hb0dgfEcyaNYthw4YxdepUAPbdd18nfTMzq1v1k/hbmPfXrVvHySefzLnnnsvIkSOZPn16VeIyMzPbmdRN4m+JpUuXcsghhzB//nxmzpzJokWL6Nu3b1uHZWZmVnX1c46/BSf5e/fuTb9+/Zg3bx6HH354FaMyMzPbudTNEX9zaX/NmjVMmzaNbdu20b9/f5YsWeKkb2ZmHU79JP4SmX/hwoUMGTKE6667jhUrVuTyHoHPzMw6nvpJ/I0c82/atIkpU6Zw3HHH0b9/f5566imGDBnSBtGZmZntHOrmHH9jB/CnnHIK8+fP57zzzmPmzJl06dKl9oGZmZntRBQRbR3DDtv9kwfGS7/7X/p0S4l927Zt7LLLLjz22GOsXbuWsWPHtnGEZmZmlSPpqYgY1pp1a97UL2m0pGclrZI0tZHlu0u6My9/QtKAsraLeOeddxg/fjwXXnghACNGjHDSNzMzK1DTxC+pE/Bd4FhgMDBO0uCiYmcD6yJiIHAtcEU5216+7NcMHTqUOXPm0LVr10qGbWZmVjdqfY7/cGBVRDwPIOkOYCzwTEGZscD0/PzHwA2SFCXOSWx97y2OPWoUffv2ZfHixYwcObI60ZuZmbVztW7q/yPgpYLXL+d5jZaJiC3AemCfUhvd+s5ajv7iaJYtW+akb2ZmVkKtj/gbu3i++Ei+nDJIOgc4J7/ctGjhf63o1avXDoZnJfQC3mjrIDoA13P1uY6rz3VcfZ9p7Yq1TvwvA58qeN0fWNNEmZcldQZ6AG8WbygibgJuApD0ZGt7N1p5XMe14XquPtdx9bmOq0/Sk61dt9ZN/b8CDpT0aUm7AacCC4rKLADOyM9PAn5W6vy+mZmZla+mR/wRsUXSZOB+oBPww4hYKelS4MmIWAD8ALhN0irSkf6ptYzRzMysntV85L6I+Cnw06J5Fxc83wic3MLN3lSB0Kw013FtuJ6rz3Vcfa7j6mt1HdfFyH1mZmZWnrq5SY+ZmZk1r10l/moN92sfKaOOz5f0jKTlkh6StH9bxNmeNVfHBeVOkhSS3Du6FcqpZ0l/l7/PKyXNqXWM7V0Zvxf7SVosaWn+zRjTFnG2Z5J+KOl1SSuaWC5J38mfwXJJQ5vdaES0i4nUGfA54ABgN+BpYHBRma8BN+bnpwJ3tnXc7Wkqs46PBD6Rn3/VdVz5Os7lugGPAEuAYW0dd3ubyvwuHwgsBXrm133aOu72NJVZxzcBX83PBwOr2zru9jYBI4GhwIomlo8BFpHGwBkOPNHcNtvTEf+Hw/1GxGagYbjfQmOBW/PzHwNHSY3dsNea0GwdR8TiiHg/v1xCGovBylfO9xjgMmAmsLGWwdWRcup5EvDdiFgHEBGv1zjG9q6cOg6ge37eg4+P22LNiIhHaGQsmwJjgdmRLAH2kvTJUttsT4m/KsP92nbKqeNCZ5P+aVr5mq1jSYcCn4qIhbUMrM6U810eBAyS9AtJSySNrll09aGcOp4OnCbpZdLVXOfVJrQOpaW/27W/nG8HVGy4X2tS2fUn6TRgGDCqqhHVn5J1LGkX0l0pz6xVQHWqnO9yZ1Jz/xdILVePSjo4It6qcmz1opw6HgfcEhFXSzqCNEbLwRGxrfrhdRgtznvt6Yi/JcP9Umq4X2tSOXWMpKOBacDxEbGpRrHVi+bquBtwMPBzSatJ5+wWuINfi5X7ezE/Ij6IiBeAZ0l/BKw85dTx2cBdABHxONCFNI6/VU5Zv9uF2lPi93C/1ddsHedm6FmkpO9zoi1Xso4jYn1E9IqIARExgNSP4viIaPW43B1UOb8X/0nqrIqkXqSm/+drGmX7Vk4d/x44CkDSn5AS///VNMr6twA4PffuHw6sj4hXS63Qbpr6w8P9Vl2ZdXwlsCcwL/eb/H1EHN9mQbczZdax7aAy6/l+4BhJzwBbgW9FxNq2i7p9KbOOLwBulvRNUvPzmT4YaxlJc0mno3rlvhKXALsCRMSNpL4TY4BVwPvAWc1u05+BmZlZx9GemvrNzMxsBznxm5mZdSBO/GZmZh2IE7+ZmVkH4sRvZmbWgTjxmzVC0pn5zniNTUe3cFsT83o1ua+BpBlF8a7Ld6us+OWtkjrnfVxUMO9ESVMaKXt0Ljui0nGUiG9gUV1slfSqpNsklRzWtMQ2h0qaLmmvSsdrVgvt5jp+szZyMmlkrELPtEUgrXBEftwH+HtgrqTdImJ2pXaQr+U+gu3HCj8RGAFcV1T8lzmmlZXafwvMAO4Fds8xXAwcJOmIfF+PlhhKupb6FsDD+1q748RvVtqyiFjV1kG0Rr5TFwCSHiANSTsFqFjiL95PM+XeJo1E2BaeK4jzYUm7k24gcwjgURGtQ3FTv1krSdpD0vWSVkp6LzchL5D0mTLWHS9pWV5vvaTlkiYWlTlS0s8kvZunRZIGtybWiPgAWAYMLNh+D0nfy3FvlvSspG8UxdBd0g2SXpK0SdIfJP23pEF5+XZN/ZJuB74C7F/QvL4qL9uuqV/STZLWSOpUtM8uuU6uKpjXR9KsXH6zpN9IOrs1dZH9Oj/uV7TvGZKWSnpb0huSHpJ0eMHyicDN+eULBe+xf0F9TMt1uUnSK5KuzH80zHYKPuI3K62T0g2fGkREbM3P98jTpcBrpCb1rwOPSzqoqXsZSBoF3EpqCr+ANNzpYKBnQZmxwN2kcbi/TPqTPpV0B7khEfFKK97Lp8lN0znZLgKGAP9Ean4/HrhO0j4RcXFe53pgNOmmTKtIN1gZQboBVmMuyWU+B5yQ521souxsYBJpLPcHCuaPJd3D/bYc617AL0jDlF4MrCYNUXpzPnXx/bLe/fYG5MfniubvC1xNOr2zJ+neH49KGhoRK4H5wAHAhaRTGg1jojd81nOBY4F/JbVufJb0/dgPOKUVcZpVXkR48uSpaCLdFjcamR4rsU4noCtpvOzzCuZPzOv2z6+nAq+X2I5Iye3+ovl7ke5BcVUzsc/I++ucp77AZXneVbnM3+bXpxWtewspUe+dX/8WmFliX53zdi4qmHc7sLqRskfnsiMK3ufzwG1F5RYCywtefxvYAPxxUbkfAX8AOpWIb2De54Qca1fSH401wB3N1GMn0p+N54CrG/k8BxSVPzLP/3LR/DPy/D9t6++1J08R4aZ+s2acABxWMG3XvCzpVEm/lLQe2AK8S2oFKNXc/yugt6TZkv5aUvHR80HA/sB/5KbjzrnV4V3gCWBkmbF/kKfXgG8B15CO3Mnb2ALcUbTO7aQOcH9eEOvZkqZK+jNJFfvNiIjI+ztBUlcASb2BL7J9P4TRwP8ALxbVx/1AH0rXdYMfkOriXeBB0hH9GcWFJB0j6eeS1pLqZzPpCL+cfYwm/Wm6pyjOhtaMvyxjG2ZV58RvVtqKiHiyYHq2YYGkE0hNuyuAcaRkeRjpqLxLUxuMiIdIzb4DSLeGfUPSA5IOzkX65Mdb+Sh5N0yjSacUytHwZ2Ug0C0iLoiITXnZ3sAb8fEe7a8VLAf4Gumc9iRSJ7jXJV0taY8yY2jObNJR+In59TjS79KcgjJ9gL/i43UxNy8vpz6+TaqLLwDfz8//rbCApMNIPf/Xk1oIhudyKyjxeRbF2YXU4lMYZ8O90cv93Myqyuf4zVrvVOC3ETGhYYakLqQm+ZIi4i7gLkl7kpLaFcAiSfsBDbeG/QdgcSOrb2pkXmP7KNVb/U3SbT47FyX/fvlxbd7GO6RTE1MlDSBd3vgvpCPbaeygiFglaQlwGumc/mnAQxGxpqDYWtLlguc3sZlnm5hfaHVBfTwsqTswUdKNEdHQ0e8k0vv6UmGdSNqbdEqhOWtJSX9UE8vXNDHfrKac+M1a7xOk5uBCp9OClrSIeBdYIGkgqVNZT9I4AS8BgyPiygrFWuxh4JvAl4A7C+Z/hZT8nmgk1tXAlZLGAwcXLy+wiXS6o1y3Ad+RdCTpCHt80fL7SOMQrI6IN1qw3VL+kfTeLyF1JoSPPs8P71Uu6RhSh7/fFKzb8Mer+D3eR+qs2TUiHq5QnGYV58Rv1nr3ATfky84WkZLW14G3S60k6Z9Jzb6LSb3C9wMmA09GxJu5zGTgJ7kFYR7paLIf8Hng+Yi4fgdjXwg8TuoZ34+U2P6G1KnxsohYl+N4AvgJqbn7PVIHts8Cs0ps+xlggqRzgKXAhohYUaL8HcC1pD8A7wH3FC2/itTS8Kika4HfAd1IfSE+HxEn0EIR8YqkG4Epkg6JiGWkz3My8CNJt+btX8THj9QbBnCanC9f/AB4OiIelDSPdI7/GtKARZBO6YwBLoiI4qsIzGqvrXsXevK0M0581Kt/YIkynYDLSYnhfVIi/xyp49i/F5Qr7tV/PKnD16uko8eXSOfR+xVt/y9I55zXkY7CXyCd1x7eTOwzyH3nminXA/hejmMzqcn8G0VlriIl7/WkjnHLgckFyxvr1d+N1IqwLi9bledv16u/aD/35GWzm4h1b9KlhatzrK8Dj1Bw9UQT6zX06j+zkWV98nu6u2DelLyPDaTEfSTwGPBg0bqX5s99a9Fn24nUkrI8f2ZvkcZPuALo3tbfa0+eIgJFfNiqZWZmZnXOvfrNzMw6ECd+MzOzDsSJ38zMrANx4jczM+tAnPjNzMw6ECd+MzOzDsSJ38zMrANx4jczM+tAnPjNzMw6kP8HHmudbz3XHFAAAAAASUVORK5CYII=\n",
+ "text/plain": [
+ "<Figure size 576x432 with 1 Axes>"
+ ]
+ },
+ "metadata": {
+ "needs_background": "light"
+ },
+ "output_type": "display_data"
+ }
+ ],
+ "source": [
+ "from sklearn.metrics import roc_curve\n",
+ "#We need to calculate hte fpt,tpr and retrieve the thresholds\n",
+ "fpr, tpr, thresholds = roc_curve(y_train_5, y_scores) \n",
+ "\n",
+ "def plot_roc_curve(fpr, tpr, label=None):\n",
+ " plt.plot(fpr, tpr, linewidth=2, label=label)\n",
+ " plt.plot([0, 1], [0, 1], 'k--')\n",
+ " plt.axis([0, 1, 0, 1])\n",
+ " plt.xlabel('False Positive Rate', fontsize=16)\n",
+ " plt.ylabel('True Positive Rate', fontsize=16)\n",
+ "\n",
+ "plt.figure(figsize=(8, 6))\n",
+ "plot_roc_curve(fpr, tpr)\n",
+ "plt.show()"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "We can calculate the area under the curve (AUC) value using the builtin python function."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 8,
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "0.9604938554008616"
+ ]
+ },
+ "execution_count": 8,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "from sklearn.metrics import roc_auc_score\n",
+ "roc_auc_score(y_train_5, y_scores)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": []
+ }
+ ],
+ "metadata": {
+ "kernelspec": {
+ "display_name": "Python 3",
+ "language": "python",
+ "name": "python3"
+ },
+ "language_info": {
+ "codemirror_mode": {
+ "name": "ipython",
+ "version": 3
+ },
+ "file_extension": ".py",
+ "mimetype": "text/x-python",
+ "name": "python",
+ "nbconvert_exporter": "python",
+ "pygments_lexer": "ipython3",
+ "version": "3.7.6"
+ }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 4
+}
diff --git a/topic_20/.ipynb_checkpoints/Untitled-checkpoint.ipynb b/topic_20/.ipynb_checkpoints/Untitled-checkpoint.ipynb
new file mode 100644
index 0000000000000000000000000000000000000000..7fec51502cbc3200b3d0ffc6bbba1fe85e197f3d
--- /dev/null
+++ b/topic_20/.ipynb_checkpoints/Untitled-checkpoint.ipynb
@@ -0,0 +1,6 @@
+{
+ "cells": [],
+ "metadata": {},
+ "nbformat": 4,
+ "nbformat_minor": 4
+}
diff --git a/topic_20/Untitled.ipynb b/topic_20/Untitled.ipynb
new file mode 100644
index 0000000000000000000000000000000000000000..f2cc0e29750742a503d6304ce00b17220514d1a5
--- /dev/null
+++ b/topic_20/Untitled.ipynb
@@ -0,0 +1,124 @@
+{
+ "cells": [
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": []
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 1,
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "image/png": "iVBORw0KGgoAAAANSUhEUgAAAOcAAADnCAYAAADl9EEgAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAGaElEQVR4nO3dPUiWfR/G8dveSyprs2gOXHqhcAh6hZqsNRqiJoPKRYnAoTGorWyLpqhFcmgpEmqIIByKXiAHIaKhFrGghiJ81ucBr991Z/Z4XPr5jB6cXSfVtxP6c2rb9PT0P0CeJfN9A8DMxAmhxAmhxAmhxAmhljXZ/Vcu/H1tM33RkxNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCLZvvG+B//fr1q9y/fPnyVz9/aGio4fb9+/fy2vHx8XK/ceNGuQ8MDDTc7t69W167atWqcr948WK5X7p0qdzngycnhBInhBInhBInhBInhBInhBInhHLOOYMPHz6U+48fP8r92bNn5f706dOG29TUVHnt8PBwuc+nLVu2lPv58+fLfWRkpOG2du3a8tpt27aV+759+8o9kScnhBInhBInhBInhBInhBInhGqbnp6u9nJsVS9evCj3gwcPlvvffm0r1dKlS8v91q1b5d7e3j7rz960aVO5b9iwody3bt0668/+P2ib6YuenBBKnBBKnBBKnBBKnBBKnBBKnBBqUZ5zTk5Olnt3d3e5T0xMzOXtzKlm997sPPDx48cNtxUrVpTXLtbz3zngnBNaiTghlDghlDghlDghlDghlDgh1KL81pgbN24s96tXr5b7/fv3y33Hjh3l3tfXV+6V7du3l/vo6Gi5N3un8s2bNw23a9euldcytzw5IZQ4IZQ4IZQ4IZQ4IZQ4IZQ4IdSifJ/zT339+rXcm/24ut7e3obbzZs3y2tv375d7idOnCh3InmfE1qJOCGUOCGUOCGUOCGUOCGUOCHUonyf80+tW7fuj65fv379rK9tdg56/Pjxcl+yxL/HrcKfFIQSJ4QSJ4QSJ4QSJ4QSJ4Tyytg8+PbtW8Otp6envPbJkyfl/uDBg3I/fPhwuTMvvDIGrUScEEqcEEqcEEqcEEqcEEqcEMo5Z5iJiYly37lzZ7l3dHSU+4EDB8p9165dDbezZ8+W17a1zXhcR3POOaGViBNCiRNCiRNCiRNCiRNCiRNCOedsMSMjI+V++vTpcm/24wsrly9fLveTJ0+We2dn56w/e4FzzgmtRJwQSpwQSpwQSpwQSpwQSpwQyjnnAvP69ety7+/vL/fR0dFZf/aZM2fKfXBwsNw3b948689ucc45oZWIE0KJE0KJE0KJE0KJE0KJE0I551xkpqamyv3+/fsNt1OnTpXXNvm79M+hQ4fK/dGjR+W+gDnnhFYiTgglTgglTgglTgglTgjlKIV/beXKleX+8+fPcl++fHm5P3z4sOG2f//+8toW5ygFWok4IZQ4IZQ4IZQ4IZQ4IZQ4IdSy+b4B5tarV6/KfXh4uNzHxsYabs3OMZvp6uoq97179/7Rr7/QeHJCKHFCKHFCKHFCKHFCKHFCKHFCKOecYcbHx8v9+vXr5X7v3r1y//Tp02/f07+1bFn916mzs7PclyzxrPhvfjcglDghlDghlDghlDghlDghlDghlHPOv6DZWeKdO3cabkNDQ+W179+/n80tzYndu3eX++DgYLkfPXp0Lm9nwfPkhFDihFDihFDihFDihFDihFCOUmbw+fPncn/79m25nzt3rtzfvXv32/c0V7q7u8v9woULDbdjx46V13rla2753YRQ4oRQ4oRQ4oRQ4oRQ4oRQ4oRQC/acc3JysuHW29tbXvvy5ctyn5iYmNU9zYU9e/aUe39/f7kfOXKk3FevXv3b98Tf4ckJocQJocQJocQJocQJocQJocQJoWLPOZ8/f17uV65cKfexsbGG28ePH2d1T3NlzZo1Dbe+vr7y2mbffrK9vX1W90QeT04IJU4IJU4IJU4IJU4IJU4IJU4IFXvOOTIy8kf7n+jq6ir3np6ecl+6dGm5DwwMNNw6OjrKa1k8PDkhlDghlDghlDghlDghlDghlDghVNv09HS1lyMwJ9pm+qInJ4QSJ4QSJ4QSJ4QSJ4QSJ4QSJ4QSJ4QSJ4QSJ4QSJ4QSJ4QSJ4QSJ4QSJ4QSJ4QSJ4QSJ4QSJ4QSJ4QSJ4QSJ4Rq9iMAZ/yWfcDf58kJocQJocQJocQJocQJocQJof4DO14Dhyk10VwAAAAASUVORK5CYII=\n",
+ "text/plain": [
+ "<Figure size 432x288 with 1 Axes>"
+ ]
+ },
+ "metadata": {
+ "needs_background": "light"
+ },
+ "output_type": "display_data"
+ }
+ ],
+ "source": [
+ "from sklearn.datasets import fetch_openml\n",
+ "mnist = fetch_openml('mnist_784', version=1)\n",
+ "mnist.keys()\n",
+ "\n",
+ "import matplotlib as mpl\n",
+ "import matplotlib.pyplot as plt\n",
+ "\n",
+ "X, y = mnist[\"data\"], mnist[\"target\"]\n",
+ "\n",
+ "some_digit = X[0]\n",
+ "some_digit_image = some_digit.reshape(28, 28)\n",
+ "plt.imshow(some_digit_image, cmap=\"binary\")\n",
+ "plt.axis(\"off\")\n",
+ "plt.show()\n",
+ "\n",
+ "import numpy as np\n",
+ "y = y.astype(np.uint8)\n",
+ "X_train, X_test, y_train, y_test = X[:60000], X[60000:], y[:60000], y[60000:]"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 11,
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "array([3], dtype=uint8)"
+ ]
+ },
+ "execution_count": 11,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "from sklearn.linear_model import SGDClassifier\n",
+ "sgd_clf = SGDClassifier(max_iter=5, tol=-np.infty, random_state=42)\n",
+ "sgd_clf.fit(X_train, y_train)\n",
+ "sgd_clf.predict([some_digit])"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 10,
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "array([[-31893.03095419, -34419.69069632, -9530.63950739,\n",
+ " 1823.73154031, -22320.14822878, -1385.80478895,\n",
+ " -26188.91070951, -16147.51323997, -4604.35491274,\n",
+ " -12050.767298 ]])"
+ ]
+ },
+ "execution_count": 10,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "sgd_clf.decision_function([some_digit])"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": []
+ }
+ ],
+ "metadata": {
+ "kernelspec": {
+ "display_name": "Python 3",
+ "language": "python",
+ "name": "python3"
+ },
+ "language_info": {
+ "codemirror_mode": {
+ "name": "ipython",
+ "version": 3
+ },
+ "file_extension": ".py",
+ "mimetype": "text/x-python",
+ "name": "python",
+ "nbconvert_exporter": "python",
+ "pygments_lexer": "ipython3",
+ "version": "3.7.6"
+ }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 4
+}
diff --git a/topic_20/topic_20.md b/topic_20/topic_20.md
new file mode 100644
index 0000000000000000000000000000000000000000..a25930c9dde856ceab74e79e4af029b03254554c
--- /dev/null
+++ b/topic_20/topic_20.md
@@ -0,0 +1,63 @@
+class: bottom, left
+background-image: url(assets/g.png)
+
+<h2 class="title_headings_sml">COSC102 - Data Science Studio 1</h2>
+
+<h1 class="title_headings_sml"> Topic 20 - Multiclass Classification </h1>
+
+<h3 class="title_headings_sml"> Dr. Mitchell Welch </h3>
+
+---
+
+
+## Reading
+
+* Chapter 3 from ***Hands-on Machine Learning with Scikit-Learn & TensorFlow***
+
+---
+
+## Summary
+
+* Multiclass Classification
+* Coded Demonstrations
+
+---
+
+## Multiclass Classification
+
+* So far we have only looked at a binary problem: building a "5" detector for the MNIST dataset.
+* We will expand on this with a full multiclass classifier.
+* Recall that we have already divided the dataset and created the full multiclass dataset for training.
+* This make things very easy:
+
+```python
+sgd_clf.fit(X_train, y_train)
+sgd_clf.predict([some_digit])
+```
+
+---
+
+## Multiclass Classification
+
+* Some machine learning algorithms are capable of multiclass classification without the need for additional processing. (e.g. Random forest, Naive bayes)
+* Others are strictly binary classifiers that require additional strategies to extend them for the multi-class problem.
+ * One-vs-rest - train one classifier for each class
+ * One-vs-one - train one classifier for each distinct pair.
+* As part of the classification process, a result is calculated from multiple binary classifiers.
+
+---
+
+## Coded Demonstrations
+
+* To demonstrate the use of multiclass classification, we are going to review two coded examples using the *iris* dataset.
+ * [Example 1]()
+ * [Example 2]()
+
+---
+
+## Summary
+
+* Multiclass Classification
+* Coded Demonstrations
+
+---
\ No newline at end of file