{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Exploratory Data Analysis\n",
    "\n",
    "The first phase of any data science project is exploratory data analysis. This involves getting to know the important features of the data. During this process, be open to accepting unexpected features.\n",
    "* The data types in columns \n",
    "* The granularity of data in rows\n",
    "* Distributions of quantitative data and measurements such as center and spread.\n",
    "* Relationships between values and items in the dataset."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Locating and Downloading Datasets\n",
    "\n",
    "With the advent of data driven technologies, data science datasets are becoming more accessible, larger, and more specific. In this notebook, we're focusing on two datasets, both obtained from kaggle. You can find a large range of other data science datasets on these website:\n",
    "* www.kaggle.com/datasets\n",
    "* azure.microsoft.com/en-au/services/open-datasets\n",
    "* archive.ics.uci.edu/ml/datasets.php\n",
    "* en.wikipedia.org/wiki/List_of_datasets_for_machine-learning_research"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "%matplotlib inline\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "def load_data(DATASET_PATH):\n",
    "    return pd.read_csv(DATASET_PATH)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Dataset 1: World Happiness Report up to 2020\n",
    "You can access this dataset here: www.kaggle.com/mathurinache/world-happiness-report \n",
    "> This is an example of a dataset that is:\n",
    "* High quality\n",
    "* Numerical"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "DATASET_PATH = './datasets/happiness/2015.csv'\n",
    "\n",
    "#create pandas object\n",
    "happiness = load_data(DATASET_PATH)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Country</th>\n",
       "      <th>Region</th>\n",
       "      <th>Happiness Rank</th>\n",
       "      <th>Happiness Score</th>\n",
       "      <th>Standard Error</th>\n",
       "      <th>Economy (GDP per Capita)</th>\n",
       "      <th>Family</th>\n",
       "      <th>Health (Life Expectancy)</th>\n",
       "      <th>Freedom</th>\n",
       "      <th>Trust (Government Corruption)</th>\n",
       "      <th>Generosity</th>\n",
       "      <th>Dystopia Residual</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Switzerland</td>\n",
       "      <td>Western Europe</td>\n",
       "      <td>1</td>\n",
       "      <td>7.587</td>\n",
       "      <td>0.03411</td>\n",
       "      <td>1.39651</td>\n",
       "      <td>1.34951</td>\n",
       "      <td>0.94143</td>\n",
       "      <td>0.66557</td>\n",
       "      <td>0.41978</td>\n",
       "      <td>0.29678</td>\n",
       "      <td>2.51738</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Iceland</td>\n",
       "      <td>Western Europe</td>\n",
       "      <td>2</td>\n",
       "      <td>7.561</td>\n",
       "      <td>0.04884</td>\n",
       "      <td>1.30232</td>\n",
       "      <td>1.40223</td>\n",
       "      <td>0.94784</td>\n",
       "      <td>0.62877</td>\n",
       "      <td>0.14145</td>\n",
       "      <td>0.43630</td>\n",
       "      <td>2.70201</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Denmark</td>\n",
       "      <td>Western Europe</td>\n",
       "      <td>3</td>\n",
       "      <td>7.527</td>\n",
       "      <td>0.03328</td>\n",
       "      <td>1.32548</td>\n",
       "      <td>1.36058</td>\n",
       "      <td>0.87464</td>\n",
       "      <td>0.64938</td>\n",
       "      <td>0.48357</td>\n",
       "      <td>0.34139</td>\n",
       "      <td>2.49204</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Norway</td>\n",
       "      <td>Western Europe</td>\n",
       "      <td>4</td>\n",
       "      <td>7.522</td>\n",
       "      <td>0.03880</td>\n",
       "      <td>1.45900</td>\n",
       "      <td>1.33095</td>\n",
       "      <td>0.88521</td>\n",
       "      <td>0.66973</td>\n",
       "      <td>0.36503</td>\n",
       "      <td>0.34699</td>\n",
       "      <td>2.46531</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Canada</td>\n",
       "      <td>North America</td>\n",
       "      <td>5</td>\n",
       "      <td>7.427</td>\n",
       "      <td>0.03553</td>\n",
       "      <td>1.32629</td>\n",
       "      <td>1.32261</td>\n",
       "      <td>0.90563</td>\n",
       "      <td>0.63297</td>\n",
       "      <td>0.32957</td>\n",
       "      <td>0.45811</td>\n",
       "      <td>2.45176</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       Country          Region  Happiness Rank  Happiness Score  \\\n",
       "0  Switzerland  Western Europe               1            7.587   \n",
       "1      Iceland  Western Europe               2            7.561   \n",
       "2      Denmark  Western Europe               3            7.527   \n",
       "3       Norway  Western Europe               4            7.522   \n",
       "4       Canada   North America               5            7.427   \n",
       "\n",
       "   Standard Error  Economy (GDP per Capita)   Family  \\\n",
       "0         0.03411                   1.39651  1.34951   \n",
       "1         0.04884                   1.30232  1.40223   \n",
       "2         0.03328                   1.32548  1.36058   \n",
       "3         0.03880                   1.45900  1.33095   \n",
       "4         0.03553                   1.32629  1.32261   \n",
       "\n",
       "   Health (Life Expectancy)  Freedom  Trust (Government Corruption)  \\\n",
       "0                   0.94143  0.66557                        0.41978   \n",
       "1                   0.94784  0.62877                        0.14145   \n",
       "2                   0.87464  0.64938                        0.48357   \n",
       "3                   0.88521  0.66973                        0.36503   \n",
       "4                   0.90563  0.63297                        0.32957   \n",
       "\n",
       "   Generosity  Dystopia Residual  \n",
       "0     0.29678            2.51738  \n",
       "1     0.43630            2.70201  \n",
       "2     0.34139            2.49204  \n",
       "3     0.34699            2.46531  \n",
       "4     0.45811            2.45176  "
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#you can change the number in the brackets to increase/decrease how much data is shown\n",
    "happiness.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 158 entries, 0 to 157\n",
      "Data columns (total 12 columns):\n",
      "Country                          158 non-null object\n",
      "Region                           158 non-null object\n",
      "Happiness Rank                   158 non-null int64\n",
      "Happiness Score                  158 non-null float64\n",
      "Standard Error                   158 non-null float64\n",
      "Economy (GDP per Capita)         158 non-null float64\n",
      "Family                           158 non-null float64\n",
      "Health (Life Expectancy)         158 non-null float64\n",
      "Freedom                          158 non-null float64\n",
      "Trust (Government Corruption)    158 non-null float64\n",
      "Generosity                       158 non-null float64\n",
      "Dystopia Residual                158 non-null float64\n",
      "dtypes: float64(9), int64(1), object(2)\n",
      "memory usage: 14.9+ KB\n"
     ]
    }
   ],
   "source": [
    "happiness.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Happiness Rank</th>\n",
       "      <th>Happiness Score</th>\n",
       "      <th>Standard Error</th>\n",
       "      <th>Economy (GDP per Capita)</th>\n",
       "      <th>Family</th>\n",
       "      <th>Health (Life Expectancy)</th>\n",
       "      <th>Freedom</th>\n",
       "      <th>Trust (Government Corruption)</th>\n",
       "      <th>Generosity</th>\n",
       "      <th>Dystopia Residual</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>158.000000</td>\n",
       "      <td>158.000000</td>\n",
       "      <td>158.000000</td>\n",
       "      <td>158.000000</td>\n",
       "      <td>158.000000</td>\n",
       "      <td>158.000000</td>\n",
       "      <td>158.000000</td>\n",
       "      <td>158.000000</td>\n",
       "      <td>158.000000</td>\n",
       "      <td>158.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>79.493671</td>\n",
       "      <td>5.375734</td>\n",
       "      <td>0.047885</td>\n",
       "      <td>0.846137</td>\n",
       "      <td>0.991046</td>\n",
       "      <td>0.630259</td>\n",
       "      <td>0.428615</td>\n",
       "      <td>0.143422</td>\n",
       "      <td>0.237296</td>\n",
       "      <td>2.098977</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>45.754363</td>\n",
       "      <td>1.145010</td>\n",
       "      <td>0.017146</td>\n",
       "      <td>0.403121</td>\n",
       "      <td>0.272369</td>\n",
       "      <td>0.247078</td>\n",
       "      <td>0.150693</td>\n",
       "      <td>0.120034</td>\n",
       "      <td>0.126685</td>\n",
       "      <td>0.553550</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>1.000000</td>\n",
       "      <td>2.839000</td>\n",
       "      <td>0.018480</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.328580</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25%</th>\n",
       "      <td>40.250000</td>\n",
       "      <td>4.526000</td>\n",
       "      <td>0.037268</td>\n",
       "      <td>0.545808</td>\n",
       "      <td>0.856823</td>\n",
       "      <td>0.439185</td>\n",
       "      <td>0.328330</td>\n",
       "      <td>0.061675</td>\n",
       "      <td>0.150553</td>\n",
       "      <td>1.759410</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>79.500000</td>\n",
       "      <td>5.232500</td>\n",
       "      <td>0.043940</td>\n",
       "      <td>0.910245</td>\n",
       "      <td>1.029510</td>\n",
       "      <td>0.696705</td>\n",
       "      <td>0.435515</td>\n",
       "      <td>0.107220</td>\n",
       "      <td>0.216130</td>\n",
       "      <td>2.095415</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75%</th>\n",
       "      <td>118.750000</td>\n",
       "      <td>6.243750</td>\n",
       "      <td>0.052300</td>\n",
       "      <td>1.158448</td>\n",
       "      <td>1.214405</td>\n",
       "      <td>0.811013</td>\n",
       "      <td>0.549092</td>\n",
       "      <td>0.180255</td>\n",
       "      <td>0.309883</td>\n",
       "      <td>2.462415</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>158.000000</td>\n",
       "      <td>7.587000</td>\n",
       "      <td>0.136930</td>\n",
       "      <td>1.690420</td>\n",
       "      <td>1.402230</td>\n",
       "      <td>1.025250</td>\n",
       "      <td>0.669730</td>\n",
       "      <td>0.551910</td>\n",
       "      <td>0.795880</td>\n",
       "      <td>3.602140</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       Happiness Rank  Happiness Score  Standard Error  \\\n",
       "count      158.000000       158.000000      158.000000   \n",
       "mean        79.493671         5.375734        0.047885   \n",
       "std         45.754363         1.145010        0.017146   \n",
       "min          1.000000         2.839000        0.018480   \n",
       "25%         40.250000         4.526000        0.037268   \n",
       "50%         79.500000         5.232500        0.043940   \n",
       "75%        118.750000         6.243750        0.052300   \n",
       "max        158.000000         7.587000        0.136930   \n",
       "\n",
       "       Economy (GDP per Capita)      Family  Health (Life Expectancy)  \\\n",
       "count                158.000000  158.000000                158.000000   \n",
       "mean                   0.846137    0.991046                  0.630259   \n",
       "std                    0.403121    0.272369                  0.247078   \n",
       "min                    0.000000    0.000000                  0.000000   \n",
       "25%                    0.545808    0.856823                  0.439185   \n",
       "50%                    0.910245    1.029510                  0.696705   \n",
       "75%                    1.158448    1.214405                  0.811013   \n",
       "max                    1.690420    1.402230                  1.025250   \n",
       "\n",
       "          Freedom  Trust (Government Corruption)  Generosity  \\\n",
       "count  158.000000                     158.000000  158.000000   \n",
       "mean     0.428615                       0.143422    0.237296   \n",
       "std      0.150693                       0.120034    0.126685   \n",
       "min      0.000000                       0.000000    0.000000   \n",
       "25%      0.328330                       0.061675    0.150553   \n",
       "50%      0.435515                       0.107220    0.216130   \n",
       "75%      0.549092                       0.180255    0.309883   \n",
       "max      0.669730                       0.551910    0.795880   \n",
       "\n",
       "       Dystopia Residual  \n",
       "count         158.000000  \n",
       "mean            2.098977  \n",
       "std             0.553550  \n",
       "min             0.328580  \n",
       "25%             1.759410  \n",
       "50%             2.095415  \n",
       "75%             2.462415  \n",
       "max             3.602140  "
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "happiness.describe()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 6536 characters]\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "happiness[\"Family\"].hist()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 57888 characters]\n",
      "text/plain": [
       "<Figure size 1440x1080 with 12 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "happiness.hist(bins=50, figsize=(20,15))\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 14560 characters]\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "happiness.plot(kind=\"scatter\", x=\"Family\", y=\"Happiness Score\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 15732 characters]\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "happiness.plot(kind=\"scatter\", x=\"Trust (Government Corruption)\", y=\"Happiness Score\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 141372 characters]\n",
      "text/plain": [
       "<Figure size 864x576 with 16 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "attributes = [\"Happiness Score\", \"Economy (GDP per Capita)\", \"Family\",\n",
    "\"Health (Life Expectancy)\"]\n",
    "pd.plotting.scatter_matrix(happiness[attributes], figsize=(12, 8))\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 37644 characters]\n",
      "text/plain": [
       "<Figure size 432x288 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "happiness.plot(kind=\"scatter\", x=\"Economy (GDP per Capita)\", y=\"Happiness Score\",\n",
    "s=50, c=\"Generosity\", cmap=plt.get_cmap(\"jet\"), colorbar=True)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Happiness Score                  1.000000\n",
       "Economy (GDP per Capita)         0.780966\n",
       "Family                           0.740605\n",
       "Health (Life Expectancy)         0.724200\n",
       "Freedom                          0.568211\n",
       "Dystopia Residual                0.530474\n",
       "Trust (Government Corruption)    0.395199\n",
       "Generosity                       0.180319\n",
       "Standard Error                  -0.177254\n",
       "Happiness Rank                  -0.992105\n",
       "Name: Happiness Score, dtype: float64"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "corr_matrix = happiness.corr()\n",
    "corr_matrix[\"Happiness Score\"].sort_values(ascending=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Dataset 2: Police Shootings\n",
    "You can access this dataset here: www.kaggle.com/washingtonpost/police-shootings\n",
    "> This is an example of a dataset that is:\n",
    "* Moderate quality\n",
    "* Categorical"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "DATASET_PATH = './datasets/police_shooting/fatal-police-shootings-data.csv'\n",
    "police = load_data(DATASET_PATH)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>name</th>\n",
       "      <th>date</th>\n",
       "      <th>manner_of_death</th>\n",
       "      <th>armed</th>\n",
       "      <th>age</th>\n",
       "      <th>gender</th>\n",
       "      <th>race</th>\n",
       "      <th>city</th>\n",
       "      <th>state</th>\n",
       "      <th>signs_of_mental_illness</th>\n",
       "      <th>threat_level</th>\n",
       "      <th>flee</th>\n",
       "      <th>body_camera</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Tim Elliot</td>\n",
       "      <td>1/2/2015</td>\n",
       "      <td>shot</td>\n",
       "      <td>gun</td>\n",
       "      <td>53.0</td>\n",
       "      <td>M</td>\n",
       "      <td>A</td>\n",
       "      <td>Shelton</td>\n",
       "      <td>WA</td>\n",
       "      <td>True</td>\n",
       "      <td>attack</td>\n",
       "      <td>Not fleeing</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Lewis Lee Lembke</td>\n",
       "      <td>1/2/2015</td>\n",
       "      <td>shot</td>\n",
       "      <td>gun</td>\n",
       "      <td>47.0</td>\n",
       "      <td>M</td>\n",
       "      <td>W</td>\n",
       "      <td>Aloha</td>\n",
       "      <td>OR</td>\n",
       "      <td>False</td>\n",
       "      <td>attack</td>\n",
       "      <td>Not fleeing</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>John Paul Quintero</td>\n",
       "      <td>1/3/2015</td>\n",
       "      <td>shot and Tasered</td>\n",
       "      <td>unarmed</td>\n",
       "      <td>23.0</td>\n",
       "      <td>M</td>\n",
       "      <td>H</td>\n",
       "      <td>Wichita</td>\n",
       "      <td>KS</td>\n",
       "      <td>False</td>\n",
       "      <td>other</td>\n",
       "      <td>Not fleeing</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Matthew Hoffman</td>\n",
       "      <td>1/4/2015</td>\n",
       "      <td>shot</td>\n",
       "      <td>toy weapon</td>\n",
       "      <td>32.0</td>\n",
       "      <td>M</td>\n",
       "      <td>W</td>\n",
       "      <td>San Francisco</td>\n",
       "      <td>CA</td>\n",
       "      <td>True</td>\n",
       "      <td>attack</td>\n",
       "      <td>Not fleeing</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Michael Rodriguez</td>\n",
       "      <td>1/4/2015</td>\n",
       "      <td>shot</td>\n",
       "      <td>nail gun</td>\n",
       "      <td>39.0</td>\n",
       "      <td>M</td>\n",
       "      <td>H</td>\n",
       "      <td>Evans</td>\n",
       "      <td>CO</td>\n",
       "      <td>False</td>\n",
       "      <td>attack</td>\n",
       "      <td>Not fleeing</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                 name      date   manner_of_death       armed   age gender  \\\n",
       "0          Tim Elliot  1/2/2015              shot         gun  53.0      M   \n",
       "1    Lewis Lee Lembke  1/2/2015              shot         gun  47.0      M   \n",
       "2  John Paul Quintero  1/3/2015  shot and Tasered     unarmed  23.0      M   \n",
       "3     Matthew Hoffman  1/4/2015              shot  toy weapon  32.0      M   \n",
       "4   Michael Rodriguez  1/4/2015              shot    nail gun  39.0      M   \n",
       "\n",
       "  race           city state  signs_of_mental_illness threat_level  \\\n",
       "0    A        Shelton    WA                     True       attack   \n",
       "1    W          Aloha    OR                    False       attack   \n",
       "2    H        Wichita    KS                    False        other   \n",
       "3    W  San Francisco    CA                     True       attack   \n",
       "4    H          Evans    CO                    False       attack   \n",
       "\n",
       "          flee  body_camera  \n",
       "0  Not fleeing        False  \n",
       "1  Not fleeing        False  \n",
       "2  Not fleeing        False  \n",
       "3  Not fleeing        False  \n",
       "4  Not fleeing        False  "
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "police.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 5416 entries, 0 to 5415\n",
      "Data columns (total 13 columns):\n",
      "name                       5416 non-null object\n",
      "date                       5416 non-null object\n",
      "manner_of_death            5416 non-null object\n",
      "armed                      5189 non-null object\n",
      "age                        5181 non-null float64\n",
      "gender                     5414 non-null object\n",
      "race                       4895 non-null object\n",
      "city                       5416 non-null object\n",
      "state                      5416 non-null object\n",
      "signs_of_mental_illness    5416 non-null bool\n",
      "threat_level               5416 non-null object\n",
      "flee                       5167 non-null object\n",
      "body_camera                5416 non-null bool\n",
      "dtypes: bool(2), float64(1), object(10)\n",
      "memory usage: 476.1+ KB\n"
     ]
    }
   ],
   "source": [
    "police.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>age</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>5181.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>37.117931</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>13.116135</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>6.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25%</th>\n",
       "      <td>27.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>35.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75%</th>\n",
       "      <td>46.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>91.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "               age\n",
       "count  5181.000000\n",
       "mean     37.117931\n",
       "std      13.116135\n",
       "min       6.000000\n",
       "25%      27.000000\n",
       "50%      35.000000\n",
       "75%      46.000000\n",
       "max      91.000000"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "police.describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Summary of people who were fleeing/mode of fleeing**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Not fleeing    3411\n",
       "Car             900\n",
       "Foot            692\n",
       "Other           164\n",
       "Name: flee, dtype: int64"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "police[\"flee\"].value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Frequency histogram showing age based distribution of shooting victims**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 6904 characters]\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "police[\"age\"].hist()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Age of youngest victim:  6.0\n",
      "Age of oldest victim:  91.0\n"
     ]
    }
   ],
   "source": [
    "print(\"Age of youngest victim: \",police[\"age\"].min())\n",
    "print(\"Age of oldest victim: \",police[\"age\"].max())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Gender distribution**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "gender\n",
       "F     238\n",
       "M    5176\n",
       "dtype: int64"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "police.groupby('gender').size()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "M    95.60399\n",
       "F     4.39601\n",
       "Name: gender, dtype: float64"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "police['gender'].value_counts(normalize=True) * 100"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 8504 characters]\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "labels = police['gender'].astype('category').cat.categories.tolist()\n",
    "counts = police['gender'].value_counts()\n",
    "sizes = [counts[var_cat] for var_cat in labels]\n",
    "fig1, ax1 = plt.subplots()\n",
    "ax1.pie(sizes, labels=labels, autopct='%1.1f%%', shadow=False) \n",
    "ax1.axis('equal')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Race Distribution**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "W    50.582227\n",
       "B    26.516854\n",
       "H    18.426966\n",
       "A     1.899898\n",
       "N     1.593463\n",
       "O     0.980592\n",
       "Name: race, dtype: float64"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "police['race'].value_counts(normalize=True) * 100"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Visualise race distribution where**\n",
    " W: White, non-Hispanic B: Black, non-Hispanic A: Asian N: Native American H: Hispanic O: Other None: unknown"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 15384 characters]\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "%matplotlib inline\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns\n",
    "\n",
    "race_count = police['race'].value_counts()\n",
    "sns.set(style=\"darkgrid\")\n",
    "sns.barplot(race_count.index, race_count.values, alpha=0.9)\n",
    "plt.title('Frequency Distribution of Races')\n",
    "plt.ylabel('Number of Occurrences', fontsize=12)\n",
    "plt.xlabel('Race', fontsize=12)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercise 1\n",
    "Use pandas and matplot lib to show the rate of shootings overtime"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercise 2\n",
    "Locate another dataset on one of the websites mentioned above. Find interesting ways to visualise patterns, and post your work on SLACK!"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
