04-02-MNIST(openml).ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Classification Based Machine Learning Algorithm\n",
    "\n",
    "[An introduction to machine learning with scikit-learn](http://scikit-learn.org/stable/tutorial/basic/tutorial.html#introduction)\n",
    "\n",
    "This notebook is inspired by Geron [Hands-on Machine Learning with Scikit-Learn and TensorFlow](http://shop.oreilly.com/product/0636920052289.do)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Scikit-learn Definition:\n",
    "\n",
    "**Supervised learning**, in which the data comes with additional attributes that we want to predict. This problem can be either:\n",
    "\n",
    "* **Classification**: samples belong to two or more *classes* and we want to learn from already labeled data how to predict the class of unlabeled data. An example of classification problem would be the handwritten digit recognition example, in which the aim is to assign each input vector to one of a finite number of discrete categories. Another way to think of classification is as a discrete (as opposed to continuous) form of supervised learning where one has a limited number of categories and for each of the n samples provided, one is to try to label them with the correct category or class.\n",
    "\n",
    "\n",
    "* **Regression**: if the desired output consists of one or more *continuous variables*, then the task is called regression. An example of a regression problem would be the prediction of the length of a salmon as a function of its age and weight."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "MNIST dataset - a set of 70,000 small images of digits handwritten. You can read more via [The MNIST Database](http://yann.lecun.com/exdb/mnist/)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Downloading the MNIST dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'0.21.1'"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import sklearn\n",
    "sklearn.__version__"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "#from sklearn.datasets import fetch_mldata\n",
    "#mnist = fetch_mldata('MNIST original')\n",
    "from sklearn.datasets import fetch_openml\n",
    "mnist = fetch_openml(name='mnist_784')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'data': array([[0., 0., 0., ..., 0., 0., 0.],\n",
       "        [0., 0., 0., ..., 0., 0., 0.],\n",
       "        [0., 0., 0., ..., 0., 0., 0.],\n",
       "        ...,\n",
       "        [0., 0., 0., ..., 0., 0., 0.],\n",
       "        [0., 0., 0., ..., 0., 0., 0.],\n",
       "        [0., 0., 0., ..., 0., 0., 0.]]),\n",
       " 'target': array(['5', '0', '4', ..., '4', '5', '6'], dtype=object),\n",
       " 'feature_names': ['pixel1',\n",
       "  'pixel2',\n",
       "  'pixel3',\n",
       "  'pixel4',\n",
       "  'pixel5',\n",
       "  'pixel6',\n",
       "  'pixel7',\n",
       "  'pixel8',\n",
       "  'pixel9',\n",
       "  'pixel10',\n",
       "  'pixel11',\n",
       "  'pixel12',\n",
       "  'pixel13',\n",
       "  'pixel14',\n",
       "  'pixel15',\n",
       "  'pixel16',\n",
       "  'pixel17',\n",
       "  'pixel18',\n",
       "  'pixel19',\n",
       "  'pixel20',\n",
       "  'pixel21',\n",
       "  'pixel22',\n",
       "  'pixel23',\n",
       "  'pixel24',\n",
       "  'pixel25',\n",
       "  'pixel26',\n",
       "  'pixel27',\n",
       "  'pixel28',\n",
       "  'pixel29',\n",
       "  'pixel30',\n",
       "  'pixel31',\n",
       "  'pixel32',\n",
       "  'pixel33',\n",
       "  'pixel34',\n",
       "  'pixel35',\n",
       "  'pixel36',\n",
       "  'pixel37',\n",
       "  'pixel38',\n",
       "  'pixel39',\n",
       "  'pixel40',\n",
       "  'pixel41',\n",
       "  'pixel42',\n",
       "  'pixel43',\n",
       "  'pixel44',\n",
       "  'pixel45',\n",
       "  'pixel46',\n",
       "  'pixel47',\n",
       "  'pixel48',\n",
       "  'pixel49',\n",
       "  'pixel50',\n",
       "  'pixel51',\n",
       "  'pixel52',\n",
       "  'pixel53',\n",
       "  'pixel54',\n",
       "  'pixel55',\n",
       "  'pixel56',\n",
       "  'pixel57',\n",
       "  'pixel58',\n",
       "  'pixel59',\n",
       "  'pixel60',\n",
       "  'pixel61',\n",
       "  'pixel62',\n",
       "  'pixel63',\n",
       "  'pixel64',\n",
       "  'pixel65',\n",
       "  'pixel66',\n",
       "  'pixel67',\n",
       "  'pixel68',\n",
       "  'pixel69',\n",
       "  'pixel70',\n",
       "  'pixel71',\n",
       "  'pixel72',\n",
       "  'pixel73',\n",
       "  'pixel74',\n",
       "  'pixel75',\n",
       "  'pixel76',\n",
       "  'pixel77',\n",
       "  'pixel78',\n",
       "  'pixel79',\n",
       "  'pixel80',\n",
       "  'pixel81',\n",
       "  'pixel82',\n",
       "  'pixel83',\n",
       "  'pixel84',\n",
       "  'pixel85',\n",
       "  'pixel86',\n",
       "  'pixel87',\n",
       "  'pixel88',\n",
       "  'pixel89',\n",
       "  'pixel90',\n",
       "  'pixel91',\n",
       "  'pixel92',\n",
       "  'pixel93',\n",
       "  'pixel94',\n",
       "  'pixel95',\n",
       "  'pixel96',\n",
       "  'pixel97',\n",
       "  'pixel98',\n",
       "  'pixel99',\n",
       "  'pixel100',\n",
       "  'pixel101',\n",
       "  'pixel102',\n",
       "  'pixel103',\n",
       "  'pixel104',\n",
       "  'pixel105',\n",
       "  'pixel106',\n",
       "  'pixel107',\n",
       "  'pixel108',\n",
       "  'pixel109',\n",
       "  'pixel110',\n",
       "  'pixel111',\n",
       "  'pixel112',\n",
       "  'pixel113',\n",
       "  'pixel114',\n",
       "  'pixel115',\n",
       "  'pixel116',\n",
       "  'pixel117',\n",
       "  'pixel118',\n",
       "  'pixel119',\n",
       "  'pixel120',\n",
       "  'pixel121',\n",
       "  'pixel122',\n",
       "  'pixel123',\n",
       "  'pixel124',\n",
       "  'pixel125',\n",
       "  'pixel126',\n",
       "  'pixel127',\n",
       "  'pixel128',\n",
       "  'pixel129',\n",
       "  'pixel130',\n",
       "  'pixel131',\n",
       "  'pixel132',\n",
       "  'pixel133',\n",
       "  'pixel134',\n",
       "  'pixel135',\n",
       "  'pixel136',\n",
       "  'pixel137',\n",
       "  'pixel138',\n",
       "  'pixel139',\n",
       "  'pixel140',\n",
       "  'pixel141',\n",
       "  'pixel142',\n",
       "  'pixel143',\n",
       "  'pixel144',\n",
       "  'pixel145',\n",
       "  'pixel146',\n",
       "  'pixel147',\n",
       "  'pixel148',\n",
       "  'pixel149',\n",
       "  'pixel150',\n",
       "  'pixel151',\n",
       "  'pixel152',\n",
       "  'pixel153',\n",
       "  'pixel154',\n",
       "  'pixel155',\n",
       "  'pixel156',\n",
       "  'pixel157',\n",
       "  'pixel158',\n",
       "  'pixel159',\n",
       "  'pixel160',\n",
       "  'pixel161',\n",
       "  'pixel162',\n",
       "  'pixel163',\n",
       "  'pixel164',\n",
       "  'pixel165',\n",
       "  'pixel166',\n",
       "  'pixel167',\n",
       "  'pixel168',\n",
       "  'pixel169',\n",
       "  'pixel170',\n",
       "  'pixel171',\n",
       "  'pixel172',\n",
       "  'pixel173',\n",
       "  'pixel174',\n",
       "  'pixel175',\n",
       "  'pixel176',\n",
       "  'pixel177',\n",
       "  'pixel178',\n",
       "  'pixel179',\n",
       "  'pixel180',\n",
       "  'pixel181',\n",
       "  'pixel182',\n",
       "  'pixel183',\n",
       "  'pixel184',\n",
       "  'pixel185',\n",
       "  'pixel186',\n",
       "  'pixel187',\n",
       "  'pixel188',\n",
       "  'pixel189',\n",
       "  'pixel190',\n",
       "  'pixel191',\n",
       "  'pixel192',\n",
       "  'pixel193',\n",
       "  'pixel194',\n",
       "  'pixel195',\n",
       "  'pixel196',\n",
       "  'pixel197',\n",
       "  'pixel198',\n",
       "  'pixel199',\n",
       "  'pixel200',\n",
       "  'pixel201',\n",
       "  'pixel202',\n",
       "  'pixel203',\n",
       "  'pixel204',\n",
       "  'pixel205',\n",
       "  'pixel206',\n",
       "  'pixel207',\n",
       "  'pixel208',\n",
       "  'pixel209',\n",
       "  'pixel210',\n",
       "  'pixel211',\n",
       "  'pixel212',\n",
       "  'pixel213',\n",
       "  'pixel214',\n",
       "  'pixel215',\n",
       "  'pixel216',\n",
       "  'pixel217',\n",
       "  'pixel218',\n",
       "  'pixel219',\n",
       "  'pixel220',\n",
       "  'pixel221',\n",
       "  'pixel222',\n",
       "  'pixel223',\n",
       "  'pixel224',\n",
       "  'pixel225',\n",
       "  'pixel226',\n",
       "  'pixel227',\n",
       "  'pixel228',\n",
       "  'pixel229',\n",
       "  'pixel230',\n",
       "  'pixel231',\n",
       "  'pixel232',\n",
       "  'pixel233',\n",
       "  'pixel234',\n",
       "  'pixel235',\n",
       "  'pixel236',\n",
       "  'pixel237',\n",
       "  'pixel238',\n",
       "  'pixel239',\n",
       "  'pixel240',\n",
       "  'pixel241',\n",
       "  'pixel242',\n",
       "  'pixel243',\n",
       "  'pixel244',\n",
       "  'pixel245',\n",
       "  'pixel246',\n",
       "  'pixel247',\n",
       "  'pixel248',\n",
       "  'pixel249',\n",
       "  'pixel250',\n",
       "  'pixel251',\n",
       "  'pixel252',\n",
       "  'pixel253',\n",
       "  'pixel254',\n",
       "  'pixel255',\n",
       "  'pixel256',\n",
       "  'pixel257',\n",
       "  'pixel258',\n",
       "  'pixel259',\n",
       "  'pixel260',\n",
       "  'pixel261',\n",
       "  'pixel262',\n",
       "  'pixel263',\n",
       "  'pixel264',\n",
       "  'pixel265',\n",
       "  'pixel266',\n",
       "  'pixel267',\n",
       "  'pixel268',\n",
       "  'pixel269',\n",
       "  'pixel270',\n",
       "  'pixel271',\n",
       "  'pixel272',\n",
       "  'pixel273',\n",
       "  'pixel274',\n",
       "  'pixel275',\n",
       "  'pixel276',\n",
       "  'pixel277',\n",
       "  'pixel278',\n",
       "  'pixel279',\n",
       "  'pixel280',\n",
       "  'pixel281',\n",
       "  'pixel282',\n",
       "  'pixel283',\n",
       "  'pixel284',\n",
       "  'pixel285',\n",
       "  'pixel286',\n",
       "  'pixel287',\n",
       "  'pixel288',\n",
       "  'pixel289',\n",
       "  'pixel290',\n",
       "  'pixel291',\n",
       "  'pixel292',\n",
       "  'pixel293',\n",
       "  'pixel294',\n",
       "  'pixel295',\n",
       "  'pixel296',\n",
       "  'pixel297',\n",
       "  'pixel298',\n",
       "  'pixel299',\n",
       "  'pixel300',\n",
       "  'pixel301',\n",
       "  'pixel302',\n",
       "  'pixel303',\n",
       "  'pixel304',\n",
       "  'pixel305',\n",
       "  'pixel306',\n",
       "  'pixel307',\n",
       "  'pixel308',\n",
       "  'pixel309',\n",
       "  'pixel310',\n",
       "  'pixel311',\n",
       "  'pixel312',\n",
       "  'pixel313',\n",
       "  'pixel314',\n",
       "  'pixel315',\n",
       "  'pixel316',\n",
       "  'pixel317',\n",
       "  'pixel318',\n",
       "  'pixel319',\n",
       "  'pixel320',\n",
       "  'pixel321',\n",
       "  'pixel322',\n",
       "  'pixel323',\n",
       "  'pixel324',\n",
       "  'pixel325',\n",
       "  'pixel326',\n",
       "  'pixel327',\n",
       "  'pixel328',\n",
       "  'pixel329',\n",
       "  'pixel330',\n",
       "  'pixel331',\n",
       "  'pixel332',\n",
       "  'pixel333',\n",
       "  'pixel334',\n",
       "  'pixel335',\n",
       "  'pixel336',\n",
       "  'pixel337',\n",
       "  'pixel338',\n",
       "  'pixel339',\n",
       "  'pixel340',\n",
       "  'pixel341',\n",
       "  'pixel342',\n",
       "  'pixel343',\n",
       "  'pixel344',\n",
       "  'pixel345',\n",
       "  'pixel346',\n",
       "  'pixel347',\n",
       "  'pixel348',\n",
       "  'pixel349',\n",
       "  'pixel350',\n",
       "  'pixel351',\n",
       "  'pixel352',\n",
       "  'pixel353',\n",
       "  'pixel354',\n",
       "  'pixel355',\n",
       "  'pixel356',\n",
       "  'pixel357',\n",
       "  'pixel358',\n",
       "  'pixel359',\n",
       "  'pixel360',\n",
       "  'pixel361',\n",
       "  'pixel362',\n",
       "  'pixel363',\n",
       "  'pixel364',\n",
       "  'pixel365',\n",
       "  'pixel366',\n",
       "  'pixel367',\n",
       "  'pixel368',\n",
       "  'pixel369',\n",
       "  'pixel370',\n",
       "  'pixel371',\n",
       "  'pixel372',\n",
       "  'pixel373',\n",
       "  'pixel374',\n",
       "  'pixel375',\n",
       "  'pixel376',\n",
       "  'pixel377',\n",
       "  'pixel378',\n",
       "  'pixel379',\n",
       "  'pixel380',\n",
       "  'pixel381',\n",
       "  'pixel382',\n",
       "  'pixel383',\n",
       "  'pixel384',\n",
       "  'pixel385',\n",
       "  'pixel386',\n",
       "  'pixel387',\n",
       "  'pixel388',\n",
       "  'pixel389',\n",
       "  'pixel390',\n",
       "  'pixel391',\n",
       "  'pixel392',\n",
       "  'pixel393',\n",
       "  'pixel394',\n",
       "  'pixel395',\n",
       "  'pixel396',\n",
       "  'pixel397',\n",
       "  'pixel398',\n",
       "  'pixel399',\n",
       "  'pixel400',\n",
       "  'pixel401',\n",
       "  'pixel402',\n",
       "  'pixel403',\n",
       "  'pixel404',\n",
       "  'pixel405',\n",
       "  'pixel406',\n",
       "  'pixel407',\n",
       "  'pixel408',\n",
       "  'pixel409',\n",
       "  'pixel410',\n",
       "  'pixel411',\n",
       "  'pixel412',\n",
       "  'pixel413',\n",
       "  'pixel414',\n",
       "  'pixel415',\n",
       "  'pixel416',\n",
       "  'pixel417',\n",
       "  'pixel418',\n",
       "  'pixel419',\n",
       "  'pixel420',\n",
       "  'pixel421',\n",
       "  'pixel422',\n",
       "  'pixel423',\n",
       "  'pixel424',\n",
       "  'pixel425',\n",
       "  'pixel426',\n",
       "  'pixel427',\n",
       "  'pixel428',\n",
       "  'pixel429',\n",
       "  'pixel430',\n",
       "  'pixel431',\n",
       "  'pixel432',\n",
       "  'pixel433',\n",
       "  'pixel434',\n",
       "  'pixel435',\n",
       "  'pixel436',\n",
       "  'pixel437',\n",
       "  'pixel438',\n",
       "  'pixel439',\n",
       "  'pixel440',\n",
       "  'pixel441',\n",
       "  'pixel442',\n",
       "  'pixel443',\n",
       "  'pixel444',\n",
       "  'pixel445',\n",
       "  'pixel446',\n",
       "  'pixel447',\n",
       "  'pixel448',\n",
       "  'pixel449',\n",
       "  'pixel450',\n",
       "  'pixel451',\n",
       "  'pixel452',\n",
       "  'pixel453',\n",
       "  'pixel454',\n",
       "  'pixel455',\n",
       "  'pixel456',\n",
       "  'pixel457',\n",
       "  'pixel458',\n",
       "  'pixel459',\n",
       "  'pixel460',\n",
       "  'pixel461',\n",
       "  'pixel462',\n",
       "  'pixel463',\n",
       "  'pixel464',\n",
       "  'pixel465',\n",
       "  'pixel466',\n",
       "  'pixel467',\n",
       "  'pixel468',\n",
       "  'pixel469',\n",
       "  'pixel470',\n",
       "  'pixel471',\n",
       "  'pixel472',\n",
       "  'pixel473',\n",
       "  'pixel474',\n",
       "  'pixel475',\n",
       "  'pixel476',\n",
       "  'pixel477',\n",
       "  'pixel478',\n",
       "  'pixel479',\n",
       "  'pixel480',\n",
       "  'pixel481',\n",
       "  'pixel482',\n",
       "  'pixel483',\n",
       "  'pixel484',\n",
       "  'pixel485',\n",
       "  'pixel486',\n",
       "  'pixel487',\n",
       "  'pixel488',\n",
       "  'pixel489',\n",
       "  'pixel490',\n",
       "  'pixel491',\n",
       "  'pixel492',\n",
       "  'pixel493',\n",
       "  'pixel494',\n",
       "  'pixel495',\n",
       "  'pixel496',\n",
       "  'pixel497',\n",
       "  'pixel498',\n",
       "  'pixel499',\n",
       "  'pixel500',\n",
       "  'pixel501',\n",
       "  'pixel502',\n",
       "  'pixel503',\n",
       "  'pixel504',\n",
       "  'pixel505',\n",
       "  'pixel506',\n",
       "  'pixel507',\n",
       "  'pixel508',\n",
       "  'pixel509',\n",
       "  'pixel510',\n",
       "  'pixel511',\n",
       "  'pixel512',\n",
       "  'pixel513',\n",
       "  'pixel514',\n",
       "  'pixel515',\n",
       "  'pixel516',\n",
       "  'pixel517',\n",
       "  'pixel518',\n",
       "  'pixel519',\n",
       "  'pixel520',\n",
       "  'pixel521',\n",
       "  'pixel522',\n",
       "  'pixel523',\n",
       "  'pixel524',\n",
       "  'pixel525',\n",
       "  'pixel526',\n",
       "  'pixel527',\n",
       "  'pixel528',\n",
       "  'pixel529',\n",
       "  'pixel530',\n",
       "  'pixel531',\n",
       "  'pixel532',\n",
       "  'pixel533',\n",
       "  'pixel534',\n",
       "  'pixel535',\n",
       "  'pixel536',\n",
       "  'pixel537',\n",
       "  'pixel538',\n",
       "  'pixel539',\n",
       "  'pixel540',\n",
       "  'pixel541',\n",
       "  'pixel542',\n",
       "  'pixel543',\n",
       "  'pixel544',\n",
       "  'pixel545',\n",
       "  'pixel546',\n",
       "  'pixel547',\n",
       "  'pixel548',\n",
       "  'pixel549',\n",
       "  'pixel550',\n",
       "  'pixel551',\n",
       "  'pixel552',\n",
       "  'pixel553',\n",
       "  'pixel554',\n",
       "  'pixel555',\n",
       "  'pixel556',\n",
       "  'pixel557',\n",
       "  'pixel558',\n",
       "  'pixel559',\n",
       "  'pixel560',\n",
       "  'pixel561',\n",
       "  'pixel562',\n",
       "  'pixel563',\n",
       "  'pixel564',\n",
       "  'pixel565',\n",
       "  'pixel566',\n",
       "  'pixel567',\n",
       "  'pixel568',\n",
       "  'pixel569',\n",
       "  'pixel570',\n",
       "  'pixel571',\n",
       "  'pixel572',\n",
       "  'pixel573',\n",
       "  'pixel574',\n",
       "  'pixel575',\n",
       "  'pixel576',\n",
       "  'pixel577',\n",
       "  'pixel578',\n",
       "  'pixel579',\n",
       "  'pixel580',\n",
       "  'pixel581',\n",
       "  'pixel582',\n",
       "  'pixel583',\n",
       "  'pixel584',\n",
       "  'pixel585',\n",
       "  'pixel586',\n",
       "  'pixel587',\n",
       "  'pixel588',\n",
       "  'pixel589',\n",
       "  'pixel590',\n",
       "  'pixel591',\n",
       "  'pixel592',\n",
       "  'pixel593',\n",
       "  'pixel594',\n",
       "  'pixel595',\n",
       "  'pixel596',\n",
       "  'pixel597',\n",
       "  'pixel598',\n",
       "  'pixel599',\n",
       "  'pixel600',\n",
       "  'pixel601',\n",
       "  'pixel602',\n",
       "  'pixel603',\n",
       "  'pixel604',\n",
       "  'pixel605',\n",
       "  'pixel606',\n",
       "  'pixel607',\n",
       "  'pixel608',\n",
       "  'pixel609',\n",
       "  'pixel610',\n",
       "  'pixel611',\n",
       "  'pixel612',\n",
       "  'pixel613',\n",
       "  'pixel614',\n",
       "  'pixel615',\n",
       "  'pixel616',\n",
       "  'pixel617',\n",
       "  'pixel618',\n",
       "  'pixel619',\n",
       "  'pixel620',\n",
       "  'pixel621',\n",
       "  'pixel622',\n",
       "  'pixel623',\n",
       "  'pixel624',\n",
       "  'pixel625',\n",
       "  'pixel626',\n",
       "  'pixel627',\n",
       "  'pixel628',\n",
       "  'pixel629',\n",
       "  'pixel630',\n",
       "  'pixel631',\n",
       "  'pixel632',\n",
       "  'pixel633',\n",
       "  'pixel634',\n",
       "  'pixel635',\n",
       "  'pixel636',\n",
       "  'pixel637',\n",
       "  'pixel638',\n",
       "  'pixel639',\n",
       "  'pixel640',\n",
       "  'pixel641',\n",
       "  'pixel642',\n",
       "  'pixel643',\n",
       "  'pixel644',\n",
       "  'pixel645',\n",
       "  'pixel646',\n",
       "  'pixel647',\n",
       "  'pixel648',\n",
       "  'pixel649',\n",
       "  'pixel650',\n",
       "  'pixel651',\n",
       "  'pixel652',\n",
       "  'pixel653',\n",
       "  'pixel654',\n",
       "  'pixel655',\n",
       "  'pixel656',\n",
       "  'pixel657',\n",
       "  'pixel658',\n",
       "  'pixel659',\n",
       "  'pixel660',\n",
       "  'pixel661',\n",
       "  'pixel662',\n",
       "  'pixel663',\n",
       "  'pixel664',\n",
       "  'pixel665',\n",
       "  'pixel666',\n",
       "  'pixel667',\n",
       "  'pixel668',\n",
       "  'pixel669',\n",
       "  'pixel670',\n",
       "  'pixel671',\n",
       "  'pixel672',\n",
       "  'pixel673',\n",
       "  'pixel674',\n",
       "  'pixel675',\n",
       "  'pixel676',\n",
       "  'pixel677',\n",
       "  'pixel678',\n",
       "  'pixel679',\n",
       "  'pixel680',\n",
       "  'pixel681',\n",
       "  'pixel682',\n",
       "  'pixel683',\n",
       "  'pixel684',\n",
       "  'pixel685',\n",
       "  'pixel686',\n",
       "  'pixel687',\n",
       "  'pixel688',\n",
       "  'pixel689',\n",
       "  'pixel690',\n",
       "  'pixel691',\n",
       "  'pixel692',\n",
       "  'pixel693',\n",
       "  'pixel694',\n",
       "  'pixel695',\n",
       "  'pixel696',\n",
       "  'pixel697',\n",
       "  'pixel698',\n",
       "  'pixel699',\n",
       "  'pixel700',\n",
       "  'pixel701',\n",
       "  'pixel702',\n",
       "  'pixel703',\n",
       "  'pixel704',\n",
       "  'pixel705',\n",
       "  'pixel706',\n",
       "  'pixel707',\n",
       "  'pixel708',\n",
       "  'pixel709',\n",
       "  'pixel710',\n",
       "  'pixel711',\n",
       "  'pixel712',\n",
       "  'pixel713',\n",
       "  'pixel714',\n",
       "  'pixel715',\n",
       "  'pixel716',\n",
       "  'pixel717',\n",
       "  'pixel718',\n",
       "  'pixel719',\n",
       "  'pixel720',\n",
       "  'pixel721',\n",
       "  'pixel722',\n",
       "  'pixel723',\n",
       "  'pixel724',\n",
       "  'pixel725',\n",
       "  'pixel726',\n",
       "  'pixel727',\n",
       "  'pixel728',\n",
       "  'pixel729',\n",
       "  'pixel730',\n",
       "  'pixel731',\n",
       "  'pixel732',\n",
       "  'pixel733',\n",
       "  'pixel734',\n",
       "  'pixel735',\n",
       "  'pixel736',\n",
       "  'pixel737',\n",
       "  'pixel738',\n",
       "  'pixel739',\n",
       "  'pixel740',\n",
       "  'pixel741',\n",
       "  'pixel742',\n",
       "  'pixel743',\n",
       "  'pixel744',\n",
       "  'pixel745',\n",
       "  'pixel746',\n",
       "  'pixel747',\n",
       "  'pixel748',\n",
       "  'pixel749',\n",
       "  'pixel750',\n",
       "  'pixel751',\n",
       "  'pixel752',\n",
       "  'pixel753',\n",
       "  'pixel754',\n",
       "  'pixel755',\n",
       "  'pixel756',\n",
       "  'pixel757',\n",
       "  'pixel758',\n",
       "  'pixel759',\n",
       "  'pixel760',\n",
       "  'pixel761',\n",
       "  'pixel762',\n",
       "  'pixel763',\n",
       "  'pixel764',\n",
       "  'pixel765',\n",
       "  'pixel766',\n",
       "  'pixel767',\n",
       "  'pixel768',\n",
       "  'pixel769',\n",
       "  'pixel770',\n",
       "  'pixel771',\n",
       "  'pixel772',\n",
       "  'pixel773',\n",
       "  'pixel774',\n",
       "  'pixel775',\n",
       "  'pixel776',\n",
       "  'pixel777',\n",
       "  'pixel778',\n",
       "  'pixel779',\n",
       "  'pixel780',\n",
       "  'pixel781',\n",
       "  'pixel782',\n",
       "  'pixel783',\n",
       "  'pixel784'],\n",
       " 'DESCR': \"**Author**: Yann LeCun, Corinna Cortes, Christopher J.C. Burges  \\n**Source**: [MNIST Website](http://yann.lecun.com/exdb/mnist/) - Date unknown  \\n**Please cite**:  \\n\\nThe MNIST database of handwritten digits with 784 features, raw data available at: http://yann.lecun.com/exdb/mnist/. It can be split in a training set of the first 60,000 examples, and a test set of 10,000 examples  \\n\\nIt is a subset of a larger set available from NIST. The digits have been size-normalized and centered in a fixed-size image. It is a good database for people who want to try learning techniques and pattern recognition methods on real-world data while spending minimal efforts on preprocessing and formatting. The original black and white (bilevel) images from NIST were size normalized to fit in a 20x20 pixel box while preserving their aspect ratio. The resulting images contain grey levels as a result of the anti-aliasing technique used by the normalization algorithm. the images were centered in a 28x28 image by computing the center of mass of the pixels, and translating the image so as to position this point at the center of the 28x28 field.  \\n\\nWith some classification methods (particularly template-based methods, such as SVM and K-nearest neighbors), the error rate improves when the digits are centered by bounding box rather than center of mass. If you do this kind of pre-processing, you should report it in your publications. The MNIST database was constructed from NIST's NIST originally designated SD-3 as their training set and SD-1 as their test set. However, SD-3 is much cleaner and easier to recognize than SD-1. The reason for this can be found on the fact that SD-3 was collected among Census Bureau employees, while SD-1 was collected among high-school students. Drawing sensible conclusions from learning experiments requires that the result be independent of the choice of training set and test among the complete set of samples. Therefore it was necessary to build a new database by mixing NIST's datasets.  \\n\\nThe MNIST training set is composed of 30,000 patterns from SD-3 and 30,000 patterns from SD-1. Our test set was composed of 5,000 patterns from SD-3 and 5,000 patterns from SD-1. The 60,000 pattern training set contained examples from approximately 250 writers. We made sure that the sets of writers of the training set and test set were disjoint. SD-1 contains 58,527 digit images written by 500 different writers. In contrast to SD-3, where blocks of data from each writer appeared in sequence, the data in SD-1 is scrambled. Writer identities for SD-1 is available and we used this information to unscramble the writers. We then split SD-1 in two: characters written by the first 250 writers went into our new training set. The remaining 250 writers were placed in our test set. Thus we had two sets with nearly 30,000 examples each. The new training set was completed with enough examples from SD-3, starting at pattern # 0, to make a full set of 60,000 training patterns. Similarly, the new test set was completed with SD-3 examples starting at pattern # 35,000 to make a full set with 60,000 test patterns. Only a subset of 10,000 test images (5,000 from SD-1 and 5,000 from SD-3) is available on this site. The full 60,000 sample training set is available.\\n\\nDownloaded from openml.org.\",\n",
       " 'details': {'id': '554',\n",
       "  'name': 'mnist_784',\n",
       "  'version': '1',\n",
       "  'format': 'ARFF',\n",
       "  'upload_date': '2014-09-29T03:28:38',\n",
       "  'licence': 'Public',\n",
       "  'url': 'https://www.openml.org/data/v1/download/52667/mnist_784.arff',\n",
       "  'file_id': '52667',\n",
       "  'default_target_attribute': 'class',\n",
       "  'tag': ['AzurePilot',\n",
       "   'OpenML-CC18',\n",
       "   'OpenML100',\n",
       "   'study_1',\n",
       "   'study_123',\n",
       "   'study_41',\n",
       "   'study_99',\n",
       "   'vision'],\n",
       "  'visibility': 'public',\n",
       "  'status': 'active',\n",
       "  'processing_date': '2018-10-03 21:23:30',\n",
       "  'md5_checksum': '0298d579eb1b86163de7723944c7e495'},\n",
       " 'categories': {},\n",
       " 'url': 'https://www.openml.org/d/554'}"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "mnist"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "70000"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(mnist['data'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Visualisation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "X, y = mnist['data'], mnist['target']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[0., 0., 0., ..., 0., 0., 0.],\n",
       "       [0., 0., 0., ..., 0., 0., 0.],\n",
       "       [0., 0., 0., ..., 0., 0., 0.],\n",
       "       ...,\n",
       "       [0., 0., 0., ..., 0., 0., 0.],\n",
       "       [0., 0., 0., ..., 0., 0., 0.],\n",
       "       [0., 0., 0., ..., 0., 0., 0.]])"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [],
   "source": [
    "y = y.astype(\"float\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([  0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,\n",
       "         0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,\n",
       "         0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,\n",
       "         0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,\n",
       "         0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,\n",
       "         0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,\n",
       "         0.,   0.,   0.,   0.,   0.,   0.,   0.,   8., 117., 254., 220.,\n",
       "        89.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,\n",
       "         0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,\n",
       "        13.,  95., 212., 253., 253., 253., 157.,   0.,   0.,   0.,   0.,\n",
       "         0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,\n",
       "         0.,   0.,   0.,   0.,  16.,  95., 209., 253., 253., 253., 245.,\n",
       "       125.,  18.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,\n",
       "         0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,  40.,  96., 206.,\n",
       "       253., 254., 253., 253., 198.,  64.,   0.,   0.,   0.,   0.,   0.,\n",
       "         0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,\n",
       "         0.,  44., 182., 240., 253., 253., 253., 254., 253., 198.,  24.,\n",
       "         0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,\n",
       "         0.,   0.,   0.,   0.,   0.,   0.,   0.,  15.,  60.,  60., 168.,\n",
       "       253., 253., 254., 200.,  23.,   0.,   0.,   0.,   0.,   0.,   0.,\n",
       "         0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,\n",
       "         0.,   0.,   0.,   0.,  70., 247., 253., 253., 245.,  21.,   0.,\n",
       "         0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,\n",
       "         0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,  75., 207.,\n",
       "       253., 253., 207.,  92.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,\n",
       "         0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,\n",
       "         0.,   0.,   0.,  79., 219., 253., 253., 253., 138.,   0.,   0.,\n",
       "         0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,\n",
       "         0.,   0.,   0.,   0.,   0.,   0.,   0.,   0., 105., 250., 253.,\n",
       "       253., 253.,  34.,   1.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,\n",
       "         0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,\n",
       "         0.,   0.,  95., 254., 254., 254., 254.,  94.,   0.,   0.,   0.,\n",
       "         0.,   0.,   3.,  13.,  13.,  13.,   8.,   0.,   0.,   0.,   0.,\n",
       "         0.,   0.,   0.,   0.,   0.,   0.,   0.,   0., 107., 253., 253.,\n",
       "       253., 204.,  15.,   0.,   0.,   0.,   0.,  21., 166., 253., 253.,\n",
       "       253., 212.,  25.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,\n",
       "         0.,   0.,  33., 217., 253., 253., 132.,  64.,   0.,   0.,  18.,\n",
       "        43., 157., 171., 253., 253., 253., 253., 253., 160.,   2.,   0.,\n",
       "         0.,   0.,   0.,   0.,   0.,   0.,   0.,   3., 166., 253., 253.,\n",
       "       242.,  49.,  17.,  49., 158., 210., 254., 253., 253., 253., 253.,\n",
       "       253., 253., 253., 253.,  11.,   0.,   0.,   0.,   0.,   0.,   0.,\n",
       "         0.,   0.,  10., 227., 253., 253., 207.,  15., 172., 253., 253.,\n",
       "       253., 254., 247., 201., 253., 210., 210., 253., 253., 175.,   4.,\n",
       "         0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,  10., 228., 253.,\n",
       "       253., 224.,  87., 242., 253., 253., 184.,  60.,  54.,   9.,  60.,\n",
       "        35., 182., 253., 253.,  52.,   0.,   0.,   0.,   0.,   0.,   0.,\n",
       "         0.,   0.,   0.,  13., 253., 253., 253., 253., 231., 253., 253.,\n",
       "       253.,  93.,  86.,  86.,  86., 109., 217., 253., 253., 134.,   5.,\n",
       "         0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,   2., 115.,\n",
       "       253., 253., 253., 253., 253., 253., 253., 253., 254., 253., 253.,\n",
       "       253., 253., 253., 134.,   5.,   0.,   0.,   0.,   0.,   0.,   0.,\n",
       "         0.,   0.,   0.,   0.,   0.,   3., 166., 253., 253., 253., 253.,\n",
       "       253., 253., 253., 254., 253., 253., 253., 175.,  52.,   5.,   0.,\n",
       "         0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,\n",
       "         0.,   7.,  35., 132., 225., 253., 253., 253., 195., 132., 132.,\n",
       "       132., 110.,   4.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,\n",
       "         0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,\n",
       "         0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,\n",
       "         0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,\n",
       "         0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,\n",
       "         0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,\n",
       "         0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,\n",
       "         0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,\n",
       "         0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,\n",
       "         0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,\n",
       "         0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,\n",
       "         0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,\n",
       "         0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,\n",
       "         0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,\n",
       "         0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,\n",
       "         0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,\n",
       "         0.,   0.,   0.])"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X[69999]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "6.0"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y[69999]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(70000, 784)"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(70000,)"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "import matplotlib\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAP8AAAD8CAYAAAC4nHJkAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAADl9JREFUeJzt3X+MXXWZx/HP0+m0hVK0VfuD2m27pHFpiJZlUnDZJZgCVmW3qEBoWFOTyvDT3WbZ7JK6G/ljSbqugs2qkKl0GRJAXbXSRCLiqEGz2jDFSitVqVKhdLaDW4Si9Md0nv1jTs3Qzvne23vuPed2nvcraebe85wfT276mXPvfO85X3N3AYhnQtUNAKgG4QeCIvxAUIQfCIrwA0ERfiAowg8ERfiBoAg/ENTEMg82ySb7FE0t85BAKAf1ex32Q1bPuoXCb2bLJa2X1CHpi+6+LrX+FE3VBbasyCEBJGzxvrrXbfhtv5l1SPq8pPdJWixppZktbnR/AMpV5DP/Ukm73P3X7n5Y0pckrWhOWwBarUj450p6YdTzPdmyNzCzbjPrN7P+IzpU4HAAmqlI+Mf6o8IJ1we7e4+7d7l7V6cmFzgcgGYqEv49kuaNev52SXuLtQOgLEXC/6SkRWa20MwmSbpW0ubmtAWg1Roe6nP3ITO7VdJjGhnq2+juP2taZwBaqtA4v7s/KunRJvUCoER8vRcIivADQRF+ICjCDwRF+IGgCD8QFOEHgiL8QFCEHwiK8ANBEX4gKMIPBEX4gaAIPxAU4QeCIvxAUIQfCIrwA0ERfiAowg8ERfiBoAg/EBThB4Ii/EBQhB8IivADQRF+ICjCDwRF+IGgCs3Sa2a7JR2QdFTSkLt3NaMptI8JSxYn679Yc1qy/uxlG3JrHZY+9/xh+HCy/u5Pr0nWz7r3qdza8MGDyW0jKBT+zHvc/bdN2A+AEvG2HwiqaPhd0rfNbKuZdTejIQDlKPq2/yJ332tmMyU9bmY/d/cnRq+Q/VLolqQpOr3g4QA0S6Ezv7vvzX4OStokaekY6/S4e5e7d3VqcpHDAWiihsNvZlPNbNqxx5Iul7SjWY0BaK0ib/tnSdpkZsf285C7f6spXQFoOXP30g52ps3wC2xZaceDZBPTv9/3/t0Jn9Te4IsfX5+snz+p46R7OubHh9L1Cwt+Srzi/dfl1oZ/urPYztvUFu/Tq77f6lmXoT4gKMIPBEX4gaAIPxAU4QeCIvxAUM24qg8VG7z5L3Jrv1tyJLntrg98rsbe00N579nx4WR9eMPM3Nq0n7+S3HZx7y+T9U/N7k/W33LPQG7tpfyXLAzO/EBQhB8IivADQRF+ICjCDwRF+IGgCD8QFOP8p4AX/iU9KP3Tm/4ztzZB6as7tx0eStb/afVNyfpp38u/PbYkyZ/LLQ2nt9TOS6enV6hx65j/mt+XW7t8+Y3JbSd968n0zscBzvxAUIQfCIrwA0ERfiAowg8ERfiBoAg/EBTj/G2gY3p6PHvNdd9I1lNj+QNH/5Dc9h9vTE9zPem76WvmW8lffz1Z/8LvFibrN785/zsGXtfNrcc3zvxAUIQfCIrwA0ERfiAowg8ERfiBoAg/EFTNcX4z2yjpCkmD7n5utmyGpC9LWiBpt6Rr3P3l1rU5vtn0NyXrq8/c0/C+L37ktmR90WNbGt53qw0fPJisP/DcBcn6zeflj/OjvjP//ZKWH7fsdkl97r5IUl/2HMAppGb43f0JSfuPW7xCUm/2uFfSlU3uC0CLNfqZf5a7D0hS9jN/TiYAbanl3+03s25J3ZI0Rae3+nAA6tTomX+fmc2RpOznYN6K7t7j7l3u3tWpyQ0eDkCzNRr+zZJWZY9XSXqkOe0AKEvN8JvZw5J+JOkdZrbHzFZLWifpMjN7VtJl2XMAp5Can/ndfWVOaVmTewnryJw3F9r+xcQ1++/Y8Epy21r3zsf4xTf8gKAIPxAU4QeCIvxAUIQfCIrwA0Fx6+428KurphTa/vIf50+jPf/p7YX2jfGLMz8QFOEHgiL8QFCEHwiK8ANBEX4gKMIPBMU4fwkmzj0rWb/nr+8rtP+On0wrtH27mnB6+rZvd/7ZppI6GZ848wNBEX4gKMIPBEX4gaAIPxAU4QeCIvxAUIzzl+D375qbrC877VCh/U9+2Qtt365sYvq/Z63X7f+GX8+tdb421FBP4wlnfiAowg8ERfiBoAg/EBThB4Ii/EBQhB8IquY4v5ltlHSFpEF3Pzdbdoek6yW9lK221t0fbVWTSJv14I7cWuQpuHtfeWdubcIPflJiJ+2pnjP//ZKWj7H8bndfkv0j+MAppmb43f0JSftL6AVAiYp85r/VzJ42s41mNr1pHQEoRaPhv0fS2ZKWSBqQ9Jm8Fc2s28z6zaz/iIp9hx1A8zQUfnff5+5H3X1Y0gZJSxPr9rh7l7t3dWpyo30CaLKGwm9mc0Y9/aCk/D83A2hL9Qz1PSzpEklvNbM9kj4p6RIzWyLJJe2WdEMLewTQAjXD7+4rx1hc7EbzQB1+c8u5Ndb4frL60L3vza3N1P+cfEPjDN/wA4Ii/EBQhB8IivADQRF+ICjCDwTFrbtLMKXv6WT9wQMzk/Xrpg02s522MXHh/GT98x+7t9D+z/rmi7k1btzNmR8Ii/ADQRF+ICjCDwRF+IGgCD8QFOEHgmKcvwR+KH37soM+qaRO2su+S89K1v9qSno0/pDXGK338Tl1ebNw5geCIvxAUIQfCIrwA0ERfiAowg8ERfiBoBjnHw/Onpdf2/ZMeX2MYeL8/N4+9PHvJretNY7/7v9Yk6zP3s3tuVM48wNBEX4gKMIPBEX4gaAIPxAU4QeCIvxAUDXH+c1snqQHJM2WNCypx93Xm9kMSV+WtEDSbknXuPvLrWt1/Pr3x/4mWV999ReS9V9d+6bc2sJtDbVUN5uY/i/0zCdm59Y2v+WR5LbfP3hasj57PeP4RdRz5h+SdJu7nyPpQkm3mNliSbdL6nP3RZL6sucAThE1w+/uA+7+VPb4gKSdkuZKWiGpN1utV9KVrWoSQPOd1Gd+M1sg6TxJWyTNcvcBaeQXhKT0nFMA2krd4TezMyR9TdIad3/1JLbrNrN+M+s/ovS97ACUp67wm1mnRoL/oLt/PVu8z8zmZPU5ksacTdLde9y9y927OjW5GT0DaIKa4Tczk3SfpJ3ufteo0mZJq7LHqySl/3QLoK3Uc0nvRZI+Imm7mR0bOForaZ2kr5jZaknPS7q6NS2Of9N3WHqFGq/sv33oodxa72cvTG479L/70juvYd+NS5P1XR/4XG5t++EjyW3vvOH6ZL1TW5N1pNUMv7v/UFLe/85lzW0HQFn4hh8QFOEHgiL8QFCEHwiK8ANBEX4gKG7d3QZmffO5ZH3bJ9K3sP7w1PwrqW//1wXJbc9Z15msP3tz4rbgkr668q5kXcqffvyqr6ZvvX32d35UY98ogjM/EBThB4Ii/EBQhB8IivADQRF+ICjCDwRl7l7awc60GX6BcRXwyTpy6fnJ+qb786+ZP8PSd0/aevhosv6u/GF6SdJEdSTrF2+/Krc27Yrnk9v6UPr7DTjRFu/Tq76/xg0iRnDmB4Ii/EBQhB8IivADQRF+ICjCDwRF+IGguJ7/FND5nfT96Zfe/w+5tf/+27uT254/qcZAfg2LNt2UrJ+zbk9ubYhx/Epx5geCIvxAUIQfCIrwA0ERfiAowg8ERfiBoGpez29m8yQ9IGm2pGFJPe6+3szukHS9pJeyVde6+6OpfXE9P9BaJ3M9fz1f8hmSdJu7P2Vm0yRtNbPHs9rd7v7pRhsFUJ2a4Xf3AUkD2eMDZrZT0txWNwagtU7qM7+ZLZB0nqQt2aJbzexpM9toZtNztuk2s34z6z+iQ4WaBdA8dYffzM6Q9DVJa9z9VUn3SDpb0hKNvDP4zFjbuXuPu3e5e1en0veTA1CeusJvZp0aCf6D7v51SXL3fe5+1N2HJW2QtLR1bQJotprhNzOTdJ+kne5+16jlc0at9kFJO5rfHoBWqeev/RdJ+oik7Wa2LVu2VtJKM1siySXtlnRDSzoE0BL1/LX/h5LGGjdMjukDaG98ww8IivADQRF+ICjCDwRF+IGgCD8QFOEHgiL8QFCEHwiK8ANBEX4gKMIPBEX4gaAIPxBUzVt3N/VgZi9J+s2oRW+V9NvSGjg57dpbu/Yl0VujmtnbfHd/Wz0rlhr+Ew5u1u/uXZU1kNCuvbVrXxK9Naqq3njbDwRF+IGgqg5/T8XHT2nX3tq1L4neGlVJb5V+5gdQnarP/AAqUkn4zWy5mf3CzHaZ2e1V9JDHzHab2XYz22Zm/RX3stHMBs1sx6hlM8zscTN7Nvs55jRpFfV2h5m9mL1228zs/RX1Ns/MvmdmO83sZ2b299nySl+7RF+VvG6lv+03sw5Jv5R0maQ9kp6UtNLdnym1kRxmtltSl7tXPiZsZhdLek3SA+5+brbsU5L2u/u67BfndHf/5zbp7Q5Jr1U9c3M2ocyc0TNLS7pS0kdV4WuX6OsaVfC6VXHmXyppl7v/2t0PS/qSpBUV9NH23P0JSfuPW7xCUm/2uFcj/3lKl9NbW3D3AXd/Knt8QNKxmaUrfe0SfVWiivDPlfTCqOd71F5Tfrukb5vZVjPrrrqZMczKpk0/Nn36zIr7OV7NmZvLdNzM0m3z2jUy43WzVRH+sWb/aachh4vc/c8lvU/SLdnbW9SnrpmbyzLGzNJtodEZr5utivDvkTRv1PO3S9pbQR9jcve92c9BSZvUfrMP7zs2SWr2c7Difv6onWZuHmtmabXBa9dOM15XEf4nJS0ys4VmNknStZI2V9DHCcxsavaHGJnZVEmXq/1mH94saVX2eJWkRyrs5Q3aZebmvJmlVfFr124zXlfyJZ9sKOOzkjokbXT3O0tvYgxm9qcaOdtLI5OYPlRlb2b2sKRLNHLV1z5Jn5T0DUlfkfQnkp6XdLW7l/6Ht5zeLtHIW9c/ztx87DN2yb39paQfSNouaThbvFYjn68re+0Sfa1UBa8b3/ADguIbfkBQhB8IivADQRF+ICjCDwRF+IGgCD8QFOEHgvp/czL6XIjMdhoAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "_ = X[1000]\n",
    "_image = _.reshape(28, 28)\n",
    "plt.imshow(_image);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.0"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y[1000]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercise: Locating the number 4 and plot the image"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "numpy.ndarray"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "type(y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([False, False,  True, ...,  True, False, False])"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y == 4"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(array([    2,     9,    20, ..., 69977, 69987, 69997]),)"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.where(y==4)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "6.0"
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y[24754]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAP8AAAD8CAYAAAC4nHJkAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAADgZJREFUeJzt3X+MXHW5x/HPw3ZpoVBo7W1tSksLNCpWaXUtRogXrdViiMVECL3G1B/cxVzwihq9wL25xT+ukquAKGhcYaUq4I8A0hj00jRGlEJlqb20WLDcssLatcUUoVr6Y7uPf+ypd2n3fGd25syc2T7vV9LMzHnOmfNk0s+emfmeOV9zdwGI55iyGwBQDsIPBEX4gaAIPxAU4QeCIvxAUIQfCIrwA0ERfiCocc3c2bE23idoYjN3CYSyV3/Vft9n1axbV/jNbKmkmyS1SbrV3a9LrT9BE3W2La5nlwAS1vvaqtet+W2/mbVJukXS+ZLOlLTczM6s9fkANFc9n/kXSXra3be5+35J35e0rJi2ADRaPeGfKem5YY/7smWvYGadZtZjZj0HtK+O3QEoUj3hH+lLhSN+H+zuXe7e4e4d7Rpfx+4AFKme8PdJmjXs8SmSttfXDoBmqSf8j0qaZ2ZzzexYSZdIWl1MWwAareahPncfMLMrJP2Phob6ut39icI6A9BQdY3zu/v9ku4vqBcATcTpvUBQhB8IivADQRF+ICjCDwRF+IGgmvp7frSevRcsStb//SvfTtav/tKlyfrUbz486p7QHBz5gaAIPxAU4QeCIvxAUIQfCIrwA0Ex1Bfcc+enr/K8+Lj0pdfGv3jExZswRnDkB4Ii/EBQhB8IivADQRF+ICjCDwRF+IGgGOc/2i16Q7J8wdkbkvXlzyxJ1ifdnd6eswBaF0d+ICjCDwRF+IGgCD8QFOEHgiL8QFCEHwiqrnF+M+uVtFvSQUkD7t5RRFMozvSbfp+s3zhjfbJ+1levSNZnHlg36p7QGoo4yecd7v6nAp4HQBPxth8Iqt7wu6QHzOwxM+ssoiEAzVHv2/5z3H27mU2TtMbMnnT3B4evkP1R6JSkCTq+zt0BKEpdR353357d7pR0r6QjJn5z9y5373D3jnaNr2d3AApUc/jNbKKZnXjovqR3S9pcVGMAGquet/3TJd1rZoee5053/1khXQFouJrD7+7bJJ1VYC+oUd/Vb8ut/WT215Lbzl/34WT91Ot/nazze/2xi6E+ICjCDwRF+IGgCD8QFOEHgiL8QFBcuvsosOeM/bm1voGXk9vO+fyBZH1wYKCmntD6OPIDQRF+ICjCDwRF+IGgCD8QFOEHgiL8QFCM848BbVNflazf/I/fy629Z93lyW3nbv7fmnrC2MeRHwiK8ANBEX4gKMIPBEX4gaAIPxAU4QeCYpx/DOj9+GuS9aXHrcmtfXobU6RhZBz5gaAIPxAU4QeCIvxAUIQfCIrwA0ERfiCoiuP8ZtYt6QJJO919frZsiqQfSJojqVfSxe7+QuPajG3vvL3Jev/BPbm10370YnLbwZo6wtGgmiP/7ZKWHrbsKklr3X2epLXZYwBjSMXwu/uDknYdtniZpFXZ/VWSLiy4LwANVutn/unu3i9J2e204loC0AwNP7ffzDoldUrSBHGeOdAqaj3y7zCzGZKU3e7MW9Hdu9y9w9072jW+xt0BKFqt4V8taUV2f4Wk+4ppB0CzVAy/md0l6WFJrzGzPjP7mKTrJC0xs62SlmSPAYwhFT/zu/vynNLigntBjomT0uP873wo/9r8czeWe13+cbNOya3936Wzk9tOfip9FsKkOx+pqScM4Qw/ICjCDwRF+IGgCD8QFOEHgiL8QFBcursFHDP/tcn6T9/clawv6f5cke28wrhTZibrfRedmqx//RM359beWuGEzzUvH5es3/DMJcm6Pcz04ykc+YGgCD8QFOEHgiL8QFCEHwiK8ANBEX4gKMb5W8CT/zIpWZ/R1rjLn+3417cl67dcmT9OL1Ueq0/53YH0T5X/Onhysn7yl/qS9RfPHXVLoXDkB4Ii/EBQhB8IivADQRF+ICjCDwRF+IGgGOdvAa/a0JZeYVm63PZyfu3Zlelx/HWXfjlZn3TMhGT9Zd+frC/Z9MHc2uRPJjeVP/uHZP24B9LHrrbXzcutHdyyNb3zADjyA0ERfiAowg8ERfiBoAg/EBThB4Ii/EBQFcf5zaxb0gWSdrr7/GzZtZL+WdLz2WrXuPv9jWryaDfliT3J+guDiYF8Se2782sLlv02uW294/iLr/5Usn7ydx/OrR1MblnZxm2vT9anvak9t3YS4/xVHflvl7R0hOU3uvuC7B/BB8aYiuF39wcl7WpCLwCaqJ7P/FeY2eNm1m1mkwvrCEBT1Br+b0g6XdICSf2Srs9b0cw6zazHzHoOaF+NuwNQtJrC7+473P2guw9K+pakRYl1u9y9w9072lXH1R4BFKqm8JvZjGEP3y9pczHtAGiWaob67pJ0nqSpZtYnaaWk88xsgSSX1Cvpsgb2CKABKobf3ZePsPi2BvQS1rht/cn68wctWX/fZb/Irf3n1E3JbRs5jl+vtknp+QymTXsxWX/+/BNzayfdUVNLRxXO8AOCIvxAUIQfCIrwA0ERfiAowg8ExaW7W8DBHTuT9ecGTkrW/7gvPSSWkrq0ttTYoTwbl/7vt/U/0j/ZffKsW5L1M799+ah7ioQjPxAU4QeCIvxAUIQfCIrwA0ERfiAowg8ExTj/GPDxhz6UrG991625tdf+4qPJbc/4wt5kfTBZreyY44/PrT31xTcmt936gfQ4/mP70xf/PuOWZ3JrA8ktY+DIDwRF+IGgCD8QFOEHgiL8QFCEHwiK8ANBMc4/Bhzbm55GO2VwMH3Zb9+aPxZejYF3vjlZ3/PZP+fWtr7x68ltH93nyfrKf/pYsq7+x9P14DjyA0ERfiAowg8ERfiBoAg/EBThB4Ii/EBQFcf5zWyWpO9IerWGft7d5e43mdkUST+QNEdSr6SL3f2FxrUa16kr1yXrb1k40izqQ558R/5v/SWp6zdzkvV7ti9M1m+dd1OyPrf9hNzaAU+P439k1SeS9dmPpF8XpFVz5B+Q9Bl3f52kt0q63MzOlHSVpLXuPk/S2uwxgDGiYvjdvd/dN2T3d0vaImmmpGWSVmWrrZJ0YaOaBFC8UX3mN7M5khZKWi9purv3S0N/ICRNK7o5AI1TdfjN7ARJd0u60t1fGsV2nWbWY2Y9B7Svlh4BNEBV4Tezdg0F/w53vydbvMPMZmT1GZJGnG3S3bvcvcPdO9o1voieARSgYvjNzCTdJmmLu98wrLRa0ors/gpJ9xXfHoBGMa8w3GJm50r6paRN+v8rOV+joc/9P5Q0W9Kzki5y912p55pkU/xsW1xvzxgFP2dBsr7w5o3J+hembUjWv/bn05L17u735tZe/cie5Lb2ULo3HGm9r9VLviv9O+5MxXF+d/+VpLwnI8nAGMUZfkBQhB8IivADQRF+ICjCDwRF+IGgKo7zF4lxfqCxRjPOz5EfCIrwA0ERfiAowg8ERfiBoAg/EBThB4Ii/EBQhB8IivADQRF+ICjCDwRF+IGgCD8QFOEHgiL8QFCEHwiK8ANBEX4gKMIPBEX4gaAIPxAU4QeCqhh+M5tlZj83sy1m9oSZfTJbfq2Z/cHMNmb/8idiB9ByxlWxzoCkz7j7BjM7UdJjZrYmq93o7l9uXHsAGqVi+N29X1J/dn+3mW2RNLPRjQForFF95jezOZIWSlqfLbrCzB43s24zm5yzTaeZ9ZhZzwHtq6tZAMWpOvxmdoKkuyVd6e4vSfqGpNMlLdDQO4PrR9rO3bvcvcPdO9o1voCWARShqvCbWbuGgn+Hu98jSe6+w90PuvugpG9JWtS4NgEUrZpv+03SbZK2uPsNw5bPGLba+yVtLr49AI1Szbf950j6kKRNZrYxW3aNpOVmtkCSS+qVdFlDOgTQENV82/8rSSPN931/8e0AaBbO8AOCIvxAUIQfCIrwA0ERfiAowg8ERfiBoAg/EBThB4Ii/EBQhB8IivADQRF+ICjCDwRl7t68nZk9L+n3wxZNlfSnpjUwOq3aW6v2JdFbrYrs7VR3/4dqVmxq+I/YuVmPu3eU1kBCq/bWqn1J9FarsnrjbT8QFOEHgio7/F0l7z+lVXtr1b4keqtVKb2V+pkfQHnKPvIDKEkp4TezpWb2lJk9bWZXldFDHjPrNbNN2czDPSX30m1mO81s87BlU8xsjZltzW5HnCatpN5aYubmxMzSpb52rTbjddPf9ptZm6TfSVoiqU/So5KWu/tvm9pIDjPrldTh7qWPCZvZ2yX9RdJ33H1+tuy/Je1y9+uyP5yT3f3fWqS3ayX9peyZm7MJZWYMn1la0oWSPqwSX7tEXxerhNetjCP/IklPu/s2d98v6fuSlpXQR8tz9wcl7Tps8TJJq7L7qzT0n6fpcnprCe7e7+4bsvu7JR2aWbrU1y7RVynKCP9MSc8Ne9yn1pry2yU9YGaPmVln2c2MYHo2bfqh6dOnldzP4SrO3NxMh80s3TKvXS0zXhetjPCPNPtPKw05nOPub5J0vqTLs7e3qE5VMzc3ywgzS7eEWme8LloZ4e+TNGvY41MkbS+hjxG5+/bsdqeke9V6sw/vODRJana7s+R+/q6VZm4eaWZptcBr10ozXpcR/kclzTOzuWZ2rKRLJK0uoY8jmNnE7IsYmdlESe9W680+vFrSiuz+Ckn3ldjLK7TKzM15M0ur5Neu1Wa8LuUkn2wo4yuS2iR1u/t/Nb2JEZjZaRo62ktDk5jeWWZvZnaXpPM09KuvHZJWSvqxpB9Kmi3pWUkXuXvTv3jL6e08Db11/fvMzYc+Yze5t3Ml/VLSJkmD2eJrNPT5urTXLtHXcpXwunGGHxAUZ/gBQRF+ICjCDwRF+IGgCD8QFOEHgiL8QFCEHwjqb0nm9MTHAbeOAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "_ = X[24754]\n",
    "_image = _.reshape(28, 28)\n",
    "plt.imshow(_image);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Splitting the train and test sets"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [],
   "source": [
    "num_split = 60000\n",
    "\n",
    "X_train, X_test, y_train, y_test = X[:num_split], X[num_split:], y[:num_split], y[num_split:]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Tips**: Typically we shuffle the training set. This ensures the training set is randomised and your data distribution is consistent. However, shuffling is a bad idea for time series data."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Shuffling the dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[Alternative Method](http://scikit-learn.org/stable/modules/generated/sklearn.model_selection.ShuffleSplit.html)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [],
   "source": [
    "shuffle_index = np.random.permutation(num_split)\n",
    "X_train, y_train = X_train[shuffle_index], y_train[shuffle_index]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Training a Binary Classifier"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To simplify our problem, we will make this an exercise of \"zero\" or \"non-zero\", making it a two-class problem.\n",
    "\n",
    "We need to first convert our target to 0 or non zero."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_train_0 = (y_train == 0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([False, False, False, ..., False, False, False])"
      ]
     },
     "execution_count": 49,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_train_0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_test_0 = (y_test == 0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([False, False, False, ..., False, False, False])"
      ]
     },
     "execution_count": 51,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_test_0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "At this point we can pick any classifier and train it. This is the iterative part of choosing and testing all the classifiers and tuning the hyper parameters"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# SGDClassifier\n",
    "\n",
    "# Training"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "SGDClassifier(alpha=0.0001, average=False, class_weight=None,\n",
       "              early_stopping=False, epsilon=0.1, eta0=0.0, fit_intercept=True,\n",
       "              l1_ratio=0.15, learning_rate='optimal', loss='hinge',\n",
       "              max_iter=1000, n_iter_no_change=5, n_jobs=None, penalty='l2',\n",
       "              power_t=0.5, random_state=0, shuffle=True, tol=0.001,\n",
       "              validation_fraction=0.1, verbose=0, warm_start=False)"
      ]
     },
     "execution_count": 52,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.linear_model import SGDClassifier\n",
    "\n",
    "clf = SGDClassifier(random_state = 0)\n",
    "clf.fit(X_train, y_train_0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Prediction"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ True])"
      ]
     },
     "execution_count": 53,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "clf.predict(X[1000].reshape(1, -1))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Performance Measures\n",
    "\n",
    "# Measuring Accuracy Using Cross-Validation\n",
    "\n",
    "## StratifiedKFold"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's try with the `StratifiedKFold` stratified sampling to create multiple folds. At each iteration, the classifier was cloned and trained using the training folds and makes predictions on the test fold. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "StratifiedKFold utilised the Stratified sampling concept\n",
    "\n",
    "* The population is divided into homogeneous subgroups called strata\n",
    "* The right number of instances is sampled from each stratum \n",
    "* To guarantee that the test set is representative of the population"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import StratifiedKFold\n",
    "from sklearn.base import clone\n",
    "clf = SGDClassifier(random_state=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [],
   "source": [
    "skfolds = StratifiedKFold(n_splits=3, random_state=100)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.9889\n",
      "0.9876\n",
      "0.9875\n"
     ]
    }
   ],
   "source": [
    "for train_index, test_index in skfolds.split(X_train, y_train_0):\n",
    "    clone_clf = clone(clf)\n",
    "    X_train_fold = X_train[train_index]\n",
    "    y_train_folds = (y_train_0[train_index])\n",
    "    X_test_fold = X_train[test_index]\n",
    "    y_test_fold = (y_train_0[test_index])\n",
    "    \n",
    "    clone_clf.fit(X_train_fold, y_train_folds)\n",
    "    y_pred = clone_clf.predict(X_test_fold)\n",
    "    n_correct = sum(y_pred == y_test_fold)\n",
    "    print(\"{0:.4f}\".format(n_correct / len(y_pred)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### `cross_val_score` using K-fold Cross-Validation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "K-fold cross-validation splits the training set into K-folds and then make predictions and evaluate them on each fold using a model trained on the remaning folds."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import cross_val_score"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0.98885056, 0.9876    , 0.98749937])"
      ]
     },
     "execution_count": 58,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cross_val_score(clf, X_train, y_train_0, cv=3, scoring='accuracy')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Exercise:\n",
    "\n",
    "What if you would like to perform 10-fold CV test? How would you do that"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0.98166972, 0.98733544, 0.98866856, 0.96333333, 0.98966667,\n",
       "       0.9895    , 0.98833333, 0.98749792, 0.98466411, 0.98566428])"
      ]
     },
     "execution_count": 59,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cross_val_score(clf, X_train, y_train_0, cv=10, scoring='accuracy')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Danger of Blindly Applying Evaluator As a Performance Measure"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's check against a dumb classifier"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.9012833333333333"
      ]
     },
     "execution_count": 60,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "1 - sum(y_train_0) / len(y_train_0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A simple check shows that 90.1% of the images are not zero. Any time you guess the image is not zero, you will be right 90.13% of the time. \n",
    "\n",
    "Bare this in mind when you are dealing with **skewed datasets**. Because of this, accuracy is generally not the preferred performance measure for classifiers."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Confusion Matrix"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import cross_val_predict"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_train_pred = cross_val_predict(clf, X_train, y_train_0, cv=3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.metrics import confusion_matrix"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[53645,   432],\n",
       "       [  289,  5634]])"
      ]
     },
     "execution_count": 64,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "confusion_matrix(y_train_0, y_train_pred)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Each row: actual class\n",
    "\n",
    "Each column: predicted class\n",
    "\n",
    "First row: Non-zero images, the negative class:\n",
    "* 53360 were correctly classified as non-zeros. **True negatives**. \n",
    "* Remaining 717 were wrongly classified as 0s. **False positive**\n",
    "\n",
    "\n",
    "Second row: The images of zeros, the positive class:\n",
    "* 395 were incorrectly classified as 0s. **False negatives**\n",
    "* 5528 were correctly classified as 0s. **True positives**\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![](img/confusion_matrix.jpg)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Precision\n",
    "\n",
    "**Precision** measures the accuracy of positive predictions. Also called the `precision` of the classifier\n",
    "\n",
    "$$\\textrm{precision} = \\frac{\\textrm{True Positives}}{\\textrm{True Positives} + \\textrm{False Positives}}$$\n",
    "\n",
    "![](img/precision.jpg)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.metrics import precision_score, recall_score"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note the result here may vary from the video as the results from the confusion matrix are different each time you run it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.9287833827893175"
      ]
     },
     "execution_count": 66,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "precision_score(y_train_0, y_train_pred) # 5618 / (574 + 5618)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.9072997416020672"
      ]
     },
     "execution_count": 67,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "5618 / (574 + 5618)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Recall\n",
    "\n",
    "`Precision` is typically used with `recall` (`Sensitivity` or `True Positive Rate`). The ratio of positive instances that are correctly detected by the classifier.\n",
    "\n",
    "$$\\textrm{recall} = \\frac{\\textrm{True Positives}}{\\textrm{True Positives} + \\textrm{False Negatives}}$$\n",
    "\n",
    "![](img\\recall.jpg)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note the result here may vary from the video as the results from the confusion matrix are different each time you run it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.9512071585345264"
      ]
     },
     "execution_count": 68,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "recall_score(y_train_0, y_train_pred) # 5618 / (305 + 5618)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.9485058247509708"
      ]
     },
     "execution_count": 69,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "5618 / (305 + 5618)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## F1 Score\n",
    "\n",
    "$F_1$ score is the harmonic mean of precision and recall. Regular mean gives equal weight to all values. Harmonic mean gives more weight to low values.\n",
    "\n",
    "\n",
    "$$F_1=\\frac{2}{\\frac{1}{\\textrm{precision}}+\\frac{1}{\\textrm{recall}}}=2\\times \\frac{\\textrm{precision}\\times \\textrm{recall}}{\\textrm{precision}+ \\textrm{recall}}=\\frac{TP}{TP+\\frac{FN+FP}{2}}$$\n",
    "\n",
    "The $F_1$ score favours classifiers that have similar precision and recall.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.metrics import f1_score"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note the result here may vary from the video as the results from the confusion matrix are different each time you run it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.939861539744766"
      ]
     },
     "execution_count": 71,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "f1_score(y_train_0, y_train_pred)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Precision / Recall Tradeoff\n",
    "\n",
    "Increasing precision reduced recall and vice versa"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![](img/precision-recall.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Our classifier is designed to pick up zeros.\n",
    "\n",
    "12 observations\n",
    "\n",
    "***\n",
    "\n",
    "**Central Arrow**\n",
    "\n",
    "Suppose the decision threshold is positioned at the central arrow: \n",
    "* We get 4 true positives (We have 4 zeros to the right of the central arrow)\n",
    "* 1 false positive which is actually seven.\n",
    "\n",
    "At this threshold, the **precision accuracy** is $\\frac{4}{5}=80\\%$\n",
    "\n",
    "However, out of the 6 zeros, the classifier only picked up 4. The **recall accuracy** is $\\frac{4}{6}=67\\%$\n",
    "\n",
    "***\n",
    "\n",
    "**Right Arrow**\n",
    "\n",
    "* We get 3 true positives\n",
    "* 0 false positive\n",
    "\n",
    "At this threshold, the **precision accuracy** is $\\frac{3}{3}=100\\%$\n",
    "However, out of the 6 zeros, the classifier only picked up 3. The **recall accuracy** is $\\frac{3}{6}=50\\%$\n",
    "\n",
    "***\n",
    "\n",
    "**Left Arrow**\n",
    "\n",
    "* We get 6 true positives\n",
    "* 2 false positive\n",
    "\n",
    "At this threshold, the **precision accuracy** is $\\frac{6}{8}=75\\%$\n",
    "Out of the 6 zeros, the classifier picked up all 6. The **recall accuracy** is $\\frac{6}{6}=100\\%$\n",
    "\n",
    "***\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "SGDClassifier(alpha=0.0001, average=False, class_weight=None,\n",
       "              early_stopping=False, epsilon=0.1, eta0=0.0, fit_intercept=True,\n",
       "              l1_ratio=0.15, learning_rate='optimal', loss='hinge',\n",
       "              max_iter=1000, n_iter_no_change=5, n_jobs=None, penalty='l2',\n",
       "              power_t=0.5, random_state=0, shuffle=True, tol=0.001,\n",
       "              validation_fraction=0.1, verbose=0, warm_start=False)"
      ]
     },
     "execution_count": 72,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "clf = SGDClassifier(random_state=0)\n",
    "clf.fit(X_train, y_train_0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.0"
      ]
     },
     "execution_count": 73,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y[1000]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([6622.3439475])"
      ]
     },
     "execution_count": 74,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_scores = clf.decision_function(X[1000].reshape(1, -1))\n",
    "y_scores"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {},
   "outputs": [],
   "source": [
    "threshold = 0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_some_digits_pred = (y_scores > threshold)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ True])"
      ]
     },
     "execution_count": 77,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_some_digits_pred"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([False])"
      ]
     },
     "execution_count": 78,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "threshold = 40000\n",
    "y_some_digits_pred = (y_scores > threshold)\n",
    "y_some_digits_pred"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_scores = cross_val_predict(clf, X_train, y_train_0, cv=3, method='decision_function')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAs8AAAHVCAYAAAAO1xbXAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAG+RJREFUeJzt3W2sZVd9H+DfHw84VZLGNh5T1y+9JplUmA8xZGQcoVYUJ7ax2xpUkMyHYBFLE7W2lEiplCF8gEKtDm0JKioxcooVE1GMS0IZMW7NxIGiSDX2mBjjwbi+mCkebNkmNg4RrVvT1Q9nTTiM78uaO/ee+/Y80tHZZ+21z1576dxzf3fdtfeu1loAAIDlvWS9GwAAAJuF8AwAAIOEZwAAGCQ8AwDAIOEZAAAGCc8AADBIeAYAgEHCMwAADBKeAQBg0I71bsBSzjzzzDY3N7fezQAAYIu77777vtta27lcvQ0dnufm5nLo0KH1bgYAAFtcVf3PkXqmbQAAwCDhGQAABgnPAAAwSHgGAIBBwjMAAAwSngEAYJDwDAAAg4RnAAAYJDwDAMAg4RkAAAYJzwAAMEh4BgCAQcIzAAAMEp4BAGCQ8AwAAIOEZwAAGCQ8AwDAIOEZAAAGCc8AADBox3o3AIDta27vgQXLj+y7asYtARhj5BkAAAYJzwAAMEh4BgCAQcIzAAAMEp4BAGCQq20AsOG4CgewURl5BgCAQcIzAAAMEp4BAGCQ8AwAAIOEZwAAGCQ8AwDAIJeqA2DTcAk7YL0ZeQYAgEHCMwAADBKeAQBgkPAMAACDhGcAABgkPAMAwKBlw3NV/URV3VNVX62qw1X1L3r5BVX15ap6pKo+VVUv6+Wn9tfzff3c1Hu9q5c/XFWXr9VBAQDAWhgZeX4+yRtba7+Q5KIkV1TVJUk+kORDrbVdSZ5Ncl2vf12SZ1trP5fkQ71equrCJNckeXWSK5L8XlWdspoHAwAAa2nZ8Nwm/qq/fGl/tCRvTPLpXn5rkjf35av76/T1l1ZV9fLbWmvPt9a+lWQ+ycWrchQAADADQ3cY7CPE9yX5uSQfSfLNJN9rrb3QqxxNck5fPifJY0nSWnuhqp5L8vJefvfU205vM72vPUn2JMn5559/gocDwHbkzoPArAydMNha+2Fr7aIk52YyWvyqhar151pk3WLlx+/r5tba7tba7p07d440DwAAZuKErrbRWvteki8muSTJaVV1bOT63CSP9+WjSc5Lkr7+Z5I8M12+wDYAALDhjVxtY2dVndaX/0aSX07yUJIvJHlrr3Ztks/25f39dfr6P22ttV5+Tb8axwVJdiW5Z7UOBAAA1trInOezk9za5z2/JMntrbXPVdXXk9xWVf8yyZ8n+Viv/7Ekf1hV85mMOF+TJK21w1V1e5KvJ3khyfWttR+u7uEAAMDaWTY8t9YeSPKaBcofzQJXy2it/e8kb1vkvW5McuOJNxMAANafOwwCAMAg4RkAAAYNXecZAE7GYtdhBthsjDwDAMAg4RkAAAYJzwAAMEh4BgCAQcIzAAAMEp4BAGCQ8AwAAIOEZwAAGCQ8AwDAIOEZAAAGuT03AKvGbbiBrc7IMwAADBKeAQBgkPAMAACDhGcAABgkPAMAwCDhGQAABgnPAAAwSHgGAIBBwjMAAAwSngEAYJDwDAAAg4RnAAAYJDwDAMAg4RkAAAYJzwAAMEh4BgCAQcIzAAAMEp4BAGDQjvVuAACslbm9BxYsP7Lvqhm3BNgqjDwDAMAg4RkAAAYJzwAAMMicZwBO2GJziQG2OiPPAAAwSHgGAIBBwjMAAAwSngEAYJDwDAAAg4RnAAAYJDwDAMAg4RkAAAYJzwAAMEh4BgCAQcIzAAAMEp4BAGCQ8AwAAIOEZwAAGCQ8AwDAIOEZAAAGCc8AADBIeAYAgEHCMwAADFo2PFfVeVX1hap6qKoOV9Vv9PL3VtV3qur+/rhyapt3VdV8VT1cVZdPlV/Ry+arau/aHBIAAKyNHQN1XkjyW621r1TVTye5r6oO9nUfaq392+nKVXVhkmuSvDrJ307yJ1X18331R5L8SpKjSe6tqv2tta+vxoEAAMBaWzY8t9aeSPJEX/5+VT2U5JwlNrk6yW2tteeTfKuq5pNc3NfNt9YeTZKquq3XFZ4BANgURkae/1pVzSV5TZIvJ3l9khuq6h1JDmUyOv1sJsH67qnNjuZHYfux48pft8A+9iTZkyTnn3/+iTQPgFU2t/fAejcBYEMZPmGwqn4qyR8l+c3W2l8muSnJzya5KJOR6Q8eq7rA5m2J8h8vaO3m1tru1trunTt3jjYPAADW3NDIc1W9NJPg/InW2h8nSWvtyan1v5/kc/3l0STnTW1+bpLH+/Ji5QAAsOGNXG2jknwsyUOttd+dKj97qtpbkjzYl/cnuaaqTq2qC5LsSnJPknuT7KqqC6rqZZmcVLh/dQ4DAADW3sjI8+uT/GqSr1XV/b3sd5K8vaouymTqxZEkv54krbXDVXV7JicCvpDk+tbaD5Okqm5IcmeSU5Lc0lo7vIrHAgAAa2rkaht/loXnK9+xxDY3JrlxgfI7ltoOAAA2MncYBACAQcIzAAAMEp4BAGDQCd0kBQC2gsVu/nJk31Uzbgmw2Rh5BgCAQcIzAAAMEp4BAGCQ8AwAAIOEZwAAGCQ8AwDAIOEZAAAGCc8AADBIeAYAgEHCMwAADBKeAQBgkPAMAACDhGcAABgkPAMAwCDhGQAABgnPAAAwSHgGAIBBwjMAAAwSngEAYJDwDAAAg4RnAAAYJDwDAMAg4RkAAAbtWO8GALC+5vYeWO8mAGwaRp4BAGCQ8AwAAIOEZwAAGCQ8AwDAIOEZAAAGCc8AADBIeAYAgEHCMwAADBKeAQBgkPAMAACDhGcAABgkPAMAwCDhGQAABgnPAAAwSHgGAIBBwjMAAAwSngEAYJDwDAAAg4RnAAAYJDwDAMAg4RkAAAYJzwAAMEh4BgCAQTvWuwEAzMbc3gPr3YQNb7E+OrLvqhm3BNiojDwDAMAg4RkAAAYJzwAAMEh4BgCAQcuG56o6r6q+UFUPVdXhqvqNXn5GVR2sqkf68+m9vKrqw1U1X1UPVNVrp97r2l7/kaq6du0OCwAAVt/IyPMLSX6rtfaqJJckub6qLkyyN8ldrbVdSe7qr5PkTUl29ceeJDclk7Cd5D1JXpfk4iTvORa4AQBgM1g2PLfWnmitfaUvfz/JQ0nOSXJ1klt7tVuTvLkvX53k423i7iSnVdXZSS5PcrC19kxr7dkkB5NcsapHAwAAa+iE5jxX1VyS1yT5cpJXtNaeSCYBO8lZvdo5SR6b2uxoL1us/Ph97KmqQ1V16Omnnz6R5gEAwJoaDs9V9VNJ/ijJb7bW/nKpqguUtSXKf7ygtZtba7tba7t37tw52jwAAFhzQ+G5ql6aSXD+RGvtj3vxk306RvrzU738aJLzpjY/N8njS5QDAMCmMHK1jUrysSQPtdZ+d2rV/iTHrphxbZLPTpW/o19145Ikz/VpHXcmuayqTu8nCl7WywAAYFPYMVDn9Ul+NcnXqur+XvY7SfYlub2qrkvy7SRv6+vuSHJlkvkkP0jyziRprT1TVe9Pcm+v977W2jOrchQAADADy4bn1tqfZeH5ykly6QL1W5LrF3mvW5LcciINBACAjcIdBgEAYJDwDAAAg4RnAAAYJDwDAMAg4RkAAAYJzwAAMGjkOs8AbCJzew+sdxMAtiwjzwAAMEh4BgCAQcIzAAAMEp4BAGCQ8AwAAIOEZwAAGORSdQCwjKUu/3dk31UzbAmw3ow8AwDAIOEZAAAGCc8AADBIeAYAgEHCMwAADBKeAQBgkPAMAACDhGcAABgkPAMAwCDhGQAABgnPAAAwSHgGAIBBwjMAAAwSngEAYJDwDAAAg4RnAAAYJDwDAMAg4RkAAAYJzwAAMGjHejcAgJWZ23tgvZsAsO0YeQYAgEHCMwAADBKeAQBgkPAMAACDhGcAABgkPAMAwCDhGQAABgnPAAAwSHgGAIBBwjMAAAwSngEAYJDwDAAAg4RnAAAYJDwDAMAg4RkAAAYJzwAAMEh4BgCAQcIzAAAMEp4BAGCQ8AwAAIOEZwAAGCQ8AwDAoGXDc1XdUlVPVdWDU2XvrarvVNX9/XHl1Lp3VdV8VT1cVZdPlV/Ry+arau/qHwoAAKytkZHnP0hyxQLlH2qtXdQfdyRJVV2Y5Jokr+7b/F5VnVJVpyT5SJI3Jbkwydt7XQAA2DR2LFehtfalqpobfL+rk9zWWns+ybeqaj7JxX3dfGvt0SSpqtt63a+fcIsBAGCdnMyc5xuq6oE+reP0XnZOksem6hztZYuVv0hV7amqQ1V16Omnnz6J5gEAwOpaaXi+KcnPJrkoyRNJPtjLa4G6bYnyFxe2dnNrbXdrbffOnTtX2DwAAFh9y07bWEhr7cljy1X1+0k+118eTXLeVNVzkzzelxcrBwCATWFFI89VdfbUy7ckOXYljv1JrqmqU6vqgiS7ktyT5N4ku6rqgqp6WSYnFe5febMBAGD2lh15rqpPJnlDkjOr6miS9yR5Q1VdlMnUiyNJfj1JWmuHq+r2TE4EfCHJ9a21H/b3uSHJnUlOSXJLa+3wqh8NAACsoZGrbbx9geKPLVH/xiQ3LlB+R5I7Tqh1AACwgbjDIAAADBKeAQBgkPAMAACDhGcAABgkPAMAwCDhGQAABgnPAAAwSHgGAIBBy94kBYD1Nbf3wHo3AYDOyDMAAAwSngEAYJBpGwBwEhabVnNk31UzbgkwC0aeAQBgkPAMAACDhGcAABgkPAMAwCDhGQAABgnPAAAwSHgGAIBBwjMAAAwSngEAYJDwDAAAg4RnAAAYJDwDAMAg4RkAAAYJzwAAMEh4BgCAQcIzAAAMEp4BAGCQ8AwAAIOEZwAAGCQ8AwDAIOEZAAAGCc8AADBIeAYAgEHCMwAADBKeAQBgkPAMAACDhGcAABi0Y70bAMDE3N4D690EAJZh5BkAAAYJzwAAMEh4BgCAQcIzAAAMEp4BAGCQ8AwAAIOEZwAAGCQ8AwDAIOEZAAAGCc8AADBIeAYAgEHCMwAADBKeAQBgkPAMAACDdqx3AwC2m7m9B9a7CQCskJFnAAAYtGx4rqpbquqpqnpwquyMqjpYVY/059N7eVXVh6tqvqoeqKrXTm1zba//SFVduzaHAwAAa2dk5PkPklxxXNneJHe11nYluau/TpI3JdnVH3uS3JRMwnaS9yR5XZKLk7znWOAGAIDNYtnw3Fr7UpJnjiu+OsmtffnWJG+eKv94m7g7yWlVdXaSy5McbK0901p7NsnBvDiQAwDAhrbSOc+vaK09kST9+axefk6Sx6bqHe1li5UDAMCmsdonDNYCZW2J8he/QdWeqjpUVYeefvrpVW0cAACcjJWG5yf7dIz056d6+dEk503VOzfJ40uUv0hr7ebW2u7W2u6dO3eusHkAALD6Vhqe9yc5dsWMa5N8dqr8Hf2qG5ckea5P67gzyWVVdXo/UfCyXgYAAJvGsjdJqapPJnlDkjOr6mgmV83Yl+T2qrouybeTvK1XvyPJlUnmk/wgyTuTpLX2TFW9P8m9vd77WmvHn4QIAAAb2rLhubX29kVWXbpA3Zbk+kXe55Ykt5xQ6wAAYANxh0EAABgkPAMAwCDhGQAABi075xkAOHFzew8sWH5k31Uzbgmwmow8AwDAIOEZAAAGCc8AADBIeAYAgEHCMwAADBKeAQBgkPAMAACDhGcAABgkPAMAwCDhGQAABgnPAAAwSHgGAIBBwjMAAAzasd4NANiq5vYeWO8mALDKjDwDAMAg4RkAAAYJzwAAMEh4BgCAQcIzAAAMEp4BAGCQ8AwAAINc5xngJLmeM8D2YeQZAAAGCc8AADBIeAYAgEHCMwAADBKeAQBgkPAMAACDXKoOAGZosUsbHtl31YxbAqyEkWcAABgkPAMAwCDhGQAABgnPAAAwSHgGAIBBwjMAAAwSngEAYJDwDAAAg4RnAAAYJDwDAMAg4RkAAAYJzwAAMGjHejcAYLOY23tgvZvAFrbY5+vIvqtm3BJgKUaeAQBgkPAMAACDhGcAABgkPAMAwCDhGQAABgnPAAAwSHgGAIBBwjMAAAwSngEAYJDwDAAAg04qPFfVkar6WlXdX1WHetkZVXWwqh7pz6f38qqqD1fVfFU9UFWvXY0DAACAWdmxCu/xD1pr3516vTfJXa21fVW1t7/+7SRvSrKrP16X5Kb+DLChzO09sN5NgG1hsZ+1I/uumnFLYNxaTNu4OsmtffnWJG+eKv94m7g7yWlVdfYa7B8AANbEyYbnluTzVXVfVe3pZa9orT2RJP35rF5+TpLHprY92ssAAGBTONlpG69vrT1eVWclOVhV31iibi1Q1l5UaRLC9yTJ+eeff5LNAwCA1XNSI8+ttcf781NJPpPk4iRPHpuO0Z+f6tWPJjlvavNzkzy+wHve3Frb3VrbvXPnzpNpHgAArKoVjzxX1U8meUlr7ft9+bIk70uyP8m1Sfb158/2TfYnuaGqbsvkRMHnjk3vAAA4xomEbGQnM23jFUk+U1XH3uc/ttb+a1Xdm+T2qrouybeTvK3XvyPJlUnmk/wgyTtPYt8AADBzKw7PrbVHk/zCAuV/keTSBcpbkutXuj8AAFhvq3GdZ4BNx7WcYfX5uWI7EJ4BgE3BXGg2grW4SQoAAGxJRp4BgE3NiDSzZOQZAAAGGXkGgA3MqCpsLEaeAQBgkJFnAGCYy9Gx3Rl5BgCAQcIzAAAMMm0D2NL8ixm2LydbshaMPAMAwCDhGQAABgnPAAAwyJxnANiEzOeF9WHkGQAABhl5BgBexJVqYGFGngEAYJCRZwDYxrbjCLP54pwMI88AADBIeAYAgEHCMwAADBKeAQBgkBMGgS1hO570BMDsGXkGAIBBRp6BTcUIMwDrSXgGgC3ENYxhbQnPAADxhwdjhGcA2AZMeYLVITwDG5Jf9ABsRK62AQAAg4RnAAAYJDwDAMAgc54BAJbgKhxME56BdeXEQAA2E9M2AABgkPAMAACDhGcAABhkzjMAwAo4kXB7MvIMAACDhGcAABhk2gYwEy5JB8BWYOQZAAAGCc8AADDItA1gVZmeAWx3S30PuhLH5ic8A0sShgHgR4RnAIAZcW3ozc+cZwAAGCQ8AwDAIOEZAAAGmfMMW9BKzvR2YiAALE94hm1GSAaAlROeAQDWmatwbB7CM2wCvlQBYGMQnmETMwUDAGZLeAYA2KD853HjEZ5hAzGSDMAIoXr9zDw8V9UVSf5dklOS/IfW2r5ZtwFWm9ALANvDTMNzVZ2S5CNJfiXJ0ST3VtX+1trXZ9kONpYT/et5JX9tr9Y+AGAjMyK99qq1NrudVf1Skve21i7vr9+VJK21f7VQ/d27d7dDhw7NrH3bxYkGwxP9gRM8AWBr2sohvKrua63tXq7erKdtnJPksanXR5O8brpCVe1Jsqe//KuqenhGbdvozkzy3fXYcX1gPfa6qtat77YI/bdy+m7l9N3J0X8rp++WsEwm2Ox993dGKs06PNcCZT829N1auznJzbNpzuZRVYdG/hrixfTdydF/K6fvVk7fnRz9t3L6buW2S9+9ZMb7O5rkvKnX5yZ5fMZtAACAFZl1eL43ya6quqCqXpbkmiT7Z9wGAABYkZlO22itvVBVNyS5M5NL1d3SWjs8yzZsYqayrJy+Ozn6b+X03crpu5Oj/1ZO363ctui7mV5tAwAANrNZT9sAAIBNS3gGAIBBwvMMVdW/qapvVNUDVfWZqjqtl89V1f+qqvv746NT2/xiVX2tquar6sNVVb38jKo6WFWP9OfTe3n1evN9P6+deq9re/1HquraWR//yVqs//q6d/VjfriqLp8qv6KXzVfV3qnyC6rqy70vPtVPYE1Vndpfz/f1c8vtYzOoqrdV1eGq+n9VtXuq3GdvGYv1XV/nc3cCquq9VfWdqc/blVPr1rwvt4PF+ms7qqoj/Tvs/qo61MtW7ftrse/Izaiqbqmqp6rqwamyNe+rxfax4bXWPGb0SHJZkh19+QNJPtCX55I8uMg29yT5pUyukf1fkrypl//rJHv78t6p97qy16sklyT5ci8/I8mj/fn0vnz6evfJKvXfhUm+muTUJBck+WYmJ6Se0pdfmeRlvc6FfZvbk1zTlz+a5J/25X+W5KN9+Zokn1pqH+vdJyfQd69K8neTfDHJ7qlyn72V953P3Yn35XuT/PMFyte8L7fDY6n+2o6PJEeSnHlc2ap9fy32HbkZH0n+fpLXZur3wSz6arF9bPSHkecZaq19vrX2Qn95dybXuV5UVZ2d5G+21v57m3yyPp7kzX311Ulu7cu3Hlf+8TZxd5LT+vtcnuRga+2Z1tqzSQ4muWK1jm0Wlui/q5Pc1lp7vrX2rSTzSS7uj/nW2qOttf+T5LYkV/e/eN+Y5NN9++P771i/fjrJpb3+YvvYFFprD7XWhu/W6bP3I0v0nc/d6plFX24HC/bXOrdpo1mV769lviM3ndbal5I8c1zxLPpqsX1saMLz+vm1TP76OuaCqvrzqvpvVfX3etk5mdxY5pijvSxJXtFaeyJJ+vNZU9scfwv0c5Yo36ym++9Ej/nlSb43FcSn++Kvt+nrn+v1t1r/TfPZWxmfu5W5of+r95apf9HOoi+3g634eTkZLcnnq+q+qtrTy1br+2up78itYhZ9tdg+NrRZ3557y6uqP0nytxZY9e7W2md7nXcneSHJJ/q6J5Kc31r7i6r6xST/uapenYHbmS/UhEW2Wcl7zdwK+2+xY1voj8Pl+mLT9t9I3y3AZy8r7jufuwUs1ZdJbkry/kyO4f1JPpjJH8Kz6MvtYDsf+0Je31p7vKrOSnKwqr6xRN0T/Rnczn297ftKeF5lrbVfXmp9n0D/D5Nc2v99kdba80me78v3VdU3k/x8Jn+dTU/tmL6d+ZNVdXZr7Yn+L5Gnevlit0A/muQNx5V/8USPb62tpP+y9G3fFyr/bib/ZtrRR6am6x97r6NVtSPJz2Tyr6wNf2v55fpukW189rKyvovP3YJG+7Kqfj/J5/rLWfTldrDpPi9rqbX2eH9+qqo+k8m0ltX6/lrqO3KrmEVfLbaPDc20jRmqqiuS/HaSf9xa+8FU+c6qOqUvvzLJriSP9n9hfL+qLulz9t6R5Ngo2P4kx85kvfa48nf0s2EvSfJcf587k1xWVaf3f5Ve1ss2jcX6L5NjvqYmZ9lfkEn/3ZNFbgffQ/cXkry1b398/x3r17cm+dNef7F9bGo+eyfF5+4E9V+Ox7wlybEz+2fRl9vBgv21zm1aF1X1k1X108eWM/neeTCr9P21zHfkVjGLvlpsHxtb2wBnLW6XRyYnwTyW5P7+OHZG+D9JcjiTM6O/kuQfTW2zO5Mf+G8m+ff50V0hX57kriSP9Oczenkl+Uiv/7X8+NUBfq23YT7JO9e7P1ar//q6d/djfjhTZzxnclbw/+jr3j1V/spMfjnPJ/lPSU7t5T/RX8/39a9cbh+b4ZFJUDmaySjzk5l8ofnsnUTf+dytqC//sH82Hsjkl+bZs+zL7fBYrL+226N/Pr7aH4eP9cVqfn8t9h25GR9JPpnJNL7/27/vrptFXy22j43+cHtuAAAYZNoGAAAMEp4BAGCQ8AwAAIOEZwAAGCQ8AwDAIOEZAAAGCc8AADDo/wOtZZ/MVAsgKwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 864x576 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.figure(figsize=(12,8)); plt.hist(y_scores, bins=100);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With the decision scores, we can compute precision and recall for all possible thresholds using the `precision_recall_curve()` function:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.metrics import precision_recall_curve"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "metadata": {},
   "outputs": [],
   "source": [
    "precisions, recalls, thresholds = precision_recall_curve(y_train_0, y_scores)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "metadata": {},
   "outputs": [],
   "source": [
    "def plot_precision_recall_vs_threshold(precisions, recalls, thresholds):\n",
    "    plt.plot(thresholds, precisions[:-1], \"b--\", label=\"Precision\")\n",
    "    plt.plot(thresholds, recalls[:-1], \"g--\", label=\"Recall\")\n",
    "    plt.xlabel(\"Threshold\")\n",
    "    plt.legend(loc=\"upper left\")\n",
    "    plt.ylim([-0.5,1.5])    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 84,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAtUAAAHkCAYAAADxfdP+AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJzs3Xd4VFXixvHvSQ+hpdG70msgdERUQFSKNAEVG4gLC6wdVn+Kuwp2V1FcBRVQEEHsrgVQsFCUgBSRIiJKqKGTRtr5/XFDCgkIzCQ35f08zzyZmTMzeSeUvDk591xjrUVERERERC6cj9sBRERERESKO5VqEREREREPqVSLiIiIiHhIpVpERERExEMq1SIiIiIiHlKpFhERERHxkFdKtTHmDWPMAWPMz2cY72aMOWaMWZd5eTjHWC9jzFZjzHZjzERv5BERERERKUzGG/tUG2O6AvHAm9baZvmMdwPutdb2Pu1+X2Ab0AOIBVYDw6y1v3gcSkRERESkkHhlptpa+y1w+AKe2g7Ybq3dYa1NAd4B+nkjk4iIiIhIYSnMNdUdjTHrjTGfG2OaZt5XHdiV4zGxmfeJiIiIiBQbfoX0edYCta218caYq4EPgfqAyeex+a5HMcaMAkYBhISEtGnUqFFBZRURERERYc2aNQettZHn8thCKdXW2uM5rn9mjHnZGBOBMzNdM8dDawB7zvAa04HpANHR0TYmJqYAE4uIiIhIaWeM+eNcH1soyz+MMVWMMSbzervMz3sI58DE+saYusaYAGAo8HFhZBIRERER8RavzFQbY+YB3YAIY0wsMAnwB7DWvgIMAkYbY9KAJGCodbYdSTPGjAW+BHyBN6y1m7yRSURERESksHhlS73CpuUfIiIiIlLQjDFrrLXR5/LYwjpQscClpqYSGxtLcnKy21GKpaCgIGrUqIG/v7/bUURERESKnRJTqmNjYylXrhx16tQhc/m2nCNrLYcOHSI2Npa6deu6HUdERESk2CnMfaoLVHJyMuHh4SrUF8AYQ3h4uGb5RURERC5QiSnVgAq1B/S1ExEREblwJapUu83X15dWrVrRrFkzBg8eTGJiosevGRMTw/jx4884vmfPHgYNGuTx5xERERGRC6dS7UXBwcGsW7eOn3/+mYCAAF555ZVc49ZaMjIyzus1o6OjmTp16hnHq1WrxsKFCy8or4iIiIh4h0p1AbnkkkvYvn07O3fupHHjxowZM4bWrVuza9cuFi1aRMeOHWndujWDBw8mPj4egNWrV9OpUydatmxJu3btOHHiBMuWLaN3794AfPPNN7Rq1YpWrVoRFRXFiRMn2LlzJ82aNQOcdeW33norzZs3JyoqiqVLlwIwa9YsBgwYQK9evahfvz7333+/O18UERERkRKqxOz+cbpu3fLed911MGYMJCbC1VfnHb/lFudy8CCcvqJi2bJz/9xpaWl8/vnn9OrVC4CtW7cyc+ZMXn75ZQ4ePMhjjz3GkiVLCAkJ4cknn+S5555j4sSJDBkyhPnz59O2bVuOHz9OcHBwrtd95plnmDZtGp07dyY+Pp6goKBc49OmTQNg48aNbNmyhZ49e7Jt2zYA1q1bx08//URgYCANGzZk3Lhx1KxZExERERHxXIkt1W5ISkqiVatWgDNTPWLECPbs2UPt2rXp0KEDAKtWreKXX36hc+fOAKSkpNCxY0e2bt1K1apVadu2LQDly5fP8/qdO3fm7rvv5oYbbmDAgAHUqFEj1/j333/PuHHjAGjUqBG1a9fOKtVXXHEFFSpUAKBJkyb88ccfKtUiIiIiXlJiS/XZZpbLlDn7eETE+c1Mn3JqTfXpQkJCsq5ba+nRowfz5s3L9ZgNGzb85Q4cEydO5JprruGzzz6jQ4cOLFmyJNds9dnOjhkYGJh13dfXl7S0tL98PyIiIiJybrSmupB16NCB5cuXs337dgASExPZtm0bjRo1Ys+ePaxevRqAEydO5Cm+v/32G82bN2fChAlER0ezZcuWXONdu3Zl7ty5AGzbto0///yThg0bFsK7EhERESndVKoLWWRkJLNmzWLYsGG0aNGCDh06sGXLFgICApg/fz7jxo2jZcuW9OjRI8/JWJ5//nmaNWtGy5YtCQ4O5qqrrso1PmbMGNLT02nevDlDhgxh1qxZuWaoRURERKRgmLMtGSiqoqOjbUxMTK77Nm/eTOPGjV1KVDLoaygiIiKSzRizxlobfS6P1Uy1iIiIiIiHVKpFRERERDykUi0iIiIi4iGVahERERERD6lUi4iIiIh4SKVaRERERMRDKtVe5OvrS6tWrWjWrBl9+vTh6NGjXn39WbNmMXbsWAAeeeQRnnnmGa++voiIiIhcGJVqLzp1mvKff/6ZsLAwpk2b5nYkERERESkEKtUFpGPHjuzevTvr9tNPP03btm1p0aIFkyZNyrr/zTffpEWLFrRs2ZLhw4cD8Mknn9C+fXuioqLo3r07+/fvL/T8IiIiInLu/NwOUFC6zeqW577rml7HmLZjSExN5Oq5V+cZv6XVLdzS6hYOJh5k0IJBucaW3bLsnD93eno6X331FSNGjABg0aJF/Prrr/z4449Ya+nbty/ffvst4eHhTJ48meXLlxMREcHhw4cB6NKlC6tWrcIYw2uvvcZTTz3Fs88+e+5vXkREREQKVYkt1W5ISkqiVatW7Ny5kzZt2tCjRw/AKdWLFi0iKioKgPj4eH799VfWr1/PoEGDiIiIACAsLAyA2NhYhgwZwt69e0lJSaFu3bruvCEREREROScltlSfbWa5jH+Zs45HlIk4r5npU06tqT527Bi9e/dm2rRpjB8/Hmst//znP7njjjtyPX7q1KkYY/K8zrhx47j77rvp27cvy5Yt45FHHjnvLCIiIiJSeLSmugBUqFCBqVOn8swzz5CamsqVV17JG2+8QXx8PAC7d+/mwIEDXHHFFSxYsIBDhw4BZC3/OHbsGNWrVwdg9uzZ7rwJERERETlnJXam2m1RUVG0bNmSd955h+HDh7N582Y6duwIQNmyZZkzZw5NmzblwQcf5NJLL8XX15eoqChmzZrFI488wuDBg6levTodOnTg999/d/ndiIiIiMjZGGut2xnOW3R0tI2Jicl13+bNm2ncuLFLiUoGfQ1FREREshlj1lhro8/lsVr+ISIiIiLiIZVqEREREREPqVSLiIiIiHioRJXq4rg+vKjQ105ERETkwpWYUh0UFMShQ4dUDi+AtZZDhw4RFBTkdhQRERGRYqnEbKlXo0YNYmNjiYuLcztKsRQUFESNGjXcjiEiIiJSLJWYUu3v76/TeYuIiIiIK0rM8g8REREREbeoVIuIiIiIeEilWkRERETEQyrVIiIiIiIeUqkWEREREfGQSrWIiIiIiIdUqkVEREREPKRSLSIiIiLiIZVqEREREREPqVSLiIiIiHhIpVpERERExENeKdXGmDeMMQeMMT+fYfwGY8yGzMsKY0zLHGM7jTEbjTHrjDEx3sgjIiIiIlKYvDVTPQvodZbx34FLrbUtgEeB6aeNX2atbWWtjfZSHhERERGRQuPnjRex1n5rjKlzlvEVOW6uAmp44/OKiIiIiBQFbqypHgF8nuO2BRYZY9YYY0ad6UnGmFHGmBhjTExcXFyBhxQREREROVdemak+V8aYy3BKdZccd3e21u4xxlQCFhtjtlhrvz39udba6WQuG4mOjraFElhERERE5BwU2ky1MaYF8BrQz1p76NT91to9mR8PAB8A7Qork4iIiIiINxRKqTbG1ALeB4Zba7fluD/EGFPu1HWgJ5DvDiIiIiIiIkWVV5Z/GGPmAd2ACGNMLDAJ8Aew1r4CPAyEAy8bYwDSMnf6qAx8kHmfH/C2tfYLb2QSERERESks3tr9Y9hfjI8ERuZz/w6gZd5niIiIiIgUHzqjooiIiIiIh1SqRUREREQ8pFItIiIiIuIhlWoREREREQ+pVIuIiIiIeEilWkRERETEQyrVIiIiIiIeUqkWEREREfGQSrWIiIiIiIdUqkVEREREPKRSLSIiIiLiIZVqEREREREPqVSLiIiIiHhIpVpERERExEMq1SIiIiIiHlKpFhERERHxkEq1iIiIiIiHVKpFRERERDykUi0iIiIi4iGVahERERERD6lUi4iIiIh4SKVaRERERMRDKtUiIiIiIh5SqRYRERER8ZBKtYiIiIiIh1SqRUREREQ8pFItIiIiIuIhlWoREREREQ+pVIuIiIiIeEilWkRERETEQyrVIiIiIiIeUqkWEREREfGQSrWIiIiIiIdUqkVEREREPKRSLSIiIiLiIZVqEREREREPqVSLiIiIiHhIpVpERERExEMq1SIiIiIiHlKpFhERERHxkEq1iIiIiIiHVKpFRERERDykUi0iIiIi4iGVahERERERD6lUi4iIiIh4yCul2hjzhjHmgDHm5zOMG2PMVGPMdmPMBmNM6xxjNxtjfs283OyNPCIiIiIihclbM9WzgF5nGb8KqJ95GQX8F8AYEwZMAtoD7YBJxphQL2USERERESkUft54EWvtt8aYOmd5SD/gTWutBVYZYyoaY6oC3YDF1trDAMaYxTjlfJ43cnlbQkoCyWnJGGPwMT4YDH4+foQEhACQnJZMhs3AYDDGYHAe5+/rD0CGzQDIGhcREfGG5GQ4eDD3fSEhEJo5TRUbm/c5ZctCxYqQkQF79uQdL1/euaSnw969eccrVIBy5SA1FfbvzztesaLzOVJS4MCBvONhYVCmDJw8CXFxecfDwyE4GJKS4NChvOORkRAYCImJcPhw3vFKlSAgAOLj4ejRvOOVK4O/P5w4AceO5R2vWhV8feH4cedyumrVwMfHee34+LzjNWo4H48cgYSE3GPGQPXqzvVDh5z3mJOPj/P64Py5JifnHvfzgypVnOtxcc7XMCd/f+f9gfNnk5qaezww0Pn6AezbB2lpuceDgiAiwrm+Z4/zdySn4GDnzwdg926wNvd4QfzdO/X1LMq8UqrPQXVgV47bsZn3nen+ImnYe8P4ZNsnue6LqhLF2jvWAtDljS6s2bsm13jX2l355pZvAGg8rTHbDm3LGjMYrmlwDZ8Mc16zzvN12HNiD8YYgv2C8fXxZWDjgUzvMx2Aui/UJSElIauQGwzDWwzn6Z5PY62l+nPOly7n+B1t7uChSx8iPiWeJtOa5Bm/s8Od3NnhTvbH76fTG52y7j/1uImdJzKi9Qh+P/I7veb2yjP+r27/4rqm17HpwCaGLByS5/Wf6P4EV9e/mjV71jD287F5nv90j6fpVLMT3//5PQ989UCe57/Q6wVaVmnJkh1LmPLdlDzj/73mv9QPr8+n2z5l6g9T84y/3vd1qpevzsJfFvLGT2/kGX+r/1uEBocyd8Nc5m+an2f8nUHvEOQXxMyfZvLpr59m3R/oF4i/jz+zrp0FwKx1s1j+5/KsH6aMMYT4h/Dslc8CMPOnmazbty7XD2ShwaH8X9f/yxr//ejvBPsF4+fjh5+PH5EhkdzY4kYAvtj+BcdPHsfH+OBjfPA1voQFh3FJ7UsAWLFrBQkpCc6Yjy8+xodA30Da12gPwK5ju/AxPgT5BVE+sDx+Pn76wU5KlYwMp3wkJDgF4uBBp4i1auV8s966FebPz/u8m26COnVg5UqYMSNvOXrhBafcfPQRzJ2bff+WLU5xWLHCuT14MLz3nlOmTl1q14bt253xPn1gyRLnfnA+NmkCq1c7t3v2hB9/zH6ujw9ER8MXXzjjDRrArpzfTYHbb4fpzrcPatbM+97uvhuefdZ5T/mNT5oEjzziFOL8xp95Bu65B37/HRo2zDv+yitwxx2wcaOT9XRz58L11ztf28suyzv+8cfZX5e+ffOOL10K3bo5X/vrr887HhMDbdrA2287OU63ZYuTe/p0uPfevOO7dzvF9rnn4F//yjt+/LjzQ8WjjzqPOd2pojlxYvafwyllyzplHmDcOJh32lRilSrZZfLWW+HTT3OP168P2zLrxODB8M03ucdbt4Y1mXXk6qth7drc45deCsuWZV/fti33eO/e8Elm3YmOzvtD1dCh2ZkbNcr7Q4W3/+75+DgFu6grrFKd33dve5b7876AMaNwlo5Qq1Yt7yU7D/0a9qN7ve5Ya7FYMmwGlUIqZY2PazeOffH7sNisx9Qsn/23ZXy78RxMPJj1XGstDcIbZI3f0eYOjp88jsWSkOL8z92mWpus8WsbXktymvPjqs38MrWs0jJrvHeD3tjMf8WnxhtGOP/T+RpfJ3vm/aceV6diHQACfAPoVLNTnudXKev8KBzoF0jrqq3zjIcFhwEQ5BdEo4hGeV6/XEA55/P7+FI+sHye5/saX4CsWf/Tn39Khs0gLSMtz/ip2ynpKcSnxOcZP/XbgaTUpKyvfX7jx08eJ/Z4bN7Xz/wYlxjHtkPbsv5ck1KT8DHZq6c27t/IZ9s/y/pztVjKB5bPKtXf/vktH275MGs8w2ZQvXz1rFL9/pb3+XRb7v81m0Y2zSrV//7m36yMXZlrvGONjqwY4XzHHvXJKDbFbco13r1edxYPXwxA5zc6s+t49ndcH+PD0GZDmTvAaQGdXu+ExRIWHEb5wPJUCKxAtzrdGNpsKNZapnw3hdDgUMoGlKVCYAX8fPyoH16fBuENSE1P5cfdP+Ln40eQXxAhASEE+gZmPV7kbJKSnFkvcL6BJic7ZSM93SklISFw8cXO+KpVsHmz8w1861ZnpjQ6Gvr3d17n6qud55w44cy+nTjhFLc+fZzyml/xe/ttGDbMeb1Jk/KOX3qpU6p/+80pGWFhzjf4U07NEB48CD/nOKrIWti0yfloDAwc6JQPa7MvFSpkP75fP2jcOPu51mbPNAJcc40znvP5tWtnj48c6fywkPPbY873O2NG3vfWrJnzMTAw//HWmUc/VaiQ/3h752d2KlfOf7xzZ+dj7dr5j7drl50zv/EWLbI/5jd+6v21a5f/+KmvT9eu+Y+f+vr27Jn/+Kk/n759858lDQx0Pg4enP1nl5/hw6Ft29z3+ftnX7/9drj88tzjp/5NgFO6+/XLPV6+fPb1++6DG2/MPX5qFhng//4v70x/1arZ1x97LO9Mfc6/R8884/w2IKd69bKvv/RS3plwb//dKy5zQOb08nLBL+Qs//jUWtssn7FXgWXW2nmZt7fiLP3oBnSz1t6R3+POJDo62sbExHglt0hRkZ6RTlpGWtbFYqkYVBGAP4/9SXxKPOkZ6WTYDDJsBsH+wTSKaATAun3rSEhJIN1mj9csX5P64fUBWLBpAceSjxGfEk9iaiLJack0iWzCsObDABi6cCgHEw9yNPkoR5OPciLlBDe1uImnez5NSnoKgY8F5sn7QJcHmHzFZA4mHiTy6cg8449f8TgTu0xk59GdDF04lIgyEVQOqUzlspWpEFiB/o370yC8AQcSDrBs5zL8fPzw9/En2D+YYL9gmkQ2ITQ4lBMnT7D7xG6C/YIJLxNOiH+IZtmLqAMHnHKZluZcDh1yfvV86hv+gw/CV185xTkx0ZlZbd7cmYEFZ9Z4/frcr9m9Oyx2fjakbl3YuTP3+LhxMHWqU6p7ZR7ZExjozPSFhzuF/O9/d34FP3u28ytpX19nhjEszPnmHhqaXVRPd2pmWERKJ2PMGmttPr9ryauwZqo/BsYaY97BOSjxmLV2rzHmS2BKjoMTewL/LKRMIkWKr48vvj6+BJK3wNaqcPbfzrSq0uqs49c1ve6s4+8MeueMYwG+ASQ9mMSx5GMcP3mcEyknSMtIy/otRvnA8iy6cRFpGWkkpCaQlJrEyfSTtK3mTM2kZ6RTMagiscdjWbt3LQcSDpBu06lVoRYNwhvkWjqU0yfDPqF3g958/fvXXDv/2lx5IspE8MGQD2hXvR1Lf1/Kaz+9RrmAcpQLKEdIQAiRZSIZ1nwYYcFhHD95nKTUJCoGVSTQL+/XVvKy1pm5/eknpwD36+cU0K+/dorp4cPOzFRysjPDtXSpsz7y3nvhrbdyv1aFCtmlOinJKbNVqjhrNq+8MntGCuDhh53yW6aMsxbWx8dZF3vK7NnOcytWdGawQ0Kyx4KD8/4KPKfQULjzzjOPqzyLiKe8UqqNMfNwZp0jjDGxODt6+ANYa18BPgOuBrYDicCtmWOHjTGPApmrxvj3qYMWRaToCPILIqhsEJXLVs4zFuAbQI+LepzxuReFXcQXN36RddtaS2JqYtbymXbV27FpzCZS01NJy0gjPiWek+kniaoSBThLoN4e8DYJqQkcSTpCXGIch5MOE1nGmR2PS4zjx90/cjT5KPEp8VlLpK6qfxVhwWFM+3EaD3z9AAZDRJkIQoNDqVauGv+95r80imjE939+z+7ju2kU0Yjq5asTFhyWa2lPaXL0KLz4orMWNufBQ2vXOqV6+3ZYtMgptgEBTjGuXj17reOECc4yBX9/50CqkBBnre8p+a07zWnAgLOPd+16Ye9LRKQweG35R2HS8g8ROZO0jDQOJh4kokwEfj5+/LT3J1bGrmT38d0cTjrMoaRDGGOYN3AePsaHmz+8mTfXv5n1/BD/EAY2Gcjsa2cDsPi3xfj6+FK1bFUuCruIAN8At96a11gLP/zgHPz0zTdw3XXOMort250DoKKinAP0mjeHiy5yDtYKKP5vW0TkvBXF5R8iIoXCz8cva2kKQFTVKKKqRp3x8dN7T2d09Gh2HdvF3vi9rNu3juMnnf2z0jPSuW/xfazf7yz0LRtQlvbV2zOk6RBub3N7wb4RL8p5MOBtt8G33zoH3gF06ODsRADOwUc7djhrl0VE5PxoplpE5Cx2HNnBzqM72X18NytjV7IqdhXR1aKZ3mc6CSkJNJ7WmEYRjWgU0YiONTrSpVYXapSv4erBlKdmoj/80NmWLCPD2T4MnO254uKcpRa9e+desywiIrmdz0y1SrWIyAXaF7+Pu768i1/ifmHrwa2cTHf2V5vTfw43tLiBzXGb+eaPb2gS2YQG4Q1yzaAXlK++ghEj4I8/nNs9ezp7AE+cWOCfWkSkxNHyDxGRQlClbBXmDXR2AE1NT+WH3T8Qsycm68DN5buWM/p/o7Me7+/jT4caHZg3cB7Vy3t+nqvERGcmet48GD3a2VIuPd1ZA33ffc6uHcXhLGQiIiWBZqpFRAqItZbth7ez48gOVuxawb74fWw+uJmvb/4aPx8/XvrxJQ4kHGBg44G5TuR0Nvv3O4X555+dbe8SE52TWLzwAgzJuzOhiIh4QMs/RESKOGst17x9DV9s/wJjDJfWvpQbmt/AjS1uzLWfdno6vPuuc6KU++5z1kcPHuzsFV23rnO2tq5dnROaiIiId6lUi4gUEwcTDzL1h6nMWjeLXcd38ciljzCp2yT+3J3KtBnxzH4llP37nf2gt2zJ3qlDREQKnkq1iEgxk56RzsdbP6Z55eYc2HIxXe94j/R+N1Iz9h7u63ELY4ZehK+vTvknIlKYzqdUl87ThomIFCEJCfDWm76kbOjPxWEX07AhDL68Eb3q9WFX3cmM316fBtMuZsLiCaSmp7odV0RE8qFSLSLikkOHnO3vqlRx9o+eMcO5Pzwc5r3QlM9vW8D2cdt5+eqXaRDegCW/L8Hf1x+ANXvWkJ6R7mJ6ERHJScs/RERc8OOPMGiQcwDiLbc45bpzZzjbOWPSMtLw8/FjX/w+6r5Ql9oVajP58skMaDzA1ZPNiIiUVFr+ISJSxAUEQJkysGIFzJwJXbqcvVCDcwp2gEohlZg7YC6+Pr4MencQzf7bjFdjXiUlPaUQkouISH40Uy0iUgishVmz4OBBuPlm5/TgaWng58EpuNIy0nhr/VtMWz2NdfvWsWnMJhpGNPRaZhGR0k4z1SIiRUhKCtx2m3P59FNn72nwrFCDM3N9a9StrL59Nev/tp6GEQ2x1vLMimfYsH+D58FFROScqVSLiBSg33+HSy5xZqnvvx+WLoWqVb37OYwxNK3UFICth7by8NKHaflKS0Z9MoqTaSe9+8lERCRfKtUiIgUkPR169XJO2rJwITz5JPgU8P+6jSIa8eddfzImegwz1s5g0LuD2Be/r2A/qYiI4OEvH0VE5HQZGU6h9veH11+HatWgXr3C+/wRZSKYds00aleszUNLH2LQgkF8d+t32iFERKQAqVSLiHiRtTBmjHM68WeecXb1cMv9ne+ne73uRJSJwBjDkaQjbDm4hY41O7oXSkSkhNLyDxERL0lNhXHj4NVXnVnqoqB11dbUqlALgHsX3UunNzrxwFcP6MyMIiJepplqEREv2LsXhgyB776DUaNgyhS3E+U19aqppNt0Hv/+cdbuXcvC6xZSNqCs27FEREoEzVSLiHjBsGGwZg1Mnw6vvPLXJ3JxQ0hACLOuncWrvV9l8Y7F9JnXh/iUeLdjiYiUCJqpFhG5QBkZzhpqX1/45z+hYkVo397tVH9tVJtRlA0oy+TvJpOQkqDZahERL9AZFUVELsDRo3DDDdC6NTz6qNtpLkxqeir+vv6kpKcQezyWeqGFuEWJiEgxoDMqiogUoP374eqrYdEi75/IpTD5+zpHU478eCSXz76crQe3upxIRKT4UqkWETkPO3ZAq1awejW8+aazfV5xN779eBJTE+n4ekeW7VzmdhwRkWJJpVpE5BylpkKPHpCSAqtWOQcnlgTR1aL5YeQPVC1XlR5v9WDmTzPdjiQiUuyoVIuInCN/f2dnj+XLoU0bt9N4V93Quiy/bTmX1bmMCUsmcCDhgNuRRESKFe3+ISLyF554AgID4a67nJnqkqpiUEX+d/3/2Be/j0ohlciwGfgYzb2IiJwL/W8pInIWc+Y42+WtWeN2ksLh7+tPzQo1ARj/+Xh6zenFrmO7XE4lIlL0qVSLiJzBsmVwxx3QtSvMLGXLjK211ChfgxW7VtDutXas3r3a7UgiIkWaSrWISD5WroTevaFGDZg/31lPXZoYY5jYZSIrRqwgyC+IrrO6smTHErdjiYgUWSrVIiL52LjRKdRLl0KVKm6ncU+zSs34YeQP1K1Yl9s+uo2TaSfdjiQiUiTpjIoiImeQnu6cglxg9/Hd7IvfR5tqJWzbExGRs9AZFUVELoC1zg4f773n3Fahzla9fPWsQv0W9z/wAAAgAElEQVTy6pfZeXSnu4FERIoYlWoRESAjA+69F55/HtaudTtN0XUg4QAPfv0g/d7px4mTJ9yOIyJSZKhUi4gAHTvCc8/B2LHw2GNupym6KoVUYv6g+fx84GcumXkJx5KPuR1JRKRIUKkWkVLvs8/gxx+d3T6mTgVj3E5UtPW8qCcfDPmAjQc2MvyD4RTHY3NERLxNpVpESr21a6F1a5g3T4X6XPVt2JfJl0/m022fsmLXCrfjiIi4Trt/iEiplXN3j5QUCAhwN09xk5aRxoGEA1QrV83tKCIiBUK7f4iI/IX9+6F5c2fZB6hQXwg/H7+sQj1nwxy2H97uciIREfeoVItIqWMt3HQT7NgBISFupyn+ElISGPvZWG58/0aS05LdjiMi4gqVahEpdV56CRYtgv/8B5o2dTtN8RcSEMKMPjP4YfcPPL/qebfjiIi4QqVaREqVzZvh/vvhqqvgb39zO03JMbjpYPo06MNDSx9i4/6NbscRESl0KtUiUqq8+SaUKwczZminD297o98blA8sz7+//be22RORUsfP7QAiIoVpyhQYMwaqV3c7SckTUSaCx694nPKB5TH6iUVEShmvzFQbY3oZY7YaY7YbYybmM/4fY8y6zMs2Y8zRHGPpOcY+9kYeEZHTzZ8Pv/7qzE7XrOl2mpJrVJtRDGk6BIA1e9a4nEZEpPB4XKqNMb7ANOAqoAkwzBjTJOdjrLV3WWtbWWtbAS8C7+cYTjo1Zq3t62keEZHTrVwJw4fDww+7naR0MMbw7qZ3iZ4RzdQfprodR0SkUHhjprodsN1au8NamwK8A/Q7y+OHAfO88HlFRP7S+vXQs6czO/3yy26nKT36NerHtY2u5R9f/IMPt3zodhwRkQLnjVJdHdiV43Zs5n15GGNqA3WBr3PcHWSMiTHGrDLGXOuFPCIiABw8CAMHOgcmfvUVhIa6naj0CPANYMGgBbSq0orR/xvNkaQjbkcSESlQ3ijV+R2NcqbDvocCC6216Tnuq5V5+sfrgeeNMRfl+0mMGZVZvmPi4uI8SywipcJDD0FsLCxYAHXquJ2m9PH39ef1vq8TlxDHfYvvczuOiEiB8kapjgVyHvZTA9hzhscO5bSlH9baPZkfdwDLgKj8nmitnW6tjbbWRkdGRnqaWURKgalTYdUq6NLF7SSlV+uqrZl8+WR6XdzL7SgiIgXKG6V6NVDfGFPXGBOAU5zz7OJhjGkIhAIrc9wXaowJzLweAXQGfvFCJhEpxf7xD9iyBfz9oVUrt9PIhC4TGNRkkNsxREQKlMf7VFtr04wxY4EvAV/gDWvtJmPMv4EYa+2pgj0MeMfmPiNAY+BVY0wGTsF/wlqrUi0iF+y995wZaoAXXnA3i2TLsBk8+f2T+Pn4cV9nLQURkZLHFMezXkVHR9uYmBi3Y4hIEbNjB0RFQYMGsHw5BAS4nUhOsdYyZOEQ3v3lXRYOXsjAJgPdjiQi8peMMWsyj/37SzpNuYiUCKmpMHSoc3KXd99VoS5qjDHMvnY2rau25paPbmFf/D63I4mIeJVKtYiUCNOmwerVMH26dvooqoL9g5k3cB5JqUmM+3yc23FERLzK4zXVIiJFwdChEBICgwe7nUTOpkF4AyZdOoknlz9JXEIckSHazUlESgatqRaRYu3ECQgOBj9NERQbJ9NOsvHARqKrndMyRRER12hNtYiUCikp0KcPtGwJyclup5FzFegXmFWo52yYw4mTJ1xOJCLiOZVqESm27r0XvvkGJkyAoCC308j52npwK8M/GM64z8eRlpHmdhwREY+oVItIsTR/Prz4Itx1F9x0k9tp5EI0jGjIhM4TmL1+Nv/+5t9uxxER8YhKtYgUOzt3wujR0L49PPWU22nEE090f4Lrm1/PlO+m8O0f37odR0TkgqlUi0ix4+sL7drB3Lk6QLEkmNprKnVD6zJ04VCOJB1xO46IyAXRtyMRKVashZo14Ysv3E4i3hJeJpwPhnxAanoqocGhbscREbkgmqkWkWJj/nxnt4/jx91OIt7WrFIzoqpGAbDjyA6X04iInD+VahEpFrZuhZEj4ehRZ19qKZmeWv4U9V+sz9e/f+12FBGR86JSLSJFXnw89O/vbJv3zjvg7+92IikoY9qOoUF4A4Z/MJzDSYfdjiMics5UqkWkSLPW2elj61Zn+UeNGm4nkoJUNqAscwfMJS4hjlGfjHI7jojIOVOpFpEibf9++OormDQJLr/c7TRSGFpXbc1DXR/ivc3vsfT3pW7HERE5J9r9Q0SKtCpVYMMGCNWmEKXKvZ3u5c0Nb1K9fHW3o4iInBPNVItIkWQtfPghpKRARISzN7WUHsH+waz/23oahDdwO4qIyDlRqRaRImnOHOfgxOnT3U4ibinjX4bktGTuX3w/b2982+04IiJnpVItIkXOb7/B3/8OnTs7BylK6eXv48/SnUu59aNbWblrpdtxRETOSKVaRIqUtDS48Ubw8XFOQ65lH6Wbr48vHwz5gMgykfSf31/b7IlIkaVSLSJFytNPw6pV8MorULu222mkKKhRvgYfDf2Iw0mHue7d68iwGW5HEhHJQ6VaRIqUnj3hoYdg6FC3k0hR0qZaG166+iV+2vcT2w9vdzuOiEgexlrrdobzFh0dbWNiYtyOISIihchay+Gkw4SXCXc7ioiUEsaYNdba6HN5rGaqRaRI+OorGD4cEhLcTiJFlTGG8DLhJKclM3fDXIrjpJCIlFw6+YuIuO7ECRgxAgIDnQMURc7mvV/e48YPbqRCUAV6N+jtdhwREUAz1SJSBDz/PPzxB7z+OgQHu51Girrrml5HnYp1mPzdZM1Wi0iRoVItIq46cACefRZ694YuXdxOI8WBv68/D3R5gFWxq3hu5XNuxxERAVSqRcRlDz4IiYkwebLbSaQ4GdF6BP0b9WfCkgnMXjfb7TgiIirVIuKuKVNg/nxo0cLtJFKc+BgfZl07i7HtxnJpnUvdjiMioi31RMQdSUkQEKAzJor3ZNgMfIzmikTEe7SlnogUeffeC127Qmqq20mkuEtOS+aKN69gyndT3I4iIqWYSrWIFLrvv4eXX4a2bcHf3+00UtwF+QUR4BvAtNXTSE5LdjuOiJRSKtUiUqiSk2HkSKhdGx57zO00UlLc0/Ee9sXv47Fv9ZdKRNyhUi0iheqxx2DrVpg+HcqWdTuNlBTd63VneIvhPPH9E2w5uMXtOCJSCqlUi0ihSUmBDz+Em26Cnj3dTiMlzRPdn8AYw5vr33Q7ioiUQjpNuYgUmoAAiImBkyfdTiIlUbVy1fj8hs+5pNYlbkcRkVJIM9UiUihmzoT16yEoCCpUcDuNlFTd63Un0C+QxNRE0jPS3Y4jIqWISrWIFLitW+G22+Chh9xOIqXB70d+p8GLDZizYY7bUUSkFFGpFpECZS2MHu3MTs+Y4XYaKQ1qVahFlbJV+McX/yD2eKzbcUSklFCpFpEC9fbbsHQpPPEEVK7sdhopDXx9fJk3cB4p6Sncs+get+OISCmhUi0iBSYuDsaPh+hoGDXK7TRSmtQPr8/o6NEs2LSApb8vdTuOiJQCKtUiUmBOnoSOHWHWLPDR/zZSyB69/FGqlK3C8l3L3Y4iIqWAttQTkQJTowZ8+qnbKaS0KuNfho2jNxJRJsLtKCJSCmjuSES8Li0Nxo2DXbvcTiKl3alCveXgFpJSk1xOIyIlmUq1iHjd1Knw0kuwcqXbSUScQt1kWhNe+OEFt6OISAmmUi0iXrVhA0yYAP36weDBbqcRgUYRjejdoDePfvsou47p1yciUjC8UqqNMb2MMVuNMduNMRPzGb/FGBNnjFmXeRmZY+xmY8yvmZebvZFHRNyRlga33+7sSf3662CM24lEHFOvmkqGzeC+xfe5HUVESiiPS7UxxheYBlwFNAGGGWOa5PPQ+dbaVpmX1zKfGwZMAtoD7YBJxphQTzOJiDteeQV+/NFZ+hEe7nYakWx1KtZhYueJzN80n7c3vu12HBEpgbyx+0c7YLu1dgeAMeYdoB/wyzk890pgsbX2cOZzFwO9gHleyCUihWz4cGfrvCFD3E4ikteDXR/kuz+/43DSYbejiEgJ5I1SXR3IuUgtFmfm+XQDjTFdgW3AXdbaXWd4bvX8PokxZhQwCqBWrVpeiC0i3pKR4Sz9qFABxoxxO41I/vx8/Fg8fDFG65JEpAB4Y011fv872dNufwLUsda2AJYAs8/juc6d1k631kZba6MjIyMvOKyIeN9LL0GbNnDwoNtJRM7uVKGet3EeO47scDmNiJQk3ijVsUDNHLdrAHtyPsBae8haezLz5gygzbk+V0SKtt274Z//hJo1tY5aioe4hDhG/280g98dTEp6ittxRKSE8EapXg3UN8bUNcYEAEOBj3M+wBhTNcfNvsDmzOtfAj2NMaGZByj2zLxPRIoBa2H0aGf5x0svabcPKR4iQyJ5ve/rrN27lse+fcztOCJSQnhcqq21acBYnDK8GVhgrd1kjPm3MaZv5sPGG2M2GWPWA+OBWzKfexh4FKeYrwb+feqgRREp+ubOhU8+gcmToV49t9OInLuBTQYyoPEApv4wlX3x+9yOIyIlgLE23yXMRVp0dLSNiYlxO4ZIqdezJ5w4AcuXO7t+iBQnm+M203p6a7rX687HQz/WAYwikocxZo21NvpcHuuN3T9EpJT67DPn4EQVaimOGkc2ZtrV06gcUlmFWkQ8plItIuft44+hSxcIC4MqVdxOI3Lhbou6Let6ekY6vj6+LqYRkeJM80sicl5Wr4ZBg+DBB91OIuI9Dy99mPavtedI0hG3o4hIMaVSLSLn7NgxGDoUqlaFKVPcTiPiPZ1rdmbD/g2M/GQkxfFYIxFxn0q1iJwTa+GOO+CPP2DePAgNdTuRiPdcefGVTL58Mu9vfp+Ptn7kdhwRKYZUqkXknMyeDfPnw6OPQqdObqcR8b67O95Nw/CG3P3l3SSkJLgdR0SKGZVqETknvXo566gnTHA7iUjB8PXxZUafGcQlxrFm7xq344hIMaN9qkXkrFJSwM9P2+ZJ6XEk6QihwVrfJCLnt0+1vk2KyBmlp8N118EVV8DJk26nESkcocGhpGWk8fTyp9l0YJPbcUSkmFCpFpEzuvtu+OgjGDAAAgPdTiNSeI4lH2PK91O4+cObybAZbscRkWJApVpE8vXf/8LUqXDnnTBunNtpRApXeJlwnu7xNGv2rmHmTzPdjiMixYBKtYjk8emnMHYs9O4NTz/tdhoRd4yIGkH76u2ZtGwScQlxbscRkSJOpVpE8qhdG/r3h3fecQ5SFCmNjDG8eNWLHEo6xIiPR7gdR0SKOH27FJEsyckQFATNm8PChW6nEXFf2+pteXvA21wcdrHbUUSkiNNMtYgAzk4f/fo5a6hFJFv/xv1pXrk51lqW7VzmdhwRKaJUqkUEa+Hvf4dFi6BJE7fTiBRNr619jctmX8aX2790O4qIFEEq1SLChAnw6qvOx1Gj3E4jUjQNbzmcxhGNufnDm4k9Hut2HBEpYlSqRUq5mTOdHT5uvx0ef9ztNCJFV5BfEAuvW0hCagL95/cnMTXR7UgiUoSoVIuUcmFhzlkTX34ZjHE7jUjR1iSyCXMHzCVmTwz3LrrX7TgiUoRo9w+RUspap0T36+dcROTc9G3Ylxd6vUC90HpuRxGRIkQz1SKl0NGj0LYt/Oc/bicRKZ7Gtx9P7wa93Y4hIkWISrVIKZOcDH37woYN0LSp22lEiq+k1CTu+uIuHv3mUbejiEgRoFItUoqkp0OtWvDdd/Dmm9Czp9uJRIqvIL8gYk/EMvm7ydoNRERUqkVKk3/8A+Li4MEHYehQt9OIFG/GGJ7q/hTpNp1nVzzrdhwRcZlKtUgp0rQpjB0Ljz3mdhKRkqFuaF0GNRnEzHUzOZx02O04IuIilWqREi4jA7Zsca6PHg0vvuhuHpGS5v5O93P85HGm/jDV7Sgi4iKVapESLCMD/vY3aNMGduxwO41IyRRVNYrnez3P39v+3e0oIuIilWqREio9HUaOhBkz4K67oG5dtxOJlFzj248nMiSS1PRUfjv8m9txRMQFKtUiJVBaGtxyi3MK8kmT4NFHdbZEkcIw6tNRXDLzEn499KvbUUSkkKlUi5RAs2fDnDlOmX7kERVqkcJyT8d7SExN5JaPbiE9I93tOCJSiFSqRUoQa52PN98MH3wA//d/7uYRKW2aVWrGC71eYMWuFby65lW344hIIVKpFikhtm6FTp1g3z7w84Nrr3U7kUjpdFPLm+hcszOPfvuottkTKUVUqkVKgA0b4NJLnR0+4uLcTiNSuhljeKHXC9StWFelWqQU8XM7gIh45ttvoW9fKFsWFi+Gxo3dTiQibaq1YfltyzE6oEGk1NBMtUgxtnIl9OoF1arBihUq1CJFiTGGw0mHGf/5eOIS9CskkZJOpVqkGGvSxFk7/c03UKuW22lE5HQ7juxgxtoZjPxkpNtRRKSAqVSLFDPp6fD885CYCBUqwNtvQ2Sk26lEJD/R1aL5V7d/8fHWj3l307tuxxGRAqRSLVKMHDsGffo4Z0h8V9+fRYqFuzrcRXS1aMZ+Ppa9J/a6HUdECohKtUgx8csv0LmzczDiK684e1GLSNHn7+vPzH4zOXHyBE98/4TbcUSkgGj3D5Fi4MsvYdAgKFMGPv8cund3O5GInI9mlZox+9rZtKnWxu0oIlJANFMtUgzUrw9dusDatSrUIsXV4KaDqRdaj6TUJGb+NBN76hSoIlIiqFSLFFEHDsCjjzqnHq9Xz5mhrl7d7VQi4qmZ62Zy28e38dra19yOIiJepFItUgQtXw5RUTBlirOWWkRKjjva3EHPi3oy+n+jWfr7UrfjiIiXqFSLFCEZGfD4484px4ODYdUqaNrU7VQi4k2+Pr4sHLyQuqF1Gf7BcJ0YRqSEUKkWKUJuvRUeeMA5KHHtWmjZ0u1EIlIQygWWY07/ORxOOsztn9zudhwR8QKv7P5hjOkFvAD4Aq9Za584bfxuYCSQBsQBt1lr/8gcSwc2Zj70T2ttX29kEikurHVmqH194aaboFMnGDUKjHE7mYgUpPY12rNg8AJaVG7hdhQR8QLj6dHHxhhfYBvQA4gFVgPDrLW/5HjMZcAP1tpEY8xooJu1dkjmWLy1tuz5fM7o6GgbExPjUW6RoiA2FkaPdmakH3vM7TQi4paTaSdZsWsFl9W9zO0oIpKDMWaNtTb6XB7rjeUf7YDt1tod1toU4B2gX84HWGuXWmsTM2+uAmp44fOKFFvWwowZ0LgxfPUVVKrkdiIRcdNzK5+j//z+xOzRhJFIceWNUl0d2JXjdmzmfWcyAvg8x+0gY0yMMWaVMeZaL+QRKdJ+/RWuuMJZ4tG2LWzaBOPHu51KRNx0Q4sbCA0OpedbPdm4f+NfP0FEihxvlOr8Vn7mu6bEGHMjEA08nePuWpnT6tcDzxtjLjrDc0dllu+YuDgdKS3FV2oqbNzonGp8yRKoW9ftRCLitloVavH1TV8T4BvAlXOu5M9jf7odSUTOkzdKdSxQM8ftGsCe0x9kjOkOPAj0tdaePHW/tXZP5scdwDIgKr9PYq2dbq2NttZGR0ZGeiG2SOFITnZO4nLbbc7tJk1g1y644w7w0f47IpKpbmhdFg9fTHxKPAMXDNQZF0WKGW/s/rEaqG+MqQvsBobizDpnMcZEAa8Cvay1B3LcHwokWmtPGmMigM7AU17IJOK69HSYNw8eegh27nS2yUtJgYAACApyO52IFEXNKzdn0fBFlPEvg9EWQCLFisel2lqbZowZC3yJs6XeG9baTcaYfwMx1tqPcZZ7lAXezfxP4tTWeY2BV40xGTiz5k/k3DVEpLhavx6GDYPNm6FVK2eZxxVXuJ1KRIqDDjU6AJBhM/juj++4tM6lLicSkXPhlX2qrbWfAZ+ddt/DOa53P8PzVgDNvZFBxG3p6bBvH1SvDjVqQIUK8O67MGCAlnmIyPmb+sNU7ll0D1/c8AU9LurhdhwR+Qv6Vi/ioeRkeO01aNAArr3W2S4vPBxWrnSWfKhQi8iFuL317TSKaMT171/P9sPb3Y4jIn9B3+5FLtDx4/DUU1CnDtx+O1Ss6JxiXETEG0ICQnj/uvdJy0ij26xubI7b7HYkETkLlWqRCzR3LkyYAC1aOGumY2Kgf3+dXlxEvKdhREO+ueUbUjNSueWjW7QjiEgR5pU11SKlwR9/wAsvQKNGzolbRo6Edu2gTRu3k4lISdaicgvmDphLeHC4dgQRKcI0Uy1yFhkZ8PXX0K8f1KsHL74Iv/3mjPn7q1CLSOHoXq87UVWjsNayZMcSt+OISD5UqkXO4tZbna3wli+HiRNhxw548km3U4lIaTVnwxx6vNWDl358ye0oInIalWqRTGlp8OWXzv7SsbHOfSNGwFtvOWdAnDwZatY8+2uIiBSk65pex7WNrmXc5+OYt3Ge23FEJAeVainVrHW2vhs3ztlfulcvp1hv2uSMd+0KN94IwcHu5hQRAQj0C2TBoAVcUusSRn4ykg37N7gdSUQyqVRLqZOeDgcOONfj4qBLF2ef6a5d4b33YM8euPJKdzOKiJyJv68/CwYvoGxAWXq/3ZuTaSfdjiQiaPcPKSUOHHBmoD//HBYtyj51eKVK8Nln0LEjlC/vdkoRkXNTpWwVlt+2nB1HdhDoF+h2HBFBpVpKqIyM7DMZ/u1v8OqrzvVKleCaa6BPn+zHalZaRIqji8Mu5uKwiwGY8t0Udh3bxZM9nqR8oGYIRNygUi0lxt692bPRS5fCr79ChQrOso4aNeCqqyAqSqcNF5GS51DiIV5Z8wqr96zmo6EfUb18dbcjiZQ6KtVS7H3/vXOg4bp1zu2qVaF3b4iPd0r19de7m09EpKA9e+WzXFb3Moa9N4zOb3RmyU1LsmaxRaRwaM5OigVrnTMavv02jB4NLVs61wHCwpzy/Pjj8NNPsHs3vPGGs5uHiEhp0btBb5bdvIz4lHi6zepGSnqK25FEShXNVEuRlJQEx49D5cpw7Bg0beqUZYBy5ZwDCytUcG43aQLLlrkWVUSkyGhTrQ0/jPyBH3f/SIBvgNtxREoVlWopElavhpgYWL/e+bhhA1x7LSxY4JTnvn2dYt2pEzRvDn76mysikq+Lwi7iorCLsNby+k+vY63l9ja3ux1LpMRTNZFCc+wYbN0KmzfDli3ODh2nTvk9ejSsWQMVK0Lr1nDPPXD55dnPfflldzKLiBRXFsv7m9/nq9+/olq5alzT4Bq3I4mUaCrV4lXWOqf43rLFWQM9cqRz/623wqxZ2Y/z84O2bbNvz5jhrI2uVQuMKdTIIiIlko/xYc6AOVw++3L6zOvD9c2v51/d/sVFYRe5HU2kRDLWWrcznLfo6GgbExPjdoxS7eRJ2L4dGjZ0CvKbb8LUqU6ZTkjIftzx484a6IUL4bffoFEjaNwY6tYFf3/38ouIlBZHk4/yr2X/YuqPU6kQWIEtY7dQKaSS27FEigVjzBprbfS5PFYz1ZKvpCTnwMCqVSEkBFascJZg7NzpzEDv3u3MSm/Z4hRrHx8ID4cRI5zS3KiRcylb1nm9QYNcfTsiIqVWxaCK/KfXfxjXfhzxKfFZhTohJYGQgBCX04mUHCrVpVB8vHOilN27oUEDqFbNOUBw0iT480+nNB8+7Dx20SLo0QMOHoTly6F2bbjiCqhTxynTlTInO2680bmIiEjRVC+0Xtb1e768h4WbF/Jcz+cY0HgARuvuRDymUl1CZGTAgQPOZf9+5+PevdClC3ToANu2Oafm3rsXTpzIft7rr8NttznP377dKc0dOjhnIKxe3dmuDpzdN/r2dee9iYiIdw1sMpAvf/uSQe8O4qqLr+Ldwe9q1lrEQyrVRVBGhrMW+fBhCAx0ym1aGrzyCsTF5b4MGgRjxzozyVWr5n2txx5zSnJoqHPClF69nJnpqlWd123RwnlcVBT8/HPhvk8REXFHp5qdWPe3dbz040vc/eXdDH53MB8N/Qh/Xx3sInKhVKq9KD3dWVqR8xIYCM2aOePz5jmzyCdOwNGjzqV5c7jzTmc8Kgp27YIjR5xiDc6uGW+8Ab6+cPfdTrkOC3OWXURGQkDm3v7h4TBtmnOylEqVnI+VK0P58s54ZKSz57OIiAiAn48fd3a4kyC/IB746gF2HNlBw4iGbscSKbZUqs/Dww87eyknJjo7XCQmQv368MEHznh0NKxbl/s53brB0qXO9UmT4NdfneshIc6ezDl3wOjSxSnTYWHOzHJYWPbyC2Ngzx7nfl/fvNl8fWHMGK++XRERKQX+Fv03BjYeSGRIJMdPHufzXz9nSLMhbscSKXZUqs/D3r3OWuUyZZyZ4Vq1nFJ9yvjxzixz2bLOJSTEWWpxynffOTPXISH5byf34otn//wREd55HyIiIjlFhkQC8P7m97n1o1t57f/bu/cor8p6j+PvLzMyoqiAXIZQERAzrwSDkZdyJYhpKzsrL6QVkpd1vJSdk6foujznlEJL83iWpifR1GqZYmVkpeKFWp4wg0RBvIBKhiAY4AUODgHP+eO3mX4MM8PgM8xvZni/1nrWb+/nefbez294/PmZPfu395PTuPHUGzmoz0EVHpnUeXifakmSBMCmzZu4/onr+dasb7Fh0wY+f/TnmXzcZPr06FPpoUkVsSP3qe62swcjSZI6h6puVVw25jIWXryQjx38Ma6ZfQ3ffOSblR6W1Cl4+YckSdrKoL0HMf2M6Tz2ymPs1X0vAOYum8uq9as4adhJFR6d1DF5plqSJDXpuAOO46jaowC45clbGP/j8Yz8n5EsWOk9WKXGDNWSJGm7po6dytXjrmb52uWMmTaG7/z+O6xZv6bSw5I6DEO1JEnarr1q9uJLx3yJ2efNZkTtCL7x6DeYtWRWpYcldRheUy1JklrtwF4H8tjnHuOp157ifYATlq0AABMsSURBVP3eB8Blv72MD+z3AT51+KeIiAqPUKoMz1RLkqQddlTtUXSvKj3W9+mVT3POz8/hiBuP4Oa5N/NW/VsVHp3U/gzVkiQpy8zPzOTWj99KdbdqLrzvQgZcPYB7n7u30sOS2pWXf0iSpCzV3aqZ9P5JnDviXGYvnc2d8+9k5MCRAMxfMZ/7F9/P0YOO5vjBx9MtPJ+nrslQLUmS2kREcMz+x3DM/scAsDlt5rL7L+PRJY8CUNuzlkkjJvHlY79Mr917VXKoUpszVEuSpJ2iW3TjkYmP8Opbr/LokkeZvnA6Ux6bwrN/e5ZfnPWLSg9PalOGakmStFMN2nsQnz7y03z6yE/z5+V/Zv3f1wPw3N+e44JfXcDZh5/NZ476DD2796zwSKV3zwubJElSuxk5cCTHHnAsAG/Xv82a9Wu4+DcXs/+1+3Ppby7liVefYHPaXOFRSjvOUC1Jkipi9KDRLLh4AbPPm83JB53MLU/ewvE/PJ53Nr4DwMLXF7J6/eoKj1JqHS//kCRJFTVmvzGM2W8MK9etZN5r89hjtz0AOPfec5n32jzGDh3L6YeezjlHnENNdU2FRys1zTPVkiSpQ+i/Z39OGnZSw/qNp97IF8d8kedXPc95M86j/9X9mfrY1AqOUGqeZ6olSVKHNOo9oxj1nlFMHTuVh156iOkLpzO412AAnn39Wc7++dkcOeBIjhpwFO/d970cvO/BDO09lKpuVRUeuXZFhmpJktShRQTjho1j3LBxDXX1m+rZp2Yf7n3uXu546o6G+ttOu42JIyby6MuPMvOlmRza71CG9BrC/vvsz8CeA9mtardKvAXtAtokVEfEycB1QBUwLaU0pVF7DXAHMApYBZyVUlpStH0VOA/YBHwhpfRAW4xJkiR1XSNqRzDr3FmklFi1fhUvrHqBv7zxFyYcPgEofclxymNTSKSGbbpFN5b+y1IG7jWQXz3/K55e8TS1PWup7VnL4F6DGdJrCHt237NSb0mdXHaojogq4AZgHLAU+FNEzEgpLSzrdh6wJqV0UERMAKYCZ0XEocAE4DDgPcBDEXFwSmlT7rgkSVLXFxH03aMvfffo2/AkR4BLjr6E80eez6LVi1j61lKWvrWUFWtXUNuzFoD7F9/P9+d8f6t99ajuwbqvrSMimPbnabzy5isM6TWE2p619O7Rm3579GNYn2Ht+v7UebTFmeqjgcUppZcAIuKnwGlAeag+DbiiWL4HuD4ioqj/aUqpHng5IhYX+5vdBuOSJEm7sJrqGg7vfziH9z98m7YbTr2Ba8Zfw4q1K1i+djlL3ljCmvVrKMUTePjlh7n7mbu3umf2IX0P4dlLngXgoz/5KAtWLqD37r3p06MPPbv35MgBR3LliVcCcN3j17F6/WpqqmuoqaqhprqGob2HcsrwUwB48MUH2bh5Y0NbTVUN/fbsx4G9DgRgxdoV7Fa1GzVVNXSv6k51t+qGsaljaotQPQj4a9n6UuADzfVJKW2MiDeBfYv6xxttO6gNxiRJktSi3at3Z3CvwQzuNZgx+43Zqu3OT97JHZ+4g1fefIWV61ay5p01dIt/3DTtxCEnUtuzljXr17B6/WpeW/safXr0aWif9uQ0FqxcsNU+Txl+SkOonvTLSSx7e9lW7WcediZ3nX4XAAdffzBv1b/V0Na9qjvnv/98bjj1BgD2umovgiAiGl4vGX0J3/7It1m7YS1DrhuyTfuXPvglLj/mclasXcHom0cDbNX+teO+xgWjLuDlNS8z/sfjt2oLgitOuIIzDzuTha8vZMI9E7Zpv/LEKzn5oJOZu2wuF/36om3avzvuuxx3wHH84a9/YPJDk7dp/97475Uu61kyi6seu6qhrSqquO/s+3L/uXe6tgjVTf3alFrZpzXblnYQcSFwIcABBxywI+OTJEnaYbtV7cawPsOavOTj8mMub3Hb+RfNJ6XEhk0bqN9UT/3G+q1C+f3n3M+6v68rtW+sp35TPQP2HNDQfs1J17Buw7qG7d+uf5vRg0Y3tF8w8gJSSiRSw+uogaMAqO5WzRmHnrFN+8H7HgyUAvrYoWO3akspsd/e+wGlM/yj3jNqm+1779671F5Vw7A+w7Zp71Hdo+Hntu8e+27TXhX/uCtLVbeqhvrNmzc39APYsGkDb77zZkNd+c+tI4stb+Bd7yDig8AVKaXxxfpXAVJKV5X1eaDoMzsiqoHXgH7A5PK+5f1aOmZdXV2aM2dO1rglSZKklkTE3JRSXWv6tkX0/xMwPCKGRER3Sl88nNGozwxgYrF8OvBIKqX5GcCEiKiJiCHAcOCJNhiTJEmS1G6yL/8orpG+FHiA0i31bk0pPRMR/wHMSSnNAG4BflR8EXE1peBN0e9uSl9q3Ahc4p0/JEmS1NlkX/5RCV7+IUmSpJ2tvS//kCRJknZphmpJkiQpk6FakiRJymSoliRJkjIZqiVJkqRMhmpJkiQpk6FakiRJymSoliRJkjIZqiVJkqRMhmpJkiQpk6FakiRJymSoliRJkjIZqiVJkqRMhmpJkiQpk6FakiRJymSoliRJkjIZqiVJkqRMhmpJkiQpk6FakiRJymSoliRJkjIZqiVJkqRMhmpJkiQpk6FakiRJymSoliRJkjIZqiVJkqRMhmpJkiQpk6FakiRJymSoliRJkjIZqiVJkqRMhmpJkiQpk6FakiRJymSoliRJkjIZqiVJkqRMhmpJkiQpk6FakiRJymSoliRJkjIZqiVJkqRMhmpJkiQpk6FakiRJymSoliRJkjIZqiVJkqRMhmpJkiQpk6FakiRJymSoliRJkjIZqiVJkqRMhmpJkiQpU1aojog+ETEzIhYVr72b6DMiImZHxDMR8XREnFXWdltEvBwR84oyImc8kiRJUiXknqmeDDycUhoOPFysN/Z/wGdTSocBJwP/FRG9ytr/LaU0oijzMscjSZIktbvcUH0acHuxfDvwicYdUkovpJQWFcvLgJVAv8zjSpIkSR1GbqgekFJaDlC89m+pc0QcDXQHXiyr/k5xWci1EVHTwrYXRsSciJjz+uuvZw5bkiRJajvbDdUR8VBELGiinLYjB4qIgcCPgEkppc1F9VeBQ4DRQB/gK81tn1L6QUqpLqVU16+fJ7olSZLUcVRvr0NKaWxzbRGxIiIGppSWF6F5ZTP99gZ+DXwjpfR42b6XF4v1EfFD4PIdGr0kSZLUAeRe/jEDmFgsTwR+2bhDRHQHfgHckVKa3qhtYPEalK7HXpA5HkmSJKnd5YbqKcC4iFgEjCvWiYi6iJhW9DkT+BBwbhO3zvtJRMwH5gN9gW9njkeSJElqd5FSqvQYdlhdXV2aM2dOpYchSZKkLiwi5qaU6lrT1ycqSpIkSZkM1ZIkSVImQ7UkSZKUyVAtSZIkZTJUS5IkSZkM1ZIkSVImQ7UkSZKUyVAtSZIkZTJUS5IkSZkM1ZIkSVImQ7UkSZKUyVAtSZIkZTJUS5IkSZkM1ZIkSVImQ7UkSZKUyVAtSZIkZTJUS5IkSZkM1ZIkSVImQ7UkSZKUyVAtSZIkZTJUS5IkSZkM1ZIkSVImQ7UkSZKUyVAtSZIkZTJUS5IkSZkM1ZIkSVImQ7UkSZKUyVAtSZIkZTJUS5IkSZkM1ZIkSVImQ7UkSZKUyVAtSZIkZTJUS5IkSZkM1ZIkSVImQ7UkSZKUyVAtSZIkZTJUS5IkSZkM1ZIkSVImQ7UkSZKUyVAtSZIkZTJUS5IkSZkM1ZIkSVImQ7UkSZKUyVAtSZIkZTJUS5IkSZmyQnVE9ImImRGxqHjt3Uy/TRExrygzyuqHRMQfi+3viojuOeORJEmSKiH3TPVk4OGU0nDg4WK9KetTSiOK8vGy+qnAtcX2a4DzMscjSZIktbvcUH0acHuxfDvwidZuGBEBfAS4591sL0mSJHUUuaF6QEppOUDx2r+ZfrtHxJyIeDwitgTnfYE3Ukobi/WlwKDM8UiSJEntrnp7HSLiIaC2iaav78BxDkgpLYuIocAjETEfeKuJfqmFcVwIXFisro2I51t57L7A33ZgrNL2OKfU1pxTakvOJ7W1XXlODW5tx+2G6pTS2ObaImJFRAxMKS2PiIHAymb2sax4fSkiZgHvB34G9IqI6uJs9X7AshbG8QPgB9sbbxNjnJNSqtvR7aTmOKfU1pxTakvOJ7U151Tr5F7+MQOYWCxPBH7ZuENE9I6ImmK5L3AssDCllIBHgdNb2l6SJEnq6HJD9RRgXEQsAsYV60REXURMK/q8D5gTEU9RCtFTUkoLi7avAP8aEYspXWN9S+Z4JEmSpHa33cs/WpJSWgWc2ET9HOD8YvkPwBHNbP8ScHTOGFphhy8ZkbbDOaW25pxSW3I+qa05p1ohSldhSJIkSXq3fEy5JEmSlKlThOqIOCMinomIzRFR16jtqxGxOCKej4jxZfUnF3WLI2JyWX2Tj0aPiJpifXHRfuD2jqGuISKuiIhXI2JeUU4pa9vp80u7rubmkbRFRCyJiPnFZ9Ocoq5PRMwsPmdmRkTvoj4i4r+L+fR0RIws28/Eov+iiJhYVj+q2P/iYtto/3epnSkibo2IlRGxoKxup8+h5o7RpaWUOnyh9GXH9wKzgLqy+kOBp4AaYAjwIlBVlBeBoUD3os+hxTZ3AxOK5ZuAi4rli4GbiuUJwF0tHaPSPxNLm86vK4DLm6jf6fPLsuuWluaRxbKlAEuAvo3qvgtMLpYnA1OL5VOA3wIBjAH+WNT3AV4qXnsXy72LtieADxbb/Bb4aKXfs6XN59CHgJHAgvacQ80doyuXTnGmOqX0bEqpqYe9nAb8NKVUn1J6GVhM6YuPRwOLU0ovpZQ2AD8FTit+e2ru0ejlj1y/Bzix6N/cMdT1tcf80q6ryXlU4TGpcyj/PGn8OXNHKnmc0rMgBgLjgZkppdUppTXATODkom3vlNLsVEo+d5TtS11ESun3wOpG1e0xh5o7RpfVKUJ1CwYBfy1b3/Ko8+bqW3o0esM2RfubRf/m9qWu5dLiT123lv2Jqj3ml3ZdfraoNRLwYETMjdKThQEGpJSWAxSv/Yv6Hf3MGlQsN65X19cec6i5Y3RZWbfUa0vRwuPQU0rNPRSmqTN9iaZ/WUgt9G9pXy1to06ipfkF3Aj8J6V/1/8ErgE+R/vML+26nBNqjWNTSssioj8wMyKea6Hvjv5/zDmoxpxDGTpMqE4tPA69BUuB/cvWyx913lT932j+0ehb9rU0IqqBfSj9uaSlY6iTaO38ioibgfuK1faYX9p1+dmi7UopLSteV0bELyhdNrQiIgamlJYXf35fWXRvbk4tBU5oVD+rqN+vif7q+tpjDjV3jC6rs1/+MQOYUNxZYQgwnNIF838Chhd3YuhO6YthM4rrfZp7NHr5I9dPBx4p+jd3DHURxX/sW/wTsOUb0u0xv7TranIeVXhM6kAiYs+I2GvLMnASpc+n8s+Txp8zny3u4DAGeLP4s/sDwEkR0bu4vO0k4IGi7e2IGFN8x+OzZftS19Yec6i5Y3Rdlf6mZGsKpaCzFKgHVlD6h9zS9nVK36B/nrJvLVP6BusLRdvXy+qHUgpGi4HpQE1Rv3uxvrhoH7q9Y1i6RgF+BMwHnqb0ITCwPeeXZdctzc0jiyWlhs+Tp4ryzJY5Qun7GA8Di4rXPkV9ADcU82k+W98t63PF589iYFJZfR2loP4icD3FQ+EsXacAdwLLgb8XWeq89phDzR2jKxefqChJkiRl6uyXf0iSJEkVZ6iWJEmSMhmqJUmSpEyGakmSJCmToVqSJEnKZKiWpA4gIvaNiHlFeS0iXi2W34iIhTvheCdExH3b77nVNrMioq6J+nMj4vq2G50kdT6GaknqAFJKq1JKI1JKI4CbgGuL5RHA5u1tXzypU5JUIYZqSer4qiLi5oh4JiIejIge0HDm+MqI+B1wWUT0i4ifRcSfinJs0e/DZWfBn9zylD6gZ0TcExHPRcRPiieiEREnFv3mR8StEVHTeEARMSkiXiiOfWw7/RwkqcMyVEtSxzccuCGldBjwBvDJsrZeKaUPp5SuAa6jdIZ7dNFnWtHncuCS4sz38cD6ov79wBeBQyk9ve/YiNgduA04K6V0BFANXFQ+mIgYCPw7pTA9rtheknZphmpJ6vheTinNK5bnAgeWtd1VtjwWuD4i5gEzgL2Ls9L/C3wvIr5AKYRvLPo/kVJamlLaDMwr9vve4ngvFH1uBz7UaDwfAGallF5PKW1oNAZJ2iV5DZ4kdXz1ZcubgB5l6+vKlrsBH0wprWdrUyLi18ApwOMRMbaZ/VYD0coxpVb2k6RdgmeqJanreBC4dMtKRIwoXoellOanlKYCc4BDWtjHc8CBEXFQsf4Z4HeN+vwROKG4Y8luwBlt9QYkqbMyVEtS1/EFoC4ini5uw/fPRf0XI2JBRDxF6Xrq3za3g5TSO8AkYHpEzKd055GbGvVZDlwBzAYeAv7c1m9EkjqbSMm/4EmSJEk5PFMtSZIkZTJUS5IkSZkM1ZIkSVImQ7UkSZKUyVAtSZIkZTJUS5IkSZkM1ZIkSVImQ7UkSZKU6f8B8h1Q0q6xAVMAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 864x576 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.figure(figsize=(12,8)); \n",
    "plot_precision_recall_vs_threshold(precisions, recalls, thresholds)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With this chart, you can select the threshold value that gives you the best precision/recall tradeoff for your task.\n",
    "\n",
    "Some tasks may call for higher precision (accuracy of positive predictions). Like designing a classifier that picks up adult contents to protect kids. This will require the classifier to set a high bar to allow any contents to be consumed by children.\n",
    "\n",
    "Some tasks may call for higher recall (ratio of positive instances that are correctly detected by the classifier). Such as detecting shoplifters/intruders on surveillance images - Anything that remotely resemble \"positive\" instances to be picked up.\n",
    "\n",
    "***"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "One can also plot precisions against recalls to assist with the threshold selection"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 85,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAtQAAAHwCAYAAACG+PhNAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJzs3XmcZXdd5//Xp/ZeqrvT6U46ne5OJ2QjwRCYNmGVXRLUxBkViaLiZIwborKM4LgN7uLITzQocVBEBIzOoPlBMCKykwAJIYGQhISsnaTT+95dy72f+eOcqr6pru6+XbdunXu7Xs/H4z76bPfcz6k6qbzP937P90RmIkmSJGlmeqouQJIkSepmBmpJkiSpBQZqSZIkqQUGakmSJKkFBmpJkiSpBQZqSZIkqQUGaknqQBHx8Yj4iSa22xsRZ81FTXMtIjIizi6n3xcRv1N1TQAR8VsR8YFZ2E9ExN9ExI6I+HK57Gcj4sny93py69VKmgsGaknHLSIeiogD5f/0nyxDweJy3acj4mC5bmtE/N+IOO0Y+7skIm6MiJ0RsT0ivhwRPzk3R9OZMvPyzPzbJrZbnJkPtLueiLguIq5p9+fMtsZQ3oFeALwCWJOZl0REP/AnwHeXv9dt1ZYnqVkGakkz9X2ZuRh4NvCdwK81rHt9ue5sYDHwx0faSUQ8F/gP4DPl9icDPwtcPpOiIqJ3Ju9rh06qZRZcBtw4dWFE9FVQy6ypuP4zgIcyc185fyowBNxVXUmSZsJALaklmfkY8HHgGdOs2wn8M3DxUXbxDuBvM/MPM3NrFm7LzFcDRMTrIuLzjW+YpivAX5Qt3PuAt0XEpsYwGxH/OSLuLKd7IuKtEfHtiNgWEddHxPJmjjUiXhwRGyPiV8vW94ci4kcb1k+t5SURMRgRfxwRj5St+X8ZEQsa3nNlRHwtInaXNV1WLv90RPy3cvrsiPhMROwqP/cfjvCzWBoR74+ILRHxcET8WkT0NP4cy1p2RMSDEXF5w35eFxEPRMSecl3jcV0E7MzMjeV2X4iId0bEduC3ym3+a0TcXe77pog4o+H9F0bEJ8pvH56MiF8tl18SETeX30w8ERF/HhEDTfweVkTERxu+0fjcxHFO2e6z5eQd5TcmP9zwO/yViNgE/E1EnFTub0tZ/0cjYk3Dfs4sf/57IuITwIopn/OciPhiWc8dEfHihnWrI+KGss77I+KnyuVXA/8beG5Z24eAe8u37YyI/zjWz0FS5zBQS2pJRKwFXgXcPs26k4H/Atx/hPcuBJ4L/FOLZfwI8LvAMEVr+D7gpVPWf7CcfgPw/cCLgNXADuDahprujIgfOcpnraIIVKcDPwFcFxHnHaGWzwN/CJxLcVFxdvm+3yg/6xLg/cBbgGXAdwEPTfOZvw38G3ASsAb4syPU9mfAUuCs8vh+HGjsOnMpRWhbAfwR8N4oLALeBVyemcPA84CvNbzvVcDHpuznAeAU4Hcj4vuBX6X4Xa8EPgd8qDzGYeDfgX+l+HmfDXyy3E8N+OWynucCLwN+7gjH1uhNwMbys04tPzunbpSZ31VOPrPsQjFxIbIKWE7RQnwNxf8L/6acXwccAP68YVcfBG4r6/xtit875fGdXv5sfqfc55uB/xMRK8tNPlTWuhr4QeD3IuJlmfle4GeAm8vargIuLN+zLDMbz19JnS4zffny5eu4XhShby+wE3gYeDewoFz3aWA/sIsi5HwNWHeE/ZxebnP+UT7rdcDnpyxL4Oxy+n3A+6es/x3gr8vpYYqAfUY5fzfwsoZtTwPGgL4mjvvFwDiwqGHZ9cCvT1cLEOVnP61h2XOBB8vp9wDvPMJnfRr4b+X0+4HrKPraTt0uKUJqLzACXNCw7qeBTzf8HO9vWLewfO8qYFH5u/yBid/jlM/4HPDChv08MmX9x4GrG+Z7ynPgDOAq4PYmz6tfAj5ylN/z75TTbwf+ZWLdMfaZjduVv8NRYOgo77kY2FFOr5vmd/5B4APl9K8Afzfl/TdRhO61FBcNww3rfh9433TnNrC+rPeY56IvX74662ULtaSZ+v7MXJaZZ2Tmz2XmgYZ1b8jMpcBFHGpVnc4OoE4Ralvx6JT5DwL/JSIGKVpNv5qZD5frzgA+Un49v5MiYNcoWjqbsSMP9XmF4oJi9RFqWUkRXG9r+Lx/LZdDEbi+3cRn/neKcP7liLgrIv7rNNusAAbKehprO71hftPERGbuLycXl8fzwxQtpk9ExMci4nyAiFgGnA988QjHCMXP9E8bjnF7We/pRzvGiDi37F6xKSJ2A7/HlO4UR/AOim89/q3spvLWJt7TaEtmHmyoY2FEvKfsJrMb+CywLIpuQ6uZ/nc+4QzghyaOvTz+F1Cc06uB7Zm5Z8p7G38nkk4ABmpJbZOZX6doLb42ImKa9fuBmylaRo9kH0UoBSAiVk33UVP2+02K4HI5T+3uAUUYvLy8GJh4DWXRF7wZJ5VdJCasAx4/Qi1bKboPXNjwWUuzuGFzopanHesDM3NTZv5UZq6maHV+dxw+csVWipb2MxqWrQOaOq7MvCkzX0ERBO8B/qpc9Urgk5lZO8IxThzHT0/5mS7IzC8e4xj/ovysczJzCUXXjcPOk2lq3ZOZb8rMs4DvA94YES9r5jiPUP+bgPOAS8s6JrqKBPAE0//OJzxK0ULdeOyLMvMPKM6L5WW3l8b3NnuuSeoSBmpJ7fa3FH1trzjC+v8OvC4i3lL2uSYinhkRHy7X3wFcGBEXR8QQ5U1wTfggRX/p7wL+sWH5X1L0+z2j/KyVEXHl8RwQ8D8jYiAiXgh875T9T8rMOkUwfWdEnFJ+3ukR8cpyk/cCPxkRL4viZsnTJ1qGG0XEDzXcJLeDIhA2BlzKwHt9eWzD5fG9ETjmeMkRcWpEXFGGxhGK7jwT+/8ephndY4q/pLgZ9MJyf0sj4ofKdR8FVkXEL0Vxg+ZwRFxarhsGdgN7y+P+2WPVWu7/e6O4UTPK99eY8vNo8CRFn/KjGaa48NkZxQ2qvzmxovxm41YO/c5fQBHiJ3wA+L6IeGVE9EbEUHnj45rMfJSiZf/3y+UXAVcDf9/McUrqHgZqSW2VmaMUN7z9+hHWf5HiBsKXAg9EMXLEdZQhLjO/RdFn9t+B+yhu9GvGhyj6y/5HZm5tWP6nwA0U3QX2ALdQ3GQHQNml4kc5sk0UofZximD0M5l5z1G2/xWK7gm3lN0J/p2iNZTM/DLFTYPvpOhz/hme2sI84TuBL0XE3rL2X8zMB6fZ7hcoWvQfoPg5fRD466PUNqGHopX2cYruGi8Cfq4MrK+g6KZyRJn5EYqbLz9cHuM3KIc9LLs7vIIihG6i+B2+pHzrmym+QdhDceHxDzTnHIqf416KbzjenZmfPsK2vwX8bdkd49VH2Ob/AxZQtPLfwuHH+yMU58h2irD9/okVZWi+kqJ1fQtFi/VbOPT/16so+kY/DnwE+M3M/ERzhympW0TmYTdGS5KmUQ6H9oHMPFKf8BNKOQrJn2fmJVXXIkmdzBZqSdLR/OaxN5Gk+a2rn3AlSWqfskuKJOkY7PIhSZIktcAuH5IkSVILDNSSJElSC7quD/WKFSty/fr1VZchSZKkE9xtt922NTNXHmu7rgvU69ev59Zbb626DEmSJJ3gIuLhZrazy4ckSZLUAgO1JEmS1AIDtSRJktQCA7UkSZLUAgO1JEmS1AIDtSRJktQCA7UkSZLUAgO1JEmS1AIDtSRJktQCA7UkSZLUAgO1JEmS1AIDtSRJktQCA7UkSZLUAgO1JEmS1IK2BeqI+OuI2BwR3zjC+oiId0XE/RFxZ0Q8u121SJIkSe3Szhbq9wGXHWX95cA55esa4C/aWIskSZLUFm0L1Jn5WWD7UTa5Enh/Fm4BlkXEae2qp1VP7DpArZ5VlyFJkqQO01fhZ58OPNowv7Fc9kQ15RzZ5+/bymvf+yX6eoLFQ32M15LzVw2zb7TG8FAfpwwPEhEEEAE95TQN05PLA7bsGWHNSQtZMNA7ua7YvJiIoJwqZuLQ5OR2h6Yblsfku56y7e6DY5yxfGGx38l6gp6Y+KyYsg5q9WTF4sGGOg6v8dDnH76scZ4p9TbW2rhtENQyOWlhP709MVnPxM9tYpuehlr7eoPhof4j//IkSZLarMpAHdMsm7YJOCKuoegWwrp169pZ07QuOXM5r3/J2WzbN8ptD2/n1CVD7Nw/xiPb9rFieJCte0fIhMwkgUyoZ5Ll0UxMJ8nmPSOTy/t7Y3J64sAn9kG5HzVnoK+H0fE6AOtPXkhPeVXSc1goLwL56Hidh7ft59KzlgNPvZiIKfMTFwSH1h+6QKhn8uj2Azz7jGX0REx+zuTn9hT/bt4zwvmrho94AdN4kUPD8j0HxycvhibqaLxIm3qRNPEf1VB/L0sX9NPXG/RGcNLCAfr7gt6eYn7igkWSJLWuykC9EVjbML8GeHy6DTPzOuA6gA0bNsx5zBzo6+HNrzxvrj/2KbJM10UwP7RsavhO8ilBfKxWZ+/I+OT7ciLclwG//pQLgWLdnpFxxmt56DOn+YyJFUda13ihMHU/TLPtxPYbd+xn0WDfU2ul4aKkrLVeL/6954k9rBweZKxe595Ne1jQ30t/b8/ke2i4uKlncbyQ3P7ITs5auegpPxsaLogmfo6NPzca1k1M37d5L309wZO7D07uv/HzMuHAWO2Yv9+qnLSw/6mBvOHCoadh+YQte0bo6w2+4/Sl7B0Z58LVSzhp4QArFg/S01NcEEyE9Xo96ekJVi4egCnfLEzsnykXMD0RjNXqnLpkiIG+oK+nh+GhPgb7e+ltuFjp7Tm0P0mSqlZloL4BeH1EfBi4FNiVmR3X3aNTTO1WUS495vuG+nvtEtEBDozWprmAeepFw+TFwuSFAGzfN0pPD9TrT71Yqk9538SFQiZs3Ts62W1nvJ58e/NeFg/2UcukVi9eX39sF6uWDFG+6xgXXOVFRMLND2xj3fKFPLbzADv3j3HX47ur+HEeZsXiAXbsH2Ogt4dzT10ME12amPJNwFO+FSgC+sGxGk/uHuHidcsO375hvrHb03TfWEz8zLfsGeGC1UuO8E1EQ1erhv+m9xwc5+RFAyxZ0E9vBD09TH7j0RPBeL3OaUsX0N9bXGSctmyIkxcNeEEhSR2ibYE6Ij4EvBhYEREbgd8E+gEy8y+BG4FXAfcD+4GfbFctUtUWDPTO6H2rlg7NciWza7xWZ7RWp1YvW+frOdlSv29knANjtWla/A//5oGGZU/sOkh/TzBWTx7auo8F/b2T4b6eWX5GEYTv3bSHU5YMMjqe3PLANs44eeHkNxSHLlamuUioQ1InE776yE5OGR7k7id2T37r8pSLnoYLmUP/TvNtBkULPsBn79v6lG3aZXiwjz0j4zxz7TJ6G74d6AkYryVb9o6w/uRFLF80ULbsF+97bOcBLly9lFOGB1m6oP9Qq3/Z8l+rJ6ctXUBfbxHoi/BevHe6ixJg8qKj8YJjuu0JWDLU58WApBNK2wJ1Zl51jPUJ/Hy7Pl9S+/X19tDXO/1gQSuHB+e4ms419ZuI+pSgvm9knJHx4sJkortQLYtuVwdG6+w+OEY9k7FanQe37ufhbfuo1ZP7ntzLyHiN4cHiT3lxMZPU67B17wgbdxxg30iNBQM91OtFUN60+yAAX7h/W0U/jcJktx0m7nc4/KbjCNh9cBwo7o14ys3MU26MnnqD88Q0k8unbNPwjcPBsRobdxzgeU87md6e4OB4nbHxOuetGp68EOnpCXbuG+OC1UvYP1pj3fKFDPT10FdeiPRGlDdJ93Ha0gUsXzTQxp+epE5TZZcPSZoXJvqnl3OHrR/qn9k3GDOVmew6MMa+0drktwq1hpb/XQfGyISxep2Htu5jURnYmdIiP/XejcZ7EcrJKa35cN+Te1ixeHDyPRMXF0+93+HQPu96fBcrFg/SX164HXZfxpT7Gqa7n+Sp3xRMXZ58+cHtnHHyQp7cc5DxWnLPpj309gTffGL3ZDep470Xor9s3Z+4EXjPSHFhcNaKRU+5QbqnPDF6yq4+EzdMP/dpJxdhvVz/5J4Rnr5qmEWDfYyO1zmjvPm6Jyi/WSheo+M1Fg/1s2LxAAmsWjJUXrzE5P5WLxs64oWwpJkxUEvSPBMRLFs4wLKFTWxc7f3YHaNeT/aNjjM6XmfbvlHGavWi1T+TWr3O1r2jZBY3Vm/eM0JvT3Fjbq1efNvw1Yd3sGZ5EYLr5bcPjTdLT1x83PbIDs5csYjt+0Ynw/xjOw6wZ2Scr2/cyWw9DqG3J1g00Mu+0RqrlgxxyZnLD4XuHti8e4QLT19KX09xUTA6XmfRYC+nDA/R1xvUE04dHiwuGMpXX09xD4NhXfORgVqSpGPo6Tk05v3Ji6vrzjTRWl7PJOuHugdNtO7vOTjGwbE6I+N1Nu7Yz0Bvz+QNyfVM7nh0F7sOFDfw3vLgNg6MHWC8XufWh7cf1i3ok/dsnlGNC/p7GRmvUU942spFDSPzBLV6snXvCGeuKPr2j9eTAE5ZMsj3fMdqzjh5ISsWD5YBvehOI3WDyJyly905smHDhrz11lurLkOSpBNa5qEW9n0jNfaNjDNWq7Pn4Dj7RoshRyda0Udrdb54/1YWDfYxXk++vnFX0Y88mGypr2fxLIZte0dZPNhHBNyzaU9TtSwe7OOC05YwWquz+8AYr3zGKl507kr6e4Penh56I+jvC5YtGJgM4729wfCgN8CqNRFxW2ZuOOZ2BmpJklSV0fE6X39sFxt37Oe+J/eyeKiPWr24Cfdrj+7kke37OXV4iLFanVsf3jHjz/lPZ5zEosE+Fk90XekJNqw/iaH+XtactICB3l77l+swBmpJknTCqdWT2x/Zwch4nfF6TvZv37x7hKH+3qKLS63OvU/uob+3h1se2MbBsTo9AQ9t20/EsYezXHPSAi5/xiouXL2Us09ZzClLBhns7WV4qM9uKPNMs4HaPtSSJKlr9PYEG9Yvb2kfo+N1Htm+j90Hx3lk2356eoL7ntzDf9yzmZ37x9i44wB/9bkHj/j+55y1nLUnLeS8VcOct2qYS888mYE+W7bnM1uoJUmSGtTqyYNb9/H4zgPct3kvA3093PPEbv7+S4+wYvEgW/eOTPu+JUN9fPeFqzhleJDliwZYt3whLz3/FLuRdDG7fEiSJLXJjn2jPLhtH3c9vpuPf/0JvvjtIz8sadnCfl7x9FNZNNjH008b5pIzT2bpgn6WDPUZtjucgVqSJGmOjdXq7Ng/ysYdB7jpG5v4zLe2FE823bz3iO+JgFc8/VTOXLmIpQv6GR2vc/HaZbzwnJX02me7UgZqSZKkDlGrJ/dv3ss9m3azde8o+0aKoQe/9ugO9o3WuHfTHnYdGJv2vauXDrFt3ygJvOkV53L6SQvIhAtWLymfvmnobhdvSpQkSeoQvT0xeRPjkdTqych4rQjeT+zhM9/awuLBPu58bBeP7yoeuPP7H7/nsPcN9vXw+peczS+87Jy21a+js4VakiSpC+w5OMa2vaMcGKvx7S17+cAtD3PLA9ufss3wUB+//PJz+YH/tIalC/orqvTEYZcPSZKkeWDHvlGuvPYLPLJ9/2HrVi8d4kXnreT3/8tFFVTW/QzUkiRJ88yeg2N8/Oub+KevbqRWT24rny558dpl/Mmrn8lZKxdXXGF3MVBLkiTNc5t2HeSKP/88m/cUY2c/a90y/uGa5/ogmiY1G6j9aUqSJJ2gVi0d4sv/4+X87x8vMuHtj+zk3F/7OO/5zLd5dJouIpoZW6glSZLmgXo9+W/vv5X/uGfzU5b/0Q9cxKu/c21FVXU2u3xIkiTpMLV68rGvP8Evffh26mUMvGjNUq7/6ecy1N9bbXEdxkAtSZKko9p9cIyLfuvfJuevumQtP/uis1l38sIKq+oc9qGWJEnSUS0Z6uf+372cF56zgovWLOVDX36U73rHp/jI7Rup17ur0bVKtlBLkiQJgH/52mP84oe/Njn/lleex8+/5OwKK6qWLdSSJEk6LldefDqfevOLuXD1EgDecdO9PLBlb8VVdT4DtSRJkiaduWIRH3vDC3nr5ecD8NL/9ZmKK+p8BmpJkiQd5nXPWz85/SN/dQvd1k14LhmoJUmSdJih/l7++eefD8AXv72NM992I996ck/FVXUmA7UkSZKmdfHaZdz+66+YnP/ud362wmo6l4FakiRJR3TSogEe+oPvmZxf/9aP8bn7tlRYUecxUEuSJOmY/v2NL5qc/rH3frnCSjqPgVqSJEnHdPYpiw9rqfZGxYKBWpIkSU27+W0vnZw+8203cnCsVmE1ncFALUmSpKadtnQB33z7Kyfnz//1f62wms5goJYkSdJxWTjQx72/c9nk/L9+Y1OF1VTPQC1JkqTjNtjXyy+//FwAfuYDt7F932jFFVXHQC1JkqQZ+cWXn8P6kxcC8Ozf/kTF1VTHQC1JkqQZ+/9/4QWT07c8sK3CSqpjoJYkSdKMDQ/18+bvLrp+vOa6W9h1YKziiuaegVqSJEktef1Lz+GslYsA+P5rv1BxNXPPQC1JkqSWXf/TzwXgwa37GBmfX2NTG6glSZLUshWLB7nimasB+N2P3V1xNXPLQC1JkqRZ8ZrvXAvA+29+mC9+e2vF1cwdA7UkSZJmxfPOXsH/vOJCAH7kr75UcTVzx0AtSZKkWfMTz1s/Of2Nx3ZVV8gcMlBLkiRpVv3RD14EwPf+2eep17PiatrPQC1JkqRZ9eoNayenP/r1JyqsZG4YqCVJkjTrPlo+QfHf7tpUcSXtZ6CWJEnSrHv6aUsA+OidtlBLkiRJx623Jyan79y4s8JK2s9ALUmSpLb44x96JgCPbN9fcSXtZaCWJElSW6xbvhCA2x7eUXEl7WWgliRJUluctnQIgL/5wkPsHRmvuJr2MVBLkiSpLdYuX8jpyxYA8IzfvKniatrHQC1JkqS2+dSbXzw5/db/c2d1hbSRgVqSJEltM9DXw41veCEAH/7Ko+w+OFZxRbPPQC1JkqS2umD1El71HasAeO7vfbLiamafgVqSJEltNzGE3r7RGvV6VlzN7DJQS5Ikqe0WDvRx1spFAGzccaDiamaXgVqSJElz4uoXnAnA3Zt2V1zJ7DJQS5IkaU6cc8owACPj9YormV0GakmSJM2JhQO9AIzXDNSSJEnScevrDQDeeP0dFVcyuwzUkiRJmhPnr1oCwDPXLqu4ktlloJYkSdKceda6Zdzx6M6qy5hVBmpJkiTNmYe37a+6hFlnoJYkSdKcufLi1QCMnkAjfRioJUmSNGc27x4B4B++8kjFlcweA7UkSZLmzNUvLB7uMlo7cR4/bqCWJEnSnDlt6RAAi8oxqU8EBmpJkiTNmZGxou/0QN+JE0NPnCORJElSx6tl0dXj3if3VFzJ7GlroI6IyyLi3oi4PyLeOs36dRHxqYi4PSLujIhXtbMeSZIkVWv5wgEA9hwcr7iS2dO2QB0RvcC1wOXABcBVEXHBlM1+Dbg+M58FvAZ4d7vqkSRJUvUWD/VVXcKsa2cL9SXA/Zn5QGaOAh8GrpyyTQJLyumlwONtrEeSJEkV6+8t4ufD2/ZVXMnsaeclwunAow3zG4FLp2zzW8C/RcQvAIuAl7exHkmSJHWIL9y/reoSZk07W6hjmmVTBxy8CnhfZq4BXgX8XUQcVlNEXBMRt0bErVu2bGlDqZIkSdLMtDNQbwTWNsyv4fAuHVcD1wNk5s3AELBi6o4y87rM3JCZG1auXNmmciVJkjQXnr1uWdUlzKp2BuqvAOdExJkRMUBx0+ENU7Z5BHgZQEQ8nSJQ2wQtSZJ0Artw9VIAxmr1iiuZHW0L1Jk5DrweuAm4m2I0j7si4u0RcUW52ZuAn4qIO4APAa/LzBPnOZSSJEk6zESQHhk/MQJ1W8ctycwbgRunLPuNhulvAs9vZw2SJEnqLMvKsajveWI3G9Yvr7ia1vmkREmSJM2pc05ZDMBffuaBiiuZHQZqSZIkzakrL14NQF/PdIPCdR8DtSRJkuZUX28PZ5y8kMH+EyOKnhhHIUmSpK4yMlZnsO/EiKInxlFIkiSpq+w+OMZHbn+s6jJmhYFakiRJc27/aI2ViwerLmNWGKglSZI051583koe33Ww6jJmhYFakiRJc26sVmfhQG/VZcwKA7UkSZLm3Lrli1g02NZnDM4ZA7UkSZLm3Mh4jS17RqouY1YYqCVJkjTn7ntyLwC3PLCt4kpaZ6CWJEnSnPvpF50FwIe+/EjFlbTOQC1JkqQ59z3fcRpQDJ/X7QzUkiRJmnMRAcBAb/fH0e4/AkmSJHWl1UuHWHACDJ1noJYkSVIl9o3WuPnb3X9T4okx+J8kSZK6zq4DY+w6MFZ1GS2zhVqSJEmVuGT98qpLmBUGakmSJFXiojVLAdi062DFlbTGQC1JkqRK9PcVUfRrj+6ouJLWGKglSZJUiReduxKAnfu7ux+1gVqSJEmVWLF4EIA7Nu6suJLWGKglSZJUiVOXFIH6o3c8UXElrTFQS5IkqRLDQ/0sHuzj/NOGqy6lJQZqSZIkVea8VcOMjterLqMlPthFkiRJldm2d4QnunzYPAO1JEmSKrNyeJCeiKrLaIldPiRJklSZZQsHGOjr7kja3dVLkiSpq43V6tyzaU/VZbTEQC1JkqTKbNs7CsDug937cBcDtSRJkipz/qpiyLz7ntxbcSUzZ6CWJElSZV5xwakA3P3E7oormTkDtSRJkiqz5qSFAHz63s0VVzJzBmpJkiRVZqLLx+LB7h3N2UAtSZKkyvT0BCsWD3LX43b5kCRJkmZk694Rli8aqLqMGTNQS5IkqVLPWresqx/u0r2VS5Ik6YQwVqsz0Nu9sbR7K5ckSdIJYWSszp6D41WXMWPdezulJEmSTghb946w80D3PinRQC1JkqRKrVu+kFpm1WXMmF0+JEmSVKklC/rtQy1JkiTN1Oh4nb4uDtR2+ZAkSVKlRmt17t+8t+oyZqx7LwUkSZJ0Qnh85wEW9PdWXcaMGaglSZJUqUvPPJlFg93bccJALUmSpEqN1eo8uHVf1WXMmIFakiRJldq0+2DVJbTEQC1JkqRKbTjjJBYN2IdakiRJmpE8VaaXAAAZmUlEQVSxWnb1sHndW7kkSZJOCKO1Ov0GakmSJGlmxsbrDPRG1WXMmIFakiRJlTowVuPxXQep17PqUmbEQC1JkqSO0J1x2kAtSZKkiq0/eRE9Ab093dntw0AtSZKkSo15U6IkSZI0c6O1OgN93RtLu7dySZIknRDGanUGbKGWJEmSZmZsPO3yIUmSJM3UWK3Opt0Hqy5jxgzUkiRJqtTGHQcAeHDrvoormRkDtSRJkip1+XesAuA9n/l2xZXMjIFakiRJlfqJ564H4OTFA9UWMkMGakmSJHWE7fvGqi5hRgzUkiRJqtTEI8c/dufjldYxUwZqSZIkVWrikeOrlg5VXMnMGKglSZJUuZeefwqDfb1VlzEjBmpJkiRVbqxWp683qi5jRgzUkiRJqtxYrc7tj+ysuowZMVBLkiSpctv3jTI81Fd1GTPS1kAdEZdFxL0RcX9EvPUI27w6Ir4ZEXdFxAfbWY8kSZI607PXncSC/u7sQ922y4CI6AWuBV4BbAS+EhE3ZOY3G7Y5B3gb8PzM3BERp7SrHkmSJHWusVrS39udnSfaWfUlwP2Z+UBmjgIfBq6css1PAddm5g6AzNzcxnokSZLUoQ6MjfPYzgPU6nnsjTtMOwP16cCjDfMby2WNzgXOjYgvRMQtEXFZG+uRJElShxooW6ef2HWg4kqOXzsD9XTjnky95OgDzgFeDFwF/O+IWHbYjiKuiYhbI+LWLVu2zHqhkiRJqtZFa4oI+Kl7uq/DQjsD9UZgbcP8GmDq8yQ3Av+SmWOZ+SBwL0XAforMvC4zN2TmhpUrV7atYEmSJFXjJecXt9ItHOi+kT7aGai/ApwTEWdGxADwGuCGKdv8M/ASgIhYQdEF5IE21iRJkqQONNF3uhsf7tK2QJ2Z48DrgZuAu4HrM/OuiHh7RFxRbnYTsC0ivgl8CnhLZm5rV02SJEnqTBOB+u4n9lRcyfFra5t6Zt4I3Dhl2W80TCfwxvIlSZKkeerMFYsAuOvxXRVXcvy6c7A/SZIknVD6y64ei+xDLUmSJB2/iOCsFYvsQy1JkiTN1Fi93pVPS+y+iiVJknRCGq8lfT22UEuSJEkzMlZL+myhliRJkmbmwOg4n7uv+56KbaCWJElSRxirJ6ctHaq6jONmoJYkSVJHeOaapfT1dF887b6B/iRJknRCGqsl927ywS6SJEnSjDy+8wCZVVdx/AzUkiRJ6ggvPm8liwa7rwPFcQfqiOiJiCXtKEaSJEnz13gt2bT7YNVlHLemAnVEfDAilkTEIuCbwL0R8Zb2liZJkqT55NEd+wH41pN7Kq7k+DTbQn1BZu4Gvh+4EVgH/FjbqpIkSdK886OXngHA9n2jFVdyfJoN1P0R0U8RqP8lM8eALuwyLkmSpE518uKBqkuYkWYD9XuAh4BFwGcj4gxgd7uKkiRJ0vwzMcLHHY/urLaQ49TUbZSZ+S7gXQ2LHo6Il7SnJEmSJM1H560aBui6kT6aqjYiBoEfANZPec/b21CTJEmS5qFavWii3rjjQMWVHJ9m4/+/ALuA24CR9pUjSZKk+WrJgn4Adh3orpsSmw3UazLzsrZWIkmSpHlt8WAfiwf7GOzrrbqU49LsTYlfjIjvaGslkiRJmvf2jY7ziW8+WXUZx6XZFuoXAK+LiAcpunwEkJl5UdsqkyRJ0ryzaKCP009aUHUZx6XZQH15W6uQJEmSKEb66O+Nqss4Lk11+cjMh4FlwPeVr2XlMkmSJGnWjNeTvp5meyV3hqaqjYhfBP4eOKV8fSAifqGdhUmSJGn+qdXr9PV0Vwt1s10+rgYuzcx9ABHxh8DNwJ+1qzBJkiTNP+O1pLfLAnWz7ekB1Brma+UySZIkadbU6klfl/WhbraF+m+AL0XER8r57wfe256SJEmSNF+N1eo8vG1/1WUcl6YCdWb+SUR8mmL4vAB+MjNvb2dhkiRJmn8e33WQJUPNtvl2hqNWGxFLMnN3RCwHHipfE+uWZ+b29pYnSZKk+eS7zlnBt7fsq7qM43Ks+P9B4HuB24BsWB7l/FltqkuSJEnzUH9vT9eNQ33UQJ2Z31v+e+bclCNJkqT5bLye9J6g41A/PyIWldOvjYg/iYh17S1NkiRJ802tnl03DnWz8f8vgP0R8UzgvwMPA3/XtqokSZI0L43Xk68/tqvqMo5Ls4F6PDMTuBL408z8U2C4fWVJkiRpPtq6Z6TqEo5bs4F6T0S8DXgt8LGI6AX621eWJEmS5qOXnL+y6hKOW7OB+oeBEeDqzNwEnA68o21VSZIkaV7afWAcgEe3d8/DXZp9sMsm4E8a5h8B3t+uoiRJkjQ/PeP0JQDs2D/K2uULK66mOUdtoY6Iz5f/7omI3Q2vPRGxe25KlCRJ0nyxZKjoVXz7IzsrrqR5xxqH+gXlv96AKEmSpLa7aO0yAIb6u2cs6mbHoX5ORAw3zC+OiEvbV5YkSZLmo4kRqD/zrS2V1nE8jmcc6r0N8/vLZZIkSdKsOW3pEABLF3TPgHLNBuoox6EGIDPrNHlDoyRJktSsiGD5ogF6u+hpic0G6gci4g0R0V++fhF4oJ2FSZIkaX46MFrjX7/xZNVlNK3ZQP0zwPOAx4CNwKXANe0qSpIkSfNXX2+wde8IDR0kOlqz41BvBl7T5lokSZIknrF6KTc/sI2R8TpD/b1Vl3NMzY7ycW5EfDIivlHOXxQRv9be0iRJkjQfPf/skwHoie7oR91sl4+/At4GjAFk5p3YYi1JkqQ2qNWLf7vlxsRmA/XCzPzylGXjs12MJEmSVC/7TndJnm46UG+NiKcBCRARPwg80baqJEmSNG/VM4kohtDrBs2OJf3zwHXA+RHxGPAg8KNtq0qSJEnzVq2e9HZJmIYmAnVE9AAbMvPlEbEI6MnMPe0vTZIkSfNRLZOebunvQRNdPsqnIr6+nN5nmJYkSVI71bushbrZPtSfiIg3R8TaiFg+8WprZZIkSZqXavXuGeEDmu9D/V8pbkj8uSnLz5rdciRJkjTf1TPZOzJOZnbFjYnNtlBfAFwL3AF8Dfgz4MJ2FSVJkqT5a+f+UQAe2ra/4kqa02yg/lvg6cC7KML008tlkiRJ0qx66dNPBWB0vF5xJc1ptsvHeZn5zIb5T0XEHe0oSJIkSQLobbbpt2LNlnl7RDxnYiYiLgW+0J6SJEmSNJ/l5JMSO7//NDTfQn0p8OMR8Ug5vw64OyK+DmRmXtSW6iRJkjTv1OonZqC+rK1VSJIkSaWJQN0tQ+c1Fagz8+F2FyJJkiRBMWwe0DVPS+ySrt6SJEmaL2rl4B7d8rREA7UkSZI6Sm2yhbriQprUJWVKkiRpvpgY5cMWakmSJGkGum2UDwO1JEmSOspkoPamREmSJOn4jZeB+sGt+yqupDkGakmSJHWUs1YsAmDTroMVV9IcA7UkSZI6yuplCwDokh4f7Q3UEXFZRNwbEfdHxFuPst0PRkRGxIZ21iNJkqTON/Fgl9se3lFxJc1pW6COiF7gWuBy4ALgqoi4YJrthoE3AF9qVy2SJEnqHmetXAzAyHi94kqa084W6kuA+zPzgcwcBT4MXDnNdr8N/BHQHZ1kJEmS1FaLB/sYHuxz2DzgdODRhvmN5bJJEfEsYG1mfrSNdUiSJKnL1DLtQw1M9yPIyZURPcA7gTcdc0cR10TErRFx65YtW2axREmSJHWieia9XZKo2xmoNwJrG+bXAI83zA8DzwA+HREPAc8BbpjuxsTMvC4zN2TmhpUrV7axZEmSJHWCeh3CLh98BTgnIs6MiAHgNcANEyszc1dmrsjM9Zm5HrgFuCIzb21jTZIkSeoCRQt11VU0p21lZuY48HrgJuBu4PrMvCsi3h4RV7TrcyVJktT96pldc1NiXzt3npk3AjdOWfYbR9j2xe2sRZIkSd0hM6mnXT4kSZKkGSmf60KvgVqSJEk6frUyUXfJIB8GakmSJHWWiUeP93RJojZQS5IkqaPUyyeOd8tNiQZqSZIkdZSJFuqxWr3iSppjoJYkSVJHmXhC4j9/7bGKK2mOgVqSJEkdZai/lyVDfZw6PFR1KU0xUEuSJKnjnHPq8GRLdaczUEuSJKnj1OpJl9yTaKCWJElS58lMW6glSZKkmaplOmyeJEmSNFP1uuNQS5IkSTNWz/TR45IkSdJM1e3yIUmSJM1cPfGmREmSJGmm6g6bJ0mSJM1c3WHzJEmSpJlz2DxJkiSpBQ6bJ0mSJLXAYfMkSZKkFjhsniRJktSCekJPlzRRG6glSZLUcUbH63z5wW1Vl9GUvqoLkCRJkqbadWCMA2O1qstoii3UkiRJ6jivfc46FvT3Vl1GUwzUkiRJ6kh99qGWJEmSZqaeEI7yIUmSJM1Mve441JIkSdKM1TPZvGeE8Vq96lKOyUAtSZKkjnNwrAjS77jp3oorOTYDtSRJkjrO7/7nZwCwY/9oxZUcm4FakiRJHWd4qJ9TlwwSdH5HagO1JEmSOlIm9HRBWu2CEiVJkjQf1ROwhVqSJEmaqaQbhqI2UEuSJKkjZXZD+7SBWpIkSR2qnklPFzRRG6glSZLUkRLs8iFJkiTNlF0+JEmSpBZkJtEFTdQGakmSJHWkTLt8SJIkSTOW4JMSJUmSpJnKTHo6P08bqCVJktSZ6nb5kCRJkmYu8aZESZIkaca8KVGSJElqQTEOdecnagO1JEmSOlLR5aPqKo7NQC1JkqSOlImjfEiSJEkzVc+0y4ckSZI0U4k3JUqSJEkzVozy0fmJ2kAtSZKkjpOZAF3Q4cNALUmSpA5U5mm7fEiSJEkzUeZperogURuoJUmS1HHqdvmQJEmSZs4uH5IkSVILsuz04SgfkiRJ0gzYQi1JkiS1YDJQd0EvagO1JEmSOs5El4+ezs/TBmpJkiR1nrpdPiRJkqSZO/SkxM5P1AZqSZIkdZyJB7vYQi1JkiTNQNaLfx02T5IkSZqByXGoK66jGQZqSZIkdZyJYfMc5UOSJEmagXr6pERJkiRpxrwpUZIkSWrBoScldr62BuqIuCwi7o2I+yPirdOsf2NEfDMi7oyIT0bEGe2sR5IkSd1h8qbELmiiblugjohe4FrgcuAC4KqIuGDKZrcDGzLzIuCfgD9qVz2SJEnqHumTEgG4BLg/Mx/IzFHgw8CVjRtk5qcyc385ewuwpo31SJIkqUsc6vLR+Ym6nYH6dODRhvmN5bIjuRr4eBvrkSRJUpeY6PLRDcPm9bVx39Mdfk6zjIh4LbABeNER1l8DXAOwbt262apPkiRJHapulw+gaJFe2zC/Bnh86kYR8XLgfwBXZObIdDvKzOsyc0Nmbli5cmVbipUkSVLnyIlxqOd5l4+vAOdExJkRMQC8BrihcYOIeBbwHoowvbmNtUiSJKmLeFMikJnjwOuBm4C7gesz866IeHtEXFFu9g5gMfCPEfG1iLjhCLuTJEnSPHIoUHd+om5nH2oy80bgxinLfqNh+uXt/HxJkiR1p8lxqCuuoxk+KVGSJEkdZ6KFuqcL0moXlChJkqT5pu5NiZIkSdLMTYy13AVdqA3UkiRJ6jwTw+Y9tHX/MbasnoFakiRJHWfhQDF2xjv//VsVV3JsBmpJkiR1nNXLFnD5M1ZVXUZTDNSSJEnqSGefspge+1BLkiRJJzYDtSRJktQCA7UkSZLUAgO1JEmS1AIDtSRJktQCA7UkSZLUAgO1JEmS1AIDtSRJktQCA7UkSZLUAgO1JEmS1AIDtSRJktQCA7UkSZLUAgO1JEmS1AIDtSRJktQCA7UkSZLUAgO1JEmS1AIDtSRJktQCA7UkSZLUAgO1JEmS1AIDtSRJktQCA7UkSZLUAgO1JEmS1AIDtSRJktQCA7UkSZLUAgO1JEmS1AIDtSRJktQCA7UkSZLUAgO1JEmS1AIDtSRJktQCA7UkSZLUAgO1JEmS1AIDtSRJktQCA7UkSZLUAgO1JEmS1AIDtSRJktQCA7UkSZLUAgO1JEmS1AIDtSRJktQCA7UkSZLUAgO1JEmS1AIDtSRJktQCA7UkSZLUAgO1JEmS1AIDtSRJktQCA7UkSZLUAgO1JEmS1AIDtSRJktQCA7UkSZLUAgO1JEmS1AIDtSRJktQCA7UkSZLUAgO1JEmS1AIDtSRJktQCA7UkSZLUAgO1JEmS1AIDtSRJktQCA7UkSZLUAgO1JEmS1AIDtSRJktQCA7UkSZLUAgO1JEmS1AIDtSRJktQCA7UkSZLUgrYG6oi4LCLujYj7I+Kt06wfjIh/KNd/KSLWt7MeSZIkaba1LVBHRC9wLXA5cAFwVURcMGWzq4EdmXk28E7gD9tVjyRJktQO7WyhvgS4PzMfyMxR4MPAlVO2uRL423L6n4CXRUS0sSZJkiRpVrUzUJ8OPNowv7FcNu02mTkO7AJObmNNkiRJ0qxqZ6CerqU5Z7ANEXFNRNwaEbdu2bJlVoqTJElSZ1u1dIiL1y6ruoxjameg3gisbZhfAzx+pG0iog9YCmyfuqPMvC4zN2TmhpUrV7apXEmSJHWSH730DP7vzz2/6jKOqZ2B+ivAORFxZkQMAK8BbpiyzQ3AT5TTPwj8R2Ye1kItSZIkdaq+du04M8cj4vXATUAv8NeZeVdEvB24NTNvAN4L/F1E3E/RMv2adtUjSZIktUPbAjVAZt4I3Dhl2W80TB8EfqidNUiSJEnt5JMSJUmSpBYYqCVJkqQWGKglSZKkFhioJUmSpBYYqCVJkqQWGKglSZKkFhioJUmSpBYYqCVJkqQWGKglSZKkFhioJUmSpBYYqCVJkqQWGKglSZKkFhioJUmSpBYYqCVJkqQWGKglSZKkFkRmVl3DcYmILcDDVdcxj6wAtlZdhDqa54iOxXNER+P5oWOp8hw5IzNXHmujrgvUmlsRcWtmbqi6DnUuzxEdi+eIjsbzQ8fSDeeIXT4kSZKkFhioJUmSpBYYqHUs11VdgDqe54iOxXNER+P5oWPp+HPEPtSSJElSC2yhliRJklpgoBYAEXFZRNwbEfdHxFunWf/GiPhmRNwZEZ+MiDOqqFPVONb50bDdD0ZERkRH342t2dfMORIRry7/jtwVER+c6xpVrSb+P7MuIj4VEbeX/695VRV1qhoR8dcRsTkivnGE9RER7yrPnzsj4tlzXePRGKhFRPQC1wKXAxcAV0XEBVM2ux3YkJkXAf8E/NHcVqmqNHl+EBHDwBuAL81thapaM+dIRJwDvA14fmZeCPzSnBeqyjT5d+TXgOsz81nAa4B3z22Vqtj7gMuOsv5y4JzydQ3wF3NQU9MM1AK4BLg/Mx/IzFHgw8CVjRtk5qcyc385ewuwZo5rVHWOeX6UfpviQuvgXBanjtDMOfJTwLWZuQMgMzfPcY2qVjPnSAJLyumlwONzWJ8qlpmfBbYfZZMrgfdn4RZgWUScNjfVHZuBWgCnA482zG8slx3J1cDH21qROskxz4+IeBawNjM/OpeFqWM08zfkXODciPhCRNwSEUdridKJp5lz5LeA10bERuBG4BfmpjR1iePNKnOqr+oC1BFimmXTDv8SEa8FNgAvamtF6iRHPT8iogd4J/C6uSpIHaeZvyF9FF/VvpjiG67PRcQzMnNnm2tTZ2jmHLkKeF9m/q+IeC7wd+U5Um9/eeoCTWeVKthCLSiu8tY2zK9hmq/aIuLlwP8ArsjMkTmqTdU71vkxDDwD+HREPAQ8B7jBGxPnlWb+hmwE/iUzxzLzQeBeioCt+aGZc+Rq4HqAzLwZGAJWzEl16gZNZZWqGKgF8BXgnIg4MyIGKG4GuaFxg/Ir/fdQhGn7Ps4vRz0/MnNXZq7IzPWZuZ6ij/0VmXlrNeWqAsf8GwL8M/ASgIhYQdEF5IE5rVJVauYceQR4GUBEPJ0iUG+Z0yrVyW4Afrwc7eM5wK7MfKLqoibY5UNk5nhEvB64CegF/joz74qItwO3ZuYNwDuAxcA/RgTAI5l5RWVFa840eX5oHmvyHLkJ+O6I+CZQA96Smduqq1pzqclz5E3AX0XEL1N8lf+69Olz80ZEfIiiS9iKsh/9bwL9AJn5lxT96l8F3A/sB36ymkqn55MSJUmSpBbY5UOSJElqgYFakiRJaoGBWpIkSWqBgVqSJElqgYFakiRJaoGBWpLmqYhYHxHfKKdfHBE+Ol6SZsBALUldpnywgX+/JalD+AdZkrpA2Zp8d0S8G/gq8GMRcXNEfDUi/jEiFpfbfWdEfDEi7oiIL0fEcPnez5XbfjUinneMz3pRRHytfN0eEcNzcYyS1K0M1JLUPc4D3g+8ArgaeHlmPhu4FXhj+UjnfwB+MTOfCbwcOABsBl5RbvvDwLuO8TlvBn4+My8GXljuQ5J0BD56XJK6x8OZeUtEfC9wAfCFiAAYAG6mCNxPZOZXADJzN0BELAL+PCIupnjs97nH+JwvAH8SEX8P/N/M3NiWo5GkE4SBWpK6x77y3wA+kZlXNa6MiIuAnOZ9vww8CTyT4pvJg0f7kMz8g4j4GPAq4JaIeHlm3tNq8ZJ0orLLhyR1n1uA50fE2QARsTAizgXuAVZHxHeWy4cjog9YStFyXQd+DOg92s4j4mmZ+fXM/EOK7iTnt/FYJKnrGaglqctk5hbgdcCHIuJOioB9fmaOUvSR/rOIuAP4BDAEvBv4iYi4haK7x75pd3zIL0XEN8p9HAA+3p4jkaQTQ2RO9+2gJEmSpGbYQi1Jkv5fu3UsAAAAADDI33oQe4siYBBqAAAYhBoAAAahBgCAQagBAGAQagAAGIQaAAAGoQYAgCHY2qTPT8A+swAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 864x576 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.figure(figsize=(12,8)); \n",
    "plt.plot(precisions, recalls);\n",
    "plt.xlabel('recalls');\n",
    "plt.ylabel('precisions');\n",
    "plt.title('PR Curve: precisions/recalls tradeoff');"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Setting High Precisions\n",
    "\n",
    "Let's aim for 90% precisions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 86,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "53685"
      ]
     },
     "execution_count": 86,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(precisions)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 87,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "53684"
      ]
     },
     "execution_count": 87,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(thresholds)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 88,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAsYAAAHVCAYAAADywj0dAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJzt3Xd4XOWB9v/7Ube6ZRXLKpZ7t3G3wYAhJpgOCSRAshBqevImu282u9mUTXIlv7TN+yZLCvCG3knAxDGYXo1775aLmtW7rDqa5/eHBlYY2R7bIz1Tvp/rOpfmnHk0cxuOR7ePnnOOsdYKAAAAiHRRrgMAAAAAwYBiDAAAAIhiDAAAAEiiGAMAAACSKMYAAACAJIoxAAAAIIliDAAAAEiiGAMAAACSKMYAAACAJCnG1RtnZmbaoqIiV28PAACACLFp06Y6a23WqcY5K8ZFRUXauHGjq7cHAABAhDDGlPgzjqkUAAAAgCjGAAAAgCSKMQAAACCJYgwAAABIohgDAAAAkijGAAAAgCSKMQAAACCJYgwAAABIohgDAAAAkijGAAAAgCSKMQAAACCJYgwAAABIohgDAAAAkijGAAAAgCQ/irEx5i/GmBpjzM4TPG+MMb8zxhQbY7YbY+YEPiYAAAAwuPw5YvygpOUnef4ySRN8y92S/nj2sQAAAIChFXOqAdbat40xRScZco2kh621VtJaY0y6MSbXWlsZoIwAAIQ9a61aOj2uYwCDKm1YrOsIJ3XKYuyHPEll/dbLfdsoxgCAoOLp9aqiqUPRUUax0VGKjjKSpEO1x2StVUx0lGKjjaKjjGKiorSjollG0qodlapo6pAxRlFGivJ9NSf42ve8kTnJ2OPHrD1Ur5rWLrf/gYBBFGWkQz+/wnWMkwpEMTYDbLMDDjTmbvVNt1BhYWEA3hoAEEm8Xqv2nl51e7wqbWjXmoN1ajzWre3lzeryeD8stTWtXerq8Som2igmyshKqmzqVEdP7xm/d176ME0dlSprrbxW8vq+9q1b2X7ber3ek4z5n/UPxlgrDYuL1oIxGbp02sjA/QcDgshAhTHYBKIYl0sq6LeeL+noQAOttfdKuleS5s2bN2B5BgBElg+mEByqbVOXx6uOnl7VtXbptT018ni9Km/sUEVjh2Sk1hNMNYiLjtK8ouGSJE+vVWZyvHJSExRl+tZ7er2akZem3LRhyk6JV3J8jDxeq16vVx6vVWpCrEamJain16ter1VPr/V99eqcgnRFGaOCjGEyJhR+tAM4U4Eoxi9I+pox5klJCyU1M78YACKL12tlpQ+nCrR09qi106Ounl7FREV9eOS2rq1bL+2s1NbyZklSTUun9la1nvS1Z+an6bzxmcpJjVd7d6+SE2I0OiNRiXExGpedpMkjU5UYF01pBXDWTlmMjTFPSFoqKdMYUy7ph5JiJcla+ydJqyRdLqlYUruk2wYrLADArYZj3dpR0axdR5vV2d2rreXNKmtoV3lju3p6/f9FYEZSnHJSEzQsLlrTRqVq0dgRSoyLVmFGorJS4jU8MU6j0ocpKyV+EP80APBR/lyV4qZTPG8lfTVgiQAAzljbN42gqb1bXR6vjjZ16L3iOh2ub9fGIw2qbO782PdMyE7WZ+YVKDM5Xsb0zaGNjTJKSYhR6rDYD+fcerx9c2oXj83Q+OwUB386ADi5QEylAACEEGutDtYeU3NHtzy9VlvLmtTl8aqmtVNrDtbrUO2xj31PblqC8tKHaemkbC0eN0Ljs5JVkDFMCbHRio3mJqoAwgPFGADCVLfHq3cO1GpjSaMk6WhTh47Ut2tfVYs6e7wfGx8XHaX8jGG6fm6+puamKiUhRl0ery6cmKWCjMShjg8AQ45iDABh6P2D9freczt0qO5/jv4mxkVr8sgUXTMrT8kJMVo8doSGxUXLa62mj0pTemIsJ7ABiGgUYwAII90er362ao8eXHNEI1MT9NvPztKs/HTlD09UXAxTHgDgZCjGABDimtt7dN87h/T2gVodrj2m1i6PLpmao1/fMCvob78KAMGEYgwAIaq5vUe/fXW/Hl9fqm5P35zhK2bk6sqZubpsRq7jdAAQeijGABBiOnt6tXJ7pf7lmW2SpMumj9TtS8bonIJ0rhABAGeBYgwAIaKls0f/55UDemZT2Ye3Rr7/lnlaNjXHcTIACA8UYwAIcgeqW/Xw+yV6ZG2JJKkgY5i+eMFY3TCvQDmpCY7TAUD4oBgDQJDq9nj1k5W79cjaEkUZKS99mH772XO0YEyG62gAEJYoxgAQhErr23X3Ixu1t6pVF03K0o+vmc5NNgBgkFGMASCIdPb06g9vHtRDa46ouaNHl07L0Z8+P5cbbwDAEKAYA0CQqGzu0Dee2KINRxq1cEyGfnT1NE3JTXUdCwAiBsUYABw72tShf39uh9YcrFdPr1c/vmaabllc5DoWAEQcijEAOPT+wXp966mtqm7t1A1z83XbeWM4SgwAjlCMAWCIdfb06jvPbte7xXVqONat1IQYrfjqeZqZn+46GgBENIoxAAyhx9eV6t+f2yFJGpEUp3+/fLJuWlColIRYx8kAABRjABgCjce69ZOVu/W3LRWSpAdum6+LJmU7TgUA6I9iDACDrLyxXdf9YY1qW7t01axR+pdPTtToEUmuYwEAjkMxBoBB9MT6Uv3b33YoOT5GK7++RNPz0lxHAgCcQJTrAAAQrl7eVaUfvbBLkvT4XQspxQAQ5DhiDAABZq3VPW8U6zev7NfMvDTdf+t8ZaXEu44FADgFijEABFCv1+rPbx/Ur1/er/PGj9D9t8zXsLho17EAAH6gGANAgBzr8ugLD6zXhiONmpmfpoduW6CYaGasAUCooBgDQADsKG/Wlx/bpPLGDt04v0A/vGoapRgAQgzFGADO0t+3HdW/PLNNcTFReuzOhTpvfKbrSACAM0AxBoCzUNbQrm89tVXTRqXqz/80TyPTElxHAgCcIX7PBwBnqNvj1V0Pb1R8TJTu+dwcSjEAhDiOGAPAGbrz4Y3aW9Wqe/9prvKHJ7qOAwA4SxwxBoAzsGJrhd7eX6vZhen65LSRruMAAAKAYgwAp+mdA7X65pNbNTM/TU/ctch1HABAgFCMAeA0dHl69dOVe1SYkagn716khFhu3gEA4YJiDAB+stbqn5/epn3VrfrX5ZOVGMdpGgAQTijGAOCnV3ZXa+X2Sv3vSyfpipm5ruMAAAKMYgwAfuj2ePV/Xj2gnNR4ffGCsa7jAAAGAb8HBAA//Omtg9pd2aI/fm4Ot3oGgDDFpzsAnMLB2jb99+vFumJmri6bwRQKAAhXFGMAOAlPr1dff3yLkuKj9f0rprqOAwAYREylAICT+Ok/9mh3ZYt+f9NsbvkMAGGOI8YAcALrDtXrwTVHtHRSlq6aNcp1HADAIKMYA8AJ/GzVHknSb26Y5TgJAGAoUIwBYACH645pW3mzbpibrxHJ8a7jAACGAMUYAAbwu9cOSJK+dvF4x0kAAEOFYgwAxzlcd0zPbanQgqIMjR6R5DoOAGCIUIwB4Dg/WLFTkvTbG89xnAQAMJQoxgDQz0s7q/TOgTotGJOhvPRhruMAAIYQxRgAfNq6PPrPv+9SXHSUHr59ges4AIAhxg0+AMDn16v3qbK5U898abESYqNdxwEADDGOGAOApJ0VzXpwzREtGZ+p+UUZruMAABygGAOIeJ09vbr5vrWSpF9eP9NxGgCAKxRjABHv5vvWqqXTo0fuWKBRnHAHABGLYgwgor1XXKfNpU361Jw8nT8hy3UcAIBDFGMAEcvrtfq3v+3Q6BGJ+tl1M1zHAQA4RjEGELH+saNSpQ3t+srScVyFAgBAMQYQmay1enRtibJS4nX93ALXcQAAQYBiDCAiPbq2ROsON+ju88cqOsq4jgMACAIUYwARx9Pr1Q9e2KW5o4frzvPHuI4DAAgSFGMAEef+dw/LWulLF46TMRwtBgD0oRgDiCgNx7r1/724VzPy0rRsSrbrOACAIEIxBhBRvvPsNknStz85kaPFAICPoBgDiBhv76/Vq3tq9LWLxuuiSRwtBgB8FMUYQETo9nj1i5f2SpK+eOFYx2kAAMGIYgwgIqzYWqFdR1t0z81zlJIQ6zoOACAIUYwBhD1rre5/57Am5iTr8hkjXccBAAQpijGAsLe5tEn7qlt167lFnHAHADghijGAsHff24ckSVfMyHWcBAAQzCjGAMJadUunXtpVpfHZyUpPjHMdBwAQxCjGAMLaf728v+/rZ2Y5TgIACHYUYwBhq7OnV6t2VOrKmbmamZ/uOg4AIMhRjAGErUfeL1Frl0c3Lyh0HQUAEAIoxgDCkrVWT24o1YKiDC0eN8J1HABACKAYAwhLG0sadbD2mK4+ZxSXaAMA+IViDCAs3ff2IaUnxupTc/JcRwEAhAiKMYCw09zRo9f21uiz8wuUGBfjOg4AIERQjAGEnTf31ajXa3XhhCzXUQAAIYRiDCCseL1Wv3+9WBOyk7VwLCfdAQD8RzEGEFbWHKxXcU2bvrx0nKKjOOkOAOA/ijGAsOH1Wv1q9V5lpcTr8hm5ruMAAEIMxRhA2Fi1s1Lbypv1vcunKCE22nUcAECIoRgDCBsv7qxSZnK8rp41ynUUAEAI8qsYG2OWG2P2GWOKjTHfHeD5QmPMG8aYLcaY7caYywMfFQBO7FiXR2/vq9XFk7MUxdxiAMAZOGUxNsZES7pH0mWSpkq6yRgz9bhh/yHpaWvtbEk3SvpDoIMCwMn85993qbXLo8/OL3QdBQAQovw5YrxAUrG19pC1tlvSk5KuOW6MlZTqe5wm6WjgIgLAyXV7vHp6Y7kkae7o4Y7TAABClT+3hMqTVNZvvVzSwuPG/EjSy8aYr0tKkrQsIOkAwA8vbOv7t/h/fWaW4yQAgFDmzxHjgSbr2ePWb5L0oLU2X9Llkh4xxnzstY0xdxtjNhpjNtbW1p5+WgAYwGPrSjQhO1nXzc5zHQUAEML8Kcblkgr6refr41Ml7pD0tCRZa9+XlCAp8/gXstbea62dZ62dl5XFrVoBnL0D1a3aUtqkz84vkDGcdAcAOHP+FOMNkiYYY8YYY+LUd3LdC8eNKZX0CUkyxkxRXzHmkDCAQffMpnLFRBldy9FiAMBZOmUxttZ6JH1N0mpJe9R39YldxpgfG2Ou9g37Z0l3GWO2SXpC0hestcdPtwCAgOrp9epvmyv0iSnZykyOdx0HABDi/Dn5TtbaVZJWHbftB/0e75Z0XmCjAcDJvbmvVnVtXbphbsGpBwMAcArc+Q5AyHp6Y5myUuK1dBLnLAAAzh7FGEBIqm3t0ut7a/SpOXmKieajDABw9vhpAiAkPbTmiHq9Vp+ek+86CgAgTFCMAYSkrWVNkqSJOSmOkwAAwgXFGEDIae/2aFNJo25dPNp1FABAGKEYAwg5z24qV0dPr5ZPz3UdBQAQRijGAELOL1/apym5qVo0NsN1FABAGKEYAwgpm0sb1dbl0TXnjOIW0ACAgKIYAwgpf9tcLkn6/CLmFwMAAotiDCBkdPb06u/bKnXlzFwlx/t1404AAPxGMQYQMl7cWanmjh7dtKDQdRQAQBiiGAMIGY+8X6KxWUlaPHaE6ygAgDBEMQYQEvZUtmhzaZM+O69AUVGcdAcACDyKMYCQ8NCaI0qIjdJn5xe4jgIACFMUYwBBr6m9W89vrdC15+QpPTHOdRwAQJiiGAMIes9vqVBnj1e3nlvkOgoAIIxRjAEENWutntxQpskjUzQlN9V1HABAGKMYAwhqL+6s0t6qVt2+ZIzrKACAMEcxBhDUHl9XKkm65pxRjpMAAMIdxRhA0DpY26Y1B+t01/ljFB8T7ToOACDMUYwBBK0n15cqOsrorgvGuo4CAIgAFGMAQanb49XfNlfooknZyk5JcB0HABABKMYAgtIru6tVf6xbNy0sdB0FABAhKMYAgtL97x5SXvowLRmf6ToKACBCUIwBBJ29VS3aUtqkW88drdhoPqYAAEODnzgAgs5zmysUHWV0/dwC11EAABGEYgwgqFhr9Y8dlbpgQqYykuJcxwEARBCKMYCgsrm0SeWNHbp02kjXUQAAEYZiDCCoPL2hTElx0bpqFne6AwAMLYoxgKDR0+vVUxvLtGxqjpLiY1zHAQBEGIoxgKCxtaxJkjS7IN1xEgBAJKIYAwgaq3dWKTrK6NNz811HAQBEIIoxgKDQ2dOrv24u1yVTcpSSEOs6DgAgAlGMAQSFP791SI3tPbpl8WjXUQAAEYpiDCAovHOgVpJ0LreABgA4QjEG4FxJ/TFtLGnUVy8a5zoKACCCUYwBOPfUhjJFGenmhUyjAAC4QzEG4JSn16u/bi7X0knZyksf5joOACCCUYwBOLXhSKOqW7p03ew811EAABGOYgzAqUfXlWh4YqwumpztOgoAIMJRjAE4U9/WpVd2VeuSqTlK5hbQAADHKMYAnFm5vVLdvV594dwxrqMAAEAxBuDOyu1HNSknRVNHpbqOAgAAxRiAG5XNHdpwpFFXzsx1HQUAAEkUYwCO/GN7pSTpylmjHCcBAKAPxRiAEyu3V2p6XqrGZCa5jgIAgCSKMQAHKps7tLWsScunjXQdBQCAD1GMAQy5J9aXyRjp6lnc1AMAEDwoxgCGVLfHqyfWl2rpxCwVjkh0HQcAgA9RjAEMqdW7qlTb2qVbFhe5jgIAwEdQjAEMqcfWlaggY5gunJjlOgoAAB9BMQYwZKqaO7X2UIM+M7dAUVHGdRwAAD6CYgxgyDyxvlTGcO1iAEBwohgDGBLWWj21oUxLxmdy7WIAQFCiGAMYElvKmlTV0qmrOFoMAAhSFGMAQ+KZjeVKjIvWZdO5qQcAIDhRjAEMurYuj1ZsrdCl00YqJSHWdRwAAAZEMQYw6F7ZXaX27l59bmGh6ygAAJwQxRjAoHth61HlpQ/TnMLhrqMAAHBCFGMAg6qls0fvHKjTlTNzuXYxACCoUYwBDKo39tbI47VaNjXHdRQAAE6KYgxgUP1je6VyUuM1l2kUAIAgRzEGMGhaO3v05v5aXTadaRQAgOBHMQYwaF7fW6Nuj1dXzsx1HQUAgFOiGAMYNCu3V2pkagJXowAAhASKMYBB0XisW2/tr9VlM0YyjQIAEBIoxgAGxYqtFer2eHXD3ALXUQAA8AvFGMCgWLWzSpNHpmjqqFTXUQAA8AvFGEDANbV3a1NJoy7h2sUAgBBCMQYQcO8V16vXa7V0UrbrKAAA+I1iDCDg3tpfo6S4aM3MT3MdBQAAv1GMAQSU12v18u5qXTptpGKj+YgBAIQOfmoBCKjdlS1qau/RkgmZrqMAAHBaKMYAAurNfTWSpPMnZDlOAgDA6aEYAwioN/bVamZ+mrJS4l1HAQDgtFCMAQRM47FubSlt5GoUAICQRDEGEDArtx+V10rLplCMAQChh2IMIGBe9N3tbkYel2kDAIQeijGAgOjs6dXGI41aMj5TxhjXcQAAOG0UYwABsamkUd29Xp03nsu0AQBCk1/F2Biz3BizzxhTbIz57gnGfMYYs9sYs8sY83hgYwIIdu8cqFNstNH8MRmuowAAcEZiTjXAGBMt6R5Jl0gql7TBGPOCtXZ3vzETJP2bpPOstY3GGM68ASLMppIGTR6ZquT4U36sAAAQlPw5YrxAUrG19pC1tlvSk5KuOW7MXZLusdY2SpK1tiawMQEEs9rWLm0sadRFk7ipBwAgdPlTjPMklfVbL/dt62+ipInGmPeMMWuNMcsHeiFjzN3GmI3GmI21tbVnlhhA0Hl9b7WslS6fmes6CgAAZ8yfYjzQ6eX2uPUYSRMkLZV0k6T7jTHpH/sma++11s6z1s7LyuLIEhAuNh5pVGpCjCblpLiOAgDAGfOnGJdLKui3ni/p6ABjVlhre6y1hyXtU19RBhDmvF6r1/fWaOmkbC7TBgAIaf4U4w2SJhhjxhhj4iTdKOmF48Y8L+kiSTLGZKpvasWhQAYFEJw2ljSq/li3PsHd7gAAIe6Uxdha65H0NUmrJe2R9LS1dpcx5sfGmKt9w1ZLqjfG7Jb0hqT/ba2tH6zQAILHq3uqFRNl9IkpOa6jAABwVvy6rpK1dpWkVcdt+0G/x1bSt30LgAhhrdWLOyt17vhMLtMGAAh53PkOwBnbXt6ssoYOXTmDq1EAAEIfxRjAGXth21HFRhtdOm2k6ygAAJw1ijGAM9LrtVqxtUIXT85WWmKs6zgAAJw1ijGAM7KtvEl1bd26nGkUAIAwQTEGcEbe2Fuj6CijCydysx4AQHigGAM4bdZavbDtqBYUZSg9Mc51HAAAAoJiDOC07a1qVUl9u66dPcp1FAAAAoZiDOC0rdpRqegoo4smc7c7AED4oBgDOG2v7qnRnMJ0ZackuI4CAEDAUIwBnJbK5g7tqWzhaDEAIOxQjAGclpd2VkmSLpvOZdoAAOGFYgzgtLy2p0Zjs5I0JjPJdRQAAAKKYgzAb+WN7Xq3uE5XzuRqFACA8EMxBuC3x9eVSpKun5PvOAkAAIFHMQbgt41HGhUXE6XCEYmuowAAEHAUYwB+aens0ebSRt1+3hjXUQAAGBQUYwB+eW1PtTxeq2VTuEwbACA8UYwB+OWlnVUamZqgOYXDXUcBAGBQUIwBnFJPr1fvFdfr4inZiooyruMAADAoKMYATmlLaZPaujy6YEKm6ygAAAwaijGAU3ptT7VioowWj6UYAwDCF8UYwElZa7V6V5XOHZ+ptMRY13EAABg0FGMAJ7WnslVH6tu1fNpI11EAABhUFGMAJ/XanmpJ0iVTcxwnAQBgcFGMAZzUu8V1mjYqVVkp8a6jAAAwqCjGAE6otbNHW0qbdO64Ea6jAAAw6CjGAE7ojX216u716pKpzC8GAIQ/ijGAE3pxR6WyUuI1dzR3uwMAhD+KMYABeXq9ere4ThdPylY0d7sDAEQAijGAAa073KDWTo+WTspyHQUAgCFBMQYwoOe3VCglPkYXTc52HQUAgCFBMQbwMd0er17aWaVLp49UQmy06zgAAAwJijGAj1l/uEGtXR5dNp2rUQAAIgfFGMDHvLmvRnHRUVo4lusXAwAiB8UYwEdYa7VqR6WWTMhUcnyM6zgAAAwZijGAj9hb1aqjzZ1aPo1pFACAyEIxBvARb+6rlSRdyGXaAAARhmIM4CPe2FejKbmpyklNcB0FAIAhRTEG8KGGY91af7iBm3oAACISxRjAh1ZuPypJWjaFm3oAACIPxRjAh/66uUKTR6Zo7ugM11EAABhyFGMAkqSyhnZtK2vSdbPzXEcBAMAJijEASdLqXVWSpMum5zpOAgCAGxRjAJKkv287qmmjUlU4ItF1FAAAnKAYA1DDsW5tr2jWJVNzXEcBAMAZijEAvbyrStZKy6ZQjAEAkYtiDEArt1eqaESipo1KdR0FAABnKMZAhKtp6dSag3W6cuYoGWNcxwEAwBmKMRDhXt5dLa+Vrpo1ynUUAACcohgDEW71rioVjUjUxJxk11EAAHCKYgxEsOaOHr1/sF6XTh/JNAoAQMSjGAMR7PW91fJ4rS6dNtJ1FAAAnKMYAxHs7f11GpEUp3Py011HAQDAOYoxEKE6e3r1yu5qXTgpS1FRTKMAAIBiDESoVTsq1dbl0fVz8l1HAQAgKFCMgQj17KZyjR6RqMXjRriOAgBAUKAYAxHoaFOH3j9Ur+tm53E1CgAAfCjGQAR6dlO5rJU+NZtpFAAAfIBiDEQYa62e21KhRWMzVDgi0XUcAACCBsUYiDDbypt1uO6Yrpud5zoKAABBhWIMRJjnt1QoLjpKy6fnuo4CAEBQoRgDEcTT69XK7Ud18eRspQ2LdR0HAICgQjEGIsh7B+tV19ata2ePch0FAICgQzEGIsiKLRVKSYjR0knZrqMAABB0KMZAhGjv9uilXVW6fHquEmKjXccBACDoUIyBCLF6V5Xau3t1DdMoAAAYEMUYiBCPryvVmMwkLRrDLaABABgIxRiIAKX17dpwpFGfnpOnqChuAQ0AwEAoxkAE+ONbBxUXE6Xr5xa4jgIAQNCiGANhrvFYt1ZsrdCVM3M1Mi3BdRwAAIIWxRgIc89sKlN7d6/uvmCs6ygAAAQ1ijEQxqy1enpjueYUpmvyyFTXcQAACGoUYyCMvVdcr+KaNt24oNB1FAAAgh7FGAhjD71/RJnJcbrmHK5dDADAqVCMgTBV1tCuV/dU67PzCxQfw53uAAA4FYoxEKYeXHNE0cbo84tGu44CAEBIoBgDYai5o0dPbSjTFTNzlZs2zHUcAABCAsUYCEN/33ZUbV0e3bFkjOsoAACEDIoxEIb+urlcE7KTNSMvzXUUAABCBsUYCDPFNa3aUtqkG+blyxjjOg4AACHDr2JsjFlujNlnjCk2xnz3JOOuN8ZYY8y8wEUEcDpWbD2qKCNdOzvPdRQAAELKKYuxMSZa0j2SLpM0VdJNxpipA4xLkfQNSesCHRKAf3p6vfrb5gqdNz5T2SkJruMAABBS/DlivEBSsbX2kLW2W9KTkq4ZYNxPJP1SUmcA8wE4DQ++d0QVTR26ZXGR6ygAAIQcf4pxnqSyfuvlvm0fMsbMllRgrV15shcyxtxtjNlojNlYW1t72mEBnJi1Vr97/YAk6eLJ2Y7TAAAQevwpxgOdvWM/fNKYKEm/lfTPp3oha+291tp51tp5WVlZ/qcEcErvFteptdOjH18zTdFRnHQHAMDp8qcYl0sq6LeeL+lov/UUSdMlvWmMOSJpkaQXOAEPGFp/efewMpPjdeP8QtdRAAAISf4U4w2SJhhjxhhj4iTdKOmFD5601jZbazOttUXW2iJJayVdba3dOCiJAXxMeWO73thXq88tLFRcDFdhBADgTJzyJ6i11iPpa5JWS9oj6Wlr7S5jzI+NMVcPdkAAp/bspnIZI10/N991FAAAQlaMP4OstaskrTpu2w9OMHbp2ccC4C9rrZ7fUqHFY0eoICPRdRwAAEIWv3MFQtyag/U6Ut+uT8/haDEAAGeDYgyEuEfXlig9MVZXzMx1HQUAgJBGMQZCWG1rl17eXa3PzCtQQmy06zgAAIQ0ijEQwu5/55C81uoz8wpOPRgAAJwUxRgIUS2dPXp0bYmunjVK47OTXccBACB9I8azAAAU5ElEQVTkUYyBEPX0hjId6+7VnUvGuo4CAEBYoBgDIcjT69UD7x3R/KLhmpGf5joOAABhgWIMhKDX9taooqlDdywZ4zoKAABhg2IMhKCH1hzRqLQELZuS4zoKAABhg2IMhJjNpY1ac7BeXzivSDHR/BUGACBQ+KkKhJjfv3ZAwxNj9bmFo11HAQAgrFCMgRCyo7xZb+yr1R1LxigpPsZ1HAAAwgrFGAghv3/9gFISYnTLuUWuowAAEHYoxkCI2FnRrJd3V+v288YoNSHWdRwAAMIOxRgIEf/1yn6lDYvVHedziTYAAAYDxRgIAVtKG/X63hrdfcFYjhYDADBIKMZACLj/3cNKSYjRF5hbDADAoKEYA0HuQHWrVu2o1OcWjuZKFAAADCKKMRDkfvvqfiXGRuvuC8a6jgIAQFijGANBbGdFs1btqNIdS8YoIynOdRwAAMIaxRgIYv9zJQqOFgMAMNgoxkCQ2lTyP1eiSBvGlSgAABhsFGMgSP3m5X3KTI7TbecVuY4CAEBEoBgDQWhNcZ3WHKzXV5aOV2IcV6IAAGAoUIyBIGOt1a9f3qfctATdvLDQdRwAACIGxRgIMqt3VWtzaZO+fvEEJcRGu44DAEDEoBgDQaTL06uf/mO3JuYk64Z5+a7jAAAQUSjGQBB5fF2pyhs79P0rpyo2mr+eAAAMJX7yAkGivduje94o1uKxI7RkfKbrOAAARByKMRAk/vLuYdW1deufPzlRxhjXcQAAiDgUYyAI1LV16Y9vHtQnp+ZoXlGG6zgAAEQkijEQBP781kF19PTqXy+b7DoKAAARi2IMOFbZ3KGH3i/RtbPzNC4r2XUcAAAiFsUYcOz3rxfLWqtvLZvoOgoAABGNYgw4VFrfrqc3lOmmBYUqyEh0HQcAgIhGMQYcsdbqJ//Yregoo69eNN51HAAAIh7FGHDkrf21emV3tb59yUTlpCa4jgMAQMSjGAMOtHd79MMXdqkwI1G3nTfGdRwAACApxnUAIBL94Y2DKqlv1xN3LVJcDP8+BQAgGPATGRhiZQ3tuvftQ7pudp4WjxvhOg4AAPChGANDyFqrn63aI0n6zvJJjtMAAID+KMbAEFq1o0ov7qzSN5dNUG7aMNdxAABAPxRjYIi0dvboP/++S9PzUvXFC8a6jgMAAI7DyXfAELns/76jmtYu3XfLPMVE829SAACCDT+dgSGwraxJ5Y0dWjopS7MK0l3HAQAAA6AYA4PM67X6ycrdGpEUp9/fNNt1HAAAcAIUY2CQPbu5XBtLGvWvyycrJSHWdRwAAHACFGNgENW0dOqnK3drftFwXT8333UcAABwEhRjYJBYa/Ufz+9Ul8erX3x6pqKijOtIAADgJCjGwCD5x45Kvby7Wt++ZKLGZiW7jgMAAE6BYgwMgvq2Lv1wxS7Nyk/THUvGuI4DAAD8QDEGAuyDKRStnR796oZZXLMYAIAQwU9sIMBWbq/Uizur9K1LJmpiTorrOAAAwE8UYyCAalo79f0VOzWrIF13nc8UCgAAQgnFGAiQXq/VN57Yos6eXv3mhplMoQAAIMTEuA4AhIv73jmktYca9KvrZ2p8NlMoAAAINRzSAgJgZ0WzfvPyPl06LYcbeQAAEKIoxsBZau7o0Zce3aTM5Hj9/FMzZQw38gAAIBQxlQI4C16v1bee2qqq5k49/aXFykiKcx0JAACcIY4YA2fhgTVH9PreGn3/yqmaUzjcdRwAAHAWKMbAGdpU0qhfvLhXy6Zk65bFo13HAQAAZ4liDJyB2tYufeOJLcpJi9evb5jFvGIAAMIAc4yB09TZ06vbHlyv+mNdevqLi5WeyLxiAADCAcUYOA29XqtvPrlFOytadP8t8zQzP911JAAAECBMpQD8ZK3Vfzy/U6t3Vev7V07Vsqk5riMBAIAAohgDfvrDmwf1xPpSfXnpON2xZIzrOAAAIMAoxoAfXtpZpV+t3qerZo3Sdy6d5DoOAAAYBBRj4BTWHKzTN5/colkF6frV9dzZDgCAcEUxBk5iw5EG3fHgRo0ekagHvjBfCbHRriMBAIBBQjEGTmBrWZNue2CDctMT9Nidi7jdMwAAYY5iDAxgZ0Wzbvl/65SRFKfH71ykrJR415EAAMAgoxgDx9lX1ap/+n/rlJIQq8fvWqiRaQmuIwEAgCFAMQb62VvVos/dv05xMVF67M6Fyh+e6DoSAAAYIhRjwGdPZYtuvHetoqOkx+5cpKLMJNeRAADAEOKW0ID6TrT7wgPrFR8Tpae/uFijR1CKAQCINBwxRsR7/2C9PnffWqUmxOqZL55LKQYAIEJxxBgR7a39tfrSI5uUP3yYHr1zoXJSOdEOAIBIRTFGxNp4pEFfemSTijKT9PDtC7gkGwAAEY6pFIhIG4406Po/va/ctARKMQAAkEQxRgR6cUelbvjT+5KkR+5cSCkGAACS/CzGxpjlxph9xphiY8x3B3j+28aY3caY7caY14wxowMfFTh7j68r1Vce36y5o4dr6w8uUV76MNeRAABAkDhlMTbGREu6R9JlkqZKuskYM/W4YVskzbPWzpT0rKRfBjoocDastfr5i3v078/t0IUTs/ToHQuVnhjnOhYAAAgi/hwxXiCp2Fp7yFrbLelJSdf0H2CtfcNa2+5bXSspP7AxgTPX0d2rbzy5VX9+65A+v6hQ990yT8Piol3HAgAAQcafq1LkSSrrt14uaeFJxt8h6cWBnjDG3C3pbkkqLCz0MyJw5sob23XXw5u0t6pF31k+SV++cJyMMa5jAQCAIORPMR6oRdgBBxrzeUnzJF040PPW2nsl3StJ8+bNG/A1gEBZd6heX35ss3p6vfrLrfN10eRs15EAAEAQ86cYl0sq6LeeL+no8YOMMcskfU/ShdbarsDEA87M4+tK9YMVO1U4IlH33TJP47KSXUcCAABBzp9ivEHSBGPMGEkVkm6UdHP/AcaY2ZL+LGm5tbYm4CkBP7V3e/TDFbv0zKZyXTgxS7+7abbShsW6jgUAAELAKYuxtdZjjPmapNWSoiX9xVq7yxjzY0kbrbUvSPqVpGRJz/jmb5Zaa68exNzAxxTXtOkrj23SgZo2ff3i8fpfyyYqOor5xAAAwD9+3RLaWrtK0qrjtv2g3+NlAc4F+M1aqyfWl+knK3drWFy0HrptgS6YmOU6FgAACDF+FWMgWB2pO6bvPb9D7xXX69xxI/RfnzlHI9MSXMcCAAAhiGKMkNTt8erPbx3U798oVnx0lH567XTdvKBQUUydAAAAZ4hijJCz7lC9vvf8ThXXtOmKGbn6wVVTlZPKUWIAAHB2KMYIGY3HuvXzF/fo6Y3lyksfpge+wLWJAQBA4FCMEfR6vVZPbyzTr1bvU3NHj7544Vh98xMTlBjH7gsAAAKHZoGgtrm0UT/++25tLWvS/KLh+vE10zUlN9V1LAAAEIYoxghKR5s69MuX9ur5rUeVlRKv3352lq49J0++62QDAAAEHMUYQaWpvVu/e61Yj64tkYz0tYvG68tLxykpnl0VAAAMLtoGgkK3x6uH3z+i3712QG1dHt0wt0Bf/8R45Q9PdB0NAABECIoxnOr2eLVia4X++41ildS36/wJmfreFVM0eSTziAEAwNCiGMOJY10e/XVzuf705kEdbe7U1NxUPfCF+Vo6KYt5xAAAwAmKMYZUS2ePHnm/RH9+66BaOj2aO3q4fvapGbpwIoUYAAC4RTHGkDja1KEH3jusJ9aXqa3Lo4snZ+srS8dp7ujhFGIAABAUKMYYNNZarT3UoEfXleilnVWSpCtm5OruC8Zqel6a43QAAAAfRTFGwDW1d+vZTeV6fH2pDtUeU2pCjG4/r0i3nlvEVSYAAEDQohgjIKy12ljSqMfXleofOyrV7fFqTmG6fn3DLF05M1cJsdGuIwIAAJwUxRhnpbm9R89t6Ts6vL+6TSnxMbpxfoFuXljIJdcAAEBIoRjjtHV5evXG3lq9sK1Cr+6uUXevVzPz0/SLT8/QVbNGKTGO3QoAAIQeGgz80uXp1XvFdXp5V7VW76pSY3uPMpLidPPCQl0/N5+T6QAAQMijGOOE2rs9enNfrV7aWaXX99aorcuj5PgYLZ2Upevn5mvJ+EzFREe5jgkAABAQFGN8xLEuj17bW6PVvjLc0dOrjKQ4XTkzV5dOH6lzx41QfAwn0gEAgPBDMY5w1lrtrWrVuwfq9Oqeam0ubVRPr1VWSryum5OnK2fmakFRBkeGAQBA2KMYR6DK5g6tO9Sg9w/W6839Napu6ZIkTcpJ0e1LxujiSdmaX5ShqCjuSAcAACIHxTgClDW0a93hBq0/XK91hxtUUt8uSUpJiNH5EzK1dFK2zp+Qqdy0YY6TAgAAuEMxDjPWWh2uO6b1hxt8ZbhBFU0dkqS0YbGaX5Shf1o0WovGjtCU3FRFc1QYAABAEsU45HV092prWZM2lzZqc0mjtpU3qa6tW5KUmRynhWNG6O4Lxmrh2AxNzE5hegQAAMAJUIxDSGdPr/ZUtmhnRbO2lzdrR0WzDtS0qddrJUnjspJ0wcQszRudoYVjMzQ2M0nGUIQBAAD8QTEOUp09vdpf3aodFc3aUd5XhPdXt8rjK8EZSXGakZemZVNyNGd0uuYUDld6Ypzj1AAAAKGLYuyYtVbljR0qrmnT3qpW7als0d6qFh2sPfbhkeD0xFjNyEvTFyeP1Yy8NM3IT9eotASOBgMAAAQQxXiIeL1WlS2d2l/dquLqNh2oadWBmjYVV7eptcvz4bhRaQmakpuqT04dqamjUjUjL035w4dRggEAAAYZxTiAuj1elTW2q7S+XaUNfUtJ/TEd8a13e7wfjs1MjteE7GRdNydPk0amaEJ2iiblpCgtMdbhnwAAACByUYz91Ou1qmvrUk1Ll2rbOlXV3KXK5g5VNHXoaFOHyho6VNncId/sB0nSsNhoFWYkamxmkj4xOVuFIxI1PitZE3NSNDyJ+cAAAADBJOKL8bEuj+rbulXb1tVXfFu7VNPSqdrWLtW29q1Xt3Sqrq3rI6VXkqKMNDI1QbnpwzSvaLgKM/JUNCJJRZlJKsxIVGZyHFMgAAAAQkREFePdR1v08xf3qKm9Rw3HutVwrFsdPb0fGxdlpIykeGWlxCsnNV5Tc1OVnRqv7NQE5aTEKzMlXiNTE5SVEq/Y6CgHfxIAAAAEWkQVY2Oklk6PRiTHaXx2sjKT45SRFK/M5DhlpsQrKzle2SnxykiKUwyFFwAAIKJEVDGekpuqFV89z3UMAAAABCEOiwIAAACiGAMAAACSKMYAAACAJIoxAAAAIIliDAAAAEiiGAMAAACSKMYAAACAJIoxAAAAIIliDAAAAEiiGAMAAACSKMYAAACAJIoxAAAAIIliDAAAAEiiGAMAAACSKMYAAACAJIoxAAAAIIliDAAAAEiiGAMAAACSJGOtdfPGxtRKKvFzeKakukGMg8jDPoVAY59CoLFPIZAifX8aba3NOtUgZ8X4dBhjNlpr57nOgfDBPoVAY59CoLFPIZDYn/zDVAoAAABAFGMAAABAUugU43tdB0DYYZ9CoLFPIdDYpxBI7E9+CIk5xgAAAMBgC5UjxgAAAMCgohgDAAAAGuJibIy5wRizyxjjNcbMO+65fzPGFBtj9hljLu23fblvW7Ex5rv9to8xxqwzxhwwxjxljInzbY/3rRf7ni861Xsg9BljfmSMqTDGbPUtl/d7btD3LUSuE+1HwAeMMUeMMTt8n00bfdsyjDGv+D5nXjHGDPdtN8aY3/n2p+3GmDn9XudW3/gDxphb+22f63v9Yt/3mqH/U2IwGWP+YoypMcbs7Ldt0PehE71HWLPWDtkiaYqkSZLelDSv3/apkrZJipc0RtJBSdG+5aCksZLifGOm+r7naUk3+h7/SdKXfY+/IulPvsc3SnrqZO8xlH9+lkHdt34k6V8G2D7o+xZL5C4n249YWD5YJB2RlHnctl9K+q7v8Xcl/cL3+HJJL0oykhZJWufbniHpkO/rcN/j4b7n1kta7PueFyVd5vrPzBLwfegCSXMk7RzKfehE7xHOy5AeMbbW7rHW7hvgqWskPWmt7bLWHpZULGmBbym21h6y1nZLelLSNb5/yVws6Vnf9z8k6dp+r/WQ7/Gzkj7hG3+i90B4G4p9C5FrwP3IcSaEhv6fJ8d/zjxs+6yVlG6MyZV0qaRXrLUN1tpGSa9IWu57LtVa+77tay8P93sthAlr7duSGo7bPBT70IneI2wFyxzjPEll/dbLfdtOtH2EpCZrree47R95Ld/zzb7xJ3othI+v+X5t9Jd+v+4Zin0LkYvPFfjDSnrZGLPJGHO3b1uOtbZSknxfs33bT/czK8/3+PjtCH9DsQ+d6D3CVkygX9AY86qkkQM89T1r7YoTfdsA26wGLu72JONP9lon+x6EgJPtW5L+KOkn6vt/+hNJv5F0u4Zm30LkYp+AP86z1h41xmRLesUYs/ckY0/3Zxj7II7HPnQWAl6MrbXLzuDbyiUV9FvPl3TU93ig7XXq+9VAjO/IXf/xH7xWuTEmRlKa+n79cLL3QAjwd98yxtwnaaVvdSj2LUQuPldwStbao76vNcaY59Q3BafaGJNrra30/Sq7xjf8RPtUuaSlx21/07c9f4DxCH9DsQ+d6D3CVrBMpXhB0o2+s/7HSJqgvongGyRN8F0lIE59Jzy94JsD84ak633ff6ukFf1e64MzLa+X9Lpv/IneA2HA9xf2A9dJ+uDM3aHYtxC5BtyPHGdCEDHGJBljUj54LOmT6vt86v95cvznzC2+KwssktTs+xX2akmfNMYM900V+6Sk1b7nWo0xi3znPNzS77UQ3oZiHzrRe4SvoTzTT32FpVxSl6Rq9f0P+eC576nv7O596ndGrfrOrtzve+57/baPVV/BKZb0jKR43/YE33qx7/mxp3oPltBfJD0iaYek7er7i5w7lPsWS+QuJ9qPWFis/fDzZJtv2fXBPqK+8xNek3TA9zXDt91Iuse3P+3QR6/gdLvv86dY0m39ts9TX9k+KOm/5burLUv4LJKekFQpqcfXo+4Yin3oRO8Rzgu3hAYAAAAUPFMpAAAAAKcoxgAAAIAoxgAAAIAkijEAAAAgiWIMAAAASKIYAwAAAJIoxgAAAIAk6f8Hc+ZicBVj/28AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 864x576 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.figure(figsize=(12,8)); \n",
    "plt.plot(thresholds, precisions[1:]);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 89,
   "metadata": {},
   "outputs": [],
   "source": [
    "idx = len(precisions[precisions < 0.9])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 90,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "-2956.070336352813"
      ]
     },
     "execution_count": 90,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "thresholds[idx]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 91,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_train_pred_90 = (y_scores > 21454)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 92,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.9904829883416607"
      ]
     },
     "execution_count": 92,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "precision_score(y_train_0, y_train_pred_90)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 93,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.7028532838088807"
      ]
     },
     "execution_count": 93,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "recall_score(y_train_0, y_train_pred_90)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Setting High Precisions\n",
    "\n",
    "Let's aim for 99% precisions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 94,
   "metadata": {},
   "outputs": [],
   "source": [
    "idx = len(precisions[precisions < 0.99])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is the same as the line above"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "idx = len(precisions) - len(precisions[precisions > 0.99])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 95,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "21040.887989945048"
      ]
     },
     "execution_count": 95,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "thresholds[idx]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 96,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_train_pred_90 = (y_scores > thresholds[idx])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 97,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.9901176470588235"
      ]
     },
     "execution_count": 97,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "precision_score(y_train_0, y_train_pred_90)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 98,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.7104507850751308"
      ]
     },
     "execution_count": 98,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "recall_score(y_train_0, y_train_pred_90)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Exercise\n",
    "\n",
    "High Recall Score. Recall score > 0.9"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 99,
   "metadata": {},
   "outputs": [],
   "source": [
    "idx = len(recalls[recalls > 0.9])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 100,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "8130.108485593395"
      ]
     },
     "execution_count": 100,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "thresholds[idx]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 101,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_train_pred_90 = (y_scores > thresholds[idx])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 102,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.966799709724238"
      ]
     },
     "execution_count": 102,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "precision_score(y_train_0, y_train_pred_90)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 103,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.8997129832854972"
      ]
     },
     "execution_count": 103,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "recall_score(y_train_0, y_train_pred_90)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The Receiver Operating Characteristics (ROC) Curve"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Instead of plotting precision versus recall, the ROC curve plots the `true positive rate` (another name for recall) against the `false positive rate`. The `false positive rate` (FPR) is the ratio of negative instances that are incorrectly classified as positive. It is equal to one minus the `true negative rate`, which is the ratio of negative instances that are correctly classified as negative.\n",
    "\n",
    "The TNR is also called `specificity`. Hence the ROC curve plots `sensitivity` (recall) versus `1 - specificity`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![](img/tnr_and_fpr.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 104,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.metrics import roc_curve"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 105,
   "metadata": {},
   "outputs": [],
   "source": [
    "fpr, tpr, thresholds = roc_curve(y_train_0, y_scores)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 106,
   "metadata": {},
   "outputs": [],
   "source": [
    "def plot_roc_curve(fpr, tpr, label=None):\n",
    "    plt.plot(fpr, tpr, linewidth=2, label=label)\n",
    "    plt.plot([0,1], [0,1], 'k--')\n",
    "    plt.axis([0, 1, 0, 1])\n",
    "    plt.xlabel('False Positive Rate')\n",
    "    plt.ylabel('True Positive Rate')\n",
    "    plt.title('ROC Curve')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 107,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAtgAAAHwCAYAAACcxBjDAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJzs3Xd8leX9//HXlTACCIqogICCCkoY4kRQnCgyVBQFBCcggiAurPJ1tdbR+rPV1tVase6qtY7aWreCWEWEMBSKAwdbVFT2SK7fHwmKSCBITu7cyev5ePAg55z73OcN4uHNlc+57hBjRJIkSVLpyEo6gCRJklSRWLAlSZKkUmTBliRJkkqRBVuSJEkqRRZsSZIkqRRZsCVJkqRSZMGWJEmSSpEFW5LKQAjh0xDCihDC0hDCghDCfSGEbTY4pmMI4dUQwpIQwrchhGdDCLkbHFMnhHBrCOHzonN9VHR7h2JeN4QQRoQQ3gshLAshzAkh/D2E0CaTv15Jqsws2JJUdo6LMW4DtAP2AUateyCE0AF4EXgG2BloBkwB3gwh7FZ0TDXgFaAVcCxQB+gIfAUcWMxr/gG4ABgBbA+0AJ4Gum9p+BBClS19jiRVRsErOUpS5oUQPgUGxRhfLrp9E9Aqxti96PYbwLQY43kbPO8/wKIY4xkhhEHA9cDuMcalJXjN5sD/gA4xxneKOeZ14KEY4z1Ft88qynlI0e0IDAcuBKoALwBLY4wj1zvHM8CYGOPvQwg7A7cBhwJLgVtijH8swW+RJFUYrmBLUhkLITQGugIfFd2uSeFK9N83cvjjwNFFX3cGni9JuS5yFDCnuHK9BXoC7YFc4BGgTwghAIQQ6gLHAI+GELKAZylceW9U9PoXhhC6bOXrS1KqWLAlqew8HUJYAswGvgCuKbp/ewrfj+dv5DnzgXXz1fWKOaY4W3p8cW6MMX4dY1wBvAFEoFPRYycDb8UY5wEHADvGGK+NMa6OMc4C/gL0LYUMkpQaFmxJKjs9Y4y1gcOBvfihOC8GCoCGG3lOQ+DLoq+/KuaY4mzp8cWZve6LWDhX+ChwatFd/YCHi77eFdg5hPDNuh/A/wH1SyGDJKWGBVuSyliMcQxwH3Bz0e1lwFvAKRs5vDeFH2wEeBnoEkKoVcKXegVoHELYfxPHLANqrne7wcYib3D7b8DJIYRdKRwd+UfR/bOBT2KM2633o3aMsVsJ80pShWDBlqRk3AocHUJoV3T7cuDMoi31aocQ6oYQrgM6AL8qOuZBCkvsP0IIe4UQskII9UII/xdC+EmJjTF+CNwJ/C2EcHgIoVoIISeE0DeEcHnRYZOBk0IINUMIewADNxc8xpgHLALuAV6IMX5T9NA7wHchhMtCCDVCCNkhhNYhhAN+zm+QJKWVBVuSEhBjXAQ8AFxVdHsc0AU4icK56c8o3MrvkKKiTIxxFYUfdPwf8BLwHYWldgdgfDEvNQK4HbgD+Ab4GDiRwg8jAtwCrAYWAvfzw7jH5vytKMsj6/2a8oHjKNyG8BMKR1vuAbYt4TklqUJwmz5JkiSpFLmCLUmSJJWijBXsEMK9IYQvQgjvFfN4CCH8segyv1NDCPtmKoskSZJUVjK5gn0fhZfyLU5XoHnRj8HAXRnMIkmSJJWJjBXsGONY4OtNHHIC8EAs9DawXQihNPZrlSRJkhKT5Ax2I9a7eAEwp+g+SZIkKbWqJPjaYSP3bXRLkxDCYArHSKhVq9Z+e+21VyZzqZxbWxAp3PwmUvTT9370BygWHrG24Kd/rH66eU784bkR1hTE7/+AxvWetP7TYoSVa/Kpkv3Tf6d+f+RG/kQXu29P/OGxlWvyqZIVCCFs9FwbO8f6v6Zlq9dSvUrW9/dFYE1+QXGvLEmq5DZWykr7pCEzr7Kpl9zyAwoKWLPkK9Yu+4ZQpRpx7eovY4w7bmmOJAv2HKDJercbA/M2dmCM8W7gboD9998/vvvuu5lPp42KsbDcFsRI/vpfF0QKIixdtZYVq9eSXwBrCwpYuaaAKbO/4atlq5j02TfUrVUVgIKCwucVxMJzFhSV14IIEz75ml22r0lBLCzHn3y5jOpVslhbUPg6ZfZr3czjAaiRodfO2crn1/mZzwuh8NcVQij6ueh+Ch8I3x8Tvj922ep8Gtet8aNz/Oic672b/fSxDV8/FPvYT9+oN/68nz5WfJ6NPV5cns2et+h2jPDhwqW03237H50jbHDcD7c3/vi6ezY8/sf3bXDMBvezkdfKLyhg4XeraLXzxv+UFPfbseHvxeZs8ve1mFcp7jmbeuVin1PMA5v8VRT7+mWQdwtfo7jXWbR0FbtsX4ucqlmbPfYnf743ctBGX34jB278/Bs73897zZKcq7gnl+jXXtLX3OC+pSvX0rhuTbKywo/+H173Hvn9uTdyXwg/vI/+cO4fjskKgbo1q/74eeHH78/rvxcTCp+z4Xt4IFCnRpUt/n+4snrooYc4/fTTGT58ODfccAN16tT57OecJ8mC/U9geAjhUQovtfttjHF+gnlSI8bIqrUFfLVsNfn5kbUFBUUlF75auor8ouJaECMFBT98HWNkycq1LFq6ilrVqhSV4vh90V26ci2ffLWMuYtXUDunCjEWluRpc76lZvUqLFqyqsx+jTMXLvnR7VVrf7r62nDbnB+9SWWFQFb48ZtWVgjMWbyCJtvXoF6t6t8f98ObVdFz1vt63Rvcx4uW0r7Z9kXn/eHc675e94b6xZJV7NWg9k/fVNcrTj8qruu9oVLcGyXw7Yo17FqvFtlZ6x3zozfXH+5f96a6/pt41ews6tasSpWsLLKyoEpWFtWrZFE7p8pPC7RvvJKkSmrx4sVMmTKFww8/nP79+9O6dWvatWu3+SduQsYKdgjhb8DhwA4hhDnANUBVgBjjn4DngG7AR8By4OxMZUmT71au4eMvlvL1stV8/vVy1uZHXvnfQt6f+x1LVq0lhI2NN2TestX5P7qdnVVUMjconPkFkeWr82lRfxuys7KokhVYtmottXOqcFTL+hTESPOdaq9XVn8ovBv+vFPtHKpkB6pmZbFtjarUrJ79/diEJEnS1nrqqac477zzWL16NZ9//jm1atXa6nINGSzYMcZTN/N4BIZl6vXLwso1+Xz0xVJmzP+O7KJyuW4lOT9GPly4hG1rVP3+vlg0SpEfIxM/W0zjujX47KvlzP56Od+tXFui11xXrqtmB9bkR+rVqkbtnCpkZ4Wilc7AJ18tW2/ltbAMr19mFy9fzXY1qtFg25yfHLM2v4Ca1bKpW6sau+24DVWzAlWys6iSHWi4bQ51cqpSq3qS3/iQJEnaOgsWLOD888/niSeeoF27dtx7773UqlWr1M5vU9qMb1esYfbXy1m2ai2ffLmMV//3BXmzvymVcYmpc74t9rGa1bJp3WhbalbLpmm9WlTNDhzVsj4tG9RxlkqSJOlnWrRoEa1atWLZsmXccMMNjBw5kqpVq5bqa1iw1/P1stX8Y+Icnsyby6q1+cxatKxEz6teJYtdtq9Jm8bbkh0KV5KzsgLZITDvmxW0a7Jd4e0NxirW5BfQbIdahBDYtV5Ndtm+JtWrZFmeJUmSStmSJUuoXbs2O+64I6NGjaJHjx5kame6Sl2wZy1ayq0vf8hbs77a7Ip0sx1qUbNaNttUr0L7ZttzQLPtadtoO7atWbr/4pEkSVLpKSgo4M477+Sqq67i1VdfZZ999mHkyJEZfc1KV7C/W7mGY28Zy7xvVxZ7TJdW9Tl4jx3Yd5e6tKhfm2pVkrwejyRJkn6O//3vfwwaNIg333yTLl26sP3225fJ61aagr3g25UMfXgieZ9/85PHOrfcif7td+Wg3epRo1p2AukkSZJUmm6++WauvPJKatasyf3338/pp59eZmO4Fb5g//fjL+n3l/E/uX/IYbtz2bF7Ou8sSZJUAS1dupTjjjuO22+/nfr165fpa1fogv3M5Llc8OjkH9132kG78OsTWlusJUmSKpCVK1dy7bXXcsghh9CtWzeuvvpqsrKSGfOtsAX7mmfe4/63fri65T1n7E/n3LL914skSZIyb9y4cQwaNIiZM2dy+eWX061bt8TKNVTQgv2XsbN+VK4/ur4rVbL9oKIkSVJFsmTJEkaNGsUdd9xB06ZNefHFFzn66KOTjkWFa50ffbGE65+b8f3tDy3XkiRJFdIzzzzDnXfeyYgRI5g2bVq5KNdQwVawF3y7ks6/H/v97YlXdqaq5VqSJKnC+Oqrr5gyZQpHHnkk/fv3p23btrRt2zbpWD9SYdrnF0tWctCNr3x/+5lhB1Nvm+oJJpIkSVJpiTHyxBNPkJubyymnnMKyZcsIIZS7cg0VpGAvXraaA6//oVzfeFIb9m6yXYKJJEmSVFrmz59Pr169OOWUU2jcuDGvvPIKtWrVSjpWsSrEiMiQhyZ+//Wjgw/ioN3qJZhGkiRJpeWLL74gNzeXlStXctNNN3HRRRdRpUr5rrDlO10JxBgZ/8nXAFxwVHPLtSRJUgWwZMkSateuzU477cSVV17JcccdR4sWLZKOVSKpHxG5Yb0dQ847YvcEk0iSJGlr5efn88c//pEmTZqQl5cHwCWXXJKacg0VoGC/NnMRALvtUIvqVbITTiNJkqSfa/r06XTq1IkLLriAjh07ssMOOyQd6WdJfcEuKIgAXHVcbsJJJEmS9HP99re/ZZ999mHmzJk8+OCD/Pvf/6ZJkyZJx/pZUj2DvXJNPp9+tQyAg5o5ey1JkpRWK1asoGfPntx2223stNNOScfZKqku2O988jUFEXbbsRY1qjkeIkmSlBYrVqzgl7/8JYceeijdu3fn6quvJisr9cMVQMoL9riPvgSggzuHSJIkpcaYMWMYNGgQH330EVWrVqV79+4VplxDymew/zl5HgAHNN0+4SSSJEnanO+++46hQ4dy+OGHk5+fz8svv8x1112XdKxSl9qCXVAQWbpqLQBNtq+RcBpJkiRtzrPPPsvdd9/NxRdfzLRp0zjqqKOSjpQRqR0RmfvNCpauWsu2Naqy7y51k44jSZKkjfjyyy+ZPHkynTt3pl+/frRr145WrVolHSujUruC/drMLwDYs0FtQggJp5EkSdL6Yow89thj5Obm0qdPH5YtW0YIocKXa0hxwV65Jj/pCJIkSdqIefPm0bNnT/r27cuuu+7K66+/Tq1atZKOVWZSOyIyY/4SANo38wOOkiRJ5cUXX3xBq1atWLlyJTfffDMXXHABVaqktnL+LKn91dbJKYzuSrYkSVLyvvvuO+rUqcNOO+3ENddcQ48ePdhjjz2SjpWI1I6IzFm8AoB2TfyAoyRJUlLy8/O55ZZbaNKkCZMmTQLgwgsvrLTlGlK8gj178XLALfokSZKS8t577zFo0CDGjx9P9+7dU3+J89KS2hXsDxYuBaDBtjkJJ5EkSap8brzxRvbdd18+/vhjHnnkEZ599lkaN26cdKxyIZUFe/2563q1qieYRJIkqXJas2YNp5xyCtOnT+fUU0912+T1pHJEZN38NUB2lv8xJUmSMm358uVcc801HHbYYfTo0YOrrrrKUl2MVK5gL16+GoBa1bITTiJJklTxvf7667Rt25abb76Zd955B8ByvQmpLNjvzf0WgE7Nd0w4iSRJUsX17bffcu6553LEEUcA8Nprr3HttdcmnKr8S2XBLoiFP3+7Yk2yQSRJkiqwZ599lnvuuYeRI0cydepUDj/88KQjpUIqZ7Cnz/sOgP12dQ9sSZKk0rRo0SImT57M0UcfTf/+/dl3333Jzc1NOlaqpHIFe9HSVQCszi9IOIkkSVLFEGPkkUceoWXLlvTt25dly5YRQrBc/wypLNi1iy6TXr+Oe2BLkiRtrTlz5nD88cfTv39/9thjD8aOHUutWrWSjpVaqRwRWb5qLQBN69VMOIkkSVK6LVy4kFatWrF27VpuueUWzj//fLKz3alta6SyYK+7iuO2NaomnESSJCmdvvnmG7bbbjvq16/Pr3/9a3r06MFuu+2WdKwKIZUjIl8tK5zBrmPBliRJ2iJr167l5ptvpkmTJkycOBGAESNGWK5LUSpXsGtVq8LKNaupW7Na0lEkSZJSY+rUqQwcOJB3332X448/noYNGyYdqUJK6Qp24ZUcq1dNZXxJkqQyd/3117Pffvvx2Wef8dhjj/H000+z8847Jx2rQkp1Q82p4gC+JElSScQY6du3LzNmzKB3795e6jyDUjcikr/uMo5AtSqp/veBJElSxixbtoyrrrqKI444guOOO44rrrjCUl1GUtdQ1y/YkiRJ+qlXXnmFNm3acMstt3z/QUbLddlJXcEuiIUFu0X9bRJOIkmSVL588803DBo0iM6dO1OlShXGjBnDL3/5y6RjVTqpK9jrVrC/W7E24SSSJEnly3PPPcd9993HZZddxpQpUzj00EOTjlQppW4Ge92EyPa13KJPkiRp4cKFTJ48mS5dunDqqaey3377seeeeyYdq1JL3Qr2uhGRXbb3MumSJKnyijHy4IMPkpubS//+/Vm2bBkhBMt1OZC6gh2LCnaOe2BLkqRK6vPPP6d79+6cccYZ7LnnnrzxxhvUqlUr6VgqkroRkaJ+TZVsC7YkSap8Fi5cSOvWrcnPz+cPf/gDw4YNIzvba4OUJ+kr2EU/V8lyqxlJklR5LF68mLp161K/fn1uuOEGevToQdOmTZOOpY1I3TLw9wU724ItSZIqvrVr13LTTTfRpEkT3n33XQCGDx9uuS7HUreCvWptPtWAbDdLlyRJFdzkyZMZOHAgkyZN4sQTT6RRo0ZJR1IJpG4Fu0oojLx4+ZqEk0iSJGXOtddey/7778/cuXN54oknePLJJ2nYsGHSsVQCqSvY+UWfcmzZsE7CSSRJkjInKyuL0047jenTp9OrV6+k42gLpG5EZMXqfKoC9bbxQjOSJKniWLp0KVdccQVHHnkkJ5xwAldccQXBkdhUSt0K9jpZ/oGTJEkVxIsvvkjr1q257bbbmDx5MoDlOsVSV7DX/VnbsXb1ZINIkiRtpcWLF3P22WfTpUsXcnJyGDt2LNdcc03SsbSVUlewV6zJB6BmNTdUlyRJ6fbcc8/x4IMP8n//939MnjyZQw45JOlIKgWpm8Fed4EZrzMjSZLSaMGCBUyePJljjz2Wfv36ccABB9CiRYukY6kUpW4Fe51tqldNOoIkSVKJxRi5//77yc3N5bTTTmP58uWEECzXFVBqC3Z2apNLkqTK5rPPPqNr166cddZZ5ObmMm7cOGrWrJl0LGVI6kZEirbBJjvLhi1Jksq/hQsX0qZNGwoKCrjttts477zzyLLHVGipK9jreKl0SZJUnn311VfUq1eP+vXr85vf/Ibu3buz6667Jh1LZSB1/3wqWsDGf/hJkqTyaM2aNdxwww3ssssuTJgwAYDzzjvPcl2JpG4Fu6BoRqSKDVuSJJUzeXl5DBgwgMmTJ3PKKaewyy67JB1JCUhdS80vKCzYbtMnSZLKk1/96lcccMABLFiwgCeffJLHH3+c+vXrJx1LCUhdwV43e13VbUQkSVI5UrVqVc4880ymT5/OiSeemHQcJSh1IyI/zGC7hC1JkpKzZMkSRo0axdFHH80JJ5zAqFGjCG7CIFK4gr2O/VqSJCXl+eefp3Xr1tx5551MnToVwHKt76W4YPuHWJIkla2vv/6aM888k65du1KzZk3GjRvHVVddlXQslTOpK9iRdR9ytGBLkqSy9fzzz/PII49w5ZVXMnnyZDp27Jh0JJVDqZvBXjeEbb+WJEllYf78+UyaNInu3btz6qmncuCBB7LHHnskHUvlWOpWsNdxBVuSJGVSjJF7772Xli1bcuaZZ7J8+XJCCJZrbVbqCvb3u4jYryVJUoZ88sknHHPMMQwcOJC9996b//73v9SsWTPpWEqJjBbsEMKxIYSZIYSPQgiXb+TxXUIIr4UQ8kIIU0MI3Up6blewJUlSJixcuJA2bdowfvx47rrrLl577TVatGiRdCylSMZmsEMI2cAdwNHAHGBCCOGfMcbp6x12JfB4jPGuEEIu8BzQtCTndx9sSZJUmr788kt22GEH6tevz80330z37t1p0qRJ0rGUQplcwT4Q+CjGOCvGuBp4FDhhg2MiUKfo622BeSU5sd1akiSVljVr1nDdddexyy678M477wAwZMgQy7V+tkzuItIImL3e7TlA+w2O+SXwYgjhfKAW0DmDeSRJkn5k4sSJDBgwgKlTp9KnTx+aNm2adCRVAJlcwd7YOnPc4PapwH0xxsZAN+DBEMJPMoUQBocQ3g0hvAtQsOFZJEmSttDVV1/NgQceyJdffsnTTz/No48+yk477ZR0LFUAmSzYc4D1v7fSmJ+OgAwEHgeIMb4F5AA7bHiiGOPdMcb9Y4z7A9Sqlp2RwJIkqfKoUaMGAwcO5P333+eEEzacYpV+vkwW7AlA8xBCsxBCNaAv8M8NjvkcOAoghNCSwoK9aHMndgcRSZK0pb777juGDh3KU089BcDll1/O3XffzXbbbZdwMlU0GSvYMca1wHDgBWAGhbuFvB9CuDaEcHzRYZcA54QQpgB/A86KMW52AMQdRCRJ0pb497//TatWrbj77ruZMWMGAMEFO2VIRi+VHmN8jsKt99a/7+r1vp4OHLyl57VfS5Kkkvjyyy+58MILefjhh2nVqhVPPPEE7dtvuOeCVLpSdyVHcEREkiSVzEsvvcTjjz/ONddcw6RJkyzXKhMZXcHOFL+lI0mSijNv3jwmTZpEjx496Nu3L+3bt2e33XZLOpYqkVSuYGenMrUkScqkGCP33HMPubm5nH322SxfvpwQguVaZS6VVdUREUmStL5Zs2bRuXNnzjnnHPbZZx/eeustatasmXQsVVKpHBGxYEuSpHUWLlxI27Ztyc7O5s9//jODBg0iKyuVa4iqIFJZsCVJkhYtWsSOO+5I/fr1+f3vf0+3bt1o3Lhx0rGkdI6IzP1mRdIRJElSQlavXs2vfvUrdtllF8aPHw/A4MGDLdcqN1K5gr37jrWSjiBJkhLwzjvvMHDgQN577z369+/P7rvvnnQk6SdSuYLtDLYkSZXPlVdeSYcOHVi8eDH/+te/eOihh9hhhx2SjiX9hAVbkiSlQu3atTnnnHN4//336d69e9JxpGKlckTEfi1JUsX37bff8otf/IIuXbpw0kkn8Ytf/MKLzSkVUlmwXcGWJKlie/bZZxkyZAgLFiygadOmgFdyVnqkc0QklaklSdLmLFq0iH79+nH88cdTr1493n77bUaNGpV0LGmLpLKquoItSVLF9Morr/DEE09w7bXX8u6773LAAQckHUnaYqkcEfFbRJIkVRyzZ88mLy+P448/nj59+tC+fXuaNWuWdCzpZ0vpCnbSCSRJ0tYqKCjgT3/6E61atWLQoEEsX76cEILlWqmX0oJtw5YkKc0+/PBDjjzySIYOHcqBBx7I22+/Tc2aNZOOJZWKVI6IuIItSVJ6LViwgHbt2lG1alVGjx7N2Wef7finKpRUFuxFS1YlHUGSJG2hhQsXUr9+fRo0aMAf/vAHunXrxs4775x0LKnUpXJEJKdqdtIRJElSCa1atYqrr76aXXfdlbfffhuAQYMGWa5VYaVyBbt+nZykI0iSpBJ4++23GThwINOnT+f000+nefPmSUeSMi6VK9jOYEuSVP6NGjWKjh07smTJEp577jkeeOAB6tWrl3QsKeNSWrBt2JIklXd169Zl6NChvPfee3Tt2jXpOFKZSeWIiP1akqTy55tvvmHkyJF07dqVXr168Ytf/CLpSFIiUlqwbdiSJJUnTz/9NOeddx5ffPEFLVq0SDqOlKiUjogknUCSJEHh1nu9e/fmxBNPZKeddmL8+PGuXKvSS2nBtmFLklQevP766zzzzDNcf/31TJgwgf322y/pSFLiUjoiknQCSZIqr88//5y8vDxOOOEEevfuzUEHHcSuu+6adCyp3EjlCrYz2JIklb2CggLuvPNOWrVqxeDBg1mxYgUhBMu1tIFUFmxHRCRJKlsffPABhx9+OMOGDaNDhw6MHz+eGjVqJB1LKpdSOSLihxwlSSo7CxYsoF27dlSvXp2//vWvnHnmmX43WdqElBZs/6eWJCnT5s+fT8OGDWnQoAG33347Xbt2pWHDhknHksq9VI6IfLdiTdIRJEmqsFauXMkVV1xB06ZNeeuttwAYMGCA5VoqoVSuYK8piElHkCSpQnrzzTcZOHAgM2fO5KyzzmLPPfdMOpKUOqlcwW5YJyfpCJIkVTiXXXYZnTp1YuXKlbzwwgv89a9/Zfvtt086lpQ6qSzYjmBLklT6dtxxR4YPH857773HMccck3QcKbVSOSJiwZYkaestXryYSy65hK5du3LKKacwcuTIpCNJFUJKV7Bt2JIkbY0nn3yS3NxcHnjgAT799NOk40gVSjoLdtIBJElKqQULFnDyySfTq1cvGjZsyLvvvsull16adCypQklnwbZhS5L0s4wZM4Z//etf3HjjjYwfP5527dolHUmqcNI5g+0atiRJJfbpp58yadIkTjrpJHr37k3Hjh1p0qRJ0rGkCiuVK9heKl2SpM0rKCjgtttuo3Xr1gwdOpQVK1YQQrBcSxmWyoLthxwlSdq0GTNm0KlTJ0aMGEGnTp2YMGECNWrUSDqWVCmkckREkiQVb/78+ey7777UrFmTBx54gNNOO83FKakMpbJgZ/kmIUnST8ybN4+dd96Zhg0bctddd9G1a1fq16+fdCyp0knpiEjSCSRJKj9WrFjBqFGjaNq0KW+99RYAZ511luVaSkgqV7Dt15IkFXrjjTcYNGgQH3zwAQMGDGCvvfZKOpJU6aVyBTvLbUQkSWLkyJEceuihrF69mpdeeonRo0dTt27dpGNJlV4qC7b1WpIkaNiwIRdccAHTpk2jc+fOSceRVCSVIyI2bElSZfTVV19x0UUX0aNHD3r37s0ll1ySdCRJG5HSFWwbtiSp8ogx8vjjj9OyZUv+9re/MWfOnKQjSdqEVK5gO4ItSaos5s2bx7Bhw3j66afZb7/9ePnll2nbtm3SsSRtQipXsJetWpt0BEmSysSbb77J888/z0033cTbb79tuZZSIJU4AiwqAAAgAElEQVQr2PkxJh1BkqSMmTVrFnl5efTq1YuTTz6Zjh070qhRo6RjSSqhVK5g16tVPekIkiSVuvz8fG699VbatGnDsGHDWLFiBSEEy7WUMqks2F7JUZJU0UyfPp1DDjmEiy66iMMPP5wJEyZQo0aNpGNJ+hlSOSLiLiKSpIpk/vz57LffftSqVYuHHnqIfv36EVxNklIrlQVbkqSKYM6cOTRu3JiGDRvy5z//mWOPPZaddtop6ViStpIjIpIklbHly5dz6aWX0qxZM/773/8CcMYZZ1iupQoilSvY9mtJUlqNGTOGQYMG8dFHH3HOOefQqlWrpCNJKmWuYEuSVEYuvvhiDj/8cAoKCnjllVe4++672XbbbZOOJamUpbRg27AlSenTpEkTLrnkEqZNm8aRRx6ZdBxJGZLKERFJktLgyy+/5MILL+S4446jT58+XHTRRUlHklQGUrqCnXQCSZKKF2Pk0UcfpWXLljz++OPMmzcv6UiSylA6C7Yfc5QklVNz586lZ8+enHrqqTRr1oyJEye6ci1VMuks2PZrSVI59d///peXXnqJ3/3ud7z11lu0adMm6UiSypgz2JIkbaWPPvqIvLw8TjnlFE4++WQOPvhgdt5556RjSUpIOlewkw4gSRKQn5/P7373O9q2bcv555/PihUrCCFYrqVKLp0F24YtSUrYe++9R4cOHRg5ciSdO3dm4sSJ1KhRI+lYksqBVI6I+CFHSVKS5s+fz/7770+dOnV49NFH6d27t9dokPS9dBZs38MkSQmYPXs2TZo0oWHDhowePZouXbqwww47JB1LUjmTyhERSZLK0vLly7nkkkvYbbfdePPNNwHo37+/5VrSRqV0BdslbElS2XjttdcYNGgQs2bNYsiQIW67J2mzUrmCbb2WJJWFCy+8kCOPPJKsrCxef/117rrrLurUqZN0LEnlXDoLtg1bklQGmjZtyqWXXsqUKVM47LDDko4jKSVSOSIiSVImfPHFF4wYMYKePXvSt29fLrzwwqQjSUqhdK5gJx1AklShxBh5+OGHyc3N5cknn2ThwoVJR5KUYuks2M6ISJJKyezZsznuuOM47bTTaN68OZMnT+aCCy5IOpakFMtowQ4hHBtCmBlC+CiEcHkxx/QOIUwPIbwfQnikZOct3ZySpMpr/PjxvPbaa9x6662MGzeO3NzcpCNJSrmMzWCHELKBO4CjgTnAhBDCP2OM09c7pjkwCjg4xrg4hLBTic6dicCSpErjww8/ZNKkSfTp04devXpxyCGH0KBBg6RjSaogMrmCfSDwUYxxVoxxNfAocMIGx5wD3BFjXAwQY/yiJCdetbagVINKkiqHtWvX8v/+3/+jbdu2XHjhhaxYsYIQguVaUqkqUcEOIVQLIeyxheduBMxe7/acovvW1wJoEUJ4M4Twdgjh2JKceHW+BVuStGWmTp1Khw4d+MUvfkGXLl2YOHEiNWrUSDqWpAposwU7hNAdmAa8VHS7XQjhqRKce2OTHHGD21WA5sDhwKnAPSGE7TaSYXAI4d0QwrsA29esVoKXlySp0Pz58znwwAP5/PPPefzxx3nqqafYeeedk44lqYIqyQr2tUB74BuAGONkoCSr2XOAJuvdbgzM28gxz8QY18QYPwFmUli4fyTGeHeMcf8Y4/7ghxwlSSXz2WefAdCwYUPuvfdepk+fzimnnOJuVJIyqiQFe02M8ZsN7ttwJXpjJgDNQwjNQgjVgL7APzc45mngCIAQwg4UjozM2tyJgx9zlCRtwrJly7jooovYfffdefPNNwHo168f9erVSziZpMqgJLuIzAgh9AayQgjNgAuAtzf3pBjj2hDCcOAFIBu4N8b4fgjhWuDdGOM/ix47JoQwHcgHLo0xfrXZRPZrSVIxXn75Zc455xw+/fRTzjvvPNq0aZN0JEmVTEkK9nDgaqAAeJLCUjyqJCePMT4HPLfBfVev93UELi76UWL2a0nSxpx//vncfvvttGjRgrFjx9KpU6ekI0mqhEoyItIlxnhZjHGfoh+XA10zHWxTnJ2TJG1M8+bNufzyy5kyZYrlWlJiSlKwr9zIfVeUdpAtkWW/liQBCxcupHfv3vztb38DYMSIEdx4443k5OQknExSZVbsiEgIoQtwLNAohPD79R6qQ+G4SGJcwJakyi3GyIMPPsiFF17IsmXLXK2WVK5sagb7C+A9YCXw/nr3LwEuz2SozXEXEUmqvD7//HPOPfdcnn/+eTp27Mjo0aPZa6+9ko4lSd8rtmDHGPOAvBDCwzHGlWWYabNcwZakymvChAm88cYb3HbbbZx33nlkZZXoosSSVGZKsotIoxDC9UAu8P1QW4yxRcZSSZK0npkzZ5KXl0ffvn3p1asXhxxyCPXr1086liRtVEn+2X8f8FcKd8frCjwOPJrBTJvlLiKSVDmsWbOG3/zmN+y9995cfPHFrFixAsByLalcK0nBrhljfAEgxvhxjPFKiq6+mBTrtSRVfHl5ebRv355Ro0bRo0cPJk2aRI0aNZKOJUmbVZIRkVWhcMn44xDCEGAusFNmY22aC9iSVLHNmzePgw46iLp16/LEE0/Qq1evpCNJUomVpGBfBGwDjACuB7YFBmQy1Oa4i4gkVUyffPIJzZo1Y+edd+b++++nS5cu1K1bN+lYkrRFNjsiEmMcH2NcEmP8PMZ4eozxeOCzMshWLFewJaliWbp0KSNGjGCPPfZg3LhxAPTt29dyLSmVNlmwQwgHhBB6hhB2KLrdKoTwAPB2maQrLleSLy5JKlUvvvgirVu35vbbb2f48OG0a9cu6UiStFWKLdghhBuBh4H+wPMhhCuA14ApQKJb9LmCLUkVw/Dhw+nSpQs1atTgjTfe4A9/+APbbLNN0rEkaatsagb7BGDvGOOKEML2wLyi2zPLJlrx3KZPktItxkgIgb322ov/+7//46qrriInJ2fzT5SkFNhUwV4ZY1wBEGP8OoTwv/JQrsEREUlKq/nz5zN8+HBOPvlkTj31VIYPH550JEkqdZsq2LuFEJ4s+joATde7TYzxpIwm2wRXsCUpXWKM3H///Vx00UWsWLGCo446KulIkpQxmyrYG246ensmg2wJ67Ukpcenn37K4MGDeemll+jUqRN/+ctf2HPPPZOOJUkZU2zBjjG+UpZBtsTKtflJR5AklVBeXh5vvfUWd9xxB0OGDCErqyQXEZak9CrJhWbKnepVspOOIEnahBkzZpCXl0e/fv048cQTmTVrFjvuuGPSsSSpTKRyGaFalVTGlqQKb82aNVx//fW0a9eOSy+9lBUrVgBYriVVKiVuqiGE6pkMIklKt0mTJnHAAQdw5ZVX0rNnTyZNmkSNGjWSjiVJZW6zBTuEcGAIYRrwYdHtvUMIt2U82aYyJfnikqSfmDdvHh06dGDhwoU89dRTPPbYY9SvXz/pWJKUiJKsYP8R6AF8BRBjnAIckclQm+MufZJUPsyaNQuAnXfemQcffJDp06fTs2fPhFNJUrJKUrCzYoyfbXBfott4BNewJSlR3333HcOGDaN58+aMGzcOgN69e1O3bt2Ek0lS8kqyi8jsEMKBQAwhZAPnAx9kNpYkqbz6z3/+w7nnnsucOXO44IIL2GeffZKOJEnlSkkK9lAKx0R2ARYCLxfdlxhHRCQpGUOHDuVPf/oTLVu25M0336RDhw5JR5KkcqckBXttjLFvxpNsAfu1JJWdGCMAIQRat27NVVddxRVXXEH16m4uJUkbU5KCPSGEMBN4DHgyxrgkw5k2z4YtSWVi3rx5DBs2jJNPPpn+/fszbNiwpCNJUrm32Q85xhh3B64D9gOmhRCeDiGUqxVtSVLpijEyevRocnNzef7551myJPm1FUlKixJdaCbG+N8Y4whgX+A74OGMptoMdxGRpMyZNWsWnTt3ZtCgQbRr145p06YxZMiQpGNJUmqU5EIz24QQ+ocQngXeARYBHTOebJOZknx1SarYpkyZwoQJE/jTn/7Eq6++yh577JF0JElKlZLMYL8HPAvcFGN8I8N5SsR+LUmla/r06eTl5dG/f39OPPFEOnXqxA477JB0LElKpZKMiOwWYzy/vJRrKPwkuyRp661evZpf//rXtGvXjssuu4yVK1cCWK4laSsUu4IdQvhdjPES4B8hhLjh4zHGkzKaTJKUURMmTGDgwIFMmzaNvn378sc//pGcnJykY0lS6m1qROSxop9vL4sgW8IFbEnaOnPnzuXggw9mxx135JlnnuH4449POpIkVRjFFuwY4ztFX7aMMf6oZIcQhgOvZDLYptivJenn+fDDD2nevDmNGjXikUceoXPnzmy33XZJx5KkCqUkM9gDNnLfwNIOsiVcwZakLfPtt98yZMgQ9txzT954o/AjNSeffLLlWpIyYFMz2H2AvkCzEMKT6z1UG/gm08EkSaXj3//+N+eeey7z58/n4osvZr/99ks6kiRVaJuawX4H+ApoDNyx3v1LgLxMhto8l7AlqSTOPfdc7r77blq3bs2TTz7JgQcemHQkSarwNjWD/QnwCfBy2cUpGUdEJKl4MRZu/BRCoF27dvzyl79k1KhRVKtWLeFkklQ5bGpEZEyM8bAQwmJg/W36AhBjjNtnPF0x7NeStHFz585l6NCh9OnTh/79+zN06NCkI0lSpbOpDzkeUfTzDsCO6/1Yd1uSVE7EGPnLX/5Cbm4uL7/8MkuXLk06kiRVWsUW7BhjQdGXTYDsGGM+0AE4F6hVBtmK5ZUcJekHH3/8MUcddRSDBw9mv/32Y9q0aZx77rlJx5KkSqsk2/Q9DcQQwu7AA0BL4JGMptoM67Uk/WDatGlMnDiRu+++m1deeYXdd9896UiSVKltaheRdQpijGtCCCcBt8YY/xhCSHQXERewJVV27733HpMmTeKMM86gZ8+ezJo1i3r16iUdS5JEyVaw14YQTgFOB/5VdF/VzEXavOAatqRKavXq1fzyl79k33335YorrmDlypUAlmtJKkdKeiXHI4CbYoyzQgjNgL9lNpYkaUPjx49n33335Ve/+hV9+vQhLy+PnJycpGNJkjaw2RGRGON7IYQRwB4hhL2Aj2KM12c+WvEcEZFU2cydO5dOnTpRv359/vWvf9G9e/ekI0mSirHZgh1C6AQ8CMyl8POFDUIIp8cY38x0OEmq7D744ANatGhBo0aNeOyxxzjqqKOoU6dO0rEkSZtQkhGRW4BuMcaDY4wdge7AHzIba9NcwZZU0X3zzTcMHjyYvfbai7FjxwJw4oknWq4lKQVKsotItRjj9HU3YowzQgiJXm83vyBu/iBJSql//vOfDB06lAULFnDppZdywAEHJB1JkrQFSlKwJ4UQ/kzhmAhAfyDRbfqqV8lO8uUlKWMGDRrE6NGjadOmDc888wz7779/0pEkSVuoJAV7CDAC+AWFM9hjgdsyGUqSKpMYC78rF0Jg//33Z9ddd+Wyyy6jWrVEv1koSfqZwro39o0+GEIbYHfg/Rjjh2WWahOqN2wep+ZNYs8GtZOOIklbbfbs2QwZMoS+ffty+umnJx1HkrSeEMLEGOMWfyux2A85hhD+j8LLpPcHXgohDNiKfJKk9RQUFHDXXXfRqlUrXn/9dVatWpV0JElSKdnUiEh/oG2McVkIYUfgOeDesoklSRXXhx9+yKBBgxg7diydO3fm7rvvplmzZknHkiSVkk0V7FUxxmUAMcZFIYSSbOknSdqM6dOnM3XqVO69917OOussgnuPSlKFsqmCvVsI4cmirwOw+3q3iTGelNFkklSBTJkyhcmTJ3PmmWdywgknMGvWLOrWrZt0LElSBmyqYPfa4PbtmQwiSRXRqlWruO666/jNb35Dw4YN6dOnDzk5OZZrSarAii3YMcZXyjLIlvC7qZLS4K233mLgwIHMmDGDM844g9///vfk5OQkHUuSlGEl2QdbkrSF5s6dy2GHHUaDBg147rnn6Nq1a9KRJEllxA8uSlIpmjFjBgCNGjXi8ccf5/3337dcS1IlU+KCHUKonskgkpRmixcvZsCAAeTm5vLGG28A0LNnT2rX9qJYklTZbLZghxAODCFMAz4sur13CMFLpUtSkaeeeorc3FweeOABRo0axQEHHJB0JElSgkqygv1HoAfwFUCMcQpwRCZDSVJaDBgwgJNOOokGDRrwzjvvcMMNN/hBRkmq5EryIcesGONnG1wIIT9DeSSp3IsxAhBC4KCDDqJ58+aMHDmSqlWrJpxMklQelKRgzw4hHAjEEEI2cD7wQWZjbZq79ElKymeffca5555Lv379OOOMMxg8eHDSkSRJ5UxJRkSGAhcDuwALgYOK7pOkSqOgoIA77riD1q1bM27cONasWZN0JElSObXZFewY4xdA3zLIIknl0syZMxk0aBDjxo3jmGOO4c9//jNNmzZNOpYkqZzabMEOIfwFiBveH2P0+6KSKoWZM2fy/vvvc99993HGGWcQvJysJGkTSjKD/fJ6X+cAJwKzMxNHksqHvLw8Jk+ezNlnn83xxx/PrFmz2G677ZKOJUlKgZKMiDy2/u0QwoPASxlLJEkJWrlyJddeey033XQTjRo14tRTTyUnJ8dyLUkqsZ9zqfRmwK6lHUSSkvbmm2/Srl07brzxRs444wwmT57sntaSpC1Wkhnsxfwwg50FfA1cnslQklTW5s6dyxFHHEGjRo144YUXOOaYY5KOJElKqU0W7FD4SZ69gblFdxXEdVdYSJCfL5JUWqZPn05ubi6NGjXiH//4B0cccQTbbLNN0rEkSSm2yRGRojL9VIwxv+hH4uVakkrD119/zVlnnUWrVq0YO3YsAMcdd5zlWpK01Uoyg/1OCGHfjCeRpDLyj3/8g9zcXB5++GGuuOIKDjzwwKQjSZIqkGJHREIIVWKMa4FDgHNCCB8Dyyi8UnmMMVq6JaXOWWedxf3338++++7L888/T7t27ZKOJEmqYDY1g/0OsC/Q8+eePIRwLPAHIBu4J8b4m2KOOxn4O3BAjPHdn/t6krQx66bbQgh07NiRli1bcskll1ClSkkuBSBJ0pbZ1N8uASDG+PHPOXEIIRu4AzgamANMCCH8M8Y4fYPjagMjgPE/53UkaVM++eQTBg8ezGmnncaZZ57J4MFehFaSlFmbKtg7hhAuLu7BGOPvN3PuA4GPYoyzAEIIjwInANM3OO7XwE3AyM3HlaSSyc/P54477mDUqFFkZWXRv3//pCNJkiqJTX3IMRvYBqhdzI/NacSPL6k+p+i+74UQ9gGaxBj/tQWZJWmTZsyYQadOnbjgggs47LDDeP/99znrrLOSjiVJqiQ2tYI9P8Z47Vace2O7VX+/zV8IIQu4BThrsycKYTAwGKBagz2KObUkFfroo4+YOXMmDz74IP379ye4eb4kqQxtdgZ7K8wBmqx3uzEwb73btYHWwOtFf/k1AP4ZQjh+ww86xhjvBu4GqN6wuXtxS/qJiRMnMmXKFAYMGMBxxx3HJ598Qp06dZKOJUmqhDY1InLUVp57AtA8hNAshFAN6Av8c92DMcZvY4w7xBibxhibAm8DPynXkrQpK1as4PLLL6d9+/b8+te/ZuXKlQCWa0lSYoot2DHGr7fmxEV7aA8HXgBmAI/HGN8PIVwbQjh+a84tSQBjx45l77335re//S1nnXUWeXl55OTkJB1LklTJZXQT2Bjjc8BzG9x3dTHHHp7JLJIqlrlz53LUUUfRpEkTXn75ZY46amu/6SZJUukoyaXSJancmDZtGgCNGjXiqaeeYtq0aZZrSVK5YsGWlApffvklp59+Om3btmXs2LEA9OjRg1q1aiWcTJKkH/M6wZLKtRgjf//73xk+fDiLFy/mmmuuoX379knHkiSpWKks2G5pK1UeZ555Jg8++CD7778/r7zyCm3atEk6kiRJm5TKgi2pYouxcLv7EAKHHXYYbdu25cILL6RKFd+yJEnlnzPYksqVWbNm0blzZ+677z4ABg4cyMiRIy3XkqTUsGBLKhfy8/O59dZbadOmDRMmTCAry7cnSVI6uSQkKXHTp09nwIABjB8/nu7du/OnP/2Jxo0bJx1LkqSfxYItKXGffPIJH3/8MY888gh9+/Yl+ElmSVKKWbAlJWLChAlMnjyZc845h+7duzNr1ixq166ddCxJkraaQ46SytTy5csZOXIkBx10EDfeeCMrV64EsFxLkiqMVBZsv3kspdPrr79O27Zt+d3vfsc555xDXl4eOTk5SceSJKlUOSIiqUzMmTOHo48+ml133ZVXX32VI444IulIkiRlRCpXsCWlx5QpUwBo3LgxzzzzDFOnTrVcS5IqNAu2pIxYtGgR/fr1o127dowZMwaAbt26UbNmzYSTSZKUWY6ISCpVMUYeffRRRowYwbfffsuvfvUrOnTokHQsSZLKjAVbUqk6/fTTefjhh2nfvj2jR4+mVatWSUeSJKlMWbAlbbWCggJCCIQQOOKII9hvv/0YMWIE2dnZSUeTJKnMOYMtaat89NFHHHXUUfz1r38FYODAgVx00UWWa0lSpZXKgu1llKXkrV27lptvvpk2bdqQl5dHtWrVko4kSVK54IiIpC323nvvcfbZZ/Puu+9ywgkncOedd7LzzjsnHUuSpHLBgi1pi33++ed89tlnPProo/Tu3dvvKkmStB4LtqQSGT9+PFOmTGHw4MF069aNWbNmsc022yQdS5KkcieVM9iSys6yZcu4+OKL6dChAzfddBOrVq0CsFxLklQMC7akYr366qu0bduWW265hSFDhjBp0iSqV6+edCxJkso1R0QkbdScOXPo0qULzZo1Y8yYMRx66KFJR5IkKRVcwZb0I3l5eQA0btyYZ599lilTpliuJUnaAqks2O5XIJW+hQsX0qdPH/bdd1/GjBkDwLHHHkuNGjUSTiZJUrqksmBLKj0xRh566CFyc3N5+umnue666+jYsWPSsSRJSi1nsKVKrl+/fjz66KN06NCB0aNH07Jly6QjSZKUahZsqRIqKCgghEAIgWOOOYYOHTowbNgwsrOzk44mSVLqOSIiVTIffPABRxxxBPfeey8AZ599NiNGjLBcS5JUSizYUiWxdu1abrrpJvbee2+mTp3qhxclScoQR0SkSmDq1KkMGDCAiRMncuKJJ3LHHXfQsGHDpGNJklQhWbClSmDOnDnMnj2bv//97/Tq1YsQ3OxSkqRMSWXBthtIm/ff//6XqVOnMmTIELp168asWbOoVatW0rEkSarwnMGWKpilS5dywQUXcMghh/C73/2OVatWAViuJUkqIxZsqQJ58cUXad26NbfddhvDhg1j0qRJVK9ePelYkiRVKqkcEZH0U7Nnz6Z79+7svvvujB07lkMOOSTpSJIkVUquYEspN3HiRACaNGnCc889x+TJky3XkiQlyIItpdSCBQs45ZRT2H///RkzZgwARx99NDk5OQknkySpcrNgSykTY+T+++8nNzeXZ599lhtuuIGOHTsmHUuSJBVxBltKmb59+/L4449z8MEHc88997DXXnslHUmSJK0nlQU74EbYqlwKCgoIIRBCoFu3bnTq1InzzjuPrCy/CSVJUnnj385SOfe///2PQw89lNGjRwNw5plnMnz4cMu1JEnllH9DS+XUmjVruOGGG9h7772ZPn0622yzTdKRJElSCaRyRESq6CZPnszZZ5/N5MmTOfnkk7ntttto0KBB0rEkSVIJWLClcmjBggUsWLCAf/zjH5x00klJx5EkSVvAgi2VE+PGjWPq1Kmcd955HHvssXz88cfUrFkz6ViSJGkLOYMtJWzJkiUMHz6cTp06ceutt7Jq1SoAy7UkSSllwZYS9MILL9C6dWvuvPNOLrjgAiZNmkT16tWTjiVJkrZCKkdEgttgqwKYPXs2PXr0YI899mDcuHFejVGSpArCFWypDMUYeeeddwBo0qQJ//nPf8jLy7NcS5JUgViwpTIyf/58evXqRfv27RkzZgwAnTt3JicnJ+FkkiSpNFmwpQyLMfLXv/6V3Nxc/vOf//Db3/6Wgw8+OOlYkiQpQ1I5gy2lSe/evXniiSfo1KkT99xzDy1atEg6kiRJyiALtpQB+fn5hBDIysriuOOO48gjj+Tcc88lK8tvGkmSVNH5t71UymbMmEGnTp0YPXo0AGeccQZDhw79/+3de3RU5b3/8fc3gaA9tlat/dlWjG21SlC8pXhB5KYoYsUKgqA0UG/nWI+tl9rDqut46q/VY/XYlhYQFKt11d6oRbGoVKUBkXDTCoIXvNTIsrXqD7UqECDP748Zz8mhARLIZGcm79daWWtmz569vxmelfnwzHf2Y7iWJKmT8B1faiMbN27ku9/9LocffjjPPfccu+++e9YlSZKkDNgiIrWBJ598knHjxrF8+XJGjRrFxIkT+eQnP5l1WZIkKQMGbKkNvP7667z55pvMnDmTYcOGZV2OJEnKkAFb2kHz5s1jxYoVfO1rX+OUU07hhRdeYNddd826LEmSlDF7sKVWevfdd7n44ovp168fEydOZMOGDQCGa0mSBBiwpVaZPXs2PXv2ZOrUqVx++eU88cQTdOvWLeuyJElSB2KLiNRCr776KsOGDeOggw5ixowZHH300VmXJEmSOiBnsKVtSClRV1cHQPfu3ZkzZw5PPPGE4VqSJG2VAVvaitdee40zzjiDY489ltraWgAGDBhARUVFxpVJkqSOrCgDdkTWFaiUpZS47bbbqKqqYs6cOdx000306dMn67IkSVKRsAdb2sKIESO455576NevH7fddhsHHHBA1iVJkqQiYsCWgM2bNxMRlJWVccYZZzB48GAuuOACysqK8kMeSZKUIdODOr2nn36aPn36MH36dADGjh3LRRddZLiWJEk7xAShTquhoYHvfOc7HHnkkbz44ovsscceWZckSZJKgC0i6pSWLVvGuHHjePrppxkzZgw//OEP2XvvvbMuS5IklQADtt9R0I4AABVySURBVDqlt956i7fffptZs2Zx2mmnZV2OJEkqIUUZsMPr9GkHzJ07lxUrVnDppZcyePBgVq9ezS677JJ1WZIkqcTYg62S984773DRRRcxcOBApkyZwoYNGwAM15IkqSAM2Cpps2bNoqqqittuu40rr7ySZcuW0a1bt6zLkiRJJawoW0Sklnj11VcZPnw4Bx98MDNnzuSLX/xi1iVJkqROoKAz2BFxSkQ8FxEvRMS/NfP45RGxKiKWR8QjEVFZyHpU+lJKPP744wB0796dOXPmsHTpUsO1JElqNwUL2BFRDkwChgBVwOiIqNpityeB6pRSL2AG8P1C1aPSt2bNGk4//XT69OlDbW0tAP3796eioiLjyiRJUmdSyBns3sALKaWXUkoNwC+BYU13SCnNTSl9kL9bB+xbwHpUohobG5k6dSpVVVU88sgj3HzzzRx//PFZlyVJkjqpQvZgfwZ4tcn9NcDR29j/POCBAtajEjV8+HBmzpzJwIEDufXWW/nc5z6XdUmSJKkTK2TAbu5i1anZHSPOBaqBflt5/ELgQoCKfQ5o9sDqXDZt2kRZWRllZWUMHz6coUOHct5553mNdEmSlLlCtoisAbo3ub8v8NqWO0XEicC3gdNTShuaO1BKaVpKqTqlVF2QSlVUli9fzrHHHsutt94KwLnnnsv5559vuJYkSR1CIQP2EuDAiPhsRFQAZwP3Nd0hIo4AppIL138rYC0qARs2bOCaa67hqKOO4pVXXmHvvffOuiRJkqR/ULAWkZTSpoi4BHgIKAduTymtjIhrgaUppfuAG4HdgN/kZx/rU0qnF6omFa8lS5Ywbtw4Vq1axdixY/nBD37AXnvtlXVZkiRJ/6CgC82klGYDs7fY9u9Nbp9YyPOrdKxdu5b33nuP2bNnM2TIkKzLkSRJ2ipXclSH9eijj7JixQq+/vWvM3jwYJ5//nmXOZckSR1eQVdylHbE22+/zQUXXMCgQYOYOnUqGzbkvvtquJYkScXAgK0O5d5776Wqqorbb7+dq666imXLlhmsJUlSUSnKFhGvxlaa6uvrOeuss+jRowf33Xcf1dVelVGSJBUfZ7CVqZQS8+fPB2C//fbj4YcfZsmSJYZrSZJUtAzYykx9fT1Dhw7lhBNOoLa2FoATTjiBioqKjCuTJEnacQZstbvGxkYmT55Mz549mTdvHhMnTuT444/PuixJkqQ2UZQ92CpuZ555Jvfeey8nnXQS06ZNY//998+6JEmSpDZjwFa72LRpE2VlZZSVlTFq1CiGDRvGuHHjCL+xKkmSSowtIiq4p556iqOPPppp06YBMHr0aMaPH2+4liRJJcmArYJZv349V199NdXV1axZs4Z99tkn65IkSZIKrihbRAJnPju6xYsXU1NTw7PPPktNTQ0333wze+65Z9ZlSZIkFVxRBmx1fO+++y7r1q3jwQcf5OSTT866HEmSpHZjwFabmTNnDitXruSyyy7jxBNP5LnnnnOZc0mS1OnYg62dtnbtWsaPH8/JJ5/M9OnT2bBhA4DhWpIkdUoGbO2Ue+65h6qqKu666y4mTJjA0qVLDdaSJKlTs0VEO6y+vp6zzz6bQw45hNmzZ3PEEUdkXZIkSVLmnMFWq6SUqK2tBWC//fbj0UcfZdGiRYZrSZKkPAO2WuyVV15hyJAh9O/f/79D9vHHH0/Xrl0zrkySJKnjKMqA7QKA7auxsZGf/OQn9OzZk8cee4wf//jH9O3bN+uyJEmSOiR7sLVdZ5xxBrNmzeLkk09m6tSpVFZWZl2SJElSh2XAVrM2btxIeXk5ZWVljB49mhEjRjB27FjCjw8kSZK2qShbRFRYTzzxBL179+aWW24BYPTo0XzlK18xXEuSJLWAAVv/bd26dUyYMIHevXvz17/+le7du2ddkiRJUtGxRUQA1NXVUVNTw/PPP89Xv/pVbrrpJvbYY4+sy5IkSSo6BmwB8P7777Nx40b+8Ic/cOKJJ2ZdjiRJUtEyYHdiDz74ICtXruSKK65g0KBBPPvss1RUVGRdliRJUlEryh5sv2q3c9566y1qamoYMmQId955Jw0NDQCGa0mSpDZQlAFbOyalxIwZM6iqquLuu+/m6quvZsmSJQZrSZKkNmSLSCdSX1/PmDFj6NWrF3PmzOGwww7LuiRJkqSS4wx2iUsp8eijjwJQWVnJH//4R+rq6gzXkiRJBWLALmEvv/wygwcPZtCgQdTW1gJw3HHH0aWLH1xIkiQVigG7BG3evJkf/ehHHHLIISxatIgpU6bQt2/frMuSJEnqFJzKLEHDhg3j97//Paeeeiq33HKLKzJKkiS1IwN2idi4cSPl5eWUlZUxduxYRo8ezZgxY4jwooaSJEntqThbRMyM/8vSpUuprq5mypQpAIwaNYpzzjnHcC1JkpSB4gzYAmDdunV861vf4uijj+aNN96gsrIy65IkSZI6PVtEitTChQupqalh9erVnH/++dx44418/OMfz7osSZKkTs+AXaTWrVtHY2MjDz/8MIMGDcq6HEmSJOUZsIvI7NmzWblyJd/85jcZOHAgzzzzDF27ds26LEmSJDVhD3YRePPNNzn33HMZOnQoP//5z2loaAAwXEuSJHVABuwOLKXEL3/5S3r06MGvf/1rrrnmGhYvXkxFRUXWpUmSJGkrbBHpwOrr66mpqeGwww5j+vTpHHrooVmXJEmSpO0oyhnsKOELYaeUePjhhwGorKyktraWhQsXGq4lSZKKRFEG7FL14osvMmjQIE466SRqa2sBOOaYYygvL8+4MkmSJLWUAbsD2Lx5MzfffDOHHnooy5YtY+rUqfTt2zfrsiRJkrQD7MHuAL70pS/xwAMPcNpppzFlyhT23XffrEuSJEnSDjJgZ6ShoYEuXbpQVlbGuHHjGDt2LGeffTYRpdtfLkmS1BnYIpKBxYsXc9RRRzF58mQARo4cyejRow3XkiRJJcCA3Y4++OADrrjiCo499ljWrl3L5z//+axLkiRJUhuzRaSdPPbYY9TU1PDSSy9x0UUXccMNN7D77rtnXZYkSZLaWFEG7GLspNi4cSPl5eXMnTuX/v37Z12OJEmSCqQoA3axmDVrFs888wxXXXUVAwYMYNWqVXTp4ksuSZJUyuzBLoA33niDMWPGcPrpp/OLX/yChoYGAMO1JElSJ2DAbkMpJe6++2569OjBjBkzuPbaa1m0aBEVFRVZlyZJkqR24pRqG6qvr2f8+PEcccQRTJ8+nZ49e2ZdkiRJktqZM9g7qbGxkYceegiAyspK5s+fz4IFCwzXkiRJnZQBeyesXr2agQMHcsoppzBv3jwAevfuTXl5ecaVSZIkKSsG7B2wadMmbrzxRnr16sWf/vQnpk+fTt++fbMuS5IkSR1AUfZgZ30Z7NNOO42HHnqIYcOGMXnyZD796U9nXJEkSZI6ikgpZV1Dq3T71IHptdVPs9du3dr1vBs2bKBr166UlZUxY8YMGhsbOeuss4hiXPVGkiRJ2xURy1JK1a19ni0iLVBXV8eRRx7JpEmTABgxYgQjR440XEuSJOkfGLC34f333+eyyy7juOOO4+9//zsHHnhg1iVJkiSpgyvKHuz2MH/+fGpqanj55Ze5+OKLuf766/nYxz6WdVmSJEnq4AzYW7Fp0ya6du1KbW0tJ5xwQtblSJIkqUgYsJuYOXMmzzzzDBMmTGDAgAGsXLmSLl18iSRJktRy9mADr7/+OiNHjuTLX/4yM2bMoKGhAcBwLUmSpFYryoDdVlfvSClx1113UVVVxb333sv3vvc96urqqKioaJPjS5IkqfPp1FO09fX1nH/++VRXVzN9+nQOPvjgrEuSJElSkSvKGeyd0djYyAMPPABAZWUlCxYsYN68eYZrSZIktYlOFbCff/55+vfvz6mnnkptbS0A1dXVlJeXZ1yZJEmSSkWnCNibNm3ihhtuoFevXqxYsYKf/vSnXnpPkiRJBdEperCHDh3KnDlzOPPMM5k0aRL77LNP1iVJkiSpREVKKesaWqXbpw5Mf3lhJXv+07av9LF+/Xq6du1KeXk5v/3tbwEYPnx4e5QoSZKkEhARy1JK1a19Xkm2iCxYsIDDDz+cSZMmAblgbbiWJElSeyjKgL21q2C/9957XHrppfTt25f169fTo0ePdq1LkiRJKpke7NraWmpqaqivr+eSSy7huuuuY7fddsu6LEmSJHUyJROwAT7ykY8wf/58+vTpk3UpkiRJ6qQK2iISEadExHMR8UJE/Fszj3eLiF/lH18UEfu35vj33HMP1113HQD9+vVjxYoVhmtJkiRlqmABOyLKgUnAEKAKGB0RVVvsdh6wNqV0APAD4IaWHPv11//KiBEjGD58OL/73e9oaGgAcMEYSZIkZa6QM9i9gRdSSi+llBqAXwLDtthnGHBn/vYMYFBEbO07jABs/uBdjjnqMO6//36uv/56Hn/8cSoqtn3JPkmSJKm9FLIH+zPAq03urwGO3to+KaVNEfEOsBfw5tYOuvndN+hx7HHc8dPpHHTQQW1csiRJkrRzChmwm5uJ3nJVm5bsQ0RcCFyYv7uhbuGCpw8++OCdLE8l5hNs4z9m6rQcF2qO40LNcVyoOTs0m1vIgL0G6N7k/r7Aa1vZZ01EdAF2B/7flgdKKU0DpgFExNIdWVFHpc1xoeY4LtQcx4Wa47hQcyJi6Y48r5A92EuAAyPisxFRAZwN3LfFPvcBNfnbI4BHU7Gt3S5JkiQ1UbAZ7HxP9SXAQ0A5cHtKaWVEXAssTSndB0wH7oqIF8jNXJ9dqHokSZKk9lDQhWZSSrOB2Vts+/cmt9cDZ7XysNPaoDSVHseFmuO4UHMcF2qO40LN2aFxEXZkSJIkSW2noCs5SpIkSZ1Nhw3YhV5mXcWpBePi8ohYFRHLI+KRiKjMok61r+2Niyb7jYiIFBFeKaATaMm4iIiR+b8ZKyPi7vauUe2vBe8j+0XE3Ih4Mv9ecmoWdar9RMTtEfG3iHh6K49HREzMj5nlEXHk9o7ZIQN2IZdZV/Fq4bh4EqhOKfUitzro99u3SrW3Fo4LIuKjwKXAovatUFloybiIiAOBCUCflFJP4BvtXqjaVQv/XlwN/DqldAS5iy9Mbt8qlYE7gFO28fgQ4MD8z4XAlO0dsEMGbAq0zLqK3nbHRUppbkrpg/zdOnLXX1dpa8nfC4D/S+4/XOvbszhlpiXj4gJgUkppLUBK6W/tXKPaX0vGRQI+lr+9O/+4hodKTEppHs2sw9LEMOBnKacO+HhEfGpbx+yoAbu5ZdY/s7V9UkqbgA+XWVfpasm4aOo84IGCVqSOYLvjIiKOALqnlO5vz8KUqZb8vfgC8IWIWBARdRGxrRkslYaWjIv/AM6NiDXkroT2r+1Tmjqw1uaPwl6mbye02TLrKikt/jePiHOBaqBfQStSR7DNcRERZeTayMa1V0HqEFry96ILuY98+5P7tGt+RBySUnq7wLUpOy0ZF6OBO1JK/xURx5Jbr+OQlFJj4ctTB9XqzNlRZ7Bbs8w621pmXSWlJeOCiDgR+DZwekppQzvVpuxsb1x8FDgE+GNE/Bk4BrjPLzqWvJa+j9ybUtqYUnoZeI5c4Fbpasm4OA/4NUBKaSGwC/CJdqlOHVWL8kdTHTVgu8y6mrPdcZFvBZhKLlzbT9k5bHNcpJTeSSl9IqW0f0ppf3K9+aenlJZmU67aSUveR2YCAwAi4hPkWkZeatcq1d5aMi7qgUEAEdGDXMB+o12rVEdzH/CV/NVEjgHeSSn9ZVtP6JAtIi6zrua0cFzcCOwG/Cb/ndf6lNLpmRWtgmvhuFAn08Jx8RAwOCJWAZuBb6aU3squahVaC8fFFcCtEXEZuTaAcU7glbaI+AW5VrFP5HvvrwG6AqSUbiHXi38q8ALwATB+u8d0zEiSJEltp6O2iEiSJElFyYAtSZIktSEDtiRJktSGDNiSJElSGzJgS5IkSW3IgC1JrRARmyPiT01+9t/GvvtHxNNtcM4/RsRzEfFUflnvg3bgGP8cEV/J3x4XEZ9u8thtEVHVxnUuiYjDW/Ccb0TER3b23JLUkRiwJal11qWUDm/y8+d2Ou85KaXDgDvJXe+9VVJKt6SUfpa/Ow74dJPHzk8prWqTKv+nzsm0rM5vAAZsSSXFgC1JOyk/Uz0/Ip7I/xzXzD49I2JxftZ7eUQcmN9+bpPtUyOifDunmwcckH/uoIh4MiJWRMTtEdEtv/0/I2JV/jw35bf9R0RcGREjgGrg5/lz7pqfea6OiH+JiO83qXlcRPx4B+tcCHymybGmRMTSiFgZEd/Jb7uUXNCfGxFz89sGR8TC/Ov4m4jYbTvnkaQOx4AtSa2za5P2kN/lt/0NOCmldCQwCpjYzPP+GfhRSulwcgF3TX4Z5lFAn/z2zcA52zn/l4AVEbELcAcwKqV0KLmVef8lIvYEvgz0TCn1Ar7b9MkppRnAUnIzzYenlNY1eXgGcGaT+6OAX+1gnaeQW4r8Q99OKVUDvYB+EdErpTQReA0YkFIakF+u/GrgxPxruRS4fDvnkaQOp0MulS5JHdi6fMhsqivwk3zP8WbgC808byHw7YjYF7gnpbQ6IgYBRwFLIgJgV3JhvTk/j4h1wJ+BfwUOAl5OKT2ff/xO4GvAT4D1wG0R8Xvg/pb+YimlNyLipYg4BlidP8eC/HFbU+c/kVuG+sgm20dGxIXk3nc+BVQBy7d47jH57Qvy56kg97pJUlExYEvSzrsMeB04jNwng+u33CGldHdELAKGAg9FxPlAAHemlCa04BznpJSWfngnIvZqbqeU0qaI6A0MAs4GLgEGtuJ3+RUwEngW+F1KKUUu7ba4TuAp4D+BScCZEfFZ4ErgiymltRFxB7BLM88N4A8ppdGtqFeSOhxbRCRp5+0O/CWl1AiMJTd7+79ExOeAl/JtEfeRa5V4BBgREZ/M77NnRFS28JzPAvtHxAH5+2OB2nzP8u4ppdnkvkDY3JU8/g58dCvHvQc4AxhNLmzT2jpTShvJtXock28v+RjwPvBORPwfYMhWaqkD+nz4O0XERyKiuU8DJKlDM2BL0s6bDNRERB259pD3m9lnFPB0RPwJOBj4Wf7KHVcDcyJiOfAHcu0T25VSWg+MB34TESuARuAWcmH1/vzxasnNrm/pDuCWD7/kuMVx1wKrgMqU0uL8tlbXme/t/i/gypTSU8CTwErgdnJtJx+aBjwQEXNTSm+Qu8LJL/LnqSP3WklSUYmUUtY1SJIkSSXDGWxJkiSpDRmwJUmSpDZkwJYkSZLakAFbkiRJakMGbEmSJKkNGbAlSZKkNmTAliRJktqQAVuSJElqQ/8fgg88t93rBF8AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 864x576 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.figure(figsize=(12,8)); \n",
    "plot_roc_curve(fpr, tpr)\n",
    "plt.show();"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 108,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.metrics import roc_auc_score"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 109,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.9953557166443254"
      ]
     },
     "execution_count": 109,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "roc_auc_score(y_train_0, y_scores)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Use PR curve whenever the **positive class is rare** or when you care more about the false positives than the false negatives\n",
    "\n",
    "Use ROC curve whenever the **negative class is rare** or when you care more about the false negatives than the false positives"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "In the example above, the ROC curve seemed to suggest that the classifier is good. However, when you look at the PR curve, you can see that there are room for improvement."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Model Comparison\n",
    "\n",
    "# Random Forest"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 110,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.ensemble import RandomForestClassifier"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 121,
   "metadata": {},
   "outputs": [],
   "source": [
    "f_clf = RandomForestClassifier(random_state=0, n_estimators=100)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 122,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_probas_forest = cross_val_predict(f_clf, X_train, y_train_0,\n",
    "                                   cv=3, method='predict_proba')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 123,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_scores_forest = y_probas_forest[:, 1]\n",
    "fpr_forest, tpr_forest, threshold_forest = roc_curve(y_train_0, y_scores_forest)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 124,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAtgAAAHwCAYAAACcxBjDAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJzs3Xd4VEXfxvHvhJbQlBpCRwklFCNVUBSkiRQBpSsgIEWpiiJib/jw2FFRBB4RK6+CFVFAqQICSSAkSAudAIIgLbRk3j9OIoikQTYnJ7k/18XFzu7Zs/eebHZ/mZ0zY6y1iIiIiIhIxvBzO4CIiIiISHaiAltEREREJAOpwBYRERERyUAqsEVEREREMpAKbBERERGRDKQCW0REREQkA6nAFhERERHJQCqwRUQygTFmuzEmzhhz3BizzxjzgTGm4EXbNDbG/GyMOWaM+csY860xJuSibQobY143xuxM3NeWxHbxZB7XGGOGG2PWG2NOGGN2G2P+zxhTy5fPV0QkJ1OBLSKSedpbawsCocD1wNikG4wxjYCfgK+B0kAlYC2wzBhzTeI2eYEFQA3gNqAw0Bg4BDRI5jHfAEYAw4GiQBXgK6BtesMbY3Kn9z4iIjmR0UqOIiK+Z4zZDgyw1s5PbE8Aalhr2ya2lwCR1tr7L7rfD8Af1trexpgBwAvAtdba42l4zGDgd6CRtfa3ZLZZCHxkrZ2S2O6bmPOmxLYFhgIjgdzAj8Bxa+3oC/bxNbDIWvuqMaY0MBG4GTgOvGatfTMNh0hEJNtQD7aISCYzxpQF2gBbEtv5cXqi/+8Sm88EWiZebgHMTUtxnag5sDu54jodOgINgRDgE6CbMcYAGGOKAK2Az4wxfsC3OD3vZRIff6QxpvUVPr6IiKeowBYRyTxfGWOOAbuAA8BTidcXxXk/jr3EfWKBpPHVxZLZJjnp3T454621f1pr44AlgAWaJN52F7DcWrsXqA+UsNY+a609Y62NAd4HumdABhERz1CBLSKSeTpaawsBTYFqnC+cDwMJQNAl7hMEHEy8fCiZbZKT3u2TsyvpgnXGFX4G9Ei8qifwceLlCkBpY8yRpH/AY0BgBmQQEfEMFdgiIpnMWrsI+AB4ObF9AlgOdLnE5l1xTmwEmA+0NsYUSONDLQDKGmPqpbDNCSD/Be1Sl4p8UftT4C5jTAWcoSNfJl6/C9hmrb36gn+FrLW3pzGviEi2oAJbRMQdrwMtjTGhie1HgT6JU+oVMsYUMcY8DzQCnkncZgZOEfulMaaaMcbPGFPMGPOYMeZfRay1djPwDvCpMaapMSavMcbfGNPdGPNo4mYRQGdjTH5jTGWgf2rBrbXhwB/AFOBHa+2RxJt+A44aY8YYYwKMMbmMMTWNMfUv5wCJiHiVCmwRERdYa/8APgSeSGwvBVoDnXHGTe/AmcrvpsRCGWvtaZwTHX8H5gFHcYra4sDKZB5qOPAW8DZwBNgKdMI5GRHgNeAMsB+YzvnhHqn5NDHLJxc8p3igPc40hNtwhrZMAa5K4z5FRLIFTdMnIiIiIpKB1IMtIiIiIpKBfFZgG2OmGWMOGGPWJ3O7Mca8mbjM7zpjTB1fZRERERERySy+7MH+AGcp3+S0AYIT/w0EJvkwi4iIiIhIpvBZgW2tXQz8mcImdwAfWscK4GpjTEbM1yoiIiIi4ho3x2CX4YLFC4DdideJiIiIiHhWbhcf21ziuktOaWKMGYgzjIQCBQrUrVat2hU9sAVOn03g1Ll4zp5L4Gx8AmfjLWfjE4hPsJxLsCRodhXxkaQXvuWfvwSptZPubQxc+PI0iRtZ+8/LWDBJf0JbSLDg5+fs0ya2c/k5D2ITEtu5EjdPpZ2Q4Ozj73a8kzej2vHxTk6/y22fc45Futp+zvG5VPvcOefyJdsWzsWfb1vr5Mnll3j8E29Pal98e7LtXPz9s/5HOwHiE1Jv5851/md7YTshwfn3r3bu8z/bf7TjnZ99Sm1rIddltuPjnWOUYvui18Y/2uec55FSO92vhQva586BX2rtNL42kmsn91pJrf2v184VvjYu+7WTg19LF97u69dGfOLtST/7hHjnsZLeJ1JtJ1ziPf9K22n8zLhU29rzxyohwXmOGdrm/LFPtR3vPC8/P+fGM0cPEX/yCCZ3Xuy5MwettSVIJzcL7N1AuQvaZYG9l9rQWjsZmAxQr149u3r16lR3fvjEGVZuO8ShE2c4cfoch06cYc/hODbvP07MweOcjXcqFAPkTfx3oVx+hkL+ucmfJxf+eXORP28u8ufJTd7cfn8XMZeSL3cu8uXxI19uP+dybr/EduLl3H7k9jPk8jP4+Rn8jCGXSbrsPK6fSbzdcMFlp7AyiQ9ucH5pTGLB9Xeki677e3tz/j5JRdr5/Zh/7Y+L9/+v/Z7fT2r7sAkGv1ywdw8smA+33GK45hr48xB8+SW0auW0D+yHzz4z3HEHXHsNREUZpkyB0aOhXDn4YQ5MnAiffAJFisCkSfDOO4b1650Mr74KU6dBdJTz2C88b/j4Y9iwwcnRpy+ErYHISOf2Hj1g0yYID3PCdursZPztN2d/DW+AUoHwzTfO7dddB8HB8OUXzv6qVoM6deCzT53bCxeGAQOcHAD58sGoUfDSS+ePxZNPwjPPwNmzkDcvPP88jBsHx49DoULw3/86z3f7dqhUCaZPh969Yf16qFULZs6ELl0gLAzq1oWvvoI77oDly6FxY/jhB7jtNpg/H1q2dJ5rzZrw3XcwciTMnQuVKzv7mTjR2b5gQfj0U5g82Wn7+8NHHzmP/eOPzhvO9OnOfb7/3nku06Y5l79MXL9v8mTYuhX+8x+nPWkS7N4NL7zgtN96Cw4ehKefdtqvvw4nTjjPHeCVV5wPnzFjnPaECc6H7oMPOu3x452cw4Y57eefh2LFYMgQp/3MM1CmjHP8wTnO114Lffo47ccfh+rVoVcvpz12LISGQrduTnvMGGjQAO6802mPHg033wwdOjjtUaOc43n77c4b80MPQZs20KoVnDkDjz4K7drBrbdCXBw88IBzn1q1nOc9ZYrzc6peHQ4cgA8+gI4doUoViI2Fjz+Gzp3hmmuc4zZzppOlQgXYuRNmzXJ+7mXKOK+Nb76Brl2hVCmIiYE5c5znUqIEbN4MP/3kvL6LFnVe4wsWQM+ecNVVzmtpyRK47z7nGG/YAFFRzuP7+TmXf//9/LGIjIQtW6BTJ6e9bp2TIenYRETAnj3Qtq3TDguD/fud4wOwZg0cOuQcK4BVq+DoUWje3GmvXAknT0KzZk57xQrnmN58s9P+9VfnmN90k9NetszJ2aiR016yxPldatjQaS9eDAEBUD9xaZtFi5zfrTqJp9H/8ovz/hGauLzPzz9D8eJQu7bTXrAAAgOd3xuAefOc4x4S4rR/+sn5uVSt6rR//NH5uQUHOwXDTz85v2PXXusURAsWOLdVquT83i9c6PzcK1RwnueiRVCtmvMed+oULF3qvE7KlHGOy6+/Qo0aEBTk/M6sWOFkCwyEY8ec41mrlvOzP3rUeS+46SYoUAD++gt27HAePyAAjhyBXbuctr+/096922nnyweHDzs/yypVnGP655/O67NqVee1cuiQ87OtWtUplA4ehD/+cNp+fk774EGnbYxz26FDzvMDp334sLN/cH4Xjh51jldS+/hx53iC81gnTzrHLqkdFwcVKzrtffucY1i+/Pn2uXNQtuz5dny8cyyT2tY6xzKpDc7vETjP1c/PObZJ7dy5nWOb1M6Tx3m9JLUDAuDqq5326dPOcUn6g+HCgvrCDhDJ2j766CPuuecehg4dyosvvkjhwoV3XM5+fDoPtjGmIvCdtbbmJW5rCwwFbsdZavdNa22D1PZ5qQLbWkvYziOs2v4nm/YdY8O+Y2zcd5SEFJ5a+aL5qVqqEBWL5SfoqgBKXeVPYOF8FC2Qj6L581LIPzd+ft77bbDW+aXOlcv5pY6Pd96UAwKcN4YzZ5wP5IoVnTfYP/902jVrOu2lS5034K5dnX38+KPzBv74486byYQJzofApMRTUocMcT783nvPeezatZ1C7733nNuLFnU+uN95x9lv5cpOgdyjh/NBfd118MUXzof56tXOh+K33zrFyk8/OQXS3LnOdt9+C6+9Bp9/7rzhzZrlFJjTpztvXLNmOXmTHvvLL53n89prTnvWLNi7F4YOddqzZztv/klF2ezZzpv7Pfc47blznWPStKnTnj/fKfJuuMFpL1zoFNVJH9zh4c4bb7nEPxvXrnVyli7ttNetc964AwOdY/X7787txYs7x3DLFudy0aLOzy02FkqWdD7oEhKcD5Z8+Zw374QE58M6T57zb95Jb+B6ExcREUm7w4cPs3btWpo2bYq1lrVr1xKa+Fe4MWaNtbZeevfpswLbGPMp0BRnhbH9wFNAHgBr7bvG6Vp9C2emkZPAvdbaVLumLy6wT5w+x+j/W8sP6/f9Y7s8uQz1KxalQrH85M+bm6IF8lKyUD6qBBaicsmCFMjnZud96s6dO/9X8Pz5TnHVvr1TeDVv7hSe997r/AVep45T+N59t9Pj9/HHTo9Z+/ZO78eNNzqFZ6tWTjHbpYtzfaNGTrHbq5dT7JUv7/Q4vv6604tx1VXw8MPw7rtOT4m10LevU/C9/76T7dFHnWJvwgSn/fDDThH4/PNOO6nn8YEHnPtv3+70Jvj7O8/l+HHIn98pFBMSnD8A8uQ5/zWSiIiIiK/Mnj2b+++/nzNnzrBz504KFCjwj9uzXIHtKxcW2AeOnaL/B6uJ3PMXhfLlpuP1ZageVJgqgQWpUfoqAvJmTpVmrdM7W7iw81VRQgJs3Hi+dzIuzvkasGZN56urY8dg6lS46y6nHRbmfPX9/ffO/Xv2dC7/9ZfTS1m/vlOIhoU5X3XeeqtTYPft62wzerTT63rzzU7ROmmSU0xXqeL02H72mdOLXLGiUzhPnw6DBzs9pXv2OPtt1swphI8dc64LDnaKXH2tJSIiItnNvn37GDZsGF988QWhoaFMmzaN66+//l/b5cgC+4GPw/g+MpbyRfMzrW99Kpcs6NPH/usvpwjNlQtGjHCGSrz7rnObn9/58bWnTzs9tC+8AI89dn487bRpTq/zpk3OGLWJE53hCj/84BTIs2dDkyZOcT13rnM7OGOCixRxil4RERERuXx//PEH1apV48SJEzz11FOMHj2aPHnyXHLbyy2ws/Y4iRR8vy6W7yNjyZPLMKN/AyoUK5D6ndLg2DGnlzguzjmx5OOPnaEUM2Y4J5tt2OCcsHH0qHMy0d69zpCHKVMg6Q+fPHmcXuNatZx2UJAzJOPaa512hQrOiUFJ7TZtnBNDkrRte/6EIXBOvhIRERGRy3fs2DEKFSpEiRIlGDt2LO3ateNKZ6ZLjid7sOf8spRbX17E8dPnGNE8mFEtq1z2/taudXqT/f2hdWtnfPGXXzpDLYoVc3qqx493iulJk5zZBpLOMBYRERGRrC0hIYF33nmHJ554gp9//vmSQ0GSc7k92G4uNHPZpi3dzvHT52hatQQjW6R/3MTZs87/ixY5UzUtWeK0O3Z0CuxcuZxZN06edIprcIZnvPqqimsRERERr/j999+5+eabGTZsGA0bNqRo0aKZ8rieLLB/WB8LQP+bKv09z3NanD3rjIVu1co5sa9RI+fkwqSxzUOGOL3XmsFCRERExNtefvllQkNDiY6OZvr06fzwww9UqFAhUx7bcwX2yTPx7Dh0ksDC+bjhmmJpus/Ysc5k/3FxzgmDFSs6wz/y5oU33zw/ab2IiIiIZA/Hjx+nffv2bNiwgd69e6erU/ZKee4kx7gzztqmTYJLkCdX6n8fJCQ4c0q3aOHM9Vy1Kvzvf75OKSIiIiKZ6dSpUzz77LPcdNNN3H777Tz55JP4+bnTl+y5HuxT55wCOySocIrbbdjgDAExxpkq78yZ80vbioiIiEj2sXTpUkJDQxk/fjxLEk+uc6u4Bg8W2PGJ658XL5Qvxe127IAVK2DePGcO6WSmNxQRERERjzp27BhDhw6lSZMmnD59mp9++onxSTNUuMhzBfbpcwkAlL7K/5K3W+v8u+02Z+7pVq0yM52IiIiIZJavv/6ad955h+HDhxMZGUnLli3djgR4sMA+G+8U2MktLPP003Djjc5CLo0aZWIwEREREfG5Q4cO8fPPPwPQq1cvIiIieOONNyhY0LcreqeH5wrs+ASLn4FiBfJe8vZ774X168+voigiIiIi3met5YsvviAkJIQuXbpw4sQJjDHUrl3b7Wj/4rkCG6B4wXz4+f1zqpXp051e6woV4M8/NZe1iIiISHYRGxvLnXfeSZcuXShbtiwLFiygQIFLj2bICjw3TR/AVQH/PGNx+3YYOhRq1oTlyyG3J5+ViIiIiFzswIEDhISEcOrUKSZMmMCoUaPIncWLvaydLhlFLxoeUrEi7NwJ+VKeWEREREREPOLYsWMUKlSIkiVL8vjjj9O+fXuqVKnidqw08eQQkRIXTNG3dSuEhztT8eXP72IoEREREbli8fHxvPnmm5QrV47w8HAAHnroIc8U1+DRArvwBUNEPvoI6tSBlStdDCQiIiIiVyw6OpomTZowYsQIGjduTPHixd2OdFk8OUQkf57zZzDeey8ULgz16rkYSERERESuyH/+8x+efPJJChYsyIwZM+jVqxfGmNTvmAV5ssAOyHu+wC5fHkaNcjGMiIiIiFyxuLg4OnbsyMSJEylZsqTbca6IJ4eI5MvtxD55EqZOhW3bXA4kIiIiIukSFxfHmDFj+P777wF48skn+fzzzz1fXINHC+w8uZzYixbBgAHOSY4iIiIi4g2LFi2idu3aTJgwgeXLlwPg5+fJsvSSPDlEJHdigX3bbXDwIGShlTFFREREJBlHjx5lzJgxvPvuu1SqVIn58+fTvHlzt2NlOE/+qZA03N0YKFZM81+LiIiIeMG3337L5MmTefDBB4mMjMyWxTV4tMDOlbhM+rBh8OmnLocRERERkWQdPHiQ+fPnA9CzZ0/WrVvHK6+8kqWXOr9Sni2wT52CBQtg1y6304iIiIjIxay1fP7554SEhNCtWzdOnDiBMYYaNWq4Hc3nvDkG28/g7w/R0ZCQ4HYaEREREbnQ3r17GTJkCN988w316tVj2rRp2brH+mKeLLCThogAZKMTTkVEREQ878CBA9SoUYNTp07x8ssvM2LECHLn9mTJedk8WZ7myeXHggXQsiXs3Ol2GhERERE5evQoACVLluSpp54iMjKShx56KMcV1+DRAjtvbj9OnYLjxyEHfdsgIiIikuXEx8fz2muvUa5cOcLCwgAYOXIklStXdjmZezz5J0VAnlw0awtt27qdRERERCTnWr9+PQMGDGDlypW0bds2W6zCmBE82YOdJ5cf5865nUJEREQk5xo/fjx16tRh69atfPLJJ3z77beULVvW7VhZgicLbGOgaVMYPNjtJCIiIiI509mzZ+nSpQvR0dH06NEDY0zqd8ohPDlExBho2BDy5HE7iYiIiEjOcPLkSZ566iluueUW2rVrxxNPPKGiOhneLLAxvPKK2ylEREREcoaFCxcyYMAAtm7dSkBAAO3atVNxnQJPDhEREREREd/766+/GDRoEM2aNQPgl19+4dlnn3U5VdbnyQI7JgaKF4cff3Q7iYiIiEj29e233zJlyhRGjx7NunXraNq0qduRPMGTQ0Ty54du3UAnqoqIiIhkrD/++IOIiAhatmxJr169qFOnDiEhIW7H8hRPFthlShveftvtFCIiIiLZh7WWTz/9lOHDh2OtZefOnRQoUEDF9WXw5BARDakXERERyTi7d++mQ4cO9OrVi8qVK7N48WIKaLnsy+bJAvuzTw0BAXDihNtJRERERLxt//791KhRg59//pnXXnuNZcuWUaNGDbdjeZonh4iEhMDw4RAQ4HYSEREREW86cuQIV199NYGBgTz33HO0a9eOa665xu1Y2YIne7Dr1jX85z/g58n0IiIiIu45d+4cL7/8MuXKlWPNmjUADB8+XMV1BvJkD/aZ03D2rFZyFBEREUmPdevW0b9/f1avXk2HDh0ICgpyO1K25Mk+4Mcec5ZKFxEREZG0eeGFF6hbty47duzg888/56uvvqJ06dJux8qWPNmD3bYtFLFupxARERHxDmst3bt35/XXX6dYsWJux8nWjLXeqlTzBQXb1atWU6vsVW5HEREREcmyTpw4wRNPPEGzZs1o37491lqM0WTH6WGMWWOtrZfe+3lyiMgfB+HoUbdTiIiIiGRNCxYsoFatWrz22mt/n8io4jrzeLLAbnObMw5bRERERM47cuQIAwYMoEWLFuTOnZtFixbx9NNPux0rx/FkgT32MWja1O0UIiIiIlnLnDlz+OCDDxgzZgxr167l5ptvdjtSjuTJMdirfnPGYOubDhEREcnp9u/fT0REBK1bt8Zay6ZNm6hatarbsbKFHDUGe89uw7FjbqcQERERcY+1lhkzZhASEkKvXr04ceIExhgV11mAJwvsO+6AmTPdTiEiIiLijp07d9K2bVt69+5N1apVWbJkCQUKFHA7liTy5DzYL42Hpre4nUJEREQk8+3fv5+aNWsSHx/PG2+8wQMPPECuXLncjiUX8GSB3a6doXKg2ylEREREMs/hw4cpUqQIgYGBvPjii7Rr146KFSu6HUsuwZNDRDZv1jzYIiIikjOcO3eOCRMmUK5cOVavXg3A0KFDVVxnYZ4ssDt2hHnz3E4hIiIi4lsRERE0bNiQMWPG0KpVK8qUKeN2JEkDTxbY418w1K7tdgoRERER33n22WepV68ee/bs4YsvvmDWrFkEBQW5HUvSwJMFdufOEBzsdgoRERER3/Hz8+Puu+8mOjqaO++80+04kg6ePMlx1Soo1xLy5HE7iYiIiEjGOH78OOPGjePWW2/ljjvuYNy4cRitqudJnuzBvvtuOHHC7RQiIiIiGeOnn36iZs2aTJw4kYiICAAV1x7myQJ70iQoVMjtFCIiIiJX5vDhw9x77720bt0af39/Fi9ezFNPPeV2LLlCniywb70VNJ+6iIiIeN2cOXOYMWMGjz32GBEREdx0001uR5IM4MkCe+VKtxOIiIiIXJ59+/Yxd+5cAHr27El0dDQvvPAC/v7+LieTjOLJAnv4cLcTiIiIiKSPtZbp06cTEhLC3XffzcmTJzHGUKVKFbejSQbzZIH9/vtuJxARERFJux07dtCmTRv69u1LSEgIS5cuJX/+/G7HEh/x5DR9WmRGREREvGL//v3UqlWLhIQEJk6cyP3334+fnyf7OCWNPPnTXb7c7QQiIiIiKTt06BAAgYGBvPTSS0RFRTF06FAV1zmAJ3/C48a5nUBERETk0s6ePcuLL75I+fLlWbVqFQD3338/FSpUcDmZZBZPDhGZNMntBCIiIiL/Fh4eTr9+/YiIiKBLly6UL1/e7UjiAk/2YFet6nYCERERkX965plnqF+/Pvv27WPWrFnMnDmTwMBAt2OJCzxZYC9bpqVDRUREJGvJkycPffr0ITo6mk6dOrkdR1xkrLVuZ0iXfEHBtkqlCCJ/LeB2FBEREcnBjh07xtixY2nZsiV33HEH1lqMUSdgdmKMWWOtrZfe+3myB/udd9xOICIiIjnZ3LlzqVmzJu+88w7r1q0DUHEtf/NkgV2hnF7AIiIikvn+/PNP+vTpQ5s2bcifPz9Lly7liSeecDuWZDGeLLAXLXI7gYiIiOREc+fO5ZNPPuHxxx8nIiKCxo0bux1JsiBPjsGuVXUtqxdqeVERERHxvdjYWMLCwmjbti3WWrZu3UrlypXdjiWZIEeNwZ6kMdgiIiLiY9Zapk2bRvXq1enTpw8nT57EGKPiWlLlyQK7VCm3E4iIiEh2tm3bNlq1akX//v257rrr+PXXX8mfX9+eS9r4tMA2xtxmjNlojNlijHn0EreXN8b8YowJN8asM8bcnpb9/vJLxmcVERERAdi/fz+1atVi5cqVTJo0iV9++YUqVaq4HUs8xGcFtjEmF/A20AYIAXoYY0Iu2uxxYKa19nqgO5CmwR+TJ2dkUhERERE4ePAgAIGBgbz88stERUUxePBg/Pw8+YW/uMiXr5gGwBZrbYy19gzwGXDHRdtYoHDi5auAvWnZ8cSJGZZRREREcrizZ8/y/PPPU758eX777TcABg8eTLly5VxOJl6V24f7LgPsuqC9G2h40TZPAz8ZY4YBBYAWadlx8eIZEU9ERERyujVr1tCvXz/WrVtHt27dqFixotuRJBvwZQ/2pVaDuXhOwB7AB9bassDtwAxjzL8yGWMGGmNWG2NWAyxYkOFZRUREJId58sknadCgAQcPHuSrr77is88+o2TJkm7HkmzAlwX2buDC71bK8u8hIP2BmQDW2uWAP/Cv/mlr7WRrbb2keQg//dQneUVERCQHCQgIoH///kRFRXHHHRePYhW5fD5baMYYkxvYBDQH9gCrgJ7W2qgLtvkB+Nxa+4ExpjqwAChjUwiVLyjYRqxcS/XymipHRERE0u7o0aOMGTOGVq1a0alTJ6y1GHOpL9xFHFluoRlr7TlgKPAjsAFntpAoY8yzxpgOiZs9BNxnjFkLfAr0Tam4TlKwoK9Si4iISHb0/fffU6NGDSZPnsyGDRsAVFyLz3hyqfTJb66lTxf1YIuIiEjKDh48yMiRI/n444+pUaMGU6dOpWHDi+dcELm0LNeD7UuzZrudQERERLxg3rx5zJw5k6eeeoqwsDAV15IpPNmDHRW2jspBAW5HERERkSxo7969hIWF0a5dO6y1bNu2jWuuucbtWOJBOaoHO79qaxEREbmItZYpU6YQEhLCvffey8mTJzHGqLiWTOfJAnvePLcTiIiISFYSExNDixYtuO+++7j++utZvnw5+fPrfC1xhycL7Dlz3E4gIiIiWcX+/fupXbs2q1ev5r333mPBggVUrlzZ7ViSg/lyqXSf+e9/3U4gIiIibvvjjz8oUaIEgYGBvPrqq9x+++2ULVvW7Vgi3uzBzu3JPwtEREQkI5w5c4ZnnnmG8uXLs3LlSgAGDhyo4lqyDE8W2BqDLSIikjP99ttv1K1bl6effpo777yTa69ZFxfHAAAgAElEQVS91u1IIv/iyQJ78WK3E4iIiEhme/zxx2nUqBGHDx/mu+++46OPPqJ48eJuxxL5F08W2M8953YCERERyWyFChXivvvuIyoqirZt27odRyRZGs0sIiIiWdJff/3FI488QuvWrencuTOPPPIIxhi3Y4mkypM92PPnu51AREREfOnbb78lJCSEKVOmsHHjRgAV1+IZniywV61yO4GIiIj4wh9//EHPnj3p0KEDxYoVY8WKFYwdO9btWCLp4skCe9xj+gtWREQkO1qwYAFffPEFzz77LKtXr6Z+/fpuRxJJN43BFhEREVft2rWL8PBwOnToQLdu3WjYsCGVKlVyO5bIZfNkD7bGYIuIiHhfQkIC7777LjVq1GDAgAGcPHkSY4yKa/E8TxbY6yPdTiAiIiJXYvPmzdx6660MGTKEBg0asGLFCvLnz+92LJEM4ckhIqMedDuBiIiIXK59+/YRGhpKnjx5mDp1Kvfee69mCJFsxZMFtoiIiHjP/v37CQwMpFSpUrzxxhvcfvvtlC5d2u1YIhnOk0NEFixwO4GIiIik1enTp3nyySepUKECK1asAGDAgAEqriXb8mQP9q5dbicQERGRtFixYgX9+/cnOjqae+65h+DgYLcjificJ3uw+/Z1O4GIiIikZuzYsTRu3Jhjx44xZ84cPvzwQ4oVK+Z2LBGf82SBLSIiIllfkSJFGDJkCOvXr6dNmzZuxxHJNJ4ssH/+2e0EIiIicrEjR44wYMAAvvzySwAeeeQR3n77bQoXLuxyMpHM5ckx2AcOuJ1ARERELvTVV19x//33c+DAAapUqeJ2HBFXebIHu2cPzZUpIiKSFezfv5+uXbvSqVMnSpYsycqVK3nkkUfcjiXiKk8W2JqLXkREJGtYuHAhX3/9NS+88AKrVq2ibt26bkcScZ0nC+xFC91OICIiknPt3LmTr7/+GoCuXbuyadMmHnvsMfLkyeNyMpGswZMF9pEj6sIWERHJbAkJCbzzzjvUqFGDgQMHEhcXhzGGChUquB1NJEvxZIHduZPbCURERHKWTZs20bRpUx544AEaNWrEypUrCQgIcDuWSJbkyVlEREREJPPs27eP0NBQ8uXLx//+9z/69OmD0QlRIsnyZA/2okVuJxAREcn+YmNjAShVqhRvvfUW0dHR9O3bV8W1SCo8WWAn/r6LiIiID5w6dYpx48ZRsWJFli9fDkC/fv0ICgpyOZmIN3hyiMhdXdxOICIikj0tW7aM/v37s3HjRvr27UvVqlXdjiTiOZ7swc6dy+0EIiIi2c+YMWNo0qQJp06d4scff+R///sfRYsWdTuWiOd4ssBessTtBCIiItlPiRIlGDp0KOvXr6dVq1ZuxxHxLE8OETkX73YCERER7zt8+DAPPfQQbdq0oUuXLowePdrtSCLZgid7sJs1dTuBiIiIt82aNYuQkBA+/PBDtm/f7nYckWzFkwW2iIiIXJ59+/Zx1113ceeddxIUFMTq1at5+OGH3Y4lkq14ssBetsztBCIiIt60aNEivvvuO8aPH8/KlSsJDQ11O5JItuPJMdgiIiKSdtu3bycsLIzOnTvTtWtXGjduTLly5dyOJZJtebIH+8Yb3U4gIiKS9SUkJDBx4kRq1qzJkCFDiIuLwxij4lrExzxZYIuIiEjKNmzYQJMmTRg+fDhNmjRh1apVBAQEuB1LJEfwZIH963K3E4iIiGRdsbGx1KlTh99//50PP/yQOXPmUL58ebdjieQYnhyDnduTqUVERHxr7969lC5dmqCgICZNmkSbNm0IDAx0O5ZIjuPJHuwG9d1OICIiknXExcUxduxYKlasyPLlzte8ffv2VXEt4hL1BYuIiHjYkiVLGDBgAJs2baJfv35Uq1bN7UgiOZ4ne7BXrHQ7gYiIiPtGjx7NzTffzJkzZ5g3bx5Tp06lSJEibscSyfE8WWD753M7gYiIiPuCgoIYMWIEkZGRtGjRwu04IpLIWGvdzpAu+YKC7b4tURQpkNftKCIiIpnq0KFDjBo1inbt2tG1a1e344hke8aYNdbaeum9nyd7sEVERHISay0zZ86kevXqfPrpp+zevdvtSCKSAk8W2L+tcjuBiIhI5ti7dy+dO3emW7dulC9fnjVr1vDggw+6HUtEUuDJAtsYtxOIiIhkjmXLljF37lwmTJjAihUrqF27ttuRRCQVnhyDvXdzFMUKagy2iIhkTzExMYSHh3PnnXdirWXv3r2UKVPG7VgiOU6OGoPtpx5sERHJhuLj43n99depVasWDzzwAHFxcRhjVFyLeIwnC+w1a9xOICIikrGio6O56aabGDVqFE2bNmXVqlUEBAS4HUtELoMnV3IsXNjtBCIiIhknNjaWunXrUqBAAT766CN69uyJ0QlHIp7lyQI7ONjtBCIiIldu9+7dlC1blqCgIN577z1uu+02SpYs6XYsEblCnhwiIiIi4mUnT57k4YcfplKlSvz6668A9O7dW8W1SDbhyQI7PNztBCIiIpdn0aJFXHfddbz88svce++91KhRw+1IIpLBPFlgX3WV2wlERETS78EHH6Rp06YkJCSwYMECJk+ezFX6UBPJdjxZYF9zjdsJRERE0q9cuXI89NBDREZGcuutt7odR0R8xJMLzezbEkWRAlpoRkREsraDBw8ycuRI2rdvT7du3dyOIyLplKMWmomIcDuBiIhI8qy1fPbZZ1SvXp2ZM2eyd+9etyOJSCbyZIFdpKjbCURERC5tz549dOzYkR49elCpUiXWrFnDqFGj3I4lIpnIkwV2hfJuJxAREbm0X3/9lXnz5vHKK6+wfPlyatWq5XYkEclkGoMtIiJyhbZs2UJ4eDhdunTBWktsbCylS5d2O5aIXKEcNQZ73Tq3E4iIiEB8fDyvvPIKtWvXZtiwYcTFxWGMUXEtksN5ssAuWsztBCIiktOtX7+eRo0aMXr0aFq0aMGaNWsICAhwO5aIZAG53Q5wOcqWcTuBiIjkZLGxsdSrV4/ChQvz2Wef0bVrV4wxbscSkSzCkwW2iIiIG3bt2kW5cuUICgpi6tSptG7dmuLFi7sdS0SyGE8OEVkf5XYCERHJSU6ePMlDDz3ENddcw7JlywDo1auXimsRuSRP9mAX1xhsERHJJL/88gsDBgwgJiaGwYMHa9o9EUmVJ3uwS5VyO4GIiOQEI0eO5NZbb8XPz4+FCxcyadIkChcu7HYsEcniPFlgi4iIZIaKFSvy8MMPs3btWm655Ra344iIR3hyoZmfv43ixnpaaEZERDLWgQMHGD58OB07dqR79+5uxxERl+WohWY0BltERDKStZaPP/6YkJAQZs2axf79+92OJCIe5skCu2RJtxOIiEh2sWvXLtq3b8/dd99NcHAwERERjBgxwu1YIuJhPi2wjTG3GWM2GmO2GGMeTWabrsaYaGNMlDHmk7Ts12OjWkREJAtbuXIlv/zyC6+//jpLly4lJCTE7Ugi4nE+G4NtjMkFbAJaAruBVUAPa230BdsEAzOBW621h40xJa21B1Lab76gYLt4ThQNr9cYbBERuTybN28mLCyMbt26Ya1l//79lNIUVSJykaw4BrsBsMVaG2OtPQN8Btxx0Tb3AW9baw8DpFZcJymmef1FROQynDt3jv/+97/Url2bkSNHEhcXhzFGxbWIZKg0FdjGmLzGmMrp3HcZYNcF7d2J112oClDFGLPMGLPCGHNbWnZcrGg6k4iISI63bt06GjVqxCOPPELr1q1Zs2YNAQEBbscSkWwo1QLbGNMWiATmJbZDjTGz07Bvc4nrLh6PkhsIBpoCPYApxpirL5FhoDFmtTFmNUCCxmCLiEg6xMbG0qBBA3bu3MnMmTOZPXs2pUuXdjuWiGRTaenBfhZoCBwBsNZGAGnpzd4NlLugXRbYe4ltvrbWnrXWbgM24hTc/2CtnWytrZc0BmbnzjQ8uoiI5Hg7duwAICgoiGnTphEdHU2XLl0w5lJ9QCIiGSMtBfZZa+2Ri65LSx/yKiDYGFPJGJMX6A58c9E2XwHNAIwxxXGGjMSktuMiRdLw6CIikmOdOHGCUaNGce2117Js2TIAevbsSbFiWkhBRHwvdxq22WCM6Qr4GWMqASOAFandyVp7zhgzFPgRyAVMs9ZGGWOeBVZba79JvK2VMSYaiAcettYeSm3fVxVOQ2oREcmR5s+fz3333cf27du5//77qVWrltuRRCSHSXWaPmNMAeBJoFXiVT8Cz1hr43yc7ZLyBQXbPZuiKF5I0/SJiMg/DRs2jLfeeosqVaowZcoUmjRp4nYkEfEwX07T19paO8Zae33iv0eBNumPmHFiY918dBERyaqCg4N59NFHWbt2rYprEXFNWnqww6y1dS66bo21tq5PkyUjX1Cw3RIRRblA9WCLiOR0+/fvZ9iwYXTq1IkePXq4HUdEspnL7cFOdgy2MaY1cBtQxhjz6gU3FQYS0h8x4xQs6Oaji4iI26y1zJgxg5EjR3LixAn1VotIlpLSSY4HgPXAKSDqguuPAY/6MlRq4l0t70VExE07d+5k0KBBzJ07l8aNGzN16lSqVavmdiwRkb8lW2Bba8OBcGPMx9baU5mYKVV/HIDihdxOISIibli1ahVLlixh4sSJ3H///fj5pWlRYhGRTJOWMdjXAi8AIYB/0vXW2iq+jXZp+YKCbcy6KMqU0BhsEZGcYuPGjYSHh9O9e3fAGXsdGBjocioRye58OYvIB8D/cJY+bwPMBD5L7wNlpPz53Xx0ERHJLGfPnuWll17iuuuu48EHHyQuzpkhVsW1iGRlaSmw81trfwSw1m611j5O4uqLbtEYbBGR7C88PJyGDRsyduxY2rVrR1hYGAEBAW7HEhFJVVpWcjxtjDHAVmPMYGAPUNK3sVL2558agy0ikp3t3buXG264gSJFivDFF19w5513uh1JRCTN0lJgjwIKAsNxxmJfBfTzZajUFFJxLSKSLW3bto1KlSpRunRppk+fTuvWrSlSpIjbsURE0iXVISLW2pXW2mPW2p3W2nustR2AHZmQLVn++dx8dBERyWjHjx9n+PDhVK5cmaVLlwLQvXt3Fdci4kkpFtjGmPrGmI7GmOKJ7RrGmA+BFZmSLhkagy0ikn389NNP1KxZk7feeouhQ4cSGhrqdiQRkSuSbIFtjBkPfAz0AuYaY8YBvwBrAVem6Ety7Jibjy4iIhll6NChtG7dmoCAAJYsWcIbb7xBQS3XKyIel9IY7DuA66y1ccaYosDexPbGzImWPE3TJyLibdZajDFUq1aNxx57jCeeeAJ/f//U7ygi4gEpFdinrLVxANbaP40xv2eF4hogbx63E4iIyOWIjY1l6NCh3HXXXfTo0YOhQ4e6HUlEJMOlVGBfY4yZlXjZABUvaGOt7ezTZCnQGGwREW+x1jJ9+nRGjRpFXFwczZs3dzuSiIjPpFRgXzzp6Fu+DJIecScBTdUnIuIJ27dvZ+DAgcybN48mTZrw/vvvU7VqVbdjiYj4TLIFtrV2QWYGSY98GqYnIuIZ4eHhLF++nLfffpvBgwfj55eWRYRFRLwrLQvNZDl5PJlaRCTn2LBhA+Hh4fTs2ZNOnToRExNDiRIl3I4lIpIpPNmNkGDdTiAiIpdy9uxZXnjhBUJDQ3n44YeJi4sDUHEtIjlKmgtsY0yWWT/x3Dm3E4iIyMXCwsKoX78+jz/+OB07diQsLIyAgAC3Y4mIZLpUC2xjTANjTCSwObF9nTFmos+TpSC3hoiIiGQpe/fupVGjRuzfv5/Zs2fz+eefExgY6HYsERFXpKUH+02gHXAIwFq7Fmjmy1Cp8TNuPrqIiCSJiYkBoHTp0syYMYPo6Gg6duzocioREXelpcD2s9buuOi6eF+ESSurMdgiIq46evQoDzzwAMHBwSxduhSArl27UqRIEZeTiYi4Ly2DLXYZYxoA1hiTCxgGbPJtrJTpJEcREff88MMPDBo0iN27dzNixAiuv/56tyOJiGQpaSmwh+AMEykP7AfmJ17nGg0RERFxx5AhQ3j33XepXr06y5Yto1GjRm5HEhHJctJSYJ+z1nb3eZJ0MCqwRUQyjU0cl2eMoWbNmjzxxBOMGzeOfPmyzORSIiJZirGpDGg2xmwFNgKfA7OstccyI1hy8gUF29jNURQtmNfNGCIiOcLevXt54IEHuOuuu+jVq5fbcUREMpUxZo21tl5675fqSY7W2muB54G6QKQx5itjjKs92urBFhHxLWstU6dOJSQkhLlz53LsmKt9KyIinpKmhWastb9aa4cDdYCjwMc+TSUiIq6JiYmhRYsWDBgwgNDQUCIjIxk8eLDbsUREPCMtC80UNMb0MsZ8C/wG/AE09nkyERFxxdq1a1m1ahXvvvsuP//8M5UrV3Y7koiIp6RlDPZ24FtgprV2SWaESkm+oGC7b0sURQpoDLaISEaJjo4mPDz873HWBw8epHjx4i6nEhFxl8/GYAPXWGuHZYXiWkREMtaZM2d47rnnCA0NZcyYMZw6dQpAxbWIyBVIdpo+Y8wr1tqHgC+NMf/q5rbWdvZpMhER8alVq1bRv39/IiMj6d69O2+++Sb+/v5uxxIR8byU5sH+PPH/tzIjiIiIZJ49e/Zw4403UqJECb7++ms6dOjgdiQRkWwj2QLbWvtb4sXq1tp/FNnGmKHAAl8GExGRjLd582aCg4MpU6YMn3zyCS1atODqq692O5aISLaSljHY/S5xXf+MDiIiIr7z119/MXjwYKpWrcqSJc4pNXfddZeKaxERH0hpDHY3oDtQyRgz64KbCgFHfB1MREQyxvfff8+gQYOIjY3lwQcfpG7dum5HEhHJ1lIag/0bcAgoC7x9wfXHgHBfhhIRkYwxaNAgJk+eTM2aNZk1axYNGjRwO5KISLaX0hjsbcA2YH7mxRERkSuVtL6BMYbQ0FCefvppxo4dS968Wj9ARCQzpDREZJG19hZjzGHgwmn6DGCttUV9nk5ERNJlz549DBkyhG7dutGrVy+GDBnidiQRkRwnpZMcmyX+XxwoccG/pLaIiGQR1lref/99QkJCmD9/PsePH3c7kohIjpVsgW2tTUi8WA7IZa2NBxoBg4ACmZBNRETSYOvWrTRv3pyBAwdSt25dIiMjGTRokNuxRERyrLRM0/cVYI0x1wIfAtWBT3yaSkRE0iwyMpI1a9YwefJkFixYwLXXXut2JBGRHC2lWUSSJFhrzxpjOgOvW2vfNMZoFhERERetX7+esLAwevfuTceOHYmJiaFYsWJuxxIREdLWg33OGNMFuAf4LvG6PL6LJCIiyTlz5gxPP/00derUYdy4cZw6dQpAxbWISBaS1pUcmwETrLUxxphKwKe+jSUiIhdbuXIlderU4ZlnnqFbt26Eh4fj7+/vdiwREblIqkNErLXrjTHDgcrGmGrAFmvtC76PJiIiSfbs2UOTJk0IDAzku+++o23btm5HEhGRZKRaYBtjmgAzgD04c2CXMsbcY61d5utwIiI53aZNm6hSpQplypTh888/p3nz5hQuXNjtWCIikoK0DBF5DbjdWnujtbYx0BZ4w7exRERytiNHjjBw4ECqVavG4sWLAejUqZOKaxERD0jLLCJ5rbXRSQ1r7QZjjNbbFRHxkW+++YYhQ4awb98+Hn74YerXr+92JBERSYe0FNhhxpj3cIaJAPQCNE2fiIgPDBgwgKlTp1KrVi2+/vpr6tWr53YkERFJp7QU2IOB4cAjOGOwFwMTfRlKRCQnsdYCYIyhXr16VKhQgTFjxpA3r74sFBHxIpP0xn7JG42pBVwLRFlrN2daqhTkCwq2+7ZEUaSAPnhExPt27drF4MGD6d69O/fcc4/bcURE5ALGmDXW2nR/lZjsSY7GmMdwlknvBcwzxvS7gnwiInKBhIQEJk2aRI0aNVi4cCGnT592O5KIiGSQlIaI9AJqW2tPGGNKAHOAaZkTS0Qk+9q8eTMDBgxg8eLFtGjRgsmTJ1OpUiW3Y4mISAZJqcA+ba09AWCt/cMYk5Yp/UREJBXR0dGsW7eOadOm0bdvX4wxbkcSEZEMlFKBfY0xZlbiZQNce0Eba21nnyYTEclG1q5dS0REBH369OGOO+4gJiaGIkWKuB1LRER8IKUC+86L2m/5MoiISHZ0+vRpnn/+eV566SWCgoLo1q0b/v7+Kq5FRLKxZAtsa+2CzAwiIpLdLF++nP79+7NhwwZ69+7Nq6++ir+/v9uxRETEx9IyD7aIiKTTnj17uOWWWyhVqhRz5syhTZs2bkcSEZFMohMXRUQy0IYNGwAoU6YMM2fOJCoqSsW1iEgOk+YC2xiTz5dBRES87PDhw/Tr14+QkBCWLFkCQMeOHSlUqJDLyUREJLOlWmAbYxoYYyKBzYnt64wxWipdRCTR7NmzCQkJ4cMPP2Ts2LHUr1/f7UgiIuKitPRgvwm0Aw4BWGvXAs18GUpExCv69etH586dKVWqFL/99hsvvviiTmQUEcnh0nKSo5+1dsdFCyHE+yiPiEiWZ60FwBjDDTfcQHBwMKNHjyZPnjwuJxMRkawgLQX2LmNMA8AaY3IBw4BNvo0lIpI17dixg0GDBtGzZ0969+7NwIED3Y4kIiJZTFqGiAwBHgTKA/uBGxKvExHJMRISEnj77bepWbMmS5cu5ezZs25HEhGRLCrVHmxr7QGgeyZkERHJkjZu3MiAAQNYunQprVq14r333qNixYpuxxIRkSwq1QLbGPM+YC++3lqr70VFJEfYuHEjUVFRfPDBB/Tu3ZuLzkkRERH5h7SMwZ5/wWV/oBOwyzdxRESyhvDwcCIiIrj33nvp0KEDMTExXH311W7HEhERD0jLEJHPL2wbY2YA83yWSETERadOneLZZ59lwoQJlClThh49euDv76/iWkRE0uxylkqvBFTI6CAiIm5btmwZoaGhjB8/nt69exMREaE5rUVEJN3SMgb7MOfHYPsBfwKP+jKUiEhm27NnD82aNaNMmTL8+OOPtGrVyu1IIiLiUSkW2MY5k+c6YE/iVQk2aYUFEZFsIDo6mpCQEMqUKcOXX35Js2bNKFiwoNuxRETEw1IcIpJYTM+21sYn/lNxLSLZwp9//knfvn2pUaMGixcvBqB9+/YqrkVE5IqlZQz2b8aYOj5PIiKSSb788ktCQkL4+OOPGTduHA0aNHA7koiIZCPJDhExxuS21p4DbgLuM8ZsBU4ABqdzW0W3iHhO3759mT59OnXq1GHu3LmEhoa6HUlERLKZlMZg/wbUATpe7s6NMbcBbwC5gCnW2peS2e4u4P+A+tba1Zf7eCIil5I0us0YQ+PGjalevToPPfQQuXOnZSkAERGR9Enp08UAWGu3Xs6OjTG5gLeBlsBuYJUx5htrbfRF2xUChgMrL+dxRERSsm3bNgYOHMjdd99Nnz59GDhQi9CKiIhvpVRglzDGPJjcjdbaV1PZdwNgi7U2BsAY8xlwBxB90XbPAROA0anHFRFJm/j4eN5++23Gjh2Ln58fvXr1cjuSiIjkECmd5JgLKAgUSuZfasrwzyXVdyde9zdjzPVAOWvtd+nILCKSog0bNtCkSRNGjBjBLbfcQlRUFH379nU7loiI5BAp9WDHWmufvYJ9m0tc9/c0f8YYP+A1oG+qOzJmIDAQIG+pylcQSURygi1btrBx40ZmzJhBr169cKb0FxERyRypjsG+AruBche0ywJ7L2gXAmoCCxM//EoB3xhjOlx8oqO1djIwGSBfULDm4haRf1mzZg1r166lX79+tG/fnm3btlG4cGG3Y4mISA6U0hCR5le471VAsDGmkjEmL9Ad+CbpRmvtX9ba4tbaitbaisAK4F/FtYhISuLi4nj00Udp2LAhzz33HKdOnQJQcS0iIq5JtsC21v55JTtOnEN7KPAjsAGYaa2NMsY8a4zpcCX7FhEBWLx4Mddddx3/+c9/6Nu3L+Hh4fj7+7sdS0REcjjjtdXP8wUF231boihSIK/bUUTERXv27KFixYqUK1eO999/n+bNr/RLNxERkX8yxqyx1tZL7/3SslS6iEiWERkZCUCZMmWYPXs2kZGRKq5FRCRLUYEtIp5w8OBB7rnnHmrXrs3ixYsBaNeuHQUKFHA5mYiIyD9pnWARydKstfzf//0fQ4cO5fDhwzz11FM0bNjQ7VgiIiLJUoEtIllanz59mDFjBvXq1WPBggXUqlXL7UgiIiIpUoEtIllO0snXxhhuueUWateuzciRI8mdW29ZIiKS9WkMtohkKTExMbRo0YIPPvgAgP79+zN69GgV1yIi4hkqsEUkS4iPj+f111+nVq1arFq1Cj8/vT2JiIg3qUtIRFwXHR1Nv379WLlyJW3btuXdd9+lbNmybscSERG5LCqwRcR127ZtY+vWrXzyySd0794dY4zbkURERC6bCmwRccWqVauIiIjgvvvuo23btsTExFCoUCG3Y4mIiFwxDXIUkUx18uRJRo8ezQ033MD48eM5deoUgIprERHJNlRgi0imWbhwIbVr1+aVV17hvvvuIzw8HH9/f7djiYiIZCgNERGRTLF7925atmxJhQoV+Pnnn2nWrJnbkURERHxCPdgi4lNr1/5/e3ceHUWdtv//fSeQBAVxQX8uIOAAQhYSIAoRwi4giziyCYoBQUBEHHR8ntFhRuWnM+4Lyk7EDUVhFERhRAGDKIssYZV9MGRGEBhBURKyfL5/dMsTMUAC6VSnc73O4ZxUd3XVlaRMLit3V60DoHr16syZM4f169erXIuISEhTwRaRgNi/fz/9+vUjISGBtLQ0ADp37sw555zjcTIREZHA0oiIiJQo5xwzZsxg5MiRHD58mEcffZSkpCSvY4mIiJQaFWwRKVH9+/dn+vTpNG3alNTUVGJiYryOJCIiUqpUsEXkrOXn52NmmBlt2rShSZMmjBw5kvDwcK+jiYiIlDrNYIvIWdmxYwft2rVj2rRpAAwaNP99lGsAACAASURBVIhRo0apXIuISLmlgi0iZyQ3N5dnnnmGuLg41q5dS0REhNeRREREgoJGRESk2DZu3MjAgQNZtWoV3bt3Z/z48Vx++eVexxIREQkKKtgiUmwZGRl88803zJgxg969e2NmXkcSEREJGirYIlIkK1asYN26dQwZMoTOnTuza9cuKleu7HUsERGRoKMZbBE5pZ9++on77ruPpKQknnrqKbKzswFUrkVERE5CBVtETmrRokU0bNiQ559/nmHDhrFmzRoiIyO9jiUiIhLUNCIiIoXKzMykY8eO1K5dm7S0NFq2bOl1JBERkTJBZ7BF5FfWrl0LQPXq1Zk7dy7r1q1TuRYRESkGFWwRAWDfvn306dOHxo0bk5aWBkCnTp2oVKmSx8lERETKFhVskXLOOcebb75JdHQ0s2fP5rHHHuO6667zOpaIiEiZpRlskXKuX79+zJgxg6SkJFJTU2nQoIHXkURERMo0FWyRcig/Px8zw8zo0KEDSUlJ3H333YSHh3sdTUREpMzTiIhIObNt2zbatGnDK6+8AsDAgQMZOXKkyrWIiEgJUcEWKSdyc3N56qmniI+PZ/369XrzooiISIBoRESkHFi/fj133HEHq1ev5ve//z3jxo3jsssu8zqWiIhISFLBFikHMjMz2bNnDzNnzqRHjx6YmdeRREREQpYKtkiI+vLLL1m/fj3Dhg2jc+fO7Nq1i3PPPdfrWCIiIiFPM9giIebIkSPce++9tGjRgmeffZbs7GwAlWsREZFSooItEkIWLFhAbGwsL730EnfffTdr1qwhMjLS61giIiLlikZERELEnj176NKlC7/73e9YsmQJLVq08DqSiIhIuaQz2CJl3OrVqwGoUaMG8+bNIz09XeVaRETEQyrYImXU3r176dWrF4mJiaSlpQFw/fXXExUV5XEyERGR8k0FW6SMcc7x2muvER0dzdy5c/nb3/7Gdddd53UsERER8dMMtkgZc8stt/Duu+/SvHlzpk6dSv369b2OJCIiIgWoYIuUAfn5+ZgZZkbnzp1JTk5m+PDhhIXpj1AiIiLBRr+dRYLcli1baNmyJampqQCkpKQwYsQIlWsREZEgpd/QIkEqJyeHv/3tb8THx7N582YqV67sdSQREREpAo2IiASh9PR0Bg4cSHp6Oj179uSll17i0ksv9TqWiIiIFIEKtkgQ2rt3L3v37uUf//gHN998s9dxREREpBhUsEWCxNKlS1m/fj3Dhw+nU6dO7Ny5k3POOcfrWCIiIlJMmsEW8diPP/7IiBEjSE5O5oUXXiA7OxtA5VpERKSMUsEW8dDHH39MbGws48eP595772XNmjVERkZ6HUtERETOgkZERDyyZ88eunbtSp06dVi6dKnuxigiIhIidAZbpBQ551i5ciUANWrUYP78+axdu1blWkREJISoYIuUkm+//ZYePXrQtGlT0tLSAGjfvj1RUVEeJxMREZGSpIItEmDOOaZNm0Z0dDTz58/nySefpHnz5l7HEhERkQDRDLZIgPXu3ZtZs2aRnJzM1KlTqVevnteRREREJIBUsEUCIC8vDzMjLCyMbt260bZtW4YOHUpYmP5oJCIiEur0216khH399dckJyeTmpoKwO23385dd92lci0iIlJO6De+SAnJycnhscceIyEhga1bt1K1alWvI4mIiIgHNCIiUgLWrl3LgAEDWL9+PX369GHs2LFccsklXscSERERD6hgi5SAffv2ceDAAWbPnk337t29jiMiIiIeUsEWOUNLlixhw4YN3H333XTq1IkdO3ZQqVIlr2OJiIiIxzSDLVJMP/zwA8OHD6dVq1aMHTuW7OxsAJVrERERAVSwRYpl3rx5xMTEMGnSJO677z7WrFlDZGSk17FEREQkiGhERKSI9uzZQ/fu3bn66quZNWsWTZs29TqSiIiIBCGdwRY5Beccy5cvB6BGjRosWLCANWvWqFyLiIjISalgi5zEf/7zH2666SaSkpJIS0sDoE2bNkRERHicTERERIKZCrbICZxzTJ06lejoaBYsWMAzzzxD8+bNvY4lIiIiZYRmsEVO0LNnT9577z1atWrF1KlTqVOnjteRREREpAxRwRYB8vLyMDPCwsK46aab6NChA3feeSdhYfojj4iIiBSP2oOUexs3bqR58+akpqYC0L9/f4YOHapyLSIiImdEDULKrWPHjvHoo4/SuHFjdu7cyQUXXOB1JBEREQkBGhGRcmn16tUMGDCAjRs30q9fP1544QUuvvhir2OJiIhICFDBlnLp4MGDHDp0iLlz59K1a1ev44iIiEgIUcGWcmPx4sVs2LCBkSNH0qFDB7Zv305UVJTXsURERCTEaAZbQt7hw4cZOnQobdu2ZcKECWRnZwOoXIuIiEhAqGBLSJs7dy7R0dFMnTqVP/7xj6xevZrIyEivY4mIiEgI04iIhKw9e/bQo0cP6tevz+zZs7nmmmu8jiQiIiLlQEDPYJtZJzPbamY7zOxPhTx/n5ltNrP1ZrbQzGoGMo+EPuccX375JQA1atRgwYIFrFq1SuVaRERESk3ACraZhQPjgBuAaKCvmUWfsNpaINE51xCYBTwVqDwS+jIzM7nxxhtp3rw5aWlpALRu3ZqIiAiPk4mIiEh5Esgz2NcCO5xzu5xzx4AZQPeCKzjnFjvnfvYvLgeqBzCPhKj8/HwmTZpEdHQ0Cxcu5LnnnqNFixZexxIREZFyKpAz2FcAewosZwJNT7H+IGB+APNIiOrRowezZ8+mbdu2TJkyhauuusrrSCIiIlKOBbJgWyGPuUJXNLsNSARaneT5IcAQgIhL65RUPinDcnNzCQsLIywsjB49etClSxcGDRqEWWGHnYiIiEjpCeSISCZQo8BydeA/J65kZu2BPwM3OueyC9uQc26ycy7ROZcYkKRSpqxfv56kpCSmTJkCwG233cbgwYNVrkVERCQoBLJgfwXUNbPaZhYB3AJ8UHAFM2sETMJXrr8LYBYJAdnZ2Tz88MM0adKEb775hosvvtjrSCIiIiK/EbAREedcrpmNAD4GwoFXnHObzGwMsMo59wHwNFAZmOk/+5jhnLsxUJmk7Prqq68YMGAAmzdvpn///jz//PNcdNFFXscSERER+Y2A3mjGOTcPmHfCY38t8HH7QO5fQsf333/PkSNHmDdvHjfccIPXcUREREROSndylKC1aNEiNmzYwL333kuHDh3Ytm2bbnMuIiIiQS+gd3IUOROHDh3izjvvpF27dkyaNInsbN97X1WuRUREpCxQwZagMmfOHKKjo3nllVf4n//5H1avXq1iLSIiImWKRkQkaGRkZNCrVy8aNGjABx98QGKirsooIiIiZY/OYIunnHN8/vnnAFx55ZV8+umnfPXVVyrXIiIiUmapYItnMjIy6NKlCy1btiQtLQ2Ali1bEhER4XEyERERkTOngi2lLj8/n/HjxxMTE8OSJUsYO3YsLVq08DqWiIiISInQDLaUuptvvpk5c+Zw/fXXM3nyZGrVquV1JBEREZESo4ItpSI3N5ewsDDCwsLo06cP3bt3Z8CAAfjv4CkiIiISMjQiIgG3bt06mjZtyuTJkwHo27cvAwcOVLkWERGRkKSCLQGTlZXF6NGjSUxMJDMzk0svvdTrSCIiIiIBpxERCYiVK1eSkpLCli1bSElJ4bnnnuPCCy/0OpaIiIhIwKlgS0D88MMPHD16lH/+85907NjR6zgiIiIipUYFW0rMggUL2LRpE6NGjaJ9+/Zs3bpVtzkXERGRckcz2HLWvv/+ewYOHEjHjh1JTU0lOzsbQOVaREREyiUVbDkr7733HtHR0bzxxhs8+OCDrFq1SsVaREREyjWNiMgZy8jI4JZbbiE2NpZ58+bRqFEjryOJiIiIeE5nsKVYnHOkpaUBcOWVV7Jo0SJWrFihci0iIiLip4ItRfbNN99www030Lp16+Mlu0WLFlSsWNHjZCIiIiLBQwVbTis/P5+XX36ZmJgYli5dyksvvURycrLXsURERESCkmaw5bRuuukm5s6dS8eOHZk0aRI1a9b0OpKIiIhI0FLBlkLl5OQQHh5OWFgYffv2pWfPnvTv3x8z8zqaiIiISFDTiIj8xpo1a7j22muZOHEiAH379uX2229XuRYREREpAhVsOe7o0aM8+OCDXHvttezdu5caNWp4HUlERESkzNGIiACwfPlyUlJS2LZtG3fccQfPPPMMF1xwgdexRERERMocFWwB4KeffiInJ4dPPvmE9u3bex1HREREpMxSwS7H/vnPf7Jp0ybuv/9+2rVrx5YtW4iIiPA6loiIiEiZphnscujgwYOkpKRwww038Nprr3Hs2DEAlWsRERGREqCCXY4455g1axbR0dG89dZbjB49mq+++krFWkRERKQEaUSkHMnIyKBfv340bNiQBQsWEB8f73UkERERkZCjM9ghzjnHokWLAKhZsyafffYZy5cvV7kWERERCRAV7BD2r3/9iw4dOtCuXTvS0tIAuO6666hQQX+4EBEREQkUFewQlJeXx4svvkhsbCwrVqxgwoQJJCcnex1LREREpFzQqcwQ1L17dz766CM6d+7MxIkTdUdGERERkVKkgh0icnJyCA8PJywsjP79+9O3b1/69euHmXkdTURERKRc0YhICFi1ahWJiYlMmDABgD59+nDrrbeqXIuIiIh4QAW7DDt69Cj/+7//S9OmTdm/fz81a9b0OpKIiIhIuacRkTJq2bJlpKSksH37dgYPHszTTz/N+eef73UsERERkXJPBbuMOnr0KPn5+Xz66ae0a9fO6zgiIiIi4qeCXYbMmzePTZs28cADD9C2bVu+/vprKlas6HUsERERESlAM9hlwIEDB7jtttvo0qUL06dP59ixYwAq1yIiIiJBSAU7iDnnmDFjBg0aNODdd9/l4YcfZuXKlURERHgdTUREREROQiMiQSwjI4OUlBTi4+NJTU0lLi7O60giIiIicho6gx1knHN8+umnANSsWZO0tDSWLVumci0iIiJSRugMdhDZuXMnd955J4sXL+azzz6jVatWNGvWzOtYIiIiUgQ5OTlkZmaSlZXldRQppqioKKpXr15i729TwQ4CeXl5vPjii4wePZqKFSsyadIkkpOTvY4lIiIixZCZmUmVKlWoVauW7qZchjjnOHjwIJmZmdSuXbtEtqmCHQS6devG/Pnz6dq1KxMmTKB69epeRxIREZFiysrKUrkug8yMiy66iP3795fYNlWwPXLs2DEqVKhAWFgYAwYMoH///txyyy36j1JERKQM0+/xsqmkv296k6MHVq5cSZMmTRg/fjwAvXv3pm/fvvqPUkRERM7a448/TkxMDA0bNiQhIYEVK1aQm5vLQw89RN26dUlISCAhIYHHH3/8+GvCw8NJSEggJiaG+Ph4nnvuOfLz8z38LMo2ncEuRT///DN/+ctfeOGFF7jsssv43e9+53UkERERCSHLli3jww8/ZM2aNURGRnLgwAGOHTvG6NGj2bt3Lxs2bCAqKooff/yRZ5999vjrKlWqRHp6OgDfffcd/fr14/Dhwzz66KNefSplmgp2KVm6dCkpKSns2rWLoUOH8uSTT1K1alWvY4mIiEgI+fbbb6lWrRqRkZEAVKtWjZ9//pkpU6awe/duoqKiAKhSpQqPPPJIodu45JJLmDx5Mtdccw2PPPKI/sJ+BjQiUkpycnIIDw9n8eLFTJw4UeVaREQkxLVuDa++6vs4J8e3/OabvuWff/Ytv/OOb/nwYd/ye+/5lg8c8C3Pnetb3ru3aPvs0KEDe/bsoV69egwfPpy0tDR27NjBlVdeSZUqVYqc/aqrriI/P5/vvvuuyK+R/6OCHUBz587lqaeeAqBNmzZs3ryZ1q1bextKREREQlblypVZvXo1kydP5uKLL6ZPnz589tlnv1pn2rRpJCQkUKNGDfbs2XPSbTnnApw2dFlZ++JFXlbX7d2xiQvOjfA6yknt37+fe++9l7fffvv4mwsiIoI3r4iIiJy9r7/+mgYNGngd41dmzZrFpEmTWLNmDbt37/7VWezY2Fg+/PBDatWqReXKlTly5Mjx53bt2sU111zDgQMHys2ISGHfPzNb7ZxLLO62dAa7BDnneOutt2jQoAGzZs1izJgxKtciIiJSarZu3cr27duPL6enp3P11VczaNAgRowYcfwuk3l5eRw7dqzQbezfv59hw4YxYsSIclOuS5re5FiCMjIyGDhwII0aNSI1NZWYmBivI4mIiEg5cuTIEe655x4OHTpEhQoVqFOnDpMnT6Zq1ar85S9/ITY2lipVqlCpUiVSUlK4/PLLATh69CgJCQnk5ORQoUIF+vfvz3333efxZ1N2aUTkLOXn5/PJJ5/QsWNH4P+ucR0eHu5xMhERESlNwTgiIkWnEZEgsX37dtq2bUunTp1YsmQJANdee63KtYiIiEg5poJ9BnJzc3n66adp2LAh6enppKamkpyc7HUsEREREQkCmsE+A127duXjjz+me/fujB8//vj8koiIiIiICnYRZWdnU7FiRcLCwhg8eDB33HEHvXr10rtrRURERORXNCJSBMuXL6dx48aMGzcOgJ49e9K7d2+VaxERERH5DRXsU/jpp58YNWoU1113HT/++CN169b1OpKIiIiIBDkV7JP4/PPPiYuL44UXXuCuu+5i48aNdOrUyetYIiIiIicVHh5OQkICsbGxdOvWjUOHDpXIdnfv3k1sbGyJbKugRx55hCuuuIKEhAQSEhL405/+VOL7+EV6ejrz5s0L2PYLUsE+idzcXCpWrEhaWhrjxo3jvPPO8zqSiIiIyClVqlSJ9PR0Nm7cyIUXXnh8vDWYjRo1ivT0dNLT03niiSeK/Lq8vLxi7UcF2yOzZ8/m73//OwBt2rRh06ZNtGzZ0uNUIiIiIsWXlJTEv//9b8B3h8d27drRuHFj4uLimDNnDuA7M92gQQPuvPNOYmJi6NChA0ePHgVg9erVxMfHk5SU9KuinpWVxcCBA4mLi6NRo0YsXrwYgFdffZWbbrqJbt26Ubt2bV5++WWee+45GjVqRLNmzfjvf/9b5OwLFy6kUaNGxMXFcccdd5CdnQ1ArVq1GDNmDC1atGDmzJns3LmTTp060aRJE5KTk9myZQsAM2fOJDY2lvj4eFq2bMmxY8f461//yjvvvENCQgLvvPPO2X+BT0FXEQH27dvHPffcw8yZM2ncuDH3338/ERERVKigL4+IiIgUX60/fRSQ7e5+okuR1svLy2PhwoUMGjQIgKioKN5//33OO+88Dhw4QLNmzbjxxhsB343z3n77baZMmULv3r35xz/+wW233cbAgQN56aWXaNWqFQ888MDxbf9Stjds2MCWLVvo0KED27ZtA2Djxo2sXbuWrKws6tSpw5NPPsnatWsZNWoUr7/+On/4wx9+k/X555/nzTffBODJJ5+kVatWDBgwgIULF1KvXj1uv/12JkyYcPy1UVFRLF26FIB27doxceJE6taty4oVKxg+fDiLFi1izJgxfPzxx1xxxRUcOnSIiIgIxowZw6pVq3j55ZfP5EtfLOX6DLZzjjfeeIPo6GjmzJnD448/zvLly4mICI7bsIuIiIgUx9GjR0lISOCiiy7iv//9L9dffz3g6zwPPfQQDRs2pH379vz73/9m3759ANSuXZuEhAQAmjRpwu7duzl8+DCHDh2iVatWAPTv3//4PpYuXXp8uX79+tSsWfN4wW7Tpg1VqlTh4osvpmrVqnTr1g2AuLg4du/eXWjmgiMiHTt2ZOvWrdSuXZt69eoBkJKScvyO2QB9+vQBfGflv/zyS3r16kVCQgJDhw7l22+/BaB58+YMGDCAKVOmFHuUpCSU61O0GRkZDB48mMTERFJTU6lfv77XkURERCQEFPVMc0n7ZQb78OHDdO3alXHjxjFy5EimT5/O/v37Wb16NRUrVqRWrVpkZWUBEBkZefz14eHhHD16FOfcSS9H7Jw76f4LbissLOz4clhYGLm5uUX6HE61fYBzzz0XgPz8fM4//3zS09N/s87EiRNZsWIFH330EQkJCYWuE0jl7gx2fn4+8+fPB6BmzZp88cUXLFmyROVaREREQkbVqlUZO3YszzzzDDk5ORw+fJhLLrmEihUrsnjxYr755ptTvv7888+natWqx0cxpk+ffvy5li1bHl/etm0bGRkZXH311SWWvX79+uzevZsdO3YA8MYbbxw/k17QeeedR+3atZk5cybgK+br1q0DYOfOnTRt2pQxY8ZQrVo19uzZQ5UqVfjxxx9LLOeplKuCvW3bNlq3bk3nzp1JS0sDIDExkfDwcI+TiYiIiJSsRo0aER8fz4wZM7j11ltZtWoViYmJTJ8+vUgnFqdNm8bdd99NUlISlSpVOv748OHDycvLIy4ujj59+vDqq6/+6sz12YqKimLatGn06tWLuLg4wsLCGDZsWKHrTp8+ndTUVOLj44mJiTn+5s0HHniAuLg4YmNjadmyJfHx8bRp04bNmzeXypsc7XSn4YNN5GV13d4dm7jg3KLPSefm5vLss8/y8MMPU6lSJZ5//nlSUlJ0J0YREREpMV9//TUNGjTwOoacocK+f2a22jmXWNxtlYsZ7C5durBgwQJuvvlmxo0bx6WXXup1JBEREREJUSFbsLOysqhYsSLh4eEMGTKEIUOG0KNHD69jiYiIiEiIC8kZ7C+++IKEhITj12ns0aOHyrWIiIiIlIqQKthHjhxh5MiRJCcnk5WVpTkoERERKVVl7b1t4lPS37eQKdhpaWnExsby8ssvM2LECDZu3Hj84uoiIiIigRYVFcXBgwdVsssY5xwHDx4kKiqqxLYZUjPY55xzDp9//jnNmzf3OoqIiIiUM9WrVyczM5P9+/d7HUWKKSoqiurVq5fY9gJ6mT4z6wS8CIQDU51zT5zwfCTwOtAEOAj0cc7tPtU2C16m77333mPLli089NBDAOTl5ema1iIiIiJSIs70Mn0BGxExs3BgHHADEA30NbPoE1YbBHzvnKsDPA88WZRt79u3l549e9KjRw/ef/99jh07BqByLSIiIiKeC+QM9rXADufcLufcMWAG0P2EdboDr/k/ngW0s9Pc/SXv5x9o1iSeDz/8kL///e98+eWXREQU/aYzIiIiIiKBFMgZ7CuAPQWWM4GmJ1vHOZdrZoeBi4ADJ9to3g/7aZB0Ha9OSy3R+96LiIiIiJSEQBbsws5EnzjwXZR1MLMhwBD/YvbyZV9srF+//lnGkxBTjVP8j5mUWzoupDA6LqQwOi6kMGd0NjeQBTsTqFFguTrwn5Osk2lmFYCqwH9P3JBzbjIwGcDMVp3JsLmENh0XUhgdF1IYHRdSGB0XUhgzW3UmrwvkDPZXQF0zq21mEcAtwAcnrPMBkOL/uCewyOnikSIiIiJShgXsDLZ/pnoE8DG+y/S94pzbZGZjgFXOuQ+AVOANM9uB78z1LYHKIyIiIiJSGgJ6oxnn3Dxg3gmP/bXAx1lAr2JudnIJRJPQo+NCCqPjQgqj40IKo+NCCnNGx0VAbzQjIiIiIlLeBHIGW0RERESk3Anagm1mncxsq5ntMLM/FfJ8pJm9439+hZnVKv2UUtqKcFzcZ2abzWy9mS00s5pe5JTSdbrjosB6Pc3MmZmuFFAOFOW4MLPe/p8Zm8zsrdLOKKWvCL9HrjSzxWa21v+7pLMXOaX0mNkrZvadmW08yfNmZmP9x8x6M2t8um0GZcEO5G3Wpewq4nGxFkh0zjXEd3fQp0o3pZS2Ih4XmFkVYCSwonQTiheKclyYWV3gQaC5cy4G+EOpB5VSVcSfF6OBd51zjfBdfGF86aYUD7wKdDrF8zcAdf3/hgATTrfBoCzYBOg261Lmnfa4cM4tds797F9cju/66xLaivLzAuD/x/c/XFmlGU48U5Tj4k5gnHPuewDn3HelnFFKX1GOCwec5/+4Kr+9h4eEGOfcEgq5D0sB3YHXnc9y4Hwzu+xU2wzWgl3YbdavONk6zrlc4JfbrEvoKspxUdAgYH5AE0kwOO1xYWaNgBrOuQ9LM5h4qig/L+oB9czsCzNbbmanOoMloaEox8UjwG1mlonvSmj3lE40CWLF7R+BvUzfWSix26xLSCny99zMbgMSgVYBTSTB4JTHhZmF4RsjG1BagSQoFOXnRQV8f/Jtje+vXZ+bWaxz7lCAs4l3inJc9AVedc49a2ZJ+O7XEeucyw98PAlSxe6cwXoGuzi3WedUt1mXkFKU4wIzaw/8GbjROZddStnEO6c7LqoAscBnZrYbaAZ8oDc6hryi/h6Z45zLcc79C9iKr3BL6CrKcTEIeBfAObcMiAKqlUo6CVZF6h8FBWvB1m3WpTCnPS78owCT8JVrzVOWD6c8Lpxzh51z1ZxztZxztfDN5t/onFvlTVwpJUX5PTIbaANgZtXwjYzsKtWUUtqKclxkAO0AzKwBvoK9v1RTSrD5ALjdfzWRZsBh59y3p3pBUI6I6DbrUpgiHhdPA5WBmf73vGY45270LLQEXBGPCylninhcfAx0MLPNQB7wgHPuoHepJdCKeFzcD0wxs1H4xgAG6AReaDOzt/GNilXzz94/DFQEcM5NxDeL3xnYAfwMDDztNnXMiIiIiIiUnGAdERERERERKZNUsEVERERESpAKtoiIiIhICVLBFhEREREpQSrYIiIiIiIlSAVbRKQYzCzPzNIL/Kt1inVrmdnGEtjnZ2a21czW+W/rffUZbGOYmd3u/3iAmV1e4LmpZhZdwjm/MrOEIrzmD2Z2ztnuW0QkmKhgi4gUz1HnXEKBf7tLab+3OufigdfwXe+9WJxzE51zr/sXBwCXF3husHNuc4mk/L+c4ylazj8AKtgiElJUsEVEzpL/TPXnZrbG/++6QtaJMbOV/rPe682srv/x2wo8PsnMwk+zuyVAHf9r25nZWjPbYGavmFmk//EnZdnAmQAAA0VJREFUzGyzfz/P+B97xMz+aGY9gURgun+flfxnnhPN7C4ze6pA5gFm9tIZ5lwGXFFgWxPMbJWZbTKzR/2PjcRX9Beb2WL/Yx3MbJn/6zjTzCqfZj8iIkFHBVtEpHgqFRgPed//2HfA9c65xkAfYGwhrxsGvOicS8BXcDP9t2HuAzT3P54H3Hqa/XcDNphZFPAq0Mc5F4fvzrx3mdmFwO+BGOdcQ+Cxgi92zs0CVuE705zgnDta4OlZwM0FlvsA75xhzk74bkX+iz875xKBhkArM2vonBsL/Ado45xr479d+Wigvf9ruQq47zT7EREJOkF5q3QRkSB21F8yC6oIvOyfOc4D6hXyumXAn82sOvCec267mbUDmgBfmRlAJXxlvTDTzewosBu4B7ga+Jdzbpv/+deAu4GXgSxgqpl9BHxY1E/MObffzHaZWTNgu38fX/i3W5yc5+K7DXXjAo/3NrMh+H7vXAZEA+tPeG0z/+Nf+PcTge/rJiJSpqhgi4icvVHAPiAe318Gs05cwTn3lpmtALoAH5vZYMCA15xzDxZhH7c651b9smBmFxW2knMu18yuBdoBtwAjgLbF+FzeAXoDW4D3nXPOfG23yDmBdcATwDjgZjOrDfwRuMY5972ZvQpEFfJaAz5xzvUtRl4RkaCjERERkbNXFfjWOZcP9Md39vZXzOwqYJd/LOIDfKMSC4GeZnaJf50LzaxmEfe5BahlZnX8y/2BNP/MclXn3Dx8byAs7EoePwJVTrLd94CbgL74yjbFzemcy8E36tHMP15yHvATcNjM/j/ghpNkWQ40/+VzMrNzzKywvwaIiAQ1FWwRkbM3Hkgxs+X4xkN+KmSdPsBGM0sH6gOv+6/cMRpYYGbrgU/wjU+clnMuCxgIzDSzDUA+MBFfWf3Qv700fGfXT/QqMPGXNzmesN3vgc1ATefcSv9jxc7pn+1+Fvijc24dsBbYBLyCb+zkF5OB+Wa22Dm3H98VTt7272c5vq+ViEiZYs45rzOIiIiIiIQMncEWERERESlBKtgiIiIiIiVIBVtEREREpASpYIuIiIiIlCAVbBERERGREqSCLSIiIiJSglSwRURERERKkAq2iIiIiEgJ+n9h1FW2CrmPlAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 864x576 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.figure(figsize=(12,8)); \n",
    "plt.plot(fpr, tpr, \"b:\", label=\"SGD\")\n",
    "plot_roc_curve(fpr_forest, tpr_forest, \"Random Forest\")\n",
    "plt.legend(loc=\"lower right\")\n",
    "plt.show();"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 125,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.999442690992666"
      ]
     },
     "execution_count": 125,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "roc_auc_score(y_train_0, y_scores_forest)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 126,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "RandomForestClassifier(bootstrap=True, class_weight=None, criterion='gini',\n",
       "                       max_depth=None, max_features='auto', max_leaf_nodes=None,\n",
       "                       min_impurity_decrease=0.0, min_impurity_split=None,\n",
       "                       min_samples_leaf=1, min_samples_split=2,\n",
       "                       min_weight_fraction_leaf=0.0, n_estimators=100,\n",
       "                       n_jobs=None, oob_score=False, random_state=0, verbose=0,\n",
       "                       warm_start=False)"
      ]
     },
     "execution_count": 126,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "f_clf.fit(X_train, y_train_0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 127,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_train_rf = cross_val_predict(f_clf, X_train, y_train_0, cv=3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 128,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.9929353585305546"
      ]
     },
     "execution_count": 128,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "precision_score(y_train_0, y_train_rf) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 129,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.9491811581968597"
      ]
     },
     "execution_count": 129,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "recall_score(y_train_0, y_train_rf) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 130,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[54037,    40],\n",
       "       [  301,  5622]])"
      ]
     },
     "execution_count": 130,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "confusion_matrix(y_train_0, y_train_rf)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}