ch04_분류(2).ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": [],
      "collapsed_sections": [],
      "authorship_tag": "ABX9TyPNqsToYY5ed2R2rFl3uQdA",
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/wlsisl/ESAA_22-2/blob/main/classification_1.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "CzwVDDw6aJtc"
      },
      "outputs": [],
      "source": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## 06 XGBoost( eXtra Gradient Boost)\n",
        "\n",
        "XGBoost : GBM에 기반하고 있지만 GBM의 단점인 느린 수행 시간 및 과적합 규제 부재 등의 문제를 해결하고있다. 특히 병렬 cpu환경에서 병렬 학습이 가능해 기존 gbm보다 빠르게 학습을 완료할 수 있다.\n",
        "\n",
        "<XGBoost의 장점>\n",
        "1. 뛰어난 예측 성능\n",
        "2. GBM대비 빠른 수행 시간\n",
        "3. 과적합 규제\n",
        "4. Tree pruning 나무 가지치기 - GBM은 분할 시 부정 손실이 발생하면 분할을 더이상 수행하지 않지만 이 방식도 지나치게 많은 분할을 발생할 수 있다. 그래서 이것을 통해 더 이상 긍정 이득이 없는 분할을 가지치기 해서 분하 ㄹ수를 더 줄임\n",
        "5. 자체 내장된 교차 검증\n",
        "6. 결손값 자체 처리\n",
        "\n",
        "파이썬 패키지명 : xgboost\n",
        "XGBoost 고유의 프레임워크를  파이썬 언어 기반에서 구현한 것으로 별도의 API기반이었다. 사이킷런 프레임워크를 기반으로 한 것이 아니기에 사이킷런의 fit(), predict())메서드와 같은 고유의 아키텍처가 적용될 수 없으며, 다양한 유틸리티와 함께 사용될 수 없었다. \n",
        "-> 래퍼 클래스를 제공하여 사이킷런과 연동할 수 있게 함. \n",
        "\n",
        "* 사이킷런의 래퍼 클래서 : XGBClassifier , XGBRegressor  를 이용한다.\n",
        "\n",
        "* 독자적인 XGBoost 프레임 워크 기반과 사이킷런과 연동되는 모듈, 이 두가지를 구분하고 사용시 주의해야함.\n",
        "\n"
      ],
      "metadata": {
        "id": "rl0JQb3LaYy3"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import xgboost as xgb\n",
        "from xgboost import XGBClassifier"
      ],
      "metadata": {
        "id": "QbwMTCYmaZcj"
      },
      "execution_count": 3,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "### 파이썬 래퍼 XGBoost 하이퍼 파라미터\n",
        "- GBM과 유사한 하이퍼 파라미터 동일 + 조기중단 + 과적합 규제 위한 파라미터\n",
        "\n",
        "여기서 아까 언급하였던 파이썬 본래의 XGBoost모듈과 사이킷런의 XGBoost모듈의 일부 하이퍼 파라미터가 다르다.( 같은기능이지만)\n",
        "\n",
        "1. 파이썬 래퍼의 XGBoost 하이퍼 파라미터 종류\n",
        " * 일반 파라미터 : 일반적으로 실행시 스레드의 개수나 silent 모드 등의 선택을 위한 파라미터로서 디폴트 파라미터 값을 바꾸는 경우는 거의 없다.\n",
        "\n",
        "    1) booster : gbtree 또는 gblinear선택. 디폴트는 전자.\n",
        "\n",
        "    2) silent: 디폴트는 0, 출력 메시지를 나타내고 싶지 않을 경우\n",
        "\n",
        "    3) nthread: cpu의 실행 스레드 개수를 조정하며, 디폴트는 cpu의 전체 스레드를 다 사용하는 것. 멀티 코어/스레드 cpu시스템에서 전체 cpu를 사용하지 않고 일부 cpu만 사용해 ml애플리케이션을 구동하는 경우 변경함\n",
        "\n",
        " * 부스터 파라미터 : 트리 최적화, 부스팅, regularization등과 관련 파라미터 등을 지칭한다.\n",
        "\n",
        "    1) eta : GBM의 학습률과 같음. 0에서 1사이의 값을 지정하며 부스팅 스텝을 반복적으로 수행할 때 업데이트 되는 학습률 값. 디폴트는 0.1이며 보통 0.01~0.2사이 값을 선호\n",
        "\n",
        "    2) num_boost_rounds: GBM의 n_estimators와 같은 파라미터\n",
        "\n",
        "    3) min_child_weight[default=1] : 트리에서 추가적으로 가지를 나눌지를 결정하기 위해 필요하 ㄴ데이터들의 weight 총합. ( 이 값이 클수록 분할을 자제한다. 즉 과적합을 조절함)\n",
        "\n",
        "    4) gamma : 트리의 리프 노드를 추가적으로 나눌지를 결정할 최소 손실 감소 값이다. 해당 값보다 큰 손실이 감소된 경우에 리프 노드를 분리한다. 값이 클수록 과적합 감소 효과가 있다. \n",
        "\n",
        "    5) max_depth[default=6] : 트리 기반 알고리즘의 max_depth와 같다. 0을 지정하면 깊이에 제한이 없다. 이 값이 높으면 특정 피처 조건에 특화되어 룰 조건이 만들어지므로 과적합 가능성이 높아지며 보통 3~10 사이 값을 적용한다.\n",
        "\n",
        "    6) sub_sample[default=1] : GBM의 subsample과 동일하다. 트리가 커져서 과적합되느 ㄴ것을 제어하기 위해 데이터를 샘플링하는 비율을 지정한다. \n",
        "\n",
        "    7) colsample_bytree[default=1] : GBM의 max_features와 유사. 트리 생성에 필요한 피처를 임의로 샘플링 하는 데 사용. 매우 많은 피처(칼럼)가 있는 경우 과적합을 조정하는데 적용\n",
        "\n",
        "    8) lambda[default=1] : L2 Regularization적용값임. 피처 개수가 많을 경우 적용을 검토하며 값이 클수록 과적합 감소효과가 있음.\n",
        "\n",
        "    9) alpha[default=0] : L1 Regularization적용값. 피처 개수가 많을 경우 적용을 검토하며 값이 클수록 과적합 감소 효과가 있다.\n",
        "\n",
        "    10) scale_pos_weight[default=1] : 특정 값으로 치우친 비대칭한 클래스로 구성된 데이터 세트의 균형을 유지하기 위한 파라미터  \n",
        "\n",
        " * 학습 태스크 파라미터: 학습 수행 시의 객체 함수, 평가를 위한 지표 등을 설정하느 ㄴ파라미터이다. \n",
        "\n",
        "    1) objective: 최솟값을 가져야할 손실 함수를 정의 많은 유형의 손실 함수를 사용할 수 있는데 이진분류인지 다중 분류인지에 따라 달라짐.\n",
        "\n",
        "    2) binary:logistic:이진 분류일 때 적용 \n",
        "\n",
        "    3) multi:softmax: 다중 분류일 때 적용. num_class를 지정해야함(레이블 클래스 개수)\n",
        "    \n",
        "    4) multi:softprob: 위와 유사하나 개별 레이블 클래스의 해당되는 예측 확률을 반환한다.\n",
        "\n",
        "    5) eval_metric:검증에 사용되는 함수를 정의한다. \n",
        "    \n",
        "    기본값은 회귀인 경우: rmse, 분류일 경우: error\n",
        "    \n",
        "      - mae , logloss, merror, mlogloss, auc 등이 있음.\n",
        "\n",
        "  \n",
        "<과적합 문제가 심각하다면 해결책>\n",
        "* eta값을 낮춘다. num_round는 반대로 높여줘야한다.\n",
        "* max_depth 값을 낮춘다.\n",
        "* min_child_weight값을 높인다.\n",
        "* gamma 값을 높인다.\n",
        "* subsample과 colsample_bytree를 조정하는 것도 트리가 너무 복잡하게 생성되는 것을 막아 과적합 문제에 도움이 된다.\n",
        "\n",
        "XGBoost자체적으로 교차검증/ 성능 평가/피처 중요도 등 시각화 기능을 가지고 있다. \n",
        "\n",
        "\n"
      ],
      "metadata": {
        "id": "_HSWcSDEaZiD"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import xgboost\n",
        "\n",
        "print(xgboost.__version__)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "A6CCa1pnaZmy",
        "outputId": "c1ee8116-574e-44f3-c72e-5b354b8d5a51"
      },
      "execution_count": 4,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "0.90\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "### 파이썬 래퍼 XGBoost적용 : 위스콘신 유방암 예측\n",
        "\n",
        "위스콘신 유방암 데이터 세트를 활용하여 파이썬 래퍼 XGBoost API의 사용법을 살표보자.\n",
        "\n",
        "데이터 설명 : 종양의 크기, 모양 등의 다양한 속성값을 기반으로 악성 종양인지 양성 종양인지를 분류한 데이터 세트이다. \n",
        "\n",
        "해당 데이터 세트에 기반해 종양의 다양한 피처에 따라 악성종양인지 일반 양성종양인지를 xgboost를 이용해 예측해보자."
      ],
      "metadata": {
        "id": "ooZ-XTeKaZqv"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import xgboost as xgb\n",
        "from xgboost import plot_importance\n",
        "import pandas as pd\n",
        "import numpy as np\n",
        "from sklearn.datasets import load_breast_cancer\n",
        "from sklearn.model_selection import train_test_split\n",
        "import warnings\n",
        "warnings.filterwarnings('ignore')\n",
        "\n",
        "dataset = load_breast_cancer()\n",
        "X_features = dataset.data\n",
        "y_label=dataset.target\n",
        "cancer_df = pd.DataFrame(data=X_features, columns=dataset.feature_names)\n",
        "cancer_df['target']=y_label\n",
        "cancer_df.head(3)\n",
        "\n"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 174
        },
        "id": "8XOTfX6RaZuX",
        "outputId": "c3fbf5a4-3fce-4b0d-f070-051db850545f"
      },
      "execution_count": 5,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "   mean radius  mean texture  mean perimeter  mean area  mean smoothness  \\\n",
              "0        17.99         10.38           122.8     1001.0          0.11840   \n",
              "1        20.57         17.77           132.9     1326.0          0.08474   \n",
              "2        19.69         21.25           130.0     1203.0          0.10960   \n",
              "\n",
              "   mean compactness  mean concavity  mean concave points  mean symmetry  \\\n",
              "0           0.27760          0.3001              0.14710         0.2419   \n",
              "1           0.07864          0.0869              0.07017         0.1812   \n",
              "2           0.15990          0.1974              0.12790         0.2069   \n",
              "\n",
              "   mean fractal dimension  ...  worst texture  worst perimeter  worst area  \\\n",
              "0                 0.07871  ...          17.33            184.6      2019.0   \n",
              "1                 0.05667  ...          23.41            158.8      1956.0   \n",
              "2                 0.05999  ...          25.53            152.5      1709.0   \n",
              "\n",
              "   worst smoothness  worst compactness  worst concavity  worst concave points  \\\n",
              "0            0.1622             0.6656           0.7119                0.2654   \n",
              "1            0.1238             0.1866           0.2416                0.1860   \n",
              "2            0.1444             0.4245           0.4504                0.2430   \n",
              "\n",
              "   worst symmetry  worst fractal dimension  target  \n",
              "0          0.4601                  0.11890       0  \n",
              "1          0.2750                  0.08902       0  \n",
              "2          0.3613                  0.08758       0  \n",
              "\n",
              "[3 rows x 31 columns]"
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-6293c3b3-50a4-4f1d-8f88-7136f9d072a7\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>mean radius</th>\n",
              "      <th>mean texture</th>\n",
              "      <th>mean perimeter</th>\n",
              "      <th>mean area</th>\n",
              "      <th>mean smoothness</th>\n",
              "      <th>mean compactness</th>\n",
              "      <th>mean concavity</th>\n",
              "      <th>mean concave points</th>\n",
              "      <th>mean symmetry</th>\n",
              "      <th>mean fractal dimension</th>\n",
              "      <th>...</th>\n",
              "      <th>worst texture</th>\n",
              "      <th>worst perimeter</th>\n",
              "      <th>worst area</th>\n",
              "      <th>worst smoothness</th>\n",
              "      <th>worst compactness</th>\n",
              "      <th>worst concavity</th>\n",
              "      <th>worst concave points</th>\n",
              "      <th>worst symmetry</th>\n",
              "      <th>worst fractal dimension</th>\n",
              "      <th>target</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>17.99</td>\n",
              "      <td>10.38</td>\n",
              "      <td>122.8</td>\n",
              "      <td>1001.0</td>\n",
              "      <td>0.11840</td>\n",
              "      <td>0.27760</td>\n",
              "      <td>0.3001</td>\n",
              "      <td>0.14710</td>\n",
              "      <td>0.2419</td>\n",
              "      <td>0.07871</td>\n",
              "      <td>...</td>\n",
              "      <td>17.33</td>\n",
              "      <td>184.6</td>\n",
              "      <td>2019.0</td>\n",
              "      <td>0.1622</td>\n",
              "      <td>0.6656</td>\n",
              "      <td>0.7119</td>\n",
              "      <td>0.2654</td>\n",
              "      <td>0.4601</td>\n",
              "      <td>0.11890</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>20.57</td>\n",
              "      <td>17.77</td>\n",
              "      <td>132.9</td>\n",
              "      <td>1326.0</td>\n",
              "      <td>0.08474</td>\n",
              "      <td>0.07864</td>\n",
              "      <td>0.0869</td>\n",
              "      <td>0.07017</td>\n",
              "      <td>0.1812</td>\n",
              "      <td>0.05667</td>\n",
              "      <td>...</td>\n",
              "      <td>23.41</td>\n",
              "      <td>158.8</td>\n",
              "      <td>1956.0</td>\n",
              "      <td>0.1238</td>\n",
              "      <td>0.1866</td>\n",
              "      <td>0.2416</td>\n",
              "      <td>0.1860</td>\n",
              "      <td>0.2750</td>\n",
              "      <td>0.08902</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>19.69</td>\n",
              "      <td>21.25</td>\n",
              "      <td>130.0</td>\n",
              "      <td>1203.0</td>\n",
              "      <td>0.10960</td>\n",
              "      <td>0.15990</td>\n",
              "      <td>0.1974</td>\n",
              "      <td>0.12790</td>\n",
              "      <td>0.2069</td>\n",
              "      <td>0.05999</td>\n",
              "      <td>...</td>\n",
              "      <td>25.53</td>\n",
              "      <td>152.5</td>\n",
              "      <td>1709.0</td>\n",
              "      <td>0.1444</td>\n",
              "      <td>0.4245</td>\n",
              "      <td>0.4504</td>\n",
              "      <td>0.2430</td>\n",
              "      <td>0.3613</td>\n",
              "      <td>0.08758</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "<p>3 rows × 31 columns</p>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-6293c3b3-50a4-4f1d-8f88-7136f9d072a7')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "        \n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "      \n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-6293c3b3-50a4-4f1d-8f88-7136f9d072a7 button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-6293c3b3-50a4-4f1d-8f88-7136f9d072a7');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n",
              "  "
            ]
          },
          "metadata": {},
          "execution_count": 5
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "-> 종양의 크기와 모양에 관련된 많은 속성이 숫자형 값으로 돼있다. 타깃 레이블 값의 종류는 악성인 'malignant'가 0 , 양성인 'benign'이 1로 돼있다. "
      ],
      "metadata": {
        "id": "nv5Tu-IDaZy8"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "#값의 분포보기\n",
        "print(dataset.target_names)\n",
        "print(cancer_df['target'].value_counts())"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "bAP430xBaZ2U",
        "outputId": "0aecd1e8-6b26-46b3-cc13-d8b3967398e6"
      },
      "execution_count": 6,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "['malignant' 'benign']\n",
            "1    357\n",
            "0    212\n",
            "Name: target, dtype: int64\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "#전체 데이터 중 80%는 학습용데이터 20%는 테스트용 데이터 추출\n",
        "X_train, X_test, y_train, y_test = train_test_split(X_features,y_label,test_size=0.2, random_state=156 )\n",
        "\n",
        "print(X_train.shape,X_test.shape)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "3xGeKKCZj_at",
        "outputId": "eb499642-6fc0-441a-9fd4-3b017386e3db"
      },
      "execution_count": 7,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "(455, 30) (114, 30)\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "-> 파이썬 래퍼 XGBoost는 사이킷런과 차이가 여러가지 있지만 눈에 띄는 차이는 학습용과 테스트용 데이터 세트를 위해 별도의 객체인 DMatrix를 생성한다는 점이다. 주요 파라미터는 data(피처 데이터 세트), label( 분류의 경우 레이블 데이터 세트, 회귀의 경우 숫자형 종솟값 데이터 세트)\n",
        "\n"
      ],
      "metadata": {
        "id": "t0l9RC9BaZ5_"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# 넘파이 형태의 학습데이터 세트와 테스트 데이터 세트를 DMatrix로 변환하는 예제  \n",
        "dtrain = xgb.DMatrix(data=X_train, label = y_train)\n",
        "dtest= xgb.DMatrix(data=X_train, label = y_train)"
      ],
      "metadata": {
        "id": "H0GAKCqOaZ9g"
      },
      "execution_count": 8,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "파이썬 래퍼 XGBoost모듈인 xgboost를 이용해 학습을 수행하기 전 먼저 XGBoost의 하이퍼 파라미터를 설정한다. \n",
        "\n",
        "* max_depth(트리 최대 깊이)는 3.\n",
        "* 학습률 eta는 0.1\n",
        "* 예제 데이터가 0또는 1 이진 분류이므로 목적함수(objective)는 이진 로지스틱(binary:logistic)\n",
        "* 오류 함수의 평가 성능지표는 logloss\n",
        "* num_rounds(부스팅 반복 횟수)는 400회."
      ],
      "metadata": {
        "id": "jwmAhwzMaaA1"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "params = { 'max_depth':3, 'eta':0.1, 'objective':'binary:logistic', 'eval_metric':'logloss', 'early_stoppings':100}\n",
        "num_rounds = 400"
      ],
      "metadata": {
        "id": "wyUvQOVOaaD8"
      },
      "execution_count": 9,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "이제 모델을 학습시켜보자. 파이썬 래퍼 XGBoost는 하이퍼 파라미터를 xgboost모듈의 train()함수에 파라미터로 전달한다. (사이킷런의 경우는 estimator의 생성자를 하이퍼 파라미터로 전달하는데 반해 차이가 있음)\n",
        "\n",
        "조기중단: xgboost의 trian()함수에 early_stopping_rounds 파라미터를 입력하여 설정\n",
        "이때 반드시 eval_set과 eval_metric이 함께 설정 돼야 한다. \n",
        "\n",
        "* eval_set : 성능 평가를 수행할 평가용 데이터 세트를 설정\n",
        "* eval_metric: 평가 세트에 적용할 성능 평가 방법. 분류일 경우 주로 error, logloss를 적용"
      ],
      "metadata": {
        "id": "Eeu1GblAaaHD"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "#train 데이터 세트는 'train', evaluation(test)데이터 세트는 'eval'로 명기\n",
        "wlist=[(dtrain, 'train'),(dtest, 'eval')]\n",
        "#하이퍼 파라미터와 early stopping 파라미터를 train()함수의 파라미터로 전달\n",
        "xgb_model = xgb.train(params=params, dtrain=dtrain, num_boost_round= num_rounds, early_stopping_rounds=100, evals=wlist)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "v5ZlP8NxaccR",
        "outputId": "c96f21cc-9792-4c86-9760-c3dd5b0e53fe"
      },
      "execution_count": 10,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "[0]\ttrain-logloss:0.609688\teval-logloss:0.609688\n",
            "Multiple eval metrics have been passed: 'eval-logloss' will be used for early stopping.\n",
            "\n",
            "Will train until eval-logloss hasn't improved in 100 rounds.\n",
            "[1]\ttrain-logloss:0.540803\teval-logloss:0.540803\n",
            "[2]\ttrain-logloss:0.483753\teval-logloss:0.483753\n",
            "[3]\ttrain-logloss:0.434457\teval-logloss:0.434457\n",
            "[4]\ttrain-logloss:0.39055\teval-logloss:0.39055\n",
            "[5]\ttrain-logloss:0.354146\teval-logloss:0.354146\n",
            "[6]\ttrain-logloss:0.321222\teval-logloss:0.321222\n",
            "[7]\ttrain-logloss:0.292593\teval-logloss:0.292593\n",
            "[8]\ttrain-logloss:0.267467\teval-logloss:0.267467\n",
            "[9]\ttrain-logloss:0.245153\teval-logloss:0.245153\n",
            "[10]\ttrain-logloss:0.225694\teval-logloss:0.225694\n",
            "[11]\ttrain-logloss:0.207938\teval-logloss:0.207938\n",
            "[12]\ttrain-logloss:0.192184\teval-logloss:0.192184\n",
            "[13]\ttrain-logloss:0.177916\teval-logloss:0.177916\n",
            "[14]\ttrain-logloss:0.165222\teval-logloss:0.165222\n",
            "[15]\ttrain-logloss:0.153622\teval-logloss:0.153622\n",
            "[16]\ttrain-logloss:0.14333\teval-logloss:0.14333\n",
            "[17]\ttrain-logloss:0.133985\teval-logloss:0.133985\n",
            "[18]\ttrain-logloss:0.125599\teval-logloss:0.125599\n",
            "[19]\ttrain-logloss:0.117286\teval-logloss:0.117286\n",
            "[20]\ttrain-logloss:0.109688\teval-logloss:0.109688\n",
            "[21]\ttrain-logloss:0.102975\teval-logloss:0.102975\n",
            "[22]\ttrain-logloss:0.097067\teval-logloss:0.097067\n",
            "[23]\ttrain-logloss:0.091428\teval-logloss:0.091428\n",
            "[24]\ttrain-logloss:0.086335\teval-logloss:0.086335\n",
            "[25]\ttrain-logloss:0.081311\teval-logloss:0.081311\n",
            "[26]\ttrain-logloss:0.076857\teval-logloss:0.076857\n",
            "[27]\ttrain-logloss:0.072836\teval-logloss:0.072836\n",
            "[28]\ttrain-logloss:0.069248\teval-logloss:0.069248\n",
            "[29]\ttrain-logloss:0.065549\teval-logloss:0.065549\n",
            "[30]\ttrain-logloss:0.062414\teval-logloss:0.062414\n",
            "[31]\ttrain-logloss:0.059591\teval-logloss:0.059591\n",
            "[32]\ttrain-logloss:0.057096\teval-logloss:0.057096\n",
            "[33]\ttrain-logloss:0.054407\teval-logloss:0.054407\n",
            "[34]\ttrain-logloss:0.052036\teval-logloss:0.052036\n",
            "[35]\ttrain-logloss:0.049751\teval-logloss:0.049751\n",
            "[36]\ttrain-logloss:0.04775\teval-logloss:0.04775\n",
            "[37]\ttrain-logloss:0.045853\teval-logloss:0.045853\n",
            "[38]\ttrain-logloss:0.044015\teval-logloss:0.044015\n",
            "[39]\ttrain-logloss:0.042263\teval-logloss:0.042263\n",
            "[40]\ttrain-logloss:0.040649\teval-logloss:0.040649\n",
            "[41]\ttrain-logloss:0.039126\teval-logloss:0.039126\n",
            "[42]\ttrain-logloss:0.037377\teval-logloss:0.037377\n",
            "[43]\ttrain-logloss:0.036106\teval-logloss:0.036106\n",
            "[44]\ttrain-logloss:0.034941\teval-logloss:0.034941\n",
            "[45]\ttrain-logloss:0.033654\teval-logloss:0.033654\n",
            "[46]\ttrain-logloss:0.032528\teval-logloss:0.032528\n",
            "[47]\ttrain-logloss:0.031485\teval-logloss:0.031485\n",
            "[48]\ttrain-logloss:0.030389\teval-logloss:0.030389\n",
            "[49]\ttrain-logloss:0.029467\teval-logloss:0.029467\n",
            "[50]\ttrain-logloss:0.028545\teval-logloss:0.028545\n",
            "[51]\ttrain-logloss:0.027525\teval-logloss:0.027525\n",
            "[52]\ttrain-logloss:0.026555\teval-logloss:0.026555\n",
            "[53]\ttrain-logloss:0.025682\teval-logloss:0.025682\n",
            "[54]\ttrain-logloss:0.025004\teval-logloss:0.025004\n",
            "[55]\ttrain-logloss:0.024297\teval-logloss:0.024297\n",
            "[56]\ttrain-logloss:0.023574\teval-logloss:0.023574\n",
            "[57]\ttrain-logloss:0.022965\teval-logloss:0.022965\n",
            "[58]\ttrain-logloss:0.022488\teval-logloss:0.022488\n",
            "[59]\ttrain-logloss:0.021854\teval-logloss:0.021854\n",
            "[60]\ttrain-logloss:0.021316\teval-logloss:0.021316\n",
            "[61]\ttrain-logloss:0.020794\teval-logloss:0.020794\n",
            "[62]\ttrain-logloss:0.020218\teval-logloss:0.020218\n",
            "[63]\ttrain-logloss:0.019701\teval-logloss:0.019701\n",
            "[64]\ttrain-logloss:0.01918\teval-logloss:0.01918\n",
            "[65]\ttrain-logloss:0.018724\teval-logloss:0.018724\n",
            "[66]\ttrain-logloss:0.018325\teval-logloss:0.018325\n",
            "[67]\ttrain-logloss:0.017867\teval-logloss:0.017867\n",
            "[68]\ttrain-logloss:0.017598\teval-logloss:0.017598\n",
            "[69]\ttrain-logloss:0.017243\teval-logloss:0.017243\n",
            "[70]\ttrain-logloss:0.01688\teval-logloss:0.01688\n",
            "[71]\ttrain-logloss:0.016641\teval-logloss:0.016641\n",
            "[72]\ttrain-logloss:0.016287\teval-logloss:0.016287\n",
            "[73]\ttrain-logloss:0.015983\teval-logloss:0.015983\n",
            "[74]\ttrain-logloss:0.015658\teval-logloss:0.015658\n",
            "[75]\ttrain-logloss:0.015393\teval-logloss:0.015393\n",
            "[76]\ttrain-logloss:0.015151\teval-logloss:0.015151\n",
            "[77]\ttrain-logloss:0.01488\teval-logloss:0.01488\n",
            "[78]\ttrain-logloss:0.014637\teval-logloss:0.014637\n",
            "[79]\ttrain-logloss:0.014491\teval-logloss:0.014491\n",
            "[80]\ttrain-logloss:0.014185\teval-logloss:0.014185\n",
            "[81]\ttrain-logloss:0.014005\teval-logloss:0.014005\n",
            "[82]\ttrain-logloss:0.013772\teval-logloss:0.013772\n",
            "[83]\ttrain-logloss:0.013568\teval-logloss:0.013568\n",
            "[84]\ttrain-logloss:0.013414\teval-logloss:0.013414\n",
            "[85]\ttrain-logloss:0.013253\teval-logloss:0.013253\n",
            "[86]\ttrain-logloss:0.013109\teval-logloss:0.013109\n",
            "[87]\ttrain-logloss:0.012926\teval-logloss:0.012926\n",
            "[88]\ttrain-logloss:0.012714\teval-logloss:0.012714\n",
            "[89]\ttrain-logloss:0.012544\teval-logloss:0.012544\n",
            "[90]\ttrain-logloss:0.012353\teval-logloss:0.012353\n",
            "[91]\ttrain-logloss:0.012226\teval-logloss:0.012226\n",
            "[92]\ttrain-logloss:0.012065\teval-logloss:0.012065\n",
            "[93]\ttrain-logloss:0.011927\teval-logloss:0.011927\n",
            "[94]\ttrain-logloss:0.011821\teval-logloss:0.011821\n",
            "[95]\ttrain-logloss:0.011649\teval-logloss:0.011649\n",
            "[96]\ttrain-logloss:0.011482\teval-logloss:0.011482\n",
            "[97]\ttrain-logloss:0.01136\teval-logloss:0.01136\n",
            "[98]\ttrain-logloss:0.011239\teval-logloss:0.011239\n",
            "[99]\ttrain-logloss:0.011132\teval-logloss:0.011132\n",
            "[100]\ttrain-logloss:0.011002\teval-logloss:0.011002\n",
            "[101]\ttrain-logloss:0.010852\teval-logloss:0.010852\n",
            "[102]\ttrain-logloss:0.010755\teval-logloss:0.010755\n",
            "[103]\ttrain-logloss:0.010636\teval-logloss:0.010636\n",
            "[104]\ttrain-logloss:0.0105\teval-logloss:0.0105\n",
            "[105]\ttrain-logloss:0.010395\teval-logloss:0.010395\n",
            "[106]\ttrain-logloss:0.010305\teval-logloss:0.010305\n",
            "[107]\ttrain-logloss:0.010197\teval-logloss:0.010197\n",
            "[108]\ttrain-logloss:0.010072\teval-logloss:0.010072\n",
            "[109]\ttrain-logloss:0.01\teval-logloss:0.01\n",
            "[110]\ttrain-logloss:0.009914\teval-logloss:0.009914\n",
            "[111]\ttrain-logloss:0.009798\teval-logloss:0.009798\n",
            "[112]\ttrain-logloss:0.00971\teval-logloss:0.00971\n",
            "[113]\ttrain-logloss:0.009628\teval-logloss:0.009628\n",
            "[114]\ttrain-logloss:0.009558\teval-logloss:0.009558\n",
            "[115]\ttrain-logloss:0.009483\teval-logloss:0.009483\n",
            "[116]\ttrain-logloss:0.009416\teval-logloss:0.009416\n",
            "[117]\ttrain-logloss:0.009306\teval-logloss:0.009306\n",
            "[118]\ttrain-logloss:0.009228\teval-logloss:0.009228\n",
            "[119]\ttrain-logloss:0.009154\teval-logloss:0.009154\n",
            "[120]\ttrain-logloss:0.009118\teval-logloss:0.009118\n",
            "[121]\ttrain-logloss:0.009016\teval-logloss:0.009016\n",
            "[122]\ttrain-logloss:0.008972\teval-logloss:0.008972\n",
            "[123]\ttrain-logloss:0.008904\teval-logloss:0.008904\n",
            "[124]\ttrain-logloss:0.008837\teval-logloss:0.008837\n",
            "[125]\ttrain-logloss:0.008803\teval-logloss:0.008803\n",
            "[126]\ttrain-logloss:0.008709\teval-logloss:0.008709\n",
            "[127]\ttrain-logloss:0.008645\teval-logloss:0.008645\n",
            "[128]\ttrain-logloss:0.008613\teval-logloss:0.008613\n",
            "[129]\ttrain-logloss:0.008555\teval-logloss:0.008555\n",
            "[130]\ttrain-logloss:0.008463\teval-logloss:0.008463\n",
            "[131]\ttrain-logloss:0.008416\teval-logloss:0.008416\n",
            "[132]\ttrain-logloss:0.008385\teval-logloss:0.008385\n",
            "[133]\ttrain-logloss:0.0083\teval-logloss:0.0083\n",
            "[134]\ttrain-logloss:0.00827\teval-logloss:0.00827\n",
            "[135]\ttrain-logloss:0.008218\teval-logloss:0.008218\n",
            "[136]\ttrain-logloss:0.008157\teval-logloss:0.008157\n",
            "[137]\ttrain-logloss:0.008076\teval-logloss:0.008076\n",
            "[138]\ttrain-logloss:0.008047\teval-logloss:0.008047\n",
            "[139]\ttrain-logloss:0.007973\teval-logloss:0.007973\n",
            "[140]\ttrain-logloss:0.007946\teval-logloss:0.007946\n",
            "[141]\ttrain-logloss:0.007898\teval-logloss:0.007898\n",
            "[142]\ttrain-logloss:0.007872\teval-logloss:0.007872\n",
            "[143]\ttrain-logloss:0.007847\teval-logloss:0.007847\n",
            "[144]\ttrain-logloss:0.007776\teval-logloss:0.007776\n",
            "[145]\ttrain-logloss:0.007752\teval-logloss:0.007752\n",
            "[146]\ttrain-logloss:0.007728\teval-logloss:0.007728\n",
            "[147]\ttrain-logloss:0.007689\teval-logloss:0.007689\n",
            "[148]\ttrain-logloss:0.007621\teval-logloss:0.007621\n",
            "[149]\ttrain-logloss:0.007598\teval-logloss:0.007598\n",
            "[150]\ttrain-logloss:0.007575\teval-logloss:0.007575\n",
            "[151]\ttrain-logloss:0.007521\teval-logloss:0.007521\n",
            "[152]\ttrain-logloss:0.007498\teval-logloss:0.007498\n",
            "[153]\ttrain-logloss:0.007464\teval-logloss:0.007464\n",
            "[154]\ttrain-logloss:0.00741\teval-logloss:0.00741\n",
            "[155]\ttrain-logloss:0.007389\teval-logloss:0.007389\n",
            "[156]\ttrain-logloss:0.007367\teval-logloss:0.007367\n",
            "[157]\ttrain-logloss:0.007345\teval-logloss:0.007345\n",
            "[158]\ttrain-logloss:0.007323\teval-logloss:0.007323\n",
            "[159]\ttrain-logloss:0.007303\teval-logloss:0.007303\n",
            "[160]\ttrain-logloss:0.007282\teval-logloss:0.007282\n",
            "[161]\ttrain-logloss:0.007261\teval-logloss:0.007261\n",
            "[162]\ttrain-logloss:0.007241\teval-logloss:0.007241\n",
            "[163]\ttrain-logloss:0.007221\teval-logloss:0.007221\n",
            "[164]\ttrain-logloss:0.0072\teval-logloss:0.0072\n",
            "[165]\ttrain-logloss:0.00718\teval-logloss:0.00718\n",
            "[166]\ttrain-logloss:0.007161\teval-logloss:0.007161\n",
            "[167]\ttrain-logloss:0.007142\teval-logloss:0.007142\n",
            "[168]\ttrain-logloss:0.007122\teval-logloss:0.007122\n",
            "[169]\ttrain-logloss:0.007103\teval-logloss:0.007103\n",
            "[170]\ttrain-logloss:0.007084\teval-logloss:0.007084\n",
            "[171]\ttrain-logloss:0.007065\teval-logloss:0.007065\n",
            "[172]\ttrain-logloss:0.007047\teval-logloss:0.007047\n",
            "[173]\ttrain-logloss:0.007028\teval-logloss:0.007028\n",
            "[174]\ttrain-logloss:0.007009\teval-logloss:0.007009\n",
            "[175]\ttrain-logloss:0.006991\teval-logloss:0.006991\n",
            "[176]\ttrain-logloss:0.006973\teval-logloss:0.006973\n",
            "[177]\ttrain-logloss:0.006955\teval-logloss:0.006955\n",
            "[178]\ttrain-logloss:0.006937\teval-logloss:0.006937\n",
            "[179]\ttrain-logloss:0.00692\teval-logloss:0.00692\n",
            "[180]\ttrain-logloss:0.006901\teval-logloss:0.006901\n",
            "[181]\ttrain-logloss:0.006884\teval-logloss:0.006884\n",
            "[182]\ttrain-logloss:0.006866\teval-logloss:0.006866\n",
            "[183]\ttrain-logloss:0.006849\teval-logloss:0.006849\n",
            "[184]\ttrain-logloss:0.006831\teval-logloss:0.006831\n",
            "[185]\ttrain-logloss:0.006815\teval-logloss:0.006815\n",
            "[186]\ttrain-logloss:0.006798\teval-logloss:0.006798\n",
            "[187]\ttrain-logloss:0.006781\teval-logloss:0.006781\n",
            "[188]\ttrain-logloss:0.006764\teval-logloss:0.006764\n",
            "[189]\ttrain-logloss:0.006747\teval-logloss:0.006747\n",
            "[190]\ttrain-logloss:0.00673\teval-logloss:0.00673\n",
            "[191]\ttrain-logloss:0.006714\teval-logloss:0.006714\n",
            "[192]\ttrain-logloss:0.006698\teval-logloss:0.006698\n",
            "[193]\ttrain-logloss:0.006682\teval-logloss:0.006682\n",
            "[194]\ttrain-logloss:0.006666\teval-logloss:0.006666\n",
            "[195]\ttrain-logloss:0.00665\teval-logloss:0.00665\n",
            "[196]\ttrain-logloss:0.006634\teval-logloss:0.006634\n",
            "[197]\ttrain-logloss:0.006618\teval-logloss:0.006618\n",
            "[198]\ttrain-logloss:0.006603\teval-logloss:0.006603\n",
            "[199]\ttrain-logloss:0.006587\teval-logloss:0.006587\n",
            "[200]\ttrain-logloss:0.006572\teval-logloss:0.006572\n",
            "[201]\ttrain-logloss:0.006556\teval-logloss:0.006556\n",
            "[202]\ttrain-logloss:0.006542\teval-logloss:0.006542\n",
            "[203]\ttrain-logloss:0.006527\teval-logloss:0.006527\n",
            "[204]\ttrain-logloss:0.006512\teval-logloss:0.006512\n",
            "[205]\ttrain-logloss:0.006497\teval-logloss:0.006497\n",
            "[206]\ttrain-logloss:0.006483\teval-logloss:0.006483\n",
            "[207]\ttrain-logloss:0.006469\teval-logloss:0.006469\n",
            "[208]\ttrain-logloss:0.006455\teval-logloss:0.006455\n",
            "[209]\ttrain-logloss:0.00644\teval-logloss:0.00644\n",
            "[210]\ttrain-logloss:0.006427\teval-logloss:0.006427\n",
            "[211]\ttrain-logloss:0.006413\teval-logloss:0.006413\n",
            "[212]\ttrain-logloss:0.006399\teval-logloss:0.006399\n",
            "[213]\ttrain-logloss:0.006385\teval-logloss:0.006385\n",
            "[214]\ttrain-logloss:0.006372\teval-logloss:0.006372\n",
            "[215]\ttrain-logloss:0.006359\teval-logloss:0.006359\n",
            "[216]\ttrain-logloss:0.006345\teval-logloss:0.006345\n",
            "[217]\ttrain-logloss:0.006332\teval-logloss:0.006332\n",
            "[218]\ttrain-logloss:0.006319\teval-logloss:0.006319\n",
            "[219]\ttrain-logloss:0.006306\teval-logloss:0.006306\n",
            "[220]\ttrain-logloss:0.006293\teval-logloss:0.006293\n",
            "[221]\ttrain-logloss:0.00628\teval-logloss:0.00628\n",
            "[222]\ttrain-logloss:0.006268\teval-logloss:0.006268\n",
            "[223]\ttrain-logloss:0.006255\teval-logloss:0.006255\n",
            "[224]\ttrain-logloss:0.006242\teval-logloss:0.006242\n",
            "[225]\ttrain-logloss:0.00623\teval-logloss:0.00623\n",
            "[226]\ttrain-logloss:0.006218\teval-logloss:0.006218\n",
            "[227]\ttrain-logloss:0.006206\teval-logloss:0.006206\n",
            "[228]\ttrain-logloss:0.006194\teval-logloss:0.006194\n",
            "[229]\ttrain-logloss:0.006182\teval-logloss:0.006182\n",
            "[230]\ttrain-logloss:0.00617\teval-logloss:0.00617\n",
            "[231]\ttrain-logloss:0.006158\teval-logloss:0.006158\n",
            "[232]\ttrain-logloss:0.006147\teval-logloss:0.006147\n",
            "[233]\ttrain-logloss:0.006135\teval-logloss:0.006135\n",
            "[234]\ttrain-logloss:0.006124\teval-logloss:0.006124\n",
            "[235]\ttrain-logloss:0.006112\teval-logloss:0.006112\n",
            "[236]\ttrain-logloss:0.006101\teval-logloss:0.006101\n",
            "[237]\ttrain-logloss:0.00609\teval-logloss:0.00609\n",
            "[238]\ttrain-logloss:0.006079\teval-logloss:0.006079\n",
            "[239]\ttrain-logloss:0.006068\teval-logloss:0.006068\n",
            "[240]\ttrain-logloss:0.006057\teval-logloss:0.006057\n",
            "[241]\ttrain-logloss:0.006046\teval-logloss:0.006046\n",
            "[242]\ttrain-logloss:0.006036\teval-logloss:0.006036\n",
            "[243]\ttrain-logloss:0.006025\teval-logloss:0.006025\n",
            "[244]\ttrain-logloss:0.006014\teval-logloss:0.006014\n",
            "[245]\ttrain-logloss:0.006004\teval-logloss:0.006004\n",
            "[246]\ttrain-logloss:0.005993\teval-logloss:0.005993\n",
            "[247]\ttrain-logloss:0.005983\teval-logloss:0.005983\n",
            "[248]\ttrain-logloss:0.005972\teval-logloss:0.005972\n",
            "[249]\ttrain-logloss:0.005962\teval-logloss:0.005962\n",
            "[250]\ttrain-logloss:0.005952\teval-logloss:0.005952\n",
            "[251]\ttrain-logloss:0.005942\teval-logloss:0.005942\n",
            "[252]\ttrain-logloss:0.005932\teval-logloss:0.005932\n",
            "[253]\ttrain-logloss:0.005922\teval-logloss:0.005922\n",
            "[254]\ttrain-logloss:0.005912\teval-logloss:0.005912\n",
            "[255]\ttrain-logloss:0.005903\teval-logloss:0.005903\n",
            "[256]\ttrain-logloss:0.005893\teval-logloss:0.005893\n",
            "[257]\ttrain-logloss:0.005883\teval-logloss:0.005883\n",
            "[258]\ttrain-logloss:0.005874\teval-logloss:0.005874\n",
            "[259]\ttrain-logloss:0.005864\teval-logloss:0.005864\n",
            "[260]\ttrain-logloss:0.005855\teval-logloss:0.005855\n",
            "[261]\ttrain-logloss:0.005845\teval-logloss:0.005845\n",
            "[262]\ttrain-logloss:0.005836\teval-logloss:0.005836\n",
            "[263]\ttrain-logloss:0.005827\teval-logloss:0.005827\n",
            "[264]\ttrain-logloss:0.005818\teval-logloss:0.005818\n",
            "[265]\ttrain-logloss:0.005809\teval-logloss:0.005809\n",
            "[266]\ttrain-logloss:0.0058\teval-logloss:0.0058\n",
            "[267]\ttrain-logloss:0.005791\teval-logloss:0.005791\n",
            "[268]\ttrain-logloss:0.005782\teval-logloss:0.005782\n",
            "[269]\ttrain-logloss:0.005773\teval-logloss:0.005773\n",
            "[270]\ttrain-logloss:0.005764\teval-logloss:0.005764\n",
            "[271]\ttrain-logloss:0.005755\teval-logloss:0.005755\n",
            "[272]\ttrain-logloss:0.005747\teval-logloss:0.005747\n",
            "[273]\ttrain-logloss:0.005738\teval-logloss:0.005738\n",
            "[274]\ttrain-logloss:0.005729\teval-logloss:0.005729\n",
            "[275]\ttrain-logloss:0.005721\teval-logloss:0.005721\n",
            "[276]\ttrain-logloss:0.005712\teval-logloss:0.005712\n",
            "[277]\ttrain-logloss:0.005704\teval-logloss:0.005704\n",
            "[278]\ttrain-logloss:0.005696\teval-logloss:0.005696\n",
            "[279]\ttrain-logloss:0.005687\teval-logloss:0.005687\n",
            "[280]\ttrain-logloss:0.005679\teval-logloss:0.005679\n",
            "[281]\ttrain-logloss:0.005671\teval-logloss:0.005671\n",
            "[282]\ttrain-logloss:0.005663\teval-logloss:0.005663\n",
            "[283]\ttrain-logloss:0.005655\teval-logloss:0.005655\n",
            "[284]\ttrain-logloss:0.005646\teval-logloss:0.005646\n",
            "[285]\ttrain-logloss:0.005639\teval-logloss:0.005639\n",
            "[286]\ttrain-logloss:0.005631\teval-logloss:0.005631\n",
            "[287]\ttrain-logloss:0.005623\teval-logloss:0.005623\n",
            "[288]\ttrain-logloss:0.005615\teval-logloss:0.005615\n",
            "[289]\ttrain-logloss:0.005608\teval-logloss:0.005608\n",
            "[290]\ttrain-logloss:0.0056\teval-logloss:0.0056\n",
            "[291]\ttrain-logloss:0.005593\teval-logloss:0.005593\n",
            "[292]\ttrain-logloss:0.005585\teval-logloss:0.005585\n",
            "[293]\ttrain-logloss:0.005577\teval-logloss:0.005577\n",
            "[294]\ttrain-logloss:0.00557\teval-logloss:0.00557\n",
            "[295]\ttrain-logloss:0.005563\teval-logloss:0.005563\n",
            "[296]\ttrain-logloss:0.005556\teval-logloss:0.005556\n",
            "[297]\ttrain-logloss:0.005548\teval-logloss:0.005548\n",
            "[298]\ttrain-logloss:0.005541\teval-logloss:0.005541\n",
            "[299]\ttrain-logloss:0.005534\teval-logloss:0.005534\n",
            "[300]\ttrain-logloss:0.005526\teval-logloss:0.005526\n",
            "[301]\ttrain-logloss:0.005519\teval-logloss:0.005519\n",
            "[302]\ttrain-logloss:0.005512\teval-logloss:0.005512\n",
            "[303]\ttrain-logloss:0.005505\teval-logloss:0.005505\n",
            "[304]\ttrain-logloss:0.005498\teval-logloss:0.005498\n",
            "[305]\ttrain-logloss:0.005491\teval-logloss:0.005491\n",
            "[306]\ttrain-logloss:0.005484\teval-logloss:0.005484\n",
            "[307]\ttrain-logloss:0.005478\teval-logloss:0.005478\n",
            "[308]\ttrain-logloss:0.005471\teval-logloss:0.005471\n",
            "[309]\ttrain-logloss:0.005464\teval-logloss:0.005464\n",
            "[310]\ttrain-logloss:0.005457\teval-logloss:0.005457\n",
            "[311]\ttrain-logloss:0.00545\teval-logloss:0.00545\n",
            "[312]\ttrain-logloss:0.005444\teval-logloss:0.005444\n",
            "[313]\ttrain-logloss:0.005437\teval-logloss:0.005437\n",
            "[314]\ttrain-logloss:0.00543\teval-logloss:0.00543\n",
            "[315]\ttrain-logloss:0.005424\teval-logloss:0.005424\n",
            "[316]\ttrain-logloss:0.005417\teval-logloss:0.005417\n",
            "[317]\ttrain-logloss:0.005411\teval-logloss:0.005411\n",
            "[318]\ttrain-logloss:0.005404\teval-logloss:0.005404\n",
            "[319]\ttrain-logloss:0.005398\teval-logloss:0.005398\n",
            "[320]\ttrain-logloss:0.005392\teval-logloss:0.005392\n",
            "[321]\ttrain-logloss:0.005385\teval-logloss:0.005385\n",
            "[322]\ttrain-logloss:0.005379\teval-logloss:0.005379\n",
            "[323]\ttrain-logloss:0.005373\teval-logloss:0.005373\n",
            "[324]\ttrain-logloss:0.005366\teval-logloss:0.005366\n",
            "[325]\ttrain-logloss:0.00536\teval-logloss:0.00536\n",
            "[326]\ttrain-logloss:0.005354\teval-logloss:0.005354\n",
            "[327]\ttrain-logloss:0.005348\teval-logloss:0.005348\n",
            "[328]\ttrain-logloss:0.005342\teval-logloss:0.005342\n",
            "[329]\ttrain-logloss:0.005336\teval-logloss:0.005336\n",
            "[330]\ttrain-logloss:0.00533\teval-logloss:0.00533\n",
            "[331]\ttrain-logloss:0.005324\teval-logloss:0.005324\n",
            "[332]\ttrain-logloss:0.005318\teval-logloss:0.005318\n",
            "[333]\ttrain-logloss:0.005312\teval-logloss:0.005312\n",
            "[334]\ttrain-logloss:0.005306\teval-logloss:0.005306\n",
            "[335]\ttrain-logloss:0.0053\teval-logloss:0.0053\n",
            "[336]\ttrain-logloss:0.005294\teval-logloss:0.005294\n",
            "[337]\ttrain-logloss:0.005288\teval-logloss:0.005288\n",
            "[338]\ttrain-logloss:0.005283\teval-logloss:0.005283\n",
            "[339]\ttrain-logloss:0.005277\teval-logloss:0.005277\n",
            "[340]\ttrain-logloss:0.005271\teval-logloss:0.005271\n",
            "[341]\ttrain-logloss:0.005265\teval-logloss:0.005265\n",
            "[342]\ttrain-logloss:0.00526\teval-logloss:0.00526\n",
            "[343]\ttrain-logloss:0.005254\teval-logloss:0.005254\n",
            "[344]\ttrain-logloss:0.005249\teval-logloss:0.005249\n",
            "[345]\ttrain-logloss:0.005243\teval-logloss:0.005243\n",
            "[346]\ttrain-logloss:0.005237\teval-logloss:0.005237\n",
            "[347]\ttrain-logloss:0.005232\teval-logloss:0.005232\n",
            "[348]\ttrain-logloss:0.005227\teval-logloss:0.005227\n",
            "[349]\ttrain-logloss:0.005221\teval-logloss:0.005221\n",
            "[350]\ttrain-logloss:0.005216\teval-logloss:0.005216\n",
            "[351]\ttrain-logloss:0.00521\teval-logloss:0.00521\n",
            "[352]\ttrain-logloss:0.005205\teval-logloss:0.005205\n",
            "[353]\ttrain-logloss:0.0052\teval-logloss:0.0052\n",
            "[354]\ttrain-logloss:0.005195\teval-logloss:0.005195\n",
            "[355]\ttrain-logloss:0.005189\teval-logloss:0.005189\n",
            "[356]\ttrain-logloss:0.005184\teval-logloss:0.005184\n",
            "[357]\ttrain-logloss:0.005179\teval-logloss:0.005179\n",
            "[358]\ttrain-logloss:0.005174\teval-logloss:0.005174\n",
            "[359]\ttrain-logloss:0.005169\teval-logloss:0.005169\n",
            "[360]\ttrain-logloss:0.005164\teval-logloss:0.005164\n",
            "[361]\ttrain-logloss:0.005159\teval-logloss:0.005159\n",
            "[362]\ttrain-logloss:0.005153\teval-logloss:0.005153\n",
            "[363]\ttrain-logloss:0.005149\teval-logloss:0.005149\n",
            "[364]\ttrain-logloss:0.005144\teval-logloss:0.005144\n",
            "[365]\ttrain-logloss:0.005139\teval-logloss:0.005139\n",
            "[366]\ttrain-logloss:0.005134\teval-logloss:0.005134\n",
            "[367]\ttrain-logloss:0.005129\teval-logloss:0.005129\n",
            "[368]\ttrain-logloss:0.005124\teval-logloss:0.005124\n",
            "[369]\ttrain-logloss:0.005119\teval-logloss:0.005119\n",
            "[370]\ttrain-logloss:0.005114\teval-logloss:0.005114\n",
            "[371]\ttrain-logloss:0.00511\teval-logloss:0.00511\n",
            "[372]\ttrain-logloss:0.005105\teval-logloss:0.005105\n",
            "[373]\ttrain-logloss:0.0051\teval-logloss:0.0051\n",
            "[374]\ttrain-logloss:0.005096\teval-logloss:0.005096\n",
            "[375]\ttrain-logloss:0.005091\teval-logloss:0.005091\n",
            "[376]\ttrain-logloss:0.005086\teval-logloss:0.005086\n",
            "[377]\ttrain-logloss:0.005082\teval-logloss:0.005082\n",
            "[378]\ttrain-logloss:0.005077\teval-logloss:0.005077\n",
            "[379]\ttrain-logloss:0.005073\teval-logloss:0.005073\n",
            "[380]\ttrain-logloss:0.005068\teval-logloss:0.005068\n",
            "[381]\ttrain-logloss:0.005064\teval-logloss:0.005064\n",
            "[382]\ttrain-logloss:0.00506\teval-logloss:0.00506\n",
            "[383]\ttrain-logloss:0.005055\teval-logloss:0.005055\n",
            "[384]\ttrain-logloss:0.005051\teval-logloss:0.005051\n",
            "[385]\ttrain-logloss:0.005047\teval-logloss:0.005047\n",
            "[386]\ttrain-logloss:0.005042\teval-logloss:0.005042\n",
            "[387]\ttrain-logloss:0.005038\teval-logloss:0.005038\n",
            "[388]\ttrain-logloss:0.005034\teval-logloss:0.005034\n",
            "[389]\ttrain-logloss:0.005029\teval-logloss:0.005029\n",
            "[390]\ttrain-logloss:0.005025\teval-logloss:0.005025\n",
            "[391]\ttrain-logloss:0.005021\teval-logloss:0.005021\n",
            "[392]\ttrain-logloss:0.005017\teval-logloss:0.005017\n",
            "[393]\ttrain-logloss:0.005013\teval-logloss:0.005013\n",
            "[394]\ttrain-logloss:0.005009\teval-logloss:0.005009\n",
            "[395]\ttrain-logloss:0.005005\teval-logloss:0.005005\n",
            "[396]\ttrain-logloss:0.005\teval-logloss:0.005\n",
            "[397]\ttrain-logloss:0.004996\teval-logloss:0.004996\n",
            "[398]\ttrain-logloss:0.004992\teval-logloss:0.004992\n",
            "[399]\ttrain-logloss:0.004988\teval-logloss:0.004988\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "-> train으로 학습을 수행하면 반복 시 train-error 와 eval-logloss가 지속적으로 감소한다. \n",
        "\n",
        "이제 모델 학습이 완료 됐으면, 이를 이용해 테스트 데이터 세트에 예측을 수행해보자.\n",
        "\n",
        "**차이점) xgboost의 predict()는 예측 결괏값이 아니라 예측 결과를 추정할 수 있는 확률 값을 반환한다.**\n",
        "\n"
      ],
      "metadata": {
        "id": "O4DyPWEKaaKH"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "pred_probs = xgb_model.predict(dtest)\n",
        "print('predict()수행 결괏값을 10개만 표시, 예측 확률값으로 표시됨')\n",
        "print(np.round(pred_probs[:10],3))\n",
        "\n",
        "#예측 확률이 0.5보다 크면 1, 그렇지 않으면 0으로 예측값 결정해 리스트 객체인 preds에 저장\n",
        "preds= [1 if x>0.5 else 0 for x in pred_probs]\n",
        "print('예측값 10개만 표시:', preds[:0])"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "-XnOziZJac15",
        "outputId": "ace3a23b-3e3c-4f4a-ca01-c0474e3a0339"
      },
      "execution_count": 11,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "predict()수행 결괏값을 10개만 표시, 예측 확률값으로 표시됨\n",
            "[0.007 0.998 1.    0.998 0.978 0.023 1.    0.001 0.001 0.998]\n",
            "예측값 10개만 표시: []\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# 테스트 실제 레이블 값을 가지는 y_test와 예측 레이블인 preds를 인자로 입력한다. \n",
        "import numpy as np\n",
        "import pandas as pd\n",
        "import matplotlib.pyplot as plt\n",
        "%matplotlib inline\n",
        "\n",
        "from sklearn.model_selection import train_test_split\n",
        "from sklearn.metrics import accuracy_score, precision_score, recall_score, roc_auc_score\n",
        "from sklearn.metrics import f1_score, confusion_matrix, precision_recall_curve, roc_curve\n",
        "from sklearn.preprocessing import StandardScaler\n",
        "from sklearn.linear_model import LogisticRegression\n",
        "\n",
        "#get_clf_eval 함수 생성\n",
        "def get_clf_eval(y_test, pred=None, pred_proba=None):\n",
        "  confusion=confusion_matrix(y_test, pred)\n",
        "  accuracy=accuracy_score(y_test, pred)\n",
        "  precision=precision_score(y_test, pred)\n",
        "  recall=recall_score(y_test, pred)\n",
        "  f1=f1_score(y_test, pred)\n",
        "  roc_auc=roc_auc_score(y_test, pred_proba)\n",
        "  print('오차 행렬')\n",
        "  print(confusion)\n",
        "  print('정확도: {0:.4f}, 정밀도: {1:.4f}, 재현율: {2:.4f}, \\\n",
        "   F1: {3:.4f}, roc_auc:{4:.4f}'.format(accuracy, precision, recall, f1, roc_auc))\n",
        "\n",
        "get_clf_eval(y_test, preds, pred_probs)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 346
        },
        "id": "u78TUtukpyiT",
        "outputId": "4165bf65-8d06-4798-f14f-2fee24ed51a4"
      },
      "execution_count": 12,
      "outputs": [
        {
          "output_type": "error",
          "ename": "ValueError",
          "evalue": "ignored",
          "traceback": [
            "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
            "\u001b[0;31mValueError\u001b[0m                                Traceback (most recent call last)",
            "\u001b[0;32m<ipython-input-12-10981739f528>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[1;32m     24\u001b[0m    F1: {3:.4f}, roc_auc:{4:.4f}'.format(accuracy, precision, recall, f1, roc_auc))\n\u001b[1;32m     25\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 26\u001b[0;31m \u001b[0mget_clf_eval\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0my_test\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mpreds\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mpred_probs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
            "\u001b[0;32m<ipython-input-12-10981739f528>\u001b[0m in \u001b[0;36mget_clf_eval\u001b[0;34m(y_test, pred, pred_proba)\u001b[0m\n\u001b[1;32m     13\u001b[0m \u001b[0;31m#get_clf_eval 함수 생성\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     14\u001b[0m \u001b[0;32mdef\u001b[0m \u001b[0mget_clf_eval\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0my_test\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mpred\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mNone\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mpred_proba\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mNone\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 15\u001b[0;31m   \u001b[0mconfusion\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mconfusion_matrix\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0my_test\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mpred\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     16\u001b[0m   \u001b[0maccuracy\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0maccuracy_score\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0my_test\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mpred\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     17\u001b[0m   \u001b[0mprecision\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mprecision_score\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0my_test\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mpred\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/sklearn/metrics/_classification.py\u001b[0m in \u001b[0;36mconfusion_matrix\u001b[0;34m(y_true, y_pred, labels, sample_weight, normalize)\u001b[0m\n\u001b[1;32m    305\u001b[0m     \u001b[0;34m(\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;36m2\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;36m1\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;36m1\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    306\u001b[0m     \"\"\"\n\u001b[0;32m--> 307\u001b[0;31m     \u001b[0my_type\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0my_true\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0my_pred\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0m_check_targets\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0my_true\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0my_pred\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    308\u001b[0m     \u001b[0;32mif\u001b[0m \u001b[0my_type\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0;32min\u001b[0m \u001b[0;34m(\u001b[0m\u001b[0;34m\"binary\"\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m\"multiclass\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    309\u001b[0m         \u001b[0;32mraise\u001b[0m \u001b[0mValueError\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"%s is not supported\"\u001b[0m \u001b[0;34m%\u001b[0m \u001b[0my_type\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/sklearn/metrics/_classification.py\u001b[0m in \u001b[0;36m_check_targets\u001b[0;34m(y_true, y_pred)\u001b[0m\n\u001b[1;32m     82\u001b[0m     \u001b[0my_pred\u001b[0m \u001b[0;34m:\u001b[0m \u001b[0marray\u001b[0m \u001b[0;32mor\u001b[0m \u001b[0mindicator\u001b[0m \u001b[0mmatrix\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     83\u001b[0m     \"\"\"\n\u001b[0;32m---> 84\u001b[0;31m     \u001b[0mcheck_consistent_length\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0my_true\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0my_pred\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     85\u001b[0m     \u001b[0mtype_true\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mtype_of_target\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0my_true\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     86\u001b[0m     \u001b[0mtype_pred\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mtype_of_target\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0my_pred\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/sklearn/utils/validation.py\u001b[0m in \u001b[0;36mcheck_consistent_length\u001b[0;34m(*arrays)\u001b[0m\n\u001b[1;32m    332\u001b[0m         raise ValueError(\n\u001b[1;32m    333\u001b[0m             \u001b[0;34m\"Found input variables with inconsistent numbers of samples: %r\"\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 334\u001b[0;31m             \u001b[0;34m%\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0mint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0ml\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;32mfor\u001b[0m \u001b[0ml\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mlengths\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    335\u001b[0m         )\n\u001b[1;32m    336\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;31mValueError\u001b[0m: Found input variables with inconsistent numbers of samples: [114, 455]"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "이제 xgboost패키지에 내장된 시각화 기능을 수행해보자. \n",
        "plot_importance() API :피처의 중요도를 막대그래프 형식으로 나타냄. 기본 평가 지표로 f1스코어를 기반으로 해 각 피처의 중요도를 나타낸다. "
      ],
      "metadata": {
        "id": "8qVj_hIlaaOO"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "from xgboost import plot_importance\n",
        "import matplotlib.pyplot as plt\n",
        "%matplotlib inline\n",
        "\n",
        "fig, ax= plt.subplots(figsize=(10,12))\n",
        "plot_importance(xgb_model, ax=ax)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 747
        },
        "id": "muiekE7xaaSo",
        "outputId": "b509b3ca-4eaa-44a9-f997-ba2c0f6403fa"
      },
      "execution_count": 13,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<matplotlib.axes._subplots.AxesSubplot at 0x7fe5cd470d10>"
            ]
          },
          "metadata": {},
          "execution_count": 13
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 720x864 with 1 Axes>"
            ],
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAmEAAALJCAYAAAD8uvTIAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nOzde3xV5Zn//c/FKaJRqA0gQjGmVDkFglDTjk5NKrRQqdbWH2qxAyjDKM/LgAMKlhF8fDqFKigUprVoWx0pamtV5pHiqMVdHQutHJLSqcDYmilQykkQNg2ShOv3x96wN0gOHNa+d5Lv+/Xar+697rXudeVbSa6se+0dc3dEREREJLNahS5AREREpCVSEyYiIiISgJowERERkQDUhImIiIgEoCZMREREJAA1YSIiIiIBqAkTkRbBzL5pZo+HrkNE5AjT54SJSEPMrBLoAtSmbb7E3f9ymnOOc/fXTq+6psfM7gd6uvstoWsRkXB0JUxEGuvL7p6b9jjlBuxMMLM2Ic9/qppq3SJy5qkJE5FTZmYdzOyHZrbNzLaa2bfMrHVy7JNmtsLMdpvZLjP7iZl1TI49BfQA/n8zi5vZPWZWYmZbjpu/0syGJJ/fb2bPmdliM9sHjKnv/Ceo9X4zW5x8nm9mbmZjzWyzme0xs9vN7NNm9jsz22tmC9OOHWNmb5nZQjP7wMw2mNnVaeMXmtl/mNn7Zvaumf3jcedNr/t24JvAjcmvvSK531gze8fM9pvZn8zsn9LmKDGzLWY22cx2JL/esWnj7c1srpn9b7K+/zKz9smxz5jZr5NfU4WZlZzS/9kicsapCROR0/EEUAP0BAYCXwDGJccMmAVcCPQGPgHcD+Du3wD+TOrq2oONPN91wHNAR+AnDZy/MYqBTwE3AvOA6cAQoC8w0syuOm7fPwJ5wEzgeTM7Pzn2DLAl+bXeAHzbzD5fR90/BL4NPJv82gck99kBjADOA8YCj5jZZWlzXAB0ALoBtwH/ZmYfS47NAQYBfwecD9wDHDazbsAy4FvJ7VOAn5tZp5PISEQioiZMRBrrxeTVlL1m9qKZdQG+BExy9wPuvgN4BLgJwN3fdfdX3f1Dd98JPAxcVff0jbLS3V9098MkmpU6z99I/5+7H3T3V4ADwNPuvsPdtwJvkmjsjtgBzHP3and/FtgIXGNmnwCuAKYm5yoHHgf+4UR1u3vViQpx92Xu/kdP+BXwCvD3abtUAw8kz/8LIA5camatgFuBie6+1d1r3f3X7v4hcAvwC3f/RfLcrwKrk7mJSGC6N0FEGusr6TfRm9nlQFtgm5kd2dwK2Jwc7wLMJ9FInJsc23OaNWxOe35RfedvpO1pz6tO8Do37fVWP/adTP9L4srXhcD77r7/uLHBddR9QmY2nMQVtktIfB1nA+vTdtnt7jVpr/+WrC8POIvEVbrjXQT8HzP7ctq2tsDrDdUjItFTEyYip2oz8CGQd1xzcMS3AQcK3f19M/sKsDBt/Pi3Zh8g0XgAkLy36/hls/RjGjr/mdbNzCytEesB/AfwF+B8Mzs3rRHrAWxNO/b4r/WY12aWA/ycxNWzpe5ebWYvkljSbcgu4CDwSaDiuLHNwFPu/o8fOUpEgtNypIicEnffRmLJbK6ZnWdmrZI34x9ZcjyXxJLZB8l7k+4+bortQEHa603AWWZ2jZm1Bf4FyDmN859pnYEyM2trZv+HxH1uv3D3zcCvgVlmdpaZ9Sdxz9bieubaDuQnlxIB2pH4WncCNcmrYl9oTFHJpdkfAQ8n3yDQ2sw+m2zsFgNfNrMvJreflbzJv/vJf/kicqapCROR0/EPJBqIP5BYanwO6Joc+3+By4APSNwc/vxxx84C/iV5j9kUd/8AmEDifqqtJK6MbaF+9Z3/TPsNiZv4dwH/Ctzg7ruTYzcD+SSuir0AzGzg889+lvzf3Wa2NnkFrQz4KYmv4+skrrI11hQSS5dvA+8D3wFaJRvE60i8G3MniStjd6Pv/SJZQR/WKiLSADMbQ+KDZa8MXYuINB/6bUhEREQkADVhIiIiIgFoOVJEREQkAF0JExEREQmgSX5OWMeOHb1nz56hywjuwIEDnHPOOaHLyArKIkVZpCiLBOWQoixSlEVK1FmsWbNml7t/5M+FNckmrEuXLqxevTp0GcHFYjFKSkpCl5EVlEWKskhRFgnKIUVZpCiLlKizMLP/PdF2LUeKiIiIBKAmTERERCQANWEiIiIiAagJExEREQlATZiIiIhIAGrCRERERAJQEyYiIiISgJowERERkQDUhImIiIgEoCZMREREJAA1YSIiIiIBqAkTERERCUBNmIiIiEgAasJEREREAlATJiIiIhKAmjARERGRANSEiYiISJOyceNGioqKjj7OO+885s2bB8CCBQvo1asXffv25Z577glcaf3aRDm5mZUBdwB/AC4ELgOmu/uc5PhZwBtATrKW59x9ZpQ1iYiISNN26aWXUl5eDkBtbS3dunXj+uuv5/XXX2fp0qVUVFSQk5PDjh07Aldav0ibMGACMAQ4BFwEfOW48Q+Bz7t73MzaAv9lZsvdfVV9k1ZV15I/bVkkBTclkwtrGKMcAGWRTlmkKIsE5ZCiLFKyPYvK2dc0ar9f/vKXfPKTn+Siiy7i7rvvZtq0aeTk5ADQuXPnKEs8bZEtR5rZo0ABsBwY5e5vA9Xp+3hCPPmybfLhUdUkIiIizcszzzzDzTffDMCmTZt48803KS4u5qqrruLtt98OXF39zD26nsfMKoHB7r4r+fp+IH5kOTK5rTWwBugJ/Ju7T61jrvHAeIC8vE6DZsx7LLK6m4ou7WF7VegqsoOySFEWKcoiQTmkKIuUbM+isFuHBveprq7mhhtu4Mc//jHnn38+Y8eOZeDAgdx5551s2LCBBx54gCVLlmBm9c4Tj8fJzc09U6V/RGlp6Rp3H3z89qiXIxvk7rVAkZl1BF4ws37u/vsT7LcIWATQo6Cnz10fvPTgJhfWoBwSlEWKskhRFgnKIUVZpGR7FpWjShrcZ+nSpRQXF/PVr34VSNwrduedd1JaWkppaSlz5syhX79+dOrUqd55YrEYJSUNn+9My5r03X2vmb0ODAM+0oSla9+2NRsbuVbcnMVisUb9R9oSKIsUZZGiLBKUQ4qySGkOWTz99NNHlyIBvvKVr/D6669TWlrKpk2bOHToEHl5eQErrF/Qj6gws07JK2CYWXtgKLAhZE0iIiKS/Q4cOMCrr7569CoYwK233sqf/vQn+vXrx0033cSTTz7Z4FJkSBm5EmZmFwCrgfOAw2Y2CegDdAWeTN4X1gr4qbu/lImaREREpOk655xz2L179zHb2rVrx+LFiwNVdPIibcLcPT/tZfcT7PI7YGCUNYiIiIhkI31ivoiIiEgAasJEREREAlATJiIiIhKAmjARERGRANSEiYiIiASgJkxEREQkADVhIiIiIgGoCRMREREJQE2YiIiISABqwkREREQCUBMmIiIiEoCaMBEREZEA1ISJiIi0YHv37uWGG26gV69e9O7dm5UrV1JRUcFnP/tZCgsL+fKXv8y+fftCl9ksRdaEmVmZmb1jZm5mvzOz9Wb2azMbkLbPj8xsh5n9Pqo6REREpG4TJ05k2LBhbNiwgYqKCnr37s24ceOYPXs269ev5/rrr+ehhx4KXWazZO4ezcRmG4AhQA/gHXffY2bDgfvdvTi5z+eAOPDv7t6vsXP3KOjprUbOj6LsJmVyYQ1z17cJXUZWUBYpyiJFWSQoh5SWlkXl7GvqHIvFYgwcOJCioiL+9Kc/YWZHxzp06MDevXsxMzZv3swXv/hF/vCHP2Si5CBisRglJSWRzW9ma9x98PHbI7kSZmaPAgXAcqDY3fckh1YB3Y/s5+5vAO9HUYOIiIjU77333qNTp06MHTuWgQMHMm7cOA4cOEDfvn1ZunQpAD/72c/YvHlz4EqbpyivhFUCg919V9q2KUAvdx+Xti0feKmhK2FmNh4YD5CX12nQjHmPRVB109KlPWyvCl1FdlAWKcoiRVkkKIeUlpZFYbcOdY7F43G2bt3KhAkTWLBgAX369GHBggWcc845DBkyhAULFvDBBx9wxRVX8Pzzzx9typqjeDxObm5uZPOXlpae8EpYxpowMysFvgdc6e670/bLpxFNWDotRya0tMvq9VEWKcoiRVkkKIeUlpZFQ8uRvXr14jOf+QyVlZUAvPnmm8yePZtly5Yd3W/Tpk3ccsst/Pa3v4263GBCLUdm5L9EM+sPPA4MT2/ATlX7tq3ZWM9/WC1FLBajclRJ6DKygrJIURYpyiJBOaQoi2NdcMEFfOITn2Djxo1ceuml/PKXv6RPnz7s2LGDzp07c/jwYb71rW9x++23hy61WYr8IyrMrAfwPPANd98U9flERESk8RYsWMCoUaPo378/5eXlfPOb3+Tpp5/mkksuoVevXlx44YWMHTs2dJnNUiauhM0APg58L/nOi5ojl+TM7GmgBMgzsy3ATHf/YQZqEhEREaCoqIjVq1cfs23ixIlMnDgxUEUtR2RNmLvnJ5+OSz5OtM/NUZ1fREREJJvpE/NFREREAlATJiIiIhKAmjARERGRANSEiYiIiASgJkxEREQkADVhIiIiIgGoCRMREREJQE2YiIiISABqwkREREQCUBMmIiIiEoCaMBEREZEA1ISJiIgEsHfvXm644QZ69epF7969Wbly5dGxuXPnYmbs2rUrYIUStSBNmJmVmdk7ZvZzM1tpZh+a2ZQQtYiIiIQwceJEhg0bxoYNG6ioqKB3794AbN68mVdeeYUePXoErlCi1ibQeScAQ4BDwEXAVwLVISIiknEffPABb7zxBk888QQA7dq1o127dgDcddddPPjgg1x33XUBK5RMyHgTZmaPAgXAcuBH7v6ImV1zMnNUVdeSP21ZJPU1JZMLaxijHABlkU5ZpCiLBOWQksksKmfX/aPtvffeo1OnTowdO5aKigoGDRrE/Pnzee211+jWrRsDBgzISI0SVsabMHe/3cyGAaXursVuERFpcWpqali7di0LFiyguLiYiRMncv/99/PGG2/wyiuvhC5PMsTcPfMnNasEBh9pwszsfiDu7nPqOWY8MB4gL6/ToBnzHstApdmtS3vYXhW6iuygLFKURYqySFAOKZnMorBbhzrH3n//fSZMmMAzzzwDwO9+9zueeOIJ3nvvPXJycgDYuXMneXl5fP/73+f8888/4/XF43Fyc3PP+LxNUdRZlJaWrnH3wcdvD3VP2Elz90XAIoAeBT197vomU3pkJhfWoBwSlEWKskhRFgnKISWTWVSOKql3/JFHHqFr165ceumlxGIxrr76ah566KGj4/n5+axevZq8vLxI6ovFYpSU1F9jSxEqiyb5r7J929ZsrGetvaWIxWIN/iNvKZRFirJIURYJyiElm7JYsGABo0aN4tChQxQUFPDjH/84dEmSYUGbMDO7AFgNnAccNrNJQB933xeyLhERkagVFRWxevXqOscrKyszV4wEEaQJc/f8tJfdQ9QgIiIiEpI+MV9EREQkADVhIiIiIgGoCRMREREJQE2YiIiISABqwkREREQCUBMmIiIiEoCaMBEREZEA1ISJiIiIBKAmTERERCQANWEiIiIiAagJExEREQlATZiIiIhIAEH+gLeIiMiZkp+fz7nnnkvr1q1p06YNq1ev5sYbb2Tjxo0A7N27l44dO1JeXh64UpFjRdqEmVkZcAfQC1gPGLAfuMPdK5L73AWMAzy5z1h3PxhlXSIi0ry8/vrr5OXlHX397LPPHn0+efJkOnToEKIskXpFfSVsAjAE6AG84+57zGw4sAgoNrNuQBnQx92rzOynwE3AE/VNWlVdS/60ZdFW3gRMLqxhjHIAlEU6ZZGiLBKacg6Vs685rePdnZ/+9KesWLHiDFUkcuZEdk+YmT0KFADLgWJ335McWgV0T9u1DdDezNoAZwN/iaomERFpfsyML3zhCwwaNIhFixYdM/bmm2/SpUsXPvWpTwWqTqRu5u7RTW5WCQx2911p26YAvdx9XPL1ROBfgSrgFXcfVcdc44HxAHl5nQbNmPdYZHU3FV3aw/aq0FVkB2WRoixSlEVCU86hsFvDy4g7d+6kU6dO7NmzhylTplBWVsaAAQMAeOSRR+jWrRsjR44EIB6Pk5ubG2nNTYWySIk6i9LS0jXuPvj47Rm9Md/MSoHbgCuTrz8GXAdcDOwFfmZmt7j74uOPdfdFJJYx6VHQ0+eu13sKJhfWoBwSlEWKskhRFglNOYfKUSUntX9FRQXV1dWUlJRQU1PDjTfeyJo1a+jePbEAE4vFKCk5uTmbK2WREiqLjH1EhZn1Bx4HrnP33cnNQ4D33H2nu1cDzwN/l6maRESkaTtw4AD79+8/+vyVV16hX79+ALz22mv06tXraAMmkm0y8quRmfUg0WB9w903pQ39GfiMmZ1NYjnyamB1Q/O1b9uajad5s2ZzEIvFTvq3xOZKWaQoixRlkdCcc9i+fTvXX389ADU1NXz9619n2LBhADzzzDPcfPPNIcsTqVemrk/PAD4OfM/MAGrcfbC7/8bMngPWAjXAOpJLjiIiIg0pKCigoqLihGNPPPFEZosROUmRNmHunp98Oi75ONE+M4GZUdYhIiIikm30Z4tEREREAlATJiIiIhKAmjARERGRANSEiYiIiASgJkxEREQkADVhIiIiIgGoCRMREREJQE2YiIiISABqwkREREQCUBMmIiIiEoCaMBEREZEA1ISJiIiIBKAmTEREsl5+fj6FhYUUFRUxePDgo9sXLFhAr1696Nu3L/fcc0/ACkVOXpuoJjazMuAO4A/AhcBlwHR3n5O2TyWwH6gFatx98AmmEhER4fXXXycvL++Y10uXLqWiooKcnBx27NgRsDqRkxdZEwZMAIYAh4CLgK/UsV+pu+86mYmrqmvJn7bsNMtr+iYX1jBGOQDKIp2ySFEWCdmeQ+Xsa07puO9///tMmzaNnJwcADp37nwmyxKJXCTLkWb2KFAALAdGufvbQHUU5xIRkebPzPjCF77AoEGDWLRoEQCbNm3izTffpLi4mKuuuoq33347cJUiJ8fcPZqJE0uNg49c5TKz+4H4ccuR7wF7AAd+4O6L6plvPDAeIC+v06AZ8x6LpO6mpEt72F4VuorsoCxSlEWKskjI9hwKu3VocJ+dO3fSqVMn9uzZw5QpUygrK2PevHkMHDiQO++8kw0bNvDAAw+wZMkSzKzOeeLxOLm5uWey/CZLWaREnUVpaemaE91yFeVyZGNc6e5bzawz8KqZbXD3N060Y7JBWwTQo6Cnz10fuvTwJhfWoBwSlEWKskhRFgnZnkPlqJKT2r+iooLq6mouvfRS7rzzTkpLSyktLWXOnDn069ePTp061XlsLBajpOTkztdcKYuUUFkE/Vfp7luT/7vDzF4ALgdO2ISla9+2NRtP8R6C5iQWi530N6/mSlmkKIsUZZHQ1HM4cOAAhw8f5txzz+XAgQO88sorzJgxg9zcXF5//XVKS0vZtGkThw4dOubGfZFsF6wJM7NzgFbuvj/5/AvAA6HqERGR7LR9+3auv/56AGpqavj617/OsGHDOHToELfeeiv9+vWjXbt2PPnkk/UuRYpkm8ibMDO7AFgNnAccNrNJQB8gD3gh+Q+mDbDE3V+Ouh4REWlaCgoKqKio+Mj2du3asXjx4gAViZwZkTVh7p6f9rL7CXbZBwyI6vwiIiIi2UyfmC8iIiISgJowERERkQDUhImIiIgEoCZMREREJAA1YSIiIiIBqAkTERERCUBNmIiIiEgAasJEREREAlATJiIiIhKAmjARERGRANSEiYiIiASgJkxEREQkADVhIiJZrLa2loEDBzJixAgA3J3p06dzySWX0Lt3b7773e8GrlBETlWbqCY2szLgDqAXsB4wYD9wh7tXJPfpCDwO9AMcuNXdV0ZVk4hIUzN//nx69+7Nvn37AHjiiSfYvHkzGzZsoFWrVuzYsSNwhSJyqiJrwoAJwBCgB/COu+8xs+HAIqA4uc984GV3v8HM2gFnN2biqupa8qcti6LmJmVyYQ1jlAOgLNIpi5Rsz6Jy9jX1jm/ZsoVly5Yxffp0Hn74YQC+//3vs2TJElq1SixkdO7cOfI6RSQakSxHmtmjQAGwHCh29z3JoVVA9+Q+HYDPAT8EcPdD7r43inpERJqiSZMm8eCDDx5tuAD++Mc/8uyzzzJ48GCGDx/O//zP/wSsUERORyRXwtz9djMbBpS6+660odtINGYAFwM7gR+b2QBgDTDR3Q+caE4zGw+MB8jL68SMwpooSm9SurRP/KYvyiKdskjJ9ixisVidYytXrqS6upr9+/dTXl7O7t27icVi/O1vf2Pr1q3MmTOHN954g6997WsN3hcWj8frPVdLoixSlEVKqCzM3aOZ2KwSGHykCTOzUuB7wJXuvtvMBpO4MnaFu//GzOYD+9z9vobm7lHQ01uNnB9J3U3J5MIa5q6PckW56VAWKcoiJduzqG858t577+Wpp56iTZs2HDx4kH379vHVr36V1atXs3z5ci6++GLcnY4dO/LBBx/Ue55YLEZJSckZrr5pUhYpyiIl6izMbI27Dz5+e0a+O5lZfxI34A93993JzVuALe7+m+Tr54BpjZmvfdvWbGzgXoqWIBaLUTmqJHQZWUFZpCiLlKacxaxZs5g1axaQ+DrmzJnD4sWLmTZtGq+//joXX3wxv/rVr7jkkksCVyoipyryJszMegDPA99w901Htrv7X81ss5ld6u4bgauBP0Rdj4hIUzZt2jRGjRrFI488Qm5uLo8//njokkTkFGXiStgM4OPA98wMoCbtktydwE+S74z8EzA2A/WIiDQpJSUlR5dKOnbsyLJl2fuOTxFpvMiaMHfPTz4dl3ycaJ9y4CNrpCIiIiLNnT4xX0RERCQANWEiIiIiAagJExEREQlATZiIiIhIAGrCRERERAJQEyYiIiISgJowERERkQDUhImIiIgEoCZMREREJAA1YSIiIiIBqAkTERERCUBNmIhIYLW1tQwcOJARI0YAMGbMGC6++GKKioooKiqivLw8cIUiEoXI/oB3fcysDLgD6AH8T1otvYFO7v5+iLpEREKYP38+vXv3Zt++fUe3PfTQQ9xwww0BqxKRqIW6EjYBGOru57h7kbsXAfcCv1IDJiItyZYtW1i2bBnjxo0LXYqIZFjGr4SZ2aNAAbDczH7k7o8kh24Gnm7MHFXVteRPWxZViU3G5MIaxigHQFmkUxYp2ZBF5exr6h2fNGkSDz74IPv37z9m+/Tp03nggQe4+uqrmT17Njk5OVGWKSIBZPxKmLvfDvwFKD3SgJnZ2cAw4OeZrkdEJJSXXnqJzp07M2jQoGO2z5o1iw0bNvD222/z/vvv853vfCdQhSISJXP3zJ/UrBIY7O67kq9vBG5x9y/Xc8x4YDxAXl6nQTPmPZaJUrNal/awvSp0FdlBWaQoi5RsyKKwW4c6xx577DFeeeUVWrduzaFDh/jb3/7G3//93zN9+vSj+5SXl/Pss88ya9asU64hHo+Tm5t7ysc3J8oiRVmkRJ1FaWnpGncffPz2IDfmn8BNNLAU6e6LgEUAPQp6+tz12VJ6OJMLa1AOCcoiRVmkZEMWlaNK6hwrKUmNxWIx5syZw0svvcS2bdvo2rUr7s6LL77IVVdddcy+JysWi53W8c2JskhRFimhsgj+ndrMOgBXAbc09pj2bVuzsYH7LFqCWCxW7zf4lkRZpCiLlKaaxahRo9i5cyfuTlFREY8++mjokkQkAsGbMOB64BV3PxC6EBGRUEpKSo7+Jr5ixYqwxYhIRgRpwtw9P+35E8ATIeoQERERCUWfmC8iIiISgJowERERkQDUhImIiIgEoCZMREREJAA1YSIiIiIBqAkTERERCUBNmIiIiEgAasJEREREAlATJiIiIhKAmjARERGRANSEiYiIiASgJkxEREQkADVhIiIZUFtby8CBAxkxYsQx28vKysjNzQ1UlYiEFFkTZmZlZvaOmf3czFaa2YdmNiVt/FIzK0977DOzSVHVIyIS0vz58+ndu/cx21avXs2ePXsCVSQiobWJcO4JwBDgEHAR8JX0QXffCBQBmFlrYCvwQmMmrqquJX/asjNabFM0ubCGMcoBUBbplEVKprKonH1NveNbtmxh2bJlTJ8+nYcffhhIXBm7++67WbJkCS+80KhvfSLSzERyJczMHgUKgOXAKHd/G6iu55CrgT+6+/9GUY+ISEiTJk3iwQcfpFWr1LfchQsXcu2119K1a9eAlYlISJFcCXP3281sGFDq7rsacchNwNP17WBm44HxAHl5nZhRWHP6hTZxXdonftMXZZFOWaRkKotYLFbn2MqVK6murmb//v2Ul5eze/dunnvuOR5//HHmzZtHLBajtra23jlOVzwej3T+pkRZpCiLlFBZmLtHM7FZJTD4SBNmZvcDcXefc9x+7YC/AH3dfXtj5u5R0NNbjZx/ZgtugiYX1jB3fZQryk2HskhRFimZyqK+5ch7772Xp556ijZt2nDw4EH27dtHTk4OOTk5nHXWWQD8+c9/pqCggHfffTeS+mKxGCUlJZHM3dQoixRlkRJ1Fma2xt0HH789G94dORxY29gGTESkKZk1axZbtmyhsrKSZ555hs9//vPs2bOHv/71r1RWVlJZWcnZZ58dWQMmItkrG35dvpkGliKP175tazY2cCNsSxCLxagcVRK6jKygLFKURYqyEJFsFnkTZmYXAKuB84DDyY+h6OPu+8zsHGAo8E9R1yEiElpJSckJlzzi8XjmixGR4CJrwtw9P+1l9zr2OQB8PKoaRERERLJVNtwTJiIiItLiqAkTERERCUBNmIiIiEgAasJEREREAlATJiIiIhKAmjARERGRANSEiYiIiASgJkxEREQkADVhIiIiIgGoCRMREREJQE2YiIiISABqwkREREQCUBMmInICtbW1DBw4kBEjRgCwcOFCevbsiZmxa9euwNWJSHMQWRNmZmVm9o6ZuZn9zszWm9mvzWxAcvwTZva6mf3BzP7bzCZGVYuIyMmaP38+vXv3Pvr6iiuu4E+1V6QAACAASURBVLXXXuOiiy4KWJWINCdtIpx7AjAE6AG84+57zGw4sAgoBmqAye6+1szOBdaY2avu/oeGJq6qriV/2rIIS28aJhfWMEY5AMoinbJIqSuLytnX1Hvcli1bWLZsGdOnT+fhhx8GYODAgZHUKCItVyRXwszsUaAAWA4Uu/ue5NAqoDuAu29z97XJ5/uBd4BuUdQjInIyJk2axIMPPkirVrpjQ0SiE8mVMHe/3cyGAaXunn7zxG0kGrNjmFk+MBD4TV1zmtl4YDxAXl4nZhTWnMmSm6Qu7RO/6YuySKcsUurKIhaL1XnMypUrqa6uZv/+/ZSXl7N79+5j9j948CBvvfUWHTp0iKDiaMTj8Xq/5pZEWaQoi5RQWUS5HHkMMysl0YRdedz2XODnwCR331fX8e6+iMRSJj0Kevrc9RkrPWtNLqxBOSQoixRlkVJXFpWjSuo85j//8z9Zs2YNY8aM4eDBg+zbt4/HH3+cxYsXA3DWWWdxxRVXkJeXF1XZZ1wsFqOkpCR0GVlBWaQoi5RQWWTkO7WZ9QceB4a7++607W1JNGA/cffnGztf+7at2djAPR0tQSwWq/eHSUuiLFKURcqpZDFr1ixmzZp19Pg5c+YcbcBERM6kyG94MLMewPPAN9x9U9p2A35I4qb9h6OuQ0TkdHz3u9+le/fubNmyhf79+zNu3LjQJYlIE5eJK2EzgI8D30v0XdS4+2DgCuAbwHozK0/u+013/0UGahIRaVBJScnRJYqysjLKysrCFiQizUpkTZi75yefjks+jh//L8CiOr+IiIhINtP7r0VEREQCUBMmIiIiEoCaMBEREZEA1ISJiIiIBKAmTERERCQANWEiIiIiAagJExEREQlATZiIiIhIAGrCRERERAJQEyYiIiISgJowERERkQDUhImIiIgEoCZMRDLu4MGDXH755QwYMIC+ffsyc+ZMAMaMGcPFF19MUVERRUVFlJeXB65URCQ6baKc3MzKgDuAXsB6wID9wB3uXmFmZwFvADnJWp5z95lR1iQi4eXk5LBixQpyc3Oprq7myiuvZPjw4QA89NBD3HDDDYErFBGJXqRNGDABGAL0AN5x9z1mNhxYBBQDHwKfd/e4mbUF/svMlrv7qvomraquJX/asohLz36TC2sYoxwAZZEuG7KonH1NveNmRm5uLgDV1dVUV1djZpkoTUQka0S2HGlmjwIFwHKg2N33JIdWAd0BPCGe3N42+fCoahKR7FFbW0tRURGdO3dm6NChFBcXAzB9+nT69+/PXXfdxYcffhi4ShGR6Jh7dD2PmVUCg919V9q2KUAvdx+XfN0aWAP0BP7N3afWMdd4YDxAXl6nQTPmPRZZ3U1Fl/awvSp0FdlBWaRkQxaF3To0et94PM59991HWVkZ5513Hueffz7V1dXMnTuXCy+8kNGjR59yHfF4/OgVt5ZMOaQoixRlkRJ1FqWlpWvcffDx26NejjyGmZUCtwFXHtnm7rVAkZl1BF4ws37u/vvjj3X3RSSWMelR0NPnrs9o6VlpcmENyiFBWaRkQxaVo0pOav+1a9eye/duxo4de3Rbu3btmDNnDiUlJzdXulgsdlrHNxfKIUVZpCiLlFBZZOw7tZn1Bx4Hhrv77uPH3X2vmb0ODAM+0oSla9+2NRsbuOekJYjFYif9w665UhYpTSGLnTt30rZtWzp27EhVVRWvvvoqU6dOZdu2bXTt2hV358UXX6Rfv36hSxURiUxGmjAz6wE8D3zD3Telbe8EVCcbsPbAUOA7mahJRMLZtm0bo0ePpra2lsOHDzNy5EhGjBjB5z//eXbu3Im7U1RUxKOPPhq6VBGRyGTqStgM4OPA95LvgKpJro12BZ5M3hfWCvipu7+UoZpEJJD+/fuzbt26j2xfsWJFgGpERMKItAlz9/zk03HJx/HjvwMGRlmDiIiISDbSJ+aLiIiIBKAmTERERCQANWEiIiIiAagJExEREQlATZiIiIhIAGrCRERERAJQEyYiIiISgJowERERkQDUhImIiIgEoCZMREREJAA1YSIiIiIBqAkTkUgcPHiQyy+/nAEDBtC3b19mzpx5zHhZWRm5ubmBqhMRCS/SJszMyszsHTP7uZmtNLMPzWzKcfvcZWb/bWa/N7OnzeysKGsSkczIyclhxYoVVFRUUF5ezssvv8yqVasAWL16NXv27AlcoYhIWG0inn8CMAQ4BFwEfCV90My6AWVAH3evMrOfAjcBT9Q3aVV1LfnTlkVScFMyubCGMcoBUBbpMplF5exr6hwzs6NXuqqrq6mursbMqK2t5e6772bJkiW88MILGalTRCQbRXYlzMweBQqA5cAod38bqD7Brm2A9mbWBjgb+EtUNYlIZtXW1lJUVETnzp0ZOnQoxcXFLFy4kGuvvZauXbuGLk9EJKjIroS5++1mNgwodfdddeyz1czmAH8GqoBX3P2VqGoSkcxq3bo15eXl7N27l+uvv5433niDn/3sZ8RisdCliYgEZ+4e3eRmlcDgI02Ymd0PxN19TvL1x4CfAzcCe4GfAc+5++ITzDUeGA+Ql9dp0Ix5j0VWd1PRpT1srwpdRXZQFimZzKKwW4dG7/vkk08CsHTpUtq1awfAjh076Nq1Kz/5yU8iqS8ej+vmf5RDOmWRoixSos6itLR0jbsPPn571PeENWQI8J677wQws+eBvwM+0oS5+yJgEUCPgp4+d33o0sObXFiDckhQFimZzKJyVEmdYzt37qRt27Z07NiRqqoq7rvvPqZOncqPf/zjo/vk5uaydevWyOqLxWKUlNRdY0uhHFKURYqySAmVReifWn8GPmNmZ5NYjrwaWN3QQe3btmZjPTcEtxSxWKzeH4ItibJIyZYstm3bxujRo6mtreXw4cOMHDmSESNGhC5LRCRrZKQJM7MLSDRX5wGHzWwSiXdE/sbMngPWAjXAOpJXu0Skaevfvz/r1q2rd594PJ6hakREsk+kTZi756e97F7HPjOBmScaExEREWmu9In5IiIiIgGoCRMREREJQE2YiIiISABqwkREREQCUBMmIiIiEoCaMBEREZEA1ISJiIiIBKAmTERERCQANWEiIiIiAagJExEREQlATZiIiIhIAGrCRERERAJQEyYikTh48CCXX345AwYMoG/fvsycOfOY8bKyMnJzcwNVJyISXmRNmJmVmdk7ZuZm9jszW29mvzazAcnxS82sPO2xz8wmRVWPiGRWTk4OK1asoKKigvLycl5++WVWrVoFwOrVq9mzZ0/gCkVEwmoT4dwTgCFAD+Add99jZsOBRUCxu28EigDMrDWwFXihMRNXVdeSP21ZNFU3IZMLaxijHABlkS6TWVTOvqbOMTM7eqWrurqa6upqzIza2lruvvtulixZwgsvNOqfvIhIsxTJlTAzexQoAJaTaLiO/Mq7Cuh+gkOuBv7o7v8bRT0iEkZtbS1FRUV07tyZoUOHUlxczMKFC7n22mvp2rVr6PJERIIyd49mYrNKYLC770rbNgXo5e7jjtv3R8Bad19Yz3zjgfEAeXmdBs2Y91gkdTclXdrD9qrQVWQHZZGSySwKu3Vo1H7xeJz77ruPMWPG8PjjjzNv3jxat27N8OHDWb58eWT1xeNx3XeGckinLFKURUrUWZSWlq5x98HHb49yOfIYZlYK3AZcedz2dsC1wL31He/ui0gsZdKjoKfPXZ+x0rPW5MIalEOCskjJZBaVo0oave/atWvZu3cvO3fu5LbbbgPgww8/ZNy4cbz77ruR1BeLxSgpaXyNzZVySFEWKcoiJVQWGXl3pJn1Bx4HrnP33ccNDydxFWx7JmoRkczYuXMne/fuBaCqqopXX32VQYMG8de//pXKykoqKys5++yzI2vARESyXeS/LptZD+B54BvuvukEu9wMPH0yc7Zv25qN9dwQ3FLEYrGTuhLRnCmLlGzJYtu2bYwePZra2loOHz7MyJEjGTFiROiyRESyRibWLGYAHwe+Z2YANUfWRc3sHGAo8E8ZqENEMqh///6sW7eu3n3i8XiGqhERyT6RNWHunp98Oi75ONE+B0g0aCIiIiItij4xX0RERCQANWEiIiIiAagJExEREQlATZiIiIhIAGrCRERERAJQEyYiIiISgJowERERkQDUhImIiIgEoCZMREREJAA1YSIiIiIBqAkTERERCUBNmIiIiEgAasJEBICDBw9y+eWXM2DAAPr27cvMmTMBWLhwIT179sTM2LVrV+AqRUSaj0ibMDMrM7N3zMzN7Hdmtt7Mfm1mA47br7WZrTOzl6KsR0TqlpOTw4oVK6ioqKC8vJyXX36ZVatWccUVV/Daa69x0UUXhS5RRKRZaRPx/BOAIUAP4B1332Nmw4FFQHHafhOBd4DzGjNpVXUt+dOWnelam5zJhTWMUQ6AskhXVxaVs6+p9zgzIzc3F4Dq6mqqq6sxMwYOHBhJnSIiLV1kV8LM7FGgAFgOFLv7nuTQKqB72n7dgWuAx6OqRUQap7a2lqKiIjp37szQoUMpLi5u+CARETkl5u7RTW5WCQx2911p26YAvdx9XPL1c8As4FxgiruPqGOu8cB4gLy8ToNmzHsssrqbii7tYXtV6Cqyg7JIqSuLwm4dGj1HPB7nvvvuo6ysjIsvvhiAm266iR/84Ad06ND4eUKLx+NHr+61ZMohRVmkKIuUqLMoLS1d4+6Dj98e9XLkMcysFLgNuDL5egSww93XmFlJfce6+yISy5j0KOjpc9dntPSsNLmwBuWQoCxS6sqiclTJSc2zdu1adu/ezdixYwE466yzuOKKK8jLyzsTZWZELBajpKQkdBnBKYcUZZGiLFJCZZGxn1pm1p/EkuNwd9+d3HwFcK2ZfQk4CzjPzBa7+y31zdW+bWs2NnB/S0sQi8VO+gdrc6UsUk41i507d9K2bVs6duxIVVUVr776KlOnTj3zBYqICJChj6gwsx7A88A33H3Tke3ufq+7d3f3fOAmYEVDDZiIRGPbtm2UlpbSv39/Pv3pTzN06FBGjBjBd7/7Xbp3786WLVvo378/48aNC12qiEizkKkrYTOAjwPfMzOAmhOtjYpIOP3792fdunUf2V5WVkZZWVmAikREmrdIm7DkFS6AcclHffvGgFiU9YiIiIhkC31ivoiIiEgAasJEREREAlATJiIiIhKAmjARERGRANSEiYiIiASgJkxEREQkADVhIiIiIgGoCRMREREJQE2YiIiISABqwkREREQCUBMmIiIiEoCaMBEREZEA1ISJtCAHDx7k8ssvZ8CAAfTt25eZM2cC8N5771FcXEzPnj258cYbOXToUOBKRUSavyBNmJmVmdk7ZvaT5OtPm1mNmd0Qoh6RliInJ4cVK1ZQUVFBeXk5L7/8MqtWrWLq1KncddddvPvuu3zsYx/jhz/8YehSRUSavTaBzjsBGOLuW8ysNfAd4JXGHlxVXUv+tGWRFddUTC6sYYxyAJRFuieGnVPnmJmRm5sLQHV1NdXV1ZgZK1asYMmSJQCMHj2a+++/nzvuuCMj9YqItFQZvxJmZo8CBcByM7sLuBP4ObAj07WItES1tbUUFRXRuXNnhg4dyic/+Uk6duxImzaJ38m6d+/O1q1bA1cpItL8ZfxKmLvfbmbDgFIgB1iSfP7p+o4zs/HAeIC8vE7MKKyJutSs16V94gqQKIt08XicWCxW7z7z5s0jHo9z33330b17d6qqqo4es2PHDg4cONDgHE1BY7JoCZRDirJIURYpobIItRx5xDxgqrsfNrN6d3T3RcAigB4FPX3u+tClhze5sAblkKAsUp4Ydg4lJSWN2nft2rUcPHiQDz/8kCuvvJI2bdqwcuVKLrnkkkbPkc1isViz+DpOl3JIURYpyiIlVBahf2oNBp5JNmB5wJfMrMbdX6zvoPZtW7Nx9jWZqC+rxWIxKkeVhC4jKyiLlPp+m9u5cydt27alY8eOVFVV8eqrrzJ16lRKS0t57rnnuOmmm3jyySe57rrrMlewiEgL1agmzMw+CWxx9w/NrAToD/y7u+89nZO7+8Vp53gCeKmhBkxETt22bdsYPXo0tbW1HD58mJEjRzJixAj69OnDTTfdxL/8y78wcOBAbrvtttCliog0e429EvZzYLCZ9SSxJLiUxL1cX4qqMBE58/r378+6des+sr2goIDf/va3ASoSEWm5GtuEHXb3GjO7Hljg7gvM7KPfyRvJ3fNPsG3Mqc4nIiIi0tQ09iMqqs3sZmA08FJyW9toShIRERFp/hrbhI0FPgv8q7u/Z2YXA09FV5aIiIhI89ao5Uh3/4OZTQV6JF+/R+JT7kVERETkFDTqSpiZfRkoB15Ovi4ys/+IsjARERGR5qyxy5H3A5cDewHcvZzEnx4SERERkVPQ6Bvz3f2D47YdPtPFiIiIiLQUjf2Iiv82s68Drc3sU0AZ8OvoyhIRERFp3hp7JexOoC/wIYkPaf0AmBRVUSIiIiLNXYNXwsysNbDM3UuB6dGXJCIiItL8NXglzN1rgcNm1iED9YiIiIi0CI29JywOrDezV4EDRza6e1kkVYmIiIg0c429J+x54D7gDWBN2kNEzoDNmzdTWlpKnz596Nu3L/Pnzz86tmDBAnr16kXfvn255557AlYpIiJnUmM/Mf/JM3lSMysD7gDOA3KB95JDz7v7A2fyXCJNQZs2bZg7dy6XXXYZ+/fvZ9CgQQwdOpTt27ezdOlSKioqyMnJYceOHaFLFRGRM6RRTZiZvQf48dvd/VQ/sHUCMAToCUxx9xEnc3BVdS3505ad4qmbj8mFNYxRDkD2Z1E5+5p6x7t27UrXrl0BOPfcc+nduzdbt27lscceY9q0aeTk5ADQuXPnyGsVEZHMaOxy5GDg08nH3wPfBRafygnN7FESn7a/HBh4KnOINGeVlZWsW7eO4uJiNm3axJtvvklxcTFXXXUVb7/9dujyRETkDGlUE+buu9MeW919HlD/r/Z1z3U78BegFFgHfNbMKsxsuZn1PZU5RZqLeDzO1772NebNm8d5551HTU0N77//PqtWreKhhx5i5MiRuH/korSIiDRBjV2OvCztZSsSV8Ya+87K+qwFLnL3uJl9CXgR+FQdNYwHxgPk5XViRmHNGTh909alfWIZTrI/i1gs1uA+NTU13HvvvRQXF3P++ecTi8U4++yzKSgo4Fe/+hUAhw4dYunSpXTs2LHOeeLxeKPO1xIoiwTlkKIsUpRFSqgsGttIzU17XkPiRvqRp3tyd9+X9vwXZvY9M8tz910n2HcRsAigR0FPn7v+TPSATdvkwhqUQ0K2Z1E5qqTecXdn9OjRXHHFFcybN+/o9ltvvZW//OUvlJSUsGnTJlq1asV1112HmdU5VywWo6Sk/vO1FMoiQTmkKIsUZZESKovG/tS6zd3/lL7BzC4+3ZOb2QXAdnd3M7ucxFW23Q0d175tazY2cKNzSxCLxRr84d5SNPUs3nrrLZ566ikKCwspKioC4Nvf/ja33nort956K/369aNdu3Y8+eST9TZgIiLSdDS2CXsOuOwE2wad5vlvAO4wsxqgCrjJdcOLtEBXXnllnfd6LV58Su+BERGRLFdvE2ZmvUj84e4OZvbVtKHzgLNO9aTunp98ujD5EBEREWlRGroSdikwAugIfDlt+37gH6MqSkRERKS5q7cJc/elwFIz+6y7r8xQTSIiIiLNXmPvCVtnZv8PiaXJo8uQ7n5rJFWJiIiINHON/cT8p4ALgC8CvwK6k1iSFBEREZFT0NgmrKe73wccSP4x72uA4ujKEhEREWneGtuEVSf/d6+Z9QM6APpLwiIiIiKnqLH3hC0ys48B9wH/AeQCMyKrSkRERKSZa1QT5u6PJ5/+CiiIrhwRERGRlqFRy5Fm1sXMfmhmy5Ov+5jZbdGWJiIiItJ8NfaesCeA/wQuTL7eBEyKoiARERGRlqCxTVieu/8UOAzg7jVAbWRViYiIiDRzjW3CDpjZxwEHMLPPAB9EVpWIiIhIM9fYJuyfSbwr8pNm9hbw78CdkVUl0gxt3ryZ0tJS+vTpQ9++fZk/fz4A999/P926daOoqIiioiJ+8YtfBK5UREQyod53R5pZD3f/s7uvNbOrSPxBbwM2unt1fcc2MG8ZcAeJT+HfTGKZswaY5O7/darzimSzNm3aMHfuXC677DL279/PoEGDGDp0KAB33XUXU6ZMCVyhiIhkUkMfUfEicFny+bPu/rUzdN4JwBBgL4lP4Xcz6w/8FOjV0MFV1bXkT1t2hkppuiYX1jBGOQDZkUXl7GvqHe/atStdu3YF4Nxzz6V3795s3bo1E6WJiEgWamg50tKen5HPBzOzR5NzLQf+0d09OXQOyXvORJq7yspK1q1bR3Fx4q9/LVy4kP79+3PrrbeyZ8+ewNWJiEgmWKoHOsGg2Vp3v+z456d9UrNKYLC77zKz64FZJP4M0jXuvrKOY8YD4wHy8joNmjHvsTNRSpPWpT1srwpdRXbIhiwKu3Vo1H5VVVVMnDiRW265hc997nO8//77dOjQATPjRz/6Ebt372bq1KmnXEc8Hic3N/eUj29OlEWCckhRFinKIiXqLEpLS9e4++DjtzfUhNUCB0hcEWsP/O3IEODuft6pFJPehKVt+xwww92HNHR8j4Ke3mrk/FM5dbMyubCGuesb+5enmrdsyKKh5UiA6upqRowYwRe/+EX++Z//+aNzVFYyYsQIfv/7359yHbFYjJKSklM+vjlRFgnKIUVZpCiLlKizMLMTNmH1/tRy99aRVfTRc71hZgVmlpfenIk0F+7ObbfdRu/evY9pwLZt23b0XrEXXniBfv36hSpRREQyKOilAzPrCfwxeWP+ZUAOsLuh49q3bc3GRlx1aO5isRiVo0pCl5EVmkIWb731Fk899RSFhYUUFRUB8O1vf5unn36a8vJyzIz8/Hx+8IMfBK5UREQyIfRa1teAfzCzaqAKuNHrWx8VacKuvPJKTvSf95e+9KUA1YiISGhBmjB3z08+/U7yISIiItKiNPYT80VERETkDFITJiIiIhKAmjARERGRANSEiYiIiASgJkxEREQkADVhIiIiIgGoCRMREREJQE2YiIiISABqwkREREQCUBMmIiIiEoCaMBEREZEA1ISJiIiIBKAm7P+2d/9RdtX1vf+fb/NjGTMh3jJBICGMc/GSxCCBZAmsYpxR5EbhW8EvqxaiLQSapdEmtKZe0O83Qu+6l7QXBJprLx1AaK1Cb0Fbv+WHuIAxWomSACFImH65MldB+RHl14Rofr3vH+fgmQ7JJIHs8zmTeT7Wyso5e++z93teKzO82HvPORpxFi1axCGHHMLs2bN/s+zxxx/nxBNPZM6cOcybN48f/vCHBSeUJGnPKi1hEbE0IjZGREbEwxGxISK+HxHHDtpmQUT0RcTjEXFRlfPowHDuuedy5513/ptlf/3Xf80XvvAFHnroIf7sz/6Mz372s4WmkyRp74yteP9LgFOA6cDGzHw+Ij4I9AAnRMQY4EvAB4Angfsj4puZ+ehwO92ybQcdF91W8eit7zPHbOfcAzCH/pWnDbt+/vz59Pf3v2b5Sy+9BMCLL77I4YcfXsVokiTtN5WVsIi4BugE7gC+nJnfr69aA0yrP3438Hhm/rj+mpuBDwPDljBpqE9/+tP86Z/+KcuXL2fnzp18//vf3/OLJEkqqLISlpmfiIgFQHdmbhq06nxqxQxgKvDTQeueBE7Y1f4iYjGwGKC9fQorjtm+/4ceYd42oXY27EDT29u7x22efvppNm/e/Jttb7nlFs4//3ze+973cu+99/KRj3yEK664otpBW9TAwMBeZTgamEWNOTSYRYNZNJTKIjKzup1H9APzXi1hEdEN/BVwcmb+IiLOAhZk5gX19R8HTsjMTw+33+mdR+WbfvfqyuYeKT5zzHau2FD1FeXm29PlSID+/n5OP/10HnnkEQDa2tp4+eWXiQgyk8mTJ//m8uRo09vbS1dXV+kxWoJZ1JhDg1k0mEVD1VlExLrMnDd0edP+Cx4R7wKuAz6Ymb+oL34KOGLQZtPqy4Y1YdwY+vbiP9QHut7eXvoXdpUeoyUcfPDBfOc736Grq4t77rmHd7zjHaVHkiRpWE0pYRExHfg68PHM/NdBq+4H3hERb6dWvn4POKcZM2nkOvvss+nt7WXTpk1MmzaNSy+9lOXLl/OZz3yG7du38+Y3v5menp7SY0qSNKxmnQlbARwM/FVEAGzPzHmZuT0iPg18CxhD7Qb+HzVpJo1QN91002uW9fb2sm7dugLTSJL0+lRawjKzo/7wgvqfXW1zO3B7lXNIkiS1Gt8xX5IkqQBLmCRJUgGWMEmSpAIsYZIkSQVYwiRJkgqwhEmSJBVgCZMkSSrAEiZJklSAJUySJKkAS5gkSVIBljBJkqQCLGGSJEkFWMJGiCuvvJJ3vvOdzJ49m7PPPptf/epXpUeSJElvQJESFhFLI2JjRHwjIv6/iFgfET+KiPNKzNPqnnrqKf7yL/+StWvX8sgjj7Bjxw5uvvnm0mNJkqQ3YGyh4y4BTgF+H5icmf9XREwB+iLiq5m5dbgXb9m2g46LbmvGnE3Tv/K0Yddv376dLVu2MG7cOF555RUOP/zwJk0mSZKq0PQzYRFxDdAJ3AEkMCkiAmgDfglsb/ZMrW7q1KksX76c6dOnc9hhhzF58mROPfXU0mNJkqQ3IDKz+QeN6AfmAb8GvgnMACYBH83MXZ7iiojFwGKA9vYpc1dcdW1zhm2SY6ZO3u26l19+mS984QusWLGCtrY2LrnkEt773vdy0kkn0dbW1sQpW9fAwIBZ1JlFg1nUmEODWTSYRUPVWXR3d6/LzHlDl5e6HPmq/wg8BLwP+PfAtyPiu5n50tANM7MH6AGY3nlUXrGh9Oj7V//Crt2u+4d/+AeOO+44zjjjDAB+9rOfsWbNGtra2ujq2v3rrwsRzAAAIABJREFURpPe3l6zqDOLBrOoMYcGs2gwi4ZSWZRuMucBK7N2Ou7xiHiC2lmxHw73ognjxtC3h3uoDiTTp09nzZo1vPLKK0yYMIG7776befNeU6glSdIIUvotKn4CvB8gIt4GHA38uOhELeiEE07grLPO4vjjj+eYY45h586dLF68uPRYkiTpDSh9Juw/AzdGxAYggP+UmZsKz9SSLr30Ui699NLSY0iSpP2kSAnLzI5BT/01P0mSNOqUvhwpSZI0KlnCJEmSCrCESZIkFWAJkyRJKsASJkmSVIAlTJIkqQBLmCRJUgGWMEmSpAIsYZIkSQVYwiRJkgqwhEmSJBVgCSvghRde4KyzzmLGjBnMnDmT++67r/RIkiSpyYp8gHdELAU+CfwY2Ar8e+BXwKLMfKTETM20bNkyFixYwC233MLWrVt55ZVXSo8kSZKarEgJA5YApwDLgIHMPDMiZgBfAt6/pxdv2baDjotuq3jE169/5Wm7Xffiiy+yevVqbrzxRgDGjx/P+PHjmzSZJElqFU2/HBkR1wCdwB3Uytg9AJn5GNAREW9r9kzN9MQTTzBlyhTOO+88jjvuOC644AI2b95ceixJktRkTS9hmfkJ4GdAN3A18BGAiHg3cCQwrdkzNdP27dt54IEH+OQnP8mDDz7IxIkTWblyZemxJElSk0VmNv+gEf3APGr3g10NHAdsAGYAf5iZD+3iNYuBxQDt7VPmrrjq2qbNu6+OmTp5t+t++ctfsmTJEm6++WYAHn74Yb72ta+9riI2MDBAW1vb657zQGIWDWbRYBY15tBgFg1m0VB1Ft3d3esyc97Q5aXuCQMgM18CzgOIiACeoHaz/q627QF6AKZ3HpVXbCg6+rD6F3YNu/7KK6/ksMMO4+ijj6a3t5f3vOc9dHUN/5pd6e3tfV2vOxCZRYNZNJhFjTk0mEWDWTSUyqJok4mItwKvZOZW4AJgdb2YDWvCuDH0DXPze6tbtWoVCxcuZOvWrXR2dnLDDTeUHkmSJDVZ6dNJM4G/iYgEfgScX3ieppgzZw5r164tPYYkSSqoSAnLzI76w03AfygxgyRJUkm+Y74kSVIBljBJkqQCLGGSJEkFWMIkSZIKsIRJkiQVYAmTJEkqwBImSZJUgCVMkiSpAEuYJElSAZYwSZKkAixhkiRJBVjCJEmSCrCEFfDCCy9w1llnMWPGDGbOnMl9991XeiRJktRklZWwiFgaERsj4taIuC8ifh0Ry4ds89aIuCUiHqtve1JV87SSZcuWsWDBAh577DHWr1/PzJkzS48kSZKabGyF+14CnAJsBY4EztjFNlcDd2bmWRExHnjL3ux4y7YddFx0234bdH/rX3nabte9+OKLrF69mhtvvBGA8ePHM378+CZNJkmSWkUlZ8Ii4hqgE7gDWJiZ9wPbhmwzGZgPXA+QmVsz84Uq5mklTzzxBFOmTOG8887juOOO44ILLmDz5s2lx5IkSU0WmVnNjiP6gXmZuan+/BJgIDMvrz+fA/QAjwLHAuuAZZm5y0YSEYuBxQDt7VPmrrjq2krm3h+OmTp5t+v6+vpYsmQJq1atYtasWaxatYqJEyeyaNGifT7OwMAAbW1tb2TUA4ZZNJhFg1nUmEODWTSYRUPVWXR3d6/LzHlDl5csYfOANcBvZ+YPIuJq4KXM/H/3tO/pnUflm3736krm3h+Guxz59NNPc+KJJ9Lf3w/Ad7/7XVauXMltt+375dXe3l66urpe55QHFrNoMIsGs6gxhwazaDCLhqqziIhdlrCSvx35JPBkZv6g/vwW4PiC8zTFoYceyhFHHEFfXx8Ad999N7NmzSo8lSRJarYqb8wfVmY+HRE/jYijM7MPeD+1S5N7NGHcGPqGOdvU6latWsXChQvZunUrnZ2d3HDDDaVHkiRJTVZ5CYuIQ4G1wEHAzoi4EJiVmS8BfwR8tf6bkT8Gzqt6nlYwZ84c1q5dW3oMSZJUUGUlLDM7Bj2dtpttHgJec41UkiTpQOc75kuSJBVgCZMkSSrAEiZJklSAJUySJKkAS5gkSVIBljBJkqQCLGGSJEkFWMIkSZIKsIRJkiQVYAmTJEkqwBImSZJUgCVMkiSpAEtYAS+88AJnnXUWM2bMYObMmdx3332lR5IkSU1WaQmLiKURsTEibo2I+yLi1xGxfMg2CyKiLyIej4iLqpynVSxbtowFCxbw2GOPsX79embOnFl6JEmS1GRjK97/EuAUYCtwJHDG4JURMQb4EvAB4Eng/oj4ZmY+OtxOt2zbQcdFt1Uz8X7Qv/K03a578cUXWb16NTfeeCMA48ePZ/z48U2aTJIktYrKzoRFxDVAJ3AHsDAz7we2Ddns3cDjmfnjzNwK3Ax8uKqZWsETTzzBlClTOO+88zjuuOO44IIL2Lx5c+mxJElSk0VmVrfziH5gXmZuqj+/BBjIzMvrz88CFmTmBfXnHwdOyMxP72Jfi4HFAO3tU+auuOrayuZ+o46ZOnm36/r6+liyZAmrVq1i1qxZrFq1iokTJ7Jo0aJ9Ps7AwABtbW1vZNQDhlk0mEWDWdSYQ4NZNJhFQ9VZdHd3r8vMeUOXV305cr/JzB6gB2B651F5xYbWHb1/Yddu182YMYPLLruMJUuWADBmzBhWrlxJV9fuX7M7vb29r+t1ByKzaDCLBrOoMYcGs2gwi4ZSWZRuMk8BRwx6Pq2+bFgTxo2hb5j7rlrZoYceyhFHHEFfXx9HH300d999N7NmzSo9liRJarLSJex+4B0R8XZq5ev3gHPKjlS9VatWsXDhQrZu3UpnZyc33HBD6ZEkSVKTNaWERcShwFrgIGBnRFwIzMrMlyLi08C3gDHAlzPzR82YqaQ5c+awdu3a0mNIkqSCKi1hmdkx6Om03WxzO3B7lXNIkiS1Gt8xX5IkqQBLmCRJUgGWMEmSpAIsYZIkSQVYwiRJkgqwhEmSJBVgCZMkSSrAEiZJklSAJUySJKkAS5gkSVIBljBJkqQCLGGSJEkFVPoB3qNZR0cHkyZNYsyYMYwdO5a1a9eWHkmSJLWQIiUsIpYCnwQeAK4FrgLGAZsy870lZqrCvffeS3t7e+kxJElSCyp1JmwJcAowAHwfWJCZP4mIQ/bmxVu27aDjotuqnG+P+leeVvT4kiRpZGv6PWERcQ3QCdwBfAr4emb+BCAzn232PFWJCE499VTmzp1LT09P6XEkSVKLicxs/kEj+oF5wP9D7TLkO4FJwNWZ+be7ec1iYDFAe/uUuSuuurY5w+7GMVMnD7v+ueeeY8qUKTz//PMsX76cpUuXcuyxx+7XGQYGBmhra9uv+xypzKLBLBrMosYcGsyiwSwaqs6iu7t7XWbOG7q89I35Y4G5wPuBCcB9EbEmM/916IaZ2QP0AEzvPCqv2FB29P6FXXu97fr169m2bRtdXXv/mr3R29u73/c5UplFg1k0mEWNOTSYRYNZNJTKonQJexL4RWZuBjZHxGrgWOA1JWywCePG0NfC92Rt3ryZnTt3MmnSJDZv3sxdd93FihUrSo8lSZJaSOkS9k/Af4+IscB44ATgyrIjvXHPPPMMZ555JgDbt2/nnHPOYcGCBYWnkiRJraRoCcvMjRFxJ/AwsBO4LjMfKTnT/tDZ2cn69etLjyFJklpYkRKWmR2DHv834L+VmEOSJKkUP7ZIkiSpAEuYJElSAZYwSZKkAixhkiRJBVjCJEmSCrCESZIkFWAJkyRJKsASJkmSVIAlTJIkqQBLmCRJUgGWMEmSpAKKfoD3gayjo4NJkyYxZswYxo4dy9q1a0uPJEmSWkilJSwilgKfBB4FDgeOBz6fmZcP2W4MsBZ4KjNPr3KmZrr33ntpb28vPYYkSWpBVZ8JWwKcAmwFjgTO2M12y4CNwEF7s9Mt23bQcdFt+2XA16t/5WlFjy9Jkka2yu4Ji4hrgE7gDmBhZt4PbNvFdtOA04DrqpqlhIjg1FNPZe7cufT09JQeR5IktZjKzoRl5iciYgHQnZmbhtn0KuCzwKSqZinhe9/7HlOnTuXZZ5/lAx/4ADNmzGD+/Pmlx5IkSS0iMrO6nUf0A/NeLWERcQkw8Oo9YRFxOvChzFwSEV3A8t3dExYRi4HFAO3tU+auuOrayubeG8dMnbzX2954441MmDCBj370o/t1hoGBAdra2vbrPkcqs2gwiwazqDGHBrNoMIuGqrPo7u5el5nzhi4v/duRvw38TkR8CHgzcFBE/F1mfmzohpnZA/QATO88Kq/YUHb0/oVdu123efNmdu7cyaRJk9i8eTOf+9znWLFiBV1du3/N69Hb27vf9zlSmUWDWTSYRY05NJhFg1k0lMqiaJPJzIuBiwEGnQl7TQEbasK4MfS18I3xzzzzDGeeeSYA27dv55xzzmHBggWFp5IkSa2kKSUsIg6l9hYUBwE7I+JCYFZmvtSM4zdbZ2cn69evLz2GJElqYZWWsMzsGPR02h627QV6KxxHkiSpZfixRZIkSQVYwiRJkgqwhEmSJBVgCZMkSSrAEiZJklSAJUySJKkAS5gkSVIBljBJkqQCLGGSJEkFWMIkSZIKsIRJkiQVYAmTJEkqoNIP8B7NOjo6mDRpEmPGjGHs2LGsXbu29EiSJKmFVFbCImIp8EngUeBw4Hjg85l5eX39EcDfAm8DEujJzKurmqeEe++9l/b29tJjSJKkFlTlmbAlwCnAVuBI4Iwh67cDn8nMByJiErAuIr6dmY/uacdbtu2g46Lb9vvA+6J/5WlFjy9Jkka2Su4Ji4hrgE7gDmBhZt4PbBu8TWb+PDMfqD9+GdgITK1inhIiglNPPZW5c+fS09NTehxJktRiIjOr2XFEPzAvMzfVn18CDLx6OXLIth3AamB2Zr60m/0tBhYDtLdPmbviqmsrmXtvHTN18rDrn3vuOaZMmcLzzz/P8uXLWbp0Kccee+x+nWFgYIC2trb9us+RyiwazKLBLGrMocEsGsyioeosuru712XmvKHLi9+YHxFtwK3AhbsrYACZ2QP0AEzvPCqv2FB29P6FXXu97fr169m2bRtdXXv/mr3R29u73/c5UplFg1k0mEWNOTSYRYNZNJTKouhbVETEOGoF7KuZ+fWSs+xPmzdv5uWXX/7N47vuuovZs2cXnkqSJLWSYqeTIiKA64GNmfnFfXnthHFj6GvhG+OfeeYZzjzzTAC2b9/OOeecw4IFCwpPJUmSWknlJSwiDgXWAgcBOyPiQmAW8C7g48CGiHiovvnnMvP2qmeqWmdnJ+vXry89hiRJamGVlbDM7Bj0dNouNvkeEFUdX5IkqZX5sUWSJEkFWMIkSZIKsIRJkiQVYAmTJEkqwBImSZJUgCVMkiSpAEuYJElSAZYwSZKkAixhkiRJBVjCJEmSCrCESZIkFWAJkyRJKsAS9gbt2LGD4447jtNPP730KJIkaQSprIRFxNKI2BgRGREPR8SGiPh+RBw7aJv++vKHImJtVbNU6eqrr2bmzJmlx5AkSSPM2Ar3vQQ4BZgObMzM5yPig0APcMKg7bozc9O+7HjLth10XHTb/pt0N/pXnjbs+ieffJLbbruNz3/+83zxi1+sfB5JknTgqORMWERcA3QCdwAnZObz9VVrgGlVHLOECy+8kL/4i7/gTW/yqq4kSdo3kZnV7DiiH5g3+CxXRCwHZmTmBfXnTwDPAwn8dWb2DLO/xcBigPb2KXNXXHVtJXMPdszUybtdd99997FmzRr++I//mIceeoi///u/57LLLqt8psEGBgZoa2tr6jFblVk0mEWDWdSYQ4NZNJhFQ9VZdHd3r8vMeUOXV3k58t+IiG7gfODkQYtPzsynIuIQ4NsR8Vhmrt7V6+sFrQdgeudRecWG6kfvX9i123Xf+ta3WLduHeeeey6/+tWveOmll7juuuv4u7/7u8rnelVvby9dXbufcTQxiwazaDCLGnNoMIsGs2golUVTSlhEvAu4DvhgZv7i1eWZ+VT972cj4hvAu4FdlrDBJowbQ98e7teq2mWXXfabM1+9vb1cfvnlTS1gkiRpZKv8ZqaImA58Hfh4Zv7roOUTI2LSq4+BU4FHqp5HkiSpFTTjTNgK4GDgryICYHv9uujbgG/Ul40FvpaZdzZhnv2uq6vLU7qSJGmfVFbCMrOj/vCC+p+h638MHDt0uSRJ0mjgeytIkiQVYAmTJEkqwBImSZJUgCVMkiSpAEuYJElSAZYwSZKkAixhkiRJBVjCJEmSCrCESZIkFWAJkyRJKsASJkmSVIAlTJIkqQBLGPDTn/6U7u5uZs2axTvf+U6uvvrq0iNJkqQDXGUlLCKWRsTGiLg1Iu6LiF9HxPIh23w5Ip6NiEeqmmNvjB07liuuuIJHH32UNWvW8KUvfYlHH3205EiSJOkAN7bCfS8BTgG2AkcCZ+ximxuB/w787b7seMu2HXRcdNs+DdO/8rTdrjvssMM47LDDAJg0aRIzZ87kqaeeYtasWft0DEmSpL1VyZmwiLgG6ATuABZm5v3AtqHbZeZq4JdVzPB69ff38+CDD3LCCSeUHkWSJB3AIjOr2XFEPzAvMzfVn18CDGTm5UO26wD+OTNn72F/i4HFAO3tU+auuOrafZrnmKmT97jNli1bWLZsGR/72MeYP3/+Pu2/hIGBAdra2kqP0RLMosEsGsyixhwazKLBLBqqzqK7u3tdZs4burzKy5H7VWb2AD0A0zuPyis27Nvo/Qu7hl2/bds2Tj/9dD7xiU/wJ3/yJ693zKbq7e2lq6ur9BgtwSwazKLBLGrMocEsGsyioVQWI6aEDTZh3Bj6hrnHa19lJueffz4zZ84cMQVMkiSNbL5FBfAv//IvfOUrX+Gee+5hzpw5zJkzh9tvv730WJIk6QBW+ZmwiDgUWAscBOyMiAuBWZn5UkTcBHQB7RHxJPCFzLy+6pmGOvnkk6nq3jhJkqRdqayEZWbHoKfTdrPN2VUdX5IkqZV5OVKSJKkAS5gkSVIBljBJkqQCLGGSJEkFWMIkSZIKsIRJkiQVYAmTJEkqwBImSZJUgCVMkiSpAEuYJElSAZYwSZKkAkZVCVu0aBGHHHIIs2fPLj2KJEka5SorYRGxNCI2RsStEXFfRPw6IpYP2WZZRDwSET+KiAurmuVV5557LnfeeWfVh5EkSdqjsRXuewlwCrAVOBI4Y/DKiJgN/CHw7vo2d0bEP2fm43va8ZZtO+i46LbXLO9fedqwr5s/fz79/f17Ob4kSVJ1KjkTFhHXAJ3AHcDCzLwf2DZks5nADzLzlczcDnwH+EgV80iSJLWaSkpYZn4C+BnQnZlX7mazR4D3RMTBEfEW4EPAEVXMI0mS1GqqvBw5rMzcGBF/DtwFbAYeAnbsbvuIWAwsBmhvn8KKY7a/Zpve3t49Hvfpp59m8+bNe7VtqxsYGDggvo79wSwazKLBLGrMocEsGsyioVQWxUoYQGZeD1wPEBH/FXhymG17gB6A6Z1H5RUbXjt6/8KuPR6zv7+fiRMn0tW1521bXW9v7wHxdewPZtFgFg1mUWMODWbRYBYNpbIoWsIi4pDMfDYiplO7H+zEvXndhHFj6NvDTfi7cvbZZ9Pb28umTZuYNm0al156Keeff/4+70eSJOmNqryERcShwFrgIGBn/a0oZmXmS8CtEXEwtZv2P5WZL1Q5y0033VTl7iVJkvZaZSUsMzsGPZ22m23eU9XxJUmSWtmoesd8SZKkVmEJkyRJKsASJkmSVIAlTJIkqQBLmCRJUgGWMEmSpAIsYZIkSQVYwiRJkgqwhEmSJBVgCZMkSSrAEiZJklSAJUySJKmAUVXCFi1axCGHHMLs2bNLjyJJkka5IiUsIpZGxMaI+GpE/GVEPB4RD0fE8VUe99xzz+XOO++s8hCSJEl7ZWyh4y4BTgHeBfwR8A7gBOB/1P8e1pZtO+i46LbXLO9fedqwr5s/fz79/f37Pq0kSdJ+1vQSFhHXAJ3AHcB/AM7NzATWRMRbI+KwzPx5s+eSJElqpqj1nyYfNKIfmAfcCKzMzO/Vl98N/KfMXLuL1ywGFgO0t0+Zu+Kqa1+z32OmTt7jsZ9++mkuvvhibrjhhjfwFbSGgYEB2traSo/REsyiwSwazKLGHBrMosEsGqrOoru7e11mzhu6vNTlyH2WmT1AD8D0zqPyig2vHb1/Ydce99Pf38/EiRPp6trztq2ut7f3gPg69gezaDCLBrOoMYcGs2gwi4ZSWZQuYU8BRwx6Pq2+bFgTxo2hbw/3f0mSJLWy0m9R8U3g96PmRODFKu8HO/vssznppJPo6+tj2rRpXH/99VUdSpIkaVilz4TdDnwIeBx4BTivyoPddNNNVe5ekiRprxUpYZnZMejpp0rMIEmSVFLpy5GSJEmjkiVMkiSpAEuYJElSAZYwSZKkAixhkiRJBVjCJEmSCrCESZIkFWAJkyRJKsASJkmSVIAlTJIkqQBLmCRJUgGWMEmSpAJGVQlbtGgRhxxyCLNnzy49iiRJGuUqK2ERsTQiNkZERsTDEbEhIr4fEccO2mZZRDwSET+KiAurmuVV5557LnfeeWfVh5EkSdqjsRXuewlwCjAd2JiZz0fEB4Ee4ISImA38IfBuYCtwZ0T8c2Y+vqcdb9m2g46LbnvN8v6Vpw37uvnz59Pf37+vX4ckSdJ+V8mZsIi4BugE7gBOyMzn66vWANPqj2cCP8jMVzJzO/Ad4CNVzCNJktRqIjOr2XFEPzAvMzcNWrYcmJGZF0TETOCfgJOALcDdwNrM/KPd7G8xsBigvX3K3BVXXfuabY6ZOnmPcz399NNcfPHF3HDDDfv8NbWagYEB2traSo/REsyiwSwazKLGHBrMosEsGqrOoru7e11mzhu6vMrLkf9GRHQD5wMnA2Tmxoj4c+AuYDPwELBjd6/PzB5qlzKZ3nlUXrHhtaP3L+za4xz9/f1MnDiRrq49b9vqent7D4ivY38wiwazaDCLGnNoMIsGs2golUVTSlhEvAu4DvhgZv7i1eWZeT1wfX2b/wo8uTf7mzBuDH17uP9LkiSplVX+FhURMR34OvDxzPzXIesOGbTNR4CvVTnL2WefzUknnURfXx/Tpk3j+uuvr/JwkiRJu9WMM2ErgIOBv4oIgO2DroveGhEHA9uAT2XmC1UOctNNN1W5e0mSpL1WWQnLzI76wwvqf3a1zXuqOr4kSVIrG1XvmC9JktQqLGGSJEkFWMIkSZIKsIRJkiQVYAmTJEkqwBImSZJUgCVMkiSpAEuYJElSAZYwSZKkAixhkiRJBVjCJEmSCrCESZIkFWAJkyRJKsASJkmSVIAlTJIkqQBLmCRJUgGRmaVn2GcR8TLQV3qOFtAObCo9RIswiwazaDCLGnNoMIsGs2ioOosjM3PK0IVjKzxglfoyc17pIUqLiLXmUGMWDWbRYBY15tBgFg1m0VAqCy9HSpIkFWAJkyRJKmCklrCe0gO0CHNoMIsGs2gwixpzaDCLBrNoKJLFiLwxX5IkaaQbqWfCJEmSRjRLmCRJUgEjqoRFxIKI6IuIxyPiotLzNFNEfDkino2IRwYt+62I+HZE/P/1v/9dyRmbJSKOiIh7I+LRiPhRRCyrLx9VeUTEmyPihxGxvp7DpfXlb4+IH9S/T/4+IsaXnrVZImJMRDwYEf9cfz4qs4iI/ojYEBEPRcTa+rJR9f0BEBFvjYhbIuKxiNgYESeN0hyOrv9bePXPSxFx4WjMAiAi/rj+M/ORiLip/rO0yM+KEVPCImIM8CXgg8As4OyImFV2qqa6EVgwZNlFwN2Z+Q7g7vrz0WA78JnMnAWcCHyq/m9htOXxa+B9mXksMAdYEBEnAn8OXJmZRwHPA+cXnLHZlgEbBz0fzVl0Z+acQe99NNq+PwCuBu7MzBnAsdT+bYy6HDKzr/5vYQ4wF3gF+AajMIuImAosBeZl5mxgDPB7FPpZMWJKGPBu4PHM/HFmbgVuBj5ceKamyczVwC+HLP4w8Df1x38DnNHUoQrJzJ9n5gP1xy9T+8E6lVGWR9YM1J+Oq/9J4H3ALfXlB3wOr4qIacBpwHX158EozWI3RtX3R0RMBuYD1wNk5tbMfIFRlsMuvB/4X5n5vxm9WYwFJkTEWOAtwM8p9LNiJJWwqcBPBz1/sr5sNHtbZv68/vhp4G0lhykhIjqA44AfMArzqF9+ewh4Fvg28L+AFzJze32T0fR9chXwWWBn/fnBjN4sErgrItZFxOL6stH2/fF24Dnghvol6usiYiKjL4ehfg+4qf541GWRmU8BlwM/oVa+XgTWUehnxUgqYRpG1t5rZFS930hEtAG3Ahdm5kuD142WPDJzR/0SwzRqZ4tnFB6piIg4HXg2M9eVnqVFnJyZx1O7feNTETF/8MpR8v0xFjge+B+ZeRywmSGX20ZJDr9Rv8/pd4B/GLputGRRv+/tw9RK+uHARF57q0/TjKQS9hRwxKDn0+rLRrNnIuIwgPrfzxaep2kiYhy1AvbVzPx6ffGozaN+meVe4CTgrfXT7DB6vk9+G/idiOindqvC+6jdDzQas3j1//bJzGep3fvzbkbf98eTwJOZ+YP681uolbLRlsNgHwQeyMxn6s9HYxanAE9k5nOZuQ34OrWfH0V+VoykEnY/8I76bzCMp3ZK9ZuFZyrtm8Af1B//AfBPBWdpmvq9PtcDGzPzi4NWjao8ImJKRLy1/ngC8AFq98fdC5xV3+yAzwEgMy/OzGmZ2UHtZ8M9mbmQUZhFREyMiEmvPgZOBR5hlH1/ZObTwE8j4uj6ovcDjzLKchjibBqXImF0ZvET4MSIeEv9vyWv/rso8rNiRL1jfkR8iNp9H2OAL2fmfyk8UtNExE1AF9AOPAN8AfhH4H8C04H/DfxuZg69ef+AExEnA98FNtC4/+dz1O4LGzV5RMS7qN1AOoba/1D9z8z8s4jopHY26LeAB4GPZeavy03aXBHRBSzPzNNHYxb1r/lkSrntAAACVElEQVQb9adjga9l5n+JiIMZRd8fABExh9ovaowHfgycR/17hVGUA/ymkP8E6MzMF+vLRt2/CYD62/l8lNpv2j8IXEDtHrCm/6wYUSVMkiTpQDGSLkdKkiQdMCxhkiRJBVjCJEmSCrCESZIkFWAJkyRJKmDsnjeRpNYXETuovW3Jq87IzP5C40jSHvkWFZIOCBExkJltTTze2EGfNSdJ+8zLkZJGhYg4LCJWR8RDEfFIRLynvnxBRDwQEesj4u76st+KiH+MiIcjYk39jXGJiEsi4isR8S/AV+qfWnBrRNxf//PbBb9ESSOMlyMlHSgmRMRD9cdPZOaZQ9afA3yr/u7xY4C3RMQU4FpgfmY+ERG/Vd/2UuDBzDwjIt4H/C0wp75uFrUPyN4SEV8DrszM70XEdOBbwMwKv0ZJBxBLmKQDxZbMnDPM+vuBL9c//P0fM/Oh+kccrc7MJwAGfWTLycD/XV92T0QcHBEH1dd9MzO31B+fAsyqfQQdAAdFRFtmDuy/L0vSgcoSJmlUyMzVETEfOA24MSK+CDz/Ona1edDjNwEnZuav9seMkkYX7wmTNCpExJHAM5l5LbUPdT4eWAPMj4i317d59XLkd4GF9WVdwKbMfGkXu70L+KNBxxjuTJwk/RueCZM0WnQBfxoR24AB4Pcz87mIWAx8PSLeBDwLfAC4hNqly4eBV4A/2M0+lwJfqm83FlgNfKLSr0LSAcO3qJAkSSrAy5GSJEkFWMIkSZIKsIRJkiQVYAmTJEkqwBImSZJUgCVMkiSpAEuYJElSAf8HMEjh7qBLN9kAAAAASUVORK5CYII=\n"
          },
          "metadata": {
            "needs_background": "light"
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "결정 트리에서 보여준 트리 기반 규칙 구조도 xgboost에서 시각화 할 수 있다. \n",
        "\n",
        "* to_graphviz() API를 이용하면 주피터 노트북에 바로 규칙 트리 구조를 그릴 수 있다. \n",
        "\n",
        "* cv() API : 데이터 세트에 대한 교차검증 수행후 최적 파라미터를 구할 수 있는 방법"
      ],
      "metadata": {
        "id": "ZmBhlymxaaVz"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "xgboost.cv(params, dtrain, num_boost_round=10, nfold=3, stratified=False, folds=None, \\\n",
        "           metrics=(), obj=None, feval=None, maximize=False, early_stopping_rounds=None, fpreproc=None, as_pandas=True,\\\n",
        "           verbose_eval=None, show_stdv=True, seed=0, callbacks=None, shuffle=True)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 364
        },
        "id": "oqjYj3jOaaZe",
        "outputId": "c2223beb-b85b-4e7c-c3d8-bc8eb73497ad"
      },
      "execution_count": 14,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "   train-logloss-mean  train-logloss-std  test-logloss-mean  test-logloss-std\n",
              "0            0.610037           0.000064           0.617447          0.004961\n",
              "1            0.541817           0.000202           0.555140          0.008989\n",
              "2            0.483717           0.000955           0.503529          0.012238\n",
              "3            0.434676           0.001311           0.459869          0.015795\n",
              "4            0.392310           0.001686           0.423058          0.016788\n",
              "5            0.355129           0.001390           0.391187          0.019785\n",
              "6            0.322530           0.001908           0.361119          0.019378\n",
              "7            0.294159           0.001618           0.337140          0.021086\n",
              "8            0.269083           0.002068           0.314903          0.019972\n",
              "9            0.246208           0.001940           0.295012          0.019237"
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-1a329f66-4ddb-4c76-8e0b-1eb27ca68087\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>train-logloss-mean</th>\n",
              "      <th>train-logloss-std</th>\n",
              "      <th>test-logloss-mean</th>\n",
              "      <th>test-logloss-std</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>0.610037</td>\n",
              "      <td>0.000064</td>\n",
              "      <td>0.617447</td>\n",
              "      <td>0.004961</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>0.541817</td>\n",
              "      <td>0.000202</td>\n",
              "      <td>0.555140</td>\n",
              "      <td>0.008989</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>0.483717</td>\n",
              "      <td>0.000955</td>\n",
              "      <td>0.503529</td>\n",
              "      <td>0.012238</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>0.434676</td>\n",
              "      <td>0.001311</td>\n",
              "      <td>0.459869</td>\n",
              "      <td>0.015795</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>0.392310</td>\n",
              "      <td>0.001686</td>\n",
              "      <td>0.423058</td>\n",
              "      <td>0.016788</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>5</th>\n",
              "      <td>0.355129</td>\n",
              "      <td>0.001390</td>\n",
              "      <td>0.391187</td>\n",
              "      <td>0.019785</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>6</th>\n",
              "      <td>0.322530</td>\n",
              "      <td>0.001908</td>\n",
              "      <td>0.361119</td>\n",
              "      <td>0.019378</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>7</th>\n",
              "      <td>0.294159</td>\n",
              "      <td>0.001618</td>\n",
              "      <td>0.337140</td>\n",
              "      <td>0.021086</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>8</th>\n",
              "      <td>0.269083</td>\n",
              "      <td>0.002068</td>\n",
              "      <td>0.314903</td>\n",
              "      <td>0.019972</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9</th>\n",
              "      <td>0.246208</td>\n",
              "      <td>0.001940</td>\n",
              "      <td>0.295012</td>\n",
              "      <td>0.019237</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-1a329f66-4ddb-4c76-8e0b-1eb27ca68087')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "        \n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "      \n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-1a329f66-4ddb-4c76-8e0b-1eb27ca68087 button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-1a329f66-4ddb-4c76-8e0b-1eb27ca68087');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n",
              "  "
            ]
          },
          "metadata": {},
          "execution_count": 14
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "* params : 부스터 파라미터\n",
        "* dtrain : 학습데이터\n",
        "* num_boost_round : 부스팅 반복 횟수\n",
        "* nfold : cv폴드 개수.\n",
        "* stratified : cv 수행 시 층화 표본 추출 수행 여부\n",
        "* metrics : cv수행시 모니터링할 성능 평가 지표\n",
        "* early_stopping_rounds : 조기 중단을 활성화시킴. 반복 횟수 지정.\n",
        "\n",
        "\n",
        "\n",
        "---\n",
        "\n",
        "### 사이킷런 래퍼 XGBoost의 개요 및 적용\n",
        "\n",
        "사이킷런의 기본 estimator을 그대로 상속해 만들었기 대문에 다른 estimator와 동일하게 fit()과 predict()만으로 학습과 예측이 가능하고 gridsearchcv, pipeline 등 사이킷런읟 ㅏ른 유틸리티를 그대로 사용할 수 있기 때문에 기존의 다른 머신러닝 알고리즘으로 만들어놓은 프로그램이 있더라도 알고리즘 클래스만 xgboost 래퍼 클래스로 바꾸면 사용 가능하다.\n",
        "\n",
        "분류를 위한 래퍼클래스 : XGBClassifier\n",
        "\n",
        "회귀를 위한 래퍼 클래스 : XGBRegressor\n",
        "\n",
        "<파이썬 래퍼 xgboost와 파라미터 차이>\n",
        "* eta -> learning_rate\n",
        "* sub_sample -> subsample\n",
        "* lambda -> reg_lambda\n",
        "* alpha -> reg_alpha\n",
        "* n_estimators가 num_boost_round보다 우선순위\n"
      ],
      "metadata": {
        "id": "N8duFM3naadH"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "#사이킷런 래퍼 XGBoost 클래스인 XGBClassifier 임포트\n",
        "from xgboost import XGBClassifier\n",
        "xgb_wrapper = XGBClassifier(n_estimators=400, learning_rate=0.1, max_depth=3)\n",
        "xgb_wrapper.fit(X_train, y_train)\n",
        "w_preds = xgb_wrapper.predict(X_test)\n",
        "w_pred_proba= xgb_wrapper.predict_proba(X_test)[:,1]"
      ],
      "metadata": {
        "id": "B_yC9bmDaagW"
      },
      "execution_count": 15,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [],
      "metadata": {
        "id": "1eZP6yEbaaj6"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "get_clf_eval(y_test, w_preds, w_pred_proba)"
      ],
      "metadata": {
        "id": "M1ekIGuPaanr",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "f75c260d-ddaf-47ef-c310-44457ea67650"
      },
      "execution_count": 16,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "오차 행렬\n",
            "[[35  2]\n",
            " [ 1 76]]\n",
            "정확도: 0.9737, 정밀도: 0.9744, 재현율: 0.9870,    F1: 0.9806, roc_auc:0.9951\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "-> 파이썬 래퍼 xgboost와 동일한 평가 결과가 나옴.\n",
        "\n",
        "여기서도 조기 중단을 수행할 수 있는데 \n",
        "\n",
        "early_stopping_rounds : 평가 지표가 향상될 수 있는 반복 횟수 정의\n",
        "\n",
        "eval_metric : 조기 중단을 위한 평가 지표\n",
        "\n",
        "eval_set : 성능 평가를 수행할 데이터세트\n",
        "\n",
        "* 성능 평가를 수행할 데이터 세트는 학습 데이터가 아니라 별도의 데이터 세트여야 한다. 데스트 데이터 세트는 학습 시에는 완전히 알려지지 않은 데이터 세트를 사용해야 한다. 평가에 테스트 데이터 세트를 사용하면 학습 시에 미리 참고가 되어 과적합할 수 있기 때문."
      ],
      "metadata": {
        "id": "Ai6P3uKuaaq-"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "#but 이 예제는 데이터 세트의 크기가 작아 테스트 데이터를 평가용으로 사용함.\n",
        "\n",
        "from xgboost import XGBClassifier\n",
        "\n",
        "xgb_wrapper = XGBClassifier(n_estimators=400, learning_rate=0.1, max_depth=3)\n",
        "evals = [(X_test, y_test)]\n",
        "xgb_wrapper.fit(X_train, y_train, early_stopping_rounds=100, eval_metric=\"logloss\", eval_set = evals, verbose=True)\n",
        "\n",
        "ws100_preds = xgb_wrapper.predict(X_test)\n",
        "ws100_pred_proba= xgb_wrapper.predict_proba(X_test)[:,1]"
      ],
      "metadata": {
        "id": "fH_ARRKMaaug",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "c11be0f2-2f71-4791-a042-0ce09f4a6742"
      },
      "execution_count": 17,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "[0]\tvalidation_0-logloss:0.61352\n",
            "Will train until validation_0-logloss hasn't improved in 100 rounds.\n",
            "[1]\tvalidation_0-logloss:0.547842\n",
            "[2]\tvalidation_0-logloss:0.494247\n",
            "[3]\tvalidation_0-logloss:0.447986\n",
            "[4]\tvalidation_0-logloss:0.409109\n",
            "[5]\tvalidation_0-logloss:0.374977\n",
            "[6]\tvalidation_0-logloss:0.345714\n",
            "[7]\tvalidation_0-logloss:0.320529\n",
            "[8]\tvalidation_0-logloss:0.29721\n",
            "[9]\tvalidation_0-logloss:0.277991\n",
            "[10]\tvalidation_0-logloss:0.260302\n",
            "[11]\tvalidation_0-logloss:0.246037\n",
            "[12]\tvalidation_0-logloss:0.231556\n",
            "[13]\tvalidation_0-logloss:0.22005\n",
            "[14]\tvalidation_0-logloss:0.208572\n",
            "[15]\tvalidation_0-logloss:0.199993\n",
            "[16]\tvalidation_0-logloss:0.190118\n",
            "[17]\tvalidation_0-logloss:0.181818\n",
            "[18]\tvalidation_0-logloss:0.174729\n",
            "[19]\tvalidation_0-logloss:0.167657\n",
            "[20]\tvalidation_0-logloss:0.158202\n",
            "[21]\tvalidation_0-logloss:0.154725\n",
            "[22]\tvalidation_0-logloss:0.148947\n",
            "[23]\tvalidation_0-logloss:0.143308\n",
            "[24]\tvalidation_0-logloss:0.136344\n",
            "[25]\tvalidation_0-logloss:0.132778\n",
            "[26]\tvalidation_0-logloss:0.127912\n",
            "[27]\tvalidation_0-logloss:0.125263\n",
            "[28]\tvalidation_0-logloss:0.119978\n",
            "[29]\tvalidation_0-logloss:0.116412\n",
            "[30]\tvalidation_0-logloss:0.114502\n",
            "[31]\tvalidation_0-logloss:0.112572\n",
            "[32]\tvalidation_0-logloss:0.11154\n",
            "[33]\tvalidation_0-logloss:0.108681\n",
            "[34]\tvalidation_0-logloss:0.106681\n",
            "[35]\tvalidation_0-logloss:0.104207\n",
            "[36]\tvalidation_0-logloss:0.102962\n",
            "[37]\tvalidation_0-logloss:0.100576\n",
            "[38]\tvalidation_0-logloss:0.098683\n",
            "[39]\tvalidation_0-logloss:0.096444\n",
            "[40]\tvalidation_0-logloss:0.095869\n",
            "[41]\tvalidation_0-logloss:0.094242\n",
            "[42]\tvalidation_0-logloss:0.094715\n",
            "[43]\tvalidation_0-logloss:0.094272\n",
            "[44]\tvalidation_0-logloss:0.093894\n",
            "[45]\tvalidation_0-logloss:0.094184\n",
            "[46]\tvalidation_0-logloss:0.09402\n",
            "[47]\tvalidation_0-logloss:0.09236\n",
            "[48]\tvalidation_0-logloss:0.093012\n",
            "[49]\tvalidation_0-logloss:0.091272\n",
            "[50]\tvalidation_0-logloss:0.090051\n",
            "[51]\tvalidation_0-logloss:0.089605\n",
            "[52]\tvalidation_0-logloss:0.089577\n",
            "[53]\tvalidation_0-logloss:0.090703\n",
            "[54]\tvalidation_0-logloss:0.089579\n",
            "[55]\tvalidation_0-logloss:0.090357\n",
            "[56]\tvalidation_0-logloss:0.091587\n",
            "[57]\tvalidation_0-logloss:0.091527\n",
            "[58]\tvalidation_0-logloss:0.091986\n",
            "[59]\tvalidation_0-logloss:0.091951\n",
            "[60]\tvalidation_0-logloss:0.091939\n",
            "[61]\tvalidation_0-logloss:0.091461\n",
            "[62]\tvalidation_0-logloss:0.090311\n",
            "[63]\tvalidation_0-logloss:0.089407\n",
            "[64]\tvalidation_0-logloss:0.089719\n",
            "[65]\tvalidation_0-logloss:0.089743\n",
            "[66]\tvalidation_0-logloss:0.089622\n",
            "[67]\tvalidation_0-logloss:0.088734\n",
            "[68]\tvalidation_0-logloss:0.088621\n",
            "[69]\tvalidation_0-logloss:0.089739\n",
            "[70]\tvalidation_0-logloss:0.089981\n",
            "[71]\tvalidation_0-logloss:0.089782\n",
            "[72]\tvalidation_0-logloss:0.089584\n",
            "[73]\tvalidation_0-logloss:0.089533\n",
            "[74]\tvalidation_0-logloss:0.088748\n",
            "[75]\tvalidation_0-logloss:0.088597\n",
            "[76]\tvalidation_0-logloss:0.08812\n",
            "[77]\tvalidation_0-logloss:0.088396\n",
            "[78]\tvalidation_0-logloss:0.088736\n",
            "[79]\tvalidation_0-logloss:0.088153\n",
            "[80]\tvalidation_0-logloss:0.087577\n",
            "[81]\tvalidation_0-logloss:0.087412\n",
            "[82]\tvalidation_0-logloss:0.08849\n",
            "[83]\tvalidation_0-logloss:0.088575\n",
            "[84]\tvalidation_0-logloss:0.08807\n",
            "[85]\tvalidation_0-logloss:0.087641\n",
            "[86]\tvalidation_0-logloss:0.087416\n",
            "[87]\tvalidation_0-logloss:0.087611\n",
            "[88]\tvalidation_0-logloss:0.087065\n",
            "[89]\tvalidation_0-logloss:0.08727\n",
            "[90]\tvalidation_0-logloss:0.087161\n",
            "[91]\tvalidation_0-logloss:0.086962\n",
            "[92]\tvalidation_0-logloss:0.087166\n",
            "[93]\tvalidation_0-logloss:0.087067\n",
            "[94]\tvalidation_0-logloss:0.086592\n",
            "[95]\tvalidation_0-logloss:0.086116\n",
            "[96]\tvalidation_0-logloss:0.087139\n",
            "[97]\tvalidation_0-logloss:0.086768\n",
            "[98]\tvalidation_0-logloss:0.086694\n",
            "[99]\tvalidation_0-logloss:0.086547\n",
            "[100]\tvalidation_0-logloss:0.086498\n",
            "[101]\tvalidation_0-logloss:0.08641\n",
            "[102]\tvalidation_0-logloss:0.086288\n",
            "[103]\tvalidation_0-logloss:0.086258\n",
            "[104]\tvalidation_0-logloss:0.086835\n",
            "[105]\tvalidation_0-logloss:0.086767\n",
            "[106]\tvalidation_0-logloss:0.087321\n",
            "[107]\tvalidation_0-logloss:0.087304\n",
            "[108]\tvalidation_0-logloss:0.08728\n",
            "[109]\tvalidation_0-logloss:0.087298\n",
            "[110]\tvalidation_0-logloss:0.087289\n",
            "[111]\tvalidation_0-logloss:0.088002\n",
            "[112]\tvalidation_0-logloss:0.087936\n",
            "[113]\tvalidation_0-logloss:0.087843\n",
            "[114]\tvalidation_0-logloss:0.088066\n",
            "[115]\tvalidation_0-logloss:0.087649\n",
            "[116]\tvalidation_0-logloss:0.087298\n",
            "[117]\tvalidation_0-logloss:0.087799\n",
            "[118]\tvalidation_0-logloss:0.087751\n",
            "[119]\tvalidation_0-logloss:0.08768\n",
            "[120]\tvalidation_0-logloss:0.087626\n",
            "[121]\tvalidation_0-logloss:0.08757\n",
            "[122]\tvalidation_0-logloss:0.087547\n",
            "[123]\tvalidation_0-logloss:0.087156\n",
            "[124]\tvalidation_0-logloss:0.08767\n",
            "[125]\tvalidation_0-logloss:0.087737\n",
            "[126]\tvalidation_0-logloss:0.088275\n",
            "[127]\tvalidation_0-logloss:0.088309\n",
            "[128]\tvalidation_0-logloss:0.088266\n",
            "[129]\tvalidation_0-logloss:0.087886\n",
            "[130]\tvalidation_0-logloss:0.088861\n",
            "[131]\tvalidation_0-logloss:0.088675\n",
            "[132]\tvalidation_0-logloss:0.088743\n",
            "[133]\tvalidation_0-logloss:0.089218\n",
            "[134]\tvalidation_0-logloss:0.089179\n",
            "[135]\tvalidation_0-logloss:0.088821\n",
            "[136]\tvalidation_0-logloss:0.088512\n",
            "[137]\tvalidation_0-logloss:0.08848\n",
            "[138]\tvalidation_0-logloss:0.088386\n",
            "[139]\tvalidation_0-logloss:0.089145\n",
            "[140]\tvalidation_0-logloss:0.08911\n",
            "[141]\tvalidation_0-logloss:0.088765\n",
            "[142]\tvalidation_0-logloss:0.088678\n",
            "[143]\tvalidation_0-logloss:0.088389\n",
            "[144]\tvalidation_0-logloss:0.089271\n",
            "[145]\tvalidation_0-logloss:0.089238\n",
            "[146]\tvalidation_0-logloss:0.089139\n",
            "[147]\tvalidation_0-logloss:0.088907\n",
            "[148]\tvalidation_0-logloss:0.089416\n",
            "[149]\tvalidation_0-logloss:0.089388\n",
            "[150]\tvalidation_0-logloss:0.089108\n",
            "[151]\tvalidation_0-logloss:0.088735\n",
            "[152]\tvalidation_0-logloss:0.088717\n",
            "[153]\tvalidation_0-logloss:0.088484\n",
            "[154]\tvalidation_0-logloss:0.088471\n",
            "[155]\tvalidation_0-logloss:0.088545\n",
            "[156]\tvalidation_0-logloss:0.088521\n",
            "[157]\tvalidation_0-logloss:0.088547\n",
            "[158]\tvalidation_0-logloss:0.088275\n",
            "[159]\tvalidation_0-logloss:0.0883\n",
            "[160]\tvalidation_0-logloss:0.08828\n",
            "[161]\tvalidation_0-logloss:0.088013\n",
            "[162]\tvalidation_0-logloss:0.087758\n",
            "[163]\tvalidation_0-logloss:0.087784\n",
            "[164]\tvalidation_0-logloss:0.087777\n",
            "[165]\tvalidation_0-logloss:0.087517\n",
            "[166]\tvalidation_0-logloss:0.087542\n",
            "[167]\tvalidation_0-logloss:0.087642\n",
            "[168]\tvalidation_0-logloss:0.08739\n",
            "[169]\tvalidation_0-logloss:0.087377\n",
            "[170]\tvalidation_0-logloss:0.087298\n",
            "[171]\tvalidation_0-logloss:0.087368\n",
            "[172]\tvalidation_0-logloss:0.087395\n",
            "[173]\tvalidation_0-logloss:0.087385\n",
            "[174]\tvalidation_0-logloss:0.087132\n",
            "[175]\tvalidation_0-logloss:0.087159\n",
            "[176]\tvalidation_0-logloss:0.086955\n",
            "[177]\tvalidation_0-logloss:0.087053\n",
            "[178]\tvalidation_0-logloss:0.08697\n",
            "[179]\tvalidation_0-logloss:0.086973\n",
            "[180]\tvalidation_0-logloss:0.087038\n",
            "[181]\tvalidation_0-logloss:0.086799\n",
            "[182]\tvalidation_0-logloss:0.086826\n",
            "[183]\tvalidation_0-logloss:0.086582\n",
            "[184]\tvalidation_0-logloss:0.086588\n",
            "[185]\tvalidation_0-logloss:0.086614\n",
            "[186]\tvalidation_0-logloss:0.086372\n",
            "[187]\tvalidation_0-logloss:0.086369\n",
            "[188]\tvalidation_0-logloss:0.086297\n",
            "[189]\tvalidation_0-logloss:0.086104\n",
            "[190]\tvalidation_0-logloss:0.086023\n",
            "[191]\tvalidation_0-logloss:0.08605\n",
            "[192]\tvalidation_0-logloss:0.086149\n",
            "[193]\tvalidation_0-logloss:0.085916\n",
            "[194]\tvalidation_0-logloss:0.085915\n",
            "[195]\tvalidation_0-logloss:0.085984\n",
            "[196]\tvalidation_0-logloss:0.086012\n",
            "[197]\tvalidation_0-logloss:0.085922\n",
            "[198]\tvalidation_0-logloss:0.085853\n",
            "[199]\tvalidation_0-logloss:0.085874\n",
            "[200]\tvalidation_0-logloss:0.085888\n",
            "[201]\tvalidation_0-logloss:0.08595\n",
            "[202]\tvalidation_0-logloss:0.08573\n",
            "[203]\tvalidation_0-logloss:0.08573\n",
            "[204]\tvalidation_0-logloss:0.085753\n",
            "[205]\tvalidation_0-logloss:0.085821\n",
            "[206]\tvalidation_0-logloss:0.08584\n",
            "[207]\tvalidation_0-logloss:0.085776\n",
            "[208]\tvalidation_0-logloss:0.085686\n",
            "[209]\tvalidation_0-logloss:0.08571\n",
            "[210]\tvalidation_0-logloss:0.085806\n",
            "[211]\tvalidation_0-logloss:0.085593\n",
            "[212]\tvalidation_0-logloss:0.085801\n",
            "[213]\tvalidation_0-logloss:0.085806\n",
            "[214]\tvalidation_0-logloss:0.085744\n",
            "[215]\tvalidation_0-logloss:0.085658\n",
            "[216]\tvalidation_0-logloss:0.085843\n",
            "[217]\tvalidation_0-logloss:0.085632\n",
            "[218]\tvalidation_0-logloss:0.085726\n",
            "[219]\tvalidation_0-logloss:0.085783\n",
            "[220]\tvalidation_0-logloss:0.085791\n",
            "[221]\tvalidation_0-logloss:0.085817\n",
            "[222]\tvalidation_0-logloss:0.085757\n",
            "[223]\tvalidation_0-logloss:0.085674\n",
            "[224]\tvalidation_0-logloss:0.08586\n",
            "[225]\tvalidation_0-logloss:0.085871\n",
            "[226]\tvalidation_0-logloss:0.085927\n",
            "[227]\tvalidation_0-logloss:0.085954\n",
            "[228]\tvalidation_0-logloss:0.085874\n",
            "[229]\tvalidation_0-logloss:0.086057\n",
            "[230]\tvalidation_0-logloss:0.086002\n",
            "[231]\tvalidation_0-logloss:0.085922\n",
            "[232]\tvalidation_0-logloss:0.086102\n",
            "[233]\tvalidation_0-logloss:0.086115\n",
            "[234]\tvalidation_0-logloss:0.086169\n",
            "[235]\tvalidation_0-logloss:0.086263\n",
            "[236]\tvalidation_0-logloss:0.086292\n",
            "[237]\tvalidation_0-logloss:0.086217\n",
            "[238]\tvalidation_0-logloss:0.086395\n",
            "[239]\tvalidation_0-logloss:0.086342\n",
            "[240]\tvalidation_0-logloss:0.08618\n",
            "[241]\tvalidation_0-logloss:0.086195\n",
            "[242]\tvalidation_0-logloss:0.086248\n",
            "[243]\tvalidation_0-logloss:0.086263\n",
            "[244]\tvalidation_0-logloss:0.086293\n",
            "[245]\tvalidation_0-logloss:0.086222\n",
            "[246]\tvalidation_0-logloss:0.086398\n",
            "[247]\tvalidation_0-logloss:0.086347\n",
            "[248]\tvalidation_0-logloss:0.086276\n",
            "[249]\tvalidation_0-logloss:0.086448\n",
            "[250]\tvalidation_0-logloss:0.086294\n",
            "[251]\tvalidation_0-logloss:0.086312\n",
            "[252]\tvalidation_0-logloss:0.086364\n",
            "[253]\tvalidation_0-logloss:0.086394\n",
            "[254]\tvalidation_0-logloss:0.08649\n",
            "[255]\tvalidation_0-logloss:0.086441\n",
            "[256]\tvalidation_0-logloss:0.08629\n",
            "[257]\tvalidation_0-logloss:0.08646\n",
            "[258]\tvalidation_0-logloss:0.086391\n",
            "[259]\tvalidation_0-logloss:0.086441\n",
            "[260]\tvalidation_0-logloss:0.086461\n",
            "[261]\tvalidation_0-logloss:0.086491\n",
            "[262]\tvalidation_0-logloss:0.086445\n",
            "[263]\tvalidation_0-logloss:0.086466\n",
            "[264]\tvalidation_0-logloss:0.086319\n",
            "[265]\tvalidation_0-logloss:0.086488\n",
            "[266]\tvalidation_0-logloss:0.086538\n",
            "[267]\tvalidation_0-logloss:0.086471\n",
            "[268]\tvalidation_0-logloss:0.086501\n",
            "[269]\tvalidation_0-logloss:0.086522\n",
            "[270]\tvalidation_0-logloss:0.086689\n",
            "[271]\tvalidation_0-logloss:0.086738\n",
            "[272]\tvalidation_0-logloss:0.08683\n",
            "[273]\tvalidation_0-logloss:0.086684\n",
            "[274]\tvalidation_0-logloss:0.08664\n",
            "[275]\tvalidation_0-logloss:0.086496\n",
            "[276]\tvalidation_0-logloss:0.086355\n",
            "[277]\tvalidation_0-logloss:0.086519\n",
            "[278]\tvalidation_0-logloss:0.086567\n",
            "[279]\tvalidation_0-logloss:0.08659\n",
            "[280]\tvalidation_0-logloss:0.086679\n",
            "[281]\tvalidation_0-logloss:0.086637\n",
            "[282]\tvalidation_0-logloss:0.086499\n",
            "[283]\tvalidation_0-logloss:0.086356\n",
            "[284]\tvalidation_0-logloss:0.086405\n",
            "[285]\tvalidation_0-logloss:0.086429\n",
            "[286]\tvalidation_0-logloss:0.086456\n",
            "[287]\tvalidation_0-logloss:0.086504\n",
            "[288]\tvalidation_0-logloss:0.08637\n",
            "[289]\tvalidation_0-logloss:0.086457\n",
            "[290]\tvalidation_0-logloss:0.086453\n",
            "[291]\tvalidation_0-logloss:0.086322\n",
            "[292]\tvalidation_0-logloss:0.086284\n",
            "[293]\tvalidation_0-logloss:0.086148\n",
            "[294]\tvalidation_0-logloss:0.086196\n",
            "[295]\tvalidation_0-logloss:0.086221\n",
            "[296]\tvalidation_0-logloss:0.086308\n",
            "[297]\tvalidation_0-logloss:0.086178\n",
            "[298]\tvalidation_0-logloss:0.086263\n",
            "[299]\tvalidation_0-logloss:0.086131\n",
            "[300]\tvalidation_0-logloss:0.086179\n",
            "[301]\tvalidation_0-logloss:0.086052\n",
            "[302]\tvalidation_0-logloss:0.086016\n",
            "[303]\tvalidation_0-logloss:0.086101\n",
            "[304]\tvalidation_0-logloss:0.085977\n",
            "[305]\tvalidation_0-logloss:0.086059\n",
            "[306]\tvalidation_0-logloss:0.085971\n",
            "[307]\tvalidation_0-logloss:0.085998\n",
            "[308]\tvalidation_0-logloss:0.085999\n",
            "[309]\tvalidation_0-logloss:0.085877\n",
            "[310]\tvalidation_0-logloss:0.085923\n",
            "[311]\tvalidation_0-logloss:0.085948\n",
            "Stopping. Best iteration:\n",
            "[211]\tvalidation_0-logloss:0.085593\n",
            "\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "-> 400으로 설정해도 311번 반복한 후 학습을 완료했다.\n",
        "\n",
        "그 이유는 211번 반복시 logloss가 0.085593이고 311번 반복시 logloss가 0.085948인데, 211번에서 311번까지 early_stopping_rounds=100으로 지정한 100번의 반복동안 성능 평가 지수가 향상되지 않았기 때문에 반복하지 않고 중단함."
      ],
      "metadata": {
        "id": "d2ToBhMoaaxy"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "#조기 중단으로 학습된 XGBClassifier의 예측 성능\n",
        "get_clf_eval(y_test, ws100_preds, ws100_pred_proba)"
      ],
      "metadata": {
        "id": "86bvS5Cgaa1i",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "e388a9ac-f445-4738-c0d2-915425bafd77"
      },
      "execution_count": 18,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "오차 행렬\n",
            "[[34  3]\n",
            " [ 1 76]]\n",
            "정확도: 0.9649, 정밀도: 0.9620, 재현율: 0.9870,    F1: 0.9744, roc_auc:0.9954\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "-> 조기 중단값을 너무 급격하게 줄이면 예측 성능이 저하될 우려가 크다. 즉  early_stopping_rounds를 너무 적게 설정하면 그 반복수 이후로 향상되는 값이 나올 수 있기 때문에 예측 성능이 나빠질 수 있다."
      ],
      "metadata": {
        "id": "fTO7jF2iaa5k"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# early_stopping_rounds를 10으로 설정하고 재학습.\n",
        "\n",
        "xgb_wrapper.fit(X_train, y_train, early_stopping_rounds=10, eval_metric=\"logloss\", eval_set = evals, verbose=True)\n",
        "\n",
        "ws10_preds = xgb_wrapper.predict(X_test)\n",
        "ws10_pred_proba= xgb_wrapper.predict_proba(X_test)[:,1]\n",
        "get_clf_eval(y_test, ws10_preds, ws10_pred_proba)"
      ],
      "metadata": {
        "id": "U6XELb9PabvM",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "d3f14214-f742-487f-bcc3-9e4794b8e84d"
      },
      "execution_count": 19,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "[0]\tvalidation_0-logloss:0.61352\n",
            "Will train until validation_0-logloss hasn't improved in 10 rounds.\n",
            "[1]\tvalidation_0-logloss:0.547842\n",
            "[2]\tvalidation_0-logloss:0.494247\n",
            "[3]\tvalidation_0-logloss:0.447986\n",
            "[4]\tvalidation_0-logloss:0.409109\n",
            "[5]\tvalidation_0-logloss:0.374977\n",
            "[6]\tvalidation_0-logloss:0.345714\n",
            "[7]\tvalidation_0-logloss:0.320529\n",
            "[8]\tvalidation_0-logloss:0.29721\n",
            "[9]\tvalidation_0-logloss:0.277991\n",
            "[10]\tvalidation_0-logloss:0.260302\n",
            "[11]\tvalidation_0-logloss:0.246037\n",
            "[12]\tvalidation_0-logloss:0.231556\n",
            "[13]\tvalidation_0-logloss:0.22005\n",
            "[14]\tvalidation_0-logloss:0.208572\n",
            "[15]\tvalidation_0-logloss:0.199993\n",
            "[16]\tvalidation_0-logloss:0.190118\n",
            "[17]\tvalidation_0-logloss:0.181818\n",
            "[18]\tvalidation_0-logloss:0.174729\n",
            "[19]\tvalidation_0-logloss:0.167657\n",
            "[20]\tvalidation_0-logloss:0.158202\n",
            "[21]\tvalidation_0-logloss:0.154725\n",
            "[22]\tvalidation_0-logloss:0.148947\n",
            "[23]\tvalidation_0-logloss:0.143308\n",
            "[24]\tvalidation_0-logloss:0.136344\n",
            "[25]\tvalidation_0-logloss:0.132778\n",
            "[26]\tvalidation_0-logloss:0.127912\n",
            "[27]\tvalidation_0-logloss:0.125263\n",
            "[28]\tvalidation_0-logloss:0.119978\n",
            "[29]\tvalidation_0-logloss:0.116412\n",
            "[30]\tvalidation_0-logloss:0.114502\n",
            "[31]\tvalidation_0-logloss:0.112572\n",
            "[32]\tvalidation_0-logloss:0.11154\n",
            "[33]\tvalidation_0-logloss:0.108681\n",
            "[34]\tvalidation_0-logloss:0.106681\n",
            "[35]\tvalidation_0-logloss:0.104207\n",
            "[36]\tvalidation_0-logloss:0.102962\n",
            "[37]\tvalidation_0-logloss:0.100576\n",
            "[38]\tvalidation_0-logloss:0.098683\n",
            "[39]\tvalidation_0-logloss:0.096444\n",
            "[40]\tvalidation_0-logloss:0.095869\n",
            "[41]\tvalidation_0-logloss:0.094242\n",
            "[42]\tvalidation_0-logloss:0.094715\n",
            "[43]\tvalidation_0-logloss:0.094272\n",
            "[44]\tvalidation_0-logloss:0.093894\n",
            "[45]\tvalidation_0-logloss:0.094184\n",
            "[46]\tvalidation_0-logloss:0.09402\n",
            "[47]\tvalidation_0-logloss:0.09236\n",
            "[48]\tvalidation_0-logloss:0.093012\n",
            "[49]\tvalidation_0-logloss:0.091272\n",
            "[50]\tvalidation_0-logloss:0.090051\n",
            "[51]\tvalidation_0-logloss:0.089605\n",
            "[52]\tvalidation_0-logloss:0.089577\n",
            "[53]\tvalidation_0-logloss:0.090703\n",
            "[54]\tvalidation_0-logloss:0.089579\n",
            "[55]\tvalidation_0-logloss:0.090357\n",
            "[56]\tvalidation_0-logloss:0.091587\n",
            "[57]\tvalidation_0-logloss:0.091527\n",
            "[58]\tvalidation_0-logloss:0.091986\n",
            "[59]\tvalidation_0-logloss:0.091951\n",
            "[60]\tvalidation_0-logloss:0.091939\n",
            "[61]\tvalidation_0-logloss:0.091461\n",
            "[62]\tvalidation_0-logloss:0.090311\n",
            "Stopping. Best iteration:\n",
            "[52]\tvalidation_0-logloss:0.089577\n",
            "\n",
            "오차 행렬\n",
            "[[34  3]\n",
            " [ 2 75]]\n",
            "정확도: 0.9561, 정밀도: 0.9615, 재현율: 0.9740,    F1: 0.9677, roc_auc:0.9947\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "-> 62번 반복까지만 수행된 후 학습이 종료됐는데 이때 예측 결과 정확도는 early_stopping_rounds가 100일때보다 낮다.\n",
        "\n"
      ],
      "metadata": {
        "id": "Ov-K19Jqaa-E"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "from xgboost import plot_importance\n",
        "import matplotlib.pyplot as plt\n",
        "%matplotlib inline\n",
        "\n",
        "fig, ax= plt.subplots(figsize=(10,12))\n",
        "#사이킷런 Wrapper 클래스를 입력해도 무방.\n",
        "plot_importance(xgb_wrapper, ax=ax)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 747
        },
        "id": "SXo38qGlyPJq",
        "outputId": "0b6970c3-91c3-4996-b304-e66875cd769d"
      },
      "execution_count": 20,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<matplotlib.axes._subplots.AxesSubplot at 0x7fe5cd4707d0>"
            ]
          },
          "metadata": {},
          "execution_count": 20
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 720x864 with 1 Axes>"
            ],
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAmEAAALJCAYAAAD8uvTIAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nOzdf5zWdZ3v/8dLICUg3RpAlMWJdJMfo5Nwos76xZlNVNJMc8sl60RgrPnd0E1Tdj1Rtue41MYKu+kx1L62VqSVrfvVdK2j1x7X1goUojSstim0/AFhOkQ6g6/zx1zYgAgzw1zzngse99ttbl7X59f1HF41PPl8Ptc1kZlIkiRpYB1QOoAkSdL+yBImSZJUgCVMkiSpAEuYJElSAZYwSZKkAixhkiRJBVjCJO0XIuKvI+K60jkkabvwc8Ik7UlEtAFjgW3dFv9RZv5yL495bmZ+a+/S1Z+I+DhwZGa+p3QWSeV4JkxST70tM0d2++pzAesPETG05Ov3Vb3mltT/LGGS+iwiDo6I6yPiVxHxWET8j4gYUl33uoi4OyI2RcTGiPhiRBxSXXcjMAH4/yOiPSIuiYiWiHh0p+O3RcSJ1ccfj4ivRsQXIuIZYO7uXn8XWT8eEV+oPm6MiIyI90fEhojYHBHnRcR/iYjvR8TTEfGZbvvOjYj7IuIzEfGbiPhRRLyl2/rDIuJfIuLXEfGTiPjATq/bPfd5wF8DZ1e/97XV7d4fEQ9HxLMR8Z8R8efdjtESEY9GxEUR8WT1+31/t/XDI2JpRPy8mu/fI2J4dd2bIuLb1e9pbUS09GnYkvqdJUzS3rgB6ASOBN4AnAScW10XwN8ChwGTgD8EPg6Qme8FfsHvz659qoev93bgq8AhwBf38Po9MQM4CjgbWAZcBpwITAHeFREn7LTtT4EG4GPALRHx6uq6LwOPVr/XPwWuiIg/eZnc1wNXADdVv/djq9s8CZwGvAp4P3BlRBzX7RiHAgcDhwPzgasi4g+q6z4NTAP+K/Bq4BLghYg4HLgd+B/V5RcDX4uI0b34M5JUI5YwST31z9WzKU9HxD9HxFjgrcCFmbklM58ErgT+DCAzf5KZ38zM5zLzKeDvgRNe/vA98h+Z+c+Z+QJdZeVlX7+H/iYzf5eZdwFbgJWZ+WRmPgbcS1ex2+5JYFlmdmTmTcB64NSI+EPgj4FLq8daA1wH/Ldd5c7MrbsKkpm3Z+ZPs8u/AXcB/0+3TTqAT1Rf/xtAO/D6iDgAmAdckJmPZea2zPx2Zj4HvAf4RmZ+o/ra3wRWVf/cJBXmvQmSeuqM7jfRR8QbgWHAryJi++IDgA3V9WOB5XQViVHVdZv3MsOGbo+P2N3r99AT3R5v3cXzkd2eP5Y7vpPp53Sd+ToM+HVmPrvTuukvk3uXImI2XWfY/oiu7+OVwLpum2zKzM5uz39bzdcAHETXWbqdHQG8MyLe1m3ZMOCePeWRVHuWMEl9tQF4DmjYqRxsdwWQQFNm/joizgA+0239zm/N3kJX8QCgem/XzpfNuu+zp9fvb4dHRHQrYhOAfwF+Cbw6IkZ1K2ITgMe67bvz97rD84g4EPgaXWfPbs3Mjoj4Z7ou6e7JRuB3wOuAtTut2wDcmJkfeMlekorzcqSkPsnMX9F1yWxpRLwqIg6o3oy//ZLjKLoumf2mem/SR3Y6xBPAxG7PHwEOiohTI2IY8N+BA/fi9fvbGGBhRAyLiHfSdZ/bNzJzA/Bt4G8j4qCIOIaue7a+sJtjPQE0Vi8lAryCru/1KaCzelbspJ6Eql6a/Rzw99U3CAyJiDdXi90XgLdFxMnV5QdVb/If3/tvX1J/s4RJ2hv/ja4C8RBdlxq/CoyrrrscOA74DV03h9+y075/C/z36j1mF2fmb4Dz6bqf6jG6zow9yu7t7vX723fouol/I/A/gT/NzE3VdXOARrrOin0d+NgePv/sK9X/boqIB6pn0BYCN9P1fbybrrNsPXUxXZcuvwf8GvgkcEC1IL6drndjPkXXmbGP4M9+aVDww1olaQ8iYi5dHyx7fOkskvYd/mtIkiSpAEuYJElSAV6OlCRJKsAzYZIkSQXU5eeEHXLIIXnkkUeWjqFe2rJlCyNGjCgdQ73k3OqTc6tfzq4+7W5uq1ev3piZL/l1YXVZwsaOHcuqVatKx1AvVSoVWlpaSsdQLzm3+uTc6pezq0+7m1tE/HxXy70cKUmSVIAlTJIkqQBLmCRJUgGWMEmSpAIsYZIkSQVYwiRJkgqwhEmSJBVgCZMkSSrAEiZJklSAJUySJKkAS5gkSVIBljBJkqQCLGGSJEkFWMIkSZIKsIRJkiQVYAmTJEkqwBImSZJUgCVMkiSpAEuYJEnaJ2zbto03vOENnHbaaQBkJpdddhl/9Ed/xKRJk/iHf/iHwgl3NLRWB46IhcAHgaOBdUAAzwIfzMy11W0+B5wGPJmZU2uVRZIk7fuWL1/OpEmTeOaZZwC44YYb2LBhAz/60Y844IADePLJJwsn3FHNShhwPnAiMAF4ODM3R8RsYAUwo7rNDcBngH/qzYG3dmyjcdHt/RhVA+Gipk7mOre649zqk3OrX86uS9uSU3u1/aOPPsrtt9/OZZddxt///d8D8L/+1//iS1/6Egcc0HXhb8yYMf2ec2/U5HJkRFwDTATuAGZk5ubqqvuB8du3y8z/A/y6FhkkSdL+48ILL+RTn/rUi4UL4Kc//Sk33XQT06dPZ/bs2fz4xz8umPClalLCMvM84JdAa2Ze2W3VfLqKmSRJUr+47bbbGDNmDNOmTdth+XPPPcdBBx3EqlWr+MAHPsC8efMKJdy1Wl6O3EFEtNJVwo7v4/4LgAUADQ2jWdzU2Y/pNBDGDu86za764tzqk3OrX86uS6VS6fG2K1eu5K677uKWW27h+eef57e//S2zZs3i1a9+NYcddhiVSoU/+IM/4MEHH+zVcXujvb2918cekBIWEccA1wGzM3NTX46RmSvoup+MCROPzKXrBqw/qp9c1NSJc6s/zq0+Obf65ey6tJ3T0uNtW1p+v22lUuHTn/40t912G4sWLWLr1q20tLRQqVSYNGnSDtv2p0ql0utj13zKETEBuAV4b2Y+0h/HHD5sCOt7ecOeyqtUKr36P5UGB+dWn5xb/XJ2/WfRokWcc845XHnllYwcOZLrrruudKQdDETVXgy8Brg6IgA6M3M6QESsBFqAhoh4FPhYZl4/AJkkSdI+qKWl5cUzUocccgi33z5432lasxKWmY3Vh+dWv3a1zZxavb4kSdJg5ifmS5IkFWAJkyRJKsASJkmSVIAlTJIkqQBLmCRJUgGWMEmSpAIsYZIkSQVYwiRJkgqwhEmSJBVgCZMkSSrAEiZJklSAJUySJKkAS5gkSVIBljBJkqQCLGGSJEkFWMIkSXXnd7/7HW984xs59thjmTJlCh/72Md2WL9w4UJGjhxZKJ3UM0NLvGhELAQ+CDwEHAYcB1yWmZ/uyf5bO7bRuOj2GiZULVzU1Mlc51Z3nFt9qse5tS05tcfbHnjggdx9992MHDmSjo4Ojj/+eGbPns2b3vQmVq1axebNm2uYVOofRUoYcD5wIvA8cARwRqEckqQ6FBEvnunq6Oigo6ODiGDbtm185CMf4Utf+hJf//rXC6eUdm/AL0dGxDXAROAO4JzM/B7QMdA5JEn1bdu2bTQ3NzNmzBhmzZrFjBkz+MxnPsPpp5/OuHHjSseT9igyc+BfNKINmJ6ZG6vPPw607+5yZEQsABYANDSMnrZ42bUDkFT9aexweGJr6RTqLedWn+pxbk2HH9yn/drb2/noRz/K3Llzue6661i2bBlDhgxh9uzZ3HHHHf2csvba29u9n60O7W5ura2tqzNz+s7LS12O7LXMXAGsAJgw8chcuq5uoqvqoqZOnFv9cW71qR7n1nZOS5/3feCBB3j66ad56qmnmD9/PgDPPfcc5557Lj/5yU/6KeHAqFQqtLS0lI6hXurL3Orr/6FVw4cNYX0vbuDU4FCpVPbqh6zKcG71aV+f21NPPcWwYcM45JBD2Lp1K9/85je59NJLefzxx1/cZuTIkXVXwLR/qcsSJknav/3qV7/ife97H9u2beOFF17gXe96F6eddlrpWFKvFC1hEXEosAp4FfBCRFwITM7MZ0rmkiQNbscccwwPPvjgbrdpb28foDRS3xQpYZnZ2O3p+BIZJEmSSvIT8yVJkgqwhEmSJBVgCZMkSSrAEiZJklSAJUySJKkAS5gkSVIBljBJkqQCLGGSJEkFWMIkSZIKsIRJkiQVYAmTJEkqwBImSZJUgCVMkiSpAEuYJElSAZYwSaojGzZsoLW1lcmTJzNlyhSWL18OwJo1a3jTm95Ec3Mz06dP5+GHHy6cVNKe1LSERcTCiHg4Ir4WEf8REc9FxMXd1h8UEd+NiLUR8cOIuLyWeSSp3g0dOpSlS5fy0EMPcf/993PVVVfx0EMPcckll/Cxj32MNWvW8IlPfILPfvazpaNK2oOhNT7++cCJwPPAEcAZO61/DviTzGyPiGHAv0fEHZl5/+4OurVjG42Lbq9JYNXORU2dzHVudce51VbbklN7tf24ceMYN24cAKNGjWLSpEk89thjRATPPPMMAL/5zW94zWte0+9ZJfWvmpWwiLgGmAjcAXwuM6+MiB1+2mRmAu3Vp8OqX1mrTJK0L2lra+PBBx9kxowZLFu2jJNPPpmLL76YF154gaVLl5aOJ2kPalbCMvO8iDgFaM3MjS+3XUQMAVYDRwJXZeZ3Xma7BcACgIaG0Sxu6qxBatXS2OFdZ1VUX5xbbVUqlT7tt3XrVi644ALOPfdcHnjgAf7hH/6B+fPnc8IJJ3DPPfewZMkSDj300P4NqwHR3t7e5/9dqJy+zC26TkbVRkS0AdO3l7CI+DjQnpmf3sW2hwBfBz6UmT/Y3XEnTDwyD3jX8v4PrJq6qKmTpetqfQVc/c251VZvL0cCdHR0cNppp3HyySfz4Q9/GICDDz6Yp59+moggMxk5ciRbtmzp77gaAJVKhZaWltIx1Eu7m1tErM7M6TsvHzQ/WTPz6Yi4BzgF2G0JGz5sCOv78INLZVUqFdrOaSkdQ73k3AaXzGT+/PlMmjTpxQIGcNhhh/Fv//ZvtLS0cPfdd3P44YcXTCmpJ4qWsIgYDXRUC9hwYBbwyZKZJGkwu++++7jxxhtpamqiubkZgCuuuIJrr72WCy64gM7OTg466CAuuuiiwkkl7cmAlLCIOBRYBbwKeCEiLgQmA+OAz1fvCzsAuDkzbxuITJJUj44//nhe7jaS1atXv/jYe4qkwa+mJSwzG7s9Hb+LTb4PvKGWGSRJkgYjPzFfkiSpAEuYJElSAZYwSZKkAixhkiRJBVjCJEmSCrCESZIkFWAJkyRJKsASJkmSVIAlTJIkqQBLmCRJUgGWMEmSpAIsYZIkSQVYwiRJkgqwhEmSJBVgCZOkGtiwYQOtra1MnjyZKVOmsHz58hfX/eM//iNHH300U6ZM4ZJLLimYUlJJQ2t14IhYCHwQOBpYBwTwLPDBzFwbEa8Hbuq2y0RgcWYuq1UmSRooQ4cOZenSpRx33HE8++yzTJs2jVmzZvHEE09w6623snbtWg488ECefPLJ0lElFVKzEgacD5wITAAezszNETEbWAHMyMz1QDNARAwBHgO+3pMDb+3YRuOi22uTWjVzUVMnc51b3XFuXdqWnNqr7ceNG8e4ceMAGDVqFJMmTeKxxx7j2muvZdGiRRx44IEAjBkzpt+zSqoPNbkcGRHX0HVm6w66Ctfm6qr7gfG72OUtwE8z8+e1yCNJJbW1tfHggw8yY8YMHnnkEe69915mzJjBCSecwPe+973S8SQVUpMzYZl5XkScArRm5sZuq+bTVcx29mfAyt0dMyIWAAsAGhpGs7ips7/iaoCMHd51VkX1xbl1qVQqfdpv69atXHDBBZx77rk88MAD/OY3v2HdunUsWbKEH/3oR5x++ul86UtfIiL6NW97e3ufM6ssZ1ef+jK3yMyahImINmD69hIWEa3A1cDxmbmp23avAH4JTMnMJ3py7AkTj8wD3rV8zxtqULmoqZOl62p5BVy14Ny69PZyJEBHRwennXYaJ598Mh/+8IcBOOWUU7j00ktpbW0F4HWvex33338/o0eP7te8lUqFlpaWfj2mBoazq0+7m1tErM7M6TsvH5B3R0bEMcB1wNu7F7Cq2cADPS1gklQPMpP58+czadKkFwsYwBlnnME999wDwCOPPMLzzz9PQ0NDqZiSCqr5P28jYgJwC/DezHxkF5vMYQ+XInc2fNgQ1vfhX6Uqq1Kp0HZOS+kY6iXn1jf33XcfN954I01NTTQ3NwNwxRVXMG/ePObNm8fUqVN5xStewec///l+vxQpqT4MxDWGxcBrgKurP2g6t5+Si4gRwCzgzwcghyQNmOOPP56Xu93jC1/4wgCnkTQY1ayEZWZj9eG51a9dbbOFroImSZK0X/ET8yVJkgqwhEmSJBVgCZMkSSrAEiZJklSAJUySJKkAS5gkSVIBljBJkqQCLGGSJEkFWMIkSZIKsIRJkiQVYAmTJEkqwBImSZJUgCVMkiSpAEuYJElSAZYwSfucDRs20NrayuTJk5kyZQrLly8H4KMf/SjHHHMMzc3NnHTSSfzyl78snFTS/qxmJSwiFkbEwxGREfH9iFgXEd+OiGO7bXNIRHw1In5U3fbNtcojaf8xdOhQli5dykMPPcT999/PVVddxUMPPcRHPvIRvv/977NmzRpOO+00PvGJT5SOKmk/NrSGxz4fOBGYADycmZsjYjawAphR3WY5cGdm/mlEvAJ4ZU8OvLVjG42Lbq9FZtXQRU2dzHVudWcwzK1tyam92n7cuHGMGzcOgFGjRjFp0iQee+wxJk+e/OI2W7ZsISL6Nack9UZNSlhEXANMBO4APpeZ366uuh8YX93mYGAmMBcgM58Hnq9FHkn7r7a2Nh588EFmzOj6t99ll13GP/3TP3HwwQdzzz33FE4naX8WmVmbA0e0AdMzc2O3ZRcDR2fmuRHRTNdZsYeAY4HVwAWZueVljrcAWADQ0DB62uJl19Ykt2pn7HB4YmvpFOqtwTC3psMP7tN+W7du5YILLuA973kPM2fO3GHdF7/4RZ5//nne//7390fEQae9vZ2RI0eWjqE+cHb1aXdza21tXZ2Z03dePmAlLCJagauB4zNzU0RMp+vM2B9n5nciYjnwTGZ+dE/HnjDxyDzgXctrklu1c1FTJ0vX1fIKuGphMMytt5cjATo6OjjttNM4+eST+fCHP/yS9b/4xS9461vfyg9+8IP+iDjoVCoVWlpaSsdQHzi7+rS7uUXELkvYgLw7MiKOAa4D3p6Zm6qLHwUezczvVJ9/FThuIPJI2rdlJvPnz2fSpEk7FLAf//jHLz6+9dZbOfroo0vEkySgtjfmAxARE4BbgPdm5iPbl2fm4xGxISJen5nrgbfQdWlyj4YPG8L6PvzLWGVVKhXazmkpHUO9VI9zu++++7jxxhtpamqiubkZgCuuuILrr7+e9evXc8ABB3DEEUdwzTXXFE4qaX82ENcYFgOvAa6uvhOps9spuQ8BX6y+M/I/gX3z5gxJA+r4449nV7davPWtby2QRpJ2rWYlLDMbqw/PrX7taps1wEuukUqSJO3r/MR8SZKkAixhkiRJBVjCJEmSCrCESZIkFWAJkyRJKsASJkmSVIAlTJIkqQBLmCRJUgGWMEmSpAIsYZIkSQVYwiRJkgqwhEmSJBVgCZMkSSrAEiZJklSAJUxSMRs2bKC1tZXJkyczZcoUli9fDsBXvvIVpkyZwgEHHMCqVasKp5Sk2hha4kUjYiHwQWAC8ONuWSYBozPz1yVySRpYQ4cOZenSpRx33HE8++yzTJs2jVmzZjF16lRuueUW/vzP/7x0REmqmSIlDDgfODEzH92+ICLeBvxlTwrY1o5tNC66vZb5VAMXNXUy17nVnd7MrW3Jqb069rhx4xg3bhwAo0aNYtKkSTz22GPMmjWr1zklqd4M+OXIiLgGmAjcERF/2W3VHGDlQOeRNDi0tbXx4IMPMmPGjNJRJGlARGYO/ItGtAHTM3Nj9fkrgUeBI1/uTFhELAAWADQ0jJ62eNm1A5RW/WXscHhia+kU6q3ezK3p8IP79Bpbt27lggsu4D3veQ8zZ858cfmFF17IBz/4QV7/+tf36bj7s/b2dkaOHFk6hvrA2dWn3c2ttbV1dWZO33l5qcuRO3sbcN/uLkVm5gpgBcCEiUfm0nWDJbp66qKmTpxb/enN3NrOaen18Ts6OjjttNM477zz+PCHP7zDukMOOYRp06YxffpLfnZpDyqVCi0tLaVjqA+cXX3qy9wGy7sj/wwvRUr7ncxk/vz5TJo06SUFTJL2dcVPS0TEwcAJwHt6us/wYUNY38sbgFVepVLp05kSlVXLud13333ceOONNDU10dzcDMAVV1zBc889x4c+9CGeeuopTj31VJqbm/nXf/3XmmSQpFKKlzDgTOCuzNxSOoikgXX88cfzcvelnnnmmQOcRpIGVpESlpmN3R7fANxQIockSVIpg+WeMEmSpP2KJUySJKkAS5gkSVIBljBJkqQCLGGSJEkFWMIkSZIKsIRJkiQVYAmTJEkqwBImSZJUgCVMkiSpAEuYJElSAZYwSZKkAixhkiRJBVjCJEmSCrCESdqjefPmMWbMGKZOnfrisrVr1/LmN7+ZpqYm3va2t/HMM88UTChJ9aemJSwiFkbEwxGREfH9iFgXEd+OiGOr6w+KiO9GxNqI+GFEXF7LPJL6Zu7cudx55507LDv33HNZsmQJ69at48wzz+Tv/u7vCqWTpPo0tMbHPx84EZgAPJyZmyNiNrACmAE8B/xJZrZHxDDg3yPijsy8f3cH3dqxjcZFt9c4uvrbRU2dzHVug0LbklN7tf3MmTNpa2vbYdkjjzzCzJkzAZg1axYnn3wyf/M3f9NfESVpn1ezM2ERcQ0wEbgDmJGZm6ur7gfGA2SX9uryYdWvrFUmSf1nypQp3HrrrQB85StfYcOGDYUTSVJ9qVkJy8zzgF8CrZl5ZbdV8+kqZgBExJCIWAM8CXwzM79Tq0yS+s/nPvc5rr76aqZNm8azzz7LK17xitKRJKmu1Ppy5A4iopWuEnb89mWZuQ1ojohDgK9HxNTM/MEu9l0ALABoaBjN4qbOAUqt/jJ2eNclSZVXqVR6vG17ezuVSoXHH3+cLVu27LDvX//1XwOwYcMGxowZ06vjqra2z031x9nVp77MbcBKWEQcA1wHzM7MTTuvz8ynI+Ie4BTgJSUsM1fQdS8ZEyYemUvXDWh/VD+4qKkT5zY4tJ3T0uNtK5UKLS0ttLW1MWLECFpauvZ98sknGTNmDC+88AJz587lIx/5yIvrVN72uan+OLv61Je5DcjfiBExAbgFeG9mPtJt+Wigo1rAhgOzgE/u6XjDhw1hfS9vLFZ5lUqlV3/5a/CYM2cOlUqFjRs3Mn78eC6//HLa29u56qqrAHjHO97B+9///sIpJam+DNRpicXAa4CrIwKgMzOnA+OAz0fEELruT7s5M28boEySemjlypW7XH7BBRcMcBJJ2nfUtIRlZmP14bnVr53Xfx94Qy0zSJIkDUZ+Yr4kSVIBljBJkqQCLGGSJEkFWMIkSZIKsIRJkiQVYAmTJEkqwBImSZJUgCVMkiSpAEuYJElSAZYwSZKkAixhkiRJBVjCJEmSCrCESZIkFWAJkyRJKsASJu1D5s2bx5gxY5g6deqLy84++2yam5tpbm6msbGR5ubmggklSdvVtIRFxMKIeDgiMiK+HxHrIuLbEXHsTtsNiYgHI+K2WuaR9nVz587lzjvv3GHZTTfdxJo1a1izZg1nnXUW73jHOwqlkyR1N7TGxz8fOBGYADycmZsjYjawApjRbbsLgIeBV/XkoFs7ttG46Pb+zqoau6ipk7nOrVfalpzaq+1nzpxJW1vbLtdlJjfffDN33313PySTJO2tmp0Ji4hrgInAHcCMzNxcXXU/ML7bduOBU4HrapVFEtx7772MHTuWo446qnQUSRI1PBOWmedFxClAa2Zu7LZqPl3FbLtlwCXAqFplkQQrV65kzpw5pWNIkqpqfTlyBxHRSlcJO776/DTgycxcHREte9h3AbAAoKFhNIubOmucVv1t7PCuS5LquUql0ut9Hn/8cbZs2bLDvtu2beOmm27is5/9bK+P2d7e3qccKsu51S9nV5/6MrcBK2ERcQxdlxxnZ+am6uI/Bk6PiLcCBwGviogvZOZ7dt4/M1fQdS8ZEyYemUvXDWh/VD+4qKkT59Y7bee09H6ftjZGjBhBS8vv973zzjtpamrine98Z6+PV6lUdjiW6oNzq1/Orj71ZW4D8jdiREwAbgHem5mPbF+emX8F/FV1mxbg4l0VsJ0NHzaE9b28YVnlVSqVPpUK9dycOXOoVCps3LiR8ePHc/nllzN//ny+/OUveylSkgaZgTotsRh4DXB1RAB0Zub0AXptab+xcuXKXS6/4YYbBjaIJGmPalrCMrOx+vDc6tfutq0AlVrmkSRJGiz8xHxJkqQCLGGSJEkFWMIkSZIKsIRJkiQVYAmTJEkqwBImSZJUgCVMkiSpAEuYJElSAZYwSZKkAixhkiRJBVjCJEmSCrCESZIkFWAJkyRJKsASJkmSVIAlTJIkqQBLmLQPmTdvHmPGjGHq1KkvLjv77LNpbm6mubmZxsZGmpubCyaUJG1XsxIWEQsj4uGIyIj4fkSsi4hvR8Sx1fV/GBH3RMRDEfHDiLigVlmk/cXcuXO58847d1h20003sWbNGtasWcNZZ53FO97xjkLpJEndDa3hsc8HTgQmAA9n5uaImA2sAGYAncBFmflARIwCVkfENzPzoT0deGvHNhoX3V7D6KqFi5o6mevceqVtyam92n7mzJm0tbXtcl1mcvPNN3P33Xf3QzJJ0t6qyZmwiLgGmAjcAczIzM3VVfcD4wEy81eZ+UD18bPAw8DhtcgjCe69917Gjh3LUUcdVTqKJIkanQnLzPMi4hSgNTM3dls1n65itoOIaATeAHzn5Y4ZEQuABQANDaNZ3NTZn8gIlA8AACAASURBVJE1AMYO7zobpp6rVCq93ufxxx9ny5YtL9n3yiuv5I1vfGOvj9ne3t6nHCrLudUvZ1ef+jK3Wl6O3EFEtNJVwo7faflI4GvAhZn5zMvtn5kr6LqUyYSJR+bSdQMWXf3koqZOnFvvtJ3T0vt92toYMWIELS2/37ezs5Ozzz6b1atXM378+F4dr1Kp7HAs1QfnVr+cXX3qy9wG5G/EiDgGuA6YnZmbui0fRlcB+2Jm3tLT4w0fNoT1vbxXRuVVKpU+lQrtvW9961scffTRvS5gkqTaqflHVETEBOAW4L2Z+Ui35QFcT9dN+39f6xzS/mDOnDm8+c1vZv369YwfP57rr78egC9/+cvMmTOncDpJUncDcSZsMfAa4Oqu3kVnZk4H/hh4L7AuItZUt/3rzPzGAGSS9kkrV67c5fIbbrhhYINIkvaoZiUsMxurD8+tfu28/t+BqNXrS5IkDWZ+Yr4kSVIBljBJkqQCLGGSJEkFWMIkSZIKsIRJkiQVYAmTJEkqwBImSZJUgCVMkiSpAEuYJElSAZYwSZKkAixhkiRJBVjCJEmSCrCESZIkFWAJkyRJKsASJu1D5s2bx5gxY5g6deqLy84++2yam5tpbm6msbGR5ubmggklSdvVtIRFxMKIeDgiMiK+HxHrIuLbEXFst21OiYj1EfGTiFhUyzzSvm7u3LnceeedOyy76aabWLNmDWvWrOGss87iHe94R6F0kqTuhtb4+OcDJwITgIczc3NEzAZWADMiYghwFTALeBT4XkT8S2Y+tLuDbu3YRuOi22scXf3toqZO5jq3Xmlbcmqvtp85cyZtbW27XJeZ3Hzzzdx99939kEyStLdqdiYsIq4BJgJ3ADMyc3N11f3A+OrjNwI/ycz/zMzngS8Db69VJml/du+99zJ27FiOOuqo0lEkSdTwTFhmnhcRpwCtmbmx26r5dBUzgMOBDd3WPQrM2NXxImIBsACgoWE0i5s6+z+0amrs8K6zYeq5SqXS630ef/xxtmzZ8pJ9r7zySt74xjf2+pjt7e19yqGynFv9cnb1qS9zq/XlyB1ERCtdJez43u6bmSvouozJhIlH5tJ1Axpd/eCipk6cW++0ndPS+33a2hgxYgQtLb/ft7Ozk7PPPpvVq1czfvz4l995FyqVyg7HUn1wbvXL2dWnvsxtwP5GjIhjgOuA2Zm5qbr4MeAPu202vrpst4YPG8L6Xt4ro/IqlUqfSoX23re+9S2OPvroXhcwSVLtDMhHVETEBOAW4L2Z+Ui3Vd8DjoqI10bEK4A/A/5lIDJJ+6I5c+bw5je/mfXr1zN+/Hiuv/56AL785S8zZ86cwukkSd0N1JmwxcBrgKsjAqAzM6dnZmdE/AXwr8AQ4HOZ+cMByiTtc1auXLnL5TfccMPABpEk7VFNS1hmNlYfnlv92tU23wC+UcsckiRJg42fmC9JklSAJUySJKkAS5gkSVIBljBJkqQCLGGSJEkFWMIkSZIKsIRJkiQVYAmTJEkqwBImSZJUgCVMkiSpAEuYJElSAZYwSZKkAixhkiRJBVjCJEmSCrCESf1k3rx5jBkzhqlTp75k3dKlS4kINm7cWCCZJGkwKlLCImJhRDwcEV+sPv8vEdEZEX9aIo/UH+bOncudd975kuUbNmzgrrvuYsKECQVSSZIGq6GFXvd84MTMfDQihgCfBO7q6c5bO7bRuOj2moVTbVzU1MncOppb25JTe7X9zJkzaWtre8nyv/zLv+RTn/oUb3/72/spmSRpXzDgJSwirgEmAndExOeABL4G/JeBziLV2q233srhhx/OscceWzqKJGmQGfASlpnnRcQpQCtwIPCl6uPdlrCIWAAsAGhoGM3ips5aR1U/Gzu862xYvahUKr3e5/HHH2fLli1UKhV+97vfsWjRIv7u7/7uxef33XcfBx98cP+HraH29vY+/VmoLOdWv5xdferL3EpdjtxuGXBpZr4QEbvdMDNXACsAJkw8MpeuKx1dvXVRUyf1NLe2c1p6v09bGyNGjKClpYV169axadMm/uIv/gKAjRs38qEPfYjvfve7HHroof2ctnYqlQotLS2lY6iXnFv9cnb1qS9zK/034nTgy9UC1gC8NSI6M/Ofy8aS9l5TUxNPPvnki88bGxtZtWoVDQ0NBVNJkgaLoiUsM1+7/XFE3ADc1pMCNnzYENb38qZplVepVPp0dqlezJkzh0qlwsaNGxk/fjyXX3458+fPLx1LkjRIlT4TJu0zVq5cudv1u3rnpCRp/1WkhGVm4y6WzR34JJIkSWX4ifmSJEkFWMIkSZIKsIRJkiQVYAmTJEkqwBImSZJUgCVMkiSpAEuYJElSAZYwSZKkAixhkiRJBVjCJEmSCrCESZIkFWAJkyRJKsASJkmSVIAlTJIkqQBLmAatefPmMWbMGKZOnfrisl//+tfMmjWLo446ilmzZrF58+aCCSVJ6rualbCIWBgRD0fE1yLiPyLiuYi4eKdt2iJiXUSsiYhVtcqi+jR37lzuvPPOHZYtWbKEt7zlLfz4xz/mLW95C0uWLCmUTpKkvTO0hsc+HzgReB44AjjjZbZrzcyNvTnw1o5tNC66fS/jaaDdcMqIXm0/c+ZM2tradlh26623UqlUAHjf+95HS0sLn/zkJ/spoSRJA6cmZ8Ii4hpgInAHcE5mfg/oqMVraf/yxBNPMG7cOAAOPfRQnnjiicKJJEnqm5qcCcvM8yLiFPZ8liuBuyIigc9m5oqX2zAiFgALABoaRrO4qbNfM6v22tvbXzyL1VOPP/44W7ZseXG/zs7OHY6xbdu2Xh9TvdOXuak851a/nF196svcank5sieOz8zHImIM8M2I+FFm/p9dbVgtaCsAJkw8MpeuKx1dvXXDKSNoaWnp1T5tbW2MGPH7/Q4//HBe//rXM27cOH71q19x2GGH9fqY6p1KpeKfcR1ybvXL2dWnvsyt6LsjM/Ox6n+fBL4OvLFkHg1+p59+Op///OcB+PznP8/b3/72wokkSeqbYqeTImIEcEBmPlt9fBLwiZ7sO3zYENYvObWm+dT/enuads6cOVQqFTZu3Mj48eO5/PLLWbRoEe9617u4/vrrOeKII7j55ptrE1aSpBqreQmLiEOBVcCrgBci4kJgMtAAfD0ituf4Umbe+bIH0n5n5cqVu1z+v//3/x7gJJIk9b+albDMbOz2dPwuNnkGOLZWry9JkjSY+Yn5kiRJBVjCJEmSCrCESZIkFWAJkyRJKsASJkmSVIAlTJIkqQBLmCRJUgGWMEmSpAIsYZIkSQVYwiRJkgqwhEmSJBVgCZMkSSqgRyUsIl4XEQdWH7dExMKIOKS20SRJkvZdPT0T9jVgW0QcCawA/hD4Us1SSZIk7eN6WsJeyMxO4EzgHzPzI8C42sXSvmr58uVMnTqVKVOmsGzZstJxJEkqpqclrCMi5gDvA26rLhvW1xetXs58OCI2R8T3I2JNRKyKiOP7ekwNfj/72c+49tpr+e53v8vatWu57bbb+MlPflI6liRJRQzt4XbvB84D/mdm/iwiXgvcuBevez5wIvA0sCUzMyKOAW4Gjt7Tzls7ttG46Pa9eHn1l7Ylp/Z425///OfMmDGDV77ylQCccMIJ3HLLLVxyySW1iidJ0qDVozNhmfkQcCnwQPX5zzLzk315wYi4BpgI3AF8IDOzumoEkC+7o+rea1/7Wu699142bdrEb3/7W77xjW+wYcOG0rEkSSqiR2fCIuJtwKeBVwCvjYhm4BOZeXpvXzAzz4uIU4DWzNwYEWcCfwuMAXp+WkV154gjjuDSSy/lpJNOYsSIETQ3NzNkyJDSsSRJKiJ+fyJqNxtFrAb+BKhk5huqy36QmVP79KIRbcD0zNzYbdlMYHFmnvgy+ywAFgA0NIyetnjZtX15afWzpsMP7vG27e3tjBw58sXn1157LaNHj+aMM86oRTT1k53npvrg3OqXs6tPu5tba2vr6sycvvPynt4T1pGZv4mI7ste6H3El5eZ/yciJkZEQ/dy1m39Cro+HoMJE4/Mpet6Gl211HZOS4+3rVQqTJ48mTFjxvCLX/yC1atXc//993PIIX7k3GBWqVRoaWkpHUO95Nzql7OrT32ZW0+bzA8j4t3AkIg4ClgIfLt38V6q+rljP63emH8ccCCwaU/7DR82hPW9uCFcg8dZZ53Fpk2bGDZsGFdddZUFTJK03+ppCfsQcBnwHF0f0vqvwP/oh9c/C/hvEdEBbAXOzp5cH1Xduvfee0tHkCRpUNhjCYuIIcDtmdlKVxHba5nZWH34yeqXJEnSfmWPH1GRmduAFyKi53dgS5Ikabd6ejmyHVgXEd8EtmxfmJkLa5JKkiRpH9fTEnZL9UuSJEn9oEclLDM/X+sgkiRJ+5OefmL+z9jFrxTKzIn9nkiSJGk/0NPLkd0/5fUg4J3Aq/s/jiRJ0v6hp7/Ae1O3r8cycxn+nkdJkqQ+6+nlyOO6PT2ArjNj/t4gSZKkPuppkVra7XEn8DPgXf0fR5Ikaf/Q0xI2PzP/s/uCiHhtDfJIkiTtF3p0Txjw1R4ukyRJUg/s9kxYRBwNTAEOjoh3dFv1KrreJSlJkqQ+2NPlyNcDpwGHAG/rtvxZ4AO1CiVJkrSv220Jy8xbgVsj4s2Z+R8DlEmSJGmf19N7wh6MiP83Iq6OiM9t/6ppMu2Tli9fztSpU5kyZQrLli0rHUeSpGJ6WsJuBA4FTgb+DRhP1yXJ3YqIhRHxcERkRHw/ItZFxLcj4thu2/xlRPwwIn4QESsjwnvN9lE/+9nPuPbaa/nud7/L2rVrue222/jJT35SOpYkSUX09CMqjszMd0bE2zPz8xHxJeDeHux3PnAiMAF4ODM3R8RsYAUwIyIOBxYCkzNza0TcDPwZcMPuDrq1YxuNi27vYXTVUtuSnv/ihJ///OfMmDGDV77ylQCccMIJ3HLLLVxyySW1iidJ0qDV0zNhHdX/Ph0RU4GDgTG72yEirgEmAncAMzJzc3XV/XSdSdtuKDA8IoYCrwR+2cNMqjOvfe1ruffee9m0aRO//e1v+cY3vsGGDRtKx5IkqYienglbERF/AHwU+BdgJLB4dztk5nkRcQrQmpkbu62aT1cxIzMfi4hPA78AtgJ3ZeZdvfweVCeOOOIILr30Uk466SRGjBhBc3MzQ4YMKR1LkqQiIjNrd/CINmD69hIWEa3A1cDxmbmpWuy+BpwNPA18BfhqZn5hF8daACwAaGgYPW3xsmtrlls913T4wT3etr29nZEjR774/Nprr2X06NGcccYZtYimfrLz3FQfnFv9cnb1aXdza21tXZ2Z03de3tNf4D0WuAI4LDNnR8Rk4M2ZeX1Pw0XEMcB1wOzM3FRdfCLws8x8qrrNLcB/BV5SwjJzBV33kjFh4pG5dJ2/P3wwaDunpcfbVioVJk+ezJgxY/jFL37B6tWruf/++znkkENqF1B7rVKp0NLSUjqGesm51S9nV5/6MreeNpkbgP8PuKz6/BHgJqBHJSwiJgC3AO/NzEe6rfoF8KaIeCVdlyPfAqza0/GGDxvC+l7cEK7B46yzzmLTpk0MGzaMq666ygImSdpv9bSENWTmzRHxVwCZ2RkR23rxOouB1wBXRwRAZ2ZOz8zvRMRXgQeATuBBqme7tG+6996evKlWkqR9X09L2JaIeA2QABHxJuA3e9opMxurD8+tfu1qm48BH+thDkmSpH1CT0vYh+l6V+TrIuI+YDTwpzVLJUmStI/bbQmLiAmZ+YvMfCAiTqDrF3oHsD4zO3a3ryRJkl7enj6s9Z+7Pb4pM3+YmT+wgEmSJO2dPZWw6PZ4Yi2DSJIk7U/2VMLyZR5LkiRpL+zpxvxjI+IZus6IDa8+pvo8M/NVNU0nSZK0j9ptCctMf7GfJElSDezpcqQkSZJqwBImSZJUgCVMkiSpAEuYJElSAZYwSZKkAixhkiRJBVjCJEmSCrCESZIkFbCnT8xXHWhsbGTUqFEMGTKEoUOHsmrVqtKRJEnSHtS0hEXEQuCDwEPAYcBxwGWZ+emdthsCrAIey8zT9nTcrR3baFx0ew0Sl9e25NQ+7XfPPffQ0NDQz2kkSVKt1PpM2PnAicDzwBHAGS+z3QXAw4C/i1KSJO0XanZPWERcA0wE7gDOyczvAR272G48cCpwXa2y7OsigpNOOolp06axYsWK0nEkSVIPRGbW7uARbcD0zNxYff5xoL375ciI+Crwt8Ao4OKXuxwZEQuABQANDaOnLV52bc1yl9R0+MG93uepp55i9OjRbN68mYsvvpiFCxdy7LHH1iDd3mlvb2fkyJGlY6iXnFt9cm71y9nVp93NrbW1dXVmTt95edEb8yPiNODJzFwdES272zYzVwArACZMPDKXrts331PQdk7LXu2/du1aOjo6aGnZu+PUQqVSGZS5tHvOrT45t/rl7OpTX+ZWusn8MXB6RLwVOAh4VUR8ITPfs7udhg8bwvo+3sC+r9myZQsvvPACo0aNYsuWLdx1110sXry4dCxJkrQHRUtYZv4V8FcA1TNhF++pgGlHTzzxBGeeeSYAnZ2dvPvd7+aUU04pnEqSJO3JgJSwiDiUro+geBXwQkRcCEzOzGcG4vX3ZRMnTmTt2rWlY0iSpF6qaQnLzMZuT8fvYdsKUKlhHEmSpEHDX1skSZJUgCVMkiSpAEuYJElSAZYwSZKkAixhkiRJBVjCJEmSCrCESZIkFWAJkyRJKsASJkmSVIAlTJIkqQBLmCRJUgGWMEmSpAIsYZIkSQVYwiRJkgoYWjqA9l5jYyOjRo1iyJAhDB06lFWrVpWOJEmS9qBmJSwiFgIfBB4CDgOOAy7LzE9X178euKnbLhOBxZm5rFaZ9mX33HMPDQ0NpWNIkqQequWZsPOBE4HngSOAM7qvzMz1QDNARAwBHgO+3pMDb+3YRuOi2/s17GDRtuTU0hEkSdIAqMk9YRFxDV1ntu4AzsnM7wEdu9nlLcBPM/Pntcizr4sITjrpJKZNm8aKFStKx5EkST0QmVmbA0e0AdMzc2P1+ceB9u2XI3fa9nPAA5n5md0cbwGwAKChYfS0xcuurUXs4poOP7jX+zz11FOMHj2azZs3c/HFF7Nw4UKOPfbYGqTbO+3t7YwcObJ0DPWSc6tPzq1+Obv6tLu5tba2rs7M6TsvL35jfkS8Ajgd+KvdbZeZK4AVABMmHplL1xWPXhNt57Ts1f5r166lo6ODlpa9O04tVCqVQZlLu+fc6pNzq1/Orj71ZW6DocnMpuss2BM93WH4sCGs994pALZs2cILL7zAqFGj2LJlC3fddReLFy8uHUuSJO3BYChhc4CVpUPUqyeeeIIzzzwTgM7OTt797ndzyimnFE4lSZL2pOYlLCIOBVYBrwJeiIgLgcmZ+UxEjABmAX9e6xz7qokTJ7J27drSMSRJUi/VrIRlZmO3p+NfZpstwGtqlUGSJGmw8tcWSZIkFWAJkyRJKsASJkmSVIAlTJIkqQBLmCRJUgGWMEmSpAIsYZIkSQVYwiRJkgqwhEmSJBVgCZMkSSrAEiZJklSAJUySJKkAS5gkSVIBljBJkqQChpYOoL3X2NjIqFGjGDJkCEOHDmXVqlWlI0mSpD2oWQmLiIXAB4GHgMOA44DLMvPT3bY5BLgOmAokMC8z/6NWmfZl99xzDw0NDaVjSJKkHqrlmbDzgROB54EjgDN2sc1y4M7M/NOIeAXwyp4ceGvHNhoX3d5vQQeTtiWnlo4gSZIGQE3uCYuIa4CJwB3AOZn5PaBjp20OBmYC1wNk5vOZ+XQt8uzrIoKTTjqJadOmsWLFitJxJElSD0Rm1ubAEW3A9MzcWH3+caB9++XIiGgGVtB1ufJYYDVwQWZueZnjLQAWADQ0jJ62eNm1NcldWtPhB/d6n6eeeorRo0ezefNmLr74YhYuXMixxx5bg3R7p729nZEjR5aOoV5ybvXJudUvZ1efdje31tbW1Zk5feflJW/MH0rXfWIfyszvRMRyYBHw0V1tnJkr6CptTJh4ZC5dt2++p6DtnJa92n/t2rV0dHTQ0rJ3x6mFSqUyKHNp95xbfXJu9cvZ1ae+zK3kR1Q8Cjyamd+pPv8qXaVMvbBlyxaeffbZFx/fddddTJ06tXAqSZK0J8VOJ2Xm4xGxISJen5nrgbfQdWlyj4YPG8J6b2AH4IknnuDMM88EoLOzk3e/+92ccsophVNJkqQ9qXkJi4hDgVXAq4AXIuJCYHJmPgN8CPhi9Z2R/wm8v9Z59jUTJ05k7dq1pWNIkqReqlkJy8zGbk/Hv8w2a4CX3KgmSZK0r/PXFkmSJBVgCZMkSSrAEiZJklSAJUySJKkAS5gkSVIBljBJkqQCLGGSJEkFWMIkSZIKsIRJkiQVYAmTJEkqwBImSZJUgCVMkiSpAEuYJElSAZYwSZKkAoaWDqC919jYyKhRoxgyZAhDhw5l1apVpSNJkqQ9qFkJi4iFwAeBh4DDgOOAyzLz09X1fwj8EzAWSGBFZi6vVZ593T333ENDQ0PpGJIkqYdqeSbsfOBE4P+2d/dBdtV1nsffX9IZN9IRVpJoKjE2vXGWhASCSRG2JqS6lbABsqvMTrlCcHiIm0IcA8uwa5Q1iltjMY5BsrW4VMfHoTTu1IIORSBCCXdQHBweEggQo6zcFSJPYXChQ5RO8t0/+sbtaUnSt83tX5/O+1XV1feee849n+5vcfn0OSf3vg68E3j/oMf3AH+emQ9HxETgoYi4KzOfONQT7+7bS8fqjYc98GhQv/ac0hEkSdIIaMk1YRFxI9AJ3AEsz8wHgL6B62Tms5n5cOP2q8A2YFor8ox1EcGZZ57J/Pnz6enpKR1HkiQNQWRma544og4syMydjfufAXr3n44ctG4HcC8wJzNfOcDzrQRWAkyaNHn+muvXtyR3aXOnHdP0Ni+++CKTJ0/m5Zdf5qqrrmLVqlWcfPLJLUj3++nt7aW9vb10DDXJuVWTc6suZ1dNB5tbd3f3Q5m5YPDy4hfmR0Q7cDNwxYEKGEBm9gA9ADM6Z+barcWjt0R9edfvtf0jjzxCX18fXV2/3/O0Qq1WG5W5dHDOrZqcW3U5u2oaztyKvkVFRIynv4B9MzNvKZmlqnbt2sWrr77629t33nknc+bMKZxKkiQdSrHDSRERwFeAbZl5XTPbThg/ju1ewA7A888/z7nnngvAnj17OP/881m6dGnhVJIk6VBaXsIi4u3Ag8BbgH0RcQUwGzgJ+BCwNSK2NFb/ZGbe3upMY0lnZyePPPJI6RiSJKlJLSthmdkx4O70N1jlh0C0av+SJEmjmR9bJEmSVIAlTJIkqQBLmCRJUgGWMEmSpAIsYZIkSQVYwiRJkgqwhEmSJBVgCZMkSSrAEiZJklSAJUySJKkAS5gkSVIBljBJkqQCLGGSJEkFWMIkSZIKaCsdQL+/jo4OJk6cyLhx42hra+PBBx8sHUmSJB1CS4+ERcSqiNgWETdHxN9HxG8i4qpB6yyNiO0R8WRErG5lnrHsnnvuYcuWLRYwSZIqotVHwi4DzgBeB94JvH/ggxExDrgBWAI8AzwQEbdm5hMHe9LdfXvpWL2xNYkLq197TukIkiRpBLTsSFhE3Ah0AncAyzPzAaBv0GqnAk9m5s8z83Xg28D7WpVprIoIzjzzTObPn09PT0/pOJIkaQhadiQsMy+NiKVAd2buPMBq04CnB9x/BljYqkxj1Q9/+EOmTZvGCy+8wJIlSzjhhBNYvHhx6ViSJOkgKnNhfkSsBFYCTJo0mTVz9xRO1Bq1Wm1Y2/3sZz8D4JRTTmHDhg3s27fvMKY6PHp7e4f986kc51ZNzq26nF01DWdupUvYDuAdA+5Pbyz7HZnZA/QAzOicmWu3lo7eGvXlXU2tv2vXLvbt28fEiRPZtWsXn/zkJ1mzZg1dXc09z0io1WqjMpcOzrlVk3OrLmdXTcOZW+km8wDwrog4nv7y9UHg/ENtNGH8OLZ7ATsAzz//POeeey4Ae/bs4fzzz2fp0qWFU0mSpEMZkRIWEW8HHgTeAuyLiCuA2Zn5SkT8GfA9YBzw1cx8fCQyjRWdnZ088sgjpWNIkqQmtbSEZWbHgLvTD7DO7cDtrcwhSZI02vixRZIkSQVYwiRJkgqwhEmSJBVgCZMkSSrAEiZJklSAJUySJKkAS5gkSVIBljBJkqQCLGGSJEkFWMIkSZIKsIRJkiQVYAmTJEkqwBImSZJUgCVMkiSpAEvYKLV3715OOeUUli1bVjqKJElqgZaVsIhYFRHbIiIj4tGI2BoRP4qIkwesU28s3xIRD7YqSxWtW7eOWbNmlY4hSZJapK2Fz30ZcAYwA9iWmS9HxFlAD7BwwHrdmbmzmSfe3beXjtUbD1/SFqtfe05T6z/zzDNs3LiRq6++muuuu65FqSRJUkktORIWETcCncAdwMLMfLnx0P3A9Fbscyy54oor+PznP89RR3m2WJKksaol/5fPzEuBX9J/lOuLAx5aQX8x++2qwJ0R8VBErGxFlqq57bbbmDJlCvPnzy8dRZIktVBkZmueOKIOLNh/qjEiuoEvAYsy86XGsmmZuSMipgB3AR/LzHsP8HwrgZUAkyZNnr/m+vUtyd0Kc6cdM+R1169fz5133sm4ceN4/fXXee211zj99NO5+uqrW5hwZPT29tLe3l46hprk3KrJuVWXs6umg82tu7v7ocxcMHj5iJSwiDgJ+A5wVmb+9ADrfwbozcwvHOq5Z3TOzKM+sO5wxm2pZq8J269Wq/GFL3yB22677TAnKqNWq9HV1VU6hprk3KrJuVWXs6umg80tIt6whLXywvz9O54B3AJ8aGABi4ijgaMy89XG7TOBzw7lOSeMH8f2YRYbSZKk0aDlJQxYAxwHfCkiAPY02uDbgO80lrUB38rMTSOQpzK6urr8a0iSpDGqZSUsMzsaNz/c+Br8+M+BbPb7YwAAE2RJREFUkwcvlyRJOhL4HgiSJEkFWMIkSZIKsIRJkiQVYAmTJEkqwBImSZJUgCVMkiSpAEuYJElSAZYwSZKkAixhkiRJBVjCJEmSCrCESZIkFWAJkyRJKsASJkmSVIAlTJIkqQBLmCRJUgGWsBHw61//mlNPPZWTTz6ZE088kU9/+tOlI0mSpMLaSuw0IlYBHwHeArQDTzUeuiUzP3uo7Xf37aVj9cYWJjy0+rXnDHndN73pTdx99920t7fT19fHokWLOOusszjttNNamFCSJI1mRUoYcBlwBjATuCozlxXKMSIigvb2dgD6+vro6+sjIgqnkiRJJY346ciIuBHoBO4AThnp/Zeyd+9e5s2bx5QpU1iyZAkLFy4sHUmSJBUUmTnyO42oAwuAOcDNwDPAL+k/Kvb4AbZZCawEmDRp8vw1168fmbAHMHfaMcParre3l0996lOsWrWK448//jCnGt16e3t/e0RQ1eHcqsm5VZezq6aDza27u/uhzFwweHmp05H7PQy8MzN7I+Js4LvAu95oxczsAXoAZnTOzLVby0avL+8a9rYPP/wwL730EhdffPHhC1QBtVqNrq6u0jHUJOdWTc6tupxdNQ1nbkWbTGa+MuD27RHxpYiYlJk7D7bdhPHj2N7EhfGlvfjii4wfP55jjz2W3bt3c9ddd/Hxj3+8dCxJklRQ0RIWEW8Hns/MjIhT6b9G7aWSmVrh2Wef5cILL2Tv3r3s27ePD3zgAyxbNqb/LYIkSTqE0qcj/wT4SETsAXYDH8wSF6m12EknncTmzZtLx5AkSaNIkRKWmR2Nm/+98SVJknRE8R3zJUmSCrCESZIkFWAJkyRJKsASJkmSVIAlTJIkqQBLmCRJUgGWMEmSpAIsYZIkSQVYwiRJkgqwhEmSJBVgCZMkSSrAEiZJklSAJUySJKkAS5gkSVIBlrBhevrpp+nu7mb27NmceOKJrFu3rnQkSZJUIUVKWESsiohtEfHNiOiKiC0R8XhE/F2JPMPR1tbG2rVreeKJJ7j//vu54YYbeOKJJ0rHkiRJFdFWaL+XAWcAvcCPgKWZ+YuImDKUjXf37aVj9cbDHqp+7TlDXnfq1KlMnToVgIkTJzJr1ix27NjB7NmzD3suSZI09ox4CYuIG4FO4A7g28AtmfkLgMx8YaTzHA71ep3NmzezcOHC0lEkSVJFRGaO/E4j6sAC4L8A44ETgYnAusz86wNssxJYCTBp0uT5a65ff9hzzZ12TNPb7N69m8svv5wLLriAxYsXH/ZMY0lvby/t7e2lY6hJzq2anFt1ObtqOtjcuru7H8rMBYOXlzodOXD/84H3AhOAv4+I+zPzp4NXzMweoAdgRufMXLv18EevL+9qav2+vj6WLVvGpZdeypVXXnnY84w1tVqNrq6u0jHUJOdWTc6tupxdNQ1nbqVL2DPAS5m5C9gVEfcCJwO/U8IGmjB+HNubuH6rFTKTFStWMGvWLAuYJElqWum3qPhbYFFEtEXEm4GFwLbCmYbkvvvu46abbuLuu+9m3rx5zJs3j9tvv710LEmSVBFFj4Rl5raI2AQ8CuwDvpyZj5XMNFSLFi2ixPV0kiRpbChSwjKzY8DtvwL+qkQOSZKkUkqfjpQkSToiWcIkSZIKsIRJkiQVYAmTJEkqwBImSZJUgCVMkiSpAEuYJElSAZYwSZKkAixhkiRJBVjCJEmSCrCESZIkFWAJkyRJKsASJkmSVIAlTJIkqQBL2DA9/fTTdHd3M3v2bE488UTWrVtXOpIkSaqQthI7jYhVwEeAnzQyzGh8/0Jmfq1Epma1tbWxdu1a3v3ud/Pqq68yf/58lixZwuzZs0tHkyRJFVCkhAGXAWcAfwock5n/JiImA9sj4puZ+frBNt7dt5eO1RsPe6j6tecMed2pU6cydepUACZOnMisWbPYsWOHJUySJA3JiJ+OjIgbgU7gDiCBiRERQDvwj8Cekc70+6rX62zevJmFCxeWjiJJkioiMnPkdxpRBxYAvwFuBU4AJgL/PjPf8BBXRKwEVgJMmjR5/prr1x/2XHOnHdP0Nrt37+byyy/nggsuYPHixYc901jS29tLe3t76RhqknOrJudWXc6umg42t+7u7ocyc8Hg5aVOR+73r4EtwHuAfwHcFRE/yMxXBq+YmT1AD8CMzpm5duvhj15f3tXU+n19fSxbtoxLL72UK6+88rDnGWtqtRpdXV2lY6hJzq2anFt1ObtqGs7cSv/ryIuBW7Lfk8BT9B8VG/UykxUrVjBr1iwLmCRJalrpI2G/AN4L/CAi3gb8S+Dnh9powvhxbG/iIvpWuO+++7jpppuYO3cu8+bNA+Bzn/scZ599dtFckiSpGkqXsP8KfD0itgIBfDwzdxbONCSLFi2ixPV0kiRpbChSwjKzY8DdM0tkkCRJKqn0NWGSJElHJEuYJElSAZYwSZKkAixhkiRJBVjCJEmSCrCESZIkFWAJkyRJKsASJkmSVIAlTJIkqQBLmCRJUgGWMEmSpAIsYZIkSQVYwiRJkgqwhEmSJBVgCRvkkksuYcqUKcyZM6d0FEmSNIYVKWERsSoitkXENyPiv0XEkxHxaES8u0SegS666CI2bdpUOoYkSRrj2grt9zLgDOAk4GPAu4CFwP9ofD+o3X176Vi9cUg7ql97TlPBFi9eTL1eb2obSZKkZo14CYuIG4FO4A7gD4GLMjOB+yPi2IiYmpnPjnQuSZKkkTTiJSwzL42IpUA38HXg6QEPPwNMA36nhEXESmAlwKRJk1kzd8+Q9ler1ZrO+Nxzz7Fr165hbasD6+3t9XdaQc6tmpxbdTm7ahrO3EqdjmxaZvYAPQAzOmfm2q1Di15f3tX0vur1OkcffTRdXc1vqwOr1Wr+TivIuVWTc6suZ1dNw5lb6X8duQN4x4D70xvLJEmSxrTSR8JuBf4sIr5N/wX5/3co14NNGD+O7U1ecD9U5513HrVajZ07dzJ9+nSuueYaVqxY0ZJ9SZKkI1fpEnY7cDbwJPAacHHZOLBhw4bSESRJ0hGgSAnLzI4Bdz9aIoMkSVJJpa8JkyRJOiJZwiRJkgqwhEmSJBVgCZMkSSrAEiZJklSAJUySJKkAS5gkSVIBljBJkqQCLGGSJEkFWMIkSZIKsIRJkiQVYAmTJEkqwBImSZJUgCVMkiSpAEvYIJdccglTpkxhzpw5paNIkqQxrEgJi4hVEbEtIjZGxHci4tGI+IeIKN58LrroIjZt2lQ6hiRJGuPaCu33MuAM4HKgNzPPjYgTgBuA9x5q4919e+lYvXFIO6pfe05TwRYvXky9Xm9qG0mSpGaNeAmLiBuBTuCOxvelAJn5k4joiIi3ZebzI51LkiRpJEVmjvxOI+rAAuBKYEJm/seIOBX4EbAwMx96g21WAisBJk2aPH/N9euHtK+5045pOt9zzz3HJz7xCb72ta81va0OrLe3l/b29tIx1CTnVk3OrbqcXTUdbG7d3d0PZeaCwctLnY7c71pgXURsAbYCm4G9b7RiZvYAPQAzOmfm2q1Di15f3tV0qHq9ztFHH01XV/Pb6sBqtZq/0wpybtXk3KrL2VXTcOZWtIRl5ivAxQAREcBTwM9LZpIkSRoJRUtYRBwLvJaZrwMfBu5tFLODmjB+HNubvOB+qM477zxqtRo7d+5k+vTpXHPNNaxYsaIl+5IkSUeu0qcjZwHfiIgEHgeKt50NGzaUjiBJko4ARUpYZnY0bu4E/rBEBkmSpJJ8x3xJkqQCLGGSJEkFWMIkSZIKsIRJkiQVYAmTJEkqwBImSZJUgCVMkiSpAEuYJElSAZYwSZKkAixhkiRJBVjCJEmSCrCESZIkFWAJkyRJKsASJkmSVIAlbJBLLrmEKVOmMGfOnNJRJEnSGNayEhYRqyJiW0RkRDwaEVsj4kcRcfKAdS6PiMci4vGIuKJVWZpx0UUXsWnTptIxJEnSGNfWwue+DDgDmAFsy8yXI+IsoAdYGBFzgP8AnAq8DmyKiNsy88lDPfHuvr10rN44pBD1a89pKvTixYup1+tNbSNJktSslhwJi4gbgU7gDmBhZr7ceOh+YHrj9izgx5n5WmbuAf4O+ONW5JEkSRptWlLCMvNS4JdAd2Z+ccBDK+gvZgCPAadHxHER8WbgbOAdrcgjSZI02rTydOQ/ERHd9JewRQCZuS0i/hK4E9gFbAH2HmT7lcBKgEmTJrNm7p4h7bdWqzWd9bnnnmPXrl3D2lYH1tvb6++0gpxbNTm36nJ21TScuY1ICYuIk4AvA2dl5kv7l2fmV4CvNNb5HPDMgZ4jM3vov56MGZ0zc+3WoUWvL+9qOm+9Xufoo4+mq6v5bXVgtVrN32kFObdqcm7V5eyqaThza3kJi4gZwC3AhzLzp4Mem5KZLzTW+WPgtKE854Tx49je5AX3Q3XeeedRq9XYuXMn06dP55prrmHFihUt2ZckSTpyjcSRsDXAccCXIgJgT2YuaDx2c0QcB/QBH83MX41AnoPasGFD6QiSJOkI0LISlpkdjZsfbny90Tqnt2r/kiRJo5nvmC9JklSAJUySJKkAS5gkSVIBljBJkqQCLGGSJEkFWMIkSZIKsIRJkiQVYAmTJEkqwBImSZJUgCVMkiSpAEuYJElSAZYwSZKkAixhkiRJBVjCJEmSCrCESZIkFWAJkyRJKsASJkmSVIAlTJIkqQBLmCRJUgGRmaUzNC0iXgW2l86hpk0CdpYOoaY5t2pybtXl7KrpYHN7Z2ZOHrywrbV5WmZ7Zi4oHULNiYgHnVv1OLdqcm7V5eyqaThz83SkJElSAZYwSZKkAqpawnpKB9CwOLdqcm7V5Nyqy9lVU9Nzq+SF+ZIkSVVX1SNhkiRJlWYJkyRJKqBSJSwilkbE9oh4MiJWl86jA4uIr0bECxHx2IBlb42IuyLiZ43v/7xkRv2uiHhHRNwTEU9ExOMRcXljubMbxSLin0XEP0TEI425XdNYfnxE/Ljxmvk/I+IPSmfV74qIcRGxOSJua9x3bqNcRNQjYmtEbImIBxvLmn6drEwJi4hxwA3AWcBs4LyImF02lQ7i68DSQctWA9/PzHcB32/c1+iyB/jzzJwNnAZ8tPHfmbMb3X4DvCczTwbmAUsj4jTgL4EvZuZM4GVgRcGMOrDLgW0D7ju3aujOzHkD3hus6dfJypQw4FTgycz8eWa+DnwbeF/hTDqAzLwX+MdBi98HfKNx+xvA+0c0lA4pM5/NzIcbt1+l/38M03B2o1r2623cHd/4SuA9wP9qLHduo1BETAfOAb7cuB84t6pq+nWySiVsGvD0gPvPNJapOt6Wmc82bj8HvK1kGB1cRHQApwA/xtmNeo1TWluAF4C7gP8N/Coz9zRW8TVzdLoe+M/Avsb943BuVZDAnRHxUESsbCxr+nWyqh9bpIrLzIwI3x9llIqIduBm4IrMfKX/j/N+zm50ysy9wLyIOBb4DnBC4Ug6hIhYBryQmQ9FRFfpPGrKoszcERFTgLsi4icDHxzq62SVjoTtAN4x4P70xjJVx/MRMRWg8f2Fwnn0BiJiPP0F7JuZeUtjsbOriMz8FXAP8K+AYyNi/x/bvmaOPn8E/NuIqNN/ic17gHU4t1EvM3c0vr9A/x89pzKM18kqlbAHgHc1/tXIHwAfBG4tnEnNuRW4sHH7QuBvC2bRG2hcj/IVYFtmXjfgIWc3ikXE5MYRMCJiArCE/uv57gH+pLGacxtlMvMTmTk9Mzvo/3/a3Zm5HOc2qkXE0RExcf9t4EzgMYbxOlmpd8yPiLPpP38+DvhqZv5F4Ug6gIjYAHQBk4DngU8D3wX+BpgB/B/gA5k5+OJ9FRQRi4AfAFv5/9eofJL+68Kc3SgVESfRfyHwOPr/uP6bzPxsRHTSf4TlrcBm4ILM/E25pDqQxunIqzJzmXMb3Rrz+U7jbhvwrcz8i4g4jiZfJytVwiRJksaKKp2OlCRJGjMsYZIkSQVYwiRJkgqwhEmSJBVgCZMkSSrAd8yXNCZExF7631pjv/dnZr1QHEk6JN+iQtKYEBG9mdk+gvtrG/D5fpLUNE9HSjoiRMTUiLg3IrZExGMRcXpj+dKIeDgiHomI7zeWvTUivhsRj0bE/Y03QyUiPhMRN0XEfcBNjXeqvzkiHmh8/VHBH1FSxXg6UtJYMSEitjRuP5WZ5w56/Hzge413th4HvDkiJgPrgcWZ+VREvLWx7jXA5sx8f0S8B/hrYF7jsdn0f3jv7oj4FvDFzPxhRMwAvgfMauHPKGkMsYRJGit2Z+a8gzz+APDVxgeUfzcztzQ+KubezHwKYMBHjCwC/l1j2d0RcVxEvKXx2K2Zubtx+wxgdv9HbgLwlohoz8zew/djSRqrLGGSjgiZeW9ELAbOAb4eEdcBLw/jqXYNuH0UcFpm/vpwZJR0ZPGaMElHhIh4J/B8Zq4Hvgy8G7gfWBwRxzfW2X868gfA8sayLmBnZr7yBk97J/CxAfs42JE4SfonPBIm6UjRBfyniOgDeoE/zcwXI2IlcEtEHAW8ACwBPkP/qctHgdeACw/wnKuAGxrrtQH3Ape29KeQNGb4FhWSJEkFeDpSkiSpAEuYJElSAZYwSZKkAixhkiRJBVjCJEmSCrCESZIkFWAJkyRJKuD/AZmyoq0bhztoAAAAAElFTkSuQmCC\n"
          },
          "metadata": {
            "needs_background": "light"
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "\n",
        "\n",
        "---\n",
        "### 07 LightGBM\n",
        "* 부스팅 계열 알고리즘에서 각광 받고 있다.\n",
        "* XGBoost보다 학습에 걸리는 시간이 훨씬 적고 메모리 사용량도 적다. \n",
        "* 단점은 적은 데이터 세트에 적용할 경우 과적합이 발생하기 쉽다. (일반적으로 만건 이하의 데이터 세트정도가 적은 데이터 세트)\n",
        "* GBM계열의 트리 분할 방법과 다르게 **리프 중심 트리 분할 방식을 사용함.**\n",
        "* 최대 손실값을 가지는 리프 노드를 지속적으로 분할해 생성된 규칙 트리는 학습을 반복할수록 결국은 균형 트리 분할 방식보다 예측 오류 손실을 최소화 할 수 있다.\n",
        "\n",
        "정리하면, \n",
        "1. 더빠른 학습과 예측 수행시간.\n",
        "\n",
        "2. 더 작은 메모리 사용량\n",
        "\n",
        "3. 카테고리형 피처의 자동변환과 최적 분할( 원-핫 인코딩 등을 사용하지 않고도 카테고리형 피처를 최적으로 변환하고 이에 따른 노드 분할 수행)\n",
        "\n",
        "패키지명 : LightGBM\n",
        "분류를 위한 클랫 : LGBMClassifier\n",
        "회귀를 위한 클래스: LGBMRegressor\n",
        "\n"
      ],
      "metadata": {
        "id": "qL24s05yyPPm"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### LightGBM 하이퍼 파라미터\n",
        "\n",
        "* 주요 파라미터\n",
        "  * num_iterations: 반복수행하려는 트리의 개수를 지정 크게 지정할수록 에측 성능이 높아질 수 있으나, 너무 크면 과적합으로 성능 저하. n_estimator와 같음.\n",
        "  * learning_rate: 0에서 1사이의 값을 지정해 부스팅 스텝을 반복적으로 수행할 때 업데이트되는 학습률 값. 일반적으로 n_estimator를 크게 하고, learning_rate를 작게 해서 예측 성능을 향상시킬 수 있으나, 과적합 이슈와 학습 시간이 길어지는 영향이 있을 수도 있음.\n",
        "  * max_depth: 트리기반 알고리즘의 max_depth와 같음.\n",
        "  * min_data_in_deaf [default=20]: 최종 결정 크래스인 리프 노드가 되기 위해서 최소한으로 필요한 레코드 수이며, 과적합을 제어하기 위한 파라미터. 사이킷런 래퍼 LightGBM 클래스인 LightGBMClassifier에서는 min_child_samples 파라미터로 이름이 변경.\n",
        "  * num_leaves: 하나의 트리가 가질 수 있는 최대 리프 개수\n",
        "  * boosting: 부스팅 트리를 생성하는 알고리즘 기술\n",
        "     * gbdt: 일반적인 그래디언트 부스팅 결정 트리\n",
        "     * rf: 랜덤 포레스트\n",
        "  * bagging_fraction [default=1.0]:트리가 커져 과적합 되는 것을 제어하기 위해 데이터를 샘플링하는 비율 지정\n",
        "  * feature_fraction [default=1.0]: 개별 트리를 학습할 때마다 무작위로 선택하는 피처의 비율. 과적합을 막기 위해 사용됨\n",
        "  * lambda_l2: 앞서와 동일\n",
        "  * labmda_l1: 앞서와 동일\n",
        "\n",
        "* Learning Task파라미터\n",
        "  * objective: 최솟값을 가져야 할 손실함수를 정의.회귀, 다중 클래스 분류, 이진 분류인지에 따라서 손실함수가 지정됨.\n",
        "\n",
        "### 하이퍼 파라미터 튜닝 방안\n",
        "num_leaves의 개수를 중심으로 min_child_sampels(min_data_in_leaf), max_depth를 함께 조정하면서 모델의 복잡도를 줄이는 것이 기본 튜닝 방안\n",
        "\n",
        "  * num_leaves: 개별 트리가 가질 수 있는 최대 리프의 개수, LightGBM 모델의 복잡도를 제어하는 주요 파라미터. 일반적으로 num_leaves의 개수를 높이면 정확도가 높아지지만, 반대로 트리의 깊이가 깊어지고 모델이 복잡도가 커져 과적합 영향도가 커진다.\n",
        "  * min_data_in_leaf: 사이킷런 래퍼 클래스에서는 min_child_samples. 과적합을 개선하기 위한 중요 파라미터. num_leaves와 학습 데이터의 크기에 따라 달라지지만, 보통 큰 값으로 설정하면 트리가 깊어지는 것을 방지한다.\n",
        "  * max_depth는 명시적으로 깊이의 크기를 제한한다. num_leaves,min_data_in_leaf와 결합해 과적합을 개선하는 데 사용.\n",
        "\n",
        "learning_rate를 작게 하면서 n_estimators를 크게 하는 것은 부스팅 계열 튜닝에서 가장 기본적인 튜닝 방안이므로 이를 적용하는 것도 좋다.\n",
        "n_estimators를 너무 크게 하는 것은 과적합으로 오히려 성능이 저하될 수 있음을 유념해야 한다.\n",
        "\n",
        "이밖에 과적합을 제어하기 위해서 reg_lambda, reg_alpha와 같은 regularization을 적용하거나 학습 데이터에 사용할 피처의 개수나 데이터 샘플링 레코드 개수를 줄이기 위해 colsample_bytree, subsample 파라미터를 적용할 수 있다.\n",
        "\n",
        "### 파이썬 래퍼 LightGBM과 사이킷런 래퍼 XGBoost, LightGBM 하이퍼 파라미터 비교\n",
        "\n",
        "LightGBM은 사이킷런과 호환하기 위해 분류를 위한 LGBMClassifier와 회귀를 위한 LGBMRegressor 클래스를 래퍼 클래스로 생성\n",
        "사이킷런 래퍼 클래스를 제공한 XGBoost는 사이킷런 하이퍼 명명 규칙에 따라 자신의 하이퍼 파라미터를 변경\n",
        "\n",
        "LightGBM은 XGBoost와 많은 유사한 기능이 있었기에 사이킷런 LightGBM의 하이퍼 파라미터를 사이킷런 XGboost에 맞춰서 변경. 이 때문에 사이킷런 래퍼 LightGBM 클래스와 사이킷런 래퍼 XGBoost 클래스는 많은 하이퍼 파라미터가 똑같다.\n",
        "\n",
        "\n",
        "---\n",
        "\n",
        "\n",
        "### LightGBM 적용 - 위스콘신 유방암 예측"
      ],
      "metadata": {
        "id": "iFb9W56EyPaT"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# LightGBM의 파이썬 패키지인 lightgbm 에서 LGBMClassifier 임포트\n",
        "from lightgbm import LGBMClassifier\n",
        "\n",
        "import pandas as pd\n",
        "import numpy as np\n",
        "from sklearn.datasets import load_breast_cancer\n",
        "from sklearn.model_selection import train_test_split\n",
        "\n",
        "dataset = load_breast_cancer()\n",
        "ftr = dataset.data\n",
        "target = dataset.target\n",
        "\n",
        "# 전체 데이터 중 80%는 학습용 데이터, 20%는 테스트용 데이터 추출\n",
        "X_train, X_test, y_train, y_test = train_test_split(ftr, target, test_size=0.2, random_state=156)\n",
        "\n",
        "# 앞서 XGBoost와 동일하게 n_estimators는 400 설정\n",
        "lgbm_wrapper = LGBMClassifier(n_estimators=400)\n",
        "\n",
        "# LightGBM도 XGBoost와 동일하게 조기 중단 수행 가능. \n",
        "evals = [(X_test, y_test)]\n",
        "lgbm_wrapper.fit(X_train, y_train, early_stopping_rounds=100, eval_metric=\"logloss\",\n",
        "                 eval_set=evals, verbose=True)\n",
        "preds = lgbm_wrapper.predict(X_test)\n",
        "pred_proba = lgbm_wrapper.predict_proba(X_test)[:, 1]"
      ],
      "metadata": {
        "id": "PgZLrcGFyPez",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "d82337dc-2d8c-4dcd-9040-dc398487a21b"
      },
      "execution_count": 21,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "[1]\tvalid_0's binary_logloss: 0.565079\tvalid_0's binary_logloss: 0.565079\n",
            "Training until validation scores don't improve for 100 rounds.\n",
            "[2]\tvalid_0's binary_logloss: 0.507451\tvalid_0's binary_logloss: 0.507451\n",
            "[3]\tvalid_0's binary_logloss: 0.458489\tvalid_0's binary_logloss: 0.458489\n",
            "[4]\tvalid_0's binary_logloss: 0.417481\tvalid_0's binary_logloss: 0.417481\n",
            "[5]\tvalid_0's binary_logloss: 0.385507\tvalid_0's binary_logloss: 0.385507\n",
            "[6]\tvalid_0's binary_logloss: 0.355846\tvalid_0's binary_logloss: 0.355846\n",
            "[7]\tvalid_0's binary_logloss: 0.330897\tvalid_0's binary_logloss: 0.330897\n",
            "[8]\tvalid_0's binary_logloss: 0.306923\tvalid_0's binary_logloss: 0.306923\n",
            "[9]\tvalid_0's binary_logloss: 0.28776\tvalid_0's binary_logloss: 0.28776\n",
            "[10]\tvalid_0's binary_logloss: 0.26917\tvalid_0's binary_logloss: 0.26917\n",
            "[11]\tvalid_0's binary_logloss: 0.250954\tvalid_0's binary_logloss: 0.250954\n",
            "[12]\tvalid_0's binary_logloss: 0.23847\tvalid_0's binary_logloss: 0.23847\n",
            "[13]\tvalid_0's binary_logloss: 0.225865\tvalid_0's binary_logloss: 0.225865\n",
            "[14]\tvalid_0's binary_logloss: 0.215076\tvalid_0's binary_logloss: 0.215076\n",
            "[15]\tvalid_0's binary_logloss: 0.205996\tvalid_0's binary_logloss: 0.205996\n",
            "[16]\tvalid_0's binary_logloss: 0.196091\tvalid_0's binary_logloss: 0.196091\n",
            "[17]\tvalid_0's binary_logloss: 0.186395\tvalid_0's binary_logloss: 0.186395\n",
            "[18]\tvalid_0's binary_logloss: 0.17942\tvalid_0's binary_logloss: 0.17942\n",
            "[19]\tvalid_0's binary_logloss: 0.174727\tvalid_0's binary_logloss: 0.174727\n",
            "[20]\tvalid_0's binary_logloss: 0.168563\tvalid_0's binary_logloss: 0.168563\n",
            "[21]\tvalid_0's binary_logloss: 0.165432\tvalid_0's binary_logloss: 0.165432\n",
            "[22]\tvalid_0's binary_logloss: 0.160356\tvalid_0's binary_logloss: 0.160356\n",
            "[23]\tvalid_0's binary_logloss: 0.155508\tvalid_0's binary_logloss: 0.155508\n",
            "[24]\tvalid_0's binary_logloss: 0.151598\tvalid_0's binary_logloss: 0.151598\n",
            "[25]\tvalid_0's binary_logloss: 0.149861\tvalid_0's binary_logloss: 0.149861\n",
            "[26]\tvalid_0's binary_logloss: 0.149873\tvalid_0's binary_logloss: 0.149873\n",
            "[27]\tvalid_0's binary_logloss: 0.147032\tvalid_0's binary_logloss: 0.147032\n",
            "[28]\tvalid_0's binary_logloss: 0.145077\tvalid_0's binary_logloss: 0.145077\n",
            "[29]\tvalid_0's binary_logloss: 0.139891\tvalid_0's binary_logloss: 0.139891\n",
            "[30]\tvalid_0's binary_logloss: 0.137416\tvalid_0's binary_logloss: 0.137416\n",
            "[31]\tvalid_0's binary_logloss: 0.138393\tvalid_0's binary_logloss: 0.138393\n",
            "[32]\tvalid_0's binary_logloss: 0.135948\tvalid_0's binary_logloss: 0.135948\n",
            "[33]\tvalid_0's binary_logloss: 0.132968\tvalid_0's binary_logloss: 0.132968\n",
            "[34]\tvalid_0's binary_logloss: 0.129697\tvalid_0's binary_logloss: 0.129697\n",
            "[35]\tvalid_0's binary_logloss: 0.131102\tvalid_0's binary_logloss: 0.131102\n",
            "[36]\tvalid_0's binary_logloss: 0.129382\tvalid_0's binary_logloss: 0.129382\n",
            "[37]\tvalid_0's binary_logloss: 0.128044\tvalid_0's binary_logloss: 0.128044\n",
            "[38]\tvalid_0's binary_logloss: 0.127325\tvalid_0's binary_logloss: 0.127325\n",
            "[39]\tvalid_0's binary_logloss: 0.128091\tvalid_0's binary_logloss: 0.128091\n",
            "[40]\tvalid_0's binary_logloss: 0.129045\tvalid_0's binary_logloss: 0.129045\n",
            "[41]\tvalid_0's binary_logloss: 0.127023\tvalid_0's binary_logloss: 0.127023\n",
            "[42]\tvalid_0's binary_logloss: 0.129314\tvalid_0's binary_logloss: 0.129314\n",
            "[43]\tvalid_0's binary_logloss: 0.129175\tvalid_0's binary_logloss: 0.129175\n",
            "[44]\tvalid_0's binary_logloss: 0.128212\tvalid_0's binary_logloss: 0.128212\n",
            "[45]\tvalid_0's binary_logloss: 0.126664\tvalid_0's binary_logloss: 0.126664\n",
            "[46]\tvalid_0's binary_logloss: 0.127662\tvalid_0's binary_logloss: 0.127662\n",
            "[47]\tvalid_0's binary_logloss: 0.126108\tvalid_0's binary_logloss: 0.126108\n",
            "[48]\tvalid_0's binary_logloss: 0.129371\tvalid_0's binary_logloss: 0.129371\n",
            "[49]\tvalid_0's binary_logloss: 0.129573\tvalid_0's binary_logloss: 0.129573\n",
            "[50]\tvalid_0's binary_logloss: 0.130876\tvalid_0's binary_logloss: 0.130876\n",
            "[51]\tvalid_0's binary_logloss: 0.131366\tvalid_0's binary_logloss: 0.131366\n",
            "[52]\tvalid_0's binary_logloss: 0.131336\tvalid_0's binary_logloss: 0.131336\n",
            "[53]\tvalid_0's binary_logloss: 0.13208\tvalid_0's binary_logloss: 0.13208\n",
            "[54]\tvalid_0's binary_logloss: 0.13306\tvalid_0's binary_logloss: 0.13306\n",
            "[55]\tvalid_0's binary_logloss: 0.132342\tvalid_0's binary_logloss: 0.132342\n",
            "[56]\tvalid_0's binary_logloss: 0.134836\tvalid_0's binary_logloss: 0.134836\n",
            "[57]\tvalid_0's binary_logloss: 0.135208\tvalid_0's binary_logloss: 0.135208\n",
            "[58]\tvalid_0's binary_logloss: 0.13328\tvalid_0's binary_logloss: 0.13328\n",
            "[59]\tvalid_0's binary_logloss: 0.134147\tvalid_0's binary_logloss: 0.134147\n",
            "[60]\tvalid_0's binary_logloss: 0.134549\tvalid_0's binary_logloss: 0.134549\n",
            "[61]\tvalid_0's binary_logloss: 0.133202\tvalid_0's binary_logloss: 0.133202\n",
            "[62]\tvalid_0's binary_logloss: 0.135726\tvalid_0's binary_logloss: 0.135726\n",
            "[63]\tvalid_0's binary_logloss: 0.134011\tvalid_0's binary_logloss: 0.134011\n",
            "[64]\tvalid_0's binary_logloss: 0.131493\tvalid_0's binary_logloss: 0.131493\n",
            "[65]\tvalid_0's binary_logloss: 0.134114\tvalid_0's binary_logloss: 0.134114\n",
            "[66]\tvalid_0's binary_logloss: 0.134525\tvalid_0's binary_logloss: 0.134525\n",
            "[67]\tvalid_0's binary_logloss: 0.131412\tvalid_0's binary_logloss: 0.131412\n",
            "[68]\tvalid_0's binary_logloss: 0.12878\tvalid_0's binary_logloss: 0.12878\n",
            "[69]\tvalid_0's binary_logloss: 0.129571\tvalid_0's binary_logloss: 0.129571\n",
            "[70]\tvalid_0's binary_logloss: 0.129671\tvalid_0's binary_logloss: 0.129671\n",
            "[71]\tvalid_0's binary_logloss: 0.129935\tvalid_0's binary_logloss: 0.129935\n",
            "[72]\tvalid_0's binary_logloss: 0.128951\tvalid_0's binary_logloss: 0.128951\n",
            "[73]\tvalid_0's binary_logloss: 0.128977\tvalid_0's binary_logloss: 0.128977\n",
            "[74]\tvalid_0's binary_logloss: 0.127121\tvalid_0's binary_logloss: 0.127121\n",
            "[75]\tvalid_0's binary_logloss: 0.128107\tvalid_0's binary_logloss: 0.128107\n",
            "[76]\tvalid_0's binary_logloss: 0.129796\tvalid_0's binary_logloss: 0.129796\n",
            "[77]\tvalid_0's binary_logloss: 0.131663\tvalid_0's binary_logloss: 0.131663\n",
            "[78]\tvalid_0's binary_logloss: 0.132483\tvalid_0's binary_logloss: 0.132483\n",
            "[79]\tvalid_0's binary_logloss: 0.131578\tvalid_0's binary_logloss: 0.131578\n",
            "[80]\tvalid_0's binary_logloss: 0.130352\tvalid_0's binary_logloss: 0.130352\n",
            "[81]\tvalid_0's binary_logloss: 0.129895\tvalid_0's binary_logloss: 0.129895\n",
            "[82]\tvalid_0's binary_logloss: 0.131587\tvalid_0's binary_logloss: 0.131587\n",
            "[83]\tvalid_0's binary_logloss: 0.132763\tvalid_0's binary_logloss: 0.132763\n",
            "[84]\tvalid_0's binary_logloss: 0.133677\tvalid_0's binary_logloss: 0.133677\n",
            "[85]\tvalid_0's binary_logloss: 0.137552\tvalid_0's binary_logloss: 0.137552\n",
            "[86]\tvalid_0's binary_logloss: 0.136055\tvalid_0's binary_logloss: 0.136055\n",
            "[87]\tvalid_0's binary_logloss: 0.137904\tvalid_0's binary_logloss: 0.137904\n",
            "[88]\tvalid_0's binary_logloss: 0.139524\tvalid_0's binary_logloss: 0.139524\n",
            "[89]\tvalid_0's binary_logloss: 0.138434\tvalid_0's binary_logloss: 0.138434\n",
            "[90]\tvalid_0's binary_logloss: 0.138402\tvalid_0's binary_logloss: 0.138402\n",
            "[91]\tvalid_0's binary_logloss: 0.139384\tvalid_0's binary_logloss: 0.139384\n",
            "[92]\tvalid_0's binary_logloss: 0.139642\tvalid_0's binary_logloss: 0.139642\n",
            "[93]\tvalid_0's binary_logloss: 0.138006\tvalid_0's binary_logloss: 0.138006\n",
            "[94]\tvalid_0's binary_logloss: 0.141612\tvalid_0's binary_logloss: 0.141612\n",
            "[95]\tvalid_0's binary_logloss: 0.142319\tvalid_0's binary_logloss: 0.142319\n",
            "[96]\tvalid_0's binary_logloss: 0.145095\tvalid_0's binary_logloss: 0.145095\n",
            "[97]\tvalid_0's binary_logloss: 0.141542\tvalid_0's binary_logloss: 0.141542\n",
            "[98]\tvalid_0's binary_logloss: 0.144993\tvalid_0's binary_logloss: 0.144993\n",
            "[99]\tvalid_0's binary_logloss: 0.147936\tvalid_0's binary_logloss: 0.147936\n",
            "[100]\tvalid_0's binary_logloss: 0.147432\tvalid_0's binary_logloss: 0.147432\n",
            "[101]\tvalid_0's binary_logloss: 0.149689\tvalid_0's binary_logloss: 0.149689\n",
            "[102]\tvalid_0's binary_logloss: 0.153542\tvalid_0's binary_logloss: 0.153542\n",
            "[103]\tvalid_0's binary_logloss: 0.154556\tvalid_0's binary_logloss: 0.154556\n",
            "[104]\tvalid_0's binary_logloss: 0.155458\tvalid_0's binary_logloss: 0.155458\n",
            "[105]\tvalid_0's binary_logloss: 0.159357\tvalid_0's binary_logloss: 0.159357\n",
            "[106]\tvalid_0's binary_logloss: 0.160176\tvalid_0's binary_logloss: 0.160176\n",
            "[107]\tvalid_0's binary_logloss: 0.163369\tvalid_0's binary_logloss: 0.163369\n",
            "[108]\tvalid_0's binary_logloss: 0.163494\tvalid_0's binary_logloss: 0.163494\n",
            "[109]\tvalid_0's binary_logloss: 0.161111\tvalid_0's binary_logloss: 0.161111\n",
            "[110]\tvalid_0's binary_logloss: 0.16332\tvalid_0's binary_logloss: 0.16332\n",
            "[111]\tvalid_0's binary_logloss: 0.1663\tvalid_0's binary_logloss: 0.1663\n",
            "[112]\tvalid_0's binary_logloss: 0.166363\tvalid_0's binary_logloss: 0.166363\n",
            "[113]\tvalid_0's binary_logloss: 0.169834\tvalid_0's binary_logloss: 0.169834\n",
            "[114]\tvalid_0's binary_logloss: 0.166509\tvalid_0's binary_logloss: 0.166509\n",
            "[115]\tvalid_0's binary_logloss: 0.165823\tvalid_0's binary_logloss: 0.165823\n",
            "[116]\tvalid_0's binary_logloss: 0.167059\tvalid_0's binary_logloss: 0.167059\n",
            "[117]\tvalid_0's binary_logloss: 0.169086\tvalid_0's binary_logloss: 0.169086\n",
            "[118]\tvalid_0's binary_logloss: 0.170012\tvalid_0's binary_logloss: 0.170012\n",
            "[119]\tvalid_0's binary_logloss: 0.168639\tvalid_0's binary_logloss: 0.168639\n",
            "[120]\tvalid_0's binary_logloss: 0.16907\tvalid_0's binary_logloss: 0.16907\n",
            "[121]\tvalid_0's binary_logloss: 0.16918\tvalid_0's binary_logloss: 0.16918\n",
            "[122]\tvalid_0's binary_logloss: 0.170233\tvalid_0's binary_logloss: 0.170233\n",
            "[123]\tvalid_0's binary_logloss: 0.165655\tvalid_0's binary_logloss: 0.165655\n",
            "[124]\tvalid_0's binary_logloss: 0.16695\tvalid_0's binary_logloss: 0.16695\n",
            "[125]\tvalid_0's binary_logloss: 0.170955\tvalid_0's binary_logloss: 0.170955\n",
            "[126]\tvalid_0's binary_logloss: 0.168916\tvalid_0's binary_logloss: 0.168916\n",
            "[127]\tvalid_0's binary_logloss: 0.172316\tvalid_0's binary_logloss: 0.172316\n",
            "[128]\tvalid_0's binary_logloss: 0.173734\tvalid_0's binary_logloss: 0.173734\n",
            "[129]\tvalid_0's binary_logloss: 0.174309\tvalid_0's binary_logloss: 0.174309\n",
            "[130]\tvalid_0's binary_logloss: 0.176719\tvalid_0's binary_logloss: 0.176719\n",
            "[131]\tvalid_0's binary_logloss: 0.176591\tvalid_0's binary_logloss: 0.176591\n",
            "[132]\tvalid_0's binary_logloss: 0.180168\tvalid_0's binary_logloss: 0.180168\n",
            "[133]\tvalid_0's binary_logloss: 0.179856\tvalid_0's binary_logloss: 0.179856\n",
            "[134]\tvalid_0's binary_logloss: 0.179251\tvalid_0's binary_logloss: 0.179251\n",
            "[135]\tvalid_0's binary_logloss: 0.18315\tvalid_0's binary_logloss: 0.18315\n",
            "[136]\tvalid_0's binary_logloss: 0.184656\tvalid_0's binary_logloss: 0.184656\n",
            "[137]\tvalid_0's binary_logloss: 0.187475\tvalid_0's binary_logloss: 0.187475\n",
            "[138]\tvalid_0's binary_logloss: 0.188721\tvalid_0's binary_logloss: 0.188721\n",
            "[139]\tvalid_0's binary_logloss: 0.188542\tvalid_0's binary_logloss: 0.188542\n",
            "[140]\tvalid_0's binary_logloss: 0.18817\tvalid_0's binary_logloss: 0.18817\n",
            "[141]\tvalid_0's binary_logloss: 0.185899\tvalid_0's binary_logloss: 0.185899\n",
            "[142]\tvalid_0's binary_logloss: 0.185452\tvalid_0's binary_logloss: 0.185452\n",
            "[143]\tvalid_0's binary_logloss: 0.186084\tvalid_0's binary_logloss: 0.186084\n",
            "[144]\tvalid_0's binary_logloss: 0.185302\tvalid_0's binary_logloss: 0.185302\n",
            "[145]\tvalid_0's binary_logloss: 0.187856\tvalid_0's binary_logloss: 0.187856\n",
            "[146]\tvalid_0's binary_logloss: 0.190334\tvalid_0's binary_logloss: 0.190334\n",
            "[147]\tvalid_0's binary_logloss: 0.192769\tvalid_0's binary_logloss: 0.192769\n",
            "Early stopping, best iteration is:\n",
            "[47]\tvalid_0's binary_logloss: 0.126108\tvalid_0's binary_logloss: 0.126108\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "get_clf_eval(y_test, preds, pred_proba)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "7AGURXzQNIBS",
        "outputId": "f69f2e2b-b955-4b1b-c496-6d6b737e5050"
      },
      "execution_count": 22,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "오차 행렬\n",
            "[[33  4]\n",
            " [ 2 75]]\n",
            "정확도: 0.9474, 정밀도: 0.9494, 재현율: 0.9740,    F1: 0.9615, roc_auc:0.9926\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "-> 정확도가 약 94.74%이다. 앞 예제의 XGBoost인 경우보다는 작지만 워낙 데이터 세트 크기가 작아 성능 비교는 큰 의미가 없다. \n"
      ],
      "metadata": {
        "id": "1rq2d0BGyPkP"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "#피처 중요도 시각화 제공\n",
        "from lightgbm import plot_importance\n",
        "import matplotlib.pyplot as plt\n",
        "%matplotlib inline\n",
        "\n",
        "fig, ax = plt.subplots(figsize=(10, 12))\n",
        "plot_importance(lgbm_wrapper, ax=ax)"
      ],
      "metadata": {
        "id": "DYiVpqu-yPpf",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 747
        },
        "outputId": "82418d8e-98f8-4724-d4e7-286c1bc63251"
      },
      "execution_count": 23,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<matplotlib.axes._subplots.AxesSubplot at 0x7fe5cc31ad50>"
            ]
          },
          "metadata": {},
          "execution_count": 23
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 720x864 with 1 Axes>"
            ],
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAokAAALJCAYAAADYj7lNAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nOzde3xV5Zn//c9XVDxAsU4EEaspHipgIAjC00ekYVqsB3SKtaKiLT7lR63TR7AIOtpSZoaxPCAjVGwV/TlYp0XHGVGLyuMBt6dpHQWJESw4alrwgIcOahAhCdfvj7WCm+wkhMPOzuH7fr3y6tr3Ya1rX5Zwcd9r7a2IwMzMzMws2z6FDsDMzMzMWh8XiWZmZmaWw0WimZmZmeVwkWhmZmZmOVwkmpmZmVkOF4lmZmZmlsNFoplZgUm6VtLthY7DzCyb/DmJZtaWSaoEegC1Wc3HR8Tbe3jO8RHx+J5F1/ZImg4cGxEXFzoWMyssrySaWXtwdkR0yfrZ7QJxb5C0byGvv7vaatxmlh8uEs2sXZLUTdL/lvSOpLckzZDUKe07RtIySR9K+kDSbyQdkvbdBRwF/E5SlaSpksokra93/kpJ30iPp0v6d0n/KuljYFxT128g1umS/jU9LpYUki6VtE7S/0i6TNLJkl6WtFHS/Ky54yQ9J2m+pI8k/VHS17P6j5D0oKS/SPpvSf+r3nWz474MuBYYk7738nTcpZJelfSJpDck/SDrHGWS1kuaLOm99P1emtV/oKQ5kv6UxvespAPTvv9L0n+m76lcUtlu/cc2s7xwkWhm7dVCoAY4FhgInAaMT/sE/Bw4AugDfAmYDhARlwB/5vPVyVnNvN7fAP8OHAL8ZifXb46hwHHAGGAucB3wDaAfcL6kr9Ub+zpQBPwMuE/SoWnf3cD69L2eB1wv6a8bift/A9cD96TvfUA65j1gFPAF4FLgRkknZZ3jcKAb0Av4PnCzpC+mfTcAg4D/GzgUmApsk9QLeAiYkbZfBfyHpMN2IUdmlkcuEs2sPbg/XY3aKOl+ST2AM4FJEbEpIt4DbgQuAIiI/46IxyJiS0S8D/wz8LXGT98sv4+I+yNiG0kx1ej1m+kfI+KziHgU2AQsioj3IuIt4BmSwrPOe8DciKiOiHuANcBZkr4EnAJcnZ5rJXA78N2G4o6IzQ0FEhEPRcTrkXgKeBQ4NWtINfAP6fUfBqqAr0jaB/h/gIkR8VZE1EbEf0bEFuBi4OGIeDi99mPAi2nezKwV8P0nZtYefCv7IRNJQ4D9gHck1TXvA6xL+3sA80gKna5p3//sYQzrso6Pbur6zbQh63hzA6+7ZL1+K3Z8CvFPJCuHRwB/iYhP6vUNbiTuBkk6g2SF8niS93EQUJE15MOIqMl6/WkaXxFwAMkqZ31HA9+RdHZW237AkzuLx8xahotEM2uP1gFbgKJ6xUud64EASiLiL5K+BczP6q//sQ+bSAojANJ7C+tvi2bP2dn197ZekpRVKB4FPAi8DRwqqWtWoXgU8FbW3PrvdYfXkjoD/0Gy+vhARFRLup9ky35nPgA+A44Byuv1rQPuioj/lTPLzFoFbzebWbsTEe+QbInOkfQFSfukD6vUbSl3JdkS/Si9N25KvVNsAHpnvV4LHCDpLEn7AT8BOu/B9fe27sAVkvaT9B2S+ywfjoh1wH8CP5d0gKT+JPcM/msT59oAFKdbxQD7k7zX94GadFXxtOYElW693wH8c/oATSdJX00Lz38Fzpb0zbT9gPQhmCN3/e2bWT64SDSz9uq7JAXOapKt5H8HeqZ9fw+cBHxE8vDEffXm/hz4SXqP41UR8RFwOcn9fG+RrCyup2lNXX9ve57kIZcPgH8CzouID9O+C4FiklXFxcDPdvL5j/em//uhpBXpCuQVwL+RvI+LSFYpm+sqkq3pF4C/AP8fsE9awP4NydPU75OsLE7Bfy+ZtRr+MG0zszZM0jiSD/4eVuhYzKx98b/YzMzMzCyHi0QzMzMzy+HtZjMzMzPL4ZVEMzMzM8vhz0ncyw455JA49thjCx1GwW3atImDDz640GEUnPOQcB4SzoNzUMd5SDgPiULmYfny5R9ERINfh+kicS/r0aMHL774YqHDKLhMJkNZWVmhwyg45yHhPCScB+egjvOQcB4ShcyDpD811uftZjMzMzPL4SLRzMzMzHK4SDQzMzOzHC4SzczMzCyHi0QzMzMzy+Ei0czMzMxyuEg0MzMzsxwuEs3MzMwsh4tEMzMzM8vhItHMzMzMcrhINDMzM7McLhLNzMzMLIeLRDMzMzPL4SLRzMzMzHK4SDQzMzOzHPsWOgAzMzOz9mTNmjWMGTNm++s33niDf/iHf+D3v/89a9asAWDjxo0ccsghrFy5slBh7pSLRDMzM7O96Ctf+cr24q+2tpZevXoxevRoJk2atH3M5MmT6datW6FCbBZFRMtfVDocmAucDGwENgCTImJtA2OLgSURcWJLxphe+5+A7wJfjIguzZlzVO9jY5/z5+U3sDZgckkNcyr8bxDnIeE8JJwH56CO85Boq3monHlWs8c++uij/P3f/z3PPffc9raI4KijjmLZsmUcd9xxZDIZysrK8hDpzklaHhGDG+pr8XsSJQlYDGQi4piIGAT8HdCjpWNpht8BQwodhJmZmbVNd999NxdeeOEObc888ww9evTguOOOK1BUzVOIB1dGANURcUtdQ0SUA89Kmi3pFUkVksbUnyhpnKT5Wa+XSCpLj6vS+askPS5piKSMpDcknZM1/z5JSyW9JmlWU4FGxB8i4p299L7NzMysA9m6dSsPPvgg3/nOd3ZoX7RoUU7h2BoVYo33RGB5A+3nAqXAAKAIeEHS07tw3oOBZRExRdJiYAYwEugL3Ak8mI4rBQYCW4A1km6KiHW79U5SkiYAEwCKig5jWknNnpyuXehxYLKN0NE5DwnnIeE8OAd1nIdEW81DJpNp1rhnn32WL3/5y7z66qu8+uqrQHKP4j333MOtt966/TxVVVXNPmdLak03AgwDFkVELbBB0lMk9yy+3Mz5W4Gl6XEFsCUiqiVVAMVZ456IiI8AJK0Gjgb2qEiMiAXAAkjuSWyL91fsbW31PpO9zXlIOA8J58E5qOM8JNpqHirHljVr3C233MLll1++w/2GS5cupaSkZIfVxULek9iUQvyXWQWct5tza9hxi/yArOPq+PwpnG0kK4VExDZJ2e9zS9ZxLXs5Bwfu14k1u3BDa3uVyWSa/YeoPXMeEs5DwnlwDuo4D4n2nIdNmzbx2GOPceutt+7Q3tA9iq1VIe5JXAZ0TrdoAZDUn+Qp5zGSOkk6DBgO/Fe9uZVAqaR9JH0JP1RiZmZmrdDBBx/Mhx9+mPMxNwsXLuSyyy4rUFS7psWLxHS1bzTwDUmvS1oF/Bz4LcnWcjlJITk1It6tN/054E1gNfALYEU+Y5U0S9J64CBJ6yVNz+f1zMzMzFqLgtwIEBFvA+c30DUl/ckeW0nysEtdgTm2kXN2yTqe3lBfRCwEFma1j9pJnFOBqU2NMTMzM2uP/N3NZmZmZpaj7T1SlAeSngc612u+JCIqChGPmZmZWaG5SAQiYmihYzAzMzNrTbzdbGZmZmY5XCSamZmZWQ4XiWZmZmaWw0WimZmZmeVwkWhmZmZmOVwkmpmZmVkOF4lmZmZmlsNFopmZmRXUxo0bOe+88zjhhBPo06cPv//97ykvL+erX/0qJSUlnH322Xz88ceFDrPDcZFoZmZmBTVx4kROP/10/vjHP1JeXk6fPn0YP348M2fOpKKigtGjRzN79uxCh9nhKCLyd3LpcGAucDKwEdgATIqItQ2MLQaWRMSJeQuo4RgPAu4FjgFqgd9FxDVp33CS+PsDF0TEv+/sfEf1Pjb2OX9eHiNuGyaX1DCnwl/o4zwknIeE8+Ac1OloeaiceVaD7ZlMhoEDB1JaWsobb7yBpO193bp1Y+PGjUhi3bp1fPOb32T16tUtFXKLymQylJWVFeTakpZHxOCG+vK2kqjkv/RiIBMRx0TEIODvgB75uuYeuCEiTgAGAqdIOiNt/zMwDvhtoQIzMzNrz958800OO+wwLr30UgYOHMj48ePZtGkT/fr144EHHgDg3nvvZd26dQWOtOPJ53bzCKA6Im6pa4iIcuBZSbMlvSKpQtKY+hMljZM0P+v1Ekll6XFVOn+VpMclDZGUkfSGpHOy5t8naamk1yTNaizIiPg0Ip5Mj7cCK4Aj09eVEfEysG1vJMTMzMx2VFNTw4oVK/jhD3/ISy+9xMEHH8zMmTO54447+OUvf8mgQYP45JNP2H///QsdaoeTz7XuE4HlDbSfC5QCA4Ai4AVJT+/CeQ8GlkXEFEmLgRnASKAvcCfwYDqulGRlcAuwRtJNEdHkP0MkHQKcDezSfrGkCcAEgKKiw5hWUrMr09ulHgcm2ykdnfOQcB4SzoNzUKej5SGTyTTYXlVVxV/+8heKiorYvHkzmUyGY445ht/+9rd8/etf59prrwVg3bp1dO/evdHztHVVVVWt8r0V4oaIYcCiiKgFNkh6iuSexZebOX8rsDQ9rgC2RES1pAqgOGvcExHxEYCk1cDRQKNFoqR9gUXALyLijV14P0TEAmABJPckdqT7TBrT0e63aYzzkHAeEs6Dc1Cno+WhcmxZg+119+LdeOON9OzZk6985StkMhlOPfVU+vbtS/fu3dm2bRvjxo1jypQpBbtvL98KeU9iU/L5/9BVwHm7ObeGHbfCD8g6ro7Pn7bZRrJSSERsSwu9OluyjmvZ+XtdALwWEXN3L+TEgft1Yk0jN+h2JJlMptFfCh2J85BwHhLOg3NQx3nY0U033cTYsWPZunUrvXv35l/+5V/49a9/zc033wzAueeey6WXXlrgKDuefBaJy4DrJU1IV9qQ1J/kKecxku4EDgWGA1PYsRCsBC6XtA/QCxiSxziRNAPoBozP53XMzMwsV2lpKS+++OIObRMnTmTixIkFisggjw+upKt9o4FvSHpd0irg5yRPCr8MlJMUklMj4t16058D3gRWA78geZgkLyQdCVxHck/jCkkrJY1P+06WtB74DnBr+h7MzMzM2r283hAREW8D5zfQNSX9yR5bSfKwS12BObaRc3bJOp7eUF9ELAQWZrWPaiLG9YAa6XuB9ElnMzMzs47E37hiZmZmZjk6zqNVgKTngc71mi+JiIpCxGNmZmbWWnWoIjEihhY6BjMzM7O2wNvNZmZmZpbDRaKZmZmZ5XCRaGZmZmY5XCSamZmZWQ4XiWZmZmaWw0WimZmZmeVwkWhmZmZmOTrU5ySamZm1JsXFxXTt2pVOnTqx77778uKLL1JeXs5ll11GVVUVxcXF/OY3v+ELX/hCoUO1DsgriWZmZgX05JNPsnLlSl588UUAxo8fz8yZM6moqGD06NHMnj27wBFaR5XXIlHS4ZLulvS6pOWSHpZ0fCNjiyW9ks94GrnuQZIekvRHSaskzczq+7Gk1ZJelvSEpKNbOj4zM+tY1q5dy/DhwwEYOXIk//Ef/1HgiKyjytt2syQBi4E7I+KCtG0A0ANYm6/r7qYbIuJJSfsDT0g6IyIeAV4CBkfEp5J+CMwCxjR1os3VtRRf81ALhNy6TS6pYZzz4DyknIeE89DxclA586wm+yVx2mmnIYkf/OAHTJgwgX79+vHAAw/wrW99i3vvvZd169a1ULRmO8rnSuIIoDoibqlriIhy4FlJsyW9IqlCUk7RJWmcpPlZr5dIKkuPq9L5qyQ9LmmIpIykNySdkzX/PklLJb0maVZjQUbEpxHxZHq8FVgBHJm+fjIiPk2H/qGu3czMbG949tlnWbFiBY888gg333wzTz/9NHfccQe//OUvGTRoEJ988gn7779/ocO0DiqfD66cCCxvoP1coBQYABQBL0h6ehfOezCwLCKmSFoMzABGAn2BO4EH03GlwEBgC7BG0k0R0eQ/xyQdApwNzGug+/vAI43MmwBMACgqOoxpJTW78Hbapx4HJisGHZ3zkHAeEs5Dx8tBJpNpsL2qqmp732uvvQbAwIEDWbRoEWPGjOHaa68FYN26dXTv3r3R87R12XnoyFprHgrxdPMwYFFE1AIbJD0FnAy83Mz5W4Gl6XEFsCUiqiVVAMVZ456IiI8AJK0GjgYaLRIl7QssAn4REW/U67sYGAx8raG5EbEAWABwVO9jY06FHxqfXFKD8+A81HEeEs5Dx8tB5diyBtszmQwnn3wy27Zto2vXrmzatIlrr72WadOm0bdvX7p37862bdsYN24cU6ZMoays4fO0dZlMpt2+t13RWvOQzz+pq4DzdnNuDTtuhR+QdVwdEZEebyNZKSQitqWFXp0tWce17Py9LgBei4i52Y2SvgFcB3wtIrY0ODPLgft1Ys1O7kHpCDKZTKO/HDsS5yHhPCScB+cg24YNGxg9ejQANTU1XHTRRZx++unMmzePm2++GYBzzz2XSy+9tJBhWgeWzyJxGXC9pAnpShuS+gMbgTGS7gQOBYYDU9ixEKwELpe0D9ALGJLHOJE0A+gGjK/XPhC4FTg9It7LZwxmZtax9O7dm/Ly8pz2iRMnMnHixAJEZLajvBWJERGSRgNzJV0NfEZS/E0CugDlQABTI+JdScVZ058D3gRWA6+SPEySF5KOJFkp/COwInkom/kRcTswO4313rT9zxFxTr5iMTMzM2st8npjSES8DZzfQNeU9Cd7bCXJwy6k28ljGzlnl6zj6Q31RcRCYGFW+6gmYlwPqJG+bzQ2z8zMzKw98zeumJmZmVmOjvOIGSDpeaBzveZLIqKiEPGYmZmZtVYdqkiMiKGFjsHMzMysLfB2s5mZmZnlcJFoZmZmZjlcJJqZmZlZDheJZmZmZpbDRaKZmZmZ5XCRaGZmZmY5XCSamZmZWQ4XiWZmZvUUFxdTUlJCaWkpgwcPBmD69On06tWL0tJSSktLefjhhwscpVl+dagP0zYzM2uuJ598kqKioh3arrzySq666qoCRWTWsvK6kijpcEl3S3pd0nJJD0s6vpGxxZJeyWc8jZH0T5LWSaqq1z5c0gpJNZLOK0RsZmZmZoWQt5VESQIWA3dGxAVp2wCgB7A2X9fdTb8D5gOv1Wv/MzAOaPY/GzdX11J8zUN7L7I2anJJDeOcB+ch5TwknIfWkYPKmWftdIwkTjvtNCTxgx/8gAkTJgAwf/58fv3rXzN48GDmzJnDF7/4xXyHa1Yw+VxJHAFUR8QtdQ0RUQ48K2m2pFckVUgaU3+ipHGS5me9XiKpLD2uSuevkvS4pCGSMpLekHRO1vz7JC2V9JqkWU0FGhF/iIh3GmivjIiXgW27mwQzM2t7nn32WVasWMEjjzzCzTffzNNPP80Pf/hDXn/9dVauXEnPnj2ZPHlyocM0y6t83pN4IrC8gfZzgVJgAFAEvCDp6V0478HAsoiYImkxMAMYCfQF7gQeTMeVAgOBLcAaSTdFxLrdeic7IWkCMAGgqOgwppXU5OMybUqPA5MVg47OeUg4DwnnoXXkIJPJNGvca68lm0sDBw5k0aJFjBnz+ZpGSUkJv/3tb5t9rvqqqqp2e2574jwkWmseCvHgyjBgUUTUAhskPQWcDLzczPlbgaXpcQWwJSKqJVUAxVnjnoiIjwAkrQaOBvJSJEbEAmABwFG9j405FX4eaHJJDc6D81DHeUg4D60jB5Vjy5rs37RpE9u2baNr165s2rSJa6+9lmnTpvGVr3yFnj17AnDjjTcydOhQysqaPldjMpnMbs9tT5yHRGvNQz7/pK4Cdvdhjxp23Ao/IOu4OiIiPd5GslJIRGyTlP1+tmQd19JCBfGB+3ViTTPud2nvMpnMTn8RdwTOQ8J5SDgPbSMHGzZsYPTo0QDU1NRw0UUXcfrpp3PJJZewcuVKJFFcXMytt95a4EjN8iufhdMy4HpJE9KVNiT1BzYCYyTdCRwKDAemsGMhWAlcLmkfoBcwJI9xmpmZbde7d2/Ky8tz2u+6664CRGNWOHl7cCVd7RsNfCP9CJxVwM+B35JsLZeTFJJTI+LdetOfA94EVgO/AFbkK04ASbMkrQcOkrRe0vS0/eS0/TvArel7MDMzM2v38roFGxFvA+c30DUl/ckeW0nysEtdgTm2kXN2yTqe3lBfRCwEFma1j9pJnFOBqQ20vwAc2dRcMzMzs/bIX8tnZmZmZjk61GN2kp4HOtdrviQiKgoRj5mZmVlr1aGKxIgYWugYzMzMzNoCbzebmZmZWQ4XiWZmZmaWw0WimZmZmeVwkWhmZmZmOVwkmpmZmVkOF4lmZmZmlsNFopmZmZnlcJFoZmYdUnFxMSUlJZSWljJ48OAd+ubMmYMkPvjggwJFZ1Z4HerDtM3MzLI9+eSTFBUV7dC2bt06Hn30UY466qgCRWXWOuR1JVHS4ZLulvS6pOWSHpZ0fCNjiyW9ks94GrnuQZIekvRHSaskzczqu1HSyvRnraSNLR2fmZm1rCuvvJJZs2YhqdChmBVU3lYSlfzpWgzcGREXpG0DgB7A2nxddzfdEBFPStofeELSGRHxSERcWTdA0v8LDNzZiTZX11J8zUP5jLVNmFxSwzjnwXlIOQ8J56Flc1A586wm+yVx2mmnIYkf/OAHTJgwgQceeIBevXoxYMCAFonRrDXL53bzCKA6Im6pa4iIciVmA2cAAcyIiHuyJ0oaBwyOiB+lr5eQFHIZSVXAr4AzgXeAa4FZwFHApIh4MJ1/DnAQcAywOCKmNhRkRHwKPJkeb5W0AjiygaEXAj/brUyYmVmr8+yzz9KrVy/ee+89Ro4cyQknnMD111/Po48+WujQzFoFRUR+TixdAXw5ezUubf82cBlwOlAEvAAMBToDSyLixJ0UiQGcGRGPSFoMHAycBfQlWbUsTedPI1n52wKsAYZFxLqdxHwIsAL4RkS8kdV+NPAH4MiIqG1g3gRgAkBR0WGDps29bRcy1T71OBA2bC50FIXnPCSch4Tz0LI5KOnVrdljFy5cyD777MPixYvp3LkzAO+//z5FRUX86le/4tBDD92rsVVVVdGlS5e9es62yHlIFDIPI0aMWB4RgxvqK8SDK8OARWmxtUHSU8DJwMvNnL8VWJoeVwBbIqJaUgVQnDXuiYj4CEDSauBooNEiUdK+wCLgF9kFYuoC4N8bKhABImIBsADgqN7HxpwKPw80uaQG58F5qOM8JJyHls1B5diyRvs2bdrEtm3b6Nq1K5s2beLaa69l2rRp3HHHHdvHFBcX8+KLL+Y82LI3ZDIZysoaj6+jcB4SrTUP+fyTugo4bzfn1rDjQzUHZB1Xx+fLn9tIVgqJiG1poVdnS9ZxLTt/rwuA1yJibgN9FwB/25zAD9yvE2t2ch9MR5DJZJr8Bd1ROA8J5yHhPLSeHGzYsIHRo0cDUFNTw0UXXcTpp59e4KjMWpd8FonLgOslTUhX2pDUH9gIjJF0J3AoMByYwo6FYCVwuaR9gF7AkDzGiaQZQDdgfAN9JwBfBH6fzxjMzKzl9O7dm/Ly8ibHVFZWtkwwZq1U3orEiAhJo4G5kq4GPiMp/iYBXYBykgdXpkbEu5KKs6Y/B7wJrAZeJblPMC8kHQlcB/wRWJF+5MH8iLg9HXIBcHfk6+ZNMzMzs1YorzeGRMTbwPkNdE1Jf7LHVgInpscBjG3knF2yjqc31BcRC4GFWe2jmohxPdDoh2HVv4aZmZlZR+Cv5TMzMzOzHB3qMTtJz5N81E62SyKiohDxmJmZmbVWHapIjIihhY7BzMzMrC3wdrOZmZmZ5XCRaGZmZmY5XCSamZmZWQ4XiWZmZmaWw0WimZmZmeVwkWhmZmZmOVwkmpmZmVmODvU5iWZmlh+1tbUMHjyYXr16sWTJEk499VQ++eQTAN577z2GDBnC/fffX+AozWxXuEg0M7M9Nm/ePPr06cPHH38MwDPPPLO979vf/jZ/8zd/U6jQzGw35XW7WdLhku6W9Lqk5ZIelnR8I2OLJb2Sz3gaue5Bkh6S9EdJqyTNzOo7StKTkl6S9LKkM1s6PjOz1m79+vU89NBDjB8/Pqfv448/ZtmyZXzrW98qQGRmtifytpIoScBi4M6IuCBtGwD0ANbm67q76YaIeFLS/sATks6IiEeAnwD/FhG/ktQXeBgobupEm6trKb7mofxH3MpNLqlhnPPgPKSch0RbzUPlzLOa7J80aRKzZs3avr2c7f777+frX/86X/jCF/IVnpnlST5XEkcA1RFxS11DRJQDz0qaLekVSRWSxtSfKGmcpPlZr5dIKkuPq9L5qyQ9LmmIpIykNySdkzX/PklLJb0maVZjQUbEpxHxZHq8FVgBHFnXDdT9ZusGvL0H+TAza3eWLFlC9+7dGTRoUIP9ixYt4sILL2zhqMxsb1BE5OfE0hXAlyPiynrt3wYuA04HioAXgKFAZ2BJRJwoaRwwOCJ+lM5ZQrLal5EUwJkR8YikxcDBwFlAX5JVy9J0/jRgILAFWAMMi4h1O4n5EJIi8RsR8YaknsCjwBfT63wjIpY3MG8CMAGgqOiwQdPm3raL2Wp/ehwIGzYXOorCcx4SzkOireahpFe3Rvtuu+02Hn30UTp16sTWrVv59NNPOfXUU7nuuuv46KOP+O53v8u9997L/vvvD0BVVRVdunRpqdBbLech4TwkCpmHESNGLI+IwQ31FeLBlWHAooioBTZIego4GXi5mfO3AkvT4wpgS0RUS6pgx63gJyLiIwBJq4GjgUaLREn7AouAX0TEG2nzhcDCiJgj6avAXZJOjIht2XMjYgGwAOCo3sfGnAo/DzS5pAbnwXmo4zwk2moeKseWNdpXVvZ5XyaT4YYbbmDJkiUA3HLLLXzrW9/itNNO22FM9pyOynlIOA+J1pqHfP62WgWct5tza9hxK/yArOPq+Hz5cxvJSiERsS0t9OpsyTquZefvdQHwWkTMzWr7PsmKJxHxe0kHkKx+vtfYSQ7crxNrdnL/TkeQyWSa/Iulo3AeEs5DoqPl4e677+aaa64pdBhmtpvyeU/iMqBzuhULgKT+wEZgjKROkg4DhgP/VW9uJVAqaR9JXwKG5DFOJM0guedwUr2uPwNfT8f0ISlW389nLJJWKXkAACAASURBVGZmbVVZWdn2VURIiuLTTz+9gBGZ2Z7I20piRISk0cBcSVcDn5EUf5OALkA5yYMhUyPiXUnFWdOfA94EVgOvktwnmBeSjgSuA/4IrEgeymZ+RNwOTAZuk3RlGuu4yNdNnGZmZmatSF5vjomIt4HzG+iakv5kj60ETkyPAxjbyDm7ZB1Pb6gvIhYCC7PaRzUR43pAjfStBk5pbK6ZmZlZe+XvbjYzMzOzHG3vMbs9IOl5ko/ayXZJRFQUIh4zMzOz1qpDFYkRMbTQMZiZmZm1Bd5uNjMzM7McLhLNzMzMLIeLRDMzMzPL4SLRzMzMzHK4SDQzMzOzHC4SzczMzCyHi0QzMzMzy+Ei0cysDaitrWXgwIGMGpV8y+j8+fM59thjkcQHH3xQ4OjMrD1ykWhm1gbMmzePPn36bH99yimn8Pjjj3P00UcXMCoza8/yWiRKOlzS3ZJel7Rc0sOSjm9kbLGkV/IZT2Mk/ZOkdZKq6rVfJqlC0kpJz0rqW4j4zKxjW79+PQ899BDjx4/f3jZw4ECKi4sLF5SZtXt5+1o+SQIWA3dGxAVp2wCgB7A2X9fdTb8D5gOv1Wv/bUTcAiDpHOCfgdObOtHm6lqKr3koL0G2JZNLahjnPDgPKech0VgeKmee1eS8SZMmMWvWLD755JN8hWZmliOfK4kjgOq6IgsgIsqBZyXNlvRKuko3pv5ESeMkzc96vURSWXpclc5fJelxSUMkZSS9kRZydfPvk7RU0muSZjUVaET8ISLeaaD946yXBwOxizkwM9sjS5YsoXv37gwaNKjQoZhZB5O3lUTgRGB5A+3nAqXAAKAIeEHS07tw3oOBZRExRdJiYAYwEugL3Ak8mI4rBQYCW4A1km6KiHW7+iYk/S3wY2B/4K8bGTMBmABQVHQY00pqdvUy7U6PA5NVk47OeUg4D4nG8pDJZBqds2jRIh599FHuu+8+tm7dyqeffsrIkSO57rrrAPjss8947rnn6NatW77C3quqqqqafL8dhfOQcB4SrTUP+SwSGzMMWBQRtcAGSU8BJwMvN3P+VmBpelwBbImIakkVQHHWuCci4iMASauBo4FdLhIj4mbgZkkXAT8BvtfAmAXAAoCjeh8bcyoKkdbWZXJJDc6D81DHeUg0lofKsWWNzikr+7wvk8lwww03sGTJku1tBxxwAKeccgpFRUV7M9S8yWQyO7ynjsp5SDgPidaah3z+1l4FnLebc2vYcSv8gKzj6oio2/bdRrJSSERsk5T9frZkHdey5+/1buBXOxt04H6dWLOT+4s6gkwm0+RffB2F85BwHhJ7Mw+/+MUvmDVrFu+++y79+/fnzDPP5Pbbb98r5zYzg/zek7gM6JxuxQIgqT+wERgjqZOkw4DhwH/Vm1sJlEraR9KXgCF5jLNRko7LenkWuQ+2mJm1mLKysu2riFdccQXr16+npqaGt99+2wWime11eVtJjIiQNBqYK+lq4DOS4m8S0AUoJ3kQZGpEvCupOGv6c8CbwGrgVWBFvuIESB9suQg4SNJ64PaImA78SNI3gGrgf2hgq9nMzMysPcrrTUIR8TZwfgNdU9Kf7LGVJA+7kG4nj23knF2yjqc31BcRC4GFWe2jdhLnVGBqA+0Tm5pnZmZm1l75G1fMzMzMLEeHetxQ0vNA53rNl0RERSHiMTMzM2utOlSRGBFDCx2DmZmZWVvg7WYzMzMzy+Ei0czMzMxyuEg0MzMzsxwuEs3MzMwsh4tEMzMzM8vhItHMzMzMcrhINDMzM7McLhLNzMzMLIeLRDOzveyzzz5jyJAhDBgwgH79+vGzn/0MgOXLl3PSSSdRWlrKsGHD+O///u8CR2pm1riCFImSDpd0t6TXJS2X9LCk4xsZWyzplQLE2FXSyqyfDyTNbek4zKzt6dy5M8uWLaO8vJyVK1eydOlS/vCHPzB37lx+85vfsHLlSi666CJmzJhR6FDNzBrV4l/LJ0nAYuDOiLggbRsA9ADWtnQ8jYmIT4DSuteSlgP37Wze5upaiq95KJ+htQmTS2oY5zw4D6n2lofKmWc12S+JLl26AFBdXU11dTWSkMTHH38MwEcffcQRRxyR91jNzHZXIVYSRwDVEXFLXUNElAPPSpot6RVJFZLG1J8oaZyk+Vmvl0gqS4+r0vmrJD0uaYikjKQ3JJ2TNf8+SUslvSZpVnMCTlc5uwPP7NE7N7MOo7a2ltLSUrp3787IkSMZOnQoV111FWeeeSZHHnkkd911F9dcc02hwzQza5QiomUvKF0BfDkirqzX/m3gMuB0oAh4ARgKdAaWRMSJksYBgyPiR+mcJcANEZGRFMCZEfGIpMXAwcBZQF+SVcvSdP40YCCwBVgDDIuIdTuJeRrwhYi4qpH+CcAEgKKiwwZNm3vbrqal3elxIGzYXOgoCs95SLS3PJT06tbssVVVVfz0pz/liiuu4LbbbuPiiy+mb9++3H333axbt44pU6bkMdLWp6qqavsqa0fmPCSch0Qh8zBixIjlETG4ob4W325uwjBgUUTUAhskPQWcDLzczPlbgaXpcQWwJSKqJVUAxVnjnoiIjwAkrQaOBposEoELgEsa64yIBcACgKN6HxtzKlpTWgtjckkNzoPzUKe95aFybNkujV+xYgUffPABf/rTn7j88ssB6N27N6effjplZbt2rrYuk8l0uPfcEOch4TwkWmseCvFbexVw3m7OrWHHLfIDso6r4/Nl0W0kK4VExDZJ2e9zS9ZxLTvJQXq/5L4Rsbw5AR64XyfW7OR+pY4gk8ns8l+k7ZHzkOhoeXj//ffZb7/9OOSQQ9i8eTOPPfYYV199NVVVVaxdu5bjjz+exx57jD59+hQ6VDOzRhWiSFwGXC9pQroCh6T+wEZgjKQ7gUOB4cAUdiwEK4HLJe0D9AKGtEC8FwKLWuA6ZtZOvPPOO3zve9+jtraWbdu2cf755zNq1Ciuuuoqvv3tb7PPPvvwxS9+kTvuuKPQoZqZNarFi8SICEmjgbmSrgY+Iyn+JgFdgHIggKkR8a6k4qzpzwFvAquBV4EVLRDy+cCZLXAdM2sn+vfvz0svvZTTfuqpp/LTn/60ABGZme26gtwkFBFvkxRf9U1Jf7LHVgInpscBjG3knF2yjqc31BcRC4GFWe2jmhFr752NMTMzM2tv/I0rZmZmZpaj/TxuuAckPU/yUTvZLomIikLEY2ZmZlZoLhKBiBha6BjMzMzMWhNvN5uZmZlZDheJZmZmZpbDRaKZmZmZ5XCRaGZmZmY5XCSamZmZWQ4XiWZmZmaWw0WimZmZmeVwkWhmZmZmOVwkmlmH99lnnzFkyBAGDBhAv379+NnPfgbAuHHj+PKXv0xpaSmlpaWsXLmywJGambWcgnzjiqTDgbnAycBGYAMwKSLWNjC2GFgSESe2ZIz1YngQ6F3IGMwsfzp37syyZcvo0qUL1dXVDBs2jDPOOAOA2bNnc9555xU4QjOzltfiRaIkAYuBOyPigrRtANADyCkSC03SuUBVc8dvrq6l+JqH8hhR2zC5pIZxzoPzkGoNeaiceVajfZLo0qULANXV1VRXV5P8qjIz67gKsd08AqiOiFvqGiKiHHhW0mxJr0iqkDSm/kRJ4yTNz3q9RFJZelyVzl8l6XFJQyRlJL0h6Zys+fdJWirpNUmzmgpUUhfgx8CMvfLOzazVqq2tpbS0lO7duzNy5EiGDk2+0v26666jf//+XHnllWzZsqXAUZqZtRxFRMteULoC+HJEXFmv/dvAZcDpQBHwAjAU6Ey63SxpHDA4In6UzlkC3BARGUkBnBkRj0haDBwMnAX0JVm1LE3nTwMGAluANcCwiFjXSKw3Ak8DL9HElrekCcAEgKKiwwZNm3vb7iWnHelxIGzYXOgoCs95SLSGPJT06tascVVVVfz0pz/liiuu4Atf+AKHHnoo1dXVzJkzhyOOOILvfe97ux1DVVXV9hXLjso5SDgPCechUcg8jBgxYnlEDG6oryD3JDZiGLAoImqBDZKeIrln8eVmzt8KLE2PK4AtEVEtqQIozhr3RER8BCBpNXA0kFMkSioFjomIK9P7IhsVEQuABQBH9T425lS0prQWxuSSGpwH56FOa8hD5diyZo9dsWIFH374IZdeeun2tv33358bbriBsrLmn6e+TCazR/PbA+cg4TwknIdEa81DIX5rrwJ29y7wGnbcIj8g67g6Pl8W3UayUkhEbJOU/T6z94tqaTwHXwUGS6pMx3SXlImIsqYCPHC/Tqxp4t6njiKTyezSX8rtlfOQaO15eP/999lvv/045JBD2Lx5M4899hhXX30177zzDj179iQiuP/++znxRD+7ZmYdRyHuSVwGdE63aAGQ1J/kKecxkjpJOgwYDvxXvbmVQKmkfSR9CRiSryAj4lcRcUREFJOscq7dWYFoZm3TO++8w4gRI+jfvz8nn3wyI0eOZNSoUYwdO5aSkhJKSkr44IMP+MlPflLoUM3MWkyLryRGREgaDcyVdDXwGUnxNwnoApQDAUyNiHfrbfU+B7wJrAZeBVa0XORm1l7179+fl156Kad92bJlBYjGzKx1KMhNQhHxNnB+A11T0p/ssZXAielxAGMbOWeXrOPpDfVFxEJgYVb7qGbGuz0GMzMzs47A37hiZmZmZjn82CUg6XmSj9rJdklEVBQiHjMzM7NCc5EIRMTQQsdgZmZm1pp4u9nMzMzMcrhINDMzM7McLhLNzMzMLIeLRDMzMzPL4SLRzMzMzHK4SDQzMzOzHC4SzczMzCyHi0QzMzMzy+Ei0czy7r333mPEiBH07duXfv36MW/ePADKy8v56le/SklJCWeffTYff/xxgSM1M7M6LhLNLO86derEnDlzWL16NX/4wx+4+eabWb16NePHj2fmzJlUVFQwevRoZs+eXehQzcwsldev5ZN0ODAXOBnYCGwAJkXE2gbGFgNLIuLEfMbUwHUPAu4FjgFqgd9FxDVZ/ecD04EAyiPioqbOt7m6luJrHspfwG3E5JIaxjkPHSoPlTPParTvr/7qrzjppJMA6Nq1K3369OGtt95i7dq1DB8+HICRI0fyzW9+k3/8x39skXjNzKxpeVtJlCRgMZCJiGMiYhDwd0CPfF1zD9wQEScAA4FTJJ0BIOk4kphPiYh+wKQCxmjWLlRWVvLSSy8xdOhQ+vXrxwMPPADAvffey7p16wocnZmZ1VFE5OfE0l8D0yNieL12AbOAM0hW52ZExD3ZK4mSxgGDI+JH6ZwlJIVcRlIV8CvgTOAd4Nr0fEeRrFI+mM4/BziIZIVwcURMbWbc84BXIuI2SbOAtRFx+07mTAAmABQVHTZo2tzbmnOpdq3HgbBhc6GjKLyOlIeSXt0a7auqqqJLly5s3ryZiRMncvHFFzN8+HD+/Oc/c9NNN/HRRx9xyimncN99920vGtujujx0ZM5BwnlIOA+JQuZhxIgRyyNicEN9+dxuPhFY3kD7uUApMAAoAl6Q9PQunPdgYFlETJG0GJgBjAT6AncCD6bjSklWBrcAayTdFBFNLlNIOgQ4G5iXNh2ftj8HdCIpepfWnxcRC4AFAEf1PjbmVOR1F79NmFxSg/PQsfJQObas0b5MJsMpp5zCqFGjuOyyy/jxj3+8ve+73/0uAGvXrmXVqlWUlTV+nrYuk8m06/fXHM5BwnlIOA+J1pqHQvztNQxYFBG1wAZJT5Hcs/hyM+dvBeoKtQpgS0RUS6oAirPGPRERHwFIWg0cDTRaJEraF1gE/CIi3kib9wWOA8qAI4GnJZVExMbGznPgfp1Y08S9WR1FJpNpsmjoKJyHRETw/e9/nz59+uxQIL733nt0796dbdu2MWPGDC677LICRmlmZtny+XTzKmDQbs6tYcfYDsg6ro7P98i3kawUEhHb2LHo3ZJ1XMvOC+IFwGsRMTerbT3wYERUR8SbwFqSotHMdsErr7zCXXfdxbJlyygtLaW0tJSHH36YRYsWcfzxx3PCCSdwxBFHcOmllxY6VDMzS+VzJXEZcL2kCel2LJL6kzzlPEbSncChwHBgCjsWgpXA5ZL2AXoBQ/IYJ5JmAN2A8fW67gcuBP5FUhHJ9vMbmNkuKSkpobH7nydOnNjC0ZiZWXPkrUiMiJA0Gpgr6WrgM5LibxLQBSgneXBlakS8mz64Uuc54E1gNfAqsCJfcUo6ErgO+COwInmuhvnpwyr/P3Baul1dC0yJiA/zFYuZmZlZa5HXexIj4m3g/Aa6pqQ/2WMrSR52Id1OHtvIObtkHU9vqC8iFgILs9pHNRHjekCN9AXw4/THzMzMrMPwN66YmZmZWY6O8dkcKUnPA53rNV8SERWFiMfMzMystepQRWJEDC10DGZmZmZtgbebzczMzCyHi0QzMzMzy+Ei0czMzMxyuEg0MzMzsxwuEs3MzMwsh4tEMzMzM8vhItHMzMzMcrhINLNdsm7dOkaMGEHfvn3p168f8+bNA2DMmDGUlpZSWlpKcXExpaWlBY7UzMz2RIf6MG0z23P77rsvc+bM4aSTTuKTTz5h0KBBjBw5knvuuWf7mMmTJ9OtW7cCRmlmZnsqr0WipMOBucDJwEZgAzApItY2MLYYWBIRJ+YzpoZI+ifgu8AXI6JLVvs4YDbwVto0PyJub+pcm6trKb7moXyF2mZMLqlhnPPQZvNQOfOsRvt69uxJz549AejatSt9+vThrbfeom/fvgBEBP/2b//GsmXLWiRWMzPLj7wViZIELAbujIgL0rYBQA8gp0gssN8B84HXGui7JyJ+1MLxmLUJlZWVvPTSSwwd+vk3Xj7zzDP06NGD4447roCRmZnZnlJE5OfE0l8D0yNieL12AbOAM4AAZkTEPdkriekK3uC64kzSEuCGiMhIqgJ+BZwJvANcm57vKJJVygfT+ecABwHHAIsjYmozYq5qYCVx8M6KREkTgAkARUWHDZo297adXard63EgbNhc6CgKr63moaTXzreKN2/ezMSJE7n44osZPvzzP+Y33ngjvXr14vzzz9/eVlVVRZcuXRo6TYfiPDgHdZyHhPOQKGQeRowYsTwiBjfUl8/t5hOB5Q20nwuUAgOAIuAFSU/vwnkPBpZFxBRJi4EZwEigL3An8GA6rhQYCGwB1ki6KSLW7cb7+Lak4SSrn1c2dI6IWAAsADiq97Exp8K3ek4uqcF5aLt5qBxb1mR/dXU1o0aN4rLLLuPHP/7x9vaamhrGjBnD8uXLOfLII7e3ZzIZysqaPmdH4Dw4B3Wch4TzkGiteSjE317DgEURUQtskPQUyT2LLzdz/lZgaXpcAWyJiGpJFUBx1rgnIuIjAEmrgaOBXS0Sf5fGukXSD0iK0L9uasKB+3ViTRP3c3UUmUxmp4VGR9Ae8xARfP/736dPnz47FIgAjz/+OCeccMIOBaKZmbVN+fwInFXAoN2cW8OOsR2QdVwdn++RbyNZKSQitrFj0bsl67iW3SiII+LDiKg7z+3s/vsxazeee+457rrrLpYtW7b9I28efvhhAO6++24uvPDCAkdoZmZ7Qz5XEpcB10uakG7HIqk/yVPOYyTdCRwKDAemsGMhWAlcLmkfoBcwJI9xNkpSz4h4J315DvBqIeIwa02GDRtGY/cyL1y4sGWDMTOzvMlbkRgRIWk0MFfS1cBnJMXfJKALUE7y4MrUiHg3fXClznPAm8BqksJsRb7iBJA0C7gIOEjSeuD2iJgOXCHpHJKVzb8A4/IZh5mZmVlrkdd7EiPibeD8BrqmpD/ZYytJHnYh3U4e28g5u2QdT2+oLyIWAguz2kftJM6pQM7TzxHxd8DfNTXXzMzMrD3y1/KZmZmZWY6299kce0DS80Dnes2XRERFIeIxMzMza606VJEYEUN3PsrMzMzMvN1sZmZmZjlcJJqZmZlZDheJZmZmZpbDRaKZmZmZ5XCRaGZmZmY5XCSamZmZWQ4XiWZmZmaWw0WimTVo3bp1jBgxgr59+9KvXz/mzZu3ve+mm27ihBNOoF+/fkydmvONlmZm1g50qA/TNrPm23fffZkzZw4nnXQSn3zyCYMGDWLkyJFs2LCBBx54gPLycjp37sx7771X6FDNzCwP8lokSjocmAucDGwENgCTImJtA2OLgSURcWI+Y2rgugcB9wLHALXA7yLimrTvMuBv0/YqYEJErG7qfJuraym+5qH8Bt0GTC6pYZzz0KrzUDnzrCb7e/bsSc+ePQHo2rUrffr04a233uK2227jmmuuoXPn5Bsuu3fvnvdYzcys5eVtu1mSgMVAJiKOiYhBwN8BPfJ1zT1wQ0ScAAwETpF0Rtr+24goiYhSYBbwzwWL0KyAKisreemllxg6dChr167lmWeeYejQoXzta1/jhRdeKHR4ZmaWB/lcSRwBVEfELXUNEVGuxGzgDCCAGRFxT/ZE/R/27j46yvrc9//7QnxA4qZ1B5CHDSlK5SGBKCjHX1kxact2ixGr7RE5+BCDh8Nx40ahQdRTf2gppWhOQfRoYy2ktKbWnnJAdHPaGkb7o/WhIJhNKrirs4sIWLTShlJIwvX7475DJ5lJMkEmM0k+r7Wy1p3vwz3XXEtWLr/f+ztjVgJMdPe54e8bCQq5iJnVAY8DU4F9wL0EBdwwglXKDeH8acDZBCuE69w94YNT7v4XYHN4fczMtgFDw9//FDO0bxhvHDObDcwGyM7uz/15DcllqBsb2CdYRevpMjkPkUgkqXFHjhxh3rx53HbbbWzbto1Dhw5RU1PDsmXLeOutt5g2bRpPP/00wf8XJlZXV5f063VnyoNy0ER5CCgPgUzNQyqLxFxga4L264B8YDyQDbxuZi934L59gWp3LzOzdcASYAowBqgENoTj8glWBo8Cu8xslbvvaevGZvYp4GpgZUzbPwPzgTOAzyea5+4VQAXAsBEXeHmNHvVckNeA8pDZeYjOLGx3TH19PcXFxcyZM4f58+cDcOGFF3LHHXdQVFREUVERDz/8MLm5ufTv37/V+0QiEQoL23+97k55UA6aKA8B5SGQqXlIx1+vyUCVuzcCB8zsJYJnFt9Mcv4xYFN4XQMcdfd6M6sBcmLGvejuhwDMrBYYDrRaJJpZb6AKeMTd32lqd/fHgMfM7L8A/wO4pa3g+px+GrvaedarJ4hEIkkVId1dV86DuzNr1ixGjx59okAE+NKXvsTmzZspKipi9+7dHDt2jOzs7DRGKiIiqZDKInEn8JWTnNtA8+clz4q5rnf3pm3f4wQrhbj78bDQa3I05rqR9t9rBfC2u69opf9HBNvcIj3Cli1bWLt2LXl5eeTn5wOwdOlSSktLKS0tJTc3lzPOOIPKyso2t5pFRKRrSmWRWA0sNbPZ4XYsZjaO4JTzdDOrBM4FCoAymheCUeB2M+sFDAEuTWGcmNkSoB9wW4v2ke7+dvjrVcDbLeeKdFeTJ0/mb/8/1twPfvCDTo5GREQ6W8qKRHd3M7sWWGFmdwN/JSj+7gSygB0EB0EWuvv+8CNwmmwB3gVqgd8C21IVp5kNBe4D3gK2hSsij7r7d4G5ZvZFoB74I+1sNYuIiIh0Fyl9JtHd3weuT9BVFv7Ejo0SHHYh3E6e2co9s2KuFyfqc/c1wJqY9uI2YnwPSLhX5u7zWpsnIiIi0p3pa/lEREREJE5mfjZHipjZq8CZLZpvcveadMQjIiIikql6VJHo7pPSHYOIiIhIV6DtZhERERGJoyJRREREROKoSBQRERGROCoSRURERCSOikQRERERiaMiUURERETiqEgUERERkTgqEkV6sD179lBUVMSYMWMYO3YsK1euBGDx4sUMGTKE/Px88vPzeeGFF9IcqYiIdLYe9WHaItJc7969KS8v5+KLL+bPf/4zEyZMYMqUKQDcddddfPWrX01zhCIiki4pLRLN7DxgBXAJ8DFwALjT3XcnGJsDbHT33FTGlOB1zwaeBc4HGoHn3H1RizFfBn4CXOLuv2nrfkfqG8lZ9Hyqwu0yFuQ1UKI8ZEQeosuuarVv0KBBDBo0CIBzzjmH0aNHs3fv3s4KTUREMljKtpvNzIB1QMTdz3f3CcA9wMBUveYn8LC7jwIuAj5nZlc2dZjZOcA84NV0BSfSGaLRKG+88QaTJgXfXvnoo48ybtw4SktL+eMf/5jm6EREpLOZu6fmxmafBxa7e0GLdgOWA1cCDixx92diVxLNrASY6O5zwzkbCQq5iJnVAY8DU4F9wL3h/YYRrFJuCOdPA84mWCFc5+4Lk4x7JfBv7v5k+PsK4OdAGfDVRCuJZjYbmA2Qnd1/wv0rnkwuSd3YwD5w4Ei6o0i/TMhD3pB+7Y45cuQI8+bN48Ybb6SgoICPPvqIfv36YWZ873vf48MPP+Tuu+8+6Rjq6urIyso66fndhfKgHDRRHgLKQyCdeSgqKtrq7hMT9aVyuzkX2Jqg/TogHxgPZAOvm9nLHbhvX6Da3cvMbB2wBJgCjAEqgQ3huHyClcGjwC4zW+Xue9q6sZl9CrgaWBn+fjHwD+7+vJmVtTbP3SuACoBhIy7w8ho96rkgrwHlITPyEJ1Z2GZ/fX09xcXFzJkzh/nz58f1jxgxguLiYgoL275PWyKRyCea310oD8pBE+UhoDwEMjUP6fjrNRmocvdG4ICZvUTwzOKbSc4/BmwKr2uAo+5eb2Y1QE7MuBfd/RCAmdUCw4FWi0Qz6w1UAY+4+ztm1gv4n0BJsm8MoM/pp7GrjWfAeopIJNJucdITZHoe3J1Zs2YxevToZgXivn37TjyruG7dOnJzO/VRYRERyQCpLBJ3Al85ybkNNH9e8qyY63r/2x75cYKVQtz9eFjoNTkac91I+++1Anjb3VeEv59DsBoaCXbIOQ/YYGbT2ju8ItJVbNmyhbVr15KXl0d+fj4AS5cupaqqiu3bt2Nm5OTk8J3vfCfNkYqISGdLZZFYDSw1s9nhdixmNo7glPN0M6sEZXjS8AAAIABJREFUzgUKCJ73iy0Eo8Dt4WreEODSFMaJmS0B+gG3NbWFq5DZMWMitPJMokhXNXnyZBI9lzx16tQ0RCMiIpkkZUWiu7uZXQusMLO7gb8SFH93AlnADoKDKwvdfX94cKXJFuBdoBb4LbAtVXGa2VDgPuAtYFu4aviou383Va8pIiIikulS+kyiu78PXJ+gqyz8iR0bJdjeJdxOntnKPbNirhcn6nP3NcCamPbiNmJ8D7A23kbTuML2xoiIiIh0F/paPhERERGJ06M+o8TMXgXObNF8k7vXpCMeERERkUzVo4pEd5+U7hhEREREugJtN4uIiIhIHBWJIiIiIhJHRaKIiIiIxFGRKCIiIiJxVCSKiIiISBwViSIiIiISR0WiiIiIiMRRkSjSA+zZs4eioiLGjBnD2LFjWblyZbP+8vJyzIyDBw+mKUIREck0PerDtEV6qt69e1NeXs7FF1/Mn//8ZyZMmMCUKVMYM2YMe/bs4Wc/+xnDhg1Ld5giIpJBUlokmtl5wArgEuBj4ABwp7vvTjA2B9jo7rmpjCkRM/sGcDPwaXfPimn/NlAU/no2MMDdP9XWvY7UN5Kz6PmUxdpVLMhroER56LQ8RJdd1Wb/oEGDGDRoEADnnHMOo0ePZu/evYwZM4a77rqL5cuXc80116Q8ThER6TpSViSamQHrgEp3vyFsGw8MBOKKxDR7DngUeDu20d3varo2szuAizo5LpFTLhqN8sYbbzBp0iTWr1/PkCFDGD9+fLrDEhGRDGPunpobm30eWOzuBS3aDVgOXAk4sMTdn4ldSTSzEmCiu88N52wEHnb3iJnVAY8DU4F9wL3h/YYRrFJuCOdPI1j9Ox9Y5+4Lk4i5LnYlsUXfr4D/191/nqBvNjAbIDu7/4T7VzzZ3kt1ewP7wIEj6Y4i/TorD3lD+iU17siRI8ybN48bb7yRSy+9lLvuuouHHnqIrKwsbrjhBr7zne/Qr19y9+qIuro6srIS/tPqUZQH5aCJ8hBQHgLpzENRUdFWd5+YqC+V2825wNYE7dcB+cB4IBt43cxe7sB9+wLV7l5mZuuAJcAUYAxQCWwIx+UTrPwdBXaZ2Sp333Myb8TMhgOfAaoT9bt7BVABMGzEBV5eo0c9F+Q1oDx0Xh6iMwvbHVNfX09xcTFz5sxh/vz51NTU8OGHHzJ37lwADh48yB133MFrr73Geeedd0rji0QiFBa2H2N3pzwoB02Uh4DyEMjUPKTjr/hkoMrdG4EDZvYSwTOLbyY5/xiwKbyuAY66e72Z1QA5MeNedPdDAGZWCwwHTqpIBG4AfhLG3KY+p5/GrnaeD+sJIpFIUoVLd5cpeXB3Zs2axejRo5k/fz4AeXl5fPDBByfG5OTk8Jvf/Ibs7Ox0hSkiIhkklR+BsxOYcJJzG2ge21kx1/X+tz3y4wQrhbj7cZoXvUdjrhv5ZAXxDUDVJ5gvklZbtmxh7dq1VFdXk5+fT35+Pi+88EK6wxIRkQyWypXEamCpmc0Ot2Mxs3EEp5ynm1klcC5QAJTRvBCMArebWS9gCHBpCuNsk5mNAj4N/DpdMYh8UpMnT6a954+j0WjnBCMiIl1CylYSw9W+a4EvmtnvzGwn8E3gaYKt5R0EheRCd9/fYvoW4F2gFngE2JaqOAHMbLmZvQecbWbvmdnimO4bgB95qk74iIiIiGSglD6T6O7vA9cn6CoLf2LHRgkOuzQVmDNbuWdWzPXiRH3uvgZYE9Ne3E6cC4GEp59bvoaIiIhIT6Cv5RMRERGROD3qM0rM7FXgzBbNN7l7TTriEREREclUPapIdPdJ6Y5BREREpCvQdrOIiIiIxFGRKCIiIiJxVCSKiIiISJykikQzO9/MzgyvC83sX8zsU6kNTURERETSJdmVxP8NNJrZBUAF8A8EH4otIiIiIt1QskXicXdvIPgGlVXuXgYMSl1YIiIiIpJOyRaJ9WY2A7gF2Bi2nZ6akEREREQk3ZItEm8FLgO+4e7vmtlngLWpC0tEOmrPnj0UFRUxZswYxo4dy8qVKwH42te+xrhx48jPz+cf//Efef/999McqYiIdAVJFYnuXgvcDWwLf3/X3b+VysBEpGN69+5NeXk5tbW1vPLKKzz22GPU1tZSVlbGm2++yfbt2ykuLubBBx9Md6giItIFJPWNK2Z2NfAwcAbwGTPLBx5092ntzDsPWAFcAnwMHADudPfdCcbmABvdPbcjb+CTMrOzgWeB84FG4Dl3XxT2nQl8H5gAfAhMd/doW/c7Ut9IzqLnUxpzV7Agr4ES5eGU5yG67KpW+wYNGsSgQcGjwueccw6jR49m7969jBkz5sSYw4cPY2anLB4REem+kv1avsXApUAEwN23m9mItiZY8JdoHVDp7jeEbeOBgUBckZhmD7v7ZjM7A3jRzK50938FZgF/dPcLzOwG4FvA9LRGKpKEaDTKG2+8waRJwTdR3nfffXz/+9+nX79+bN68Oc3RiYhIV2Du3v4gs1fc/T+Z2RvuflHY9qa7j2tjzueBxe5e0KLdgOXAlYADS9z9mdiVRDMrASa6+9xwzkaCQi5iZnXA48BUYB9wb3i/YQSrlBvC+dOAswlWCNe5+8KkEmK2Evg3d3/SzP5v+B5+bWa9gf1Af2+RNDObDcwGyM7uP+H+FU8m81Ld2sA+cOBIuqNIv1Odh7wh/dodc+TIEebNm8eNN95IQUGzf3788Ic/5NixY9x6662nLqgk1NXVkZWV1amvmYmUB+WgifIQUB4C6cxDUVHRVnefmKgv2ZXEnWb2X4DTzGwk8C/Ar9qZkwtsTdB+HZAPjAeygdfN7OUk4wDoC1S7e5mZrQOWAFOAMUAlsCEclw9cBBwFdpnZKnff09aNww8IvxpYGTYNAfYAuHuDmR0C/h44GDvP3SsIPj+SYSMu8PKaZNPafS3Ia0B5OPV5iM4sbLO/vr6e4uJi5syZw/z58+P6R4wYwdSpU6msrDxlMSUjEolQWFjYqa+ZiZQH5aCJ8hBQHgKZmodk/3rdAdxHUHA9DfxfguLsZEwGqty9EThgZi8RPLP4ZpLzjwGbwusa4Ki715tZDZATM+5Fdz8EYGa1wHDCgi+RcKWwCnjE3d/pwPtpps/pp7GrjefGeopIJNJuQdMTdGYe3J1Zs2YxevToZgXi22+/zciRIwFYv349o0aN6pR4RESka2u3SDSz04Dn3b2IoFBM1k7gKycZVwPNT16fFXNdH7Pde5ygcMXdj4eFXpOjMdeNtP9eK4C33X1FTNtegm+XeS+8dz+CAywiGWfLli2sXbuWvLw88vPzAVi6dClPPfUUu3btolevXgwfPpwnnngizZGKiEhX0G6R6O6NZnbczPo1rcwlqRpYamazw+1YzGwcwSnn6WZWCZwLFABlNC8Eo8DtZtaLYMv30g68boeZ2RKCAvC2Fl0bCD5A/NcEBW91y+cRRTLF5MmTSfSf59SpU9MQjYiIdHXJbjfXATVm9nPgcFOju/9LaxPc3c3sWmCFmd0N/JWg+LsTyAJ2EBxcWeju+8ODK022AO8CtcBvCT+fMRXMbCjBCulbwLbw40EedffvAk8Ba83s34GPgBtSFYeIiIhIJkm2SPxp+NMh7v4+cH2CrrLwJ3ZslOCwC+Fq3cxW7pkVc704UZ+7rwHWxLQXtxHje0DCD45z978C/7m1uSIiIiLdVVJFort37lFIEREREUmrZL9x5V2CreFm3L3ND9TONGb2KnBmi+ab3L0mHfGIiIiIZKpkt5tjP2TxLIIt2HNPfTip5e6T0h2DiIiISFfQq/0h4O4fxvzsDT8mRh8GKCIiItJNJbvdfHHMr70IVhb1dRoiIiIi3VSyhV55zHUDwcfTJDq1LCIiIiLdQLJF4qyWX1VnZp9JQTwiIiIikgGSeiYR+EmSbSIiIiLSDbS5kmhmo4CxQD8zuy6m6+9o/jV6IiIiItKNtLfdfCFQDHwKuDqm/c/Af01VUCIiIiKSXm0Wie6+HlhvZpe5+687KSYROQl79uzh5ptv5sCBA5gZs2fPZt68eXzta19j/fr19OrViwEDBrBmzRoGDx6c7nBFRCTDJftM4htm9s9m9r/M7HtNPymNTEQ6pHfv3pSXl1NbW8srr7zCY489Rm1tLWVlZbz55pts376d4uJiHnzwwXSHKiIiXUCyp5vXAm8BVwAPAjOB357si5rZecAK4BLgY+AAcKe7704wNgfY6O65J/t6J8vMJgBrgD7AC8A8d4/7esJYR+obyVn0fCdEl9kW5DVQojyc8jxEl7X+GfaDBg1i0KBBAJxzzjmMHj2avXv3MmbMmBNjDh8+jJmdsnhERKT7SrZIvMDd/7OZXePulWb2NPDLk3lBC/5CrQMq3f2GsG08MBCIKxLT7HGCZy9fJSgS/wn417RGJJKEaDTKG2+8waRJwTdR3nfffXz/+9+nX79+bN68Oc3RiYhIV2DtLIwFg8xec/dLzexl4HZgP/Cau4/o8AuafR5Y7O4FLdoNWA5cCTiwxN2fiV1JNLMSYKK7zw3nbAQedveImdURFHVTgX3AveH9hhGsUm4I508DzgbOB9a5+8JW4hwEbHb3UeHvM4BCd/9vCcbOBmYDZGf3n3D/iic7mpZuZ2AfOHAk3VGk36nOQ96Qfu2OOXLkCPPmzePGG2+koKDZPzN++MMfcuzYMW699dZTF1QS6urqyMrK6tTXzETKg3LQRHkIKA+BdOahqKhoq7tPTNSX7EpihZl9GvgasAHIAu4/yXhyga0J2q8D8oHxQDbweliUJqsvUO3uZWa2DlgCTAHGAJVh3ISvcRFwFNhlZqvcfU+C+w0B3ov5/b2wLY67VwAVAMNGXODlNfrGwgV5DSgPpz4P0ZmFbfbX19dTXFzMnDlzmD9/flz/iBEjmDp1KpWVlacspmREIhEKCws79TUzkfKgHDRRHgLKQyBT85DUXy93/254+RLQ4dXDJE0Gqty9EThgZi8RPLP4ZpLzjwGbwusa4Ki715tZDZATM+5Fdz8EYGa1wHAgUZF4Uvqcfhq72nhurKeIRCLtFjQ9QWfmwd2ZNWsWo0ePblYgvv3224wcORKA9evXM2rUqE6JR0REurakikQzGwgsBQa7+5VmNga4zN2fOonX3Al85STmQfC90bEnsmM/0Ls+5lDJcYKVQtz9uJnFvs+jMdeNtJ6DvcDQmN+Hhm0iGWnLli2sXbuWvLw88vPzAVi6dClPPfUUu3btolevXgwfPpwnnngizZGKiEhXkOw+2BpgNXBf+Ptu4BngZIrEamCpmc0Ot2kxs3EEp5ynm1klcC5QAJTRvBCMArebWS+Crd9LT+L1k+Lu+8zsT2b2nwgOrtwMrErV64l8UpMnTybRM8ZTp05NQzQiItLVJfs5idnu/mOCFTrcvYFgFa7DwtW+a4EvmtnvzGwn8E3gaYKt5R0EheRCd9/fYvoW4F2gFngE2HYyMXTA7cB3gX8HfodONouIiEgPkexK4mEz+3uCU8eEq2uHTvZF3f194PoEXWXhT+zYKMFhl6YCc2Yr98yKuV6cqM/d1xCsija1F7cT52+aXltERESkJ0m2SJxPcDr4fDPbAvTn5J8rFBEREZEM12aRaGbD3P337r7NzC4HLgQM2OXu9Z0SYScws1eBM1s03+TuNemIR0RERCTd2ltJ/D/AxeH1M+7+5RTHkxbuPindMYiIiIhkkvYOrsR+yWuqPh9RRERERDJMe0Wit3ItIiIiIt1Ye9vN483sTwQrin3Ca8Lf3d3/LqXRiYiIiEhatFkkuvtpnRWIiIiIiGSOZD9MW0RERER6EBWJIiIiIhJHRaKIiIiIxFGRKJIme/bsoaioiDFjxjB27FhWrlwJwEcffcSUKVMYOXIkU6ZM4Y9//GOaIxURkZ5IRaJImvTu3Zvy8nJqa2t55ZVXeOyxx6itrWXZsmV84Qtf4O233+YLX/gCy5YtS3eoIiLSAyX73c2nlJmdB6wALgE+Bg4Ad7r77gRjc4CN7p7bmTGGr30G8ChQCBwH7nP3/93WnCP1jeQser4TostsC/IaKFEeWPNPfVvtGzRoEIMGDQLgnHPOYfTo0ezdu5f169cTiUQAuOWWWygsLORb3/pWZ4QrIiJyQqcXiWZmwDqg0t1vCNvGAwOBuCIxze4DPnD3z5pZL+DcdAck3VM0GuWNN95g0qRJHDhw4ETxeN5553HgwIE0RyciIj2RuXfuF6mY2eeBxe5e0KLdgOXAlQTf7rLE3Z+JXUk0sxJgorvPDedsBB5294iZ1QGPA1OBfcC94f2GEaxSbgjnTwPOBs4H1rn7wjZi3QOMcvfD7byn2cBsgOzs/hPuX/FkBzLSPQ3sAweOpDuK9PtMv9PIyspqc8yRI0eYN28eN954IwUFBRQXF7Nx48YT/VdffTXPPfdcqkNNqbq6unbz0BMoD8pBE+UhoDwE0pmHoqKire4+MVFfOrabc4GtCdqvA/KB8UA28LqZvdyB+/YFqt29zMzWAUuAKcAYoBLYEI7LBy4CjgK7zGyVu+9peTMz+1R4+XUzKwR+B8x197hlHXevACoAho24wMtr0rKLn1EW5DWgPATbzYWFha3219fXU1xczJw5c5g/fz4AQ4YM4cILL2TQoEHs27ePwYMHt3mPriASiXT593AqKA/KQRPlIaA8BDI1D5n0V3wyUOXujcABM3uJ4JnFN5OcfwzYFF7XAEfdvd7MaoCcmHEvuvshADOrBYYDcUUiQW6GAr9y9/lmNh94GLiprSD6nH4au5ZdlWTI3VckEiE6szDdYaRd07OFibg7s2bNYvTo0ScKRIBp06ZRWVnJokWLqKys5JprrumESEVERJpLx+nmncCEk5zbQPOYz4q5rve/7Z0fJ1gpxN2P07wYPhpz3UjrhfKHwF+An4a/PwtcfHJhi8TbsmULa9eupbq6mvz8fPLz83nhhRdYtGgRP//5zxk5ciS/+MUvWLRoUbpDFRGRHigdK4nVwFIzmx1u02Jm4whOOU83s0qCAyIFQBnNC8EocHt4iGQIcGmqgnR3N7PnCE42VwNfAGpT9XrS80yePJnWngl+8cUXOzkaERGR5jq9SAyLr2uBFWZ2N/BXguLvTiAL2EFwcGWhu+8PD6402QK8S1Cs/RbYluJw7wbWmtkK4A/ArSl+PREREZGMkJZnEt39feD6BF1l4U/s2CjBYRfC7eSZrdwzK+Z6caI+d18DrIlpL24nzv8gWNEUERER6VH0jSsiIiIiEieTTjenjZm9CpzZovkmd69JRzwiIiIi6aYiEXD3SemOQURERCSTaLtZREREROKoSBQRERGROCoSRURERCSOikQRERERiaMiUURERETiqEgUERERkTgqEkVEREQkjopEkRQrLS1lwIAB5ObmnmjbsWMHl112GXl5eVx99dX86U9/SmOEIiIi8VQkiqRYSUkJmzZtatZ22223sWzZMmpqarj22mt56KGH0hSdiIhIYin9xhUzOw9YAVwCfAwcAO50990JxuYAG909t2VfqpnZN4CbgU+7e1ZM+zCgEvgUcBqwyN1faOteR+obyVn0fCrD7RIW5DVQ0kPyEF12VZv9BQUFRKPRZm27d++moKAAgClTpnDFFVfw9a9/PVUhioiIdFjKVhLNzIB1QMTdz3f3CcA9wMBUveYn8BxwaYL2/wH82N0vAm4A/lenRiXd1tixY1m/fj0Azz77LHv27ElzRCIiIs2Zu6fmxmafBxa7e0GLdgOWA1cCDixx92diVxLNrASY6O5zwzkbgYfdPWJmdcDjwFRgH3BveL9hBKuUG8L504CzgfOBde6+MImY61qsJH4HeMfdv2VmlwHl7v7/JJg3G5gNkJ3df8L9K55MNk3d1sA+cOBIuqPoHHlD+rXaV1dXR1ZWFvv37+eee+5h9erVAPz+979n1apVHDp0iM997nP89Kc/PVE0dkdNeejplAfloInyEFAeAunMQ1FR0VZ3n5ioL5XbzbnA1gTt1wH5wHggG3jdzF7uwH37AtXuXmZm64AlwBRgDMHW8IZwXD5wEXAU2GVmq9y9o8s1i4Gfmdkd4et+MdEgd68AKgCGjbjAy2tSuovfJSzIa6Cn5CE6s7DVvkgkQmFhIdFolL59+1JY+LexN998MxBsPe/cubNZX3fTlIeeTnlQDpooDwHlIZCpeUjHX/HJQJW7NwIHzOwlgmcW30xy/jGg6RRADXDU3evNrAbIiRn3orsfAjCzWmA40NEicQawxt3Lw5XEtWaW6+7HW5vQ5/TT2NXOM2o9QSQSabN46uk++OADBgwYwPHjx1myZAlz5sxJd0giIiLNpPJ0805gwknObaB5bGfFXNf73/bIjxOsFBIWbrFF79GY60ZOriCeBfw4vP+vwziyT+I+0oPNmDGDyy67jF27djF06FCeeuopqqqq+OxnP8uoUaMYPHgwt956a7rDFBERaSaVK4nVwFIzmx1ux2Jm4whOOU83s0rgXKAAKKN5IRgFbjezXsAQEh8q6Qy/B74ArDGz0QQx/iFNsUgXVVVVlbB93rx5nRyJiIhI8lK2khiu9l0LfNHMfmdmO4FvAk8TbC3vICgkF7r7/hbTtwDvArXAI8C2VMUJYGbLzew94Gwze8/MFoddC4D/amY7gCqgxFN10kdEREQkg6T0mUR3fx+4PkFXWfgTOzZKcNilqcCc2co9s2KuFyfqc/c1wJqY9uJ24lwIxJ1+dvda4HNtzRURERHpjvSNKyIiIiISp2d8RknIzF4FzmzRfJO716QjHhEREZFM1aOKRHeflO4YRERERLoCbTeLiIiISBwViSIiIiISR0WiiIiIiMRRkSgiIiIicVQkioiIiEgcFYkiIiIiEkdFooiIiIjEUZEokmKlpaUMGDCA3NzcE207duzgsssuIy8vj6uvvpo//elPaYxQREQknopEkRQrKSlh06ZNzdpuu+02li1bRk1NDddeey0PPfRQmqITERFJLKXfuGJm5wErgEuAj4EDwJ3uvjvB2Bxgo7vntuxLNTP7BnAz8Gl3z4ppnw/cBjQAfwBK3f0/2rrXkfpGchY9n8pwu4QFeQ2U9JA8RJdd1WZ/QUEB0Wi0Wdvu3bspKCgAYMqUKVxxxRV8/etfT1WIIiIiHZaylUQzM2AdEHH38919AnAPMDBVr/kJPAdcmqD9DWCiu48DfgIs79SopNsaO3Ys69evB+DZZ59lz549aY5IRESkOXP31NzY7PPAYncvaNFuBMXWlYADS9z9mdiVRDMrISjO5oZzNgIPu3vEzOqAx4GpwD7g3vB+wwhWKTeE86cBZwPnA+vcfWESMdfFriS26LsIeNTdP5egbzYwGyA7u/+E+1c82d5LdXsD+8CBI+mOonPkDenXal9dXR1ZWVns37+fe+65h9WrVwPw+9//nlWrVnHo0CE+97nP8dOf/vRE0dgdNeWhp1MelIMmykNAeQikMw9FRUVb3X1ior5UbjfnAlsTtF8H5APjgWzgdTN7uQP37QtUu3uZma0DlgBTgDFAJbAhHJcPXAQcBXaZ2Sp3/yTLNbOAf03U4e4VQAXAsBEXeHlNSnfxu4QFeQ30lDxEZxa22heJRCgsLCQajdK3b18KC/829uabbwaCreedO3c26+tumvLQ0ykPykET5SGgPAQyNQ/p+Cs+Gahy90bggJm9RPDM4ptJzj8GNJ0CqAGOunu9mdUAOTHjXnT3QwBmVgsMB06qSDSzG4GJwOXtje1z+mnsaucZtZ4gEom0WTz1dB988AEDBgzg+PHjLFmyhDlz5qQ7JBERkWZSebp5JzDhJOc20Dy2s2Ku6/1ve+THCVYKcffjNC96j8ZcN3KSBbGZfRG4D5jm7kfbGy/S0owZM7jsssvYtWsXQ4cO5amnnqKqqorPfvazjBo1isGDB3PrrbemO0wREZFmUrmSWA0sNbPZ4XYsZjaO4JTzdDOrBM4FCoAymheCUeB2M+sFDCHxoZKUC59D/A7wT+7+QTpikK6vqqoqYfu8efM6ORIREZHkpWwlMVztuxb4opn9zsx2At8EnibYWt5BUEgudPf9LaZvAd4FaoFHgG2pihPAzJab2XvA2Wb2npktDrseArKAZ81su5ltaPUmIiIiIt1ISp9JdPf3gesTdJWFP7FjowSHXZoKzJmt3DMr5npxoj53XwOsiWkvbifOhUDc6Wd3/2Jb80RERES6K33jioiIiIjE6RmfURIys1eBM1s03+TuNemIR0RERCRT9agi0d0npTsGERERka5A280iIiIiEkdFooiIiIjEUZEoIiIiInFUJIqIiIhIHBWJIiIiIhJHRaKIiIiIxFGRKCIiIiJxVCSKnAKlpaUMGDCA3NzcE23bt2/n9ttvJz8/n4kTJ/Laa6+lMUIREZGOUZEocgqUlJSwadOmZm0LFy7klltuYfv27Tz44IMsXBj39eAiIiIZK6XfuGJm5wErgEuAj4EDwJ3uvjvB2Bxgo7vntuxLcYxnA88C5wONwHPuvijsKwEeAvaGwx919++2db8j9Y3kLHo+dQF3EQvyGijpZnmILruq1b6CggKi0WizNjPj8OHDABw6dIjBgwenMjwREZFTKmVFopkZsA6odPcbwrbxwEAgrkhMs4fdfbOZnQG8aGZXuvu/hn3PuPvcdAYnXdOKFSsoLCxk9erVHD9+nF/96lfpDklERCRpqVxJLALq3f2JpgZ332GBh4ArAQeWuPszsRPDFbyJTcWZmW0kKOQiZlYHPA5MBfYB9wLLgWEEq5QbwvnTgLMJVgjXuXvCvT53/wuwObw+ZmbbgKEdeaNmNhuYDZCd3Z/78xo6Mr1bGtgnWE3sTiKRSJv9+/fv5/DhwyfGPfLII5SWlnLFFVewefNmrrvuOsrLy1MfaAaqq6trN389gfKgHDRRHgLKQyBT85DKIjEX2Jqg/TogHxgPZAOvm9nLHbhvX6CHJbwyAAAgAElEQVTa3cvMbB2wBJgCjAEqgQ3huHzgIuAosMvMVrn7nrZubGafAq4GVsY0f9nMCghWP+9KdA93rwAqAIaNuMDLa1K6i98lLMhroLvlITqzsO3+aJS+fftSWBiMu+aaa7jjjjsoLCzk8ssv59vf/vaJvp4mEon02PceS3lQDpooDwHlIZCpeUjHX/HJQJW7NwIHzOwlgmcW30xy/jGg6YRADXDU3evNrAbIiRn3orsfAjCzWmA40GqRaGa9gSrgEXd/J2x+Loz1qJn9N4Ii9PNtBdfn9NPY1cazaz1FJBJpt6jq7gYPHsyOHTsoKiqiurqakSNHpjskERGRpKWySNwJfOUk5zbQ/OT1WTHX9e7u4fVxgpVC3P14WOg1ORpz3Uj777UCeNvdVzQ1uPuHMf3fJdjWFokzY8YMIpEIBw8eZOjQoTzwwAM8+eSTlJaWsnr1as466ywqKirSHaaIiEjSUlkkVgNLzWx2uB2LmY0jOOU83cwqgXOBAqCM5oVgFLjdzHoBQ4BLUxgnZrYE6Afc1qJ9kLvvC3+dBvw2lXFI11VVVZWwvaKiIiO3EERERNqTsiLR3d3MrgVWmNndwF8Jir87gSxgB8HBlYXuvj/8CJwmW4B3gVqCwmxbquI0s6HAfcBbwLbgUPaJj7r5FzObRrCy+RFQkqo4RERERDJJSp9JdPf3gesTdJWFP7FjowSHXQi3k2e2cs+smOvFifrcfQ2wJqa9uI0Y3wOslb57gHtamysiIiLSXekbV0REREQkTvf6jJJ2mNmrwJktmm9y95p0xCMiIiKSqXpUkejuk9Idg4iIiEhXoO1mEREREYmjIlFERERE4qhIFBEREZE4KhJFREREJI6KRBERERGJoyJRREREROKoSBQRERGROCoSRU6B0tJSBgwYQG5u7om27du3c/vtt5Ofn8/EiRN57bXX0hihiIhIx6hIFDkFSkpK2LRpU7O2hQsXcsstt7B9+3YefPBBFi5cmKboREREOi4t37hiZucBK4BLgI+BA8Cd7r47wdgcYKO757bsSzUzmwHcCzjwPnCjux9sa86R+kZyFj3fGeFltAV5DZR0szxEl13Val9BQQHRaLRZm5lx+PBhAA4dOsTgwYNTGZ6IiMgp1elFopkZsA6odPcbwrbxwEAgrkhMFzPrDawExrj7QTNbDswFFqc1MOkyVqxYQWFhIatXr+b48eP86le/SndIIiIiSUvHSmIRUO/uTzQ1uPsOCzwEXEmwcrfE3Z+JnWhmJcBEd58b/r4ReNjdI2ZWBzwOTAX2EawALgeGEaxSbgjnTwPOBs4H1rl7a3uAFv70NbMPgb8D/j3hQLPZwGyA7Oz+3J/X0MGUdD8D+wSrid1JJBJps3///v0cPnz4xLhHHnmE0tJSrrjiCjZv3sx1111HeXl56gPNQHV1de3mrydQHpSDJspDQHkIZGoe0lEk5gJbE7RfB+QD44Fs4HUze7kD9+0LVLt7mZmtA5YAU4AxQCWwIRyXD1wEHAV2mdkqd9/T8mbuXm9m/x2oAQ4DbwP/nOiF3b0CqAAYNuICL69Jyy5+RlmQ10B3y0N0ZmHb/dEoffv2pbAwGHfNNddwxx13UFhYyOWXX863v/3tE309TSQS6bHvPZbyoBw0UR4CykMgU/OQSX/FJwNV7t4IHDCzlwieWXwzyfnHgKaTAzXA0bDQqwFyYsa96O6HAMysFhgOxBWJZnY68N8JCsp3gFXAPQTFZ6v6nH4au9p4dq2niEQi7RZV3d3gwYPZsWMHRUVFVFdXM3LkyHSHJCIikrR0FIk7ga+c5NwGmp/IPivmut7dPbw+TrBSiLsfD58vbHI05rqR1nOQH87/HYCZ/RhYdJJxSzc3Y8YMIpEIBw8eZOjQoTzwwAM8+eSTlJaWsnr1as466ywqKirSHaaIiEjS0lEkVgNLzWx2uE2LmY0jOOU83cwqgXOBAqCM5oVgFLjdzHoBQ4BLUxjnXmCMmfV39z8QbF3/NoWvJ11YVVVVwvaKioqM3EIQERFpT6cXie7uZnYtsMLM7gb+SlD83QlkATsIDq4sdPf94UfgNNkCvAvUEhRs21IY5/tm9gDwspnVA/8BlKTq9UREREQySVqeSXT394HrE3SVhT+xY6MEh10It5NntnLPrJjrxYn63H0NsCamvbidOJ8AnmhrjIiIiEh3pG9cEREREZE4mXS6OW3M7FXgzBbNN7l7TTriEREREUk3FYmAu09KdwwiIiIimUTbzSIiIiISR0WiiIiIiMRRkSgiIiIicVQkioiIiEgcFYkiIiIiEkdFooiIiIjEUZEoIiIiInFUJIp0QGlpKQMGDCA3N/dE2/Tp08nPzyc/P5+cnBzy8/PTGKGIiMipoQ/TFumAkpIS5s6dy80333yi7ZlnnjlxvWDBAvr165eO0ERERE6plBaJZnYesAK4BPgYOADc6e67E4zNATa6e27LvhTHeDbwLHA+0Ag85+6Lwr7hwPeA/sBHwI3u/l5b9ztS30jOoudTG3QXsCCvgZIumIfosqva7C8oKCAajSbsc3d+/OMfU11dnYLIREREOlfKtpvNzIB1QMTdz3f3CcA9wMBUveYn8LC7jwIuAj5nZlc2tQPfd/dxwIPAN9MVoGS+X/7ylwwcOJCRI0emOxQREZFPzNw9NTc2+zyw2N0LWrQbsBy4EnBgibs/E7uSaGYlwER3nxvO2UhQyEXMrA54HJgK7APuDe83jGCVckM4fxpwNsEK4Tp3X5hk3CuBf3P3J81sJ/BP7r4njPuQu/9dgjmzgdkA2dn9J9y/4smk89RdDewDB46kO4qOyxvS/lbx/v37ueeee1i9enWz9m9/+9sMGTKE66+//kRbXV0dWVlZpzzOrkZ5CCgPykET5SGgPATSmYeioqKt7j4xUV8qt5tzga0J2q8D8oHxQDbwupm93IH79gWq3b3MzNYBS4ApwBigEtgQjssnWBk8Cuwys1XuvqetG5vZp4CrgZVh044w3pXAtcA5Zvb37v5h7Dx3rwAqAIaNuMDLa/So54K8BrpiHqIzC9sfE43St29fCgv/NrahoYHp06ezdetWhg4deqI9Eok0G9dTKQ8B5UE5aKI8BJSHQKbmIR2nmycDVe7e6O4HgJcInllM1jFgU3hdA7zk7vXhdU7MuBfd/ZC7/xWoBYa3dVMz6w1UAY+4+zth81eBy83sDeByYC/Bc4sizfziF79g1KhRzQpEERGRriyVSz07ga+c5NwGmhewZ8Vc1/vf9siPE6wU4u7Hw0KvydGY60baf68VwNvuvqKpwd3fJ1hJxMyygC+7+8dt3aTP6aexq53DDz1BJBJJalWuq5kxYwaRSISDBw8ydOhQHnjgAWbNmsWPfvQjZsyYke7wRERETplUFonVwFIzmx1ux2Jm4whOOU83s0rgXKAAKKN5IRgFbjezXsAQ4NIUxomZLQH6Abe1aM8GPnL34wSHbr6Xyjgk81VVVSVsX7NmTecGIiIikmIp224OV/uuBb5oZr8LD4F8E3gaeJPgeb9qYKG7728xfQvwLsE28SPAtlTFaWZDgfsInmncZmbbzaypWCwkeJ5xN8Gp7G+kKg4RERGRTJLSkwXhdu31CbrKwp/YsVGCwy5NBebMVu6ZFXO9OFGfu68B1sS0F7cR43uAtdL3E+Anrc0VERER6a70tXwiIiIiEqfrfUbJJ2BmrwJntmi+yd1r0hGPiIiISKbqUUWiu09KdwwiIiIiXYG2m0VEREQkjopEEREREYmjIlFERERE4qhIFBEREZE4KhJFREREJI6KRBERERGJoyJRREREROKoSBRpobS0lAEDBpCbm9usfdWqVYwaNYqxY8eycOHCNEUnIiLSOVQkirRQUlLCpk2bmrVt3ryZ9evXs2PHDnbu3MlXv/rVNEUnIiLSOVL6jStmdh6wArgE+Bg4ANzp7rsTjM0BNrp7bsu+VDOzbwA3A59296yY9uHA94D+wEfAje7+Xlv3OlLfSM6i51MZbpewIK+BkgzNQ3TZVW32FxQUEI1Gm7U9/vjjLFq0iDPPDL7VccCAAakKT0REJCOkbCXRzAxYB0Tc/Xx3nwDcAwxM1Wt+As8BlyZofxj4vruPAx4EvtmpUUnG2L17N7/85S+ZNGkSl19+Oa+//nq6QxIREUmpVK4kFgH17v5EU4O777DAQ8CVgANL3P2Z2IlmVgJMdPe54e8bgYfdPWJmdcDjwFRgH3AvsBwYRrBKuSGcPw04GzgfWOfurT5E5u6vhK/TsmsMMD+83gz8n0TzzWw2MBsgO7s/9+c1tJGWnmFgn2A1MRNFIpF2x+zfv5/Dhw+fGHvo0CFqampYtmwZb731FtOmTePpp59O9N9MM3V1dUm9XnenPASUB+WgifIQUB4CmZqHVBaJucDWBO3XAfnAeCAbeN3MXu7AffsC1e5eZmbrgCXAFIKCrhLYEI7LBy4CjgK7zGyVu+/p4HvYEca7ErgWOMfM/t7dP4wd5O4VQAXAsBEXeHlNSnfxu4QFeQ1kah6iMwvbHxON0rdvXwoLg7EXXnghd9xxB0VFRRQVFfHwww+Tm5tL//7927xPJBI5cY+eTHkIKA/KQRPlIaA8BDI1D+k4uDIZqHL3Rnc/ALxE8Mxiso4BTacKaoCX3L0+vM6JGfeiux9y978CtcDwk4j1q8DlZvYGcDmwF2g8iftIF/elL32JzZs3A8HW87Fjx8jOzk5zVCIiIqmTyqWencBXTnJuA80L2LNiruvd3cPr4wQrhbj7cTOLfT9HY64bOYn36u7vE6wkYmZZwJfd/eO25vQ5/TR2tXMwoieIRCJJrdhlohkzZhCJRDh48CBDhw7lgQceoLS0lNLSUnJzcznjjDOorKxsd6tZRESkK0tlkVgNLDWz2eF2LGY2juCU83QzqwTOBQqAMpoXglHgdjPrBQwh8aGSlDOzbOAjdz9OcOjme+mIQzpXVVVVwvYf/OAHnRyJiIhI+qRsuzlc7bsW+KKZ/c7MdhKcDn4aeJPgeb9qYKG7728xfQvwLsE28SPAtlTFCWBmy83sPeBsM3vPzBaHXYUEzzPuJjiV/Y1UxiEiIiKSKVJ6siDcrr0+QVdZ+BM7Nkpw2KWpwJzZyj2zYq4XJ+pz9zXAmpj24nbiXAjEnX52958AP2lrroiIiEh3pG9cEREREZE4mfkZJSliZq8CZ7Zovsnda9IRj4iIiEim6lFFortPSncMIiIiIl2BtptFREREJI6KRBERERGJoyJRREREROKoSBQRERGROCoSRURERCSOikQRERERiaMiUURERETiqEiUbq20tJQBAwaQm5sb11deXo6ZcfDgwTREJiIiktlUJEq3VlJSwqZNm+La9+zZw89+9jOGDRuWhqhEREQyX0q/ccXMzgNWAJcAHwMHgDvdfXeCsTnARnePX/JJMTP7BnAz8Gl3z0rQ/2XgJ8Al7v6btu51pL6RnEXPpybQLmRBXgMlnZCH6LKr2uwvKCggGo3Gtd91110sX76ca665JkWRiYiIdG0pW0k0MwPWARF3P9/dJwD3AANT9ZqfwHPApYk6zOwcYB7waqdGJCmzfv16hgwZwvjx49MdioiISMZK5XZzEVDv7k80Nbj7DuD/M7OHzOzfzKzGzKa3nGhmJWb2aMzvG82sMLyuC+fvNLNfmNmlZhYxs3fMbFrM/J+a2SYze9vMlrcVqLu/4u77Wun+OvAt4K8dTYBknr/85S8sXbqUBx98MN2hiIiIZLRUbjfnAlsTtF8H5APjgWzgdTN7uQP37QtUu3uZma0DlgBTgDFAJbAhHJcPXAQcBXaZ2Sp339ORN2BmFwP/4O7Pm1lZG+NmA7MBsrP7c39eQ0deplsa2CfYck61SCTS7pj9+/dz+PBhIpEI77zzDrt37+bCCy8E4A9/+ANjx47l8ccf59xzzz3l8dXV1SUVY3enPASUB+WgifIQUB4CmZqHlD6T2IrJQJW7NwIHzOwlgmcW30xy/jGg6SRCDXDU3evNrAbIiRn3orsfAjCzWmA4kHSRaGa9gP8JlLQ31t0rgAqAYSMu8P+/vXuPsqo88zz+/QnES3FJtFDTECnxluUgFG2CSbfDKmNwzMhoSBQkjC0D3WS13TPtdFKEJCsOpL0QjdOkNWsIndiQXp2iM0YSvKQyChxs7I5RCFBcUhK1ujUKCUZNqNaiqnjmj72LHOrUFevUPtT5fdY6y33e/b57P/uRc3h4995n39uQRVpLy2cuaWMw8tA0r6b3Pk1NVFRUUFNTQ01NDQsWLDi6rqqqimeffZbKysqixJfL5aip6T3Goc55SDgPzkEH5yHhPCRKNQ/F/Ft8N3D9cY5t49hT4afkLbdGRKTLR0hmComII5Lyj6clb7md/h/rKJLZ0FxyeSVnA+slXdvTzSunjhhGYy83U5SDXC7XpwKu2ObOnUsul+PgwYOMHz+eZcuWsXDhwqzDMjMzK3nFLBI3AndKWpTOtCFpMsldznMkrQFOB6YDtRxbCDYBt6SzeePo5qaSYkpnIY9OL0nKAZ/t7e5mKy11dXU9ru/qzmczMzMr4o0r6WzfLOCjkp6XtBu4C/gOyanlHSSF5OKI2N9p+FPAi8Ae4G+AbcWKE0DS3ZJeBk6T9LKkpcXcn5mZmVmpK+pFYxHxCjC7i1W16Su/bxPJ6d2OAnNeN9scmbe8tKt1EbEaWJ3XPrOXOBcDi3vpU9PTejMzM7OhxE9cMTMzM7MCZXUbrqSngZM7Nd8UEQ1ZxGNmZmZWqsqqSIyIy7KOwczMzOxE4NPNZmZmZlbARaKZmZmZFXCRaGZmZmYFXCSamZmZWQEXiWZmZmZWwEWimZmZmRVwkWhmZmZmBVwk2glvwYIFnHnmmUyaNOlo25e+9CUmT55MdXU1V111Fa+88kqGEZqZmZ14XCTaCW/+/PnU19cf01ZbW8vOnTvZvn07M2fO5Mtf/nJG0ZmZmZ2YivrEFUlnAyuADwJvAAeAWyPiuS76VgGPRMSkzuuKTdIdwB8B74mIkZ3WzQaWAgHsiIhP9bStt1rbqVryaLFCPWF85pI25g9gHpqWX9PtuunTp9PU1HRM2+jRo48uNzc3I2nAYjEzMysHRSsSlfytvA5YExE3pm1TgLOAgiIxYw8D9wP78hslXQB8HvjDiHhd0plZBGfH54tf/CLf/va3GTNmDJs2bco6HDMzsxNKMU83XwG0RsTKjoaI2AFskXSPpF2SGiTN6TxQ0nxJ9+e9f0RSTbp8KB2/W9ITkqZJykl6QdK1eeMfklQvaZ+ku3sKNCJ+HBGvdrHqT4CvR8Trab9fHkceLCN33HEHL730EvPmzeP+++/vfYCZmZkdVczTzZOArV20fwKoBqYAlcAzkp7sx3YrgI0RUStpHXA7MAO4GFgDrE/7VQNTgRagUdJ9EfFSP4/hQgBJTwHDgKURUd+5k6RFwCKAysqx3HZJWz93M/ScdWpyynmg5HK5Htfv37+f5ubmLvtNnDiRJUuWcMUVVwxYPH116NChXmMvB85DwnlwDjo4DwnnIVGqeSjqNYnduByoi4h24ICkzSTXLO7s4/jDQEeh1gC0RESrpAagKq/fhoh4E0DSHmAC0N8icThwAVADjAeelHRJRLyR3ykiVgGrAM6ZeH7c25BFWkvLZy5pYyDz0DSvpuf1TU1UVFRQU5P027dvHxdccAEA9913H5deeunRdYMpl8tlst9S4zwknAfnoIPzkHAeEqWah2JWM7uB649zbBvHngo/JW+5NSIiXT5CMlNIRByRlH88LXnL7Rzfsb4MPB0RrcCLkp4jKRqf6W7AqSOG0djDTRblIpfL9VrYDZS5c+eSy+U4ePAg48ePZ9myZTz22GM0NjZy0kknMWHCBFauXNn7hszMzOyoYhaJG4E7JS1KZ9qQNJnkLuc5ktYApwPTgVqOLQSbgFsknQSMA6YVMc6efB+YC/ydpEqS088vZBSLdaOurq6gbeHChRlEYmZmNnQU7caVdLZvFvBRSc9L2g3cBXyH5NTyDpJCcnFE7O80/CngRWAP8DfAtmLFCSDpbkkvA6dJelnS0nTVj4DX0tPVm4DaiHitmLGYmZmZlYKiXjwXEa8As7tYVZu+8vs2kdzs0lFgzutmmyPzlpd2tS4iVgOr89pn9hLnYmBxF+0B/GX6MjMzMysbfuKKmZmZmRUoq9twJT0NnNyp+aaIaMgiHjMzM7NSVVZFYkRclnUMZmZmZicCn242MzMzswIuEs3MzMysgItEMzMzMyvgItHMzMzMCrhINDMzM7MCLhLNzMzMrICLRDMzMzMr4CLRThgLFizgzDPPZNKkSUfbamtref/738/kyZOZNWsWb7zxRoYRmpmZDR0uEu2EMX/+fOrr649pmzFjBrt27WLnzp1ceOGF3HXXXRlFZ2ZmNrQU9Ykrks4GVgAfBN4ADgC3RsRzXfStAh6JiEmd1xWbpDuAPwLeExEj89pPBr4NXAq8BsyJiKaetvVWaztVSx4tYrQnhs9c0sb8fuahafk1Pa6fPn06TU1Nx7RdddVVR5c/9KEP8eCDD/Zrn2ZmZta1os0kShKwDshFxHkRcSnweeCsYu3zHXgYmNZF+0Lg9Yg4H/hr4CuDGpX1ywMPPMDHPvaxrMMwMzMbEop5uvkKoDUiVnY0RMQOYIukeyTtktQgaU7ngZLmS7o/7/0jkmrS5UPp+N2SnpA0TVJO0guSrs0b/5Ckekn7JN3dU6AR8eOIeLWLVdcBa9LlB4Er0+LXSswdd9zB8OHDmTdvXtahmJmZDQnFPN08CdjaRfsngGpgClAJPCPpyX5stwLYGBG1ktYBtwMzgItJCrr1ab9qYCrQAjRKui8iXurnMYwDXgKIiDZJbwJnAAfzO0laBCwCqKwcy22XtPVzN0PPWacmp5z7I5fL9dpn//79NDc3H9O3vr6ehx9+mHvvvZfNmzf3M9LiOnToUJ+Oa6hzHhLOg3PQwXlIOA+JUs1DUa9J7MblQF1EtAMHJG0muWZxZx/HHwY67l5oAFoiolVSA1CV129DRLwJIGkPMIG04BtoEbEKWAVwzsTz496GLNJaWj5zSRv9zUPTvJre+zQ1UVFRQU1N0re+vp7169ezefNmxo4dexyRFlculzsaazlzHhLOg3PQwXlIOA+JUs1DMauZ3cD1xzm2jWNPhZ+St9waEZEuHyGZKSQijkjKP56WvOV2ju9YfwG8D3g53fYYkhtYunXqiGE09nIDRjnI5XJ9Kvr6Y+7cueRyOQ4ePMj48eNZtmwZd911Fy0tLcyYMQNIbl5ZuXJlL1syMzOz3hSzSNwI3ClpUTrThqTJJHc5z5G0BjgdmA7Ucmwh2ATcIukkklO+Xd1UMhjWAzcD/0JS8G7MK1BtkNXV1RW0LVy4MINIzMzMhr6iFYkREZJmASskfQ54m6T4uxUYCewAAlgcEfvTn8Dp8BTwIrAH2AtsK1acAOmNLZ8CTpP0MvDNiFgKfAv4e0k/B34N3FjMOMzMzMxKRVEvnouIV4DZXayqTV/5fZtIbnYhna3r8jbV/N8xTAu5gnURsRpYndc+s5c4FwOLu2h/G7ihp7FmZmZmQ5GfuGJmZmZmBcrqNlxJTwMnd2q+KSIasojHzMzMrFSVVZEYEZdlHYOZmZnZicCnm83MzMysgItEMzMzMyvgItHMzMzMCrhINDMzM7MCLhLNzMzMrICLRDMzMzMr4CLRzMzMzAq4SLRMLFiwgDPPPJNJkyYdbfv1r3/NjBkzuOCCC5gxYwavv/56hhGamZmVNxeJlon58+dTX19/TNvy5cu58sor2bdvH1deeSXLly/PKDozMzPLpEiUdLaktZKel7RV0mOSLuymb5WkXYMdY7rvnKRGSdvT15lZxDEUTZ8+ndNPP/2Yth/84AfcfPPNANx88818//vfzyI0MzMzI4PH8kkSsA5YExE3pm1TgLOA5wY7nj6YFxHP9rXzW63tVC15tJjxnBBWX13R7zEHDhzgve99LwBnn302Bw4cGOiwzMzMrI+ymEm8AmiNiJUdDRGxA9gi6R5JuyQ1SJrTeaCk+ZLuz3v/iKSadPlQOn63pCckTUtnAl+QdG3e+Ick1UvaJ+nuoh+tHRdJJP+eMDMzsywM+kwiMAnY2kX7J4BqYApQCTwj6cl+bLcC2BgRtZLWAbcDM4CLgTXA+rRfNTAVaAEaJd0XES/1sN2/k9QOfA+4PSKicwdJi4BFAJWVY7ntkrZ+hD00HTp0iFwu12Of/fv309zcfLTf6NGj+d73vscZZ5zBa6+9xqhRo3rdRqnrSx7KgfOQcB6cgw7OQ8J5SJRqHrIoErtzOVAXEe3AAUmbgQ8CO/s4/jDQcSdEA9ASEa2SGoCqvH4bIuJNAEl7gAlAd0XivIj4haRRJEXiTcC3O3eKiFXAKoBzJp4f9zaUUlqzsfrqCmpqanrs09TUREXF7/rNmTOHffv28clPfpLly5dz44039rqNUpfL5U74YxgIzkPCeXAOOjgPCechUap5yKKa2Q1cf5xj2zj2FPkpecutebN8R0hmComII5Lyj7Mlb7mdHnIQEb9I//tbSd8BptFFkZjv1BHDaFx+TW/HMeT19i+iuXPnksvlOHjwIOPHj2fZsmUsWbKE2bNn861vfYsJEybw3e9+d3CCNTMzswJZFIkbgTslLUpn4JA0GXgDmCNpDXA6MB2o5dhCsAm4RdJJwDiSoq0o0sLy3RFxUNIIYCbwRLH2V27q6uq6bN+wYcMgR2JmZmZdGfQiMSJC0ixghaTPAW+TFH+3AiOBHUAAiyNiv6SqvOFPAS8Ce4C9wLYihnoy8KO0QBxGUiD+bRH3Z2ZmZlYyMrl4LiJeAWZ3sao2feX3bSK52YX0dPK8brY5Mm95aVfrImI1sDqvfWYPMTYDl/ZwGGZmZmZDlp+4YmZmZmYFfBsuIOlpktPL+dfNTQIAABP+SURBVG6KiIYs4jEzMzPLmotEICIuyzoGMzMzs1Li081mZmZmVsBFopmZmZkVcJFoZmZmZgVcJJqZmZlZAReJZmZmZlbARaKZmZmZFXCRaGZmZmYFXCRanzQ2NlJdXX30NXr0aFasWJF1WGZmZlYk/jFt65OLLrqI7du3A9De3s64ceOYNWtWxlGZmZlZsWQykyjpbElrJT0vaaukxyRd2E3fKkm7BjvGdN9zJO2UtFvSV7KIoRRt2LCB8847jwkTJmQdipmZmRXJoM8kShKwDlgTETembVOAs4DnBjue7kg6A7gHuDQifiVpjaQrI2JDT+Peam2nasmjgxPkAGpafk2f+65du5a5c+cWMRozMzPLWhYziVcArRGxsqMhInYAWyTdI2mXpAZJczoPlDRf0v157x+RVJMuH0rH75b0hKRpknKSXpB0bd74hyTVS9on6e4e4pwI7IuIX6XvnwA++Y6P/gR3+PBh1q9fzw033JB1KGZmZlZEWVyTOAnY2kX7J4BqYApQCTwj6cl+bLcC2BgRtZLWAbcDM4CLgTXA+rRfNTAVaAEaJd0XES91sb2fAxdJqgJeBj4OvKurHUtaBCwCqKwcy22XtPUj7NKQy+X61G/Lli2ce+657N27l71793bb79ChQ33e5lDmPCSch4Tz4Bx0cB4SzkOiVPNQSjeuXA7URUQ7cEDSZuCDwM4+jj8M1KfLDUBLRLRKagCq8vptiIg3ASTtASYABUViRLwu6U+BfwSOAP8MnNfVjiNiFbAK4JyJ58e9DaWU1r5pmlfTp34rV67klltuoaam5/65XK7XPuXAeUg4DwnnwTno4DwknIdEqeYhi2pmN3D9cY5t49hT5KfkLbdGRKTLR0hmComII5Lyj7Mlb7mdHnIQEQ8DD8PR2cL23gI8dcQwGvtxfd+JpLm5mccff5xvfOMbWYdiZmZmRZbFNYkbgZPTogsASZOBN4A5koZJGgtMB37SaWwTUC3pJEnvA6YVM1BJZ6b/fQ9wC/DNYu6v1FVUVPDaa68xZsyYrEMxMzOzIhv0mcSICEmzgBWSPge8TVL83QqMBHYAASyOiP3pNYEdngJeBPYAe4FtRQ73a+md1wBfjoiSufvazMzMrJgyuXguIl4BZnexqjZ95fdtIrnZhfR08rxutjkyb3lpV+siYjWwOq99Zi9x+ndezMzMrCz5sXxmZmZmVuDEuw23CCQ9DZzcqfmmiGjIIh4zMzOzrLlIBCLisqxjMDMzMyslPt1sZmZmZgVcJJqZmZlZAReJZmZmZlbARaKZmZmZFXCRaGZmZmYFXCSamZmZWQEXiWZmZmZWwEWi9UljYyPV1dVHX6NHj2bFihVZh2VmZmZF4h/Ttj656KKL2L59OwDt7e2MGzeOWbNmZRyVmZmZFUsmM4mSzpa0VtLzkrZKekzShd30rZK0a7BjTPddL2mHpN2SVkoalkUcpWbDhg2cd955TJgwIetQzMzMrEgGfSZRkoB1wJqIuDFtmwKcBTw32PH0YnZE/CaN+UHgBmBtTwPeam2nasmjgxLcQGpafk2f+65du5a5c+cWMRozMzPLWhYziVcArRGxsqMhInYAWyTdI2mXpAZJczoPlDRf0v157x+RVJMuH0rH75b0hKRpknKSXpB0bd74h9IZwn2S7u4p0Ij4Tbo4HHgXEO/04E90hw8fZv369dxwww1Zh2JmZmZFlMU1iZOArV20fwKoBqYAlcAzkp7sx3YrgI0RUStpHXA7MAO4GFgDrE/7VQNTgRagUdJ9EfFSdxuV9CNgGvBDktnErvosAhYBVFaO5bZL2voRdmnI5XJ96rdlyxbOPfdc9u7dy969e7vtd+jQoT5vcyhzHhLOQ8J5cA46OA8J5yFRqnkopRtXLgfqIqIdOCBpM/BBYGcfxx8G6tPlBqAlIlolNQBVef02RMSbAJL2ABOAbovEiPhPkk4B/gH4CPB4F31WAasAzpl4ftzbUEpp7ZumeTV96rdy5UpuueUWamp67p/L5XrtUw6ch4TzkHAenIMOzkPCeUiUah6yqGZ2A9cf59g2jj1FfkrecmtEdJwOPkIyU0hEHJGUf5wtecvt9CEHEfG2pB8A19FFkZjv1BHDaOzH9X0nkubmZh5//HG+8Y1vZB2KmZmZFVkW1yRuBE5OT9ECIGky8AYwR9IwSWOB6cBPOo1tAqolnSTpfSSngYtC0khJ702XhwPXAD8r1v5OBBUVFbz22muMGTMm61DMzMysyAZ9JjEiQtIsYIWkzwFvkxR/twIjgR0kN4gsjoj9kqryhj8FvAjsAfYC24oYagWwXtLJJMX0JmBlz0PMzMzMhoZMLp6LiFeA2V2sqk1f+X2bSG52IT2dPK+bbY7MW17a1bqIWA2szmuf2UOMB0iuiTQzMzMrO34sn5mZmZkVOPFuwy0CSU8DJ3dqvikiGrKIx8zMzCxrLhKBiLgs6xjMzMzMSolPN5uZmZlZAReJZmZmZlbARaKZmZmZFXCRaGZmZmYFXCSamZmZWQEXiWZmZmZWwEWimZmZmRXw7ySWoaqqKkaNGsWwYcMYPnw4zz77bNYhmZmZWYlxkVimNm3aRGVlZdZhmJmZWYnK5HSzpLMlrZX0vKStkh6TdGE3fask7cogxtMkPSrpZ5J2S1o+2DGYmZmZZWXQZxIlCVgHrImIG9O2KcBZwHODHU8vvhoRmyS9C9gg6WMR8cOeBrzV2k7VkkcHKbyuNS2/psf1krjqqquQxKc//WkWLVo0SJGZmZnZiSKLmcQrgNaIWNnREBE7gC2S7pG0S1KDpDmdB0qaL+n+vPePSKpJlw+l43dLekLSNEk5SS9IujZv/EOS6iXtk3R3d0FGxL9HxKZ0+TCwDRg/QDnI1JYtW9i2bRs//OEP+frXv86TTz6ZdUhmZmZWYrK4JnESsLWL9k8A1cAUoBJ4RlJ/qpcKYGNE1EpaB9wOzAAuBtYA69N+1cBUoAVolHRfRLzU04YlvRv4L8DXulm/CFgEUFk5ltsuaetH2AMvl8v12mffvn0ATJ06lbq6Oo4cOTKgMRw6dKhPcQx1zkPCeUg4D85BB+ch4TwkSjUPpXTjyuVAXUS0AwckbQY+COzs4/jDQH263AC0RESrpAagKq/fhoh4E0DSHmAC0G2RKGk4UAf8TUS80FWfiFgFrAI4Z+L5cW9DtmltmlfT7brm5maOHDnCqFGjaG5u5gtf+AK33XYbNTXdjzkeuVxuwLd5InIeEs5DwnlwDjo4DwnnIVGqeciimtkNXH+cY9s49hT5KXnLrRER6fIRkplCIuJIWuh1aMlbbqf3HKwC9kXEir4EeOqIYTT2ck1glg4cOMCsWbMAaGtr41Of+hRXX311xlGZmZlZqcmiSNwI3ClpUToDh6TJwBvAHElrgNOB6UAtxxaCTcAtkk4CxgHTihmopNuBMcAfF3M/g2nixIns2LEj6zDMzMysxA16kRgRIWkWsELS54C3SYq/W4GRwA4ggMURsV9SVd7wp4AXgT3AXpKbSYpC0njgi8DPgG3JTdncHxHfLNY+zczMzEpFJhfPRcQrwOwuVtWmr/y+TSQ3u5CeTp7XzTZH5i0v7WpdRKwGVue1z+whxpcB9XAYZmZmZkOWn91sZmZmZgVK6e7mzEh6Gji5U/NNEdGQRTxmZmZmWXORCETEZVnHYGZmZlZKfLrZzMzMzAq4SDQzMzOzAi4SzczMzKyAi0QzMzMzK+Ai0czMzMwKuEg0MzMzswIuEs3MzMysgItEMzMzMyvgItHMzMzMCrhINDMzM7MCLhLNzMzMrICLRDMzMzMroIjIOoYhRdJvgcas4ygBlcDBrIMoAc5DwnlIOA/OQQfnIeE8JLLMw4SIGNvViuGDHUkZaIyID2QdRNYkPes8OA8dnIeE8+AcdHAeEs5DolTz4NPNZmZmZlbARaKZmZmZFXCROPBWZR1AiXAeEs5DwnlIOA/OQQfnIeE8JEoyD75xxczMzMwKeCbRzMzMzAq4SDQzMzOzAi4SB5CkqyU1Svq5pCVZxzNYJD0g6ZeSduW1nS7pcUn70v++J8sYi03S+yRtkrRH0m5Jf5G2l1seTpH0E0k70jwsS9vPlfR0+tn4R0nvyjrWwSBpmKSfSnokfV92eZDUJKlB0nZJz6ZtZfW5AJD0bkkPSvqZpL2SPlxueZB0UfrnoOP1G0m3lmEe/mf6/bhLUl36vVmS3w0uEgeIpGHA14GPARcDcyVdnG1Ug2Y1cHWntiXAhoi4ANiQvh/K2oDPRMTFwIeAP0v//5dbHlqAj0TEFKAauFrSh4CvAH8dEecDrwMLM4xxMP0FsDfvfbnm4YqIqM77Hbhy+1wAfA2oj4j3A1NI/lyUVR4iojH9c1ANXAr8O7COMsqDpHHA/wA+EBGTgGHAjZTod4OLxIEzDfh5RLwQEYeBtcB1Gcc0KCLiSeDXnZqvA9aky2uAjw9qUIMsIl6NiG3p8m9J/gIYR/nlISLiUPp2RPoK4CPAg2n7kM8DgKTxwDXAN9P3ogzz0I2y+lxIGgNMB74FEBGHI+INyiwPnVwJPB8R/0r55WE4cKqk4cBpwKuU6HeDi8SBMw54Ke/9y2lbuTorIl5Nl/cDZ2UZzGCSVAVMBZ6mDPOQnmLdDvwSeBx4HngjItrSLuXy2VgBLAaOpO/PoDzzEMD/k7RV0qK0rdw+F+cCvwL+Lr384JuSKii/POS7EahLl8smDxHxC+CrwL+RFIdvAlsp0e8GF4lWdJH8zlJZ/NaSpJHA94BbI+I3+evKJQ8R0Z6eThpPMsP+/oxDGnSSZgK/jIitWcdSAi6PiN8nuRTnzyRNz19ZJp+L4cDvA/8nIqYCzXQ6pVomeQAgvd7uWuD/dl431POQXm95Hck/HH4PqKDwcq2S4SJx4PwCeF/e+/FpW7k6IOm9AOl/f5lxPEUnaQRJgfgPEfFQ2lx2eeiQnk7bBHwYeHd6agXK47Pxh8C1kppILj35CMk1aeWWh46ZEyLilyTXn02j/D4XLwMvR8TT6fsHSYrGcstDh48B2yLiQPq+nPLwUeDFiPhVRLQCD5F8X5Tkd4OLxIHzDHBBeofSu0im0tdnHFOW1gM3p8s3Az/IMJaiS683+xawNyL+d96qcsvDWEnvTpdPBWaQXJ+5Cbg+7Tbk8xARn4+I8RFRRfJdsDEi5lFmeZBUIWlUxzJwFbCLMvtcRMR+4CVJF6VNVwJ7KLM85JnL7041Q3nl4d+AD0k6Lf17o+PPQkl+N/iJKwNI0n8muQ5pGPBARNyRcUiDQlIdUANUAgeA/wV8H/gucA7wr8DsiOh8c8uQIely4J+ABn53DdoXSK5LLKc8TCa56HoYyT9CvxsRX5Y0kWRG7XTgp8B/jYiW7CIdPJJqgM9GxMxyy0N6vOvSt8OB70TEHZLOoIw+FwCSqkluYnoX8ALw30g/I5RXHipICqWJEfFm2lZWfx7SnwabQ/KrGD8F/pjkGsSS+25wkWhmZmZmBXy62czMzMwKuEg0MzMzswIuEs3MzMysgItEMzMzMyvgItHMzMzMCrhINLOyIKld0va8V9VxbOPjki4e+OhA0u9JerD3ngO6z+r0p7vMzAoM772LmdmQ8Fb6uMB34uPAIyQ/ftsnkobnPZO1WxHxCr/7Md2iS5/uUA18AHhssPZrZicOzySaWdmSdKmkzZK2SvpR3qPB/kTSM5J2SPpe+nSEPyB53uw96UzkeZJykj6QjqlMH8OHpPmS1kvaCGxInzzygKSfSPqppOu6iKVK0q688d+X9LikJkl/Lukv07E/lnR62i8n6WtpPLskTUvbT0/H70z7T07bl0r6e0lPAX8PfBmYk46fI2mapH9J9/PPHU8ISeN5SFK9pH2S7s6L+2pJ29JcbUjbej1eMyt9nkk0s3JxqqTt6fKLwGzgPuC6iPiVpDnAHcAC4KGI+FsASbcDCyPiPknrgUci4sF0XU/7+31gckT8WtKdJI/mW5A+tvAnkp6IiOYexk8CpgKnAD8HPhcRUyX9NfBHJE93AjgtIqolTQceSMctA34aER+X9BHg2ySzhgAXA5dHxFuS5gMfiIg/T49nNPAfI6JN0keBO4FPpuOq03hagEZJ9wFvA38LTI+IFzuKV+CLx3G8ZlZiXCSaWbk45nSzpEkkBdXjabE3DHg1XT0pLQ7fDYwEfnQc+3s879FiVwHXSvps+v4UkkeQ7e1h/KaI+C3wW0lvAg+n7Q3A5Lx+dQAR8aSk0WlRdjlpcRcRGyWdkRaAAOsj4q1u9jkGWCPpAiCAEXnrNuQ9Rm0PMAF4D/BkRLyY7uudHK+ZlRgXiWZWrgTsjogPd7FuNfDxiNiRzrbVdLONNn532c4pndblz5oJ+GRENPYjvvznth7Je3+EY7+7Oz9btbdnrfY0m/dXJMXprPTGnlw38bTT898fx3O8ZlZifE2imZWrRmCspA8DSBoh6T+k60YBr0oaAczLG/PbdF2HJuDSdLmnm05+BPx3pVOWkqa+8/CPmpNu83LgzXS2759I45ZUAxyMiN90Mbbz8YwBfpEuz+/Dvn8MTJd0brqvjtPNxTxeMxskLhLNrCxFxGGSwu4rknYA24E/SFd/CXgaeAr4Wd6wtUBtejPGecBXgT+V9FOgsofd/RXJqdudknan7wfK2+n+VwIL07alwKWSdgLLgZu7GbsJuLjjxhXgbuCudHu9nmmKiF8Bi4CH0hz+Y7qqmMdrZoNEEb2dmTAzs1IkKQd8NiKezToWMxt6PJNoZmZmZgU8k2hmZmZmBTyTaGZmZmYFXCSamZmZWQEXiWZmZmZWwEWimZmZmRVwkWhmZmZmBf4/rt0WCHKCdFYAAAAASUVORK5CYII=\n"
          },
          "metadata": {
            "needs_background": "light"
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## 10 스태킹 앙상블\n",
        "\n",
        " 스태킹(Staking)은 개별적인 여러 알고리즘을 서로 결합해 예측 결과를 도출한다는 점에서 앞에 소개한 배깅(Bagging) 및 부스팅(Boosting)과 공통점을 가지고 있다.\n",
        "\n",
        "가장 큰 차이점은 개별 알고리즘으로 예측한 데이터를 기반으로 다시 예측을 수행한다는 것이다. 즉, 개별 알고리즘의 예측 결과 데이터 세트를 최종적인 메타 데이터 세트로 만들의 별도의 ML 알고리즘으로 최종 학습을 수행하고 테스트 데이터를 기반으로 다시 최종 예측을 수행하는 방식이다.\n",
        "\n",
        "메타 모델: 개별 모델의 예측된 데이터 세트를 다시 기반으로 하여 학습하고 예측하는 방식\n",
        "\n",
        "------------------\n",
        "스태킹 모델은 두 종류의 모델이 필요하다.\n",
        "\n",
        "1. 개별적인 기반 모델\n",
        "\n",
        "2. 최종 메타 모델\n",
        "\n",
        "핵심: 여러 개별 모델의 예측 데이터를 각각 스태킹 형태로 결합해 최종 메타 모델의 학습용 피처 데이터 세트와 테스트용 피처 데이터 세트를 만드는 것.\n",
        "\n",
        "스태킹을 적용할 때는 많은 개별 모델이 필요함: 2~3개의 개별 모델만을 결합해서는 쉽게 예측 성능을 향상시킬 수 없으며, 스태킹을 적용한다고 해서 반드시 성능 향상이 되리라는 보장도 없음. 일반적으로 성능이 비슷한 모델을 결합해 좀 더 나은 성능 향상을 도출하기 위해 적용된다.\n",
        "\n",
        "즉, 여러 개의 모델에 대한 예측값을 합한 후, 스태킹 형태로 쌓은 뒤 이에 대한 예측을 다시 수행하는 것\n",
        "\n",
        "\n",
        "\n",
        "---\n",
        "### 기본 스태킹 모델\n"
      ],
      "metadata": {
        "id": "2y0kxdtayPuR"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import numpy as np\n",
        "\n",
        "from sklearn.neighbors import KNeighborsClassifier\n",
        "from sklearn.ensemble import RandomForestClassifier\n",
        "from sklearn.ensemble import AdaBoostClassifier\n",
        "from sklearn.tree import DecisionTreeClassifier\n",
        "from sklearn.linear_model import LogisticRegression\n",
        "\n",
        "from sklearn.datasets import load_breast_cancer\n",
        "from sklearn.model_selection import train_test_split\n",
        "from sklearn.metrics import accuracy_score\n",
        "\n",
        "cancer_data = load_breast_cancer()\n",
        "\n",
        "X_data = cancer_data.data\n",
        "y_label = cancer_data.target\n",
        "\n",
        "X_train, X_test, y_train, y_test = train_test_split(X_data, y_label, test_size=0.2, random_state=0)"
      ],
      "metadata": {
        "id": "kHs0delpyPzO"
      },
      "execution_count": 24,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "스태킹에 사용될 머신러닝 알고리즘 클래스 생성\n",
        "\n",
        "개별 모델은 KNN, 랜덤 포레스트, 결정트리, 에이다부스트이며, 이들 모델의 예측 결과를 합한 데이터 세트로 학습/예측하는 최종 모델은 로지스틱 회귀이다."
      ],
      "metadata": {
        "id": "-DNwXiE1yP4L"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# 개별 ML 모델 생성\n",
        "knn_clf = KNeighborsClassifier(n_neighbors=4)\n",
        "rf_clf = RandomForestClassifier(n_estimators=100, random_state=0)\n",
        "dt_clf = DecisionTreeClassifier()\n",
        "ada_clf = AdaBoostClassifier(n_estimators=100)\n",
        "\n",
        "# 스태킹으로 만들어진 데이터 세트를 학습, 예측할 최종 모델\n",
        "lr_final = LogisticRegression(C=10)"
      ],
      "metadata": {
        "id": "FIdfrP0PyP9r"
      },
      "execution_count": 25,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [],
      "metadata": {
        "id": "ZQz-6jrmyQCK"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# 개별 모델들을 학습.\n",
        "knn_clf.fit(X_train, y_train)\n",
        "rf_clf.fit(X_train, y_train)\n",
        "dt_clf.fit(X_train, y_train)\n",
        "ada_clf.fit(X_train, y_train)"
      ],
      "metadata": {
        "id": "viJjxiUTyQHf",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "ec59b3e3-99a4-40bf-d29e-6c7dd5d50fc8"
      },
      "execution_count": 26,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "AdaBoostClassifier(n_estimators=100)"
            ]
          },
          "metadata": {},
          "execution_count": 26
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "개별 모델의 예측 데이터 세트를 반환하고 각 모델의 예측 정확도살펴보자."
      ],
      "metadata": {
        "id": "gC1-qOWRyQMj"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "knn_pred = knn_clf.predict(X_test)\n",
        "rf_pred = rf_clf.predict(X_test)\n",
        "dt_pred = dt_clf.predict(X_test)\n",
        "ada_pred = ada_clf.predict(X_test)\n",
        "\n",
        "print('KNN 정확도: {0:.4f}'.format(accuracy_score(y_test, knn_pred)))\n",
        "print('랜덤 포레스트 정확도: {0:.4f}'.format(accuracy_score(y_test, rf_pred)))\n",
        "print('결정 트리 정확도: {0:.4f}'.format(accuracy_score(y_test, dt_pred)))\n",
        "print('에어다부스트 정확도: {0:.4f}'.format(accuracy_score(y_test, ada_pred)))"
      ],
      "metadata": {
        "id": "u8TIu16tyQQ_",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "65567538-0126-4335-e2e7-12ae86b96c8c"
      },
      "execution_count": 28,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "KNN 정확도: 0.9211\n",
            "랜덤 포레스트 정확도: 0.9649\n",
            "결정 트리 정확도: 0.9035\n",
            "에어다부스트 정확도: 0.9561\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "개별 알고리즘으로부터 예측된 예측값을 칼럼 레벨로 옆으로 붙여서 피처 값으로 만들어, 최종 메타 모델인 로지스틱 회귀에서 학습 데이터로 다시 사용한다.\n",
        "\n",
        "반환된 예측 데이터 세트는 1차원 형태의 ndarray이므로 먼저 반환된 예측 결과를 행 형태로 붙인 뒤, 넘파이의 transpose()를 이용해 행과 열 위치를 바꾼 ndarray로 변환하면 된다."
      ],
      "metadata": {
        "id": "hYpA5004yQVH"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "pred = np.array([knn_pred, rf_pred, dt_pred, ada_pred])\n",
        "print(pred.shape)\n",
        "\n",
        "# transpose를 이용해 행과 열의 위치 교환. 칼럼 레벨로 각 알고리즘의 예측 결과를 피처로 만듦\n",
        "pred = np.transpose(pred)\n",
        "print(pred.shape)"
      ],
      "metadata": {
        "id": "EVyezoAtyQal",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "166882e1-f397-44d4-85ad-a9d9908c06c6"
      },
      "execution_count": 29,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "(4, 114)\n",
            "(114, 4)\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "이렇게 예측 데이터로 생성된 데이터 세트를 기반으로 최종 메타 모델인 로지스틱 회귀를 학습하고 예측 정확도를 측정한다."
      ],
      "metadata": {
        "id": "juym3TieyQex"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "lr_final.fit(pred, y_test)\n",
        "final = lr_final.predict(pred)\n",
        "\n",
        "print('최종 메타 모델의 예측 정확도: {0:.4f}'.format(accuracy_score(y_test, final)))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "iyokTkwiPHt4",
        "outputId": "f6038491-bd64-4165-a75e-7325578fc426"
      },
      "execution_count": 30,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "최종 메타 모델의 예측 정확도: 0.9649\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "-> 개별 모델의 예측 데이터를 스태킹으로 재구성해 최종 메타 모델에서 학습하고 예측한 결과, 정확도가 97.37%로 개별 모델 정확도보다 향상됨.\n",
        "\n",
        "-------------\n",
        "이제 과적합 개선하기 위한 cv세트 기반의 스태킹 모델을 살펴보자.\n",
        "\n",
        "### CV세트 기반의 스태킹\n",
        "\n",
        "\n",
        "이 모델은 과적합을 개선하기 위해 최종 메타 모델을 위한 데이터 세트를 만들 때 교차 검증 기반으로 예측된 결과 데이터 세트를 이용한다. \n",
        "\n",
        "개별 모델들이 각각 교차 검증으로 메타 모델을 위한 학습용 스태킹 데이터 생성과 예측을 위한 테스트용 스태킹 데이터를 생성한 뒤 이를 기반으로 메타 모델이 학습과 예측을 수행한다. \n",
        "\n",
        "* 스텝1: 각 모델별로 원본 학습/테스트 데이터를 예측한 결과 값을 기반으로 메타 모델을 위한 학습용/테스트용 데이터를 생성한다.\n",
        "\n",
        "* 스텝2: 스텝1에서 개별 모델들이 생성한 학습용 데이터를 모두 스태킹 형태로 합쳐서 메타 모델이 학습할 최종 학습용 데이터 세트를 생성. 마찬가지로 각 모델들이 생성한 테스트용 데이터를 모두 스태킹 형태로 합쳐서 메타 모델이 예측할 최종 테스트 데이터 세트를 생성. 메타 모델은 최종적으로 생성된 학습 데이터 세트와 원본 학습 데이터의 레이블 데이터를 기반으로 학습한 뒤, 최종적으로 생성된 테스트 데이터 세트를 예측하고, 원본 테스트 데이터의 레이블 데이터를 기반으로 평가한다.\n",
        "\n",
        "\n",
        "\n",
        "---\n",
        "\n",
        "### step 1구현"
      ],
      "metadata": {
        "id": "lG7HPVP5PH0i"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "from sklearn.model_selection import KFold\n",
        "from sklearn.metrics import mean_absolute_error\n",
        "\n",
        "# 개별 기반 모델에서 최종 메타 모델이 사용할 학습 및 테스트용 데이터를 생성하기 위한 함수\n",
        "def get_stacking_base_datasets(model, X_train_n, y_train_n, X_test_n, n_folds) :\n",
        "  # 지정된 n_folds값으로 KFold 생성\n",
        "  kf = KFold(n_splits=n_folds, shuffle=False)\n",
        "  # 추후에 메타 모델이 사용할 학습 데이터 반환을 위해 넘파이 배열 초기화\n",
        "  train_fold_pred = np.zeros((X_train_n.shape[0], 1))\n",
        "  test_pred = np.zeros((X_test_n.shape[0], n_folds))\n",
        "  print(model.__class__.__name__, 'model 시작')\n",
        "\n",
        "  for folder_counter, (train_index, valid_index) in enumerate(kf.split(X_train_n)):\n",
        "    # 입력된 학습 데이터에서 기반모델이 학습/예측할 폴드 데이터 세트 추출\n",
        "    print('\\t 폴드 세트: ', folder_counter, '시작')\n",
        "    X_tr = X_train_n[train_index]\n",
        "    y_tr = y_train_n[train_index]\n",
        "    X_te = X_train_n[valid_index]\n",
        "\n",
        "    # 폴드 세트 내부에서 다시 만들어진 학습 데이터로 기반 모델의 학습 수행\n",
        "    model.fit(X_tr, y_tr)\n",
        "    # 폴드 세트 내부에서 다시 만들어진 검증 데이터로 기반 모델 예측 후 데이터 저장\n",
        "    train_fold_pred[valid_index, :] = model.predict(X_te).reshape(-1,1)\n",
        "    # 입력된 원본 테스트 데이터를 폴드 세트 내 학습된 기반 모델에서 예측 후 데이터 저장\n",
        "    test_pred[:, folder_counter] = model.predict(X_test_n)\n",
        "\n",
        "  # 폴드 세트 내에서 원본 테스트 데이터를 예측한 데이터를 평균하여 테스트 데이터로 생성\n",
        "  test_pred_mean = np.mean(test_pred, axis=1).reshape(-1,1)\n",
        "\n",
        "  # train_fold_pred는 최종 메타 모델이 사용하는 학습 데이터, test_pred_mean은 테스트 데이터\n",
        "  return train_fold_pred, test_pred_mean"
      ],
      "metadata": {
        "id": "liETx9WsPH5h"
      },
      "execution_count": 34,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "이제 여러 개의 분류 모델별로 stack_base_model()함수를 수행한다."
      ],
      "metadata": {
        "id": "r4t2nwPXPIFR"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "knn_train, knn_test = get_stacking_base_datasets(knn_clf, X_train, y_train, X_test, 7)\n",
        "rf_train, rf_test = get_stacking_base_datasets(rf_clf, X_train, y_train, X_test, 7)\n",
        "dt_train, dt_test = get_stacking_base_datasets(dt_clf, X_train, y_train, X_test, 7)\n",
        "ada_train, ada_test = get_stacking_base_datasets(ada_clf, X_train, y_train, X_test, 7)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "B8pY1270PIJW",
        "outputId": "fc83feea-28c1-43f3-f7b0-1da12214b9e9"
      },
      "execution_count": 33,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "KNeighborsClassifier model 시작\n",
            "\t 폴드 세트:  0 시작\n",
            "\t 폴드 세트:  1 시작\n",
            "\t 폴드 세트:  2 시작\n",
            "\t 폴드 세트:  3 시작\n",
            "\t 폴드 세트:  4 시작\n",
            "\t 폴드 세트:  5 시작\n",
            "\t 폴드 세트:  6 시작\n",
            "RandomForestClassifier model 시작\n",
            "\t 폴드 세트:  0 시작\n",
            "\t 폴드 세트:  1 시작\n",
            "\t 폴드 세트:  2 시작\n",
            "\t 폴드 세트:  3 시작\n",
            "\t 폴드 세트:  4 시작\n",
            "\t 폴드 세트:  5 시작\n",
            "\t 폴드 세트:  6 시작\n",
            "DecisionTreeClassifier model 시작\n",
            "\t 폴드 세트:  0 시작\n",
            "\t 폴드 세트:  1 시작\n",
            "\t 폴드 세트:  2 시작\n",
            "\t 폴드 세트:  3 시작\n",
            "\t 폴드 세트:  4 시작\n",
            "\t 폴드 세트:  5 시작\n",
            "\t 폴드 세트:  6 시작\n",
            "AdaBoostClassifier model 시작\n",
            "\t 폴드 세트:  0 시작\n",
            "\t 폴드 세트:  1 시작\n",
            "\t 폴드 세트:  2 시작\n",
            "\t 폴드 세트:  3 시작\n",
            "\t 폴드 세트:  4 시작\n",
            "\t 폴드 세트:  5 시작\n",
            "\t 폴드 세트:  6 시작\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "### step 2구현\n",
        " 앞의 예제에서 get_stacking_base_datasets()호출로 반환된 각 모델별 학습 데이터와 테스트데이터를 합치기만 하면 된다.\n",
        "\n",
        " * concatenate() : 여러 개의 넘파이 배열을 칼럼 또는 로우 레벨로 합쳐주는 기능 제공"
      ],
      "metadata": {
        "id": "SDnsFWBwPIMe"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "Stack_final_X_train = np.concatenate((knn_train, rf_train, dt_train, ada_train), axis=1)\n",
        "Stack_final_X_test = np.concatenate((knn_test, rf_test, dt_test, ada_test), axis=1)\n",
        "print('원본 학습 피처 데이터 Shape:', X_train.shape, '원본 테스트 피처 Shape:', X_test.shape)\n",
        "print('스태킹 학습 피처 데이터 Shape:', Stack_final_X_train.shape,\n",
        "      '스태킹 테스트 피처 데이터 Shape:', Stack_final_X_test.shape)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "EpEDEfeePIPw",
        "outputId": "821742de-152e-457d-9cef-a8e3f563e6fb"
      },
      "execution_count": 35,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "원본 학습 피처 데이터 Shape: (455, 30) 원본 테스트 피처 Shape: (114, 30)\n",
            "스태킹 학습 피처 데이터 Shape: (455, 4) 스태킹 테스트 피처 데이터 Shape: (114, 4)\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Stack_final_X_train: 메타 모델이 학습할 학습용 피처 데이터 세트\n",
        "\n",
        "Stack_final_X_test: 메타 모델이 예측할 테스트용 피처 데이터 세트\n",
        "\n",
        "스태킹 학습 피처 데이터는 원본 학습 피처 데이터와 로우 크기는 같으며, 4개의 개별 모델 예측값을 합친 것이므로, 칼럼 크기는 4\n",
        "\n",
        "\n",
        "---\n",
        "\n",
        "\n",
        "\n",
        "이제 최종 메타 모델인 로지스틱 회귀를 스태킹된 학습용 피처 데이터 세트와 원본 학습 레이블 데이터로 학습한 후, 스태킹된 테스트 데이터 세트로 예측하고, 예측 결과를 원본 테스트 레이블 데이터와 비교해 정확도 측정해보자. "
      ],
      "metadata": {
        "id": "nOEFrEwGPITq"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "lr_final.fit(Stack_final_X_train, y_train)\n",
        "stack_final = lr_final.predict(Stack_final_X_test)\n",
        "\n",
        "print('최종 메타 모델의 예측 정확도: {0:.4f}'.format(accuracy_score(y_test, stack_final)))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "RqM1wS1dPIWU",
        "outputId": "301a0918-01d8-4607-ad20-e5ffdaeb4b43"
      },
      "execution_count": 36,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "최종 메타 모델의 예측 정확도: 0.9737\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "### 정리\n",
        "\n",
        "1. 앙상블 기법 : 결정 트리 기반의 다수의 약한 학습기를 결합해 변동성을 줄여 예측 오류를 줄이고 성능을 개선하고 있다.\n",
        "\n",
        "2. 배깅: 학습 데이터를 중복을 허용하면서 다수의 세트로 샘플링하여 이를 다수의 약한 학습기가 학습한 뒤 최종 결과를 결합해 예측 하는 방식\n",
        "   - 랜덤 포레스트 \n",
        "\n",
        "3. 부스팅: 학습기들이 순차적으로 학습을 진행하면서 예측이 ㅊ틀린 데이터에 대해서는 가중치를 부여해 다음번 학습기가 학습할 때에는 이전에 예측이 틀린 데이터에 대해서는 보다 높은 정확도로 예측할 수 있도록 함\n",
        "   - XGBoost, LightGBM\n",
        "\n",
        "4. 스태킹 : 여러 개의 개별 모델들이 생성한 예측 데이터를 기반으로 최종 메타 모델이 학습할 별도의 학습 데이터 세트와 예측할 테스트 데이터 세트를 재 생성하는 기법"
      ],
      "metadata": {
        "id": "QXK5IjVTyQxk"
      }
    }
  ]
}