앙상블_연습문제_(1).ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": [],
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/wlsisl/ESAA_/blob/main/%EC%95%99%EC%83%81%EB%B8%94_%EC%97%B0%EC%8A%B5%EB%AC%B8%EC%A0%9C_(1).ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## **| 앙상블 학습과 랜덤 포레스트 연습 문제**\n",
        "___\n",
        "- 출처 : 핸즈온 머신러닝 Ch07 앙상블 학습과 랜덤 포레스트 연습문제 2, 7, 8, 9번\n",
        "- 이론적 지식을 묻는 문제의 경우 텍스트 셀을 추가하여 정답을 적어주세요."
      ],
      "metadata": {
        "id": "zCu72vDHGMHo"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# import libraries\n",
        "import numpy as np"
      ],
      "metadata": {
        "id": "hQLO-M61qSVA"
      },
      "execution_count": 1,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "### **1. 직접 투표와 간접 투표 분류기 사이의 차이점은 무엇일까요?**\n",
        "___\n"
      ],
      "metadata": {
        "id": "7ifUj-jvi5NE"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "직접투표: 각 분류기의 예측을 모아서 가장 많이 선택된 클래스를 예측하는 것\n",
        "\n",
        "간접투표: 모든 분류기가 클래스의 확률을 예측할 수 있으면 개별 분류기의 예측을 평균내어 확률이 가장 높은 클래스를 예측할 수 있다.\n",
        "\n",
        "-> 가장 많이 선택된 클래스를 예측하는 것과, 예측을 평균내어 확률이 가장 높은 클래스를 예측하는 것에 차이가 있음."
      ],
      "metadata": {
        "id": "pslif99x5h6M"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### **2. 그레디언트 부스팅 앙상블이 훈련 데이터에 과대 적합되었다면 학습률을 어떻게 해야 할까요?**\n",
        "___"
      ],
      "metadata": {
        "id": "JK-WcK3ujU0E"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "감소시켜야한다. "
      ],
      "metadata": {
        "id": "AIhEKabV5iis"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### **3. [실습] 다음 지시에 따라 투표 기반 분류 모델을 만들어 보세요**\n",
        "___"
      ],
      "metadata": {
        "id": "j3g-_Dq9GiuT"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "#### **STEP 1. MNIST 데이터를 불러들이고, 훈련, 검증, 테스트 데이터로 나누세요.**"
      ],
      "metadata": {
        "id": "7aLghZCVj34L"
      }
    },
    {
      "cell_type": "code",
      "execution_count": 2,
      "metadata": {
        "id": "188lZyYEGJZ7",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "88e84719-d9a5-480a-ab75-51b49f65036e"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "/usr/local/lib/python3.9/dist-packages/sklearn/datasets/_openml.py:968: FutureWarning: The default value of `parser` will change from `'liac-arff'` to `'auto'` in 1.4. You can set `parser='auto'` to silence this warning. Therefore, an `ImportError` will be raised from 1.4 if the dataset is dense and pandas is not installed. Note that the pandas parser may return different data types. See the Notes Section in fetch_openml's API doc for details.\n",
            "  warn(\n"
          ]
        }
      ],
      "source": [
        "# import MNIST dataset\n",
        "from sklearn.datasets import fetch_openml\n",
        "\n",
        "mnist = fetch_openml('mnist_784', version=1, as_frame = False)\n",
        "X, y = mnist[\"data\"], mnist[\"target\"]"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# train/valid/test dataset\n",
        "from sklearn.model_selection import train_test_split\n",
        "\n",
        "X_train_val, X_test, y_train_val, y_test = train_test_split(X, y, test_size=10000, random_state=42)\n",
        "X_train, X_val, y_train, y_val = train_test_split(X_train_val, y_train_val, test_size=10000, random_state=42)"
      ],
      "metadata": {
        "id": "lXF4M2MdIpKa"
      },
      "execution_count": 5,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "####  **STEP 2. 랜덤 포레스트 분류기, 엑스트라 트리 분류기, SVM 분류기, MLP 분류기를 훈련시키세요.**\n",
        "- 모델 파라미터는 `n_estimators=100`, `random_state=42`로 설정합니다. "
      ],
      "metadata": {
        "id": "mHOTxVK3kjnM"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# import package\n",
        "from sklearn.ensemble import RandomForestClassifier, ExtraTreesClassifier\n",
        "from sklearn.svm import LinearSVC\n",
        "from sklearn.neural_network import MLPClassifier"
      ],
      "metadata": {
        "id": "udX8yKD4k9DE"
      },
      "execution_count": 4,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# model fitting\n",
        "from sklearn.ensemble import RandomForestClassifier\n",
        "from sklearn.ensemble import VotingClassifier\n",
        "from sklearn.linear_model import LogisticRegression\n",
        "from sklearn.svm import SVC\n",
        "\n",
        "rnd_clf = RandomForestClassifier(n_estimators=100, random_state=42)\n",
        "extra_clf = ExtraTreesClassifier(n_estimators=100, random_state=42)\n",
        "svm_clf = LinearSVC(random_state=42)\n",
        "mlp_clf = MLPClassifier(random_state=42)\n",
        "\n",
        "model = [rnd_clf, extra_clf,svm_clf,mlp_clf]\n",
        "for i in model:\n",
        "  i.fit(X_train, y_train)"
      ],
      "metadata": {
        "id": "iEG8FFJ1lBw4",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "9073cb23-12d1-482a-d5dc-ae581563dcbc"
      },
      "execution_count": 7,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "/usr/local/lib/python3.9/dist-packages/sklearn/svm/_base.py:1244: ConvergenceWarning: Liblinear failed to converge, increase the number of iterations.\n",
            "  warnings.warn(\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "####  **STEP 3-1. 앞에서 훈련시킨 각 모델을 직접 투표 방법을 사용해 앙상블로 연결하고 훈련시킨 후, `score()`메서드를 이용하여 검증 데이터셋에서의 성능을 평가해보세요.**"
      ],
      "metadata": {
        "id": "lB4OagE2lglK"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "from sklearn.ensemble import VotingClassifier"
      ],
      "metadata": {
        "id": "6gGlXpJrlqzz"
      },
      "execution_count": 8,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# model fitting\n",
        "voting_clf = VotingClassifier(estimators=[('rnd', rnd_clf), ('extra',extra_clf),('svm',svm_clf),('mlp',mlp_clf )],voting='hard')\n",
        "voting_clf.fit(X_train, y_train)"
      ],
      "metadata": {
        "id": "LTnt0h4Hmitr",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 155
        },
        "outputId": "372aab59-3f6a-430e-a57f-f2d1bc4465b6"
      },
      "execution_count": 9,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "/usr/local/lib/python3.9/dist-packages/sklearn/svm/_base.py:1244: ConvergenceWarning: Liblinear failed to converge, increase the number of iterations.\n",
            "  warnings.warn(\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "VotingClassifier(estimators=[('rnd', RandomForestClassifier(random_state=42)),\n",
              "                             ('extra', ExtraTreesClassifier(random_state=42)),\n",
              "                             ('svm', LinearSVC(random_state=42)),\n",
              "                             ('mlp', MLPClassifier(random_state=42))])"
            ],
            "text/html": [
              "<style>#sk-container-id-1 {color: black;background-color: white;}#sk-container-id-1 pre{padding: 0;}#sk-container-id-1 div.sk-toggleable {background-color: white;}#sk-container-id-1 label.sk-toggleable__label {cursor: pointer;display: block;width: 100%;margin-bottom: 0;padding: 0.3em;box-sizing: border-box;text-align: center;}#sk-container-id-1 label.sk-toggleable__label-arrow:before {content: \"▸\";float: left;margin-right: 0.25em;color: #696969;}#sk-container-id-1 label.sk-toggleable__label-arrow:hover:before {color: black;}#sk-container-id-1 div.sk-estimator:hover label.sk-toggleable__label-arrow:before {color: black;}#sk-container-id-1 div.sk-toggleable__content {max-height: 0;max-width: 0;overflow: hidden;text-align: left;background-color: #f0f8ff;}#sk-container-id-1 div.sk-toggleable__content pre {margin: 0.2em;color: black;border-radius: 0.25em;background-color: #f0f8ff;}#sk-container-id-1 input.sk-toggleable__control:checked~div.sk-toggleable__content {max-height: 200px;max-width: 100%;overflow: auto;}#sk-container-id-1 input.sk-toggleable__control:checked~label.sk-toggleable__label-arrow:before {content: \"▾\";}#sk-container-id-1 div.sk-estimator input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-1 div.sk-label input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-1 input.sk-hidden--visually {border: 0;clip: rect(1px 1px 1px 1px);clip: rect(1px, 1px, 1px, 1px);height: 1px;margin: -1px;overflow: hidden;padding: 0;position: absolute;width: 1px;}#sk-container-id-1 div.sk-estimator {font-family: monospace;background-color: #f0f8ff;border: 1px dotted black;border-radius: 0.25em;box-sizing: border-box;margin-bottom: 0.5em;}#sk-container-id-1 div.sk-estimator:hover {background-color: #d4ebff;}#sk-container-id-1 div.sk-parallel-item::after {content: \"\";width: 100%;border-bottom: 1px solid gray;flex-grow: 1;}#sk-container-id-1 div.sk-label:hover label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-1 div.sk-serial::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: 0;}#sk-container-id-1 div.sk-serial {display: flex;flex-direction: column;align-items: center;background-color: white;padding-right: 0.2em;padding-left: 0.2em;position: relative;}#sk-container-id-1 div.sk-item {position: relative;z-index: 1;}#sk-container-id-1 div.sk-parallel {display: flex;align-items: stretch;justify-content: center;background-color: white;position: relative;}#sk-container-id-1 div.sk-item::before, #sk-container-id-1 div.sk-parallel-item::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: -1;}#sk-container-id-1 div.sk-parallel-item {display: flex;flex-direction: column;z-index: 1;position: relative;background-color: white;}#sk-container-id-1 div.sk-parallel-item:first-child::after {align-self: flex-end;width: 50%;}#sk-container-id-1 div.sk-parallel-item:last-child::after {align-self: flex-start;width: 50%;}#sk-container-id-1 div.sk-parallel-item:only-child::after {width: 0;}#sk-container-id-1 div.sk-dashed-wrapped {border: 1px dashed gray;margin: 0 0.4em 0.5em 0.4em;box-sizing: border-box;padding-bottom: 0.4em;background-color: white;}#sk-container-id-1 div.sk-label label {font-family: monospace;font-weight: bold;display: inline-block;line-height: 1.2em;}#sk-container-id-1 div.sk-label-container {text-align: center;}#sk-container-id-1 div.sk-container {/* jupyter's `normalize.less` sets `[hidden] { display: none; }` but bootstrap.min.css set `[hidden] { display: none !important; }` so we also need the `!important` here to be able to override the default hidden behavior on the sphinx rendered scikit-learn.org. See: https://github.com/scikit-learn/scikit-learn/issues/21755 */display: inline-block !important;position: relative;}#sk-container-id-1 div.sk-text-repr-fallback {display: none;}</style><div id=\"sk-container-id-1\" class=\"sk-top-container\"><div class=\"sk-text-repr-fallback\"><pre>VotingClassifier(estimators=[(&#x27;rnd&#x27;, RandomForestClassifier(random_state=42)),\n",
              "                             (&#x27;extra&#x27;, ExtraTreesClassifier(random_state=42)),\n",
              "                             (&#x27;svm&#x27;, LinearSVC(random_state=42)),\n",
              "                             (&#x27;mlp&#x27;, MLPClassifier(random_state=42))])</pre><b>In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook. <br />On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.</b></div><div class=\"sk-container\" hidden><div class=\"sk-item sk-dashed-wrapped\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-1\" type=\"checkbox\" ><label for=\"sk-estimator-id-1\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">VotingClassifier</label><div class=\"sk-toggleable__content\"><pre>VotingClassifier(estimators=[(&#x27;rnd&#x27;, RandomForestClassifier(random_state=42)),\n",
              "                             (&#x27;extra&#x27;, ExtraTreesClassifier(random_state=42)),\n",
              "                             (&#x27;svm&#x27;, LinearSVC(random_state=42)),\n",
              "                             (&#x27;mlp&#x27;, MLPClassifier(random_state=42))])</pre></div></div></div><div class=\"sk-parallel\"><div class=\"sk-parallel-item\"><div class=\"sk-item\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><label>rnd</label></div></div><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-2\" type=\"checkbox\" ><label for=\"sk-estimator-id-2\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">RandomForestClassifier</label><div class=\"sk-toggleable__content\"><pre>RandomForestClassifier(random_state=42)</pre></div></div></div></div></div></div><div class=\"sk-parallel-item\"><div class=\"sk-item\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><label>extra</label></div></div><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-3\" type=\"checkbox\" ><label for=\"sk-estimator-id-3\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">ExtraTreesClassifier</label><div class=\"sk-toggleable__content\"><pre>ExtraTreesClassifier(random_state=42)</pre></div></div></div></div></div></div><div class=\"sk-parallel-item\"><div class=\"sk-item\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><label>svm</label></div></div><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-4\" type=\"checkbox\" ><label for=\"sk-estimator-id-4\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">LinearSVC</label><div class=\"sk-toggleable__content\"><pre>LinearSVC(random_state=42)</pre></div></div></div></div></div></div><div class=\"sk-parallel-item\"><div class=\"sk-item\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><label>mlp</label></div></div><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-5\" type=\"checkbox\" ><label for=\"sk-estimator-id-5\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">MLPClassifier</label><div class=\"sk-toggleable__content\"><pre>MLPClassifier(random_state=42)</pre></div></div></div></div></div></div></div></div></div></div>"
            ]
          },
          "metadata": {},
          "execution_count": 9
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# model test\n",
        "from sklearn.metrics import accuracy_score\n",
        "voting_clf.score(X_val, y_val)"
      ],
      "metadata": {
        "id": "6M8p66FHmt0U",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "c8fec25b-c7e6-47de-b6c7-280b77beea75"
      },
      "execution_count": 10,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "0.9721"
            ]
          },
          "metadata": {},
          "execution_count": 10
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "####  **STEP 3-2. 검증 데이터셋에서 각 분류 모델의 성능을 `score()` 메서드를 이용하여 확인해보고, 가장 성능이 낮은 모델을 제거하여 그 결과를 비교해보세요.**\n",
        "- Hint : 가장 성능이 낮은 모델을 제거할 때 `del`를 활용해보세요"
      ],
      "metadata": {
        "id": "tgmsaye3oLYQ"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# 각 분류 모델 학습\n",
        "voting_clf = VotingClassifier(estimators=[('rnd', rnd_clf), ('extra',extra_clf),('svm',svm_clf),('mlp',mlp_clf )],voting='hard')\n",
        "voting_clf.fit(X_train, y_train)"
      ],
      "metadata": {
        "id": "-Ykw6-_4rV9S",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 155
        },
        "outputId": "261093b9-070d-4081-f4ed-81fc6bb11996"
      },
      "execution_count": 32,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "/usr/local/lib/python3.9/dist-packages/sklearn/svm/_base.py:1244: ConvergenceWarning: Liblinear failed to converge, increase the number of iterations.\n",
            "  warnings.warn(\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "VotingClassifier(estimators=[('rnd', RandomForestClassifier(random_state=42)),\n",
              "                             ('extra', ExtraTreesClassifier(random_state=42)),\n",
              "                             ('svm', LinearSVC(random_state=42)),\n",
              "                             ('mlp', MLPClassifier(random_state=42))])"
            ],
            "text/html": [
              "<style>#sk-container-id-4 {color: black;background-color: white;}#sk-container-id-4 pre{padding: 0;}#sk-container-id-4 div.sk-toggleable {background-color: white;}#sk-container-id-4 label.sk-toggleable__label {cursor: pointer;display: block;width: 100%;margin-bottom: 0;padding: 0.3em;box-sizing: border-box;text-align: center;}#sk-container-id-4 label.sk-toggleable__label-arrow:before {content: \"▸\";float: left;margin-right: 0.25em;color: #696969;}#sk-container-id-4 label.sk-toggleable__label-arrow:hover:before {color: black;}#sk-container-id-4 div.sk-estimator:hover label.sk-toggleable__label-arrow:before {color: black;}#sk-container-id-4 div.sk-toggleable__content {max-height: 0;max-width: 0;overflow: hidden;text-align: left;background-color: #f0f8ff;}#sk-container-id-4 div.sk-toggleable__content pre {margin: 0.2em;color: black;border-radius: 0.25em;background-color: #f0f8ff;}#sk-container-id-4 input.sk-toggleable__control:checked~div.sk-toggleable__content {max-height: 200px;max-width: 100%;overflow: auto;}#sk-container-id-4 input.sk-toggleable__control:checked~label.sk-toggleable__label-arrow:before {content: \"▾\";}#sk-container-id-4 div.sk-estimator input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-4 div.sk-label input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-4 input.sk-hidden--visually {border: 0;clip: rect(1px 1px 1px 1px);clip: rect(1px, 1px, 1px, 1px);height: 1px;margin: -1px;overflow: hidden;padding: 0;position: absolute;width: 1px;}#sk-container-id-4 div.sk-estimator {font-family: monospace;background-color: #f0f8ff;border: 1px dotted black;border-radius: 0.25em;box-sizing: border-box;margin-bottom: 0.5em;}#sk-container-id-4 div.sk-estimator:hover {background-color: #d4ebff;}#sk-container-id-4 div.sk-parallel-item::after {content: \"\";width: 100%;border-bottom: 1px solid gray;flex-grow: 1;}#sk-container-id-4 div.sk-label:hover label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-4 div.sk-serial::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: 0;}#sk-container-id-4 div.sk-serial {display: flex;flex-direction: column;align-items: center;background-color: white;padding-right: 0.2em;padding-left: 0.2em;position: relative;}#sk-container-id-4 div.sk-item {position: relative;z-index: 1;}#sk-container-id-4 div.sk-parallel {display: flex;align-items: stretch;justify-content: center;background-color: white;position: relative;}#sk-container-id-4 div.sk-item::before, #sk-container-id-4 div.sk-parallel-item::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: -1;}#sk-container-id-4 div.sk-parallel-item {display: flex;flex-direction: column;z-index: 1;position: relative;background-color: white;}#sk-container-id-4 div.sk-parallel-item:first-child::after {align-self: flex-end;width: 50%;}#sk-container-id-4 div.sk-parallel-item:last-child::after {align-self: flex-start;width: 50%;}#sk-container-id-4 div.sk-parallel-item:only-child::after {width: 0;}#sk-container-id-4 div.sk-dashed-wrapped {border: 1px dashed gray;margin: 0 0.4em 0.5em 0.4em;box-sizing: border-box;padding-bottom: 0.4em;background-color: white;}#sk-container-id-4 div.sk-label label {font-family: monospace;font-weight: bold;display: inline-block;line-height: 1.2em;}#sk-container-id-4 div.sk-label-container {text-align: center;}#sk-container-id-4 div.sk-container {/* jupyter's `normalize.less` sets `[hidden] { display: none; }` but bootstrap.min.css set `[hidden] { display: none !important; }` so we also need the `!important` here to be able to override the default hidden behavior on the sphinx rendered scikit-learn.org. See: https://github.com/scikit-learn/scikit-learn/issues/21755 */display: inline-block !important;position: relative;}#sk-container-id-4 div.sk-text-repr-fallback {display: none;}</style><div id=\"sk-container-id-4\" class=\"sk-top-container\"><div class=\"sk-text-repr-fallback\"><pre>VotingClassifier(estimators=[(&#x27;rnd&#x27;, RandomForestClassifier(random_state=42)),\n",
              "                             (&#x27;extra&#x27;, ExtraTreesClassifier(random_state=42)),\n",
              "                             (&#x27;svm&#x27;, LinearSVC(random_state=42)),\n",
              "                             (&#x27;mlp&#x27;, MLPClassifier(random_state=42))])</pre><b>In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook. <br />On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.</b></div><div class=\"sk-container\" hidden><div class=\"sk-item sk-dashed-wrapped\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-12\" type=\"checkbox\" ><label for=\"sk-estimator-id-12\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">VotingClassifier</label><div class=\"sk-toggleable__content\"><pre>VotingClassifier(estimators=[(&#x27;rnd&#x27;, RandomForestClassifier(random_state=42)),\n",
              "                             (&#x27;extra&#x27;, ExtraTreesClassifier(random_state=42)),\n",
              "                             (&#x27;svm&#x27;, LinearSVC(random_state=42)),\n",
              "                             (&#x27;mlp&#x27;, MLPClassifier(random_state=42))])</pre></div></div></div><div class=\"sk-parallel\"><div class=\"sk-parallel-item\"><div class=\"sk-item\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><label>rnd</label></div></div><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-13\" type=\"checkbox\" ><label for=\"sk-estimator-id-13\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">RandomForestClassifier</label><div class=\"sk-toggleable__content\"><pre>RandomForestClassifier(random_state=42)</pre></div></div></div></div></div></div><div class=\"sk-parallel-item\"><div class=\"sk-item\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><label>extra</label></div></div><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-14\" type=\"checkbox\" ><label for=\"sk-estimator-id-14\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">ExtraTreesClassifier</label><div class=\"sk-toggleable__content\"><pre>ExtraTreesClassifier(random_state=42)</pre></div></div></div></div></div></div><div class=\"sk-parallel-item\"><div class=\"sk-item\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><label>svm</label></div></div><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-15\" type=\"checkbox\" ><label for=\"sk-estimator-id-15\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">LinearSVC</label><div class=\"sk-toggleable__content\"><pre>LinearSVC(random_state=42)</pre></div></div></div></div></div></div><div class=\"sk-parallel-item\"><div class=\"sk-item\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><label>mlp</label></div></div><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-16\" type=\"checkbox\" ><label for=\"sk-estimator-id-16\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">MLPClassifier</label><div class=\"sk-toggleable__content\"><pre>MLPClassifier(random_state=42)</pre></div></div></div></div></div></div></div></div></div></div>"
            ]
          },
          "metadata": {},
          "execution_count": 32
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# 각 분류 모델의 성능 확인\n",
        "[i.score(X_val, y_val) for i in model]"
      ],
      "metadata": {
        "id": "ghPFb287om1Z",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "4a7a01bc-c325-4183-f6b2-a7fc1d9a3cca"
      },
      "execution_count": 12,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[0.9692, 0.9715, 0.8695, 0.9614]"
            ]
          },
          "metadata": {},
          "execution_count": 12
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "- Q. 어떤 모델의 성능이 가장 낮나요?\n",
        "- A."
      ],
      "metadata": {
        "id": "GN-HIeWhosGs"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# 가장 성능이 낮은 모델 제거\n",
        "del voting_clf.estimators_[2]"
      ],
      "metadata": {
        "id": "shHFXdWcoqf6"
      },
      "execution_count": 14,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "voting_clf.estimators_"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "o1_dGPOMELbP",
        "outputId": "3d17c5ad-4d7b-4d06-d53d-1922569e1daa"
      },
      "execution_count": 33,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[RandomForestClassifier(random_state=42),\n",
              " ExtraTreesClassifier(random_state=42),\n",
              " LinearSVC(random_state=42),\n",
              " MLPClassifier(random_state=42)]"
            ]
          },
          "metadata": {},
          "execution_count": 33
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# model fitting\n",
        "model= voting_clf.estimators_\n",
        "for i in model:\n",
        "  i.fit(X_train, y_train)"
      ],
      "metadata": {
        "id": "PpV5S__I2U3F"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# 모델 제거 후 성능 확인\n",
        "voting_clf.score(X_val, y_val)"
      ],
      "metadata": {
        "id": "BMTkSkbNpRQi"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "### **4. 다음 단계를 따라 앞에서 훈련시킨 분류 모델들을 이용하여 스태킹 앙상블을 구성해보자.**\n",
        "___"
      ],
      "metadata": {
        "id": "-pDjW5XcHPOt"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "#### **STEP 1. 3번 문제의 각 분류 모델을 실행해서 검증 세트에서 예측을 만들고, 그 결과로 훈련 세트를 만들어 보세요.**"
      ],
      "metadata": {
        "id": "_xhEB_KtH47q"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "X_val_predictions = np.empty((len(X_val), len(model)), dtype=np.float32)\n",
        "\n",
        "for index, i in enumerate(model):\n",
        "    X_val_predictions[:, index] = i.predict(X_val)"
      ],
      "metadata": {
        "id": "3CLxYCROIAk6"
      },
      "execution_count": 19,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "####  **STEP 2. 새로운 훈련 세트를 이용하여 랜덤 포레스트 분류 모델을 학습시켜 보세요.**"
      ],
      "metadata": {
        "id": "AHgSqi-zIBCd"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "rnd_forest_blender = RandomForestClassifier(n_estimators=200, random_state=42)\n",
        "rnd_forest_blender.fit(X_val_predictions, y_val)"
      ],
      "metadata": {
        "id": "SsBT_d0MIH-V",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 77
        },
        "outputId": "4b26a58b-c289-4d7e-a8af-d28ec2733f83"
      },
      "execution_count": 20,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "RandomForestClassifier(n_estimators=200, random_state=42)"
            ],
            "text/html": [
              "<style>#sk-container-id-3 {color: black;background-color: white;}#sk-container-id-3 pre{padding: 0;}#sk-container-id-3 div.sk-toggleable {background-color: white;}#sk-container-id-3 label.sk-toggleable__label {cursor: pointer;display: block;width: 100%;margin-bottom: 0;padding: 0.3em;box-sizing: border-box;text-align: center;}#sk-container-id-3 label.sk-toggleable__label-arrow:before {content: \"▸\";float: left;margin-right: 0.25em;color: #696969;}#sk-container-id-3 label.sk-toggleable__label-arrow:hover:before {color: black;}#sk-container-id-3 div.sk-estimator:hover label.sk-toggleable__label-arrow:before {color: black;}#sk-container-id-3 div.sk-toggleable__content {max-height: 0;max-width: 0;overflow: hidden;text-align: left;background-color: #f0f8ff;}#sk-container-id-3 div.sk-toggleable__content pre {margin: 0.2em;color: black;border-radius: 0.25em;background-color: #f0f8ff;}#sk-container-id-3 input.sk-toggleable__control:checked~div.sk-toggleable__content {max-height: 200px;max-width: 100%;overflow: auto;}#sk-container-id-3 input.sk-toggleable__control:checked~label.sk-toggleable__label-arrow:before {content: \"▾\";}#sk-container-id-3 div.sk-estimator input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-3 div.sk-label input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-3 input.sk-hidden--visually {border: 0;clip: rect(1px 1px 1px 1px);clip: rect(1px, 1px, 1px, 1px);height: 1px;margin: -1px;overflow: hidden;padding: 0;position: absolute;width: 1px;}#sk-container-id-3 div.sk-estimator {font-family: monospace;background-color: #f0f8ff;border: 1px dotted black;border-radius: 0.25em;box-sizing: border-box;margin-bottom: 0.5em;}#sk-container-id-3 div.sk-estimator:hover {background-color: #d4ebff;}#sk-container-id-3 div.sk-parallel-item::after {content: \"\";width: 100%;border-bottom: 1px solid gray;flex-grow: 1;}#sk-container-id-3 div.sk-label:hover label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-3 div.sk-serial::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: 0;}#sk-container-id-3 div.sk-serial {display: flex;flex-direction: column;align-items: center;background-color: white;padding-right: 0.2em;padding-left: 0.2em;position: relative;}#sk-container-id-3 div.sk-item {position: relative;z-index: 1;}#sk-container-id-3 div.sk-parallel {display: flex;align-items: stretch;justify-content: center;background-color: white;position: relative;}#sk-container-id-3 div.sk-item::before, #sk-container-id-3 div.sk-parallel-item::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: -1;}#sk-container-id-3 div.sk-parallel-item {display: flex;flex-direction: column;z-index: 1;position: relative;background-color: white;}#sk-container-id-3 div.sk-parallel-item:first-child::after {align-self: flex-end;width: 50%;}#sk-container-id-3 div.sk-parallel-item:last-child::after {align-self: flex-start;width: 50%;}#sk-container-id-3 div.sk-parallel-item:only-child::after {width: 0;}#sk-container-id-3 div.sk-dashed-wrapped {border: 1px dashed gray;margin: 0 0.4em 0.5em 0.4em;box-sizing: border-box;padding-bottom: 0.4em;background-color: white;}#sk-container-id-3 div.sk-label label {font-family: monospace;font-weight: bold;display: inline-block;line-height: 1.2em;}#sk-container-id-3 div.sk-label-container {text-align: center;}#sk-container-id-3 div.sk-container {/* jupyter's `normalize.less` sets `[hidden] { display: none; }` but bootstrap.min.css set `[hidden] { display: none !important; }` so we also need the `!important` here to be able to override the default hidden behavior on the sphinx rendered scikit-learn.org. See: https://github.com/scikit-learn/scikit-learn/issues/21755 */display: inline-block !important;position: relative;}#sk-container-id-3 div.sk-text-repr-fallback {display: none;}</style><div id=\"sk-container-id-3\" class=\"sk-top-container\"><div class=\"sk-text-repr-fallback\"><pre>RandomForestClassifier(n_estimators=200, random_state=42)</pre><b>In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook. <br />On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.</b></div><div class=\"sk-container\" hidden><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-11\" type=\"checkbox\" checked><label for=\"sk-estimator-id-11\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">RandomForestClassifier</label><div class=\"sk-toggleable__content\"><pre>RandomForestClassifier(n_estimators=200, random_state=42)</pre></div></div></div></div></div>"
            ]
          },
          "metadata": {},
          "execution_count": 20
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "- 이 랜덤 포레스트 분류 모델이 바로 블렌더에 해당합니다. "
      ],
      "metadata": {
        "id": "43MJatnzquTH"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "####  **STEP 3. 이제 테스트셋에서 스태킹 앙상블 모델을 평가해보세요.**\n",
        "- 성능 평가 지표로 **정확도**를 이용하세요."
      ],
      "metadata": {
        "id": "zWozt0n2IJZL"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# 각 분류 모델의 예측을 만들어 새로운 데이터셋 생성\n",
        "X_test_predictions = np.empty((len(X_test), len(model)), dtype=np.float32)\n",
        "\n",
        "for index, estimator in enumerate(model):\n",
        "    X_test_predictions[:, index] = estimator.predict(X_test)"
      ],
      "metadata": {
        "id": "Yo4H-hTRIW7-"
      },
      "execution_count": 21,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# 새로운 데이터셋을 이용하여 블렌더로 예측\n",
        "y_pred = rnd_forest_blender.predict(X_test_predictions)"
      ],
      "metadata": {
        "id": "fVdwwmSYKDmF"
      },
      "execution_count": 22,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# model test\n",
        "from sklearn.metrics import accuracy_score\n",
        "accuracy_score(y_test, y_pred)"
      ],
      "metadata": {
        "id": "ijlm1VbOKFSN",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "ddb7ed70-ca1b-4152-f634-c4fca9ded032"
      },
      "execution_count": 23,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "0.9687"
            ]
          },
          "metadata": {},
          "execution_count": 23
        }
      ]
    }
  ]
}