三_数据预处理与特征工程.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 数据挖掘的五大流程  \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1. 获取数据\n",
    "2. **数据预处理**\n",
    "数据预处理是从数据中检测，纠正或删除损坏，不准确或不适用于模型的记录的过程\n",
    "可能面对的问题有：数据类型不同，比如有的是文字，有的是数字，有的含时间序列，有的连续，有的间断。\n",
    "也可能，数据的质量不行，有噪声，有异常，有缺失，数据出错，量纲不一，有重复，数据是偏态，数据量太\n",
    "大或太小\n",
    "数据预处理的目的：让数据适应模型，匹配模型的需求\n",
    "3. **特征工程**：\n",
    "特征工程是将原始数据转换为更能代表预测模型的潜在问题的特征的过程，可以通过挑选最相关的特征，提取\n",
    "特征以及创造特征来实现。其中创造特征又经常以降维算法的方式实现。\n",
    "可能面对的问题有：特征之间有相关性，特征和标签无关，特征太多或太小，或者干脆就无法表现出应有的数\n",
    "据现象或无法展示数据的真实面貌\n",
    "特征工程的目的：1) 降低计算成本，2) 提升模型上限\n",
    "4. 建模，测试模型并预测出结果\n",
    "5. 上线，验证模型效果"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## sklearn中的数据预处理与特征工程模块"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "+ 模块preprocessing：几乎包含数据预处理的所有内容\n",
    "+ 模块Impute：填补缺失值专用\n",
    "+ 模块feature_selection：包含特征选择的各种方法的实践\n",
    "+ 模块decomposition：包含降维算法"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 数据预处理 Precessing &Impute"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 数据无量纲化(去单位化)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "在机器学习算法实践中，我们往往有着将不同规格的数据转换到同一规格，或不同分布的数据转换到某个特定分布\n",
    "的需求，这种需求统称为将数据“无量纲化”。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "譬如梯度和矩阵为核心的算法中，譬如逻辑回归，支持向量机，神经\n",
    "网络，无量纲化可以加快求解速度；而在距离类模型，譬如K近邻，K-Means聚类中，无量纲化可以帮我们提升模\n",
    "型精度，避免某一个取值范围特别大的特征对距离计算造成影响。（一个特例是决策树和树的集成算法们，对决策\n",
    "树我们不需要无量纲化，决策树可以把任意数据都处理得很好。）"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "数据的无量纲化可以是线性的，也可以是非线性的。线性的无量纲化包括**中心化**（Zero-centered或者Mean-subtraction）处理和**缩放处理**（Scale）。**中心化的本质是让所有记录减去一个固定值，即让数据样本数据平移到\n",
    "某个位置**。**缩放的本质是通过除以一个固定值，将数据固定在某个范围之中**，取对数也算是一种缩放处理。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### preprocessing.MinMaxScaler"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "当数据(x)按照**最小值**中心化后，再按**极差（最大值 - 最小值）缩放**，数据移动了最小值个单位，并且会被收敛到\n",
    "[0,1]之间，而这个过程，就叫做数据**归一化**(Normalization，又称Min-Max Scaling)。注意，Normalization是归\n",
    "一化，不是正则化，真正的正则化是regularization，不是数据预处理的一种手段。归一化之后的数据服从正态分\n",
    "布，公式如下:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### 归一化"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    ">归一化：１）把数据变成(０，１)之间的小数。主要是为了数据处理方便提出来的，把数据映射到0～1范围之内处理，更加便捷快速。２）把有量纲表达式变成无量纲表达式，便于不同单位或量级的指标能够进行比较和加权。归一化是一种简化计算的方式，即将有量纲的表达式，经过变换，化为无量纲的表达式，成为纯量。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "归一化:  \n",
    "+ Max-min-normalization  \n",
    "$$\\dfrac{x - x_{min}}{x_{max} - x_{min}}$$  \n",
    "+ 均值归一化  \n",
    "$$\\dfrac{x - \\mu}{x_{max} - x_{min}}$$  \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "缺点：当有新数据加入时，可能导致max和min的变化，需要重新定义。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "+ 非线性归一化\n",
    "    + 对数函数转换：y = log10(x)\n",
    "    + 反余切函数转换：y = atan(x) * 2 / π\n",
    "\n",
    "经常用在数据分化比较大的场景，有些数值很大，有些很小。通过一些数学函数，将原始值进行映射。该方法包括 log、指数，正切等。需要根据数据分布的情况，决定非线性函数的曲线，比如log(V, 2)还是log(V, 10)等。\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### 标准化"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    ">标准化：在机器学习中，我们可能要处理不同种类的资料，例如，音讯和图片上的像素值，这些资料可能是高维度的，资料标准化后会使每个特征中的数值平均变为0(将每个特征的值都减掉原始资料中该特征的平均)、标准差变为1，这个方法被广泛的使用在许多机器学习算法中(例如：支持向量机、逻辑回归和类神经网络)。\n",
    "\n",
    "标准化：  \n",
    "$$\\dfrac{x - \\mu}{\\sigma} \\;\\;\\;(其中，\\mu为x的平均值,\\sigma为x的标准差)$$\n",
    "\n",
    ">中心化：平均值为0，对标准差无要求\n",
    "\n",
    "##### 中心化\n",
    "中心化：  \n",
    "$$x - \\mu$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 为什么要归一化/标准化？"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "归一化/标准化实质是一种线性变换，线性变换有很多良好的性质，这些性质决定了对数据改变后不会造成“失效”，反而能提高数据的表现，这些性质是归一化/标准化的前提。比如有一个很重要的性质：线性变换不会改变原始数据的数值排序。  \n",
    "1. 某些模型求解需要  \n",
    "\n",
    "    + 在使用梯度下降的方法求解最优化问题时， 归一化/标准化后可以加快梯度下降的求解速度，即提升模型的收敛速度。如左图所示，未归一化/标准化时形成的等高线偏椭圆，迭代时很有可能走“之”字型路线（垂直长轴），从而导致迭代。  很多次才能收敛。而如右图对两个特征进行了归一化，对应的等高线就会变圆，在梯度下降进行求解时能较快的收敛。\n",
    "\n",
    "![](images/3_1.png)  \n",
    "\n",
    "+ 一些分类器需要计算样本之间的距离(如欧氏距离)，例如KNN。如果一个特征值域范围非常大，那么距离计算就主要取决于这个特征，从而与实际情况相悖(比如这时实际情况是值域范围小的特征更重要)。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 归一化sklearn实现"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-12-10T09:56:23.272238Z",
     "start_time": "2019-12-10T09:56:23.262322Z"
    }
   },
   "outputs": [],
   "source": [
    "from sklearn.preprocessing import MinMaxScaler\n",
    "import numpy as np \n",
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-12-10T09:56:23.641266Z",
     "start_time": "2019-12-10T09:56:23.629359Z"
    }
   },
   "outputs": [],
   "source": [
    "data = np.array([[-1, 2], [-0.5, 6], [0, 10], [1, 18]])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-12-10T09:56:23.948894Z",
     "start_time": "2019-12-10T09:56:23.931426Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>-1.0</td>\n",
       "      <td>2.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>-0.5</td>\n",
       "      <td>6.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.0</td>\n",
       "      <td>10.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1.0</td>\n",
       "      <td>18.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     0     1\n",
       "0 -1.0   2.0\n",
       "1 -0.5   6.0\n",
       "2  0.0  10.0\n",
       "3  1.0  18.0"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.DataFrame(data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 在归一化到0-1范围"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-12-10T09:56:58.503965Z",
     "start_time": "2019-12-10T09:56:58.497518Z"
    }
   },
   "outputs": [],
   "source": [
    "# 实现归一化\n",
    "scaler = MinMaxScaler()\n",
    "scaler = scaler.fit(data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-12-10T09:57:13.138722Z",
     "start_time": "2019-12-10T09:57:13.119858Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[0.  , 0.  ],\n",
       "       [0.25, 0.25],\n",
       "       [0.5 , 0.5 ],\n",
       "       [1.  , 1.  ]])"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "scaler.transform(data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-12-10T09:57:58.674959Z",
     "start_time": "2019-12-10T09:57:58.656553Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[0.  , 0.  ],\n",
       "       [0.25, 0.25],\n",
       "       [0.5 , 0.5 ],\n",
       "       [1.  , 1.  ]])"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 一步到位\n",
    "result = MinMaxScaler().fit_transform(data)\n",
    "result"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-12-10T09:59:43.412879Z",
     "start_time": "2019-12-10T09:59:43.397297Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[0.  , 0.  ],\n",
       "       [0.25, 0.25],\n",
       "       [0.5 , 0.5 ],\n",
       "       [1.  , 1.  ]])"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 实现复原效果\n",
    "scaler = MinMaxScaler().fit(data)\n",
    "result = scaler.transform(data)\n",
    "result"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-12-10T10:00:02.344722Z",
     "start_time": "2019-12-10T10:00:02.329349Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[-1. ,  2. ],\n",
       "       [-0.5,  6. ],\n",
       "       [ 0. , 10. ],\n",
       "       [ 1. , 18. ]])"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "scaler.inverse_transform(result)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 归一化到指定范围\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-12-10T10:01:41.243321Z",
     "start_time": "2019-12-10T10:01:41.228816Z"
    }
   },
   "outputs": [],
   "source": [
    "scaler = MinMaxScaler(feature_range=(5, 10)).fit(data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-12-10T10:02:06.383114Z",
     "start_time": "2019-12-10T10:02:06.362181Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ 5.  ,  5.  ],\n",
       "       [ 6.25,  6.25],\n",
       "       [ 7.5 ,  7.5 ],\n",
       "       [10.  , 10.  ]])"
      ]
     },
     "execution_count": 51,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "scaler.transform(data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 使用numpy实现归一化  \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-12-10T10:03:31.968948Z",
     "start_time": "2019-12-10T10:03:31.961015Z"
    }
   },
   "outputs": [],
   "source": [
    "data_norm = (data - data.min(axis=0))  / (data.max(axis=0) - data.min(axis=0))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-12-10T10:03:42.764169Z",
     "start_time": "2019-12-10T10:03:42.743333Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[0.  , 0.  ],\n",
       "       [0.25, 0.25],\n",
       "       [0.5 , 0.5 ],\n",
       "       [1.  , 1.  ]])"
      ]
     },
     "execution_count": 56,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data_norm"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-12-10T10:06:05.495954Z",
     "start_time": "2019-12-10T10:06:05.477601Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[-1. ,  2. ],\n",
       "       [-0.5,  6. ],\n",
       "       [ 0. , 10. ],\n",
       "       [ 1. , 18. ]])"
      ]
     },
     "execution_count": 61,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 反归一化\n",
    "data_norm * (data.max(axis=0) - data.min(axis=0)) + data.min(axis=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-12-10T10:05:12.841601Z",
     "start_time": "2019-12-10T10:05:12.832173Z"
    }
   },
   "source": [
    "## 标准化sklearn实现"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "当数据(x)按均值(μ)中心化后，再按标准差(σ)缩放，数据就会服从为均值为0，方差为1的正态分布（即标准正态分\n",
    "布），而这个过程，就叫做数据标准化(Standardization，又称Z-score normalization)，公式如下："
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$x^{*} = \\dfrac{x- \\mu}{\\sigma}$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-12-10T10:57:54.287021Z",
     "start_time": "2019-12-10T10:57:54.274036Z"
    }
   },
   "outputs": [],
   "source": [
    "from sklearn.preprocessing import StandardScaler"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-12-10T10:58:12.169192Z",
     "start_time": "2019-12-10T10:58:12.161255Z"
    }
   },
   "outputs": [],
   "source": [
    "data = np.array([[-1, 2], [-0.5, 6], [0, 10], [1, 18]])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-12-10T10:58:26.509811Z",
     "start_time": "2019-12-10T10:58:26.499818Z"
    }
   },
   "outputs": [],
   "source": [
    "scaler = StandardScaler().fit(data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-12-10T10:59:13.817997Z",
     "start_time": "2019-12-10T10:59:13.799469Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([-0.125,  9.   ])"
      ]
     },
     "execution_count": 67,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "scaler.mean_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-12-10T10:59:24.126056Z",
     "start_time": "2019-12-10T10:59:24.115409Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ 0.546875, 35.      ])"
      ]
     },
     "execution_count": 69,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "scaler.var_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-12-10T10:59:24.494528Z",
     "start_time": "2019-12-10T10:59:24.473390Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[-1.18321596, -1.18321596],\n",
       "       [-0.50709255, -0.50709255],\n",
       "       [ 0.16903085,  0.16903085],\n",
       "       [ 1.52127766,  1.52127766]])"
      ]
     },
     "execution_count": 70,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "scaler.fit_transform(data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-12-10T11:00:23.060712Z",
     "start_time": "2019-12-10T11:00:23.038180Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[-1.18321596, -1.18321596],\n",
       "       [-0.50709255, -0.50709255],\n",
       "       [ 0.16903085,  0.16903085],\n",
       "       [ 1.52127766,  1.52127766]])"
      ]
     },
     "execution_count": 73,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 一步到位\n",
    "StandardScaler().fit_transform(data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "对于StandardScaler和MinMaxScaler来说，空值NaN会被当做是缺失值，在fit的时候忽略，在transform的时候\n",
    "保持缺失NaN的状态显示。并且，尽管去量纲化过程不是具体的算法，但在fit接口中，依然只允许导入至少二维数\n",
    "组，一维数组导入会报错。通常来说，我们输入的X会是我们的特征矩阵，现实案例中特征矩阵不太可能是一维所\n",
    "以不会存在这个问题。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## StandardScaler与MinMaxScaler如何选择？"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "看情况。大多数机器学习算法中，会选择StandardScaler来进行特征缩放，因为MinMaxScaler对异常值非常敏\n",
    "感。在PCA，聚类，逻辑回归，支持向量机，神经网络这些算法中，StandardScaler往往是最好的选择。  \n",
    "\n",
    "MinMaxScaler在不涉及距离度量、梯度、协方差计算以及数据需要被压缩到特定区间时使用广泛，比如数字图像\n",
    "处理中量化像素强度时，都会使用MinMaxScaler将数据压缩于[0,1]区间之中。  \n",
    "\n",
    "建议先试试看StandardScaler，效果不好换MinMaxScaler。  \n",
    "\n",
    "除了StandardScaler和MinMaxScaler之外，sklearn中也提供了各种其他缩放处理（中心化只需要一个pandas广\n",
    "播一下减去某个数就好了，因此sklearn不提供任何中心化功能）。比如，在希望压缩数据，却不影响数据的稀疏\n",
    "性时（不影响矩阵中取值为0的个数时），我们会使用MaxAbsScaler；在异常值多，噪声非常大时，我们可能会选\n",
    "用分位数来无量纲化，此时使用RobustScaler。更多详情请参考以下列表："
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![](images/3_2.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 处理缺失值"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 175,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-12-10T12:49:06.256307Z",
     "start_time": "2019-12-10T12:49:06.218552Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Age</th>\n",
       "      <th>Sex</th>\n",
       "      <th>Embarked</th>\n",
       "      <th>Survived</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>22.0</td>\n",
       "      <td>male</td>\n",
       "      <td>S</td>\n",
       "      <td>No</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>38.0</td>\n",
       "      <td>female</td>\n",
       "      <td>C</td>\n",
       "      <td>Yes</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>26.0</td>\n",
       "      <td>female</td>\n",
       "      <td>S</td>\n",
       "      <td>Yes</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>35.0</td>\n",
       "      <td>female</td>\n",
       "      <td>S</td>\n",
       "      <td>Yes</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>35.0</td>\n",
       "      <td>male</td>\n",
       "      <td>S</td>\n",
       "      <td>No</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    Age     Sex Embarked Survived\n",
       "0  22.0    male        S       No\n",
       "1  38.0  female        C      Yes\n",
       "2  26.0  female        S      Yes\n",
       "3  35.0  female        S      Yes\n",
       "4  35.0    male        S       No"
      ]
     },
     "execution_count": 175,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "\n",
    "data = pd.read_csv('data/Narrativedata.csv', index_col=0)\n",
    "data_copy = data.copy()\n",
    "data.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### impute.SimpleImputer\n",
    "![](images/3_3.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 176,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-12-10T12:49:06.794050Z",
     "start_time": "2019-12-10T12:49:06.782429Z"
    }
   },
   "outputs": [],
   "source": [
    "from sklearn.impute import SimpleImputer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 177,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-12-10T12:49:07.317351Z",
     "start_time": "2019-12-10T12:49:07.291115Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "Int64Index: 891 entries, 0 to 890\n",
      "Data columns (total 4 columns):\n",
      "Age         714 non-null float64\n",
      "Sex         891 non-null object\n",
      "Embarked    889 non-null object\n",
      "Survived    891 non-null object\n",
      "dtypes: float64(1), object(3)\n",
      "memory usage: 34.8+ KB\n"
     ]
    }
   ],
   "source": [
    "data.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 178,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-12-10T12:49:07.809307Z",
     "start_time": "2019-12-10T12:49:07.781983Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[29.69911765]\n",
      " [54.        ]\n",
      " [ 2.        ]\n",
      " [27.        ]\n",
      " [14.        ]]\n",
      "[[28.]\n",
      " [54.]\n",
      " [ 2.]\n",
      " [27.]\n",
      " [14.]]\n",
      "[[ 0.]\n",
      " [54.]\n",
      " [ 2.]\n",
      " [27.]\n",
      " [14.]]\n"
     ]
    }
   ],
   "source": [
    "age = data.loc[:, 'Age'].values.reshape(-1, 1)  # 这里将数据升维是为了让sklearn能处理\n",
    "\n",
    "# 实例化模型, 分别填充mean, median, 0\n",
    "impute_mean = SimpleImputer(missing_values=np.nan, strategy='mean')\n",
    "impute_median = SimpleImputer(missing_values=np.nan, strategy='median')\n",
    "impute_0 = SimpleImputer(missing_values=np.nan, strategy='constant', fill_value=0)\n",
    "\n",
    "impute_mean = impute_mean.fit_transform(age)\n",
    "impute_median = impute_median.fit_transform(age)\n",
    "impute_0 = impute_0.fit_transform(age)\n",
    "\n",
    "print(impute_mean[5:10])\n",
    "print(impute_median[5:10])\n",
    "print(impute_0[5:10])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 179,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-12-10T12:49:08.316680Z",
     "start_time": "2019-12-10T12:49:08.305227Z"
    }
   },
   "outputs": [],
   "source": [
    "# 修改原始数据\n",
    "data.loc[:, 'Age'] = impute_median"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 180,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-12-10T12:49:09.054726Z",
     "start_time": "2019-12-10T12:49:09.028430Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "Int64Index: 891 entries, 0 to 890\n",
      "Data columns (total 4 columns):\n",
      "Age         891 non-null float64\n",
      "Sex         891 non-null object\n",
      "Embarked    889 non-null object\n",
      "Survived    891 non-null object\n",
      "dtypes: float64(1), object(3)\n",
      "memory usage: 34.8+ KB\n"
     ]
    }
   ],
   "source": [
    "data.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 181,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-12-10T12:49:09.592981Z",
     "start_time": "2019-12-10T12:49:09.568043Z"
    }
   },
   "outputs": [],
   "source": [
    "# 用总数填补Embarked\n",
    "embarked = data.loc[:, 'Embarked'].values.reshape(-1, 1)\n",
    "impute_most_frequent = SimpleImputer(strategy='most_frequent').fit_transform(embarked)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 182,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-12-10T12:49:10.100403Z",
     "start_time": "2019-12-10T12:49:10.088844Z"
    }
   },
   "outputs": [],
   "source": [
    "data.loc[:, 'Embarked'] = impute_most_frequent"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 183,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-12-10T12:49:10.607707Z",
     "start_time": "2019-12-10T12:49:10.582368Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "Int64Index: 891 entries, 0 to 890\n",
      "Data columns (total 4 columns):\n",
      "Age         891 non-null float64\n",
      "Sex         891 non-null object\n",
      "Embarked    891 non-null object\n",
      "Survived    891 non-null object\n",
      "dtypes: float64(1), object(3)\n",
      "memory usage: 34.8+ KB\n"
     ]
    }
   ],
   "source": [
    "data.info()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 处理分类类型特征：编码与哑变量"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "在机器学习中，大多数算法，譬如逻辑回归，支持向量机SVM，k近邻算法等都只能够处理数值型数据，不能处理\n",
    "文字，在sklearn当中，除了专用来处理文字的算法，其他算法在fit的时候全部要求输入数组或矩阵，也不能够导\n",
    "入文字型数据（其实手写决策树和朴素贝叶斯可以处理文字，但是sklearn中规定必须导入数值型）。  \n",
    "\n",
    "然而在现实中，许多标签和特征在数据收集完毕的时候，都不是以数字来表现的。比如说，学历的取值可以是[\"小\n",
    "学\"，“初中”，“高中”，\"大学\"]，付费方式可能包含[\"支付宝\"，“现金”，“微信”]等等。在这种情况下，为了让数据适\n",
    "应算法和库，我们必须将数据进行编码，即是说，将文字型数据转换为数值型"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### LabelEncoder标签(target)专用，能够将分类转换为分类数值（由于是target专用，所以可以接受一维数据）"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 184,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-12-10T12:49:12.748556Z",
     "start_time": "2019-12-10T12:49:12.728062Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0, 2, 2, 2, 0])"
      ]
     },
     "execution_count": 184,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.preprocessing import LabelEncoder\n",
    "\n",
    "y = data.iloc[:, -1]\n",
    "label = LabelEncoder()\n",
    "# label.fit_transform(y)\n",
    "label = label.fit(y)\n",
    "y_encoded = label.transform(y)\n",
    "y_encoded[:5]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 185,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-12-10T12:49:13.348143Z",
     "start_time": "2019-12-10T12:49:13.336197Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array(['No', 'Unknown', 'Yes'], dtype=object)"
      ]
     },
     "execution_count": 185,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 查看分类\n",
    "label.classes_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 186,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-12-10T12:49:14.317934Z",
     "start_time": "2019-12-10T12:49:14.295465Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array(['No', 'Yes', 'Yes', 'Yes', 'No'], dtype=object)"
      ]
     },
     "execution_count": 186,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "label.inverse_transform(y_encoded)[:5]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 187,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-12-10T12:49:14.947799Z",
     "start_time": "2019-12-10T12:49:14.936293Z"
    }
   },
   "outputs": [],
   "source": [
    "data.iloc[:, -1] = y_encoded"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 188,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-12-10T12:49:15.701229Z",
     "start_time": "2019-12-10T12:49:15.676788Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Age</th>\n",
       "      <th>Sex</th>\n",
       "      <th>Embarked</th>\n",
       "      <th>Survived</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>22.0</td>\n",
       "      <td>male</td>\n",
       "      <td>S</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>38.0</td>\n",
       "      <td>female</td>\n",
       "      <td>C</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>26.0</td>\n",
       "      <td>female</td>\n",
       "      <td>S</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>35.0</td>\n",
       "      <td>female</td>\n",
       "      <td>S</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>35.0</td>\n",
       "      <td>male</td>\n",
       "      <td>S</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    Age     Sex Embarked  Survived\n",
       "0  22.0    male        S         0\n",
       "1  38.0  female        C         2\n",
       "2  26.0  female        S         2\n",
       "3  35.0  female        S         2\n",
       "4  35.0    male        S         0"
      ]
     },
     "execution_count": 188,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "将上面代码简化"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 189,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-12-10T12:49:16.594039Z",
     "start_time": "2019-12-10T12:49:16.572562Z"
    }
   },
   "outputs": [],
   "source": [
    "from sklearn.preprocessing import LabelEncoder\n",
    "# 这里可以接收1维数据\n",
    "data.iloc[:, -1] = LabelEncoder().fit_transform(data.iloc[:, -1])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 190,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-12-10T12:49:17.808113Z",
     "start_time": "2019-12-10T12:49:17.780323Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Age</th>\n",
       "      <th>Sex</th>\n",
       "      <th>Embarked</th>\n",
       "      <th>Survived</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>22.0</td>\n",
       "      <td>male</td>\n",
       "      <td>S</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>38.0</td>\n",
       "      <td>female</td>\n",
       "      <td>C</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>26.0</td>\n",
       "      <td>female</td>\n",
       "      <td>S</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>35.0</td>\n",
       "      <td>female</td>\n",
       "      <td>S</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>35.0</td>\n",
       "      <td>male</td>\n",
       "      <td>S</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    Age     Sex Embarked  Survived\n",
       "0  22.0    male        S         0\n",
       "1  38.0  female        C         2\n",
       "2  26.0  female        S         2\n",
       "3  35.0  female        S         2\n",
       "4  35.0    male        S         0"
      ]
     },
     "execution_count": 190,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### OrdinalEncoder特征(feature)专用，能够将分类特征转换为分类数值(由于是特征专用，所以必须输入2维数组)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 191,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-12-10T12:49:19.177557Z",
     "start_time": "2019-12-10T12:49:19.160195Z"
    }
   },
   "outputs": [],
   "source": [
    "data2 = data.copy()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 192,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-12-10T12:49:19.761353Z",
     "start_time": "2019-12-10T12:49:19.749984Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[array(['female', 'male'], dtype=object), array(['C', 'Q', 'S'], dtype=object)]"
      ]
     },
     "execution_count": 192,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.preprocessing import OrdinalEncoder\n",
    "\n",
    "orc = OrdinalEncoder().fit(data2.iloc[:, 1: -1])\n",
    "orc.categories_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 193,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-12-10T12:49:21.484148Z",
     "start_time": "2019-12-10T12:49:21.462134Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[1., 2.],\n",
       "       [0., 0.],\n",
       "       [0., 2.],\n",
       "       ...,\n",
       "       [0., 2.],\n",
       "       [1., 0.],\n",
       "       [1., 1.]])"
      ]
     },
     "execution_count": 193,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "orc.transform(data2.iloc[:, 1: -1])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 194,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-12-10T12:49:23.869364Z",
     "start_time": "2019-12-10T12:49:23.854840Z"
    }
   },
   "outputs": [],
   "source": [
    "# 把第2列到最后一列(不包括最后一列的取出来)\n",
    "data2.iloc[:, 1: -1] = OrdinalEncoder().fit_transform(data2.iloc[:, 1: -1])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 195,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-12-10T12:49:24.669047Z",
     "start_time": "2019-12-10T12:49:24.629102Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Age</th>\n",
       "      <th>Sex</th>\n",
       "      <th>Embarked</th>\n",
       "      <th>Survived</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>22.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>38.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>26.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>35.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>35.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    Age  Sex  Embarked  Survived\n",
       "0  22.0  1.0       2.0         0\n",
       "1  38.0  0.0       0.0         2\n",
       "2  26.0  0.0       2.0         2\n",
       "3  35.0  0.0       2.0         2\n",
       "4  35.0  1.0       2.0         0"
      ]
     },
     "execution_count": 195,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data2.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## OneHotEncoder独热变量，哑变量"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-12-10T12:51:43.497552Z",
     "start_time": "2019-12-10T12:51:43.484075Z"
    }
   },
   "source": [
    "如上面所示，我们把文字型数据变成(分类)数据值型数据，但这样并不一定正确。因为数值有大有小，大小可以用来表示他们之间的层次关系。但是，有些数据之间根本没有层次关系，思考下面几种不同类型是数据：  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1. 舱门（S, C, Q）  \n",
    "三种取值S，C，Q是相互独立的，也就是它们之间是没有什么特殊的关系。（即，S≠C≠Q）  \n",
    "我们把这种类型的变量称做**名义变量**\n",
    "2. 学历（小学， 初中，高中）  \n",
    "三种取值并非相互独立，可以明显的看出高中>初中>小学，但是学历之间的取值并不能计算的，我们不能说高中=小学+初中。  \n",
    "我们把这种类型的变量称做**有序变量**  \n",
    "3. 体重(>45kg, >90kg, >135kg)  \n",
    "这些类型的数据是可以用数值来衡量的，我们可以说90kg = 30kg + 60kg。  \n",
    "我们把这种类型的变量称做**有距变量**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "然而在对特征进行编码的时候，这三种分类数据都会被我们转换为[0,1,2]，这三个数字在算法看来，是连续且可以\n",
    "计算的，这三个数字相互不等，有大小，并且有着可以相加相乘的联系。所以算法会把舱门，学历这样的分类特\n",
    "征，都误会成是体重这样的分类特征。这是说，我们把分类转换成数字的时候，忽略了数字中自带的数学性质，所\n",
    "以给算法传达了一些不准确的信息，而这会影响我们的建模。\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "类别OrdinalEncoder可以用来处理有序变量，但**对于名义变量，我们只有使用哑变量的方式来处理**，才能够尽量\n",
    "向算法传达最准确的信息"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![](images/3_4.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 处理名义变量"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 202,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-12-10T13:08:08.990227Z",
     "start_time": "2019-12-10T13:08:08.961461Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Age</th>\n",
       "      <th>Sex</th>\n",
       "      <th>Embarked</th>\n",
       "      <th>Survived</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>22.0</td>\n",
       "      <td>male</td>\n",
       "      <td>S</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>38.0</td>\n",
       "      <td>female</td>\n",
       "      <td>C</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>26.0</td>\n",
       "      <td>female</td>\n",
       "      <td>S</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>35.0</td>\n",
       "      <td>female</td>\n",
       "      <td>S</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>35.0</td>\n",
       "      <td>male</td>\n",
       "      <td>S</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    Age     Sex Embarked  Survived\n",
       "0  22.0    male        S         0\n",
       "1  38.0  female        C         2\n",
       "2  26.0  female        S         2\n",
       "3  35.0  female        S         2\n",
       "4  35.0    male        S         0"
      ]
     },
     "execution_count": 202,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 229,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-12-10T13:21:07.888976Z",
     "start_time": "2019-12-10T13:21:07.871476Z"
    }
   },
   "outputs": [],
   "source": [
    "from sklearn.preprocessing import OneHotEncoder\n",
    "temp = data.iloc[:, 1: -1]\n",
    "ohe = OneHotEncoder().fit(temp)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 230,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-12-10T13:21:08.596104Z",
     "start_time": "2019-12-10T13:21:08.570308Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[0., 1., 0., 0., 1.],\n",
       "       [1., 0., 1., 0., 0.],\n",
       "       [1., 0., 0., 0., 1.],\n",
       "       ...,\n",
       "       [1., 0., 0., 0., 1.],\n",
       "       [0., 1., 1., 0., 0.],\n",
       "       [0., 1., 0., 1., 0.]])"
      ]
     },
     "execution_count": 230,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "result = ohe.transform(temp).toarray()\n",
    "result"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 231,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-12-10T13:21:09.876381Z",
     "start_time": "2019-12-10T13:21:09.846022Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>x0_female</th>\n",
       "      <th>x0_male</th>\n",
       "      <th>x1_C</th>\n",
       "      <th>x1_Q</th>\n",
       "      <th>x1_S</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   x0_female  x0_male  x1_C  x1_Q  x1_S\n",
       "0        0.0      1.0   0.0   0.0   1.0\n",
       "1        1.0      0.0   1.0   0.0   0.0\n",
       "2        1.0      0.0   0.0   0.0   1.0\n",
       "3        1.0      0.0   0.0   0.0   1.0\n",
       "4        0.0      1.0   0.0   0.0   1.0"
      ]
     },
     "execution_count": 231,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ohe_data = pd.DataFrame(result ,columns=ohe.get_feature_names())\n",
    "ohe_data.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 232,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-12-10T13:21:11.570754Z",
     "start_time": "2019-12-10T13:21:11.537383Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Age</th>\n",
       "      <th>Sex</th>\n",
       "      <th>Embarked</th>\n",
       "      <th>Survived</th>\n",
       "      <th>x0_female</th>\n",
       "      <th>x0_male</th>\n",
       "      <th>x1_C</th>\n",
       "      <th>x1_Q</th>\n",
       "      <th>x1_S</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>22.0</td>\n",
       "      <td>male</td>\n",
       "      <td>S</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>38.0</td>\n",
       "      <td>female</td>\n",
       "      <td>C</td>\n",
       "      <td>2</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>26.0</td>\n",
       "      <td>female</td>\n",
       "      <td>S</td>\n",
       "      <td>2</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>35.0</td>\n",
       "      <td>female</td>\n",
       "      <td>S</td>\n",
       "      <td>2</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>35.0</td>\n",
       "      <td>male</td>\n",
       "      <td>S</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    Age     Sex Embarked  Survived  x0_female  x0_male  x1_C  x1_Q  x1_S\n",
       "0  22.0    male        S         0        0.0      1.0   0.0   0.0   1.0\n",
       "1  38.0  female        C         2        1.0      0.0   1.0   0.0   0.0\n",
       "2  26.0  female        S         2        1.0      0.0   0.0   0.0   1.0\n",
       "3  35.0  female        S         2        1.0      0.0   0.0   0.0   1.0\n",
       "4  35.0    male        S         0        0.0      1.0   0.0   0.0   1.0"
      ]
     },
     "execution_count": 232,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "new_data = pd.concat([data, ohe_data], axis=1)\n",
    "new_data.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 233,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-12-10T13:21:18.244947Z",
     "start_time": "2019-12-10T13:21:18.214583Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Age</th>\n",
       "      <th>Survived</th>\n",
       "      <th>x0_female</th>\n",
       "      <th>x0_male</th>\n",
       "      <th>x1_C</th>\n",
       "      <th>x1_Q</th>\n",
       "      <th>x1_S</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>22.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>38.0</td>\n",
       "      <td>2</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>26.0</td>\n",
       "      <td>2</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>35.0</td>\n",
       "      <td>2</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>35.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    Age  Survived  x0_female  x0_male  x1_C  x1_Q  x1_S\n",
       "0  22.0         0        0.0      1.0   0.0   0.0   1.0\n",
       "1  38.0         2        1.0      0.0   1.0   0.0   0.0\n",
       "2  26.0         2        1.0      0.0   0.0   0.0   1.0\n",
       "3  35.0         2        1.0      0.0   0.0   0.0   1.0\n",
       "4  35.0         0        0.0      1.0   0.0   0.0   1.0"
      ]
     },
     "execution_count": 233,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "new_data.drop(['Embarked', 'Sex'], axis=1).head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 239,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-12-10T13:24:46.391162Z",
     "start_time": "2019-12-10T13:24:46.361379Z"
    }
   },
   "outputs": [],
   "source": [
    "# 另一种方式实现\n",
    "from sklearn.preprocessing import OneHotEncoder\n",
    "\n",
    "data3 = data.copy()\n",
    "\n",
    "# fit， transform一步到位\n",
    "result = OneHotEncoder().fit_transform(data3.iloc[:, 1: -1]).toarray()\n",
    "new_data2 = pd.concat([data3.drop(['Embarked', 'Sex'], axis=1),\n",
    "                       pd.DataFrame(result, columns=['Female', 'male', 'Embarked_C','Embarked_Q', 'Embarked_S'])],\n",
    "                     axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 241,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-12-10T13:24:56.742747Z",
     "start_time": "2019-12-10T13:24:56.706427Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Age</th>\n",
       "      <th>Survived</th>\n",
       "      <th>Female</th>\n",
       "      <th>male</th>\n",
       "      <th>Embarked_C</th>\n",
       "      <th>Embarked_Q</th>\n",
       "      <th>Embarked_S</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>22.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>38.0</td>\n",
       "      <td>2</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>26.0</td>\n",
       "      <td>2</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>35.0</td>\n",
       "      <td>2</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>35.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    Age  Survived  Female  male  Embarked_C  Embarked_Q  Embarked_S\n",
       "0  22.0         0     0.0   1.0         0.0         0.0         1.0\n",
       "1  38.0         2     1.0   0.0         1.0         0.0         0.0\n",
       "2  26.0         2     1.0   0.0         0.0         0.0         1.0\n",
       "3  35.0         2     1.0   0.0         0.0         0.0         1.0\n",
       "4  35.0         0     0.0   1.0         0.0         0.0         1.0"
      ]
     },
     "execution_count": 241,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "new_data2.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## LabelBinarize标签做哑变量\n",
    "特征可以做哑变量，标签也可以吗？可以，使用类sklearn.preprocessing.LabelBinarizer可以对做哑变量，许多算\n",
    "法都可以处理多标签问题（比如说决策树），但是这样的做法在现实中不常见。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![](images/3_5.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 数据类型以及常用统计量"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![](images/3_5.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Binarizer二值化"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "根据阈值将数据二值化（将特征值设置为0或1），用于处理连续型变量。大于阈值的值映射为1，而小于或等于阈\n",
    "值的值映射为0。默认阈值为0时，特征中所有的正值都映射到1。二值化是对文本计数数据的常见操作，分析人员\n",
    "可以决定仅考虑某种现象的存在与否。它还可以用作考虑布尔随机变量的估计器的预处理步骤（例如，使用贝叶斯\n",
    "设置中的伯努利分布建模）"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 244,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-12-10T13:38:28.214955Z",
     "start_time": "2019-12-10T13:38:28.198383Z"
    }
   },
   "outputs": [],
   "source": [
    "from sklearn.preprocessing import Binarizer\n",
    "data10 = data.copy()\n",
    "\n",
    "# 将年龄二值化\n",
    "age = data10.iloc[:, 0].values.reshape(-1, 1)  # 由于Binarizer是特征专用的类，所以不接受一维数据"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 246,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-12-10T13:39:41.339024Z",
     "start_time": "2019-12-10T13:39:41.301800Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[0.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [1.]])"
      ]
     },
     "execution_count": 246,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "result = Binarizer(threshold=30).fit_transform(age)  # 设置阈值为30，即30岁以上为1，30岁一下为0\n",
    "result"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## KBinsDiscretizer(分段化，离散化)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "这是将连续型变量划分为分类变量的类，能够将连续型变量排序后按顺序分箱后编码。总共包含三个重要参数："
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![](images/3_6.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 252,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-12-10T13:48:49.032391Z",
     "start_time": "2019-12-10T13:48:48.989573Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[1.],\n",
       "       [2.],\n",
       "       [1.],\n",
       "       [2.],\n",
       "       [2.],\n",
       "       [1.],\n",
       "       [3.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [3.],\n",
       "       [1.],\n",
       "       [2.],\n",
       "       [0.],\n",
       "       [3.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [2.],\n",
       "       [2.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [2.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [2.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [4.],\n",
       "       [1.],\n",
       "       [2.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [2.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [3.],\n",
       "       [1.],\n",
       "       [4.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [2.],\n",
       "       [2.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [2.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [2.],\n",
       "       [1.],\n",
       "       [3.],\n",
       "       [1.],\n",
       "       [4.],\n",
       "       [1.],\n",
       "       [2.],\n",
       "       [2.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [2.],\n",
       "       [2.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [2.],\n",
       "       [1.],\n",
       "       [2.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [4.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [2.],\n",
       "       [2.],\n",
       "       [3.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [2.],\n",
       "       [2.],\n",
       "       [1.],\n",
       "       [2.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [2.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [2.],\n",
       "       [2.],\n",
       "       [3.],\n",
       "       [1.],\n",
       "       [3.],\n",
       "       [2.],\n",
       "       [1.],\n",
       "       [3.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [2.],\n",
       "       [2.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [2.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [3.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [3.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [3.],\n",
       "       [1.],\n",
       "       [2.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [2.],\n",
       "       [2.],\n",
       "       [2.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [2.],\n",
       "       [3.],\n",
       "       [1.],\n",
       "       [2.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [2.],\n",
       "       [2.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [2.],\n",
       "       [1.],\n",
       "       [2.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [2.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [3.],\n",
       "       [1.],\n",
       "       [2.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [2.],\n",
       "       [1.],\n",
       "       [3.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [2.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [2.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [2.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [2.],\n",
       "       [3.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [3.],\n",
       "       [1.],\n",
       "       [2.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [2.],\n",
       "       [3.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [3.],\n",
       "       [2.],\n",
       "       [1.],\n",
       "       [2.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [3.],\n",
       "       [2.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [2.],\n",
       "       [2.],\n",
       "       [1.],\n",
       "       [3.],\n",
       "       [2.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [2.],\n",
       "       [4.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [2.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [2.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [2.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [3.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [2.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [3.],\n",
       "       [1.],\n",
       "       [2.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [2.],\n",
       "       [3.],\n",
       "       [2.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [2.],\n",
       "       [2.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [2.],\n",
       "       [2.],\n",
       "       [2.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [2.],\n",
       "       [1.],\n",
       "       [2.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [2.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [2.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [2.],\n",
       "       [1.],\n",
       "       [2.],\n",
       "       [2.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [3.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [2.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [2.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [2.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [2.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [2.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [2.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [2.],\n",
       "       [3.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [2.],\n",
       "       [1.],\n",
       "       [2.],\n",
       "       [1.],\n",
       "       [2.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [2.],\n",
       "       [1.],\n",
       "       [3.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [3.],\n",
       "       [1.],\n",
       "       [2.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [2.],\n",
       "       [0.],\n",
       "       [3.],\n",
       "       [2.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [3.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [4.],\n",
       "       [1.],\n",
       "       [3.],\n",
       "       [1.],\n",
       "       [2.],\n",
       "       [2.],\n",
       "       [2.],\n",
       "       [2.],\n",
       "       [1.],\n",
       "       [2.],\n",
       "       [1.],\n",
       "       [3.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [2.],\n",
       "       [2.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [2.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [3.],\n",
       "       [3.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [2.],\n",
       "       [3.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [3.],\n",
       "       [4.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [3.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [2.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [2.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [2.],\n",
       "       [3.],\n",
       "       [1.],\n",
       "       [2.],\n",
       "       [2.],\n",
       "       [1.],\n",
       "       [2.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [2.],\n",
       "       [1.],\n",
       "       [2.],\n",
       "       [3.],\n",
       "       [1.],\n",
       "       [2.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [2.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [2.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [3.],\n",
       "       [3.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [2.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [3.],\n",
       "       [2.],\n",
       "       [1.],\n",
       "       [2.],\n",
       "       [2.],\n",
       "       [1.],\n",
       "       [2.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [3.],\n",
       "       [3.],\n",
       "       [2.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [2.],\n",
       "       [2.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [2.],\n",
       "       [3.],\n",
       "       [2.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [2.],\n",
       "       [3.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [2.],\n",
       "       [3.],\n",
       "       [2.],\n",
       "       [1.],\n",
       "       [2.],\n",
       "       [2.],\n",
       "       [1.],\n",
       "       [3.],\n",
       "       [1.],\n",
       "       [3.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [2.],\n",
       "       [2.],\n",
       "       [2.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [2.],\n",
       "       [2.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [2.],\n",
       "       [1.],\n",
       "       [2.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [2.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [3.],\n",
       "       [3.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [4.],\n",
       "       [3.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [2.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [2.],\n",
       "       [1.],\n",
       "       [3.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [3.],\n",
       "       [3.],\n",
       "       [2.],\n",
       "       [2.],\n",
       "       [2.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [2.],\n",
       "       [1.],\n",
       "       [2.],\n",
       "       [1.],\n",
       "       [4.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [2.],\n",
       "       [2.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [3.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [3.],\n",
       "       [3.],\n",
       "       [2.],\n",
       "       [1.],\n",
       "       [3.],\n",
       "       [2.],\n",
       "       [1.],\n",
       "       [2.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [2.],\n",
       "       [2.],\n",
       "       [2.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [2.],\n",
       "       [1.],\n",
       "       [3.],\n",
       "       [1.],\n",
       "       [2.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [2.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [2.],\n",
       "       [3.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [2.],\n",
       "       [2.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [2.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [4.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [2.],\n",
       "       [1.],\n",
       "       [2.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [2.],\n",
       "       [2.],\n",
       "       [1.],\n",
       "       [2.],\n",
       "       [1.],\n",
       "       [2.],\n",
       "       [0.],\n",
       "       [3.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [2.],\n",
       "       [3.],\n",
       "       [1.],\n",
       "       [3.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [2.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [2.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [2.],\n",
       "       [3.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [2.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [2.],\n",
       "       [1.],\n",
       "       [2.],\n",
       "       [2.],\n",
       "       [1.],\n",
       "       [2.],\n",
       "       [2.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [2.],\n",
       "       [0.],\n",
       "       [3.],\n",
       "       [1.],\n",
       "       [2.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [3.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [2.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [2.],\n",
       "       [1.],\n",
       "       [2.],\n",
       "       [1.],\n",
       "       [2.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [4.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [2.],\n",
       "       [1.],\n",
       "       [2.],\n",
       "       [3.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [2.],\n",
       "       [1.],\n",
       "       [2.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [2.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [2.],\n",
       "       [2.],\n",
       "       [2.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [3.],\n",
       "       [1.],\n",
       "       [2.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [2.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [1.]])"
      ]
     },
     "execution_count": 252,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.preprocessing import KBinsDiscretizer\n",
    "# 对年龄进行操作\n",
    "X = data.iloc[:, 0].values.reshape(-1, 1)\n",
    "est = KBinsDiscretizer(n_bins=5, encode='ordinal', strategy='uniform')\n",
    "est.fit_transform(X)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 253,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-12-10T13:48:59.745693Z",
     "start_time": "2019-12-10T13:48:59.731251Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0., 1., 2., 3., 4.])"
      ]
     },
     "execution_count": 253,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.unique(est.fit_transform(X))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 257,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-12-10T13:50:02.900246Z",
     "start_time": "2019-12-10T13:50:02.882338Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[0., 1., 0., 0., 0.],\n",
       "       [0., 0., 1., 0., 0.],\n",
       "       [0., 1., 0., 0., 0.],\n",
       "       ...,\n",
       "       [0., 1., 0., 0., 0.],\n",
       "       [0., 1., 0., 0., 0.],\n",
       "       [0., 1., 0., 0., 0.]])"
      ]
     },
     "execution_count": 257,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "est = KBinsDiscretizer(n_bins=5, encode='onehot', strategy='uniform').fit_transform(X)\n",
    "est.toarray()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 特征工程\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "特征工程的基本思想：  \n",
    "$$全部特征\\Rightarrow最佳特征子集\\Rightarrow算法\\Rightarrow模型评估$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "当数据预处理完了，我们可以进行特征工程。特征工程分为下面3个部分：  \n",
    "![](images/3_7.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**NOTE：**在进行特征工程时必须要了解数据的结构，了解每一个特征代表的意义。  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 特征选择（过滤法，嵌入法，包装法，和降维算法）"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 过滤法"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-12-11T23:50:14.986439Z",
     "start_time": "2019-12-11T23:50:01.793737Z"
    }
   },
   "outputs": [],
   "source": [
    "# 先导入数据\n",
    "import pandas as pd\n",
    "data = pd.read_csv('data/digit recognizor.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-12-11T23:51:17.745509Z",
     "start_time": "2019-12-11T23:51:12.137653Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>label</th>\n",
       "      <th>pixel0</th>\n",
       "      <th>pixel1</th>\n",
       "      <th>pixel2</th>\n",
       "      <th>pixel3</th>\n",
       "      <th>pixel4</th>\n",
       "      <th>pixel5</th>\n",
       "      <th>pixel6</th>\n",
       "      <th>pixel7</th>\n",
       "      <th>pixel8</th>\n",
       "      <th>...</th>\n",
       "      <th>pixel774</th>\n",
       "      <th>pixel775</th>\n",
       "      <th>pixel776</th>\n",
       "      <th>pixel777</th>\n",
       "      <th>pixel778</th>\n",
       "      <th>pixel779</th>\n",
       "      <th>pixel780</th>\n",
       "      <th>pixel781</th>\n",
       "      <th>pixel782</th>\n",
       "      <th>pixel783</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>42000.000000</td>\n",
       "      <td>42000.0</td>\n",
       "      <td>42000.0</td>\n",
       "      <td>42000.0</td>\n",
       "      <td>42000.0</td>\n",
       "      <td>42000.0</td>\n",
       "      <td>42000.0</td>\n",
       "      <td>42000.0</td>\n",
       "      <td>42000.0</td>\n",
       "      <td>42000.0</td>\n",
       "      <td>...</td>\n",
       "      <td>42000.000000</td>\n",
       "      <td>42000.000000</td>\n",
       "      <td>42000.000000</td>\n",
       "      <td>42000.00000</td>\n",
       "      <td>42000.000000</td>\n",
       "      <td>42000.000000</td>\n",
       "      <td>42000.0</td>\n",
       "      <td>42000.0</td>\n",
       "      <td>42000.0</td>\n",
       "      <td>42000.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>4.456643</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.219286</td>\n",
       "      <td>0.117095</td>\n",
       "      <td>0.059024</td>\n",
       "      <td>0.02019</td>\n",
       "      <td>0.017238</td>\n",
       "      <td>0.002857</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>2.887730</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>6.312890</td>\n",
       "      <td>4.633819</td>\n",
       "      <td>3.274488</td>\n",
       "      <td>1.75987</td>\n",
       "      <td>1.894498</td>\n",
       "      <td>0.414264</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.00000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25%</th>\n",
       "      <td>2.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.00000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>4.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.00000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75%</th>\n",
       "      <td>7.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.00000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>9.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>254.000000</td>\n",
       "      <td>254.000000</td>\n",
       "      <td>253.000000</td>\n",
       "      <td>253.00000</td>\n",
       "      <td>254.000000</td>\n",
       "      <td>62.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>8 rows × 785 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "              label   pixel0   pixel1   pixel2   pixel3   pixel4   pixel5  \\\n",
       "count  42000.000000  42000.0  42000.0  42000.0  42000.0  42000.0  42000.0   \n",
       "mean       4.456643      0.0      0.0      0.0      0.0      0.0      0.0   \n",
       "std        2.887730      0.0      0.0      0.0      0.0      0.0      0.0   \n",
       "min        0.000000      0.0      0.0      0.0      0.0      0.0      0.0   \n",
       "25%        2.000000      0.0      0.0      0.0      0.0      0.0      0.0   \n",
       "50%        4.000000      0.0      0.0      0.0      0.0      0.0      0.0   \n",
       "75%        7.000000      0.0      0.0      0.0      0.0      0.0      0.0   \n",
       "max        9.000000      0.0      0.0      0.0      0.0      0.0      0.0   \n",
       "\n",
       "        pixel6   pixel7   pixel8  ...      pixel774      pixel775  \\\n",
       "count  42000.0  42000.0  42000.0  ...  42000.000000  42000.000000   \n",
       "mean       0.0      0.0      0.0  ...      0.219286      0.117095   \n",
       "std        0.0      0.0      0.0  ...      6.312890      4.633819   \n",
       "min        0.0      0.0      0.0  ...      0.000000      0.000000   \n",
       "25%        0.0      0.0      0.0  ...      0.000000      0.000000   \n",
       "50%        0.0      0.0      0.0  ...      0.000000      0.000000   \n",
       "75%        0.0      0.0      0.0  ...      0.000000      0.000000   \n",
       "max        0.0      0.0      0.0  ...    254.000000    254.000000   \n",
       "\n",
       "           pixel776     pixel777      pixel778      pixel779  pixel780  \\\n",
       "count  42000.000000  42000.00000  42000.000000  42000.000000   42000.0   \n",
       "mean       0.059024      0.02019      0.017238      0.002857       0.0   \n",
       "std        3.274488      1.75987      1.894498      0.414264       0.0   \n",
       "min        0.000000      0.00000      0.000000      0.000000       0.0   \n",
       "25%        0.000000      0.00000      0.000000      0.000000       0.0   \n",
       "50%        0.000000      0.00000      0.000000      0.000000       0.0   \n",
       "75%        0.000000      0.00000      0.000000      0.000000       0.0   \n",
       "max      253.000000    253.00000    254.000000     62.000000       0.0   \n",
       "\n",
       "       pixel781  pixel782  pixel783  \n",
       "count   42000.0   42000.0   42000.0  \n",
       "mean        0.0       0.0       0.0  \n",
       "std         0.0       0.0       0.0  \n",
       "min         0.0       0.0       0.0  \n",
       "25%         0.0       0.0       0.0  \n",
       "50%         0.0       0.0       0.0  \n",
       "75%         0.0       0.0       0.0  \n",
       "max         0.0       0.0       0.0  \n",
       "\n",
       "[8 rows x 785 columns]"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 数据集非常大，而且包含很多特征（784个），如果用svm，或者神经网络可能还跑不出来，用KNN也要跑很久\n",
    "data.describe()  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-12-11T23:53:29.603315Z",
     "start_time": "2019-12-11T23:53:29.581288Z"
    }
   },
   "outputs": [],
   "source": [
    "# 提取特征\n",
    "X = data.iloc[:, 1:]\n",
    "# 提取标签\n",
    "y = data.iloc[:, 0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "过滤方法通常用作预处理步骤，特征选择完全独立于任何机器学习算法。它是根据各种统计检验中的分数以及相关\n",
    "性的各项指标来选择特征。\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 方差过滤"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### VarianceThreshold(方差阈值法)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "我们可以认为，当数据的方差很小(0)时，我们可以认为这个特征基本没有差异或者说(这个特征没有贡献)，很有可能就是这个特征的数值基本相等。所以，对于机器学习来说，**进行特征工程的第一步就是优先消除方差为0的特征**。  \n",
    "`VarianceThreshold`的一个重要参数就是`threshold`，这个参数控制过滤的方差阈值，默认为0。方差过滤法会舍弃方差小于threshold的特征\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-12-12T00:01:52.175104Z",
     "start_time": "2019-12-12T00:01:51.150207Z"
    }
   },
   "outputs": [],
   "source": [
    "from sklearn.feature_selection import VarianceThreshold\n",
    "\n",
    "X_var0 = VarianceThreshold(threshold=0).fit_transform(X)  # 默认Threshold为0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-12-12T00:03:17.392213Z",
     "start_time": "2019-12-12T00:03:17.375845Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "X_shape: (42000, 784) , X_var0_shape : (42000, 708)\n"
     ]
    }
   ],
   "source": [
    "\"\"\"可以看到，经过方差过滤，特征由原来的784变成708个\"\"\"\n",
    "print('X_shape: {} , X_var0_shape : {}'.format(X.shape, X_var0.shape))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "将特征砍掉一半：通过过滤方差的中位数。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-12-12T00:06:55.273963Z",
     "start_time": "2019-12-12T00:06:54.224426Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(42000, 784) (42000, 392)\n"
     ]
    }
   ],
   "source": [
    "import numpy as np\n",
    "\n",
    "X_fsvar = VarianceThreshold(threshold=X.var().median()).fit_transform(X)\n",
    "\"\"\"可以明显地看到，特征变为原来的一半\"\"\"\n",
    "print(X.shape, X_fsvar.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-12-12T00:07:27.513986Z",
     "start_time": "2019-12-12T00:07:27.493686Z"
    }
   },
   "source": [
    "过滤特定阈值方差的特征。  \n",
    "例如，在二分类，特征的取值服从伯努利分布，它的方差为：  \n",
    "$$DX = p(1- p)$$  \n",
    "其中，X为特征矩阵， p为二分类特征中，某一类特征所占的比例。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-12-12T00:11:16.028004Z",
     "start_time": "2019-12-12T00:11:15.112957Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(42000, 784) (42000, 685)\n"
     ]
    }
   ],
   "source": [
    "# 若特征是伯努利随机变量，假设p=0.8，即二分类特征中某种分类占到80%以上的时候删除特征\n",
    "X_bvar = VarianceThreshold(threshold=0.8*(1 - 0.8)).fit_transform(X)\n",
    "print(X.shape, X_bvar.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 方差过滤对模型的影响"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "我为大家准备了KNN和随机森林分别在方差过滤前和\n",
    "方差过滤后运行的效果和运行时间的对比。KNN是K近邻算法中的分类算法，其原理非常简单，是利用每个样本到\n",
    "其他样本点的距离来判断每个样本点的相似度，然后对样本进行分类。KNN必须遍历每个特征和每个样本，因而特\n",
    "征越多，KNN的计算也就会越缓慢。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-12-12T00:49:27.485501Z",
     "start_time": "2019-12-12T00:49:17.361229Z"
    }
   },
   "outputs": [],
   "source": [
    "from sklearn.feature_selection import VarianceThreshold\n",
    "from sklearn.ensemble import RandomForestClassifier as RFC\n",
    "from sklearn.neighbors import KNeighborsClassifier as KNN\n",
    "from sklearn.model_selection import cross_val_score\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "\n",
    "data = pd.read_csv('data/digit recognizor.csv')\n",
    "X = data.iloc[:, 1:]\n",
    "y = data.iloc[:, 0]\n",
    "\n",
    "# 这里用中位数过滤法\n",
    "X_fsvar = VarianceThreshold(threshold=X.var().median()).fit_transform(X)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### KNN方差过滤前的表现\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "KNN_pre_score = cross_val_score(KNN(), X, y, cv=5).mean()  # 这里暂时不对算法进行修改"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "cross_val_score:0.9658569700264943"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "spend time : 2357.26731539$≈39min$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#####  KNN方差过滤后"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "spend time : 17.24842890min  \n",
    "cross_val_score:0.9659997478150573  \n",
    "可见，经过特征过滤后的运行时间大大下降了，分数也有所提高。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-12-12T00:48:08.917275Z",
     "start_time": "2019-12-12T00:48:08.911871Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.952952798247698\n"
     ]
    }
   ],
   "source": [
    "print(RFC_pre_score)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### 随机森林方差过滤后的表现"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-12-12T00:51:09.128030Z",
     "start_time": "2019-12-12T00:50:16.874410Z"
    }
   },
   "outputs": [],
   "source": [
    "RFC_pro_score = cross_val_score(RFC(n_estimators=20, random_state=1), X_fsvar, y, cv=5).mean()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-12-12T00:51:09.591298Z",
     "start_time": "2019-12-12T00:51:09.585050Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.9521912618525812\n"
     ]
    }
   ],
   "source": [
    "print(RFC_pro_score)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "由上面可以看出，方差(中位数)过滤并没有提高模型的精确度(可能方差过滤掉一些重要的特征)，而且对时间的节省只有一点作用。  \n",
    "为什么随机森林运行如此之快？为什么方差过滤对随机森林没很大的有影响？   \n",
    "这是由于两种算法的原理中涉及到的\n",
    "计算量不同。最近邻算法KNN，单棵决策树，支持向量机SVM，神经网络，回归算法，都需要遍历特征或升维来进\n",
    "行运算，所以他们本身的运算量就很大，需要的时间就很长，因此方差过滤这样的特征选择对他们来说就尤为重\n",
    "要。但对于不需要遍历特征的算法，比如随机森林，它随机选取特征进行分枝，本身运算就非常快速，因此特征选\n",
    "择对它来说效果平平。这其实很容易理解，无论过滤法如何降低特征的数量，随机森林也只会选取固定数量的特征\n",
    "来建模；而最近邻算法就不同了，特征越少，距离计算的维度就越少，模型明显会随着特征的减少变得轻量。因\n",
    "此，**过滤法的主要对象是：需要遍历特征或升维的算法**们，而**过滤法的主要目的**是：在维持算法表现的前提下，帮\n",
    "助算法们降低计算成本"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "|思考：过滤法对随机森林无效，却对树模型有效？|\n",
    "|:-------|\n",
    "|从算法原理上来说，传统决策树需要遍历所有特征，计算不纯度后进行分枝，而随机森林却是随机选择特征进行计算和分枝，因此随机森林的运算更快，过滤法对随机森林无用，对决策树却有用。  \n",
    "在sklearn中，决策树和随机森林都是随机选择特征进行分枝（不记得的小伙伴可以去复习第一章：决策树，参数random_state），但决策树在建模过程中随机抽取的特征数目却远远超过随机森林当中每棵树随机抽取的特征数目（比如说对于这个780维的数据，随机森林每棵树只会抽取10\\~20个特征，而决策树可能会抽取300\\~400个特征），因此，过滤法对随机森林无用，却对决策树有用。  \n",
    "也因此，在sklearn中，随机森林中的每棵树都比单独的一棵决策树简单得多，高维数据下的随机森林的计算比决策树快很多。  |"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "|  | 阈值很小,被过滤掉得特征比较少| 阈值比较大,被过滤掉的特征有很多|\n",
    "|---|----|----|\n",
    "|模型表现| 可能不会有很大影响| 可能变更好，代表被滤掉的特征大部分是噪音<br>也可能变糟糕，代表被滤掉的特征中很多都是有效特征|\n",
    "|运行时间| 可能降低运行时间，取决于过滤的特征数目以及采用的模型|一定可以降低模型的运行时间，算法越复杂，下降的幅度越大|"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 超参数threshold的选择"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "一般先过滤掉方差为0的特征。然后进行其他特征选择操作。  \n",
    "我们怎样知道，方差过滤掉的到底时噪音还是有效特征呢？过滤后模型到底会变好还是会变坏呢？答案是：每个数\n",
    "据集不一样，只能自己去尝试。这里的方差阈值，其实相当于是一个超参数，要选定最优的超参数，我们可以画学\n",
    "习曲线，找模型效果最好的点。但现实中，我们往往不会这样去做，因为这样会耗费大量的时间。我们只会使用阈\n",
    "值为0或者阈值很小的方差过滤，来为我们优先消除一些明显用不到的特征，然后我们会选择更优的特征选择方法\n",
    "继续削减特征数量。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 相关性过滤"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "我们希望选出与标签相关且有意义的特征，因为这样的特征能够为我们提供大量信息。如果特征与标签无关，那只会白白浪费我们的计算内存，可能还会给模型带来噪音。在sklearn当中，我们有三种常用的方法来评判特征与标签之间的相关性：卡方，F检验，互信息。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### 卡方过滤"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "卡方过滤是专门针对离散型标签（即分类问题）的相关性过滤。卡方检验类feature_selection.chi2计算每个非负\n",
    "特征和标签之间的卡方统计量，并依照卡方统计量由高到低为特征排名。再结合feature_selection.SelectKBest\n",
    "这个可以输入”评分标准“来选出前K个分数最高的特征的类，我们可以借此除去最可能独立于标签，与我们分类目\n",
    "的无关的特征。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-12-12T01:55:37.471223Z",
     "start_time": "2019-12-12T01:55:37.361830Z"
    }
   },
   "source": [
    "另外，如果卡方检验检测到某个特征中所有的值都相同，会提示我们使用方差先进行方差过滤。并且，刚才我们已\n",
    "经验证过，当我们使用方差过滤筛选掉一半的特征后，模型的表现时提升的。因此在这里，我们使用threshold=中\n",
    "位数时完成的方差过滤的数据来做卡方检验（如果方差过滤后模型的表现反而降低了，那我们就不会使用方差过滤\n",
    "后的数据，而是使用原数据）："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-12-12T02:00:00.486479Z",
     "start_time": "2019-12-12T01:59:57.225150Z"
    }
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "\n",
    "data = pd.read_csv('data/digit recognizor.csv')\n",
    "X = data.iloc[:, 1:]\n",
    "y = data.iloc[:, 0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-12-12T02:02:48.149621Z",
     "start_time": "2019-12-12T02:02:48.143671Z"
    }
   },
   "outputs": [],
   "source": [
    "from sklearn.feature_selection import VarianceThreshold\n",
    "from sklearn.ensemble import RandomForestClassifier as RFC\n",
    "from sklearn.model_selection import cross_val_score\n",
    "from sklearn.feature_selection import SelectKBest\n",
    "from sklearn.feature_selection import chi2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-12-12T02:24:03.716359Z",
     "start_time": "2019-12-12T02:24:02.027838Z"
    }
   },
   "outputs": [],
   "source": [
    "X_fsvar = VarianceThreshold().fit_transform(X)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-12-12T02:24:17.712340Z",
     "start_time": "2019-12-12T02:24:17.067614Z"
    }
   },
   "outputs": [],
   "source": [
    "X_fschi = SelectKBest(chi2, k=300).fit_transform(X_fsvar, y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-12-12T02:24:27.495000Z",
     "start_time": "2019-12-12T02:24:27.469706Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(42000, 300)"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_fschi.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-12-12T02:25:57.895127Z",
     "start_time": "2019-12-12T02:25:35.278265Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[0.953837   0.95275497 0.95320872 0.9540312  0.95509767]\n"
     ]
    }
   ],
   "source": [
    "# 检验模型\n",
    "pre_score = cross_val_score(RFC(n_estimators=20, random_state=1), X_fsvar, y, cv=5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-12-12T02:26:47.784100Z",
     "start_time": "2019-12-12T02:26:47.779635Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.9537859105492401\n"
     ]
    }
   ],
   "source": [
    "print(pre_score.mean())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-12-12T02:26:47.448801Z",
     "start_time": "2019-12-12T02:26:29.620289Z"
    }
   },
   "outputs": [],
   "source": [
    "pro_score = cross_val_score(RFC(n_estimators=20, random_state=1), X_fschi, y, cv=5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-12-12T02:26:47.623895Z",
     "start_time": "2019-12-12T02:26:47.619922Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.9482860428801871\n"
     ]
    }
   ],
   "source": [
    "print(pro_score.mean())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "模型的分数反而下降了，说明我们过滤掉一些有用的特征"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### 超参数k的选择  \n",
    "那如何设置一个最佳的K值呢？在现实数据中，数据量很大，模型很复杂的时候，我们也许不能先去跑一遍模型看\n",
    "看效果，而是希望最开始就能够选择一个最优的超参数k。那第一个方法，就是我们之前提过的学习曲线："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-12-12T02:35:37.607570Z",
     "start_time": "2019-12-12T02:32:01.534422Z"
    }
   },
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "\n",
    "scores = []\n",
    "for i in range(300, 400+1, 10):\n",
    "    X_fschi = SelectKBest(chi2, k=i).fit_transform(X_fsvar, y)\n",
    "    score = cross_val_score(RFC(n_estimators=20, random_state=1), X_fschi, y, cv=5).mean()\n",
    "    scores.append(score)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-12-12T02:35:38.018059Z",
     "start_time": "2019-12-12T02:35:37.835530Z"
    },
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYAAAAD6CAYAAACoCZCsAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJzt3Xd0VWXa9/HvlQ4BQgudEEoIIp1QBQXpOjjoWFCKymBFRx0V9Xn09RllxhEVR8eKlWZBsSuCIE0JJQm9hFASQiAkQEIISUi73z9yGCOTkJPknLNPuT5rsdbJyZ29r9sV9y9n73tfW4wxKKWU8j1+VheglFLKGhoASinlozQAlFLKR2kAKKWUj9IAUEopH6UBoJRSPkoDQCmlfJQGgFJK+SgNAKWU8lEBVhdwMU2bNjWRkZFWl6GUUh4lPj7+hDEmvKpxbh0AkZGRxMXFWV2GUkp5FBFJsWecngJSSikfpQGglFI+SgNAKaV8lAaAUkr5KA0ApZTyURoASinlozQAlFLKR7k8AEQkVERGiEgbV+9bKaU8wQe/HuKn3cedvh+7AkBE3hORWBF5spLvtxeR70VknYi8ZHsvQEQOi8hq27/uIhII/AAMAr4VkUsdNhOllPICp/OLmP1jIj/tTnf6vqq8E1hErgP8jTGDROR9EYkyxiRdMOx54FljzAYR+VREhgE5wMfGmMfKbetS4AVjzHcikg0MAXY5bDZKKeXhPo8/Qn5RCVMHRTp9X/Z8AhgGLLa9Xk7ZQftCnYEE2+sMIAwYCPxBRDbZPkEEGGN22Q7+vYFrbdv7HRG5U0TiRCQuMzOzmtNRSinPVVpqWBCbTN92jejWOszp+7MnAEKBNNvrU0DzCsZ8DjwtIuOBscBKYDMw0hjTHwgErio3frxt32cu3JAxZq4xJsYYExMeXmUvI6WU8hprkzJJPpnH1EHtXLI/ewIgF6hje12vop8xxswClgLTgXnGmFxguzHmmG1IHBBVbvwzwELgzzUvXSmlvMv82BSa1gtmXLeWLtmfPQEQz2+nfXoCyZWM2wpEAHNsXy8QkZ4i4g9MALaJyE0i8pTt+w2B7BpVrZRSXibl5FlWJWZwy4AIggJcs0DTnnbQXwHrRKQVMA6YKCKzjDEXrgh6FJhjjMmzff0M8BEgwDfGmBUiEgR8LCJrgaPAbY6YhFJKebqFG1LwF2HSgAiX7bPKADDG5NhW9YwCZhtj0oFtFYx7+oKvdwI9LnivEPhTbQpWSilvk19YwqebUxnbrQXNG4S4bL92PRDGGJPFbyuBlFJKOdDXW9PIKSjm1sGRLt2vtoJQSikLGWOYF5vCJS0bENOukUv3rQGglFIWikvJYs+xHG4d1A4Rcem+NQCUUspC89Yn0yAkgD/2au3yfWsAKKWURY7nFPDjznRu6teWOkH+Lt+/BoBSSlnko42HKTGGyQNdc+fvhTQAlFLKAoXFpXy06TDDo5vRrkmoJTVoACillAV+3JVO5plzLuv7UxENAKWUssD89clENqnL5VHWNb3UAFBKKRfbmXaauJQspgyKxM/PtUs/y9MAUEopF1sQm0KdQH+u72vtk3E1AJRSyoWy8wr5amsa1/ZpTVidQEtr0QBQSikXWhyXyrniUksv/p6nAaCUUi5SUmpYsCGFAe0b06VFA6vL0QBQSilXWbU3g9RT+S7v+lkZDQCllHKRebHJtGgQwqiuFT1a3fU0AJRSygUOZOayLukEkwZEEOjvHode96hCKaW83ILYFAL9hYn9XffIx6poACillJPlnitmSfwRru7ekvD6wVaX8x8aAEop5WRfbknjzLliprrJxd/zNACUUsqJjDHMX59M99Zh9G7b0OpyfkcDQCmlnCj24EmSMnKZasEjH6uiAaCUUk40f30KjeoGMr5nK6tL+S8aAEop5SRp2fks353OTf0iCAl0/SMfq6IBoJRyWyWlhv/5cgePfraN0lJjdTnV9tHGFAAmDXCfpZ/lBVhdgFJKVaS01DDz8+0sSTgCQGTTUGYM72RxVfYrKCrh402pjLikOW0b17W6nArpJwCllNsxxvDU1ztZknCEB0dGcU3PVry0PJFf95+wujS7/bDjGKfOFnLroEirS6mUBoBSyq0YY3j2uz0s2niYu6/oyAMjonjuuu50CK/HXz7eQvrpAqtLtMu82BQ6hIdyWacmVpdSKQ0ApZRbeWn5Pt7/9RC3DY7ksbHRiAihwQG8NbkP+UUlzPgogaKSUqvLvKitqdlsS83m1kGRbrf0szwNAKWU23jt5yReW7Wfif3a8v/+0PV3B89Ozerz/J96EJ+SxXM/7LWwyqrNj00mNMif6/q0trqUi9IAUEq5hXfXHeTF5fu4tndr/n5t9woflj6+ZytuGxzJ+78e4vvtxyyosmonc8/x3bZj/KlvG+qHWPvIx6poACilLLdgQwqzvt/DuG4teOH6HvhXcPA/73+uuoTeEQ2Z+fk2DmTmurBK+3yyOZXCEvd45GNVNACUUpb6PP4IT321kxFdmvHKxN4EVNErPyjAjzcm9SE40J97FsaTV1jsokqrVlxSyqINKVzWqQmdmtW3upwq2RUAIvKeiMSKyJOVfL+9iHwvIutE5CXbewEiclhEVtv+dReRQBH5VESWi8jPItLIkZNRSnmWb7cdZebn2xjSqSmvT+pDUIB9f5O2DKvDKxN7kZSRy/9+uRNj3OMmsRV7Mjh6uoCpbrz0s7wq/2uLyHWAvzFmENBBRKIqGPY88KwxZijQRkSGAT2Aj40xw2z/dgDjgB+NMaOBZcAUR01EKeVZlu9K58FPtxLTrjFzp/atdquEoVHhPDSyM19uSWPRxsNOqrJ65scm07phHUZ0aWZ1KXaxJ26HAYttr5cDQyoY0xlIsL3OAMKAgcAfRGST7RNEgDHmG2PMB7Zx4baxSikfszoxg/s+2kK31mG8d1sMdYNq1pTgvuGdGBYdzjPf7mZbaraDq6yepONnWH/gJJMGRlR5Gstd2FNlKJBme30KqOhpxp8DT4vIeGAssBLYDIw0xvQHAoGrzg8WkQ7AlcCSCzckIneKSJyIxGVmZlZnLkopDxB74CR3LYinU7N6zL+9f61Wyvj5CS/f2Ivw+sHcuyiBrLOFDqy0eubHphAU4MdNMW0tq6G67AmAXKCO7XW9in7GGDMLWApMB+YZY3KB7caY8+u04oAoABEJBj4E7jTGFFWwrbnGmBhjTEx4eHg1p6OUcmfxKVn8ed5mIhrXZcGf+xNWt/bLJBuFBvHGpD5knjnHQ4u3WtI0LqegiCUJRxjfoxVN6rnPIx+rYk8AxPPbaZ+eQHIl47YCEcAc29cLRKSniPgDE4Bttvc/AD40xsTVqGKllEfaceQ0t72/iWb1g1k0fYBDD5Q92zbkqfFdWZ2YyWur9jtsu/b6Iv4IeYUl3DrY/Zd+lmdPAHwFTBGROcCNwC4RmVXBuEeBOcaYPNvXzwALKAuGWGPMChEZB1wLTLWtDHqg9lNQSrm7vek5THl/Iw3qBLLojoE0axDi8H1MHhDBtb1b8/KKfaxLct3p49JSw/zYFHq1bUiPNu71yMeqVHnlxRiTY1vVMwqYbYxJ57e/5suPe/qCr3dSthKo/HtL+e10klLKBxzIzGXyuxsJDvDj4zsG0rqhcw4BIsLfr+3GrqOneeCTrXx3/xBaOWlf5f164AQHT5zl5Zt6On1fjmbXpWpjTJYxZrHt4K+UUnY5fDKPSe9sBGDR9IFENHFuX/y6QQG8ObkvhcWlzPgogcJi5zeNm7c+hSahQVzVvaXT9+VonrFWSSnlcY5m53PLuxsoKC5h4fQBdGpWzyX77Rhej9nX92DL4Wz+8cMep+4r9VQeK/ce5+b+EQQHuN8jH6uiAaCUcriMnAJueWcDp/OKWDBtAF1aNHDp/q/q3pJpl7Xnw/XJfLPtqNP2s3BjCn4i3OKmj3ysigaAUsqhTuaeY9K7G8k4c44Pp/Wje5swS+p44qou9G3XiMeXbGd/xhmHb7+gqIRPN6cyumtzl1xrcAYNAKWUw5zOK2LKe5s4fCqPd2+NoW+7xpbVEujvx+u39KFOoD93L0zg7DnHNo37ZttRsvOKuHVwpEO360oaAEoph8g9V8ytH2xif0Yub0/py+COTa0uiRZhIbx6c28OZubyxBc7HNY0zhjDvPXJRDevz4D21oVcbWkAKKVqLb+whGkfbGZH2mleu6U3w6LdpxnaZZ2a8vDoaL7ZdpQFG1Icss2Ew9nsOprD1MHt3PqRj1XRAFBK1UpBUQl3LogjLuUU/7qpF6MvbWF1Sf/lnis6MqJLM579bjcJh7Nqvb35scnUDwlgQi/3fuRjVTQAlFI1VlhcyoxFCaxLOsHs63syvmcrq0uqkJ+fMOfGXjRvEMJ9ixI4VYumcRlnCvhhxzFu6NuW0OCadTF1FxoASqkaKS4p5cFPt7BybwazJnTj+r5trC7posLqBvLmpL6cyC3kgU+2UFLDpnGfbEqlqMQwxQMe+VgVDQClVLWVlhoe/Xw7P+xI58mrL2HyQM84GHZvE8b/XXMp65JO8OrKpGr/fFFJKYs2pnBF53DaNw11QoWupQGglKoWYwz/+9UOvtySxiOjOzN9aAerS6qWm/u35bo+rXn15yRWJ1bvmVTLdx3neM45j+v6WRkNAKWU3Ywx/O3b3Xy8KZUZwzty35UVPSHWvYkIf5/Qnejm9Xnw062kZefb/bPzYpOJaFyXKzq7zyqn2tAAUErZxRjD7GWJfLg+mT8Pac8jo6OtLqnG6gT58+bkvpSUGO5dlMC54pIqf2bPsRw2HTrFlIHt8Pfz3KWf5WkAKKXs8u+f9/Pm6gNMGhDBk1df4tHr3wHaNw3lhRt6sC01m1nfVd00bn5sCiGBftwQ494Xu6tDA0ApdVE5BUU88MkW5vy0jz/1acOzf+zm8Qf/88Z2a8kdQ9uzYEMKX29Nq3Tc6bwivtqSxoRerWlYN8iFFTqXZy9iVUo5VVzyKR74ZCvpOQX8dVRnZgzvhJ+XnP44b+bYLmxNzebxJTu4pGUDOjev/19jPotPJb+oxCuWfpannwCUUv+luKSUOT/t48a3Y/H3Ez67exB/GRHlNee+ywv09+O1W/oQGhzA3Qvjyb2gaVxpqWHBhhT6RTbi0lbWdDZ1Fg0ApdTvHD6Zx41vx/LqyiQm9G7N938ZQp+IRlaX5VTNG4Tw75t7k3ziLI8t2f67pnFrkjJJOZnH1EGR1hXoJHoKSCn1H19uOcJTX+1CBF69uTfXuGlrB2cY1LEJj4yJZvaPicS0a8Ttl7UHYP76ZJrVD2aMG/Y4qi0NAKUUOQVFPPXVTr7eepR+kY14+aZetGnk3Of3uqO7L+9IQkoWf/9+Dz3aNKRJaBCr92XywIgoggK874SJBoBSPq78hd6HR3Xm3uGdvPJcvz38/ISXbujF+Nd+YcaiBAZ3aoK/CLf098xHPlbF+yJNKWWXii703u+lF3qrI6xuIG9M6sOpvEK+SEhjXPeWNGsQYnVZTqEBoJQP8sULvdXRrXUYsyZ0o06gP9Mui7S6HKfRU0BK+RhfvtBbHTfGtOWanq0ICfS3uhSn0QBQykfohd7q8+aDP2gAKOUT9EKvqogGgFJerLiklFd/3s9rPyfRplFdPrt7kJ7rV/+hAaCUlzp8Mo8HP91CwuFsruvTmr9dcyn1QwKtLku5EQ0ApbyQXuhV9tAAUMqL6IVeVR0aAEp5Cb3Qq6pLA0ApD6cXelVNaQAo5cH0Qq+qDbsCQETeA7oC3xtjZlXw/fbAa0ADYJMx5mERCQAO2v4B3G+M2SEidYH1xpheDpmBUj5KL/Sq2qoyAETkOsDfGDNIRN4XkShjTNIFw54HnjXGbBCRT0VkGJADfGyMeazctvyBxUBDx01BKd+iF3qVo9jzCWAYZQdtgOXAEODCAOgMJNheZwBhlH1i+IOIDAd2AHcBBrgT+KiynYnInbYxRER4ZwtWpWpqa2o2932UwLHTeqFX1Z493UBDgTTb61NA8wrGfA48LSLjgbHASmAzMNIY0x8IBK4yxpQYY45ebGfGmLnGmBhjTEx4eLi981DK6/289zgT58YCaOtm5RD2fALIBerYXtejgtAwxswSkSHAo8A8Y0yuiGw3xpyzDYkDohxRsFK+aEn8EWYu2U7Xlg344PZ+NK0XbHVJygvY8wkgnrLTPgA9geRKxm0FIoA5tq8XiEhP23n/CcC2WtSplM96e80BHv5sGwM7NObjOwfqwV85jD2fAL4C1olIK2AcMFFEZhljnrxg3KPAHGNMnu3rZyg71y/AN8aYFY4qWilfUFpqeG7pHt5Zd4ire7Rkzo09CQ7w7vbEyrXEGFP1IJFGwChgrTEm3elV2cTExJi4uDhX7U4pt1FUUspjn2/niy1pTB3UjqfHX6rn+5XdRCTeGBNT1Ti77gMwxmTx20ogpZQT5RUWM2NRAqsSM/nrqM7cf2UnRPTgrxxP7wRWyo1knS1k2rzNbEvN5h/XdueWAboUWjmPBoBSbuJodj5T39/E4VN5vDGpD2O7tbS6JOXlNACUcgNJx88w9f1N5BYUM+/2/gzq2MTqkpQP0ABQymLxKVn8ed5mAvz8+OSugVzaKszqkpSP0ABQykKr9mZwz6J4mjcIYcG0AUQ00Z4+ynU0AJSyyPm7e7u0qM+Ht/cnvL7e4KVcSwNAKQvMXXuAf/ywl8Edm/D2lL7aw19ZQgNAKRcqLTX888e9zF17kKu7t2TOTXp3r7KOBoBSLlJUUspjS7bzRUIaUwa24/+u0bt7lbU0AJRygfJ39z40sjN/GaF39yrraQAo5WTZeYXc/mHZ3b1/v7Ybkwa0s7okpQANAKWc6j93957Uu3uV+9EAUMpJ9mecYcp7mzhTUMyH0/oxuGNTq0tS6nc0AJRygoTDWUz70HZ3750D6dZa7+5V7kcDQCkHW5WYwT0Ly+7unT+tP+2ahFpdklIV0gBQyoG+SDjCzM+3E6139yoPoAGglIO8s/Ygf/9hD4M6NGHuVL27V7k/DQClaskYwz+X7uXttQe5qnsL5tzYi5BAvbtXuT8NAKVqoaiklMeX7GBJwhEmD4zgb9d007t7lcfQAFCqhvILS5jxUQI/783gwZFRPDAiSu/uVR5FA0CpGsjOK2Tah5vZkprNrAndmDxQ7+5VnkcDQKlqKigqYeLcDRzMPMvrt/Thqu56d6/yTBoASlXTwg0p7E0/wztTYxjVtbnV5ShVY35WF6CUJzmdX8Rrq/YzNKqpHvyVx9MAUKoa3lpzgOy8Ih4f18XqUpSqNQ0ApeyUfrqA9385xIRerbi0lfb2UZ5PA0ApO/1rxT6MgYdHR1tdilIOoQGglB2Sjp9hcVwqkwe2o23julaXo5RDaAAoZYfZyxKpGxTAfVd2sroUpRxGA0CpKsQln+Kn3ce5+4oONA4NsrocpRxGA0CpizDG8NzSvTSrH8y0Ie2tLkcph9IAUOoiftp9nPiULB4c2Zm6QXrfpPIudgWAiLwnIrEi8mQl328vIt+LyDoRecn2XoCIHBaR1bZ/3W3v/01ENovI646bhlKOV1xSyuxliXQID+XGmDZWl6OUw1UZACJyHeBvjBkEdBCRqAqGPQ88a4wZCrQRkWFAD+BjY8ww278dItIXGAL0BzJEZKTDZqKUg30ef4T9GbnMHNOFAH/9sKy8jz2/1cOAxbbXyyk7gF+oM5Bge50BhAEDgT+IyCbbJ4gA4ApgiTHGAMuAobWoXSmnyS8s4eUV++gd0ZAxl2rLB+Wd7AmAUCDN9voUUNH/DZ8DT4vIeGAssBLYDIw0xvQHAoGr7NmWiNwpInEiEpeZmVmduSjlMB+sP8TxnHM8Me4S7fGvvJY9AZAL1LG9rlfRzxhjZgFLgenAPGNMLrDdGHPMNiQOiLJzW3ONMTHGmJjw8PDqzEUph8g6W8ibqw8w8pJm9G/f2OpylHIaewIgnt9O+/QEkisZtxWIAObYvl4gIj1FxB+YAGyrxraUsszrq/Zz9lwxM8dqwzfl3exZ1/YVsE5EWgHjgIkiMssYc+GKoEeBOcaYPNvXzwAfAQJ8Y4xZISJ+wHMi8gplp4rGOmQWSjnIkaw85semcH3fNnRuXt/qcpRyqioDwBiTY1vVMwqYbYxJp+yv+QvHPX3B1zspWwlU/r1S28qfq4FXjDGHalG7chMlpcZrHoQ+Z/k+RODBkZ2tLkUpp7NrbZsxJssYs9h28K8VY0y+MeZzY8zB2m5LWa+opJSrX13HQ59upWxxl+fafTSHL7emcdtlkbRqWKfqH1DKw+mtjapWvt12lL3pZ9ibfoaO4aHcd2VFt4l4htnL9tIgJJB7r9CGb8o36N0tqsZKSw1vrTlAdPP6TOjVipd+2seK3cetLqtG1h84werETGYM70hY3UCry1HKJTQAVI2tSsxg3/Fc7h7WgX/+qQfdWoXx4Kdb2Z9xxurSqsUYwz+X7qVVWAhTB0VaXY5SLqMBoGrsrTUHaN2wDn/o0YqQQH/entKXkEA/7pgfz+n8IqvLs9v3O46x/chp/jo6mpBAf6vLUcplNABUjcQln2JzchbTh7Yn0NYnp1XDOrw5uS9HsvL4y8dbKCl1/4vCRSWlvLAskejm9bm2d2ury1HKpTQAVI28teYAjeoGclO/tr97v19kY/52TTfW7MvkhWWJFlVnv082HSblZB6PjYv2mqWsStlLA8DB8gqLyS8ssboMp0pMP8OKPRncOjiywh75twyIYNKACN5ac4Cvt6ZVsAX3kHuumFdWJjGgfWOGRzezuhylXE4DwIFKSg03vBXLlPc2evya+It5e+0B6gT6c+tFLpg+Pf5S+kc25rEl29mZdtp1xVXDu+sOciK3kMfHddGGb8onaQA40Dfb0th1NIe4lCyWe+hyyKqkZefzzdajTOzflkYXeT5uUIAfb0zuQ+O6Qdw5P44TuedcWGXVMs+c4521BxnXrQW9IxpZXY5SltAAcJDC4lJeWr6Pri0b0DE8lBeWJXrERdDqendd2Q3c04d2qHJs03rBzJ0aw8mzhdy7MIHC4lJnl2e3f/+cREFxKY+Oiba6FKUsowHgIB9vOsyRrHxmjo3mkdHR7M/I5YuEI1aX5VBZZwv5ZFMq1/RqRWs7WyV0ax3G7Ot7sCn5FM98t8vJFdon+cRZPtp4mIn92tIhvJ7V5ShlGW0F4QBnzxXz75/LLiZe0bnsGQY92oTxrxVJjO/ZymvWls+LTSa/qIS7r+hYrZ/7Y6/W7D6Ww9trDtK1ZRi3DIhwToF2enF5IoH+fjwwwnPbVijlCPoJwAHe/+UQJ3ILmTm27GKiiPDY2C6kZeezaONhq8tziLzCYuatT2bkJc1q1CZ55pguXNE5nKe/2Ulc8iknVGifbanZfLf9GHcMbU+zBiGW1aGUO9AAqKVTZwuZu/Ygo7o2p2+73y4mXtapKUM6NeX1VfvJPVdsYYWO8enmVLLyiqr91/95/n7CqxN707phHe5emMDR7HwHV1i18y0fGocGccflVV/DUMrbaQDU0pur95NbWFzhxcRHx0Rz6mzhfy6ceqqiklLeXXeIfpGNiIms+SMSw+oG8s7UGAqKSrhrQTwFRa69X2Jt0gliD57k/is7UT9EG74ppQFQC0ez85kXm8J1vSt+elTPtg25qnsL3ll7kJNutgyyOr7ddpS07Pwa//VfXlTz+rx8Uy92pJ3miS92uOx+idLSsr/+2zauw6QB7VyyT6XcnQZALbyyIgkMPDiy8ouJD4+OpqC4lNdXHXBhZY5jjOHtNQeJbl7fYXfLjuranIdHdebLLWm8u841D4X7elsae47l8MjoaIIC9NdeKdAAqLH9Gbl8Fp/KpIERtG1ct9JxHcPrcUPfNizckMKRrLxKx7mrVYkZJB4/w11XdMDPgb1y7ruyE+O6teC5pXtYuy/TYdutyLniEl5cto9urRswvkcrp+5LKU+iAVBDLy1PpE6gPzOGV/30qAdGRoHAv1YkuaAyx3pzdVnL5/E9HXvgFBFevKEnnZvX576PEkg+cdah2y9vQWwKadn5PD72EoeGmFKeTgOgBralZrN0ZzrTh3agab3gKse3DKvDrYPa8UXCEfYd95yHpVTU8tmRQoMDeGdqDH5+wh3z45yyWiqnoIjXVu1naFRThkQ1dfj2lfJkGgA18MKyRBqHBjF9aHu7f+beYZ0IDQrgRQ9okXxeZS2fHalt47q8fksfDp44y0OfbqXUwe0z3l5zgOy8Ih4b28Wh21XKG2gAVNMvSSf4Zf8JZgyv3lLCRqFB3Hl5B5bvPk7C4SwnVugY+45fvOWzI13WqSn/e9Ul/LT7OK+sdNxpsvTTBbz3yyH+2KsV3VqHOWy7SnkLDYBqMMYwe9leWjesw6QatDOYNqQ9TesF8fzSvW7fLvqtNVW3fHak2y+L5Pq+bXhlZRI/7jzmkG2+snIfJaWGR0ZrwzelKqIBUA0/7kxn+5HTPDgyqkb9fUKDA7j/yig2HjrF2qQTTqjQMext+exIIsKsCd3o2bYhf128jcT02l0r2Z+Ry6ebU5k8sN1FV2kp5cs0AOxUXFLKC8sTiWpWj+v6tKnxdm7uH0GbRnWY/eNeh5/vdpT3bGvz7Wn57Eghgf7MndKXesEB3DE/juy8whpva/aPe6kbFMB9dqzSUspXaQDYaUnCEQ5mnuWRMbV7dmxQgB8Pj+7MrqM5/OCgUx2OlHW2kI83Ha5Wy2dHat4ghLem9CX9dAH3fbSF4pLqP0MgPuUUy3cf567LO9DEjlVaSvkqDQA7FBSV8K8VSfRq25DRXZvXenvX9GxNlxb1eWn5PopqcIBzpvmxKTVq+exIfSIaMWtCN37Zf4J/Lt1brZ81xvDcD3sJrx/Mn6uxSkspX6QBYIcFsSkcO13AzLHRDnl2rL+f8OiYaA6dOMtnce7z0Ji8wmI+XH+IEV1q1vLZkW7s15bbBkfy7i+HWBJv/3+jFXsyiEvJ4sGRUU5fvaSUp9MAqEJOQRGvry67kWhwR8fdSHRll2bEtGvEKyv3kV/o2q6YlVlsa/l8zzDr/vov73+vvoRBHZrwxJc72JaaXeX44pJSZv+4lw5NQ7kxxnkRCjOXAAAMfElEQVT3LijlLTQAqvDO2oNk5xUxc4xjbyQSEWaO7cLxnHPMi0126LZroqiklHfWHSKmXe1aPjtSoL8fr0/qQ3i9YO5aEE/GmYKLjl+ScISkjFxmjo12yp3LSnkb/b/kIjLPnOPddYe4ukdLurdx/I1E/ds3Znh0OG+s2s/pvCKHb786vtte1vLZXf76P69xaBDvTI3hdH4R9yxM4FxxxZ+W8gtLePmnsus0Yy5t4eIqlfJMGgAX8drPSRSWlPLwqM5O28ejY7qQU1DM22utaxdtjOGt1Qfp3Lyew1o+O1LXVg148YaexKdk8fTXuyq8ie7D9cmk5xTwxLguDrlOo5Qv0ACoxOGTeXy06TA3xrSlQ3g9p+2na6sG/LFXK97/9RAZORc/xeEs51s+331FR7ftlnl1j5bMGN6RTzansnBDyu++l3W2kDdW72dEl2YM6NDEogqV8jx2BYCIvCcisSLyZCXfby8i34vIOhF56YLvNReRLVWNczcvr9iHnwgPjKj8YS+O8tdRnSkuMbz6szXtot9afdApLZ8d7eFR0Yzo0oy/fbubDQdP/uf9N1aXPXd5pjZ8U6paqgwAEbkO8DfGDAI6iEhFR8TngWeNMUOBNiIyrNz3XgTq2DHObew5lsNXW9O47bJIWoSFOH1/7ZqEcnP/CD7ZlErKSef1xa9IfMopNiWf4s9DnNPy2ZH8/ISXJ/Yiokld7l2UwJGsPI5k5TFvfQp/6tOG6BbWLl1VytPY83/8MGCx7fVyYEgFYzoDCbbXGUAYgIhcCZwF0i82rjwRuVNE4kQkLjPTuU+KqsyLyxKpFxzAPS68Ger+KzsR6O/HnJ/2uWyfAG+uPkjDuoFM7O8ZyyYbhJQ9WL6opJQ758fz3NK9IGWfopRS1WNPAIQCabbXp4CKboX9HHhaRMYDY4GVIhIEPAU8frFxF27IGDPXGBNjjIkJDw+3fyYOEpd8ipV7M7j7io40rOuaRmgAzRqEMG1IJF9vPcquo6ddss+yls/HuXWQ81s+O1LH8Hq8OrE3e9Jz+H77MW4fHEkrC9pWKOXp7AmAXH47hVOvop8xxswClgLTgXnGmFzKDvxvGGOyqxjnNowxPP9jWRuB2y+LdPn+77y8I2F1Al320Ji31xwsa/k8ONIl+3Ok4V2a8dTVXenSor7bLV1VylPYEwDx/HbapyeQXMm4rUAEMMf29UhghoisBnqJyLuVjHMbqxMz2ZycxV9GWNNGIKxOIPcO68iqxEw2lrvI6Qxp2fl8vTWNm/q1pbGLWj472rQh7fnxwctd+klNKW9iTwB8BUwRkTnAjcAuEZlVwbhHgTnGmDwAY8zlxphhxphhwFZjzPSKxrmL0tKyv/7bNanLRCc+ArEqtw6OpHmDYGYvS3TqQ2N+a/msDdOU8lVVBoAxJoeyC8EbgOHGmG3GmP9aDmqMedoYs6CSbQyzZ5yVvt1+lL3pZ/jrqM6WroYJCfTngRGdiU/JYuWeDKfsI+tsIZ9sPsw1PVvRppE+LEUpX2XXkc4Yk2WMWWyMSa96tOcpLC7lpeX7uKRlA8b3sH4t/A0xbWjfNJQXliVS4oSHxsyPTSGvsIS7LGz5rJSynnsv/HaRTzcf5vCpPGaOjXaLO2ED/cseGpN4/Axfb02r+geqoXzLZ103r5Rv8/kAyCss5pWV++kf2ZhhnV2/7LQyV3VrSbfWDZjz075KG6DVxPmWz3fryhmlfJ7PB8AHvyZzIvecwx724ih+fsLMMV04kpXPxxsPO2Sb5Vs+93OTls9KKev4dABknS3krdUHGHlJM7fpgV/e0KimDOrQhNdW7efsueJab+98y2crH/eolHIfPh0Ab605QG5hMY+Miba6lAqVPTQmmhO5hbz/y6Fabat8y+cru7hfy2ellOv5bAAcO53Ph+uTubZXa7q0aGB1OZXqHdGIMZc2Z+7ag5w6W1jj7Zxv+XzX5e7b8lkp5Vo+GwCvrkyi1Bge8oAmYo+MjuZsYTFvrt5f4228tfogrcJCuKaX9ctclVLuwScD4EBmLovjjjBpQDvaNnb/G6Gimtfnuj5tmBebwtHs/Gr//PmWz9OHdnD7ls9KKdfxyaPBnOX7CA7wY8bwTlaXYrcHR0aBgVdWVP+hMZ7W8lkp5Ro+FwDbj2Tz/Y5jTB/SnvD6wVaXY7c2jeoyeWA7PotPZX+G/U1Ukzy05bNSyvl8LgBeWJZIo7qBTL+8g9WlVNuM4R2pE+jPS8vtbxf91pqDhAT6eWTLZ6WUc/lUAKzff4J1SSeYMbwTDUICrS6n2prUC+aOyzuwdGc621Kzqxx/1NbyeWK/CI9t+ayUch6fCQBjDM8vS6RlWAiTB7azupwamz60A41Dg5i9bG+VY99ddwiDtnxWSlXMZwJg2a7jbEvN5qGRnQkJ9Le6nBqrFxzAfcM78ev+k/ySdKLScdl5ZS2f/6gtn5VSlfCJACguKeXF5Yl0DA/luj6trS6n1iYNjKB1wzrMXra30ofGaMtnpVRVfCIAvtiSxv6MXB4dE02AF6yDDw7w56FRndl+5DQ/7vzvRzTkF5bw4fpkrtSWz0qpi/D8o2EVCopK+NdP++jZJowxl7awuhyHubZ3a6Ka1eOF5YkUl5T+7nuL41I5dbZQH5aulLoorw+AhRtSOHq6gMfGdnGrds+15e8nPDImmoOZZ1mScOQ/7xeVlDJ37UH6astnpVQVvDoAzhQU8fqq/Qzp1JTBnZpaXY7Dje7anF5tG/KvFUkUFJU9NOb77cdIy87nHj33r5SqglcHwDvrDpGVV8SjbtruubZEhMfGduHY6QIWxKaUtXxec4CoZtryWSlVNa/tDXAi9xzvrjvIVd1b0LNtQ6vLcZpBHZtweedwXl+9n2YNgtmbfoaXbuipLZ+VUlXy2k8Ar/28n3PFpTw82jv/+i9v5phosvOKeOSzbdryWSllN68MgNRTeSzamMINfdvQMbye1eU4XbfWYfyhR0uKSoy2fFZK2c0rTwEVlZQyNCqcB0ZGWV2Ky/zPVZcQXj+Ym/tHWF2KUspDSGV3krqDmJgYExcXZ3UZSinlUUQk3hgTU9U4PVeglFI+SgNAKaV8lAaAUkr5KA0ApZTyURoASinlozQAlFLKR2kAKKWUj9IAUEopH+XWN4KJSCaQUotNNAUqf3Cu9/G1+YLO2VfonKunnTEmvKpBbh0AtSUicfbcDectfG2+oHP2FTpn59BTQEop5aM0AJRSykd5ewDMtboAF/O1+YLO2VfonJ3Aq68BKKWUqpy3fwJQSilVCY8PABFpLCKjRKSp1bW4is7ZN+iclbN5dACISCPgO6A/sEpEwkXkPRGJFZEny437r/c8VQVzbiciS0VkuYh8KSJBtnHePOdw2/vNRWRLuXG+MOc3RGR8uXFePWcR+UFE4kTk7XLjvGbO55X/XXblMcyjAwDoAfzVGPN3YBlwJeBvjBkEdBCRKBG57sL3LKzXES6c8wRgjjFmNJAOjPWBOfexvf8iUAfAF+YsIkOBFsaYb8En5nwLsMi2Fr6+iMR44ZzPexGoU9H8nDlnjw4AY8waY8wGEbmcsr8axgCLbd9eDgwBhlXwnseqYM4fGGN+sn07HMjA++ccKyJXAmcpCz3w/jlvAN4BkkXkj7Zhw/DuOWcD3USkIdAWSMXL5gxwwe/yMFx4DPPoAAAQEQFuArIAA6TZvnUKaA6EVvCeR7tgzkW29wYBjYwxG/D+OQvwFPB4uSHePufJwG5gNtBfRO7H++e8GmgH/AXYQ9kcvWrOtlO25X+XK5qf0+bs8QFgyswAtgODsZ0SAOpRNr/cCt7zaBfM+RoRaQz8G5hmG+Ltc34QeMMYk11uiLfP+T5grjEmHVgIDMf757wauNsY8wywF7gd75vz4/z+d7mi+Tltzh79H09EHhORqbYvGwL/5LePRz2BZCC+gvc8VgVzzgY+A54wxpxvnOftcx4LzBCR1UAvEXkX75/zh0AH29cxlDVJ9PY5NwS6i4g/MICyT/heNWdgJOV+l4HxuPAY5tE3gtlWDSwGgoGdwBPAWmAlMA4YSNkvzbry7xljTltSsANUMOcdwD+AbbYhbwJL8e45zzC2X1wRWW2MGSYiDfDuOT8GvE/Zx/9A4HrgDN4953mUzbkdEAtcS9kfrV4z5/JsIXANF8wPJx7DPDoAKmL7JRoFrLV9XK7wPW+nc9Y5eytvn7Mrj2FeFwBKKaXs49HXAJRSStWcBoBSSvkoDQCllPJRGgBKKeWjNACUUspH/X/IWPVSzhNbCgAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(range(300, 400+1, 10), scores)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-12-12T02:40:26.182668Z",
     "start_time": "2019-12-12T02:36:09.184080Z"
    }
   },
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "\n",
    "scores = []\n",
    "for i in range(390, 500+1, 10):\n",
    "    X_fschi = SelectKBest(chi2, k=i).fit_transform(X_fsvar, y)\n",
    "    score = cross_val_score(RFC(n_estimators=20, random_state=1), X_fschi, y, cv=5).mean()\n",
    "    scores.append(score)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-12-12T02:40:43.299582Z",
     "start_time": "2019-12-12T02:40:43.188478Z"
    },
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAD6CAYAAABOIFvoAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJzt3Xl8VPW9//HXJzPZSEKAEJCwI2FRICwhgKKiRQUVqrZVe1t71fZ6u2l7ba2219bftXbRtrR2r61btbWiVuqG4lJcSTAJCYsCYhZ2CclAEkL2z++POQkxJGYgkzmZmc/z8ciDOWe+c+bzJTDvOed7zveIqmKMMcYAxLhdgDHGmP7DQsEYY0w7CwVjjDHtLBSMMca0s1AwxhjTzkLBGGNMOwsFY4wx7SwUjDHGtLNQMMYY084bSCMRuQ84DXhOVe/s4vnxwG+BgcB6Vf2WiHiBEucH4AZV3eS0TwS2qOoEZ7kIOOS0+5GqvtRdLUOHDtVx48YFUrYxxhhHQUHBQVVN76ldj6EgIpcDHlVdICL3i0imqr7fqdldwA9VNVdEHhORRUA18Kiq3tLFZm8DRjjbTwO2qupVPdUCMG7cOPLz8wNpaowxxiEi5YG0C+Tw0SJgpfN4DbCwizaTgELn8QEgFZgPXCIi60XkPmfPARGZAswA8pz284AcEXlbRFaJSEoghRtjjAm+QEIhCdjjPK4ChnfR5gngdhFZBiwBXgHeARarag4QC1zktP05cGOH15YAF6rqGcBG4NrOGxeR60UkX0TyKyoqAijZGGPMyQgkFGqBROdxclevccYZVgNfAh5S1Vpgo6ruc5rkA5ki8gXgNVUt7fDyEmBHx3ZdbP9eVc1W1ez09B4PiRljjDlJgYRCAccOGWUBZd20KwLGACuc5YdFJEtEPMClQDH+vYjlIrIWmCkizwI/ApY5r/m0084YY4wLAjn7aBXwhohkAEuBq0TkTlW9rVO7m4EVqlrnLN8B/B0Q4GlVfRl4ua2xiKxV1UtEZASwSkR+DKwDHupdl4wxxpwsCeQmOyIyGDgfeF1V9/d5VR8jOztb7ewjY4w5MSJSoKrZPbUL6DoFVfVx7AwkY4wxEcquaDbG9Jm8kkoKyqvcLsOcAAsFY0zQqSq/+/cOrvpzLl95pJCWVrsXfLiwUDDGBFV9Uws3rSzmZy9uY/LwFA7UNJBXWul2WSZAFgrGmKA5UFPPVffm8tSGPXzr/En886tnkBTn4ZnifT2/2PQLFgrGmKDYvOcwn/ztW2zbX8MfPz+bGz6RyYA4L+efNpzVm/fR2NzqdokmABYKxpheW71pH5/54zoEeOIrC1gybUT7c8tnZnCorok3d9gUNeHAQsEYc9JUlV+/8j5f+VshU0aksOrrZ3J6RupH2iycmE5qYixPF+11qUpzIgK6TsEYYzqrb2rh248X8+zGfVw+ayQ/vnw6CbGe49rFeWO4aPop/KtoL0cbW0iMO76N6T9sT8EYc8I+rK7nij+t47lN+7hlyRR+cUVWl4HQZllWBnWNLbyy9cMQVmlOhoWCMeaEbNx9iOW/fZMdB2q59+psvrLoVETkY18zb3waw1Li7RBSGLBQMMYE7JnivXzmj+vwxsTw5FfO4PzTurq9yvE8McIlMzJYu62C6vqmPq7S9IaFgjGmR62tyoo127jh0Q3MGJXK018/k6kjBp7QNpbPzKCxpZUXN7s6p6bpgYWCMeZj1TU287W/F/LrV3fwmTmjeORL80hLjj/h7WSNSmXMkAE8XWyHkPozCwVjTLf2HjrKZ/64jhe37Oe2i6dy96dnEO89ubOHRIRlWSN4+4NKDtY2BLlSEywWCsaYLm3Y6WP5b9+ivLKO+/5zLl86a0KPA8o9WZ41kpZW5flNNu1Ff2WhYIw5zqoNe7jy3lwGxHl46qtncO6UYUHZ7uRTUpg8PMXOQurHLBSMMe1aW5W7X9jKNx8rYtboQaz62plkDk8J6nssn5lBfrmPPYeOBnW7JjgsFIwxABxpaOa/Hyng92s/4LM5Y3j4i/MYkhQX9PdZNiMDgGdtwLlfslAwxrDbV8en/vA2r7z3IbcvO40fXzaNOG/ffDyMSRvAzNGD7CykfspCwZgol19WxSd/+xZ7Dh3lwWtzuPbM8b0eUO7J8qwMtuyt5oOK2j59H3PiLBSMiWKP5+/iP/6cx8DEWFZ97UzOnpQekve9eMYIRLAB537IQsGYKNTSqvz4+fe4+YmNzB0/mKe+eganpieH7P2HD0xg/vg0ninei6rdv7k/sVAwJsrU1Ddx/V/zuff1Eq6eP5YHr81h0IDgDyj3ZPnMDEoOHmHL3uqQv7fpXkD3UxCR+4DTgOdU9c4unh8P/BYYCKxX1W+JiBcocX4AblDVTU77RGCLqk5wlv8PuMh57dd62SdjopKq0tDcSk19MzX1TdQ2NDuPP7r87Ma9fFBxhB9+8nSuXjDOtXqXTjuFH/xrM08X72XayNSeX2BCosdQEJHLAY+qLhCR+0UkU1Xf79TsLuCHqporIo+JyCKgGnhUVW/pYrO3ASOc7c8BFgI5wA9EZLGqvtyLPhkTdppa/B/mtfXN1DQ0feRxbX0z1fXNzoe6f7m2wVnXoU1NfTPNrT0fihk+MJ6/XpfDmROHhqBn3Rs0II6zM9N5pngvty6ZQkxM3w5um8AEsqewCFjpPF6D/wO8cyhMAgqdxweAVPx7FpeIyLnAJuC/VbVZRKYAM4A8p/05wJOqqiLyIrAUsFAwUeFoYwsX/Oo1dlX1fCFXrEdISYglOd5LcryXlAQvGYMSSElIaV9OTvCSEu9tb9e2bqCznBTv7bNTTU/G8pkZvLL1AAU7fcwdN8TtcgyBhUISsMd5XAXM7qLNE8DtIpILLAG+C0wFFqvqPhH5K/7DQ08DPwduAB7osP0POmz/uAnaReR64HqAMWPGBFCyMeFhwy4fu6qO8tmcMUwenkxyQiwpzgd7csJHP9zjvTF9fqpoqC2eOpyE2BieLtprodBPBBIKtUCi8ziZLganVfVOEVkI3Aw8pKq1IrJRVdumQswHMkXkC8Brqlra4R93INu/F7gXIDs7205VMBGjsNwHwK1LppA6INblakIvKd7LJ6YO5/lN+7h92Wl4Pf1nLyZaBfIbKMB/yAggCyjrpl0RMAZY4Sw/LCJZIuIBLgWK8e9FLBeRtcBMEXn2BLZvTMTJL/eROSw5KgOhzfKsDCqPNPLWB5Vul2IIbE9hFfCGiGTgP95/lYjcqaq3dWp3M7BCVeuc5TuAvwMCPO0MHrePFYjIWlW9RERigJ+IyD34Q2NJ77pkTHhobVUKy31cPGOE26W4atHkdFISvDxdtJdzQnTxnOlej6GgqtXO2UTnA3er6n783/o7t7u90/Jm/APK3W13kfNnq4gsBi4G7lHV0hPpgDHhakdFLdX1zcwZG93H0uO9HpacfgovbN5PfdM0EmJP7iY+JjgCOoCnqj5VXekEQtCp6lFVfUJVS3pubUxkKHDGE+aMHexyJe5bPjODmoZm1m474HYpUc9GdYxxSX6Zj7SkOMalDXC7FNctmJDG0OQ4nim2O7K5zULBGJcU7vQxe+zgiDvN9GR4PTFcNH0EL7/3IbUNzW6XE9UsFIxxwcHaBkoPHiHbDh21W56VQUNzKy+92ydHqU2ALBSMcUGhjSccZ/aYwYwclGjTabvMQsEYFxSU+4jzxNhEcB3ExAiXZI3gjfcP4jvS6HY5UctCwRgXFJT7mDZyoJ1+2cnyrAyaW5XnN9uAs1ssFIwJsYbmFjbuOUy2zfVznNNGDOTU9CQ7hOQiCwVjQmzznmoam1uZPcbGEzoTEZZnjWR9WRX7D9e7XU5UslAwJsQKyqsAG2TuzrKsEajCsxttb8ENFgrGhFhBuY+xaQNIT4l3u5R+aUJ6MtNGDuSZYgsFN1goGBNCqkpBuc/2EnqwPCuD4t2HKTt4xO1Soo6FgjEhtLOqjoO1jRYKPbhkRgaA7S24wELBmBDKL/NftJYd5TOj9iRjUCI544bwdPFeVO2+WqFkoWBMCBXs9JGS4CVzWLLbpfR7y2Zm8P6BWrbur3G7lKhioWBMCBWU+Zg9ZjAxMTYJXk8umnYKnhixQ0ghZqFgTIgcPtrE9gM1Np4QoLTkeM6cOJRnNtohpFCyUDAmRDbs9KGKzYx6ApZnZbCr6igbdh1yu5SoYaFgTIgUlvvwxAhZowe5XUrYuPD04cR5Y2zaixCyUDAmRPLLfUwdkUJSfI+3RjeOlIRYzps8jOc27aOl1Q4hhYKFgjEh0NzSStGuQ8yx+Y5O2PKZGVTUNJBbUul2KVHBQsGYENi6v4a6xhbm2MyoJ+y8KcNIjvfaIaQQsVAwJgQK7E5rJy0h1sMFpw1n9eZ9NDS3uF1OxAt5KIhIkoh8QkRGhfq9jXFLfrmPEakJjByU6HYpYWlZVgbV9c28sf2g26VEvIBCQUTuE5F1InJbN8+PF5HnROQNEfmFs84rIjtFZK3zM11EYoHngQXAMyJyutO2qEO784PUN2P6jcJyH7NtL+GkLcwcyuABsTxtF7L1uR5PgxCRywGPqi4QkftFJFNV3+/U7C7gh6qaKyKPicgioBp4VFVv6bCt04GfqeqzInIIWCgi+4GtqnpV0HplTD+y7/BR9hw6ypfOGu92KWEr1hPD0ukjeKpwD3WNzQyIi74zuFpaFU8IroQPZE9hEbDSebwGWNhFm0lAofP4AJAKzAcuEZH1zp6GV1W3OIEwC7jM2d48IEdE3haRVSKS0ov+GNPvtE2CZ+MJvbM8K4OjTS28/N4Bt0txxff/tZmv/72wz6/uDiQUkoA9zuMqYHgXbZ4AbheRZcAS4BXgHWCxquYAscBFHdovc967BigBLlTVM4CNwLWdNy4i14tIvojkV1RUBNQxY/qLgnIfibEepo4Y6HYpYS1n3BBOGZgQlWch7ThQw2Pv7CItKQ6Rvt1bCCQUaoG20bHkrl6jqncCq4EvAQ+pai2wUVX3OU3ygcwO7e8AHgG+iD8UdnTVrkP7e1U1W1Wz09PTA+mXMf1GQbmPrNGpxHrsZL/eiIkRLpkxgte2H+BwXZPb5YTUXS9sIzHWw42fOO7jMegC+VdawLFDRllAWTftioAxwApn+WERyRIRD3ApUCwiV4rI953nBwGHgB/h33MA+DRQfEI9MKYfq2ts5t191Xb/hCBZPjODphblhS37em4cIdaXVvHSux/ylUWnkpbc97dwDSQUVgFXi8gK4Apgi4jc2UW7m4EVqlrnLN8BPIw/LNap6svAU8BMEXkdmAs8hD9E/ldENgMNzjpjIkLRrkO0tKqNJwTJ9JGpjE0bwDPF0REKqsqPn3+P4QPjue7M0Jyo0OMQvqpWO2cTnQ/crar76eLbvKre3ml5MzCj07pG4FOdXroP/2CzMRGn0LlobbZNbxEUIsLyrAx+9+8dHKipZ1hKgtsl9anVm/dTtOsQd39qBolxnpC8Z0AHOVXVp6ornUAwxgQov9xH5rBkUgfEul1KxFielUGrwvMbI3tvobG5lbtf2Mqk4cl8ak7orvW1kS9j+khrq1JY7iN7nO0lBFPm8BSmnJIS8ReyPbp+J2WVdXx36dSQXJ/QxkLBmD6yo6KW6vpmO3TUB5bPzKBw5yF2VdX13DgM1dQ3cc8r77NgQhqLJof2jEsLBWP6SNskeNk2M2rQLZuRAcAzGyNzb+FPr5VQdaSR7140pc+vS+jMQsGYPpJf5iMtKY5xaQPcLiXijB4ygNljBkXkhWz7D9fzlzdLWJ6VwYxRob9Ln4WCMX2kcKd/ErxQf9OLFsuyMti6v4b3P6xxu5Sg+uVL22lpVW6+cLIr72+hYEwfOFjbQOnBI3Z9Qh+6eMYIYgSeiaAB5237a3i8YBdfWDCO0UPc2cO0UDCmD7Rdn5BtodBnhqUksODUNJ4u3tvnk8SFyl0vbCUp3svXz53oWg0WCsb0gYJyH3GeGKaNTHW7lIi2PCuDsso6Nu057HYpvfb2Bwd5desBvnbuRAYnxblWh4WCMX2goNzHtJEDSYgNzVWo0WrJ6SOI9UjYDzi3tio/Xb2VjNQErjljnKu1WCgYE2QNzS1s3HPYxhNCIHVALOdMGsazG/fR2hq+h5Ce3bSPjbsP860LJrv+RcJCIcw1tbSyasMeLv/9W1z7wHr2HDrqdklRb/OeahqbW5ljM6OGxLKsEeyvrmd9WZXbpZyUhuYWfvbiVqaOGMils0a6XY6FQrg6VNfI79fu4Ky7/s03HyviUF0T60urWPqr13l+U2TPCdPfFZT7P5xsTyE0zj9tOImxnrA9C+mR3J3sqjrKd5dOCel0Ft2JvhudhrmSiloeeKuMJwp2c7SphTMnpvGTy6dzzqR0dvnquPEfRXz1b4VcNXc0P1h2WlTey9ZtBeU+xqYNID2l7+e+NzAgzsvi04bz/KZ9/L/lp4fVzYwOH23iN6++z1mZQzl7Uv+4gZh9YoQBVWVdSSX3v1nKK1sPEBsTwydnZnDdwvEfucXj2LQknvjyAn750nb+8NoHrC+t4tefnWVnwISQqlJQ7us3/8GjxfKsDJ4p3subOw5y7uRhbpcTsD+s/YDDR5u4ZckUt0tpZ6HQjzU2t/JM8V7ue7OUd/dVMyQpjhvOy+Tz88d0O498rCeG7yyZwsLModz0WDGX/f4tblkyhevOHE9MP9g1jXQ7q+o4WNtoh45C7OxJQxmY4OWZor1hEwp7Dh3l/rdKuWzmyH71xc1CoR+qOtLI3/PKeWhdORU1DWQOS+anl0/n0lkjAz4z4YxTh7L6G2dxy5MbufO593htewW/uCIr4m9K4rb8sraL1myQOZTivR6WThvBsxv3Ut/U4voZPIFYsWY7ADddMMnlSj4qfA6+RYEdB2r47j83seAnr/DzNduZOmIgD12Xw5r/OZurcsac8D/0wUlx/OnqOfzosmm8U1bF0l+9watbP+yj6g1AwU4fKQleMoclu11K1Fk+M4MjjS08tWGP26X06N291fxzw26uPWMcowb3rwkTbU/BZarKmzsOct+bpazdVkGcN4bLZ43kuoXjmTQ8pdfbFxE+N28sOeOGcMOjG7juwXyuOWMcty6dEhbfpsJNQZmP2WMG26E6F8yfkEb22MF8f9VmBiXGsnT6CLdL6tZPX9jKwIRYvrrIveksumOh4JL6phaeLtrL/W+VsnV/DUOT47np/El8bt4Y0pKDf9ZK5vAUVn3tTO5+YRv3v1VKbkklv/7srKAEj/E7fLSJ7QdquHhG//0wimSeGOGBa+dyzQPv8PVHN/Bb6JfB8Mb7Fby+vYLbLp7aL2/TaoePQuxgbQO/enk7C+96le88uRGAn316Bm/dei43fiKzTwKhTUKshx8sO40HrplLRU0Dy37zJg/nlkfMZGJu27DTh6pNguemlIRYHrouh5mjB/H1Rzf0u2t2WluVnzy/lVGDE7l6wVi3y+mS7SmEyLb9Ndz/ZilPFe2hsbmV86YM44sLx3PGqWkhn2//3CnDWP3Ns/j24xv5/qrNvL69grs+NYMhLk7CFQkKy314YoSs0aG/MYo5Jjney0PX5XDN/eu54dENqNJv9t5WFe3h3X3V3HPVTOK9/fPwrYVCH1JVXttewX1vlvLG+wdJiI3hM3NGce2Z45no8kDksJQEHrxmLg+8XcZdq7ey9J7X+eUVMzlj4lBX6wpn+eU+po5IISne/lu5LTney4PX5XDtA+u58R8bUJRLnFt4uqW+qYWfv7iN6SNT228n2h/Zv94+UnrwCNf/NZ/3D9QyLCWemy+czH/kjHF1StzOYmKELy4cz7zxQ7jxHxv43H15fPmcU7np/ElhdVVof9Dc0krRrkN8Zs4ot0sxjuR4Lw9c6w+Gb/yjCFX/3drc8tDbZew9XM/Pr8jq1ycihPx/vogkicgnRCSi//f8Pa+c8so6fnllFm/ecp7rc6R/nGkjU3n2hoVcNXcMf1j7AZ/+w9uUHTzidllhZev+GuoaW5gzzq5P6E+S4708eG0Oc8YM5hv/2MDTLs2P5DvSyG//vYNzJ6dzxqn9e288oFAQkftEZJ2I3NbN8+NF5DkReUNEfuGs84rIThFZ6/xMF5FY4HlgAfCMiJzutP0/EXlHRH4XpH65LrekilljBnHZrFHEefv/t+4BcV5+cvl0/vC52ZRV1nHxr9/giYLdNggdoALnTmt2JXP/kxTv5YFr55I9bgjf/McG/lUU+usYfvfvHRxpaObWpVND/t4nqsdPKxG5HPCo6gJggohkdtHsLuCHqnoWMEpEFgEzgEdVdZHzswmYBPxMVe8E7gMWisgcYCGQAxwQkcVB6ZmLquub2LL3MPMmpLldyglbOn0Eq79xFtNGpvLtx4v5xj+KqK5vcrusfi+/3MeI1ARGDkp0uxTThaR4Lw9c4w+G/3msKKTBsKuqjr+uK+fTc0Yx+ZT+fwp4IF9hFwErncdr8H+AdzYJKHQeHwBSgfnAJSKy3tnT8KrqFlV9VkRmAZc52zsHeFL9X0lfBM7qvHERuV5E8kUkv6Ki4gS6546CMh+tCvPHh+ehhIxBifz9v+bz7Qsm8dymfVx0zxvt34RN1wrLfcy2vYR+LSney4PXziVnvD8YVoXoyuefr9lGTAz8z/n9azqL7gQSCklA299eFTC8izZPALeLyDJgCfAK8A6wWFVzgFjgog7tlznvXRPI9lX1XlXNVtXs9PT+P/tkbkklcZ4YZo0J3w8JT4zw9fMyefzLCxCBK/60jl+/8j4tYXx3q76y7/BR9hw6atcnhIEBcV7uv2Yu88ancdPKIp7asLtP32/T7sP8q2gvX1w4nhGp4bEXGUgo1AJtvUnu6jXO4aDVwJeAh1S1Ftioqm1XjuQDmR3a3wE8AnwxkO2Hm9zSKrJGp5IY1z/PQz4Rs8cM5vkbz2LZjBGseGk7n7031+7u1omNJ4SXjsHwrZXFfRYMqsqPn3+PIUlx/Pc5p/bJe/SFQD6ACzh2yCgLKOumXREwBljhLD8sIlki4gEuBYpF5EoR+b7z/CDg0AlsPyzUNjSzec9h5o0Pv/GE7qQkxPKrq2bxyyuzeHdftd3drZP8Mh+JsZ6P3NvC9G+JcR7uv2Yu8yekcdPKYv5ZGPxgWLu9gnUlldx43kQGJvS/6Sy6E0gorAKuFpEVwBXAFhG5s4t2NwMrVLXOWb4DeBh/WKxT1ZeBp4CZIvI6MBd4CHgTmCUi9wC3Ao/2pkNuyy+roqVVmR+Gg8w9uWzWKJ67cSET0pP56t8KeeydnW6X1C8U7vSRNTrVru0IM4lxHu77z7mccWoa33q8mCcLghcMLa3KT5/fyti0AfzHvP45nUV3evxXrKrV+Aebc4FzVbVYVY87NVVVb1fVhzssb1bVGao6XVX/11nXqKqfUtWzVfUqVa1X1VZgMfAGsFRVS4PUN1fklVbhjRFmj43MqQ7GpiXx+JcXkDNuCD97cRu1Dc1ul+SqusZmtuyttvsnhKnEOA9/+cJczjx1KN9+opgnghQMTxbuZtuHNXznwilhcUp6RwFVq6o+VV2pqvv7oghVPaqqT6hqSV9sP5TySiqZMSo1ou+NHOuJ4XsXT+VgbSP3vh72v7JeKd51mJZWtfGEMJYY5+Ev/5nNmacO5eYnink8f1evtne0sYUVa7aTNXoQF00/JUhVhk54RVg/V9fYzMbd4Xl9womaOXoQF88YwZ9fL+HD6nq3y3FNQXkV4B+QN+ErIdYfDAsnDuU7T25kZS+C4f63StlfXc/3lk4J+WSXwWChEEQF5T6aI3Q8oSvfuXAyza2t/Orl7W6X4pqCch+Zw5L75bz45sQkxHr48xf8wXDLkxtZ+c6JB0NlbQN/WPsBi6cOD9svhxYKQZRXUoUnRqLmUMLYtCQ+P38sj72zi/c/rHG7nJBrbVUKyn1kj4uO33c0aAuGszLT+c6TG0/4ZIrfvLqDusZmbl06uY8q7HsWCkGUW1LJtJGpJEfR1Mk3nJdJUpyXu17Y6nYpIbejopbq+mY7dBRhEmI93Hv1HM6elM4tT27iH+sDC4ayg0d4JLecK+eOYeKw/j+dRXcsFILkaGMLxbsPMX9CdJ2FMiQpjq+ceyovv3eA3JJKt8sJqbaL1rJtZtSI0xYM50xK59Z/buLRAILhZ2u2EeuJ4X8WdzU9XPiwUAiSDTt9NLUo8yPoorVAXXfmeEakJvCT59+LqllV88t8pCXFMS5tgNulmD6QEOvhT1fPYdHkdL77z038Pa/7YNiw08dzG/fxX2dPYNjAhBBWGXwWCkGSW1JJjBCVx5cTYj3cdP4kincf5rkoutK5cKd/ErxwPMPEBCYh1sMfPz+Hcyen872nNvG3vPLj2qgqP1m9laHJcVx/9gQXqgwuC4UgyS2tYtrIVFLC6HL2YLp89iimnJLC3S9so7G51e1y+tzB2gZKDx6JmpMKollCrIc/Xu0Phv99ajOP5H40GF557wDrS6v4xuJJETGeaKEQBPVNLRTtOsS8MJ0qOxg8McKtS6ews6ruuP80kaiwbTzBQiEqxHv9wXDelGHctmozDzv/xptbWvnpC1uZMDSJq+aOdrnK4Aj/WOsHinYdorG5NaImwTsZ50xKZ+HEofzm1ff51JxRpCZG7l5TQbmPOE8M00amul2KCZF4r4c/fH42X32kkO+v2gyqeD0x7DhQyx8/Pydi5r6KjF64LLekEhGYG8V7CgAi/r0FX10Tf3ztA7fL6VMF5T6mjRxIQmz4T49uAhfv9fD7z89m8dRhfP9fW7jz2XfJHjuYC0/v6jYz4clCIQjySqo4bcTAiP5mHKhpI1O5bNZI7n+zlL0Ret+FhuYWNu45bOMJUSre6+F3n/MHQ11TC9+9KDyns+iOhUIvNTS3ULjTF/WHjjr61gWTUIUVL0Xm9Beb91TT2NzKHJsZNWrFe/1nJb1+87kR9+/AQqGXincdpqG5NeouWvs4owYP4Jozx/Fk4W7e21ftdjlB1zYJnu0pRDevJ4bRQyLvGhULhV7Kc8YTcqJ8PKGzry3y323qp6sjb/qLgnIfY9MGkJ4S73YpxgSdhUIv5ZVWMXl4CoMGxLldSr+SOiCWr587kde2V/Dm+wdd+uz0AAASEklEQVTdLidoVP2T4M2x+Y5MhLJQ6IXG5lbyy6uiZqrsE3X1grGMHJTIT1a/R2trZEx/sbOqjoO1jcyJwivXTXSwUOiFTXsOUd9k4wndSYj18J0lk9myt5p/Fe9xu5ygyC/zX7Rm4wkmUlko9EJuiX/AMcfOPOrWshkZTBs5kJ+/uJ36pha3y+m1gp0+UuK9TArjqZGN+TgWCr2QW1LJ5OEpDEmy8YTuxMQI31s6lT2HjvLXdWVul9NrBWU+Zo0dTExM5JyXbkxHFgonqamllYJyH/Ps0FGPzpg4lEWT0/ntqzs4VNfodjkn7fDRJrYfqLH5jkxEs1A4SZv3HKauscUuWgvQrUunUNPQzO/+vcPtUk7ahp0+VG08wUQ2C4WT1DaeYHsKgZlyykA+PXsUD71dzq6qOrfLOSmF5T5iBGaOHuR2Kcb0mYBCQUTuE5F1InJbN8+PF5HnROQNEfmFs84rIjtFZK3zM11EYkXkMRFZIyKvishgp21Rh3bnB697fSevtJKJw5IZmmwXMAXqpgsmIQK/WLPN7VJOSn65j6kjBpIUAXPmG9OdHkNBRC4HPKq6AJggIl3dgPQu4IeqehYwSkQWATOAR1V1kfOzCVgKvKCqFwAvAleLSBqwtUO7l4LUtz7T3NJKfpkvqu+fcDJGpCbyxYXjWVW0l817DrtdzglpbmmlaNchG08wES+QPYVFwErn8RpgYRdtJgGFzuMDQCowH7hERNY7expeVX1aVR9w2qU7becBOSLytoisEpHjzvUTketFJF9E8isqKgLuXF/Zsrea2oZmu2jtJHx50akMSYrjx2F2P+et+2uoa2xhzjj7ImAiWyChkAS0XXlUBXQ1cfgTwO0isgxYArwCvAMsVtUcIBa4qK2xiEwAzgOeBEqAC1X1DGAjcG3njavqvaqararZ6enpgfatz+SVVgI2nnAyBibEcuN5E3n7g0rWbnc/4ANVUG4XrZnoEEgo1AKJzuPkrl6jqncCq4EvAQ+pai2wUVXb7uKeD2QCiEg88CBwvao24Q+FHZ3b9Wd5JVVMGJrEsJQEt0sJS/8xbyxj0wbw0+e30hIm01/kl/sYkZrAyEGJPTc2JowFEgoFHDtklAWUddOuCBgDrHCWHxaRLBHxAJcCxc76B4AHVTXfWf4RsMx5/OkO7fqlllZlfWkV8+zQ0UmL88bwnQunsO3DGp4s3O12OQEpLPcx2/YSTBQIJBRW4R8QXgFcAWwRkTu7aHczsEJV2843vAN4GH9YrFPVl0VkKXAZ8AXnTKNv4A+R/xWRzUAD8FDvutS33ttXTU1Ds8131EsXTT+FrNGDWLFmO0cb+/f0F/sOH2XPoaM2yGyiQo/n1qlqtXM20fnA3aq6ny6+zavq7Z2WN+M/A6njutUcOxTV0bwTqNlVuSXOeIJdtNYrIsL3lk7hyntzuf+tUr527kS3S+qWjSeYaBLQdQqq6lPVlU4gRLW80irGpQ3glFQbT+iteRPSWDx1OH9Y+wGVtQ1ul9Ot/DIfibEepo4Y6HYpxvQ5u6L5BLS2jSfYXkLQ3Lp0MkebWvjNq/13+ovCnT6yRqcS67H/Liby2b/yE7B1fw2HjzbZqahBNHFYClfOHc0jueWUHTzidjnHqWtsZsvearIj7ObsxnTHQuEEHLs+wfYUgumbizOJ88bwsxf73/QXxbsO09KqNp5gooaFwgnILalk9JBEO1c9yIalJPBfZ03guU372LDT53Y5H1FQ7p/4cLbdk9lECQuFANl4Qt/6r7MnMDQ5np88v7VfTX9RUO4jc1gyqQNi3S7FmJCwUAjQ+wdq8dU12XxHfSQ53ss3F2eyvqyKl9874HY5gP+LQEG5j+xxtpdgooeFQoCOXZ9gA4595cq5o5mQnsRPV79Hc0ur2+XwQUUt1fXNdujIRBULhQDllVYyclAio4cMcLuUiBXrieGWJVP4oOIIK/Pdn/4i37loLdtmRjVRxEIhAKpKXkmV7SWEwAWnDSd77GB++fJ2jjQ0u1pLQbmPtKQ4xqXZFwETPSwUArDjQC2VRxptPCEERITvXTyVipoG/vJGqau1FDiT4ImIq3UYE0oWCgHILbX7MYfS7DGDuWj6Kfzp9Q+oqHFn+ovK2gZKDx6x6xNM1LFQCEBeSSWnDExgjI0nhMzNF06hsbmVX7283ZX3b5sEz2ZGNdHGQqEHqkpuSRXzJwyxwwghNH5oEp+bN4Z/vLOLHQdqQ/7+BTt9xHlimDYyNeTvbYybLBR6UHLwCAdrG2xqCxfc8IlMEmM93P3C1pC/d0GZj2kjB5IQ6wn5exvjJguFHuSVOOMJduZRyA1NjufL50xgzbsf8k5ZVcjet6G5hY17Dtt4golKPd5kJ9rlllQyLCWe8UOT3C4lKn1x4QQezi3nB//awmdzRpOS4CU5Ptb508vAhFiSE7ykJHiDNrX15j3VNDa3MsdmRjVRyELhY6gqeaWVzJuQZuMJLkmM83Dbxadx08oifvCvLR/bNt4bQ0rCscA49qd/XcfltiBJ6bCcHO//aZsEz/YUTDSyUPgY5ZV1fFjdYIeOXLYsK4PzTxtObUMzNfXN1NY3U1PfRE37cpP/z4Zmqp0/a+qbqK1vpry27thyQzOtAcy154kRxqYNID0lvu87Z0w/Y6HwMdrmO7KL1tyXEOshIdbD0OST/6BWVeoaW5wAafIHSH1z+3KN87imvpkzJ9rv3EQnC4WPkVdaxdDkeE5Nt/GESCAiJMV7SYr3AnaPbWO6YmcfdcM/31El88bb9QnGmOhhodCN3b6j7D1cz3yb2sIYE0UsFLqxrsTux2yMiT4BhYKI3Cci60Tktm6eHy8iz4nIGyLyC2edV0R2isha52e6iMSKyGMiskZEXhWRwU7b/xORd0Tkd8HrWu/klVQxJCmOzGHJbpdijDEh02MoiMjlgEdVFwATRCSzi2Z3AT9U1bOAUSKyCJgBPKqqi5yfTcBS4AVVvQB4EbhaROYAC4Ec4ICILA5Kz3opr9TGE4wx0SeQPYVFwErn8Rr8H+CdTQIKnccHgFRgPnCJiKx39jS8qvq0qj7gtEt32p4DPKn+u7W/CJx1Uj0Jot2+Onb7jtr1CcaYqBNIKCQBe5zHVcDwLto8AdwuIsuAJcArwDvAYlXNAWKBi9oai8gE4DzgyUC2LyLXi0i+iORXVFQE0q9eaZ/vyMYTjDFRJpBQqAUSncfJXb1GVe8EVgNfAh5S1Vpgo6ruc5rkA5kAIhIPPAhcr6pNAW7/XlXNVtXs9PT0ALt28vJKKxk0IJbJw1P6/L2MMaY/CSQUCjh2yCgLKOumXREwBljhLD8sIlki4gEuBYqd9Q8AD6pq/gluP2RyS6rIGTeEmBgbTzDGRJdAQmEV/gHhFcAVwBYRubOLdjcDK1S1zlm+A3gYf1isU9WXRWQpcBnwBeeMpG8AbwKzROQe4Fbg0d51qXf2HT7Kzqo6O3RkjIlKPU5zoarVztlE5wN3q+p+jn3r79ju9k7Lm/GfgdRx3WqOHSpq55xxdDFwj6q6erf2tvEEu2jNGBONApr7SFV9HDsDKehU9Sj+wWrX5ZZUMjDBy5RTBrpdijHGhJxd0dxJXmkVOeOH4LHxBGNMFLJQ6ODD6npKDx6xqbKNMVHLQqGDtvsnzBtvoWCMiU4WCh3klVaREu/ltAwbTzDGRCcLhQ7ySiqZa+MJxpgoZqHgOFBTzwcVR2y+I2NMVLNQcKwvtfmOjDHGQsGRV1JFUpyHaTaeYIyJYhYKjtySSrLHDcHrsb8SY0z0sk9AoLK2gfcP1DLPprYwxkQ5CwWOjSfYRWvGmGhnoYD/+oQBcR6mj0x1uxRjjHGVhQL+8YQ5YwcTa+MJxpgoF/Wfgr4jjWzdX2OHjowxBgsF1pc51yfYRWvGGGOhkFtSSUJsDDNGDXK7FGOMcV3Uh0JeSRWzxwwmzhv1fxXGGBPdoXC4ron39lfbeIIxxjiiOhTWl1WhauMJxhjTJqpDIa+kkjhvDFmjbTzBGGMg2kOhtIrZYwaREOtxuxRjjOkXojYUquub2LL3sN160xhjOojaUMgvq6JVsUnwjDGmg4BCQUTuE5F1InJbN8+PF5HnROQNEfmFs84rIjtFZK3zM91ZP0BEijq8tst2fS2vpIo4TwyzxwwOxdsZY0xY6DEURORywKOqC4AJIpLZRbO7gB+q6lnAKBFZBMwAHlXVRc7PJhHxACuBjiO7x7XrbacCkVtSyczRNp5gjDEdBbKnsAj/BznAGmBhF20mAYXO4wNAKjAfuERE1jt7Gl7n+euBsg6v7a5dn6ltaGbz3mo7dGSMMZ0EEgpJwB7ncRUwvIs2TwC3i8gyYAnwCvAOsFhVc4BY4CJVbVHVvZ1ee1y7zhsXketFJF9E8isqKgLp18fKL6uipVXtojVjjOkkkFCoBRKdx8ldvUZV7wRWA18CHlLVWmCjqu5zmuQDXR12IpB2qnqvqmaranZ6enoAJX+83JIqYj1i4wnGGNNJIKFQwLFDRll89NBPR0XAGGCFs/ywiGQ54wiXAsXdvC7QdkGTV1rJjFGDSIyz8QRjjOkokFBYBVwtIiuAK4AtInJnF+1uBlaoap2zfAfwMP6wWKeqL3ez/UDbBcWRhmY27T7MfBtPMMaY4/Q4qKuq1c7ZROcDd6vqfrr4Nq+qt3da3oz/zKKutrkokHZ9oaDcR3Or2kVrxhjThYDO9FFVH8fOQApreaWVeGKEOWNtPMEYYzqLuiua80qqmDEqlaT4Pj/z1Rhjwk5UhcLRxhaKdx+yQ0fGGNONqAqFwp0+mlrULlozxphuRFUo5JX4xxOybTzBGGO6FFWhkFtaxbSMgaQkxLpdijHG9EtREwr1TS0U7TzEPJvawhhjuhU1oVBd38SSaaewaFLvp8kwxphIFTXnZQ5LSeDXn53ldhnGGNOvRc2egjHGmJ5ZKBhjjGlnoWCMMaadhYIxxph2FgrGGGPaWSgYY4xpZ6FgjDGmnYWCMcaYdqKqbtdwQkSkAih3u44ADQUOul1EH4rk/lnfwlck9683fRurqj1O6RB2oRBORCRfVbPdrqOvRHL/rG/hK5L7F4q+2eEjY4wx7SwUjDHGtLNQ6Fv3ul1AH4vk/lnfwlck96/P+2ZjCsYYY9rZnoIxxph2FgrGGGPaWSgEiYgMF5ENzuP7RGSdiNzW4fnj1oWLtr6JSKqIrBaRNSLylIjEOc+Hbd/go7+7bpbDtn9d9OX3IrKsw3LY901EBovI8yKSLyJ/6vB8WPZNRLwislNE1jo/00Xk/0TkHRH5XYd2x60LBguF4Pk5kCgilwMeVV0ATBCRzK7WuVrpifs5kAh8DlihqhcA+4ElEdA3ONa/45YjoH8d+3IWcIqqPuMsR0rfrgb+5py/nyIi2WHetxnAo6q6SFUXAXHAQiAHOCAii0VkTud1wXpzC4UgEJHzgCP4PygXASudp9bg/8V1tS4sdOybqv5eVV9ynkoHDhDGfYPjfnfHLRPG/evYFxGJBf4MlInIJ50mi4iAvgGVwDQRGQSMBnYRxn0D5gOXiMh6EbkP+ATwpPrPCnoROAs4p4t1QWGh0EvOIZTvA7c6q5KAPc7jKmB4N+v6vS761rZ+ATBYVXMJ077B8f3rpr9h2b8u+vIF4F3gbiBHRG4gcvr2JjAWuBF4D39fwrJvjneAxaqaA8Ti3xsK2WeKhULv3Qr8XlUPOcu1HDsUkYz/77irdeGgc98QkSHAb4DrnFXh2jc4vn/H9Zfw7V/nvswC7lXV/cAjwLlETt9uB76sqncAW4FrCd++AWxU1X3O43xC/JkSTn9R/dVi4GsishaYCSzj2K5qFlAGFHSxLhx8pG8icj/wOPBdVW2blDBc+wbH/+6u4aP9/Qvh27/OffsyMMF5Lhv/pJKR0rfxwHQR8QDzACV8+wbwsIhkOf25FP9eQeg+U1TVfoL0A6wFBgLFwAr8u7KpXa1zu9aT7NtXAJ/zeC1wZST0ra1/XS1HQv+c31UK/kB/HVgHjIygvuUAW/B/e34J/zfnsO0bMA3YCGwCfoT/y/tbwD3ANvwheNy6YL2/XdHcB0RkMHA+8Lr6d9e7XBcpIrlvENn9s76FBxFJBC4GClW1pLt1QXkvCwVjjDFtbEzBGGNMOwsFY4wx7SwUjDHGtLNQMMYY085CwRhjTLv/D8ZvbyINccqAAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(range(390, 500+1, 10), scores)  # 可以看到在470左右取到最大值\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-12-12T02:41:31.482673Z",
     "start_time": "2019-12-12T02:41:31.456881Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.9543812394392164"
      ]
     },
     "execution_count": 49,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "max(scores)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    ">用卡方检验找出k值"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    ">p值法的大概原理是：利用小概率事件不可能发生，如果我们认为小概率事件发生的概率正常为p，如果超过p，那么我们就可以认为这件事不是小概率了。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "卡方检验的本质是推测两组数据之间的差异，其检验的原假设是”两组数据是相互独立的”（即，两组数据不相关）。卡方检验返回卡方值和\n",
    "P值两个统计量，其中卡方值很难界定有效的范围，而p值，我们一般使用0.01或0.05作为显著性水平，即p值判断\n",
    "的边界，具体我们可以这样来看："
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![](images/3_8.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "从特征工程的角度，我们希望选取卡方值很大，p值小于0.05的特征，即和标签是相关联的特征。而调用\n",
    "SelectKBest之前，我们可以直接从chi2实例化后的模型中获得各个特征所对应的卡方值和P值"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-12-12T02:45:41.074745Z",
     "start_time": "2019-12-12T02:45:40.477926Z"
    }
   },
   "outputs": [],
   "source": [
    "chi_values, chi_pvalues = chi2(X_fsvar, y) # 卡方检验会返回两个变量，一个是卡方分布的分布值，一个是P值"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-12-12T02:53:10.775460Z",
     "start_time": "2019-12-12T02:53:10.762534Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0"
      ]
     },
     "execution_count": 65,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 可以通过p过滤，\n",
    "p = 0.05\n",
    "(chi_pvalues > p).sum()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-12-12T02:53:30.169423Z",
     "start_time": "2019-12-12T02:53:30.161983Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "708"
      ]
     },
     "execution_count": 68,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#k取多少？我们想要消除所有p值大于设定值，比如0.05或0.01的特征：\n",
    "p = 0.05\n",
    "k = chi_values.shape[0] - (chi_pvalues > p).sum()\n",
    "#X_fschi = SelectKBest(chi2, k=填写具体的k).fit_transform(X_fsvar, y)\n",
    "#cross_val_score(RFC(n_estimators=10,random_state=0),X_fschi,y,cv=5).mean()\n",
    "k"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    ">用F检验选择k值"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "F检验，又称ANOVA，方差齐性检验，是用来捕捉每个特征与标签之间的线性关系的过滤方法。它即可以做回归也\n",
    "可以做分类，因此包含**feature_selection.f_classif**（F检验分类）和**feature_selection.f_regression**（F检验回\n",
    "归）两个类。其中F检验分类用于标签是离散型变量的数据，而F检验回归用于标签是连续型变量的数据"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "和卡方检验一样，这两个类需要和类**SelectKBest**连用，并且我们也可以直接通过输出的统计量来判断我们到底要\n",
    "设置一个什么样的K。需要注意的是，F检验在数据服从正态分布时效果会非常稳定，因此如果使用F检验过滤，我\n",
    "们会先将数据转换成服从正态分布的方式"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "F检验的本质是寻找两组数据之间的线性关系，其原假设是”数据不存在显著的线性关系“。它返回F值和p值两个统\n",
    "计量。和卡方过滤一样，**我们希望选取p值小于0.05或0.01的特征，这些特征与标签时显著线性相关的**，而p值大于\n",
    "0.05或0.01的特征则被我们认为是和标签没有显著线性关系的特征，应该被删除。以F检验的分类为例，我们继续\n",
    "在数字数据集上来进行特征选择"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### 小结\n",
    "+ 卡方检验(一般检验特征与标签的关系)：用于检验两组数据是否相互独立(**原假设两组数据是相互独立的**)。  \n",
    "卡方检验会返回一个chi_values和一个chi_pvalues。当chi_pvalues中values>p，（我们认为小概率事件发生的概率为p）我们就可以认为这两组数据是相互独立的。  \n",
    "既然两组数据是相互独立(即，特征与标签是相互独立)，也就是他们是不相关的，那我们就不能用该特征预测标签。  \n",
    "所以，我们可以用$特征的数量 - 线性不是相互独立的特征的数量 = k$\n",
    "+ F检验(检验特征与标签的关系)：用于检验两组数据是否线性相关(**原假设两组数据不存在显著的线性关系**)  \n",
    "F检验与卡方检验一样会返回一个F_values和一个F_pvalues。当F_pvalues中的values>p，我们就可以认为这两组数据是显著性相关的。所以我们需要过滤这些数据\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-12-12T03:06:27.586014Z",
     "start_time": "2019-12-12T03:06:26.995247Z"
    }
   },
   "outputs": [],
   "source": [
    "from sklearn.feature_selection import f_classif\n",
    "\n",
    "F_values, F_pvalues = f_classif(X_fsvar, y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-12-12T03:09:51.441287Z",
     "start_time": "2019-12-12T03:09:51.430376Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "48"
      ]
     },
     "execution_count": 73,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "p = 0.05\n",
    "(F_pvalues>p).sum()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-12-12T03:12:45.800115Z",
     "start_time": "2019-12-12T03:12:45.112132Z"
    }
   },
   "outputs": [],
   "source": [
    "p = 0.05\n",
    "k = X_fsvar.shape[1] - (F_pvalues>p).sum()\n",
    "X_fsf = SelectKBest(f_classif, k=k).fit_transform(X_fsvar, y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 85,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-12-12T03:13:49.640669Z",
     "start_time": "2019-12-12T03:13:27.668801Z"
    }
   },
   "outputs": [],
   "source": [
    "score = cross_val_score(RFC(n_estimators=20, random_state=1), X_fsf, y, cv=5).mean()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 89,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-12-12T03:17:18.806761Z",
     "start_time": "2019-12-12T03:17:18.780971Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "只过滤方差为0的：0.9537859105492401， F检验后的:0.9537625033693994\n"
     ]
    }
   ],
   "source": [
    "\"\"\"虽然分数下降了，但下降的幅度不是很明显，我们可以在运行时间与准确度去平衡\"\"\"\n",
    "print('只过滤方差为0的：{}， F检验后的:{}'.format(pre_score.mean(), score))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    ">使用互信息法选出k值"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "互信息法是用来捕捉每个特征与标签之间的任意关系（包括线性和非线性关系）的过滤方法。和F检验相似，它既\n",
    "可以做回归也可以做分类，并且包含两个类**feature_selection.mutual_info_classif**（互信息分类）和\n",
    "**feature_selection.mutual_info_regression**（互信息回归）。这两个类的用法和参数都和F检验一模一样，不过\n",
    "互信息法比F检验更加强大，F检验只能够找出线性关系，而互信息法可以找出任意关系。  \n",
    "\n",
    "互信息法不返回p值或F值类似的统计量，它返回“每个特征与目标之间的互信息量的估计”，这个估计量在[0,1]之间\n",
    "取值，为0则表示两个变量独立，为1则表示两个变量完全相关。以互信息分类为例的代码如下"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 91,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-12-12T03:30:08.143578Z",
     "start_time": "2019-12-12T03:25:53.152160Z"
    }
   },
   "outputs": [],
   "source": [
    "from sklearn.feature_selection import mutual_info_classif as MIC\n",
    "\n",
    "result = MIC(X_fsvar, y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 96,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-12-12T03:41:36.643118Z",
     "start_time": "2019-12-12T03:37:21.459007Z"
    }
   },
   "outputs": [],
   "source": [
    "k = X_fsvar.shape[1] - (result<=0).sum()\n",
    "X_fsmic = SelectKBest(MIC, k=k).fit_transform(X_fsvar, y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 97,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-12-12T03:41:58.247127Z",
     "start_time": "2019-12-12T03:41:36.882510Z"
    }
   },
   "outputs": [],
   "source": [
    "score = cross_val_score_val_score_val_score(RFC(n_estimators=20, random_state=1), X_fsmic, y, cv=5).mean()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 101,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-12-12T03:44:08.596934Z",
     "start_time": "2019-12-12T03:44:08.586493Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.9537859105492401 0.9527383252946129\n"
     ]
    }
   ],
   "source": [
    "print(pre_score.mean(), score)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "当然了，无论是F检验还是互信息法，大家也都可以使用学习曲线，只是使用统计量的方法会更加高效。当统计量\n",
    "判断已经没有特征可以删除时，无论用学习曲线如何跑，删除特征都只会降低模型的表现。当然了，如果数据量太\n",
    "庞大，模型太复杂，我们还是可以牺牲模型表现来提升模型速度，一切都看大家的具体需求"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 过滤法总结"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "到这里我们学习了常用的基于过滤法的特征选择，包括方差过滤，基于卡方，F检验和互信息的相关性过滤，讲解\n",
    "了各个过滤的原理和面临的问题，以及怎样调这些过滤类的超参数。通常来说，我会建议，先使用方差过滤，然后\n",
    "使用互信息法来捕捉相关性，不过了解各种各样的过滤方式也是必要的。所有信息被总结在下表："
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![](images/3_9.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Embedded嵌入法"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "嵌入法是一种让算法自己决定使用哪些特征的方法，即特征选择和算法训练同时进行。在使用嵌入法时，我们先使\n",
    "用某些机器学习的算法和模型进行训练，得到各个特征的权值系数，根据权值系数从大到小选择特征。这些权值系\n",
    "数往往代表了特征对于模型的某种贡献或某种重要性，比如决策树和树的集成模型中的feature_importances_属\n",
    "性，可以列出各个特征对树的建立的贡献，我们就可以基于这种贡献的评估，找出对模型建立最有用的特征。因此\n",
    "相比于过滤法，嵌入法的结果会更加精确到模型的效用本身，对于提高模型效力有更好的效果。并且，由于考虑特\n",
    "征对模型的贡献，因此无关的特征（需要相关性过滤的特征）和无区分度的特征（需要方差过滤的特征）都会因为\n",
    "缺乏对模型的贡献而被删除掉，可谓是过滤法的进化版"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![](images/3_10.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "过滤法中使用的统计量可以使用统计知识和常识来查找范围（如p值应当低于显著性水平0.05），而嵌入法中使用\n",
    "的权值系数却没有这样的范围可找——我们可以说，权值系数为0的特征对模型丝毫没有作用，但当大量特征都对\n",
    "模型有贡献且贡献不一时，我们就很难去界定一个有效的临界值。这种情况下，模型权值系数就是我们的超参数，\n",
    "我们或许需要学习曲线，或者根据模型本身的某些性质去判断这个超参数的最佳值究竟应该是多少。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "另外，嵌入法引入了算法来挑选特征，因此其计算速度也会和应用的算法有很大的关系。如果采用计算量很大，计\n",
    "算缓慢的算法，嵌入法本身也会非常耗时耗力。并且，在选择完毕之后，我们还是需要自己来评估模型。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "SelectFromModel是一个元变换器，可以与任何在拟合后具有coef_，feature_importances_属性或参数中可选惩\n",
    "罚项的评估器一起使用（比如随机森林和树模型就具有属性feature_importances_，逻辑回归就带有l1和l2惩罚\n",
    "项，线性支持向量机也支持l2惩罚项）。  \n",
    "\n",
    "对于有feature_importances_的模型来说，若重要性低于提供的阈值参数，则认为这些特征不重要并被移除。\n",
    "feature_importances_的取值范围是[0,1]，如果设置阈值很小，比如0.001，就可以删除那些对标签预测完全没贡\n",
    "献的特征。如果设置得很接近1，可能只有一两个特征能够被留下"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    ">带有惩罚项的模型的嵌入法\n",
    "\n",
    "对于带有惩罚项的模型来说，正则化的惩罚项越大，特征在模型中的系数就越小。当正则惩罚项越大(最大为1)时，特征在模型中的系数就越接近于0。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![](images/3_11.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-12-13T07:00:48.861512Z",
     "start_time": "2019-12-13T07:00:31.376976Z"
    }
   },
   "outputs": [],
   "source": [
    "from sklearn.feature_selection import SelectFromModel \n",
    "from sklearn.ensemble import RandomForestClassifier as RFC\n",
    "\n",
    "from sklearn.feature_selection import VarianceThreshold\n",
    "from sklearn.ensemble import RandomForestClassifier as RFC\n",
    "from sklearn.neighbors import KNeighborsClassifier as KNN\n",
    "from sklearn.model_selection import cross_val_score\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "\n",
    "\n",
    "data = pd.read_csv('data/digit recognizor.csv')\n",
    "X = data.iloc[:, 1:]\n",
    "y = data.iloc[:, 0]\n",
    "\n",
    "# 这里用方差过滤法\n",
    "X_fsvar = VarianceThreshold(threshold=0).fit_transform(X)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-12-13T07:01:20.104266Z",
     "start_time": "2019-12-13T07:01:13.721464Z"
    }
   },
   "outputs": [],
   "source": [
    "rfc = RFC(n_estimators=10, random_state=1)\n",
    "# 在这里我只想取出来有限的特征。0.005这个阈值对于有780个特征的数据来说，是非常高的阈值，因为平均每个特征\n",
    "# 只能够分到大约0.001的feature_importances\n",
    "x_embedded = SelectFromModel(rfc, threshold=0.005).fit_transform(X_fsvar, y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-12-13T07:01:24.831438Z",
     "start_time": "2019-12-13T07:01:24.804101Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(42000, 49)"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x_embedded.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-12-13T07:02:16.230250Z",
     "start_time": "2019-12-13T07:02:04.987934Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.899524461464263"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cross_val_score(rfc, x_embedded, y, cv=5).mean()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-12-13T07:15:35.652096Z",
     "start_time": "2019-12-13T07:15:29.426431Z"
    }
   },
   "outputs": [],
   "source": [
    "# 绘制学习曲线\n",
    "rfc = RFC(n_estimators=10, random_state=1)\n",
    "threshold = np.linspace(0, rfc.fit(X,y).feature_importances_.max(), 20)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-12-13T07:20:57.424403Z",
     "start_time": "2019-12-13T07:15:35.870862Z"
    }
   },
   "outputs": [],
   "source": [
    "scores = []\n",
    "for i in threshold:\n",
    "    X_embedded = SelectFromModel(rfc, threshold=i).fit_transform(X, y)\n",
    "    score = cross_val_score(rfc, X_embedded, y, cv=5).mean()\n",
    "    scores.append(score)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-12-13T07:23:37.131637Z",
     "start_time": "2019-12-13T07:23:36.785668Z"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXcAAAD6CAYAAABamQdMAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJzt3Xl4lfWd/vH3J3vIBiEhYYdABFFBMYLI7lIVt5a22E6L1VpR62g7nfaqjvZypsP82jqOjrbVikU71WpLW+tCteICgsgWUFCUfdOwhS1hCyHJ5/dHjooxIYfkhOeck/t1Xbny5JzvOec2mDtPnvM836+5OyIiEl8Sgg4gIiKRp3IXEYlDKncRkTikchcRiUMqdxGROKRyFxGJQyp3EZE4pHIXEYlDKncRkTiUFMSL5uXleZ8+fYJ4aRGRmLV06dJd7p4fzthAyr1Pnz6UlpYG8dIiIjHLzDaHO1aHZURE4pDKXUQkDqncRUTikMpdRCQOqdxFROKQyl1EJA6p3EVE4lAg57m31PaKKp5avOX4g8JYNjA1OZHUpATSkhNJS04kPTmRtOSPv04gNSmR9JT6+9KOGZeYYBH6LxERaVsxVe47Kqt48LW1zY6z43Rwa5aMTUlMIC05gfysVLrmpFOYk0bXnLRPP2fX39apQzJ2vBAiIm0spsp9SM+ObPr5Za16DnenuraOqqN1HDlaS9XROqpqajlcXUvV0VqqaurqP3/yUffp55paDh2pYef+I2yrqGL+ul3sqKyirsEvjNSkBApz0ijM/rj80+mak0ZB6OtuHdPJy0zRLwARaTMxVe6RYGakJiWSmpQI6cmtfr6a2jp2HahmW8VhtldUsa2iih2V9Z+3V1SxdMtedlRsp7q27jOPS0lKoFuo6Lt3TP/M524d629PS05sdT4RaZ/aXblHWlJiaC89J63JMXV1zp5D1Z+U/7aKw5TtO0zZ3sNs3XeYeWt3sWN/1ecOGeVlptSXfc6npd+jUzqDe3SkW8f0Nv4vE5FYpnI/CRISjLzMVPIyUzm9e06jY47W1rG9ooqyffWFv3XfYcr2VbF132HWlx9g7tpyDlXXfjK+R6d0hvftzPC+uQzrm0vvzh10mEdEPqFyjxLJiQn0zO1Az9wOjd7v7lQcPsrm3YdYunkvizfuYfbqnfx12UcAFGSnMqxvZ4b1zWV431yKu2Sq7EXaMfPWnD7SQiUlJa4pf1vP3Vm38wCLNu5h8cY9LNq4mx2VRwDIzUjhnD6dGBbauz+1a7ZO5RSJcWa21N1LwhmrPfcYZmYUF2RRXJDFN8/tjbuzZc+hT8p+8cY9vLxyBwBZqUmUhMr+9O7ZDCjMIj8zVXv3InEqrHI3s+nAIODv7j61kfv7Ar8CsoHF7v6vEU0pYTEzenfOoHfnDCaV9ARg677DLNm0h0Ub97Bow25mry7/ZHynDskMKMxiYGE2pxRkMaCw/iMzVb/zRWJdsz/FZjYRSHT3EWb2mJkVu3vDK4l+Afynuy80sz+Z2Th3n9MWgeXEdOuYzlVndueqM7sDsOdgNau2V7J6+35Wb9/Pqu37mVH64eferB1wTNkPKMyiKC+TlCTNViESK8LZRRsHzAhtzwJGAQ3L/RRgWWh7J9D4KSESuNyMFM7rl8d5/fI+ua2uzinbd5hV2/ezenslq3ccYPX2St5YU05N6AqtpASjX34mAwqzmDi0O2NPydchHZEoFk65ZwBloe09wNBGxvwFuNvMFgKXAHc0HGBmU4ApAL169WpRWGkbCQn2yZk6Fw0q+OT26po6Nuw68Mke/urt+3lr/S6eX76VwT1yuO38Yi44tYtKXiQKhVPuB4CPr5jJpJGZJN19qpmNAn4E/J+7H2hkzDRgGtSfLdPixHLSpCQlMLAwm4GF2VwVuq26po5nln3Er+es4zu/L+W0btncen5/vjCokASdjSMSNcI5iLqU+kMxAEOATU2MewfoBdzX+lgSrVKSEvjasF68/q/juPerQzh4pIabnlzGpQ/M44XlW6ltONGOiASi2fPczSwbmAe8BlwKfA34qrvf1WDcfwDr3P2J5l5U57nHj5raOmau2MYvX1/L+vKD9MvP4Nbzi7l8cFeSEvUGrEgknch57mFdxGRmnYCLgLnuvr2V+VTucai2znnpvW388rV1rN6xn755GXx3XD++eFZ3klXyIhER8XKPNJV7/Kqrc2a9v4MHX1vL+9sq6ZmbznfH9efLQ3voVEqRVlK5S+Dcndc+2MkvX1/L8o8q6JaTxs3j+zOppEf9dMsicsJU7hI13J031pTz4GtrWbZlHwXZqdw8th9fG9ZL89WLnCCVu0Qdd+et9bt54NW1LN60h4LsVL47rj9Xn9NTJS8SJpW7RC13Z8H63dz/6hqWbNpLYXYa3x3fj0klKnmR5qjcJeo1VvK3jO/HpHN66pi8SBNU7hIzPj5cc/8rayjdvJeuOWl8d5xKXqQxKneJOe7O/HX1e/JLPy55nV0j8hkqd4lZ7s6b63Zx/ytrWLZlH91CJf9VlbyIyl1in7szb+0u7n91DW+HSv6W8/vz1bN76mIoabdOpNz1UyJRycwYc0o+z9x8Hr//9jAKctK482/vMf7eOSzcsDvoeCJRT+UuUe3Ykv+/bw8jNSmB63+3hHc+3Bd0NJGopnKXmGBmjD0ln6ennEtuZgrXPr6YNTv2Bx1LJGqp3CWmFGSn8eT1w0lOTOCbv13Elt2Hgo4kEpVU7hJzenfO4Mnrh3Okpo5vTl/EzsqqoCOJRB2Vu8SkAYVZ/O66c9h14AiTpy9m36HqoCOJRBWVu8Sss3p14tFrSti46yDXPr6Eg0dqgo4kEjVU7hLTRvbP48Gvn8W7ZRVMeaKUqqO1QUcSiQoqd4l5l5xeyD1fHsz8dbu57em3qamtCzqSSODCKnczm25mC8zsribu72RmL5pZqZk9EtmIIs378tk9uPuKQcx6fwc//uu71NWd/CuvRaJJs+VuZhOBRHcfARSZWXEjwyYDfwhdFptlZmFdHisSSdeN7Mv3Lyzmr8s+4j///j5BTK0hEi2SwhgzDpgR2p4FjALWNhizGzjdzDoCPYEPGz6JmU0BpgD06tWrhXFFju97FxRTcfgoj8/fRE56Mt+/8JSgI4kEIpzDMhlAWWh7D1DQyJg3gd7AbcAHoXGf4e7T3L3E3Uvy8/NbGFfk+MyMn1w2iK+c3YP/fXUtj725MehIIoEIZ8/9AJAe2s6k8V8IdwM3uXulmf0AuA6YFpmIIicmIcH4+cQz2F91lJ/OfJ/s9GS+cnaPoGOJnFTh7Lkvpf5QDMAQYFMjYzoBZ5hZIjAc0MFOCVRSYgIPfv0sRvXP48d/XcHLK7cHHUnkpAqn3J8FJpvZfcAkYKWZTW0w5mfU76lXALnA0xFNKdICqUmJPDL5bAb3yOHWp95m/rpdQUcSOWmaLXd3r6T+TdWFwHh3X+7udzUYs9jdT3P3THe/yN0PtE1ckROTkZrE49eeQ9+8DG74fSlvb9kbdCSRkyKs89zdfa+7z3B3/W0rMadjhxSeuH4Y+VmpXPv4Etbt1FTBEv90haq0C11CUwUnJhh3PPOuzoGXuKdyl3ajZ24HfnTxAJZs2svMFduCjiPSplTu0q5MKunJad2y+dmLH3C4WpOMSfxSuUu7kphg3H3FaWytqOI3b6wPOo5Im1G5S7szrG8ulw/uym/eWE/ZvsNBxxFpEyp3aZfumHAqAD978YOAk4i0DZW7tEvdO6Zz09h+zFyxjcUbPzcVkkjMU7lLu3XT2H50y0njP15YSa3mf5c4o3KXdis9JZE7JpzKyq2V/Ln0c7NUi8Q0lbu0a5cP7so5fTrx3y+vpuLw0aDjiESMyl3aNbP6UyP3HKrml681XINGJHap3KXdO717DleX9OR3b21ifbnmvJP4oHIXAf71CwNIT05k6sz3g44iEhEqdxEgPyuV2y4oZvbqcmav2hl0HJFWU7mLhHzrvD4U5WXwnzPfp7qmLug4Iq2ichcJSUlK4CeXD2LDroP8fsGmoOOItIrKXeQY4wd2YdyAfB54dS27DhwJOo5Ii6ncRRq467JBHD5ay//MWh10FJEWC6vczWy6mS0ws7uauP9mM5sT+njHzB6JbEyRk6d/l0y+dV4f/rjkQ94rqwg6jkiLNFvuZjYRSHT3EUCRmRU3HOPuD7v7OHcfB8wDHo14UpGT6LYLisntkMJPX3hfS/JJTApnz30cMCO0PQsY1dRAM+sOFLh7aSP3TTGzUjMrLS8vb0lWkZMmJz2ZH148gMWb9mhJPolJ4ZR7BlAW2t4DFBxn7C3Aw43d4e7T3L3E3Uvy8/NPLKVIACaV9GRQVy3JJ7EpnHI/AKSHtjObeoyZJQDjgTkRSSYSsPol+QaxtaKKR+ZqST6JLeGU+1I+PRQzBNjUxLjRwCLXAUqJI8OLOnOZluSTGBROuT8LTDaz+4BJwEozm9rIuIuBuZEMJxIN7rh0IO7w85dWBR1FJGzNlru7V1L/pupCYLy7L3f3z50S6e7/5u7PRD6iSLB6dOrATWP78cLyrVqST2JGWOe5u/ted5/h7tvbOpBINLppbD+6akk+iSFJQQcQiQUfL8l329Nvc+7PXiMpwVr0PIkJxr9NOJUJZ3SNcEKRz1K5i4TpisFd2V5xmHU7W76gx9LNe7nr2fcYVZxHdlpyBNOJfJbKXSRMZsaUMf1a9RwrPtrHlb+az0Oz13P7pQMjlEzk8zRxmMhJNLhHR750Vncem7+RD/ccCjqOxDGVu8hJ9qOLB2DAf7+sWSel7ajcRU6ybh3T+c7ovjy/fCvvfLgv6DgSp1TuIgG4eVx/8jJTmDpTs05K21C5iwQgMzWJH1w0gNLNe/nHe7p8RCJP5S4SkEklPTilIJOf/2MVR2o066RElspdJCBJiQn824RT2bz7EE8s2Bx0HIkzKneRAI0b0IXRxXk8+Npa9h6sDjqOxBGVu0jA7rzsVA4cqeHB19cGHUXiiMpdJGADC7OZVNKTJxZsZuOug0HHkTihcheJAj/4wimkJCXw85c+CDqKxAmVu0gU6JKVxk1j+/Hyyh0s2rA76DgSB1TuIlHihtFFFGan8V8vfkCd5oyXVlK5i0SJ9JREfnTxAFZ8VMHzy7cGHUdinMpdJIp86azunN49m3v+sYqqo7qwSVourHI3s+lmtsDMPrd2aoNxD5nZFZGJJtL+JCQYd04YxNaKKqa/uTHoOBLDmi13M5sIJLr7CKDIzIqbGDcaKHT3FyKcUaRdGdGvMxeeWsBDs9dRvv9I0HEkRoWz5z4OmBHangWMajjAzJKBR4FNZnZVY09iZlPMrNTMSsvLy1sYV6R9uGPCQI7U1HH/q2uCjiIxKpxyzwDKQtt7gIJGxlwDvA/cAwwzs1sbDnD3ae5e4u4l+fn5Lc0r0i70y8/kG8N78cfFW1izY3/QcSQGhVPuB4D00HZmE485C5jm7tuBJ4HxkYkn0n5978JTyEhN4v+9qAub5MSFU+5L+fRQzBBgUyNj1gFFoe0SQFPcibRSbkYKt57fnzmry5m3Vocy5cSEU+7PApPN7D5gErDSzKY2GDMdGG9mc4HvAvdGNqZI+/St8/rQMzed//r7B9TqwiY5Ac2Wu7tXUv+m6kJgvLsvd/e7GozZ7+5fdfcx7j7C3csaey4ROTGpSYn8+JKBrNq+n78s/TDoOBJDwjrP3d33uvuM0DF1ETmJLjujK0N7deTeWWs4eKQm6DgSI3SFqkiUMzPuvGwQ5fuP8Mgb64OOIzFC5S4SA87u3YnLBndl2rwNlO07HHQciQEqd5EYcfslA6lzOP/eOfzzU8uYvWonNbV1QceSKJUUdAARCU/P3A48d8tInlq0hRdWbGXmim3kZaZwxZBufHloD07rlo2ZBR1TooS5n/zTq0pKSry0tPSkv65IvKiuqWP26p38bVkZr6/aSXVtHcVdMpk4tAdfPKsbXXPSm38SiTlmttTdS8Iaq3IXiW37DlXz93e38cyyMpZu3osZjCjqzMShPbjk9EIyU/UHerxQuYu0U5t3H+Rvb5fxzLIytuw5RFpyAhefVsjEoT0Y2a8zSYl6my2WqdxF2jl3Z9mWvfx1WRkzl2+lsqqG/KxUrhrSjf5dMlv8vGZwXr88euZ2iGBaCZfKXUQ+caSmltmrdvLMsjJmr97J0drW/cynJyfyw4sHcO15fUhM0Bu4J5PKXUQadeBIDfurjrb88VU1/OylVby+aidDe3Xknq8Mpn+XrAgmlONRuYtIm3F3nntnK//xwkoOHqnltgv6c+PYfiTreH6bO5Fy17+GiJwQM+OLZ3XnlR+M5aLTCrh31hqu/NV83iurCDqaHEPlLiItkpeZyq//aSiPTD6b3QeOcNWv5/OLf6yi6mht0NEElbuItNLFpxXyyr+M5ctDu/PwnPVMeGAeSzbtCTpWu6dyF5FWy+mQzD1fGcIT1w+juraOSY8s4O7n3tMUxQFSuYtIxIwuzufl74/hWyP68PuFm/nC/XOZu0ZLBAZB5S4iEZWRmsS/X3kaf75xBKnJCVzz2GJ++OflVBxq+SmYcuLCKnczm25mC8zsribuTzKzLWY2J/RxRmRjikisKemTy4u3jeaW8f3429tlXHj/G/zjPS3mdrI0W+5mNhFIdPcRQJGZFTcybDDwtLuPC328G+mgIhJ70pIT+dHFA3nulpHkZ6Zy05NLeXiOVpM6GcLZcx8HzAhtzwJGNTLmXOByM1sc2svXNHQi8onTu+fw3D+PZPyAfH7zxnq90XoShFPuGUBZaHsPUNDImCXAhe4+DEgGJjQcYGZTzKzUzErLy/UGi0h7k5yYwK0XFFNx+CgzSj8MOk7cC6fcDwAfz/yf2cRjVrj7ttB2KfC5QzfuPs3dS9y9JD8/v0VhRSS2De3ViZLenZj+5kYtEdjGwin3pXx6KGYIsKmRMU+Y2RAzSwS+CCyPTDwRiTdTxhTx0d7DvKg3V9tUOOX+LDDZzO4DJgErzWxqgzE/BZ4A3gEWuPurkY0pIvHiwlMLKMrPYNrc9QQxcWF70Wy5u3sl9W+qLgTGu/tyd7+rwZj33H2wu5/h7ne2TVQRiQcJCcYNo4t4r6ySBRt2Bx0nboV1nru773X3Ge6uv6NEpNW+dFZ38jJTmDZ3Q9BR4pauUBWRky4tOZFvjejDnNXlrN6+P+g4cUnlLiKB+Oa5vUlPTuTRedp7bwsqdxEJRKeMFK4+pyfPvVPG9oqqoOPEHZW7iATm+lF9qa1zHn9rY9BR4o7KXUQC0zO3A5ee0ZWnFm5p1cLd8nkqdxEJ1I1jith/pIY/LdGUBJGkcheRQA3u0ZHhfXN57M2NHNWUBBGjcheRwN04toitFVXMXLE16ChxQ+UuIoEbd0oXirtkMm3uRk1JECEqdxEJXEKCccOYIj7YVsmb63YFHScuqNxFJCpcdWY3umSlakqCCFG5i0hUSE1K5NqRfZi3dhfvb60MOk7MU7mLSNT4xvDeZKRoSoJIULmLSNTISU/ma8N68cLyrWzddzjoODFN5S4iUeW6kX1w4LE3NSVBa6jcRSSq9OjUgcsHd+XpxVuoOKwpCVpK5S4iUWfKmCIOVtfy9OItQUeJWSp3EYk6p3XLYVT/PB6fv5HqGk1J0BJhlbuZTTezBWZ2VzPjCszs7chEE5H27IYxReyoPMLzyzUlQUs0W+5mNhFIdPcRQJGZFR9n+L1AeqTCiUj7NaY4j4GFWTw6d4OmJGiBcPbcxwEzQtuzgFGNDTKz84GDgBbRFpFWMzOmjCli9Y79zFlTHnScmBNOuWcAZaHtPUBBwwFmlgL8BLi9qScxsylmVmpmpeXl+ocSkeZdMaQbhdlpTHtDFzWdqHDK/QCfHmrJbOIxtwMPufu+pp7E3ae5e4m7l+Tn5594UhFpd5ITE/j2qD4s2LCbdz+qCDpOTAmn3Jfy6aGYIcCmRsZcCNxiZnOAM83stxFJJyLt3teH9SIrNYlpmpLghIRT7s8Ck83sPmASsNLMph47wN3HuPs4dx8HvOPu34l8VBFpj7LSkvmn4b148d1tfLjnUNBxYkaz5e7uldS/qboQGO/uy929yVMiQwUvIhIx147sgwHTNSVB2MI6z93d97r7DHfXmTAictJ1zUnnyjO7MaP0Q/Ydqg46TkzQFaoiEhOmjCniUHUtU//+AVVHa4OOE/VU7iISEwYWZnPjmCL+svQjJjw4j6Wb9wYdKaqp3EUkZtwx4VSevH44R47W8ZXfvMXUme9zuFp78Y1RuYtITBlVnMfL/zKGbwzvxW/f3MiEB+exZNOeoGNFHZW7iMSczNQkpn7xDJ76znCO1tYx6ZEF/PQF7cUfS+UuIjHrvP55vPz9MUw+tzePzd/IpQ/MZfFG7cWDyl1EYlxGahI/vep0nr7hXGrduXraAv79+ZUcqq4JOlqgVO4iEhdG9OvMy98fw7dG9OF3b23ikv+dx8INu4OOFRiVu4jEjQ4pSfz7lafxxynnYgZfm7aQu597j4NH2t9evMpdROLOuUWdeel7o7luZB9+v3Azlzwwl7fW7wo61kmlcheRuNQhJYm7rziNGTeOINGMf3p0EXc9+2672YtXuYtIXDunTy4vfW8M14/qyx8WbeH2Z94NOtJJoXIXkbiXnpLITy4fxJQxRfx9xdZ2MXWwyl1E2o3rzutLYoK1i6mDVe4i0m4U5qRx5ZDu7WLqYJW7iLQrN4zpy6HqWv6waEvQUdqUyl1E2pWBhdmMOSWf3721iSM18TsXTcTK3cxyzewiM8uL1HOKiLSFKaOLKN9/hOfe2Rp0lDYTVrmb2XQzW2Bmja6damadgJnAMGC2meVHMKOISESN7N+ZU7tm8+jcDbh70HHaRLPlbmYTgUR3HwEUmVlxI8MGAz9w9/8CXgaGRjamiEjkmBlTxvRl7c4DzFlTHnScNhHOnvs4YEZoexYwquEAd3/D3Rea2Rjq994XRCyhiEgbuHxwNwqz03h07oago7SJcMo9AygLbe8BChobZGYGXA3sBY42cv8UMys1s9Ly8vj8TSkisSM5MYFvj+rDW+t3815ZRdBxIi6ccj8ApIe2M5t6jNe7BVgBXNnI/dPcvcTdS/LzdUheRIL3tWG9yExN4tF58bf3Hk65L+XTQzFDgE0NB5jZj83smtCXHYF9EUknItKGstOS+fqwnsxcsY2yfYeDjhNR4ZT7s8BkM7sPmASsNLOpDcZMC42ZCyRSf2xeRCTqXTeyLwY8HmdTEiQ1N8DdK81sHHARcI+7bweWNxizN3S/iEhM6dYxncsHd+XpxVu49YJictKTg44UEWGd5+7ue919RqjYRUTiyndGF3GwupY/Lo6fKQk0/YCItHund89hZP/OPD5/E9U1dUHHiQiVu4gIcMPoIrZXVjFzRXxMSaByFxEBxp6Sz4CCLKbFyZQEKncREeqnJLh+dF9Wbd/Pm+tifzFtlbuISMhVZ3YjPyuVaXEwJYHKXUQkJDUpkWvP68O8tbv4YFtl0HFaReUuInKMbwzvRYeUxJifkkDlLiJyjI4dUphU0pPn39nKtorYnZJA5S4i0sD1o/pS587v3toUdJQWU7mLiDTQM7cDl57RlacWbmF/1edmMI8JKncRkUZMGV3E/iM1/GnJh0FHaRGVu4hII4b07Miwvrk8Pn8TR2tjb0oClbuISBOmjC6ibN9hXnx3W9BRTpjKXUSkCecP7EJRfgaPzou9KQlU7iIiTUhIMG4YXcR7ZZUs2LA76DgnROUuInIcXzqrO3mZKTwaY1MSqNxFRI4jLTmRa0b0Yfbqctbs2B90nLCp3EVEmvHNc3uTlpzAb2NoSoKwyt3MppvZAjO7q4n7c8zsJTObZWZ/M7OUyMYUEQlObkYKXz27J8++vZWdlVVBxwlLswtkm9lEINHdR5jZY2ZW7O5rGwz7BnCfu79iZg8DlwDPt0FeEZFAXD+qL08u2syVv5pPVlqz1dmkcQPyufOyQRFM1rhwEo4DZoS2ZwGjgM+Uu7s/dMyX+cDOSIQTEYkWffIyuHPCqSzbsrdVz1OQnRahRMcXTrlnAGWh7T3A0KYGmtkIoJO7L2zkvinAFIBevXqdeFIRkYB9Z3RR0BHCFs4x9wNAemg7s6nHmFku8Evg243d7+7T3L3E3Uvy8/NbklVERMIUTrkvpf5QDMAQYFPDAaE3UP8M3OHumyOWTkREWiSccn8WmGxm9wGTgJVmNrXBmOupP1xzp5nNMbOrI5xTREROQLPH3N290szGARcB97j7dmB5gzEPAw+3SUIRETlhYZ3P4+57+fSMGRERiXK6QlVEJA6p3EVE4pDKXUQkDlkQE9CbWTnQmlMm84BdEYoTSdGaC5StpZStZZStZZrL1tvdw7pQKJByby0zK3X3kqBzNBStuUDZWkrZWkbZWiaS2XRYRkQkDqncRUTiUKyW+7SgAzQhWnOBsrWUsrWMsrVMxLLF5DF3ERE5vljdcxcRkeNQuYuIxKGoKPfm1mhtaky4t0VDtrZYZzaS37fQ7QVm9nZrc7VRtofM7IpoymZmnczsRTMrNbNHAs5WYGbzjvk62cxeMLP5ZtboGgsBZusVmj32dTObZmYWLdmOuf10M3ultbnaKNsLZnZmc68beLnbMWu0AkVmVhzOmHBvi5ZsfLrO7BeA7dSvMxst2T52L58uzBI12cxsNFDo7i9EWbbJwB9C5yVnmVmrzk9uRbZOwP9Rv2rax24Flrr7SOArZpYVRdluBG529/OBnsAZUZSN0C+b+4Dk1uRqo2zfANa7+zvNvXbg5U7ja7SGMybc26Iim7s/5O4f7wlEYp3ZiGUDMLPzgYPU/+JprYhlM7Nk4FFgk5ldFU3ZgN3A6WbWkfqS+jCgbLXA1UBlE+PmAq29MCZi2dz9Tnf/IPRlZ1p/tWjEsoVcB8xuZaaIZ7P61e7+B9hrZuObe+FoKPeGa7QWhDkm3NuiJRtw/HVmg8pm9YeIfgLc3spMEc8GXAO8D9wDDDOzW6Mo25tAb+A24IPQ7Sc9m7tXuntFC54rqGwAWP2iPivdfWu0ZDOzzsA3qf8rNhIi+X37F+pXvHsEuMbMrjzeC0dDuYezRmtjY8K9LVpp7fz7AAABfklEQVSyNbvObIDZbgcecvd9EcgV6WxnAdNCi8Q8CTS7x3ISs90N3OTuPwVWUb/HF0S2lj5XUNkwsyLgh8D3W5kr0tl+Tv1yoUcjkCvS2c4Cfh36WZhB/R5/k6Kh3Jtdo7WJMeHeFhXZLPLrzEby+3YhcIuZzQHONLPfRlG2dcDHS86X0LoJ5yKdrRNwhpklAsOB1l400tJsLX2uQLKFjic/DXy7qb36oLIBY4FfHPOz0HBJ0SCzndjPgrsH+gFkU79s333U/2k7BJjazJiccG+Lomw3A3uBOaGPq6MlW4PHzImyf9Ms6n8pzgUWAN2jKNswYCX1e16vAJlBZGvs3476w0UrgQeAJdS/YRct2X4BbDvmZ2FstGSLlp+FJr5v3YAXgfmh/9+yjvvarQ0fiQ/q94AmUX9GRNhjwr0tWrJF8/dN2WI7WxPjuoXGtWonpy2yRfP3LV6yafoBEZE4FA3H3EVEJMJU7iIicUjlLiISh1TuIiJxSOUuIhKH/j8QQ46jCFayxwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "plt.plot(threshold, scores)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "从上面我们可以看出，随着阈值的增加，模型的效果越来越差。阈值在0-0.002已经很好了，但是此时过滤的特征确是很少，说明原始数据的特征都对模型有很大影响。不过我们可以牺牲一点模型的准确度，降低特征的数量，从而使模型跑的更快。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-12-13T07:28:19.049768Z",
     "start_time": "2019-12-13T07:28:12.211657Z"
    }
   },
   "outputs": [],
   "source": [
    "\"\"\"使用阈值为0.001看看过滤后的特征数量\"\"\"\n",
    "X_embedded = SelectFromModel(rfc, threshold=0.001).fit_transform(X, y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-12-13T07:29:00.428095Z",
     "start_time": "2019-12-13T07:29:00.420337Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(42000, 282)"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_embedded.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-12-13T07:29:49.835779Z",
     "start_time": "2019-12-13T07:29:25.715978Z"
    }
   },
   "outputs": [],
   "source": [
    "score = cross_val_score(rfc, X_embedded, y, cv=5).mean()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-12-13T07:31:35.352088Z",
     "start_time": "2019-12-13T07:31:07.221329Z"
    }
   },
   "outputs": [],
   "source": [
    "\"\"\"与没过滤之前的做对比\"\"\"\n",
    "pre_score = cross_val_score(rfc, X, y, cv=5).mean()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-12-13T07:31:35.707778Z",
     "start_time": "2019-12-13T07:31:35.695523Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.9381432010052206 0.9366670499328583\n"
     ]
    }
   ],
   "source": [
    "print(pre_score, score)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-12-13T07:41:29.234045Z",
     "start_time": "2019-12-13T07:33:54.646559Z"
    }
   },
   "outputs": [],
   "source": [
    "\"\"\"寻找更好的阈值\"\"\"\n",
    "pieces = 15\n",
    "scores = []\n",
    "for i in np.linspace(0, 0.002, pieces):\n",
    "    X_embedded = SelectFromModel(rfc, threshold=i).fit_transform(X, y)\n",
    "    score = cross_val_score(rfc, X_embedded, y, cv=5)\n",
    "    scores.append(score.mean())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-12-13T07:41:29.882976Z",
     "start_time": "2019-12-13T07:41:29.505828Z"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYAAAAD6CAYAAACoCZCsAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJzt3Xl8VPW5+PHPk8kGIQtZSAh7IGEJEJawCLIpCFbBSluxWrcWtUqv19prvb31/ryttL3aSldtxaUXtVoQ9w1QZFNBCPu+L2ELAQIhhOzP748MFWECkzAzZ5J53q8Xr9eZM99zznPOK5xnzvd8F1FVjDHGhJ4wpwMwxhjjDEsAxhgToiwBGGNMiLIEYIwxIcoSgDHGhChLAMYYE6IsARhjTIiyBGCMMSHKEoAxxoSocKcDuJjk5GTt2LGj02EYY0yjsnLlyqOqmnKpckGdADp27EheXp7TYRhjTKMiInu9KWdVQMYYE6IsARhjTIiyBGCMMSHKEoAxxoQoSwDGGBOiLAEYY0yIsgRgjDEhyhJAPRQUl/Hy0j0UFJc5HYoxxly2oO4IFmx+O3crs1fu57F3N3JVt1ZMGtCeUV1TCHdZHjXGND6WALxUXFbJB+sO8Y1eaXRMiuH1lfv5ZHMerWKj+Hb/ttyU246OyTFOh2mMMV6zBOCl99Ye5ExlNfcO70xOuwQeGpPFgq2FzFyxj2cX7+KZhTsZnJHIzQPaM65nGtERLqdDNsaYi7IE4KWZK/LplhZL77bxAIS7whjTI5UxPVIpKC5j9sr9zFyRz4Mz1xD3Tjg39m3DpAHt6ZEe53DkxhjjmSUAL2w6WMy6/Sd5bHwPROSC71Pjopkyqgv3jejMsl3HmJmXz2sr8pmxdC+928YzaUA7JuSkExsd4UD0xhjjmSUAL8zKyycyPIwb+7a5aLmwMGFIl2SGdEnmF6UVvL36AP9ckc/P39rA1Pc3c13v1tw8oB39O7T0mEiMMSaQvEoAIvIC0AP4QFWnevi+E/AXIA5Yrqo/Oee7VGCOqvb1Zl/BpqyymjdX7WdcdhoJzSO93i6heSR3Du3EHUM6sm7/Sf65Ip931xxg9sr9ZKTEcPOAdkzs15bkFlF+jN4YY+p2yfaLIjIRcKnqFUCGiGR6KPYE8LiqDgPaisjIc777HdCsHvsKKnM3Hqa4rIpJA9o1aHsRIaddAr+Z2IvlPx/Nk9/uTcvmkfz6wy0M/vV87ntlJQu3HqG6Rn0cuTHGXJw3TwAjgVnu5XnAlcD288pkAavcy0eAeAARuQo4DRyux76Cyj+X59MusRlXZCRd9r5iosK5KbcdN+W2Y3vBKWauyOeNVfv5aMNhruvdmqdv6eeDiI0xxjve9GCKAQ64l48DqR7KzAYeE5HxwDhgvohEAv8N/Gd99iUi94hInojkFRYWencWfrL32GmW7jrGpNx2hIX5ts4+MzWWR6/vwbL/uprvDmzPnA2HKTpd4dNjGGPMxXiTAEpwV+EALTxt467L/wiYDMxQ1RJqb/zPqOqJeu5ruqrmqmpuSsolp7T0q1l5+YQJfLt/w6p/vBEV7uLmAe2orlHmbznit+MYY8z5vEkAK6mtqgHIAfbUUW4N0B6Y5v48GpgiIguBPiLyfD325biq6hpez9vPqK6tSIuP9uuxereNp3V8NHM3Hr50YWOM8RFv3gG8DSwRkXTgWuBmEZmqqo+eV+5hYJqqlgKo6vCzX4jIQlWdLCJx5+1rsE/Owg8Wbi3kyKlybmrgy9/6EBHGZqfx2vJ9lFZU0TzSWucaY/zvkk8AqlpM7cvbZcAoVV3r4eaPqj6mqi/XsY+RdezrZIMj97OZefkkt4jiqm6tAnK8a7JTKa+qYdFWZ997GGNCh1fDWKpqkarOUtXLrqPw5b785UhxGZ9uOcK3+7clIkAjfQ7smEjL5hFWDWSMCRgbx9iD2av2U12jDW773xDhrjBGd09l/pYjVFTVBOy4xpjQZQngPKrKzBX5DOyUSKcAD+88NjuNU2VVLNt1LKDHNcaEJksA51m26zh7j5VycwB//Z91ZWYyzSNdVg1kjAkISwDnmZWXT2x0ONf2bB3wY0dHuBjZNYV5mwqosaEhjDF+ZgngHCdLK/lw/SG+2acNzSKdmdBlbHYahafKWZ1f5MjxjTGhwxLAOd5Ze4DyqpqAvvw936hurYhwCXM3FjgWgzEmNFgCcFNVXlueT3Z6HD3bxDsWR1x0BEM6JzN342FUrRrIGOM/lgDcNhwoZvOhYkde/p5vbHYae4+VsrXglNOhGGOaMEsAbjPz9hEVHsaEPhef9SsQxvRIRQTmbLDWQMYY/7EEAJypqOad1Qe5rldr4ps5P29vSmwU/du3tPcAxhi/sgQAfLj+EKfKqwIy8Ju3xvVMY/OhYvKPlzodijGmibIEAMxckU/HpOYM6pTodCj/MjY7DcA6hRlj/CbkE8CuwhKW7znOpAHtEfHtrF+Xo11ic7q3jrP3AMYYvwn5BDAzLx9XmPCt/s6//D3f2OxUVu4rovBUudOhGGOaoJBOAJXVNbyxcj9Xd2tFq1j/zvrVEON6pqEKH2+yl8HGGN8L6QQwf/MRjpZUONrz92K6psbSIam5vQcwxviFXxOAiCSKyBgRSfbncRpq5op9pMZFMSLL2cnn63J2qsgvdh6luKzS6XCMMU2MVwlARF4QkaUicsFUkO7vO4nIByKyRESecq9rCbwPDAQWiEiKp3JOOXTyDIu2FfKd/u0ID9CsXw0xNjuVymplwZYjTodijGliLnnnE5GJgEtVrwAyRCTTQ7EngMdVdRjQVkRGAr2Bh1T1V8BcoF8d5RwxO28/NQo35QZn9c9Zfdu1JCU2yqqBjDE+581P35HALPfyPOBKD2WygFXu5SNAvKouUtVlIjKc2qeApZ7Knb8jEblHRPJEJK+w0D8TpNfUKDPz8hnSOYn2Sc39cgxfCQsTrumRysKthZRVVjsdjjGmCfEmAcQAB9zLx4FUD2VmA4+JyHhgHDAfQGob1k8CioDKusqdS1Wnq2ququampPinbv6LncfYX3QmaF/+nm9sdhqlFdV8tv2o06EYY5oQbxJACdDMvdzC0zaqOhX4CJgMzFDVEvd6VdUpwDpgQl3lAu2fK/YR3yziX71tg93gjCRio8OtGsgY41PeJICVfFXtkwPsqaPcGqA9MA1ARB4Rkdvd3yUAJzyVC7Si0xXM21jAjX3bEB3hzKxf9RUZHsbV3VrxyeYCqqprnA7HGNNEeJMA3gZuE5FpwE3ARhGZ6qHcw8A0VT07etl093aLARe17w88lQuot1YfoKLa2Vm/GmJsdhpFpZUs33Pc6VCMMU1E+KUKqGqxu7XOGOBJVT0MrPVQ7rHzPhe5t7louUBSVWauyCenbTzdW8c5FUaDjOiaQlR4GPM2FjCkc1B2qzDGNDJeNYBX1SJVneW++Tdaa/JPsLXgFJMGtHc6lHprHhnO8KwU5tlUkcYYHwneHlB+MCsvn2YRLsbntHY6lAYZm53GwZNlrD9w0ulQjDFNQMgkgNPlVby75iDX925NbLTzs341xOjurXCFiQ0RbYzxiZBJAB+sO8TpiupG9/L3XAnNIxmckWjNQY0xPhEyCeCfK/bROSWG/h1aOh3KZRmbncbOwtPsOOJIFwpjTBMSEglge8EpVu07wc1BNutXQ1zTw6aKNMb4RkgkgJkr8olwCTf2C75Zv+orLT6anHYJzGtiCWD2yv18d/oyfjd3Kyv2HLcOb8YEwCX7ATR25VXVvLn6AGN6pJLcIsrpcHxibHYqT87ZysETZ0hPaHbpDYLcrLx8fjp7Henx0Szfc5y/LNhBbFQ4Q7skM6JrCsOzUmjTBM7TmGDT5BPAJ5uOcPx0RdAP+1wf47LTeHLOVuZtPMydQzs5Hc5leWfNAR55Yx3DMpN57vZcyqtq+GLHURZtK2TxtkLmuJ90urRqwYis2mQwqFNioxnGw5hg1uQTwD9X7CM9PpphmcE561dDZKS0ILNVC+ZuLGjUCWDOhkM8NGstAzsmMv22XKIjXERHuLi2V2uu7dUaVWXHkRIWbStk0bZCXl62lxc+201UeBiDM5IYnpXCiKwUOqfENPp3O8Y4oUkngP1FpXy24ygPXJWJK6xp3SDGZqfx10U7OX66gsSYSKfDqbf5mwv4t9dW06ddAi/eOYBmkRf+ohcRMlNjyUyNZfKwDM5UVPPl7mP/SgiPv7+Jx4E2Cc3+lQyGdklqtP08jAm0Jp0AXs/bD8B3cts6HInvjc1O4y8LdvDJ5oJGV721eFsh972yiu6t4/j7XQOIifLuz7BZpIuRXVsxsmsrAPKPl7J4eyGLthby3tqDvLZ8H+FhQr/2LRnRtTYh9GgdR1gTS/7G+EqTTQDVNcrreflc2SWZti2De9avhujZJo42Cc2Yt/Fwo0oAS3ce456X8+jcqgUvfX8gcZfxa71dYnNuHdSBWwd1oLK6hlV7i2oTwrZCfjt3K7+du5WkmEgeuDqTO4Z09N1JGNNENNkEsGR7IQdPlvHz63o4HYpfiAjXZKfyjy/3cbq8yutf0U5aufc4P5ixgnYtm/PKDwaS0Nx3VVcRrjAGZSQxKCOJh8d2o/BUOZ/tKOTVL/cx9YNNXNWtFe0Sm94PAWMuR5PtBzBzRT6JMZGM7tHK6VD8Zmx2GhVVNSza5p+5k31pbf4J7nxxBWlx0fzj7kEk+blJbkpsFDf2bcufv9uPMBGmfbzNr8czpjFqkgngaEk5n2wuYGLfNkSFN93mggM6JpIYExn0vYI3HjzJ7S8uJyEmgn/cPYhWsdEBO3ZafDR3De3E22sOsPlQccCOa0xj4NcEICKJIjJGRAI6g8nhk2V0TmnRqAd+84YrTBjdvRWfbj5CRVVw9pzdVnCK215YTkyki1cnD6Z1fOA7dN03ojOxUeE8OWdLwI9tTDDzKgGIyAsislREHq3j+04i8oGILBGRp9zrWgLvAwOBBSKSIiItReRDEckTkWd9dhbn6dkmnjkPDiczNdZfhwgaY7PTOFVexRc7jzodygV2FZZwy3NfEh4mvHr3YMfq4OObR3D/qC4s2FrIl7uOORKDMcHokglARCYCLlW9AsgQkUwPxZ4AHlfVYUBb9xSSvYGHVPVXwFygH3Ab8A9VzQViRSTXR+cRsoZ2SSYm0sXcjQVOh/I1+46VcstzXwLKq3cPpmNyjKPx3DmkI2lx0fzvnC02o5oxbt48AYwEZrmX5wFXeiiTBaxyLx8B4lV1kaouE5Hh1D4FLAWOAT1FJAFoB+RfRuwGiI5wMbJbKz7eVEB1TXDc2A6cOMN3n1tGWVU1r0weRJdWLZwOiegIFw+OzmT1vhPM2xRcydIYp3iTAGKAA+7l40CqhzKzgcdEZDwwDpgPILX98ycBRUAl8BnQAXgA2Oze39eIyD3uKqK8wsLgb90SDMZmp3G0pJzV+4qcDoWC4jJueW4ZxWWVvPz9QXRLi3M6pH/5dv+2dE6J4bdzt9poo8bgXQIoAc6+uWvhaRtVnQp8BEwGZqhqiXu9quoUYB0wAXgM+KGq/hLYAtzlYV/TVTVXVXNTUprO+D3+NKprCpGuMMeniiw8Vc4tzy3j6KlyZnx/IL3axjsaz/nCXWE8PLYrO46U8OaqA5fewJgmzpsEsJKvqn1ygD11lFsDtAemAYjIIyJyu/u7BOAE0BLoJSIuYBAQHHUWjVxsdARDuyQxd9Nhx+q3j5+u4HvPf8nBE2X8/a6B9GsfnDOvjc1Oo0+7BH7/yTbKKqudDscYR3mTAN4GbhORacBNwEYRmeqh3MPANFUtdX+e7t5uMeCi9v3Bb9zrTwKJwGuXGb9xG5udRv7xM2w+dCrgxz55ppLbXviS3cdO8/wduQzslBjwGLwlIjwyrhuHTpbx0tI9TodjjKMuOX6Aqha7W/WMAZ5U1cPAWg/lHjvvc5F7m3MtB7IbHK2p0+geqYS9tZ65Gw/TIz1w9e4l5VXc8eJythWcYvrtuQztEtAuHw1yReckRmSl8PSCnUwa0J74ZjZ6qAlNXvUDUNUiVZ3lvvmbIJTcIorcDokB7RVcWlHF9/++gg0HTvL0Lf0Y1bXxDLvx03FdOXmmkmcX7XQ6FGMc0ySHgghV12SnsuXwKfYeO+33Y5VVVjN5Rh55e4/zh5v7cE12mt+P6UvZ6fHc0CedFz/fTUFxmdPhGOMISwBNyFj3TdjfTwHlVdX88JWVLN11jKduyuH63ul+PZ6//GRMV6prlD/O3+50KMY4whJAE9IusTnZ6XF+7RVceKqcH726moVbC/nNjb24sW/jnWynfVJzbhnYnpkr8tlVWOJ0OMYEnCWAJmZsdhqr9hVx5JTvqjXOVFTzzpoD3Pn35Qz+zXw+3lTAL2/I5uaB7X12DKf86KpMosLDeGqeDRdtQk/wzyJi6mVsdhrTPt7Gx5sKuHVQhwbvp6ZGWbbrGG+uPsCcDYcpKa+idXw09wzPYGLfNk1moL2U2CgmD8vgT/O3c+/+E/Rum+B0SMYEjCWAJiYrtQUdk5ozZ8PhBiWA7QWneHP1Ad5efYBDJ8toERXOtT3TuLFfGwZ3SmqS8+vePawTryzbyxNztvCPyYOdDseYgLEE0MSICGN7pvHCkt2cPFPpVRv3wlPlvLv2IG+t3s+GA8W4woThmcn87BvdGdM9lWaRTXdSHajtSf2jUV345fubWLK9kGGZNgSJCQ2WAJqgsdlpPLtoFwu2HOGbfdt4LHOmopp5mw7z1uoDLNl+lOoapWebOP77+h5MyEknJda/UzYGm1sHt+eFz3bzxJwtDO2c3CSfdIw5nyWAJqhP2wRaxUYxd+PhryUAT/X66U2wXr8hosJdPDQmi5+8vpYP1h9ifE7jbNrqyawV+Xyw/hB/v3OAJTbzNZYAmqCwMOGa7FTeWHmAsspq9h0v5c1VB3hnTejU6zfEN/u2YfriXTw1byvjeqYR4Wr8jeTKKqt5cu4WjpZUsGpfEbkdg3ecJhN4lgCaqHHZrXll2T5GT1vE/qIzIVev3xCuMOGn47rygxl5/HNFPrcNbngrqmAxc0U+R0sqCBN4d+1BSwDmaywBNFGDMhLp0qoFzSJcfH9oJ8aHYL1+Q1zVrRUDOrbkT/O3861+bWge2Xj/i1RU1fDsop3kdmhJalw0H64/xP+7vgfhTeDJxviG/SU0URGuMD55aATv/duVfP/KTnbz95KI8J/XdqPwVDkvfrbb6XAuy9trDnDwZBlTRnVhfE46R0sq+GLnMafDMkHEEoAx5+nfIZHR3VN5dtEuik5XOB1Og1TXKH9buJMereMY2TWFkV1TiI0K5921B50OzQQRSwDGePDTcV05XVHF0wt2OB1Kg3y04RC7jp5myqguiAjRES7G9kxj7obDNhOa+RdLAMZ4kJUay8R+bXlp6V4OnDjjdDj1oqo8vWAnGSkxjOv51TDdE3LSOVVexcKthQ5GZ4KJXxOAiCSKyBgRCf5poow5z4/HZIHA7z9uXAPFLdxayOZDxdw3ojOuc5r4DumcRHKLSN5de8DB6Eww8SoBiMgLIrJURB6t4/tOIvKBiCwRkafc61oC7wMDgQUikiIi94nIQve/NSLyrM/OxBgfa5PQjNsHd+DNVfvZVhD4uZYbQlX5y4IdtElodkEv8HBXGN/o1Zr5m49wqqzSoQhNMLlkAhCRiYBLVa8AMkQk00OxJ4DHVXUY0NY9h3Bv4CFV/RUwF+inqn9V1ZGqOhJYAjzno/Mwxi+mjOpCTGQ4T87Z6nQoXvly93FW7i3i3hEZHjuy3dAnnfKqGj7e5L85I0zj4c0TwEhglnt5HnClhzJZwCr38hEgXlUXqeoyERlO7VPA0rOFRaQNkKqqeefvSETuEZE8EckrLLS6SuOsljGR3Dsig082F5C357jT4VzS0wt2kNwiipty23n8vl/7lrRJaGatgQzgXQKIAc5WGh4HUj2UmQ08JiLjgXHAfAAREWASUASc+8w5Bfirp4Op6nRVzVXV3JQUG5XROO9sP4on5mxBVZ0Op05r80+wZPtRJg/rRHSE557eIsL4nHQ+236U4420iavxHW8SQAnQzL3cwtM2qjoV+AiYDMxQ1RL3elXVKcA6YAKAiIQBo4CFlxu8MYHQPDKcB67OZMWeIj7dcsTpcOr0zMIdxEWHc+ugi8/UNiEnnaoa5cP1hwIUmQlW3iSAlXxV7ZMD7Kmj3BqgPTANQEQeEZHb3d8lACfcy8OALzWYf0oZc56bB7SjY1Jznpyzleqa4PvT3V5wirkbC7hzSEdioy8+B0T31rF0adXCqoGMVwngbeA2EZkG3ARsFJGpHso9DExT1VL35+nu7RYDLmrfHwCMBRZfXtjGBFaEK4yfXNOVrQWneHt18DWjfGbhTppHurhraKdLlhURJuSks3z3cQ42sj4OxrcumQBUtZjaF8HLgFGqulZVL2gOqqqPqerL53wuUtUxqjpcVe8/+4tfVf9LVd/03SkYExjX9WpNzzZxTPt4G+VVwdObdt+xUt5de5BbBranZUykV9tMcM938P46ewoIZV71A3DfzGep6mF/B2RMsAoLEx4Z140DJ87wyrJ9TofzL39bvBOXCHcPz/B6m47JMfRuG2/VQCHOhoIwph6GZaYwtEsSTy/YERSdqQqKy5idt59v57YlNS66XttOyElnw4FidhWW+Ck6E+wsARhTT4+M68bx0xX8+sMtTofC80t2Ua3KD4d3rve243PSEfdEMSY0WQIwpp56t03ghyM689ryfbyybK9jcRSdruAfX+5jQk467ZOa13v71LhoBnVK5N21B4O6f4PxH0sAxjTAw2O7MrJrCv/z7ka+3OXMJCt//2IPpRXV3Dey/r/+z5qQ04ZdhafZeLDYh5GZxsISgDEN4AoT/nhzX9onNef+f6xif1HppTfyoZLyKv7v892MzU4lKzW2wfu5tmca4WHCe1YNFJIsARjTQPHNInju9lwqqmq456WVlFZUBezYryzbS3FZFfeP7HJZ+2kZE8nwrBTeXXuQmiDs4Gb8yxKAMZehc0oL/vTdvmw+XMzDs9cFpC69rLKa55fsZlhmMjntEi57fxNy0jl0soy8vUU+iM40JpYAjLlMo7q14pFx3fhg3SGeWbjT78eblZfP0ZJypoy6vF//Z43pkUp0RJhNFBOCLAEY4wP3Ds/ghj7p/G7eVj7x41j7ldU1PLtoF/07tGRQp0Sf7DMmKpzR3VP5cP1hKqtrfLJP0zhYAjDGB0SEJ77Vm+z0OB6cuYbtfppB7J01Bzlw4gw/ck/27isTctI5frqCz3cc9dk+TfCzBGCMj0RHuJh+Wy7RES7ufimPk6W+7SlcXaM8s3AH3VvHMbKrb+fKGNE1hbjocOsUFmIsARjjQ+kJzfjb9/rV/kp/bRVVPqxSmbvxMLsKTzNlVGef/voHiAp3Ma5nGvM2FlBWGTwD3Rn/sgRgjI/ldkzk8Rt6smT7Uf73I98MF6GqPL1gBxnJMVzbs7VP9nm+CTltKCmvYkEQT3pjfMsSgDF+cPPA9txxRQee/2w3b6zcf9n7W7itkI0Hi/nhyM64wnz76/+sKzonkdwiinfWWDVQqLAEYIyfPHp9D67ISOJnb61nTf6JS29wEc8s2EGbhGbc2LeNj6K7kCtMuL53az7deoTiIBjp1PifXxOAiCSKyBgRSfbncYwJRhGuMJ6+tR+tYqO456U8CorLGrSfL3cdY8WeIu4ZnkGEy7+/2cbnpFNRVcO8jf5rymqCh1d/TSLygogsFZELZgJzf99JRD4QkSUi8pR7XUvgfWAgsEBEUs4p/4yIjPdB/MYEtcSYSJ67PZeS8irufXllg16wPr1wJ8ktIpk0oJ0fIvy6fu0TaNuymbUGChGXTAAiMhFwqeoVQIaIZHoo9gTwuKoOA9qKyEigN/CQqv4KmAv0c+9vGJCmqu/56ByMCWrdW8fx1HdyWJN/gp+/taFew0Ws33+SxdsK+cGVGURHuPwYZa2z8wV/vuMox0rK/X484yxvngBGArPcy/OAKz2UyQJWuZePAPGqukhVl4nIcGqfApaKSATwHLBHRG64rMiNaUSu7dWaB67O5I1V+3nx8z1eb/f0gh3ERofzvcHt/RfceSb0Sae6Rvlw/aGAHdM4w5sEEAOcHSTkOJDqocxs4DF3tc44YD6A1DZWngQUAZXA7cAm4ElgoIj82/k7EpF7RCRPRPIKCwvreTrGBK8Hr87kmh6p/OqDTXy2/dI9brcXnGLOxsPcOaQjsdERAYiwVre0OLJSW1g1UAjwJgGUAM3cyy08baOqU4GPgMnADFUtca9XVZ0CrAMmAH2B6e7J5V8BRnnY13RVzVXV3JQU3/Z2NMZJYWHCtEl96NKqBVNeXcWeo6cvWv6vi3bSLMLFXUM7BSjCr0zISWfFniIOnDgT8GObwPEmAazkq2qfHGBPHeXWAO2BaQAi8oiI3O7+LgE4AewAMtzrcgHn5tMzxgEtosJ5/vYBiMDdL+VRUu55DoH846W8s+YgtwxqT2JMZICjrG0NBNhEMU2cNwngbeA2EZkG3ARsFJGpHso9DExT1bNTI013b7cYcFH7/uAFYJR73f3A7y73BIxpbNonNefpW/qx6+hpfjxzjceJWJ5dvBOXCHcPy/CwB//rkBRDTrsE3rVOYU3aJROAqhZT+yJ4GTBKVdeq6gXNQVX1MVV9+ZzPRao6RlWHq+r97uqgU6r6Hfe6K1TVBiA3IWlol2Qeva47H28q4A+fbPvad0eKy5iVt59v9W9LWny0QxHWVgNtOlTMjiMljsVg/MurfgDum/ksd929McYH7hzSke/0b8ufPt3BB+u+anHz/Ge7qaqu4YcjnPn1f9b43q0RwV4GN2E2FIQxDhERpt7Yk77tE/iP19ey6WAxJ0oreGXZXibkpNMhKcbR+FrFRXNFRhLvrT0YkKkuTeBZAjDGQVHhLp79Xn/im0Vw90t5/P7jbZRWVHPfZU727isTctLZffQ0Gw4UOx2K8QNLAMY4rFVcNM/e1p/CknJmLN3LNT1S6ZoW63RYAFzbszURLrH5gpsoSwDGBIGcdgk8+a3etGwewQNXexptxRnxzSMYkZXCe2tsPc0tAAAS5ElEQVQPeWytZBo3SwDGBIlv9m3DykfH0LNNvNOhfM34nHQOF5exfM9xp0MxPmYJwJggEuanyV4ux5geqTSLcFlroCbIEoAx5qKaR4YzpkcqH60/RKUP5zg2zrMEYIy5pAk56RSVVno1iJ1pPCwBGGMuaXhWCvHNIqwaqImxBGCMuaTI8DCu7ZnGvI2HOVNR/1nNTHCyBGCM8cqEnHROV1Tz6ZYjTodifMQSgDHGK4MykmgVG8U7axpfp7D5mwsYM20RGw6cdDqUoGIJwBjjFVeYcF3v1izcWsjJM5VOh+O1bQWneOC11Ww/UsLkGXkcKS5zOqSgYQnAGOO1CTnpVFTXMHdj4xgY+ERpBXe/lEfzqHD+fucAissqufulPMoq7T0GWAIwxtRDn3YJtE9s3ihmCquqruFHr67m0Iky/va9/ozq1orfT+rD2v0neXj2OhvhFD8nABFJFJExIpLsz+MYYwJDRJiQk87nO45SeKrc6XAu6tcfbuGzHUeZemNP+ndoCcDY7DR+Oq4r7609yJ8/3eFwhM7zKgGIyAsislRELpgJzP19JxH5QESWiMhT7nUtgfeBgcACEUkRkXAR2SciC93/evnsTIwxATGhTzo1Ch+uP3Tpwg55PS+fFz/fzV1DO3JTbruvfXffiM5M7NuGaR9v46MgPodAuGQCEJGJgEtVrwAyRMTTUIVPAI+r6jCgrYiMBHoDD6nqr4C5QD/3utdUdaT733pfnYgxJjCyUmPplhYbtJ3CVu0r4udvbWBolyR+/o3uF3wvIvx6Yi/6tU/gx7PWhHTLIG+eAEYCs9zL84ArPZTJAla5l48A8aq6SFWXichwap8ClgKDgetFZLn7qSL8sqI3xjhifE46K/cWkX+81OlQvubwyTLufXklafHR/OW7/Qh3eb7FRUe4ePa2XJJiokK6ZZA3CSAGONvw9ziQ6qHMbOAxERkPjAPmA4iIAJOAIqASWAGMVtWBQATwjfN3JCL3iEieiOQVFhbW83SMMYEwIScdIKj6BJRVVnPvy3mUllfx3O25tIyJvGj5lNgonrs9N6RbBnmTAEqAZu7lFp62UdWpwEfAZGCGqpa416uqTgHWAROAdap6ttItD7igOklVp6tqrqrmpqSk1Pd8jDEB0C6xOUO7JPGHT7bz2vJ9ToeDqvJfb65n7f6T/H5SH69nVOuRHsfvJ/Vh3YHQbBnkTQJYyVfVPjnAnjrKrQHaA9MAROQREbnd/V0CcAJ4WURyRMQFfBNY28C4jTEO++v3+jO0SzI/e3M9v3xvE9UOzhj2/JLdvLn6AA+NyeKa7LR6bTs2O42Hx4ZmyyBvEsDbwG0iMg24CdgoIlM9lHsYmKaqZysFp7u3Wwy4qH1/8EvgZWqTxVJV/eRyT8AY44y46AheuCOXu4Z25MXPd/ODGSs4VRb4HsKLthXym482c23PNH40qkuD9nFuy6Bgbt3ka+LNI4+7SecYYLGqBqwLYG5urubl5QXqcMaYBvrHl3t57J2NdEqO4YU7BtA+qXlAjrv76Glu+MtnpCc04437hhAT1fB2JWWV1dzy3DI2HSrm9XuH0KttcE3NWR8islJVcy9Vzqt+AKpapKqzAnnzN8Y0HrcO6sBL3x/IkVPlfPOZz1m+2//zB58qq2TyjBW4woTnbs+9rJs/fL1l0N0vhUbLIBsKwhjjE0O6JPPW/UNIaBbBrc8v4/W8fL8dq7pGefCfa9h7rJRnbu1Pu0TfPHGEWssgSwDGGJ/JSGnBW/cPZWCnRB6evY7ffLTZLy+Hp328lflbjvDY+B5c0TnJp/vukR7HH0KkZZAlAGOMT8U3j+D/7hrI9wa359lFu7j35ZWcLq/y2f7fW3uQpxfs5LsD2/O9wR18tt9zXRMiLYMsARhjfC7CFcbjN/TkFxOy+XRLAd/66xccOHHmsve74cBJHp69lgEdW/KLCdnU9jX1j1BoGWQJwBjjFyLCHUM68ve7BnKg6Aw3/OUzVu4tavD+jpaUc89LeSQ2j+SZW/sTGe7f29e5YwY9NGsN6/c3vTGDLAEYY/xqRFYKb00ZQvPIcL773DLeXl3/4SMqqmq475WVHC+tYPrtuaTERvkh0gs19ZZBlgCMMX7XpVUs70wZSt92CTw4cw2/m7uVmnq8HP6f9zayYk8Rv/12Dj3bBLZ9flNuGWQJwBgTEC1jInn5B4OYlNuOvyzYwZRXV1FacemXwy8v28urX+7j/pGdGe8ehC7QmmrLIEsAxpiAiQwP43+/1YtHr+vOnI2HuenZpRw+WXe1yrJdx/jFuxu5qlsrfnJN1wBGeqFrstP46dhuTaplkCUAY0xAiQiTh2Xwwh257C48zYS/fMa6/ScuKJd/vJT7/7GKDknN+cPNfXCF+a/Fj7d+OCKDif1qWwZ9sK7xtwyyBGCMccRV3VJ54/4hRLjC+M7flvL+uq9mGCutqOLul/Koqq7h+TsGEBcd4WCkXxERfn1jbcugn7ze+FsGWQIwxjimW1oc7/xoKD3bxPOjV1fzx0+2U1Oj/Mfra9lWcIo/39KPTskxTof5Nee3DCpoxC2DLAEYYxyV3CKKV+8exMR+bfj9J9sY98fFfLj+MD+7tjsjsoJzUqiU2Ciev6O2ZdB9r6xstC+FLQEYYxwXFe7iqe/k8Mi4bmw/UsLEvm2YPKyT02FdVPfWcfzPhGxW7TvB3I0FTofTIF7NB+AUmw/AmNCzv6iU1vHNguKl76VUVddwze8XExkexocPDCMsSGL26XwAxhgTKG1bNm8UN3+AcFcYD1ydyZbDp5izsfFNl+LXBCAiiSIyRkSS/XkcY4xxyvicdDqnxPzrBXZj4lUCEJEXRGSpiDxax/edROQDEVkiIk+517UE3gcGAgtEJOWc8qkistoH8RtjjKNcYcK/j85ia8EpPtzQuPoGXDIBiMhEwKWqVwAZIpLpodgTwOOqOgxoKyIjgd7AQ6r6K2Au0O+c8r8Dml1u8MYYEwyu69WazFYt+MMn2/0yAY6/ePMEMBKY5V6eB1zpoUwWsMq9fASIV9VFqrpMRIZT+xSwFEBErgJOA42vwswYYzyofQrIZMeRkq91aAt23iSAGODs+K3HgVQPZWYDj4nIeGAcMB9AamdrmAQUAZUiEgn8N/CfdR1MRO4RkTwRySssLPT6RIwxxknf6Nmarqmx/HF+43kK8CYBlPBVdU0LT9uo6lTgI2AyMENVS9zrVVWnAOuACdTe+J9R1QsH/vhqX9NVNVdVc1NSgrMTiDHGnC8sTHhwdCa7Ck/z7tr6z3ngBG8SwEq+qvbJAfbUUW4N0B6YBiAij4jI7e7vEoATwGhgiogsBPqIyPMNC9sYY4LP2Ow0uqXF8qf5O6iqrnE6nEvyJgG8DdwmItOAm4CNIjLVQ7mHgWmqWur+PN293WLABcxT1eGqOlJVRwJrVHXy5Z+CMcYEh7Aw4cdjsth99DTvrAn+dwFe9QR2N+kcAyxW1YC9vLWewMaYxkZVuf7Pn1FSXsX8h0YQ7gp8f1uf9gRW1SJVnRXIm78xxjRGIsKDo7PYe6yUNxsw/3Eg2VAQxhjjY6O7t6JXm3j+/Ol2KoP4XYAlAGOM8TER4cdjMsk/foY3Vu53Opw6WQIwxhg/GNW1FTntEvjzpzuoqArOpwBLAMYY4wciwo9HZ3LgxBleX5nvdDgeWQIwxhg/GZGVQt/2CTz96Q7Kq6qdDucClgCMMcZPap8Csjh4soxZecH3LsASgDHG+NGwzGT6d2jJ05/uoKwyuJ4CLAEYY4wfiQgPjcnicHEZM1cE17sASwDGGONnQzonMbBjIs8sDK6nAEsAxhjjZyLCg2MyKSgu59Uv9zkdzr9YAjDGmAAY0jmZwRmJ/HXRzqB5CrAEYIwxAfLj0VkUnirnlWV7nQ4FsARgjDEBMygjiSGdk/jbop2UVlQ5HY4lAGOMCaQfj8niaElFUDwFWAIwxpgAGtAxkWGZyTy7aBeny519CvBrAhCRRBEZIyLJF1tnjDGh5MHRWRw7XcFLS519CvAqAYjICyKyVEQereP7TiLygYgsEZGn3OtaAu8DA4EFIpLiaZ1vTsMYYxqP/h1aMiIrhemLd1Li4FPAJROAiEwEXKp6BZAhIpkeij0BPK6qw4C2IjIS6A08pKq/AuYC/epYZ4wxIefHY7IoKq1kxhd7HIvBmyeAkcAs9/I84EoPZbKAVe7lI0C8qi5S1WUiMpzaX/xLPa27rOiNMaaR6tMugVFdU5i+eBenyiodicGbBBADnJ3Y8jiQ6qHMbOAxERkPjAPmA4iIAJOAIqCyrnXnEpF7RCRPRPIKCwvrdzbGGNOI/HhMFifPVPJ/n+9x5PjeJIASoJl7uYWnbVR1KvARMBmYoaol7vWqqlOAdcCEutadt6/pqpqrqrkpKfaKwBjTdPVum8Do7q14bskuih14CvAmAazkq2qfHGBPHeXWAO2BaQAi8oiI3O7+LgE44WldQ4I2xpim4sHRWRSXVfHiZ7sDfmxvEsDbwG0iMg24CdgoIlM9lHsYmKaqpe7P093bLQZc1L4/8LTOGGNCVs828VzTI5UXPtvNyTOBfQoQVb10odrmm2OAxap62O9RueXm5mpeXl6gDmeMMY7YdLCYb/xpCQ9cnclDY7Iue38islJVcy9Vzqt+AKpapKqzAnnzN8aYUNEjPY5x2Wm8+NluTpRWBOy4NhSEMcYEgQfHZFJSXsXzSwL3LsASgDHGBIFuaXFc16s1f/98N0WnA/MUYAnAGGOCxL+PzqS0sprpS3YF5HiWAIwxJkhkpcZyXa/WzPhiD8dKyv1+PEsAxhgTRB4cncmZAD0FWAIwxpgg0qVVLHdc0ZH0+GaXLnyZwv1+BGOMMfXyPxOyA3IcewIwxpgQZQnAGGNClCUAY4wJUZYAjDEmRFkCMMaYEGUJwBhjQpQlAGOMCVGWAIwxJkR5NSGMU0SkENh7GbtIBo76KBxfsrjqx+KqH4urfppiXB1U9ZKTqgd1ArhcIpLnzaw4gWZx1Y/FVT8WV/2EclxWBWSMMSHKEoAxxoSopp4ApjsdQB0srvqxuOrH4qqfkI2rSb8DMMYYU7em/gRgjDGmDpYAjDEmRAVtAhCRF0RkqYg8Wp8yvl4XyLhEJF5EPhKReSLylohEiki4iOwTkYXuf70ciMtjDCLyCxFZISJPO3S97jsnpjUi8qy318sPsaWKyJJzPkeIyHsi8rmIfL+udQ7E1d59XT4VkelSq42I7D/nmnlsP+7nuDzG4MD/yfPj+sU5MW0RkZ8F+nqJh/tCfc7pYoIyAYjIRMClqlcAGSKS6U0ZX68LdFzArcA0Vb0GOAyMA3oDr6nqSPe/9Q7EdUEMItIfuBIYCBwRkdGBjktV/3o2JmAJ8Jw318sPsbUEZgAx52z+b8BKVR0KfFtEYutYF+i47gXuU9WrgHZAL2AQ8KtzrlmhA3FdEIMD/ycviEtVHzvnb2wD8FKgrxce7gsNvYedLygTADASmOVenkftjcabMr5eF9C4VPUZVf3YvS4FOAIMBq4XkeXu7O5pGk+/xlVHDCOAN7S2FcFcYJgDcQG1vx6BVFXNqyNWT3wZWzUwCSiuY9vFQG4d6wIal6r+XFU3uz8mUdvTdDAwWURWicivPRzP73HVEUNDj+nLuAAQkQHAflU9UEesfourjvuCt+d0UcGaAGKAA+7l40Cql2V8vS7QcQEgIlcALVV1GbACGK2qA4EI4BsOxOUphqC5XsAU4K/uZW+ul09jU9ViVT3ZwPMKdFwAiMgkYKOqHgQ+ovbmMQC4QkR6OxCXpxiC5noB/w78+SKx+i2us4XPuy809O/ra4I1AZQAzdzLLfAcp6cyvl4X6LgQkURq/9DO1hGvU9VD7uU8wNNjnb/j8hRDsFyvMGAUsND9nTfXy9exebv/QF8zj0QkA/gP4EH3qi9U9ZSqVgOr8f/fmCeeYgiW65UAtFLVnReJ1a9xebgvXNY5nRWsCWAlXz2+5AB7vCzj63UBjcv9cud14GeqenYQvJdFJEdEXMA3gbWBjquOGBy/Xu7lYcCX+lWHFm+ul69j83b/gb5mF3DXc78GfP+cX7tzRaS1iDQHrqG2rjugcdURg+PXy+0G4MNLxOq3uOq4L1zuOdVS1aD7B8RR+x93GrDZfTJTL1Em3tfrHIjrPqCI2l+zC6mtj+wJrAPWU/viyYnrdUEM1P54+Bz4I7AV6BTouNzb/xqYeM7+Lnm9fB3bOeUXnrPcAdjovj4rAJendQ7E9QRw6Jy/sRHUPkFtcV+3Hzl0vS6I4WLbBiou9+dXgX4Xi9XPf/ue7gsNuoddEKfTN/uL/AdtCdwEpNWnjK/XBTquYL1edeyvGfBtICOY4nLimtWxbbq7XPzF1gU6rmC9Xr48ZihcL1+ckw0FYYwxISpY3wEYY4zxM0sAxhgToiwBGGNMiLIEYIwxIcoSgDHGhKj/D+sNy0YmG2LcAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(np.linspace(0, 0.002, pieces), scores)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-12-13T07:42:31.961819Z",
     "start_time": "2019-12-13T07:42:25.761299Z"
    }
   },
   "outputs": [],
   "source": [
    "\"\"\"再一次更正模型\"\"\"\n",
    "X_embedded = SelectFromModel(rfc, threshold=0.00125).fit_transform(X, y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-12-13T07:42:48.626869Z",
     "start_time": "2019-12-13T07:42:48.552524Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(42000, 249)"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_embedded.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-12-13T07:43:57.753497Z",
     "start_time": "2019-12-13T07:43:34.729360Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.9374765994297645"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "\"\"\"\n",
    "这里嵌入法的阈值为0.00125过滤的特征比之前的还多(threshold=0.001)，\n",
    "但模型的评分却比前面还高，说明其中含有一些噪音。\n",
    "\"\"\"\n",
    "cross_val_score(rfc, X_embedded, y, cv=5).mean()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-12-13T07:51:34.882151Z",
     "start_time": "2019-12-13T07:47:57.104457Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.9610005206702571"
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 使用更精确的随机森林模型\n",
    "cross_val_score(RFC(n_estimators=100, random_state=1), X_embedded, y, cv=5).mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Wrapper包装法"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "包装法也是一个**特征选择和算法训练**同时进行的方法，与嵌入法十分相似，它也是依赖于算法自身的选择，比如\n",
    "coef_属性或feature_importances_属性来完成特征选择。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "但不同的是，我们往往使用一个目标函数作为黑盒来帮\n",
    "助我们选取特征，而不是自己输入某个评估指标或统计量的阈值。包装法在初始特征集上训练评估器，并且通过\n",
    "coef_属性或通过feature_importances_属性获得每个特征的重要性。然后，从当前的一组特征中修剪最不重要的\n",
    "特征。在修剪的集合上**递归地重复该过程**，直到最终到达所需数量的要选择的特征。区别于过滤法和嵌入法的一次\n",
    "训练解决所有问题，包装法要使用特征子集进行多次训练，因此它所需要的计算成本是最高的。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![](images/3_12.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "这里的算法指的时数据挖掘时的算法(即，目标函数)。最典型的目标函数是递归特征消去法(Recursive feature elimination, 简写为RFE)。这是一种贪婪的优化算法，它会反复创建模型， 并在每次迭代中剔除那些表现最差的特征，选出那些表现最好的特征。然后从上一次没选到的特征中构建模型，如此往复，直到所有的特征都筛选完毕(可见，这种算法是成效明显，但计算量要比嵌入法和过滤法多得多)。  \n",
    "\n",
    "最后，会根据所选的特征进行排名，选出最佳子集。包装法的效果是所有特征选择方法中最利于提升模型\n",
    "表现的，它可以使用很少的特征达到很优秀的效果。除此之外，在特征数目相同时，包装法和嵌入法的效果能够匹\n",
    "敌，不过它比嵌入法算得更见缓慢，所以也不适用于太大型的数据。相比之下，包装法是最能保证模型效果的特征\n",
    "选择方法"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "参数**estimator**是需要填写的实例化后的评估器，**n_features_to_select**是想要选择的特征个数，**step**表示每次迭\n",
    "代中希望移除的特征个数。除此之外，RFE类有两个很重要的属性，**.support_**：返回所有的特征的是否最后被选\n",
    "中的布尔矩阵，以及**.ranking_**返回特征的按数次迭代中综合重要性的排名。类feature_selection.RFECV会在交叉\n",
    "验证循环中执行RFE以找到最佳数量的特征，增加参数cv，其他用法都和RFE一模一样"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-12-13T07:55:59.712516Z",
     "start_time": "2019-12-13T07:55:59.634520Z"
    }
   },
   "outputs": [],
   "source": [
    "from sklearn.feature_selection import RFE"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-12-13T07:58:36.823591Z",
     "start_time": "2019-12-13T07:57:26.639347Z"
    }
   },
   "outputs": [],
   "source": [
    "rfc = RFC(n_estimators=10, random_state=1)\n",
    "rfe = RFE(rfc, n_features_to_select=300, step=50).fit(X, y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-12-13T07:59:08.148567Z",
     "start_time": "2019-12-13T07:59:08.067477Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "300"
      ]
     },
     "execution_count": 56,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "rfe.support_.sum()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-12-13T08:00:18.318603Z",
     "start_time": "2019-12-13T08:00:18.290781Z"
    },
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([11, 10,  9,  8,  7,  7,  7,  7,  7,  7,  7,  8,  7,  7,  8,  7,  7,\n",
       "        7,  7,  8,  7,  7,  7,  8,  7,  7,  7,  8,  8,  7,  7,  7,  7,  7,\n",
       "        7,  7,  7,  7,  7,  7,  7,  7,  7,  6,  7,  8,  7,  8,  8,  8,  8,\n",
       "        8,  8,  8,  8,  8,  8,  8,  8,  8,  8,  8,  8,  7,  7,  8,  5,  5,\n",
       "        5,  4,  4,  5,  5,  5,  5,  6,  6,  9,  9,  9,  9,  9,  9,  9,  9,\n",
       "        9,  9,  9,  9,  7,  9,  7,  6,  5,  4,  1,  3,  1,  1,  1,  1,  1,\n",
       "        1,  3,  4,  6,  5,  5,  7,  9,  9,  9, 10, 10, 10, 10,  9,  6,  6,\n",
       "        4,  3,  4,  1,  1,  1,  1,  1,  1,  1,  3,  1,  2,  3,  4,  5,  5,\n",
       "        5, 10, 10, 10, 11, 11, 11, 11, 11, 11,  4,  4,  2,  1,  1,  1,  1,\n",
       "        1,  1,  1,  1,  1,  1,  1,  1,  3,  2,  4,  5,  6, 10, 11, 11, 11,\n",
       "       10, 10,  5,  4,  4,  2,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,\n",
       "        1,  1,  1,  1,  1,  3,  8, 10, 11, 11, 11, 11,  5,  5,  4,  3,  1,\n",
       "        1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  3,  1,\n",
       "        6,  7, 11, 11, 11, 10,  6,  5,  4,  3,  3,  2,  1,  1,  1,  1,  1,\n",
       "        1,  1,  1,  1,  1,  1,  1,  1,  1,  2,  2,  4, 11, 11, 10, 10,  8,\n",
       "        7,  5,  4,  3,  2,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,\n",
       "        1,  1,  1,  2,  4,  6, 10, 10, 10, 10,  6,  6,  6,  4,  4,  1,  1,\n",
       "        1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  2,  3,  4,  4,\n",
       "        6, 10, 10, 10,  9,  6,  6,  4,  3,  1,  1,  1,  1,  1,  1,  1,  1,\n",
       "        1,  1,  1,  1,  1,  1,  1,  2,  2,  4,  6,  8, 10, 11, 10, 10,  6,\n",
       "        5,  3,  2,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,\n",
       "        1,  1,  4,  4,  6, 10, 11, 11, 11, 11, 10,  5,  3,  2,  1,  1,  1,\n",
       "        1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  4,  6, 11,\n",
       "        9, 11, 11, 11,  6,  5,  3,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,\n",
       "        1,  1,  1,  1,  1,  1,  1,  1,  4,  6,  8, 11, 11, 11, 11,  6,  6,\n",
       "        4,  2,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,\n",
       "        1,  2,  5,  6, 11, 10, 11, 11, 11, 10,  5,  3,  1,  1,  1,  1,  1,\n",
       "        1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  2,  3,  3,  4,  5,  6,  9,\n",
       "       11, 11,  9,  5,  5,  3,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,\n",
       "        1,  1,  1,  1,  1,  2,  3,  4,  5, 11, 10, 11, 11,  9,  5,  5,  2,\n",
       "        1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  2,\n",
       "        3,  3,  5,  6, 11, 11, 11,  6,  4,  3,  2,  2,  1,  1,  1,  1,  1,\n",
       "        1,  1,  1,  1,  1,  1,  1,  1,  2,  3,  3,  4,  5,  7, 11, 11, 11,\n",
       "       11, 11,  5,  5,  3,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,\n",
       "        1,  1,  1,  2,  3,  4,  4,  6, 10, 10, 10, 10, 10,  5,  5,  4,  2,\n",
       "        1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  2,  2,  3,  5,\n",
       "        6,  7, 10, 10, 10, 10,  9,  6,  4,  4,  3,  1,  1,  1,  1,  1,  1,\n",
       "        1,  1,  1,  1,  2,  2,  1,  3,  3,  4,  6,  6,  7, 10, 10, 10, 10,\n",
       "       10,  6,  5,  4,  3,  3,  1,  1,  1,  1,  1,  1,  1,  1,  1,  2,  3,\n",
       "        3,  3,  4,  5,  6,  8, 10, 10, 10, 10,  9,  9,  6,  5,  5,  4,  3,\n",
       "        2,  3,  1,  3,  1,  1,  1,  2,  2,  3,  3,  3,  4,  5,  6,  6,  6,\n",
       "        9,  9,  9,  9,  9,  9,  9,  6,  5,  4,  4,  3,  3,  1,  2,  1,  3,\n",
       "        3,  3,  4,  3,  5,  4,  6,  6,  6,  9,  8,  9,  8,  8,  8,  8,  8,\n",
       "        8,  7,  6,  7,  6,  5,  6,  4,  5,  5,  4,  5,  5,  4,  4,  6,  5,\n",
       "        6,  6,  7,  8,  8,  8,  8,  8,  8,  8,  8, 10,  8,  8,  9,  9,  9,\n",
       "        9,  7,  8,  9,  9,  9,  9,  9,  7,  8,  9,  9,  9,  9,  9,  9, 10,\n",
       "       11,  8])"
      ]
     },
     "execution_count": 62,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "rfe.ranking_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-12-13T07:58:55.554486Z",
     "start_time": "2019-12-13T07:58:55.457508Z"
    }
   },
   "outputs": [],
   "source": [
    "X_wrapper = rfe.transform(X)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-12-13T08:00:51.122691Z",
     "start_time": "2019-12-13T08:00:25.430793Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.9384047458885167"
      ]
     },
     "execution_count": 63,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cross_val_score(rfc, X_wrapper, y, cv=5).mean()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-12-13T08:24:46.753633Z",
     "start_time": "2019-12-13T08:03:13.924803Z"
    }
   },
   "outputs": [],
   "source": [
    "# 对包装法绘制学习曲线\n",
    "scores = []\n",
    "for i in range(1, X.shape[1]+1, 50):\n",
    "    X_wrapper = RFE(rfc, n_features_to_select=i, step=50).fit_transform(X, y)\n",
    "    score = cross_val_score(rfc, X_wrapper, y, cv=5).mean()\n",
    "    scores.append(score)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-12-13T08:24:47.326578Z",
     "start_time": "2019-12-13T08:24:46.977375Z"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXYAAAD6CAYAAAC1W2xyAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAGRlJREFUeJzt3XuMXOd93vHvM7MXLndJipfVipJtSZQUBI1lOupWMW3ZWbtiY9dxHQipbMCVi6aBYENI/0hRRGkVGE3lXohAQNFGgukyQGAXRt0WFeq0gZUCYaTKlO2lY8mV0yKSQSehd8jlbZZL7uxl5tc/5uxtdoZ7SA53Zs55PsBgzpx5Z+c3h9xn3z37nvdVRGBmZtlR6HQBZmbWXg52M7OMcbCbmWWMg93MLGMc7GZmGeNgNzPLGAe7mVnGONjNzDLGwW5mljF9nXjTffv2xT333NOJtzYz61knT548FxGjm7XrSLDfc889TE5OduKtzcx6lqQfp2nnUzFmZhnjYDczyxgHu5lZxjjYzcwyxsFuZpYxDnYzs4xxsJuZZUxHxrGb3Qq1WlBZqnJ1ocrcQpXKYpW5xfrjymKVxkUg1fhY2uT5hveL+ntWa0E1or4d9ce1CKo1VvYt1WKlbS0aXlODarJEZX9B9BUL9BdFf7FAX3LfXxR9hbX7C63bFurbfQXV37taf7+l2vJ9bf3jaov9taBaq617fbEgBooFBvrW3IoN98n2YF+B/jX7+gracIxv5N94+RhHsHq8l49n2qU+r2NF0L7ks7TrM2wFB7tdU7UWLCzVWKjW1t0vJvfzS/X7tUFVawy3WLuv8ZtzOeRYsx1Uq8FcEsxzCy3uF6tUFqpcTfbNL9U6fbjsGiRWwn859AvSuv8vtWBdUDfu77TGzzDYV2z+Q61//b7Blf1FfubOnXzyvXfd0jod7BlTqwXluUUuXF3g4pUFzl9Zf3/hygIXri5QWaw2BHWsCepq/XG11tFvpv6i2NZfZKi/yNDA+vtdQ/0b9q2939ZfZPvy4/4ig/1FCms6Wo2famNHL675fAAFQUGiWNDK/bptiUIB+goFCgUoLrdNnlvbdrm2paQHvVirsbhUY6kWLFbr/z5Ly/e12pp9rdsu1YK+pKa+oigWCquPV+4La55vsn/d67X6g375/0q1/t4rP/iT7eUf+IsN+5dft7y/GrHhuBTE+mPUZH+hsHrc1h57SRt+02olTcc7ApaqyWdaXP8Z1n7m+aX1309Xryyta7N8PBaWanz03Xc42PNuqVpjenae87P1UL54dYHzs8l9k9C+eHWBVlm8faDInuEBdm8fYGigyPBgH7uL639d7l/za+dAw3MDRa3ZLtK/5nE9CNYH3drQWvuNKrESfGoIwbXfpP3F/P0JqH5aBYYodroU62EO9g67XFnkJ5cq/OTSHH91aY6fJLfTF+v3pZlK06AuCHZvH2D38AB7hge4b3SEv3HvAHu21x83u23rd1iY5YGD/Raq1oKzl+uhfToJ7+XAPp3cLleW1r2mryD237aNO3cN8b4De7nztiH237aNvcOD7B2p97b3Dg+wc6ifYqH7/4hjZlvPwd4mM5VFvv2jC7z61jl+ODVT722XKyw1dLd3buvjrt3becfuIR6+dw933jbEXbcNrdyP7hh0YJvZTXGw36DKYpXv/fgir759jlffOs8bf3WJWsC2/gLvvnMX43fv5s7lwN5dD+39u7axY1t/p0s3s4xzsKdUrQU/OF3m1bfO8a23zzF56iLzSzWKBXHwHbt46sP38/779vHQ3bcx2Odz2WbWOQ72FiKCt87O8upb53j17fO89qPzK+fDf/qOHXzm5+7mA/fv5eF797gXbmZdJVWwSzoG/DXgf0TEs02evxf498BO4DsR8Y/bWuUWOX1prt4jf+sc33r7PGcvzwPwzj1DfPzB/bz//n28/7697BsZ7HClZmatbRrskh4DihFxSNLvSXogIv68odm/Af5FRLwm6T9JmoiI47ei4HZ77Ufn+e+v/4RvvXWOU+evArBvZIBD9+3jA/ft5QP37+Ode7Z3uEozs/TS9NgngK8n2y8BjwCNwf5TwPeS7bPArsYvIulJ4EmAd73rXTdQ6q3xua+eZHGpxvsO7OWJQ/fwyP37+KmxkZ6YD8LMrJk0wT4MnE62LwAPNWnzX4AvSHoN+Cjwm40NIuIocBRgfHy885M+AFfml7h0dZGnP/bTfO7n7+t0OWZmbZHmmu1ZYCjZHmn2muS8+x8Cvwr8fkTMtq3CW6g0UwHgjp3bOlyJmVn7pAn2k9RPvwAcBE61aPd94F3Aczdf1tYolZNg3+VgN7PsSBPsLwJPSHoOeBx4U9KGkTHAPwGei4ir7SzwVppKgn2/g93MMmTTc+wRMSNpAjgMHImIEvB6k3ZfaH95t1apPAfAmE/FmFmGpBrHHhEXWR0ZkxlT5YpnPTSzzMnfhNdrlMoV/+HUzDIn18E+Va74/LqZZU6ug/3MTIUxB7uZZUxug72yWOX8lQX2+1SMmWVMboP97Ex9gi+PYTezrMltsE8lQx337xrapKWZWW/JbbCvTCfgHruZZUxug33K0wmYWUblNthL5Qo7BvsYGfQiUmaWLbkOdvfWzSyLchvsUzMOdjPLptwGe6k856tOzSyTchnsi9UaZy/Pc4eHOppZBuUy2KcvzxPhedjNLJtyGewe6mhmWZbLYD/jtU7NLMNyGexeEs/MsiyXwV4qz7Gtv8Cuof5Ol2Jm1napgl3SMUknJD3T4vndkv6npElJX2pvie1XX2BjCEmdLsXMrO02DXZJjwHFiDgEHJD0QJNmTwD/MSLGgR2SxttcZ1t5STwzy7I0PfYJVheyfgl4pEmb88C7Jd0GvBP4y7ZUd4t4STwzy7I0wT4MnE62LwBjTdr8b+Bu4B8Bf5a0W0fSk8mpmsnp6ekbLPfm1WrhJfHMLNPSBPsssHyJ5kiL13wB+FxE/Dbwf4F/0NggIo5GxHhEjI+Ojt5ovTft/JUFlmrhHruZZVaaYD/J6umXg8CpJm12Aw9KKgI/B0RbqrsFSmWPYTezbEsT7C8CT0h6DngceFPSsw1t/hVwFCgDe4CvtbXKNvKSeGaWdZuuMhERM5ImgMPAkYgoAa83tPkO8DO3pMI285J4ZpZ1qZYPioiLrI6M6WlT5Qr9RbF3eKDTpZiZ3RK5u/K0VK4wtnMbhYIvTjKzbMplsPsPp2aWZfkLdi+JZ2YZl6tgjwimvCSemWVcroK9PLdIZbHmJfHMLNNyFeyeh93M8iBXwV7yknhmlgO5CvYpTydgZjmQq2AvzVQoCEZ3DHa6FDOzWyZfwV6eY3THIP3FXH1sM8uZXCXcVLniETFmlnm5CvZSucJ+n183s4zLXbB7RIyZZV1ugn12fonL80sOdjPLvNwEe8kXJ5lZTuQu2D2G3cyyLjfB7iXxzCwvchPsyz3223f64iQzy7ZUwS7pmKQTkp5p8fznJR1Pbt+X9KX2lnnzpmYq7B0eYFt/sdOlmJndUpsGu6THgGJEHAIOSHqgsU1EvBARExExAbwCfLntld4kD3U0s7xI02OfYHUh65eAR1o1lHQXMBYRkzdfWntNeUk8M8uJNME+DJxOti8AY9do+xTwQrMnJD0paVLS5PT09PVV2QZnvCSemeVEmmCfBZaHkoy0eo2kAvBh4Hiz5yPiaESMR8T46OjoDZR64yqLVS5cWfAYdjPLhTTBfpLV0y8HgVMt2n0Q+HZERBvqaqszM8sLbHioo5llX5pgfxF4QtJzwOPAm5KebdLuF4CX21lcu3hJPDPLk77NGkTEjKQJ4DBwJCJKwOtN2v3T9pfXHl4Sz8zyZNNgB4iIi6yOjOk5XhLPzPIkF1eenpmpsGNbH8ODqX6OmZn1tFwE+1R5zufXzSw3chHsJS+JZ2Y5kotgn/KSeGaWI5kP9sVqjenZeY+IMbPcyHywn708T4THsJtZfmQ+2EvJAhtjDnYzy4kcBPs84B67meVH5oN9ZUm8nR4VY2b5kPlgL5UrDPUX2Tnki5PMLB8yH+xTMxX279qGpE6XYma2JTIf7F4Sz8zyJh/B7ouTzCxHMh3stVp4STwzy51MB/u5K/Ms1cJDHc0sVzId7KsLbHioo5nlR6aD3UvimVkeZTrYvSSemeVRpoN9qlxhoFhgz/aBTpdiZrZlUgW7pGOSTkh6ZpN2z0v6RHtKu3ml8hy37xykUPDFSWaWH5sGu6THgGJEHAIOSHqgRbsPAndExDfaXOMNKyVXnZqZ5UmaHvsE8PVk+yXgkcYGkvqBLwOnJH2y2ReR9KSkSUmT09PTN1ju9fGSeGaWR2mCfRg4nWxfAMaatPks8EPgCPCwpF9rbBARRyNiPCLGR0dHb7Te1CKiviSee+xmljNpgn0WWO72jrR4zc8CRyOiBHwV+HB7yrtxl64uMr9U83QCZpY7aYL9JKunXw4Cp5q0eQs4kGyPAz++6cpuksewm1lepZmk/EXgFUl3Ah8DPi3p2YhYO0LmGPB7kj4N9AO/3P5Sr09pxkvimVk+bRrsETEjaQI4DBxJTre83tDmMvB3b0mFN8g9djPLq1TLCkXERVZHxvSEM+UKBcHoyGCnSzEz21KZvfJ0qlzh9h3b6Ctm9iOamTWV2dQreR52M8upzAa7x7CbWV5lNthL5QpjHsNuZjmUyWC/XFlkdn7JPXYzy6VMBvuZGc/Dbmb5lclgXx3D7gnAzCx/Mh7s7rGbWf5kMtiXl8S7facvTjKz/MlksE+VK+wbGWCwr9jpUszMtlwmg71UnvNQRzPLrUwGuy9OMrM8y2Swn/F0AmaWY5kL9spilYtXFz3U0cxyK3PBvjwixkvimVleZS7YPYbdzPIuc8HuJfHMLO8yF+xTPhVjZjmXKtglHZN0QtIzLZ7vk/QXko4ntwfbW2Z6Z8oVdm7rY3gw1ap/ZmaZs2mwS3oMKEbEIeCApAeaNHsP8LWImEhuP2h3oWnVx7B7RIyZ5VeaHvsEqwtZvwQ80qTN+4BflPSdpHffse6yl8Qzs7xLE+zDwOlk+wIw1qTNd4FHI+JhoB/4240NJD0paVLS5PT09I3WuylfdWpmeZcm2GeB5XMbIy1e80ZETCXbk8CG0zURcTQixiNifHR09IaK3czCUo1zs/PusZtZrqUJ9pOsnn45CJxq0uYrkg5KKgK/BLzenvKuz9nLFSI8IsbM8i1NsL8IPCHpOeBx4E1Jzza0+W3gK8D3gRMR8b/aW2Y6K1edusduZjm26R85I2JG0gRwGDgSESUaeuQR8X+oj4zpqNKMl8QzM0s1eiUiLrI6MqZrucduZpaxK0+nyhW2DxTZuc0XJ5lZfmUq2Evl+hh2SZ0uxcysYzIV7FPlOY+IMbPcy1SwL/fYzczyLDPBXq0FZy/P+6pTM8u9zAT7+dl5lmrBHR7qaGY5l5lgX1k5yefYzSznMhfsPsduZnmXmWAvletL4jnYzSzvMhPsUzMVBooF9mwf6HQpZmYdlZlgL5UrjO0apFDwxUlmlm+ZCvb9Oz0ixswsO8HuJfHMzICMBHtEeEk8M7NEJoL94tVFFpZq7rGbmZGRYJ9aHuroi5PMzLIR7F5gw8xsVSaCfWU6Ac8TY2aWLtglHZN0QtIzm7Qbk/Sn7SktvTMzFYoFMbpjcKvf2sys62wa7JIeA4oRcQg4IOmBazT/HWDLu81T5Qq37xik6IuTzMxS9dgnWF3I+iXgkWaNJH0EuAKU2lLZdfACG2Zmq9IE+zBwOtm+AIw1NpA0APwW8HSrLyLpSUmTkianp6dvpNaWvCSemdmqNME+y+rplZEWr3kaeD4iLrX6IhFxNCLGI2J8dHT0+itt/XWZco/dzGxFmmA/yerpl4PAqSZtHgWeknQceK+k/9CW6lK4PL/E1YWqrzo1M0v0pWjzIvCKpDuBjwGflvRsRKyMkImIDy1vSzoeEb/a/lKbO7Myht1DHc3MIEWwR8SMpAngMHAkIkrA69doP9G26lJYHcPuHruZGaTrsRMRF1kdGdNVVq469R9PzcyADFx5utxjH3Owm5kBGQj20swc+0YGGOjr+Y9iZtYWPZ+GHupoZrZezwd7qVzhDi+JZ2a2oveDfcYrJ5mZrdXTwT63UOXS1UWfijEzW6Ong7004zHsZmaNejrYvSSemdlGPR3sXhLPzGyjng72KQe7mdkGPR3sZ2Yq7BrqZ/tAqpkRzMxyoaeDfarsoY5mZo16Oti9JJ6Z2UY9HezusZuZbdSzwb6wVOPc7LxndTQza9CzwX7GFyeZmTXVs8G+fNWpl8QzM1uvd4PdS+KZmTXVtmCXtEfSYUn72vU1r8VXnZqZNZcq2CUdk3RC0jMtnt8N/AHwMPDHkkbbWGNTU+UKwwNFdgz64iQzs7U2DXZJjwHFiDgEHJD0QJNm7wF+PSK+CHwTeKi9ZW5UmpljbNc2JN3qtzIz6ylpeuwTwNeT7ZeARxobRMSfRMRrkj5Evdd+orGNpCclTUqanJ6evomS6zyG3cysuTTBPgycTrYvAGPNGqnedf4UcBFYbHw+Io5GxHhEjI+O3vyZGi+JZ2bWXJpgnwWWE3Sk1Wui7ingDeDvtKe85qq14OzleffYzcyaSBPsJ1k9/XIQONXYQNJvSPps8vA24FJbqmvh3Ow81Vp4RIyZWRNpgv1F4AlJzwGPA29KerahzdGkzctAkfq5+FtmymPYzcxa2nSsYETMSJoADgNHIqIEvN7Q5mLy/JYoJUvieZ4YM7ONUg0CT4L765s23CLusZuZtdaTUwqUyhUGigX2DA90uhQzs67Tk8E+lSyw4YuTzMw26slgL8145SQzs1Z6M9h91amZWUs9F+wR4bVOzcyuoeeC/cKVBRaqNe7wUEczs6Z6Ltg91NHM7Np6LthXF9jwBGBmZs30XLBPeRFrM7Nr6rlgP1OuUCyIfSODnS7FzKwr9VywT5UrjO0YpFjwxUlmZs30XLAvL4lnZmbN9Vywe0k8M7Nr66lgX7k4yUvimZm11FPBPlNZ4upC1T12M7Nr6KlgPzOzPIbdwW5m1kpPBXuxID7+4H7uGx3pdClmZl0r1QpK3eK+0RF+9zMPdboMM7OulqrHLumYpBOSnmnx/C5JfyjpJUn/TZKXNjIz65BNg13SY0AxIg4BByQ90KTZZ4DnIuJvASXgo+0t08zM0kpzKmaC1YWsXwIeAf58bYOIeH7Nw1HgbDuKMzOz65fmVMwwcDrZvgCMtWoo6RCwOyJea/Lck5ImJU1OT0/fULFmZra5NME+CyxfETTS6jWS9gD/DviVZs9HxNGIGI+I8dHR0Rup1czMUkgT7Cepn34BOAicamyQ/LH0PwO/GRE/blt1ZmZ23dIE+4vAE5KeAx4H3pT0bEObfwg8BPwzScclfarNdZqZWUqb/vE0ImYkTQCHgSMRUQJeb2jzAvDCLanQzMyuiyJi699UmgZu9JTNPuBcG8u5Fbq9xm6vD7q/xm6vD1xjO3RbfXdHxKZ/pOxIsN8MSZMRMd7pOq6l22vs9vqg+2vs9vrANbZDt9fXSk/NFWNmZptzsJuZZUwvBvvRTheQQrfX2O31QffX2O31gWtsh26vr6meO8duZmbX1os9djMzuwYHu3WUpD2SDkva1+lazLKip4J9s3nht5qkMUmvJNv9kr4h6VVJv9Jq3xbWtmGO/GbHr5PHVNJu4A+Ah4E/ljTabTUm7z8m6U9b1dLhY9gn6S+SK76PS3pQ0j+X9F1Jv7um3YZ9Haj1eUmfSLa75jhK+vya4/d9SV/qpvpuRM8Ee8p54beynt3A71Of/RLg14CTEfEB4Jcl7Wixb6s0zpH/aRqOXxcc0/cAvx4RXwS+CXykC2sE+B1gqFktXVDfe4CvRcREREwAA9TndnoYOCvpUUl/vXHfFteIpA8Cd0TEN7rtOEbEC2uO3yvA291U343omWCn+bzwnVQFPgXMJI8nWK3vZWC8xb4tERHPR8QfJQ9Hgb/HxuM30WTflomIP4mI1yR9iHro/EK31SjpI8AV6j8cm9XS0fqA9wG/KOk7ko4BfxP4r1EfFfFN4IPAzzfZt2Uk9QNfBk5J+iTdeRyRdBf1acnf0Y31XY9eCvbU88JvhYiYiYjyml3N6ut4zctz5AN/2aX1ifoPyItAdFONyaylvwU8nezqxn/j7wKPRsTDQD/1Kba7rcbPAj8EjlD/Af5UF9ZIUtcLLWrphvpS66VgTzUvfAc1q6+jNTfMkd919QFE3VPAG8D7u6zGp4HnI+JS8rgbj+EbETGVbE+2qKfTNf4scDSZQPCr1H977aoaJRWADwPHW9TS6WN4Xbq6uAabzgvfYc3q61jN2jhHflfVl9T4G5I+mzy8DfjXTerpZI2PAk9JOg68F/hEl9UH8BVJByUVgV+i3rPsthrfAg4k2+PAPU3q6XSNHwS+nZyu6rrvlesWET1xA3ZSny74OeDPgF2drimp63hyfzfwJvBvqf96XGy2bwvr+jz10xvHk9vfbzx+nT6m1E8R/RH1HtzzSU1dVePaf+dmtXS6PuDd1H/b+QHwReqdtVeT/3P/D7i32b4trnEH9U7Gy8CJ5Pui247jvwQeS7a77t/5em89deVpMhLlMPBy1H+t6yqS7qT+U/2bkZx/b7avg/VtOH7ddky7vcZury+pZwj4OPC9iPhRq32d1O3Hsdvr20xPBbuZmW2ul86xm5lZCg52M7OMcbCbmWWMg93MLGMc7GZmGfP/AcflYNCFRYlZAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(range(1, X.shape[1]+1, 50), scores)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "由上图可以看出，模型在第一次过滤时就已经很好了，后面的效果不会很明显。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-12-13T09:09:04.220887Z",
     "start_time": "2019-12-13T09:08:28.178617Z"
    }
   },
   "outputs": [],
   "source": [
    "X_wrapper = RFE(rfc, n_features_to_select=200, step=50).fit_transform(X, y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-12-13T09:09:13.129276Z",
     "start_time": "2019-12-13T09:09:04.296888Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.9337380105246142"
      ]
     },
     "execution_count": 68,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cross_val_score(rfc, X_wrapper, y, cv=5).mean()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-12-13T09:10:34.170485Z",
     "start_time": "2019-12-13T09:09:13.209309Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.9578335813015357"
      ]
     },
     "execution_count": 69,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cross_val_score(RFC(n_estimators=100, random_state=1), X_wrapper, y, cv=5).mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 小结"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "在特征过滤算法中，  \n",
    "过滤法的速度最快，但效果最粗糙。  \n",
    "嵌入法和包装法更精确，具体到每个模型，但开销也由模型的复杂度增加而增加。  \n",
    "优先使用过滤法，简单地过滤一些没有用的特征，再进一步考虑嵌入法和包装法。  \n",
    "当数据量很大的时候，优先使用方差过滤和互信息法调整，再上其他特\n",
    "征选择方法。使用逻辑回归时，优先使用嵌入法。使用支持向量机时，优先使用包装法。迷茫的时候，从过滤法走\n",
    "起，看具体数据具体分析"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": true
  },
  "varInspector": {
   "cols": {
    "lenName": 16,
    "lenType": 16,
    "lenVar": 40
   },
   "kernels_config": {
    "python": {
     "delete_cmd_postfix": "",
     "delete_cmd_prefix": "del ",
     "library": "var_list.py",
     "varRefreshCmd": "print(var_dic_list())"
    },
    "r": {
     "delete_cmd_postfix": ") ",
     "delete_cmd_prefix": "rm(",
     "library": "var_list.r",
     "varRefreshCmd": "cat(var_dic_list()) "
    }
   },
   "types_to_exclude": [
    "module",
    "function",
    "builtin_function_or_method",
    "instance",
    "_Feature"
   ],
   "window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}