Neural Network Facial Recognition.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "from pandas.plotting import scatter_matrix\n",
    "data = pd.read_csv('New_Data.csv')\n",
    "\n",
    "X = data.iloc[:,:4]\n",
    "Y = data['Name']\n",
    "from sklearn.preprocessing import MinMaxScaler\n",
    "from sklearn.model_selection import train_test_split"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train_org, X_test_org, y_train, y_test = train_test_split(X, Y, random_state = 0, test_size = 0.2)\n",
    "#Split the data into training and testing sets.\n",
    "\n",
    "scaler = MinMaxScaler() #Scaling the data because sometimes, the data varies a lot(check X.describe())\n",
    "X_train = scaler.fit_transform(X_train_org)\n",
    "X_test = scaler.fit_transform(X_test_org)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Check this Keras documentation for everything about the Model :https://keras.io/models/model/ or Sequential which is the most basic type: https://keras.io/getting-started/sequential-model-guide/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from keras.models import Sequential\n",
    "from keras.layers import Dense, Activation\n",
    "from keras import optimizers"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Creating the model\n",
    "### First, we must create the model and specify the number of layers using model.add(Dense()). We also specify the type of activation which is how the layer will convert input to an output that the next layer will use as input. Read this for more on activations: https://keras.io/activations/\n",
    "#### Dense(num...: Specifies how many layers to add.\n",
    "#### Activation: The activation parameter in Dense() specifies the type of activation function being used. The purpose of an activation function is to convert a node of input to an output signal. This output signal will now be used as input in the next layer. One common type is sigmoid: S-shaped range of 0 to 1. Another is tanh: Range of -1 to 1. Suffers from vanishing gradient problem. A POPULAR ONE is ReLu: Avoids vanishing gradient problem. Should be used within hidden layers. \n",
    "#### input_dim is the number of features(columns).\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = Sequential()\n",
    "model.add(Dense(8, activation = \"softmax\", input_dim = 4)) #input_dim is # of features\n",
    "\n",
    "#model.add(Dense(2, activation = \"sigmoid\"))\n",
    "model.add(Dense(1)) #activation default is linear"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Compilation\n",
    "### Before traning a model, we must compile the model by configuring the optimization process. We use .compile() and pass in 3 arguments: optimizer function type, loss function we are trying to minimize with our model, (optional) list of metrics to calculate. (The specified metrics are returned in a list after evaluation).\n",
    "\n",
    "#### Learning rate is essentially how quickly a model abandons old beliefs for new ones. It is also how fast the weights change. For example, if the model sees 10 orange cats it will think all cats are orange. If it sees a black cat and has a low learning rate, it will be slow to learn that cats can also be black and will think that it is an outlier. With a very high learning rate, it will change its mind very quickly for the new data. A good learning rate is low enough that the network converges to something useful, but high enough that it doesn't take too long to train.\n",
    "#### Optimization procedures are methods that define a loss and cost function to minimize both. They are used to minimize loss in the training process. The optimization parameter specifies the type of optimizer being used. To see all the optimizer parameters and their information, check this documentation by Keras: https://keras.io/optimizers/ \n",
    "#### The loss parameter specifies how the loss is calculated. Loss is calculated on training and validation. Loss is NOT a percentage. It is the summation of the errors made for each example in the training and validation sets. Therefore, the goal of a model is to reduce the loss by changing weight values through different optimization methods. Loss value implies how well or poorly a certain model behaves after each iteration of optimization. Ideally, one would expect the reduction of loss after each, or several, iteration(s). The accuracy of a model is usually determined after the model parameters are learned and fixed and no learning is taking place. It is done by dividing correctly classified samples by all samples.  For more about loss, check this SO question: https://stackoverflow.com/questions/34518656/how-to-interpret-loss-and-accuracy-for-a-machine-learning-model For the TYPES of loss functions, check this KERAS DOCUMENTATION: https://keras.io/losses/\n",
    "#### metrics: List of metrics to be evaluated by the model during training and testing. Typically you will use metrics=['accuracy']. To specify different metrics for different outputs of a multi-output model, you could also pass a dictionary, such as metrics={'output_a': 'accuracy'}. \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "opt = optimizers.Adamax(lr=0.002, beta_1=0.9, beta_2=0.999, epsilon=None, decay=0.0)\n",
    "model.compile(optimizer = opt, loss = \"mean_absolute_error\", metrics = ['accuracy'])\n",
    "#optimizes the model.\n",
    "#rmsprop optimizer: 47.62%, mean_absolute error, softplus"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Training\n",
    "### We use model.fit() to train model on numpy arrays given a number of epoches (iterations) and batch_size. The model learns from previous iterations and uses its optimizer to try to reduce the loss. Possible parameters are below. \n",
    "\n",
    "#### fit(x=None, y=None, batch_size=None, epochs=1, verbose=1, callbacks=None, validation_split=0.0, validation_data=None, shuffle=True, class_weight=None, sample_weight=None, initial_epoch=0, steps_per_epoch=None, validation_steps=None)\n",
    "#### Epoch: the number of iterations. An (1) epoch is an iteration over the ENTIRE x and y data provided.\n",
    "#### Batch size: the number of samples (from the data) per gradient update. If unspecified, batch_size will default to 32. What it does is whatever the batch size is, it takes that number of rows from the data and trains the model on that. Then, it goes to the next batch of that size and trains it. Small batch sizes are good because the neural network gets to update the weights after running each batch. It also requires less memory and trains faster. Disadvantage is that the gradient estimate will be less accurate.  The gradient is an optimization procedure that refers to calculating the derivative from all training data before calculating an update. For more about batch sizes, check this question: https://stats.stackexchange.com/questions/153531/what-is-batch-size-in-neural-network\n",
    "#### class_weight: Optional dictionary mapping class indices (integers) to a weight (float) value, used for weighting the loss function (during training only). This can be useful to tell the model to \"pay more attention\" to samples from an under-represented class."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.fit(X_train, y_train, epochs = 2500, batch_size = 10, class_weight = None)\n",
    "#Trains from previous iterations."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Evaluation\n",
    "### Now that we have trained the model, we feed it testing data and evaluate it. model.evaluate(X_test, y_test) returns us the accuracy (it is the second and last element in the array). We can also use model.metrics_names[] to get an array with the metrics and their values that we specified in compilation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "scores = model.evaluate(X_test, y_test)\n",
    "print(\"\\n%s: %.2f%%\" % (model.metrics_names[1], scores[1]*100))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Automated Testing - Activation, Optimizer, Loss\n",
    "### The below code runs 40 different tests for 5 different activations: ReLU, SELU, ELU, Softplus,Softmax and 4             optimizers: RMSProp, Adam, Adagrad, Adamax "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "activations = [\"relu\", \"selu\", \"elu\", \"softplus\", \"softmax\"]\n",
    "opt1 = optimizers.RMSprop(lr=0.001, rho=0.9, epsilon=None, decay=0.0)\n",
    "opt2 = optimizers.Adam(lr=0.001, beta_1=0.9, beta_2=0.999, epsilon=None, decay=0.0, amsgrad=False)\n",
    "opt3 = optimizers.Adagrad(lr=0.01, epsilon=None, decay=0.0)\n",
    "opt4 =optimizers.Adamax(lr=0.002, beta_1=0.9, beta_2=0.999, epsilon=None, decay=0.0)\n",
    "optimizern = [1, 2, 3, 4]\n",
    "loss = ['mean_squared_error', 'mean_absolute_error']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "info_list = []\n",
    "float count = 0\n",
    "for a in range (len(activations)):\n",
    "    for b in range (4):\n",
    "        for c in range (len(loss)):\n",
    "            activename = activations[a]\n",
    "            model = Sequential()\n",
    "            model.add(Dense(8, activation = activations[a], input_dim = 4)) #input_dim is # of features\n",
    "            model.add(Dense(1)) \n",
    "            optimname = \"hi\"\n",
    "            opt = optimizers.Adam(lr=0.001, beta_1=0.9, beta_2=0.999, epsilon=None, decay=0.0, amsgrad=False)\n",
    "            if(b == 0):\n",
    "                opt = opt1\n",
    "                optimname = \"RMSProp\"\n",
    "            elif(b == 1):\n",
    "                opt = opt2\n",
    "                optimname = \"Adam\"\n",
    "            elif(b == 2):\n",
    "                opt = opt3\n",
    "                optimname = \"Adagrad\"\n",
    "            elif(b == 3):\n",
    "                opt = opt4\n",
    "                optimname = \"Adamax\"\n",
    "            lossname = loss[c]\n",
    "            print(activename+ \": \" + optimname + \" , \" + lossname)\n",
    "            model.compile(optimizer = opt, loss = loss[c], metrics = ['accuracy'])\n",
    "            model.fit(X_train, y_train, epochs = 2500, batch_size = 10, class_weight = None)\n",
    "            scores = model.evaluate(X_test, y_test)\n",
    "            info_list.append([scores[1]*100, activename+ \": \" + optimname + \" , \" + lossname])\n",
    "            print(scores[1]*100,\"%\")\n",
    "            count += 1\n",
    "            print(count*5/2, \"% Complete!\")\n",
    "print(info_list)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "for i in info_list:\n",
    "    print(i)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Visualizing Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "import itertools\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "objects = ('ReLu', 'SELU', 'ELU', 'Softplus', 'Softmax')\n",
    "y_pos = np.arange(len(objects))\n",
    "performance = [90.47,95.24,76.19, 76.19, 90]\n",
    " \n",
    "## IMPORTANT DATA\n",
    "# ELU AVERAGE - 29.5\n",
    "# ADAM AVERAGE - 25.7\n",
    "# -------\n",
    "# ReLU - 23.81\n",
    "# SELU - 26.19\n",
    "# Softplus - 19.05\n",
    "# Softmax - 26.19\n",
    "\n",
    "#plt.bar(y_pos, performance, color = 'g', align='center', alpha=0.5)\n",
    "#plt.xticks(y_pos, objects)\n",
    "n_groups = 5\n",
    "fig, ax = plt.subplots()\n",
    "index = np.arange(n_groups)\n",
    "bar_width = 0.2\n",
    "opacity = 0.8\n",
    "rms_vals = (33.33, 23.81, 33.33, 19.05, 14.29) #Average - 24.762 \n",
    "rects1 = plt.bar(index-0.2, rms_vals, bar_width,\n",
    "                 alpha=opacity,\n",
    "                 color='r',\n",
    "                 label='RMSProp')\n",
    "adam_vals = (19.05, 33.33, 33.33, 14.29, 28.57) #Average - 25.714 \n",
    "rects2 = plt.bar(index, adam_vals, bar_width,\n",
    "                 alpha=opacity,\n",
    "                 color='g',\n",
    "                 label='Adam')\n",
    "adagrad_vals = (23.81, 23.81, 33.33, 23.81, 23.81) #Average - 25.714 \n",
    "rects3 = plt.bar(index + 0.2, adagrad_vals, bar_width,\n",
    "                 alpha=opacity,\n",
    "                 color='c',\n",
    "                 label='Adagrad')\n",
    "adamax_vals = (19.05, 23.81, 23.81, 19.05, 38.1) #Average - 24.764\n",
    "rects4 = plt.bar(index + 0.4, adamax_vals, bar_width,\n",
    "                 alpha=opacity,\n",
    "                 color='m',\n",
    "                 label='Adamax')\n",
    "plt.plot(objects, (23.81, 26.19, 29.532, 19.05, 26.19) , label = \"Average\", c= \"k\")\n",
    "plt.xlabel('Layer Activation Functions', size = 22)\n",
    "#plt.ylabel('Accuracy (Unoptimized Learning Rate, Batch Size, Epoch)', size = 17)\n",
    "#plt.suptitle('Keras Neural Network Sequential Model Accuracy Under Different\\nOptimizers, Activations and Mean_Absolute_Error Loss', size = 25)\n",
    "plt.ylabel('Accuracy (Unoptimized Hyperparameters)', size = 23)\n",
    "\n",
    "plt.suptitle('Keras Neural Network Sequential Model Accuracy Under\\nDifferent Optimizers and Activation Functions', size = 25)\n",
    "\n",
    "plt.xticks(index + bar_width, objects, fontsize=18)\n",
    "plt.yticks([0,5,10,15,20,25,30,35,40], [0,5,10,15,20,25,30,35,40], fontsize=18)\n",
    "leg = plt.legend(title = \"Optimization Algorithms\", loc = 1, prop={'size': 18})\n",
    "plt.rcParams[\"figure.figsize\"] = [16, 9]\n",
    "#plt.tight_layout()\n",
    "plt.show()\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Average Accuracies for relu, selu, elu, softplus, softmax"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "x_axis = range(1, 8) # x_axis values\n",
    "%matplotlib inline\n",
    "#x-values, y-values, Name for legend, color\n",
    "plt.plot(x_axis, relu_score , label = \"ReLu RMSProp\", c= \"g\") #Plots a green line\n",
    "plt.plot(x_axis, softmax_score, label = \"Softmax Adamax\", c= \"b\")  #Plots a blue line\n",
    "plt.xlabel('[0.0001,      0.001,      0.002,      0.005,      0.01,      0.05,      0.1]')\n",
    "plt.ylabel('Accuracy')\n",
    "plt.title(\"Optimal learning rate for Adamax Optimizer and RMSProp Optimizer.\")\n",
    "plt.legend()\n",
    "print(\"The best is: Softmax, Adamax, mean_absolute_error with learning_rate of 0.05.\\nadamax = optimizers.Adamax(lr= 0.05, beta_1=0.9, beta_2=0.999, epsilon=None, decay=0.0)\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Optimal values were softmax activation, adamax optimizer, and mean_absolute_error.\n",
    "#optimizers.Adamax(lr=0.002, beta_1=0.9, beta_2=0.999, epsilon=None, decay=0.0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Automated Testing - Learning Rate\n",
    "### This automated test is for ReLu, RMSProp, mean_squared_error and SoftMax, Adamax, mean_absolute_error for different learning rates."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "relu_score = []\n",
    "softmax_score = []\n",
    "rms = optimizers.RMSprop(lr=0.001, rho=0.9, epsilon=None, decay=0.0)\n",
    "adamax =optimizers.Adamax(lr=0.002, beta_1=0.9, beta_2=0.999, epsilon=None, decay=0.0)\n",
    "learning_rates = [0.0001, 0.001, 0.002, 0.005, 0.01, 0.05, 0.1]\n",
    "counter = 0;\n",
    "for i in range(len(learning_rates)):\n",
    "    model = Sequential()\n",
    "    model.add(Dense(8, activation = \"relu\", input_dim = 4)) #input_dim is # of features\n",
    "    model.add(Dense(1)) \n",
    "    counter += 1\n",
    "    print(\"ReLu\", learning_rates[i], \"{:.4f}% Complete\".format(100*counter/14),\"------------------------------------------------------------------\")\n",
    "    rms = optimizers.RMSprop(lr= learning_rates[i], rho=0.9, epsilon=None, decay=0.0)\n",
    "    model.compile(optimizer = rms, loss = 'mean_squared_error', metrics = ['accuracy'])\n",
    "    model.fit(X_train, y_train, epochs = 2500, batch_size = 10, class_weight = None)\n",
    "    scores = model.evaluate(X_test, y_test)\n",
    "    relu_score.append(scores[1]*100)\n",
    "    print(scores[1]*100, learning_rates[i])\n",
    "for i in range(len(learning_rates)):\n",
    "    model = Sequential()\n",
    "    model.add(Dense(8, activation = \"softmax\", input_dim = 4)) #input_dim is # of features\n",
    "    model.add(Dense(1)) \n",
    "    counter += 1\n",
    "    print(\"SoftMax\", learning_rates[i], \"{:.4f}% Complete\".format(100*counter/14),\"------------------------------------------------------------------\")\n",
    "    adamax = optimizers.Adamax(lr= learning_rates[i], beta_1=0.9, beta_2=0.999, epsilon=None, decay=0.0)\n",
    "    model.compile(optimizer = adamax, loss = 'mean_absolute_error', metrics = ['accuracy'])\n",
    "    model.fit(X_train, y_train, epochs = 2500, batch_size = 10, class_weight = None)\n",
    "    scores = model.evaluate(X_test, y_test)\n",
    "    softmax_score.append(scores[1]*100)\n",
    "    print(scores[1]*100, learning_rates[i])\n",
    "print(relu_score)\n",
    "print(softmax_score)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(relu_score)\n",
    "print(softmax_score)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "x_axis = range(1, 8) # x_axis values\n",
    "%matplotlib inline\n",
    "#x-values, y-values, Name for legend, color\n",
    "plt.plot(x_axis, relu_score , label = \"ReLu RMSProp\", c= \"g\") #Plots a green line\n",
    "plt.plot(x_axis, softmax_score, label = \"Softmax Adamax\", c= \"b\")  #Plots a blue line\n",
    "plt.xlabel('[0.0001,      0.001,      0.002,      0.005,      0.01,      0.05,      0.1]')\n",
    "plt.ylabel('Accuracy')\n",
    "plt.title(\"Optimal learning rate for Adamax Optimizer and RMSProp Optimizer.\")\n",
    "plt.legend()\n",
    "print(\"The best is: Softmax, Adamax, mean_absolute_error with learning_rate of 0.05.\\nadamax = optimizers.Adamax(lr= 0.05, beta_1=0.9, beta_2=0.999, epsilon=None, decay=0.0)\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Automated Testing - Batch Size\n",
    "### This automated test is for ReLu, RMSProp, mean_squared_error and SoftMax, Adamax, mean_absolute_error for different batch sizes.¶"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "rms_score = []\n",
    "adamax_score = []\n",
    "rms = optimizers.RMSprop(lr=0.001, rho=0.9, epsilon=None, decay=0.0)\n",
    "adamax =optimizers.Adamax(lr=0.002, beta_1=0.9, beta_2=0.999, epsilon=None, decay=0.0)\n",
    "batch_sizes = [5, 10, 15, 20, 30, 50, 75, 100]\n",
    "counter = 0;\n",
    "for i in range(len(batch_sizes)):\n",
    "    model = Sequential()\n",
    "    model.add(Dense(8, activation = \"relu\", input_dim = 4)) #input_dim is # of features\n",
    "    model.add(Dense(1)) \n",
    "    counter += 1\n",
    "    print(\"ReLu\", batch_sizes[i], \"{:.4f}% Complete\".format(100*counter/16),\"------------------------------------------------------------------\")\n",
    "    rms = optimizers.RMSprop(lr= 0.01, rho=0.9, epsilon=None, decay=0.0)\n",
    "    model.compile(optimizer = rms, loss = 'mean_squared_error', metrics = ['accuracy'])\n",
    "    model.fit(X_train, y_train, epochs = 2500, batch_size = batch_sizes[i], class_weight = None)\n",
    "    scores = model.evaluate(X_test, y_test)\n",
    "    rms_score.append(scores[1]*100)\n",
    "    print(scores[1]*100, batch_sizes[i])\n",
    "for i in range(len(batch_sizes)):\n",
    "    model = Sequential()\n",
    "    model.add(Dense(8, activation = \"softmax\", input_dim = 4)) #input_dim is # of features\n",
    "    model.add(Dense(1)) \n",
    "    counter += 1\n",
    "    print(\"SoftMax\", batch_sizes[i], \"{:.4f}% Complete\".format(100*counter/16),\"------------------------------------------------------------------\")\n",
    "    adamax = optimizers.Adamax(lr= 0.05, beta_1=0.9, beta_2=0.999, epsilon=None, decay=0.0)\n",
    "    model.compile(optimizer = adamax, loss = 'mean_absolute_error', metrics = ['accuracy'])\n",
    "    model.fit(X_train, y_train, epochs = 2500, batch_size = batch_sizes[i], class_weight = None)\n",
    "    scores = model.evaluate(X_test, y_test)\n",
    "    adamax_score.append(scores[1]*100)\n",
    "    print(scores[1]*100, batch_sizes[i])\n",
    "print(rms_score)\n",
    "print(adamax_score)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(rms_score)\n",
    "print(adamax_score)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "x_axis = range(1, 9) # x_axis values\n",
    "%matplotlib inline\n",
    "#x-values, y-values, Name for legend, color\n",
    "plt.plot(x_axis, rms_score , label = \"ReLu RMSProp\", c= \"g\") #Plots a green line\n",
    "plt.plot(x_axis, adamax_score, label = \"Softmax Adamax\", c= \"b\")  #Plots a blue line\n",
    "plt.xlabel('[5        10         15           20          30           50          75      100]')\n",
    "plt.ylabel('Accuracy')\n",
    "plt.title(\"Optimal batch size for Adamax Optimizer and RMSProp Optimizer.\")\n",
    "plt.legend()\n",
    "print(\"The best is: Softmax, Adamax, mean_absolute_error with batch size of 5 and learning_rate of 0.05.\\nadamax = optimizers.Adamax(lr= 0.05, beta_1=0.9, beta_2=0.999, epsilon=None, decay=0.0)\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Automated Testing - Epoch Size\n",
    "### This test is for improving the accuracy by finding the best epoch value"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "erms_score = []\n",
    "eadamax_score = []\n",
    "rms = optimizers.RMSprop(lr=0.001, rho=0.9, epsilon=None, decay=0.0)\n",
    "adamax =optimizers.Adamax(lr=0.002, beta_1=0.9, beta_2=0.999, epsilon=None, decay=0.0)\n",
    "epoches = [500, 1000, 2500, 5000, 8000]\n",
    "counter = 0;\n",
    "for i in range(len(epoches)):\n",
    "    model = Sequential()\n",
    "    model.add(Dense(8, activation = \"relu\", input_dim = 4)) #input_dim is # of features\n",
    "    model.add(Dense(1)) \n",
    "    counter += 1\n",
    "    print(\"ReLu\", epoches[i], \"{:.1f}% Complete\".format(100*counter/10),\"------------------------------------------------------------------\")\n",
    "    rms = optimizers.RMSprop(lr= 0.01, rho=0.9, epsilon=None, decay=0.0)\n",
    "    model.compile(optimizer = rms, loss = 'mean_squared_error', metrics = ['accuracy'])\n",
    "    model.fit(X_train, y_train, epochs = epoches[i], batch_size = 5, class_weight = None)\n",
    "    scores = model.evaluate(X_test, y_test)\n",
    "    erms_score.append(scores[1]*100)\n",
    "    print(scores[1]*100, epoches[i])\n",
    "for i in range(len(epoches)):\n",
    "    model = Sequential()\n",
    "    model.add(Dense(8, activation = \"softmax\", input_dim = 4)) #input_dim is # of features\n",
    "    model.add(Dense(1)) \n",
    "    counter += 1\n",
    "    print(\"SoftMax\", epoches[i], \"{:.1f}% Complete\".format(100*counter/10),\"------------------------------------------------------------------\")\n",
    "    adamax = optimizers.Adamax(lr= 0.05, beta_1=0.9, beta_2=0.999, epsilon=None, decay=0.0)\n",
    "    model.compile(optimizer = adamax, loss = 'mean_absolute_error', metrics = ['accuracy'])\n",
    "    model.fit(X_train, y_train, epochs = epoches[i], batch_size = 5, class_weight = None)\n",
    "    scores = model.evaluate(X_test, y_test)\n",
    "    eadamax_score.append(scores[1]*100)\n",
    "    print(scores[1]*100, epoches[i])\n",
    "print(erms_score)\n",
    "print(eadamax_score)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(erms_score)\n",
    "print(eadamax_score)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "x_axis = range(1, 6) # x_axis values\n",
    "%matplotlib inline\n",
    "#x-values, y-values, Name for legend, color\n",
    "plt.plot(x_axis, erms_score , label = \"ReLu RMSProp\", c= \"g\") #Plots a green line\n",
    "plt.plot(x_axis, eadamax_score, label = \"Softmax Adamax\", c= \"b\")  #Plots a blue line\n",
    "plt.xlabel('[500               1000               2500                 5000               8000]')\n",
    "plt.ylabel('Accuracy')\n",
    "plt.rcParams[\"figure.figsize\"] = [15, 9]\n",
    "plt.title(\"Optimal Epoch Number for Adamax Optimizer and RMSProp Optimizer.\")\n",
    "plt.legend()\n",
    "plt.figure(num=None, figsize=(15, 9), dpi=80, facecolor='w', edgecolor='k')\n",
    "plt.show()\n",
    "print(\"The best is: Softmax, Adamax, mean_absolute_error with batch size of 5 and learning_rate of 0.05.\\nadamax = optimizers.Adamax(lr= 0.05, beta_1=0.9, beta_2=0.999, epsilon=None, decay=0.0)\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Results for Accuracy based on Softmax Activated Layers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "layer_scores = [23.80952388048172, 19.0476194024086, 42.85714328289032, 23.80952388048172, 57.14285969734192, 47.61904776096344, 76.1904776096344, 52.3809552192688, 71.42857313156128, 61.90476417541504, 71.42857313156128, 71.42857313156128, 76.1904776096344, 42.85714328289032, 71.42857313156128]\n",
    "import matplotlib.pyplot as plt\n",
    "from matplotlib.pyplot import figure\n",
    "x_axis = range(1, 16) # x_axis values\n",
    "%matplotlib inline\n",
    "#x-values, y-values, Name for legend, color\n",
    "plt.plot(x_axis, layer_scores , label = \"Softmax, Adamax\", c= \"b\") #Plots a green line\n",
    "plt.xlabel('Number of Layers', size = 13)\n",
    "plt.ylabel('Accuracy', size = 13)\n",
    "plt.title(\"Accuracy of the Sequential Model with Different Numbers of Softmax-Activated Layers\", size = 10)\n",
    "plt.legend()\n",
    "#plt.figure(num=None, figsize=(15, 9), dpi=500, facecolor='w', edgecolor='k')\n",
    "plt.rcParams[\"figure.figsize\"] = [16, 9]\n",
    "#plt.tight_layout()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Optimal Model out of All Tests"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = Sequential()\n",
    "model.add(Dense(7, activation = \"softmax\", input_dim = 4)) #input_dim is # of features\n",
    "model.add(Dense(1)) \n",
    "adamax = optimizers.Adamax(lr= 0.05, beta_1=0.9, beta_2=0.999, epsilon=None, decay=0.0)\n",
    "model.compile(optimizer = adamax, loss = 'mean_absolute_error', metrics = ['accuracy'])\n",
    "model.fit(X_train, y_train, epochs = 5000, batch_size = 5, class_weight = None)\n",
    "scores = model.evaluate(X_test, y_test)\n",
    "print(\"ACCURACY: {:.2f}\".format(scores[1]*100))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = Sequential()\n",
    "model.add(Dense(7, activation = \"softmax\", input_dim = 4)) #input_dim is # of features\n",
    "model.add(Dense(1)) \n",
    "adamax = optimizers.Adamax(lr= 0.05, beta_1=0.9, beta_2=0.999, epsilon=None, decay=0.0)\n",
    "model.compile(optimizer = adamax, loss = 'mean_absolute_error', metrics = ['accuracy'])\n",
    "model.fit(X_train, y_train, epochs = 5000, batch_size = 5, class_weight = None)\n",
    "scores2 = model.evaluate(X_test, y_test)\n",
    "print(\"ACCURACY: {:.2f}\".format(scores2[1]*100))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}