11 - Tune Hyperparameters.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Tune Hyperparameters\n",
    "\n",
    "There are many machine learning algorithms that require *hyperparameters* (parameter values that influence training, but can't be determined from the training data itself). For example, when training a logistic regression model, you can use a *regularization rate* hyperparameter to counteract bias in the model; or when training a convolutional neural network, you can use hyperparameters like *learning rate* and *batch size* to control how weights are adjusted and how many data items are processed in a mini-batch respectively. The choice of hyperparameter values can significantly affect the performance of a trained model, or the time taken to train it; and often you need to try multiple combinations to find the optimal solution.\n",
    "\n",
    "In this case, you'll train a classification model with two hyperparameters, but the principles apply to any kind of model you can train with Azure Machine Learning.\n",
    "\n",
    "## Install the Azure Machine Learning SDK\n",
    "\n",
    "The Azure Machine Learning SDK is updated frequently. Run the following cell to upgrade to the latest release, along with the additional package to support notebook widgets."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install --upgrade azureml-sdk azureml-widgets"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Connect to your workspace\n",
    "\n",
    "With the latest version of the SDK installed, now you're ready to connect to your workspace.\n",
    "\n",
    "> **Note**: If you haven't already established an authenticated session with your Azure subscription, you'll be prompted to authenticate by clicking a link, entering an authentication code, and signing into Azure."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import azureml.core\n",
    "from azureml.core import Workspace\n",
    "\n",
    "# Load the workspace from the saved config file\n",
    "ws = Workspace.from_config()\n",
    "print('Ready to use Azure ML {} to work with {}'.format(azureml.core.VERSION, ws.name))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Prepare data\n",
    "\n",
    "In this lab, you'll use a dataset containing details of diabetes patients. Run the cell below to create this dataset (if it already exists, the existing version will be used)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from azureml.core import Dataset\n",
    "\n",
    "default_ds = ws.get_default_datastore()\n",
    "\n",
    "if 'diabetes dataset' not in ws.datasets:\n",
    "    default_ds.upload_files(files=['./data/diabetes.csv', './data/diabetes2.csv'], # Upload the diabetes csv files in /data\n",
    "                        target_path='diabetes-data/', # Put it in a folder path in the datastore\n",
    "                        overwrite=True, # Replace existing files of the same name\n",
    "                        show_progress=True)\n",
    "\n",
    "    #Create a tabular dataset from the path on the datastore (this may take a short while)\n",
    "    tab_data_set = Dataset.Tabular.from_delimited_files(path=(default_ds, 'diabetes-data/*.csv'))\n",
    "\n",
    "    # Register the tabular dataset\n",
    "    try:\n",
    "        tab_data_set = tab_data_set.register(workspace=ws, \n",
    "                                name='diabetes dataset',\n",
    "                                description='diabetes data',\n",
    "                                tags = {'format':'CSV'},\n",
    "                                create_new_version=True)\n",
    "        print('Dataset registered.')\n",
    "    except Exception as ex:\n",
    "        print(ex)\n",
    "else:\n",
    "    print('Dataset already registered.')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Prepare a training script\n",
    "\n",
    "Now let's create a folder for the training script you'll use to train the model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "\n",
    "experiment_folder = 'diabetes_training-hyperdrive'\n",
    "os.makedirs(experiment_folder, exist_ok=True)\n",
    "\n",
    "print('Folder ready.')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now create the Python script to train the model. In this example, you'll use a *Gradient Boosting* algorithm to train a classification model. The script must include:\n",
    "\n",
    "- An argument for each hyperparameter you want to optimize (in this case, the learning rate and number of estimators for the Gradient Boosting algorithm)\n",
    "- Code to log the performance metric you want to optimize for (in this case, you'll log both AUC and accuracy, so you can choose to optimize the model for either of these)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%writefile $experiment_folder/diabetes_training.py\n",
    "# Import libraries\n",
    "import argparse, joblib, os\n",
    "from azureml.core import Run\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.ensemble import GradientBoostingClassifier\n",
    "from sklearn.metrics import roc_auc_score, roc_curve\n",
    "\n",
    "# Get the experiment run context\n",
    "run = Run.get_context()\n",
    "\n",
    "# Get script arguments\n",
    "parser = argparse.ArgumentParser()\n",
    "\n",
    "# Input dataset\n",
    "parser.add_argument(\"--input-data\", type=str, dest='input_data', help='training dataset')\n",
    "\n",
    "# Hyperparameters\n",
    "parser.add_argument('--learning_rate', type=float, dest='learning_rate', default=0.1, help='learning rate')\n",
    "parser.add_argument('--n_estimators', type=int, dest='n_estimators', default=100, help='number of estimators')\n",
    "\n",
    "# Add arguments to args collection\n",
    "args = parser.parse_args()\n",
    "\n",
    "# Log Hyperparameter values\n",
    "run.log('learning_rate',  np.float(args.learning_rate))\n",
    "run.log('n_estimators',  np.int(args.n_estimators))\n",
    "\n",
    "# load the diabetes dataset\n",
    "print(\"Loading Data...\")\n",
    "diabetes = run.input_datasets['training_data'].to_pandas_dataframe() # Get the training data from the estimator input\n",
    "\n",
    "# Separate features and labels\n",
    "X, y = diabetes[['Pregnancies','PlasmaGlucose','DiastolicBloodPressure','TricepsThickness','SerumInsulin','BMI','DiabetesPedigree','Age']].values, diabetes['Diabetic'].values\n",
    "\n",
    "# Split data into training set and test set\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.30, random_state=0)\n",
    "\n",
    "# Train a Gradient Boosting classification model with the specified hyperparameters\n",
    "print('Training a classification model')\n",
    "model = GradientBoostingClassifier(learning_rate=args.learning_rate,\n",
    "                                   n_estimators=args.n_estimators).fit(X_train, y_train)\n",
    "\n",
    "# calculate accuracy\n",
    "y_hat = model.predict(X_test)\n",
    "acc = np.average(y_hat == y_test)\n",
    "print('Accuracy:', acc)\n",
    "run.log('Accuracy', np.float(acc))\n",
    "\n",
    "# calculate AUC\n",
    "y_scores = model.predict_proba(X_test)\n",
    "auc = roc_auc_score(y_test,y_scores[:,1])\n",
    "print('AUC: ' + str(auc))\n",
    "run.log('AUC', np.float(auc))\n",
    "\n",
    "# Save the model in the run outputs\n",
    "os.makedirs('outputs', exist_ok=True)\n",
    "joblib.dump(value=model, filename='outputs/diabetes_model.pkl')\n",
    "\n",
    "run.complete()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create compute\n",
    "\n",
    "Hyperparameter tuning involves running multiple training iterations with different hyperparameter values and comparing the performance metrics of the resulting models. To do this efficiently, we'll take advantage of on-demand cloud compute and create a cluster - this will allow multiple training iterations to be run concurrently.\n",
    "\n",
    "Use the following code to specify an Azure Machine Learning compute cluster (it will be created if it doesn't already exist).\n",
    "\n",
    "> **Important**: Change *your-compute-cluster* to the name of your compute cluster in the code below before running it! Cluster names must be globally unique names between 2 to 16 characters in length. Valid characters are letters, digits, and the - character."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from azureml.core.compute import ComputeTarget, AmlCompute\n",
    "from azureml.core.compute_target import ComputeTargetException\n",
    "\n",
    "cluster_name = \"your-compute-cluster\"\n",
    "\n",
    "try:\n",
    "    # Check for existing compute target\n",
    "    training_cluster = ComputeTarget(workspace=ws, name=cluster_name)\n",
    "    print('Found existing cluster, use it.')\n",
    "except ComputeTargetException:\n",
    "    # If it doesn't already exist, create it\n",
    "    try:\n",
    "        compute_config = AmlCompute.provisioning_configuration(vm_size='STANDARD_DS11_V2', max_nodes=2)\n",
    "        training_cluster = ComputeTarget.create(ws, cluster_name, compute_config)\n",
    "        training_cluster.wait_for_completion(show_output=True)\n",
    "    except Exception as ex:\n",
    "        print(ex)\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Run a hyperparameter tuning experiment\n",
    "\n",
    "Azure Machine Learning includes a hyperparameter tuning capability through *hyperdrive* experiments. These experiments launch multiple child runs, each with a different hyperparameter combination. The run producing the best model (as determined by the logged target performance metric for which you want to optimize) can be identified, and its trained model selected for registration and deployment.\n",
    "\n",
    "> **Mote**: In this example, we aren't specifying an early stopping policy. Such a policy is only relevant if the training script performs multiple training iterations, logging the primary metric for each iteration. This approach is typically employed when training deep neural network models over multiple *epochs*."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from azureml.core import Experiment, ScriptRunConfig, Environment\n",
    "from azureml.core.conda_dependencies import CondaDependencies\n",
    "from azureml.train.hyperdrive import GridParameterSampling, HyperDriveConfig, PrimaryMetricGoal, choice\n",
    "from azureml.widgets import RunDetails\n",
    "\n",
    "# Create a Python environment for the experiment\n",
    "sklearn_env = Environment(\"sklearn-env\")\n",
    "\n",
    "# Ensure the required packages are installed (we need scikit-learn, Azure ML defaults, and Azure ML dataprep)\n",
    "packages = CondaDependencies.create(conda_packages=['scikit-learn','pip'],\n",
    "                                    pip_packages=['azureml-defaults','azureml-dataprep[pandas]'])\n",
    "sklearn_env.python.conda_dependencies = packages\n",
    "\n",
    "# Get the training dataset\n",
    "diabetes_ds = ws.datasets.get(\"diabetes dataset\")\n",
    "\n",
    "# Create a script config\n",
    "script_config = ScriptRunConfig(source_directory=experiment_folder,\n",
    "                                script='diabetes_training.py',\n",
    "                                # Add non-hyperparameter arguments -in this case, the training dataset\n",
    "                                arguments = ['--input-data', diabetes_ds.as_named_input('training_data')],\n",
    "                                environment=sklearn_env,\n",
    "                                compute_target = training_cluster)\n",
    "\n",
    "# Sample a range of parameter values\n",
    "params = GridParameterSampling(\n",
    "    {\n",
    "        # Hyperdrive will try 6 combinations, adding these as script arguments\n",
    "        '--learning_rate': choice(0.01, 0.1, 1.0),\n",
    "        '--n_estimators' : choice(10, 100)\n",
    "    }\n",
    ")\n",
    "\n",
    "# Configure hyperdrive settings\n",
    "hyperdrive = HyperDriveConfig(run_config=script_config, \n",
    "                          hyperparameter_sampling=params, \n",
    "                          policy=None, # No early stopping policy\n",
    "                          primary_metric_name='AUC', # Find the highest AUC metric\n",
    "                          primary_metric_goal=PrimaryMetricGoal.MAXIMIZE, \n",
    "                          max_total_runs=6, # Restict the experiment to 6 iterations\n",
    "                          max_concurrent_runs=2) # Run up to 2 iterations in parallel\n",
    "\n",
    "# Run the experiment\n",
    "experiment = Experiment(workspace=ws, name='mslearn-diabetes-hyperdrive')\n",
    "run = experiment.submit(config=hyperdrive)\n",
    "\n",
    "# Show the status in the notebook as the experiment runs\n",
    "RunDetails(run).show()\n",
    "run.wait_for_completion()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can view the experiment run status in the widget above. You can also view the main Hyperdrive experiment run and its child runs in [Azure Machine Learning studio](https://ml.azure.com).\n",
    "\n",
    "> **Note**: If a message indicating that a non-numeric can't be visualized is displayed, you can ignore it.\n",
    "\n",
    "## Determine the best performing run\n",
    "\n",
    "When all of the runs have finished, you can find the best one based on the performance metric you specified (in this case, the one with the best AUC)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Print all child runs, sorted by the primary metric\n",
    "for child_run in run.get_children_sorted_by_primary_metric():\n",
    "    print(child_run)\n",
    "\n",
    "# Get the best run, and its metrics and arguments\n",
    "best_run = run.get_best_run_by_primary_metric()\n",
    "best_run_metrics = best_run.get_metrics()\n",
    "script_arguments = best_run.get_details() ['runDefinition']['arguments']\n",
    "print('Best Run Id: ', best_run.id)\n",
    "print(' -AUC:', best_run_metrics['AUC'])\n",
    "print(' -Accuracy:', best_run_metrics['Accuracy'])\n",
    "print(' -Arguments:',script_arguments)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that you've found the best run, you can register the model it trained."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from azureml.core import Model\n",
    "\n",
    "# Register model\n",
    "best_run.register_model(model_path='outputs/diabetes_model.pkl', model_name='diabetes_model',\n",
    "                        tags={'Training context':'Hyperdrive'},\n",
    "                        properties={'AUC': best_run_metrics['AUC'], 'Accuracy': best_run_metrics['Accuracy']})\n",
    "\n",
    "# List registered models\n",
    "for model in Model.list(ws):\n",
    "    print(model.name, 'version:', model.version)\n",
    "    for tag_name in model.tags:\n",
    "        tag = model.tags[tag_name]\n",
    "        print ('\\t',tag_name, ':', tag)\n",
    "    for prop_name in model.properties:\n",
    "        prop = model.properties[prop_name]\n",
    "        print ('\\t',prop_name, ':', prop)\n",
    "    print('\\n')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> **More Information**: For more information about Hyperdrive, see the [Azure ML documentation](https://docs.microsoft.com/azure/machine-learning/how-to-tune-hyperparameters)."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3.6 - AzureML",
   "language": "python",
   "name": "python3-azureml"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}