notebooks/POC Use Xingu.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "6a854604",
   "metadata": {},
   "source": [
    "# POC Use Xingu\n",
    "\n",
    "Use this notebook to get you started with Xingu.\n",
    "\n",
    "**Do not use it here. Copy this notebook and feel free to edit, modify and experiment in there.**\n",
    "\n",
    "**Never commit your changes to the repo**, since this notebook is just a standard example and will be used by others to play with Xingu.\n",
    "\n",
    "<hr/>\n",
    "\n",
    "Start by importing configuration bundles to train, or batch predict, or explore metrics."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "12b3e96f",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# this is a simple `.py` file in notebooks folder, full of configuration parameters for Xingu\n",
    "import config_my_xingu"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f3224b8e",
   "metadata": {
    "tags": []
   },
   "source": [
    "## Setup environment\n",
    "### The `env` bundle controls locations for files and databases\n",
    "#### `config_my_xingu.bundles['env']['alpha_explorer']`\n",
    "Use when working on everyday Xingu improvements\n",
    "* Xingu database: local SQLite\n",
    "* DVC: off\n",
    "* Query cache: on, in `../data`\n",
    "* Trained models in: `../models`\n",
    "\n",
    "#### `config_my_xingu.bundles['env']['beta_explorer']`\n",
    "Use when working with staging database\n",
    "* Xingu database: staging PostgreSQL\n",
    "* DVC: on\n",
    "* Query cache: on, in `../data`\n",
    "* Trained models in: staging S3\n",
    "\n",
    "#### `config_my_xingu.bundles['env']['staging']`\n",
    "Similar to `beta_explorer`, used in GitHub staging workflow\n",
    "* Xingu database: staging PostgreSQL\n",
    "* DVC: on\n",
    "* Query cache: off\n",
    "* Trained models in: staging S3\n",
    "\n",
    "#### `config_my_xingu.bundles['env']['production']`\n",
    "Do not use in your laptop, this is just documented as how to configure for production\n",
    "* Xingu database: production PostgreSQL\n",
    "* DVC: on\n",
    "* Query cache: off\n",
    "* Trained models in: production S3\n",
    "\n",
    "### The `parallel` bundle controls parallelism and modus operandi\n",
    "#### `config_my_xingu.bundles['parallel']['train_and_predict']`\n",
    "Use when working on everyday Xingu improvements\n",
    "* Train: yes\n",
    "    * Train parallelism: maximum\n",
    "    * Hyper-parameters optimization: use what is found in DB, or estimator default\n",
    "* Post process (pickle, metrics etc): yes\n",
    "    * Batch predict: yes\n",
    "    * Post-process parallelism: maximum\n",
    "\n",
    "#### `config_my_xingu.bundles['parallel']['predict_only']`\n",
    "Use with pre-trained models\n",
    "* Train: no\n",
    "    * Hyper-parameters optimization: no\n",
    "* Post process (pickle, metrics etc): no\n",
    "    * Post-process parallelism: maximum\n",
    "    * Batch predict: yes\n",
    "\n",
    "#### `config_my_xingu.bundles['parallel']['hyper_optimize_only']`\n",
    "Use when working on hyper-parameters optimization\n",
    "* Train: yes\n",
    "    * Train parallelism: one model at a time\n",
    "    * Hyper-parameters optimization: compute\n",
    "    * Hyper-parameters optimization parallelism: maximum\n",
    "* Post process (pickle, metrics etc): no\n",
    "    * Batch predict: no\n",
    "\n",
    "#### `config_my_xingu.bundles['parallel']['do_all']`\n",
    "Use when working on hyper-parameters optimization\n",
    "* Train: yes\n",
    "    * Train parallelism: 3 models at a time\n",
    "    * Hyper-parameters optimization: compute\n",
    "    * Hyper-parameters optimization parallelism: 6\n",
    "* Post process (pickle, metrics etc): yes\n",
    "    * Post-process parallelism: 3 models at a time\n",
    "    * Batch predict: yes\n",
    "\n",
    "Choose one **env** bundle and one **parallel** bundle"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f2cbe98f",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import os\n",
    "import sys\n",
    "import pathlib\n",
    "\n",
    "os.environ.update(config_my_xingu.bundles['env']['alpha_explorer'])\n",
    "os.environ.update(config_my_xingu.bundles['parallel']['train_and_predict'])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d2c555e3",
   "metadata": {},
   "source": [
    "Amend anything you want to change. All values must be text."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b5ec05dd",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "os.environ.update(\n",
    "    dict(\n",
    "        HYPEROPT_STRATEGY     = 'dp',\n",
    "        BATCH_PREDICT         = 'false',\n",
    "        DEBUG                 = 'true'\n",
    "    )\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "836ae362",
   "metadata": {},
   "source": [
    "## Import Xingu and configure Logging\n",
    "Next line is required if `xingu` folder not in `PYTHON_PATH` or xingu not installed by pip."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "83376138",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Give priority to local packages (not needed in case Xingu was installed by pip)\n",
    "sys.path.insert(0, os.path.abspath(os.path.join(os.path.dirname(''), '..')))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a81e7c4d",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import logging\n",
    "import decouple\n",
    "import pandas as pd\n",
    "import numpy\n",
    "from xingu import DataProviderFactory\n",
    "from xingu import ConfigManager\n",
    "from xingu import Coach\n",
    "from xingu import Model\n",
    "\n",
    "%config InlineBackend.figure_format='retina'\n",
    "\n",
    "# Configure logging for Xingu\n",
    "logger=logging.getLogger('xingu')\n",
    "FORMATTER = logging.Formatter(\"%(asctime)s|%(levelname)s|%(name)s|%(message)s\")\n",
    "HANDLER = logging.StreamHandler()\n",
    "HANDLER.setFormatter(FORMATTER)\n",
    "logger.addHandler(HANDLER)\n",
    "logger.setLevel(logging.DEBUG)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "063214ad",
   "metadata": {
    "tags": []
   },
   "source": [
    "## POC 1. Train some Models\n",
    "A `Coach` is needed to train anything. Put more DataProvider IDs in the `data_providers` list. If you want to train models that have pre-reqs and are not training their dependencies in the same train session, pre-trained pre-req models will be efficiently loaded upfront."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a38fdf3f",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "data_providers=['mydp1']\n",
    "\n",
    "dpf=DataProviderFactory(providers_list=data_providers)\n",
    "coach=Coach(dpf)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a551177b",
   "metadata": {},
   "source": [
    "!rm xingu.db* ../models/*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "717fcc3a",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "%%time\n",
    "coach.team_train()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "15693466",
   "metadata": {},
   "source": [
    "Also try `config_my_xingu.bundles['parallel']['hyper_optimize_only']` config bundle to radically change what `team_train()` does.\n",
    "\n",
    "Trained models are here:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d50e8b22",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "coach.trained"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "af7fe6eb",
   "metadata": {
    "jp-MarkdownHeadingCollapsed": true
   },
   "source": [
    "Trained models can be used now to compute estimations."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c0c030e6",
   "metadata": {
    "tags": []
   },
   "source": [
    "## POC 2. Use Pre-Trained Models for Batch Predict\n",
    "\n",
    "Reset this notebook before continuing. Run again only cells before \"POC 1\" just to setup environment.\n",
    "\n",
    "A `Coach` is needed to eficiently load pre-trained models"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4e30ebf9",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "data_providers=['mydp1']\n",
    "dpf=DataProviderFactory(providers_list=data_providers)\n",
    "coach=Coach(dpf)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "022b91a3",
   "metadata": {},
   "source": [
    "Notice that `anuncios` is not in the `data_providers` list, but it will be loaded since it is a pre-req for `cartorios`, which is in the list.\n",
    "\n",
    "Pre-trained pickles will be search in and loaded from whatever is set in `TRAINED_MODELS_PATH` environment variable. This is usually set to `models` local folder or to some S3 URL.\n",
    "\n",
    "Models will be loaded in parallel."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "496e6d22",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "print(os.environ['TRAINED_MODELS_PATH'])\n",
    "coach.team_load()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "06205b32",
   "metadata": {},
   "source": [
    "Use embedded DataProvider to load some data. The following logic is barely what happens in `Model::batch_predict()` method. See also `Model::fit()` method for the training data preparation logic."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "efc6f4f8-deb2-43c3-89b6-e0fcf2c97356",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "model=coach.trained['mydp1']\n",
    "\n",
    "# Following line is here just to force use of cached parquet, if available\n",
    "model.context='batch_predict'\n",
    "\n",
    "# Get DP’s batch predict SQL queriesp\n",
    "dict_with_queries     = model.dp.get_dataset_sources_for_batch_predict()\n",
    "\n",
    "# Use queries to get multiple DataFrames\n",
    "dict_with_dataframes  = model.data_sources_to_data(dict_with_queries)\n",
    "\n",
    "# Integrate into one DataFrame and apply logic to clean data\n",
    "df                    = model.dp.clean_data_for_batch_predict(dict_with_dataframes)\n",
    "\n",
    "# Feature engineering\n",
    "df                    = model.dp.feature_engineering_for_batch_predict(df)\n",
    "\n",
    "# Resulting DataFrame used for batch predict\n",
    "df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fc6fd94f-43aa-434a-b786-fd45b45f2378",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "dict_with_dataframes"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d50e743b",
   "metadata": {},
   "source": [
    "Compute estimations, finaly"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bbb4d4d4",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Illustrative only. For you to see what pred_quantiles() does internally\n",
    "X_features=model.dp.get_features_list()\n",
    "\n",
    "# Don't need to filter by X_features, it will be filtered internally\n",
    "Y_pred=model.predict_proba(df)\n",
    "\n",
    "Y_pred"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6da7e4ee",
   "metadata": {},
   "source": [
    "### Compute metrics\n",
    "\n",
    "Put data in right places so we can use convenient internal methods"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "862cf321",
   "metadata": {},
   "outputs": [],
   "source": [
    "model.batch_predict_data=df\n",
    "model.batch_predict_valuations=Y_pred"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ef7db53a",
   "metadata": {},
   "source": [
    "Compute all metrics available for model, including methods provided by its DataProvider"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "98856d22",
   "metadata": {},
   "outputs": [],
   "source": [
    "model.compute_model_metrics()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cb8b2c19",
   "metadata": {},
   "outputs": [],
   "source": [
    "model.compute_estimation_metrics()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6a8fc5ed",
   "metadata": {},
   "source": [
    "If `model.sets['train']`, `model.sets['val']` and `model.sets['test']` are defined and have data, this should work too:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5acab05f",
   "metadata": {},
   "outputs": [],
   "source": [
    "model.compute_trainsets_model_metrics()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bafb4502-cffe-41c0-bdd7-3ed492ecbf76",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b647770f",
   "metadata": {},
   "source": [
    "## POC 3. Assess Metrics and create Comparative Reports\n",
    "Since all metrics are stored in DB, they can be assessed and compared.\n",
    "The `RobsonCoach` class has reporting tools."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9aa78ae4",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Get metrics from staging and development DB\n",
    "\n",
    "os.environ.update(\n",
    "    dict(\n",
    "        XINGU_DB_URL=config_my_xingu.bundles['env']['beta_explorer']['XINGU_DB_URL']\n",
    "    )\n",
    ")\n",
    "\n",
    "coach=Coach()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "16c2572c",
   "metadata": {},
   "source": [
    "Retrieve all metrics and metadata about 4 specific `train_id`s and show it in a comparative way."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2cd96f6c",
   "metadata": {},
   "outputs": [],
   "source": [
    "report=coach.report(train_ids=['salmon-participant','wise-jacquard'])\n",
    "\n",
    "display(report['meta'])\n",
    "display(report['metrics'])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bc2dc52c",
   "metadata": {},
   "source": [
    "### Display a subset of metrics: only the m² values for São Paulo."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "014a71fb",
   "metadata": {},
   "outputs": [],
   "source": [
    "report['metrics'][['value per m²:São Paulo' in s for s in report['metrics'].index]].xs('global', level='set', axis=1).dropna()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e7ae1466",
   "metadata": {
    "tags": []
   },
   "source": [
    "### Display a subset of metrics: only the ones related to the `test` split part."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8008a49a",
   "metadata": {},
   "outputs": [],
   "source": [
    "report['metrics'].xs('test', level='set', axis=1).dropna()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "eb61e861",
   "metadata": {},
   "source": [
    "### Save all metrics as Excel file"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "011fb93f",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Excel won't support time with timezone - how typical. Make it naïve.\n",
    "report['meta'].loc['time_utc']=report['meta'].loc['time_utc'].apply(lambda x: x.tz_convert(None))\n",
    "\n",
    "with pd.ExcelWriter(f'Metrics for Comitee Report — {pd.Timestamp.now().strftime(\"%Y.%m.%d-%H.%M.%S\")}.xlsx') as writer:\n",
    "\n",
    "    report_aux = report['meta'].sort_values(\"dataprovider_id\", axis=1)\n",
    "    report_aux.to_excel(writer, sheet_name=\"meta\")\n",
    "\n",
    "    dataprovider_list = list(set(report_aux.loc[\"dataprovider_id\", :]))\n",
    "\n",
    "    for dataprovider_id in dataprovider_list:\n",
    "\n",
    "        train_ids = list(report_aux.loc[:, report_aux.loc[\"dataprovider_id\", :] == dataprovider_id].columns)\n",
    "        train_session_ids = report_aux.loc[\"train_session_id\", report_aux.loc[\"dataprovider_id\", :] == dataprovider_id]\n",
    "\n",
    "        sheet = report[\"metrics\"].loc[:, report[\"metrics\"].columns.get_level_values(1).isin(train_ids)]\n",
    "\n",
    "        aux_list = {id: id + '|'+ train_session_ids[id] for id in sheet.columns.get_level_values(1)}\n",
    "\n",
    "        sheet = sheet.rename(columns=aux_list)\n",
    "\n",
    "        sheet.to_excel(writer, sheet_name=dataprovider_id)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "98e5289d",
   "metadata": {},
   "source": [
    "## POC 4. Check and report how Metrics evolved\n",
    "\n",
    "This example reports how metrics of same estimator evolved throughout time. We’ll use the production database."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c0e2318c",
   "metadata": {},
   "outputs": [],
   "source": [
    "os.environ.update(config_my_xingu.bundles['env']['production'])\n",
    "\n",
    "coach=Coach()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5667c93a",
   "metadata": {},
   "outputs": [],
   "source": [
    "dp='mydp1'\n",
    "\n",
    "query=\"\"\"\n",
    "    select * from metrics_model\n",
    "    where dataprovider_id = '{dp}'\n",
    "    -- and set='global';\n",
    "\"\"\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ad595a81",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Extract from DB\n",
    "report=pandas.read_sql(query.format(dp=dp),con=coach.get_db_connection('xingu'))\n",
    "\n",
    "# Make time human readable\n",
    "report['time']=pd.to_datetime(report['time'], unit='s', utc=True)\n",
    "\n",
    "# Display a simple evolution report with just OKRs\n",
    "print(f\"Evolution of metrics for {dp}\")\n",
    "\n",
    "(\n",
    "    report[report['name'].str.contains('OKR')]\n",
    "    .set_index(['name','time'])\n",
    "    .drop(columns='dataprovider_id train_session_id train_id set value_text'.split())\n",
    "    .unstack()\n",
    "    .sort_index()\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "af5251e4",
   "metadata": {},
   "source": [
    "How `OKR error > 15%:proportion` metric evolved through time"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ac81d9c2",
   "metadata": {},
   "outputs": [],
   "source": [
    "KPI=\"myKPI\"\n",
    "\n",
    "(\n",
    "    report\n",
    "    .query('name==@KPI')\n",
    "    [['time','value_number']]\n",
    "    .sort_values('time')\n",
    "    .set_index('time')\n",
    "    .plot\n",
    "    .line(title=f'{KPI} @ {dp}')\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f3f2452c",
   "metadata": {},
   "source": [
    "## POC 5. Play with Xingu barebones\n",
    "\n",
    "`Coach` is handy to coordinate full trains, full batch predict process (including metrics computation) and multi-model loading. But you can play with `Model` objects directly too. A coach is still needed for DB access, though."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "143d3609",
   "metadata": {},
   "outputs": [],
   "source": [
    "data_providers=['mydp1']\n",
    "\n",
    "dpf=DataProviderFactory(providers_list=data_providers)\n",
    "\n",
    "coach=Coach(dpf)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "30519e67",
   "metadata": {},
   "source": [
    "Get an untrained object for `anuncios_scs`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "51cd818e",
   "metadata": {},
   "outputs": [],
   "source": [
    "model=Model(\n",
    "    dp                     = next(coach.dp_factory.produce()),\n",
    "    coach                  = coach,\n",
    "    trained                = False,\n",
    "    delayed_prereq_binding = True\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5c25a02e",
   "metadata": {},
   "source": [
    "Manualy load and bind pre-req models"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b60ed73a",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Use the coach to load them efficiently\n",
    "# coach.team_load(explicit_list=model.dp.pre_req)\n",
    "\n",
    "# Bind them to current model\n",
    "model.load_pre_req_model()\n",
    "\n",
    "# See result\n",
    "model.dp.pre_req_model"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c1688273",
   "metadata": {},
   "source": [
    "Get DP’s SQL queries and related data, clean, integrate and engineer some features"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2e587943",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Following line is here just to force use of cached parquet, if available\n",
    "model.context='train_dataprep'\n",
    "\n",
    "# Get DP’s batch predict SQL queries\n",
    "dict_with_queries     = model.dp.get_dataset_sources_for_train()\n",
    "\n",
    "# Use queries to get multiple DataFrames\n",
    "dict_with_dataframes  = model.data_sources_to_data(dict_with_queries)\n",
    "\n",
    "# Integrate into one DataFrame and apply logic to clean data\n",
    "df                    = model.dp.clean_data_for_train(dict_with_dataframes)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6b6b5fbf",
   "metadata": {},
   "outputs": [],
   "source": [
    "df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1e7e71e5",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Feature engineering\n",
    "df=model.dp.feature_engineering_for_train(df)\n",
    "\n",
    "# Resulting DataFrame used for batch predict\n",
    "df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9d1502b3",
   "metadata": {},
   "outputs": [],
   "source": [
    "model.dp.data_split_for_train(df)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3f82f2b1",
   "metadata": {
    "tags": []
   },
   "source": [
    "## POC 6. Play with `ConfigManager`\n",
    "\n",
    "Reset this notebook before continuing. Run again only cells **before “POC 1”** just to setup environment.\n",
    "\n",
    "Here is `XINGU_DB_URL` env var with AWS secrets and parameters. Use `ConfigManager` to resolve them."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b93f39ce",
   "metadata": {},
   "outputs": [],
   "source": [
    "config_my_xingu.bundles['env']['beta_explorer']['XINGU_DB_URL']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f78abbb9",
   "metadata": {},
   "outputs": [],
   "source": [
    "os.environ.update(\n",
    "    dict(\n",
    "        XINGU_DB_URL = config_my_xingu.bundles['env']['beta_explorer']['XINGU_DB_URL']\n",
    "    )\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "80e7b7c6",
   "metadata": {},
   "outputs": [],
   "source": [
    "ConfigManager.get('XINGU_DB_URL')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b21bf477",
   "metadata": {},
   "source": [
    "One more try. Reset its cache first."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5adcfc63",
   "metadata": {},
   "outputs": [],
   "source": [
    "ConfigManager.cache={}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3500bb8b",
   "metadata": {},
   "outputs": [],
   "source": [
    "os.environ.update(\n",
    "    dict(\n",
    "        XYZ = '{%AWS_PARAM:xingu-staging-url%}/{%AWS_PARAM:xingu-staging-database-name%}'\n",
    "    )\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9155c38d",
   "metadata": {},
   "outputs": [],
   "source": [
    "ConfigManager.get('XYZ')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "04505673",
   "metadata": {},
   "source": [
    "## POC 7. Xingu Estimators in the Command Line\n",
    "\n",
    "### All Xingu features can be controlled in the command line; see them all here\n",
    "\n",
    "```shell\n",
    "xingu -h\n",
    "```\n",
    "\n",
    "### Train and Batch Predict 2 models in your laptop:\n",
    "\n",
    "This is fully parallel. One model will execute post-train actions (batch predict, data and pickle saving, metrics etc) while other model is being trained.\n",
    "\n",
    "```shell\n",
    "xingu \\\n",
    "    --models-db \"sqlite:///xingu.db?check_same_thread=False\" \\\n",
    "    --databases athena \"awsathena+rest://athena.us-east-1.amazonaws.com:443/mydatabase?work_group=mlops&compression=snappy\" \\\n",
    "    --databases databricks \"databricks+connector://token:dapi170fe...a3@abc123.cloud.databricks.com/default?http_path=/sql/1.0/endpoints/123abc\" \\\n",
    "    --datasource-cache-path data \\\n",
    "    --trained-models-path models \\\n",
    "    --debug \\\n",
    "    --project-home . \\\n",
    "    --dps mydp1,mydp2\n",
    "```\n",
    "\n",
    "### Batch Predict only in Production environment\n",
    "\n",
    "Note the `--no-train` parameter.\n",
    "\n",
    "```shell\n",
    "xingu \\\n",
    "    --no-train \\\n",
    "    --models-db \"postgresql+psycopg2://{%AWS_PARAM:xingu-production-user%}:{%AWS_SECRET:xingu-production-rds-secret%}@{%AWS_PARAM:xingu-production-url%}/{%AWS_PARAM:xingu-production-database-name%}\" \\\n",
    "    --databases athena \"awsathena+rest://athena.us-east-1.amazonaws.com:443/mydatabase?work_group=mlops&compression=snappy\" \\\n",
    "    --databases databricks \"databricks+connector://token:dapi170fe...a3@abc123.cloud.databricks.com/default?http_path=/sql/1.0/endpoints/123abc\" \\\n",
    "    --datasource-cache-path data \\\n",
    "    --trained-models-path \"s3://{%AWS_PARAM:xingu-production-bucket%}/trained-models\" \\\n",
    "    --debug \\\n",
    "    --project-home . \\\n",
    "    --dps mydp1,mydp2\n",
    "```\n",
    "\n",
    "### Hyper-parameters optimization only\n",
    "\n",
    "Notice how everything is turned off and disabled most parallelism to let Ray/SKOpt/Optimizer consume all CPUs\n",
    "\n",
    "```shell\n",
    "python3 -m xingu \\\n",
    "    --models-db \"sqlite:///xingu.db?check_same_thread=False\" \\\n",
    "    --databases athena \"awsathena+rest://athena.us-east-1.amazonaws.com:443/mydatabase?work_group=mlops&compression=snappy\" \\\n",
    "    --databases databricks \"databricks+connector://token:dapi170fe...a3@abc123.cloud.databricks.com/default?http_path=/sql/1.0/endpoints/123abc\" \\\n",
    "    --datasource-cache-path data \\\n",
    "    --trained-models-path models \\\n",
    "    --debug \\\n",
    "    --project-home . \\\n",
    "    --no-post-process \\\n",
    "    --no-batch-predict \\\n",
    "    --hyperopt-strategy self \\\n",
    "    --parallel-train-max-workers 1 \\\n",
    "    --dps mydp1,mydp2\n",
    "```\n",
    "\n",
    "### Control Parallelism\n",
    "\n",
    "Explore these options to avoid over-subscribing and over-loading your CPU and RAM.\n",
    "\n",
    "```shell\n",
    "python3 -m xingu \\\n",
    "    --models-db \"sqlite:///xingu.db?check_same_thread=False\" \\\n",
    "    --databases athena \"awsathena+rest://athena.us-east-1.amazonaws.com:443/mydatabase?work_group=mlops&compression=snappy\" \\\n",
    "    --databases databricks \"databricks+connector://token:dapi170fe...a3@abc123.cloud.databricks.com/default?http_path=/sql/1.0/endpoints/123abc\" \\\n",
    "    --query-cache-path data \\\n",
    "    --trained-models-path models \\\n",
    "    --debug \\\n",
    "    --project-home . \\\n",
    "    --hyperopt-strategy self \\\n",
    "    --parallel-train-max-workers 3 \\\n",
    "    --parallel-hyperopt-max-workers 6 \\\n",
    "    --parallel-post-process-max-workers 3 \\\n",
    "    --parallel-estimators-max-workers 3\n",
    "```\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ecce9fda-52c0-4878-bcc8-569447e7a005",
   "metadata": {},
   "source": [
    "## POC 8. Xingu with Docker and Containers\n",
    "\n",
    "Xingu was designed to run everywhere, from your Windows, Mac, Linux laptop with no database at all, to large cloud environments with multiple data lakes and object sotrage. This is not mandatory, but on clouds usually containers are used to ship applications in controlled and reproducible environments. So this is how to use Xingu with Docker.\n",
    "\n",
    "Make sure your project folder has all the pieces:\n",
    "\n",
    "```shell\n",
    "$ cd mymodels\n",
    "$ find\n",
    "dataproviders/mydp1.py\n",
    "dataproviders/mydp2.py\n",
    "models/\n",
    "data/\n",
    "plots/\n",
    ".env\n",
    "Dockerfile\n",
    "```\n",
    "\n",
    "Use [the `Dockerfile` on Xingu repository](https://github.com/avibrazil/xingu/blob/main/Dockerfile) as a starting point.\n",
    "\n",
    "The `.env` file may contain environment variables to configure Xingu. It is not mandatory and configuration can be supplied to Xingu as real environment variables (not in a file) or via its command line parameters.\n",
    "\n",
    "### Build a general container image named `xingu`\n",
    "Include in the `Dockerfile`´s `pip install` and `dnf install` everything that you´ll need except your own project files.\n",
    "```shell\n",
    "cat Dockerfile | docker build --build-arg UID=$(id -u) --build-arg USER=some_user_name -t xingu -\n",
    "```\n",
    "Now this image can be used multiple times in various Xingu projects.\n",
    "\n",
    "### Use the image to train and predict your Xingu models\n",
    "Here we mount the current folder (your project folder) into container´s `/home/some_user_name/mymodels`. And then run a plain `xingu` command that will execute the plan defined by your environment.\n",
    "```shell\n",
    "docker run \\\n",
    "    --mount type=bind,source=\"`pwd`\",destination=/home/some_user_name/mymodels \\\n",
    "    -t xingu \\\n",
    "    /bin/sh -c \"cd mymodels; xingu\"\n",
    "```\n",
    "\n",
    "Or overwrite your environment with some command line parameters, for example to train only one of your models:\n",
    "\n",
    "```shell\n",
    "docker run \\\n",
    "    --mount type=bind,source=\"`pwd`\",destination=/home/some_user_name/mymodels \\\n",
    "    -t xingu \\\n",
    "    /bin/sh -c \"cd mymodels; xingu -d --dps mymodel1\"\n",
    "```\n",
    "\n",
    "Or use pre-trained models to just batch predict, no training:\n",
    "\n",
    "```shell\n",
    "docker run \\\n",
    "    --mount type=bind,source=\"`pwd`\",destination=/home/some_user_name/mymodels \\\n",
    "    -t xingu \\\n",
    "    /bin/sh -c \"cd mymodels; xingu -d --dps mymodel1 --no-train\"\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d2a946ab",
   "metadata": {},
   "source": [
    "## POC 9. Deploy Xingu Data and Estimators between environments\n",
    "### Staging to Production\n",
    "\n",
    "```shell\n",
    "python -m xingu.deploy \\\n",
    "    --source-models-db \"postgresql+psycopg2://{%AWS_PARAM:xingu-staging-user%}:{%AWS_SECRET:xingu-staging-rds-secret%}@{%AWS_PARAM:xingu-staging-url%}/{%AWS_PARAM:xingu-staging-database-name%}\" \\\n",
    "    --target-models-db \"postgresql+psycopg2://{%AWS_PARAM:xingu-production-user%}:{%AWS_SECRET:xingu-production-rds-secret%}@{%AWS_PARAM:xingu-production-url%}/{%AWS_PARAM:xingu-production-database-name%}\" \\\n",
    "    --source-trained-models-path \"s3://{%AWS_PARAM:xingu-staging-bucket%}/trained-models\" \\\n",
    "    --target-trained-models-path \"s3://{%AWS_PARAM:xingu-production-bucket%}/trained-models\" \\\n",
    "    --project-home . \\\n",
    "    --debug\n",
    "```\n",
    "### Build API Container with Production Estimators\n",
    "Note how `--dps` is not being used, causing it to act on all DataProviders. Note the `--no-db` parameter, to not copy DB entries, because the production API doesn’t use the Xingu database.\n",
    "```shell\n",
    "git clone git@github.com:avibrazil/xingu.git;\n",
    "cd xingu;\n",
    "# Change to production branch\n",
    "git checkout deploy-command;\n",
    "\n",
    "python -m xingu.deploy \\\n",
    "    --source-models-db \"postgresql+psycopg2://{%AWS_PARAM:xingu-staging-user%}:{%AWS_SECRET:xingu-staging-rds-secret%}@{%AWS_PARAM:xingu-staging-url%}/{%AWS_PARAM:xingu-staging-database-name%}\" \\\n",
    "    --source-trained-models-path \"s3://{%AWS_PARAM:xingu-staging-bucket%}/trained-models\" \\\n",
    "    --target-trained-models-path models \\\n",
    "    --project-home . \\\n",
    "    --no-db \\\n",
    "    --debug;\n",
    "```\n",
    "\n",
    "### Production to Laptop or SageMaker\n",
    "```shell\n",
    "git clone git@github.com:avibrazil/xingu.git;\n",
    "cd xingu;\n",
    "# Change to production branch\n",
    "git checkout deploy-command;\n",
    "\n",
    "python -m xingu.deploy \\\n",
    "    --dps mydp1,mydp2 \\\n",
    "    --source-models-db \"postgresql+psycopg2://{%AWS_PARAM:xingu-staging-user%}:{%AWS_SECRET:xingu-staging-rds-secret%}@{%AWS_PARAM:xingu-staging-url%}/{%AWS_PARAM:xingu-staging-database-name%}\" \\\n",
    "    --target-models-db \"sqlite:///xingu.db?check_same_thread=False\" \\\n",
    "    --source-trained-models-path \"s3://{%AWS_PARAM:xingu-production-bucket%}/trained-models\" \\\n",
    "    --target-trained-models-path models \\\n",
    "    --project-home . \\\n",
    "    --debug;\n",
    "```\n",
    "\n",
    "### Staging to Laptop or SageMaker\n",
    "Manually edit `inventory.yaml` to correctly map desired `train_ids` to `dataprovider_ids`, and then:\n",
    "```shell\n",
    "python -m xingu.deploy \\\n",
    "    --dps mydp1,mydp2 \\\n",
    "    --source-models-db \"postgresql+psycopg2://{%AWS_PARAM:xingu-staging-user%}:{%AWS_SECRET:xingu-staging-rds-secret%}@{%AWS_PARAM:xingu-staging-url%}/{%AWS_PARAM:xingu-staging-database-name%}\" \\\n",
    "    --target-models-db \"sqlite:///xingu.db?check_same_thread=False\" \\\n",
    "    --source-trained-models-path \"s3://{%AWS_PARAM:xingu-staging-bucket%}/trained-models\" \\\n",
    "    --target-trained-models-path models \\\n",
    "    --project-home . \\\n",
    "    --debug;\n",
    "```\n",
    "\n",
    "### Laptop or SageMaker to Staging (go to committee)\n",
    "Your `inventory.yaml` has the `train_id` of an estimator that you just trained for a certain `dataprovider_ids`.\n",
    "```shell\n",
    "python -m xingu.deploy \\\n",
    "    --dps mydp1,mydp2 \\\n",
    "    --source-models-db \"sqlite:///xingu.db?check_same_thread=False\" \\\n",
    "    --source-models-db \"postgresql+psycopg2://{%AWS_PARAM:xingu-staging-user%}:{%AWS_SECRET:xingu-staging-rds-secret%}@{%AWS_PARAM:xingu-staging-url%}/{%AWS_PARAM:xingu-staging-database-name%}\" \\\n",
    "    --source-trained-models-path models \\\n",
    "    --target-trained-models-path \"s3://{%AWS_PARAM:xingu-staging-bucket%}/trained-models\" \\\n",
    "    --project-home . \\\n",
    "    --debug;\n",
    "```\n",
    "\n",
    "### Partial deployment or deployment failed?\n",
    "Low RAM can hurt data extraction bacause `SELECT`s might return several million lines of data. Deploy command tries to transfer data in chunks of variable size, based on the detected RAM. If it fails, use the `--db-page-size` parameter with values as low as 200000. It will take longer but it won’t fail.\n",
    "\n",
    "```shell\n",
    "python -m xingu.deploy \\\n",
    "    ...\n",
    "    --db-page-size 200000 \\\n",
    "    ...\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c185c259-7e91-438e-a65f-fc22214274da",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "instance_type": "ml.m5.4xlarge",
  "kernelspec": {
   "display_name": "python3.10",
   "language": "python",
   "name": "python3.10"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}