juliaL16demo.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Regularization demo",
    "\n",
    "2018 Winter Term 1\n",
    "\n",
    "By Mike Gelbart (converted to Julia by Jason Chau)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [],
   "source": [
    "using Distributions\n",
    "using Plots\n",
    "using Statistics\n",
    "using Printf\n",
    "using Plots.PlotMeasures\n",
    "include(\"__init__.jl\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"utf-8\"?>\n",
       "<svg xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\" width=\"450\" height=\"375\" viewBox=\"0 0 1800 1500\">\n",
       "<defs>\n",
       "  <clipPath id=\"clip8300\">\n",
       "    <rect x=\"0\" y=\"0\" width=\"1800\" height=\"1500\"/>\n",
       "  </clipPath>\n",
       "</defs>\n",
       "<polygon clip-path=\"url(#clip8300)\" points=\"\n",
       "0,1500 1800,1500 1800,0 0,0 \n",
       "  \" fill=\"#ffffff\" fill-rule=\"evenodd\" fill-opacity=\"1\"/>\n",
       "<defs>\n",
       "  <clipPath id=\"clip8301\">\n",
       "    <rect x=\"360\" y=\"0\" width=\"1261\" height=\"1261\"/>\n",
       "  </clipPath>\n",
       "</defs>\n",
       "<polygon clip-path=\"url(#clip8300)\" points=\"\n",
       "185.016,1379.34 1701.57,1379.34 1701.57,59.0551 185.016,59.0551 \n",
       "  \" fill=\"#ffffff\" fill-rule=\"evenodd\" fill-opacity=\"1\"/>\n",
       "<defs>\n",
       "  <clipPath id=\"clip8302\">\n",
       "    <rect x=\"185\" y=\"59\" width=\"1518\" height=\"1321\"/>\n",
       "  </clipPath>\n",
       "</defs>\n",
       "<polyline clip-path=\"url(#clip8302)\" style=\"stroke:#000000; stroke-width:2.5; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  403.403,1379.34 403.403,59.0551 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip8302)\" style=\"stroke:#000000; stroke-width:2.5; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  673.349,1379.34 673.349,59.0551 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip8302)\" style=\"stroke:#000000; stroke-width:2.5; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  943.295,1379.34 943.295,59.0551 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip8302)\" style=\"stroke:#000000; stroke-width:2.5; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  1213.24,1379.34 1213.24,59.0551 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip8302)\" style=\"stroke:#000000; stroke-width:2.5; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  1483.19,1379.34 1483.19,59.0551 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip8302)\" style=\"stroke:#000000; stroke-width:2.5; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  185.016,1341.98 1701.57,1341.98 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip8302)\" style=\"stroke:#000000; stroke-width:2.5; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  185.016,926.792 1701.57,926.792 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip8302)\" style=\"stroke:#000000; stroke-width:2.5; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  185.016,511.607 1701.57,511.607 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip8302)\" style=\"stroke:#000000; stroke-width:2.5; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  185.016,96.4218 1701.57,96.4218 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip8300)\" style=\"stroke:#000000; stroke-width:5; stroke-opacity:1; fill:none\" points=\"\n",
       "  185.016,1379.34 1701.57,1379.34 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip8300)\" style=\"stroke:#000000; stroke-width:5; stroke-opacity:1; fill:none\" points=\"\n",
       "  185.016,1379.34 185.016,59.0551 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip8300)\" style=\"stroke:#000000; stroke-width:5; stroke-opacity:1; fill:none\" points=\"\n",
       "  403.403,1379.34 403.403,1359.54 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip8300)\" style=\"stroke:#000000; stroke-width:5; stroke-opacity:1; fill:none\" points=\"\n",
       "  673.349,1379.34 673.349,1359.54 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip8300)\" style=\"stroke:#000000; stroke-width:5; stroke-opacity:1; fill:none\" points=\"\n",
       "  943.295,1379.34 943.295,1359.54 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip8300)\" style=\"stroke:#000000; stroke-width:5; stroke-opacity:1; fill:none\" points=\"\n",
       "  1213.24,1379.34 1213.24,1359.54 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip8300)\" style=\"stroke:#000000; stroke-width:5; stroke-opacity:1; fill:none\" points=\"\n",
       "  1483.19,1379.34 1483.19,1359.54 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip8300)\" style=\"stroke:#000000; stroke-width:5; stroke-opacity:1; fill:none\" points=\"\n",
       "  185.016,1341.98 207.764,1341.98 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip8300)\" style=\"stroke:#000000; stroke-width:5; stroke-opacity:1; fill:none\" points=\"\n",
       "  185.016,926.792 207.764,926.792 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip8300)\" style=\"stroke:#000000; stroke-width:5; stroke-opacity:1; fill:none\" points=\"\n",
       "  185.016,511.607 207.764,511.607 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip8300)\" style=\"stroke:#000000; stroke-width:5; stroke-opacity:1; fill:none\" points=\"\n",
       "  185.016,96.4218 207.764,96.4218 \n",
       "  \"/>\n",
       "<g clip-path=\"url(#clip8300)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:59px; text-anchor:middle;\" transform=\"rotate(0, 403.403, 1439.94)\" x=\"403.403\" y=\"1439.94\">-2</text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip8300)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:59px; text-anchor:middle;\" transform=\"rotate(0, 673.349, 1439.94)\" x=\"673.349\" y=\"1439.94\">-1</text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip8300)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:59px; text-anchor:middle;\" transform=\"rotate(0, 943.295, 1439.94)\" x=\"943.295\" y=\"1439.94\">0</text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip8300)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:59px; text-anchor:middle;\" transform=\"rotate(0, 1213.24, 1439.94)\" x=\"1213.24\" y=\"1439.94\">1</text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip8300)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:59px; text-anchor:middle;\" transform=\"rotate(0, 1483.19, 1439.94)\" x=\"1483.19\" y=\"1439.94\">2</text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip8300)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:59px; text-anchor:end;\" transform=\"rotate(0, 167.016, 1363.48)\" x=\"167.016\" y=\"1363.48\">0</text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip8300)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:59px; text-anchor:end;\" transform=\"rotate(0, 167.016, 948.292)\" x=\"167.016\" y=\"948.292\">10</text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip8300)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:59px; text-anchor:end;\" transform=\"rotate(0, 167.016, 533.107)\" x=\"167.016\" y=\"533.107\">20</text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip8300)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:59px; text-anchor:end;\" transform=\"rotate(0, 167.016, 117.922)\" x=\"167.016\" y=\"117.922\">30</text>\n",
       "</g>\n",
       "<polygon clip-path=\"url(#clip8302)\" points=\"\n",
       "268.43,1217.42 268.43,1341.98 403.403,1341.98 403.403,1217.42 268.43,1217.42 268.43,1217.42 \n",
       "  \" fill=\"#009af9\" fill-rule=\"evenodd\" fill-opacity=\"1\"/>\n",
       "<polyline clip-path=\"url(#clip8302)\" style=\"stroke:#000000; stroke-width:5; stroke-opacity:1; fill:none\" points=\"\n",
       "  268.43,1217.42 268.43,1341.98 403.403,1341.98 403.403,1217.42 268.43,1217.42 \n",
       "  \"/>\n",
       "<polygon clip-path=\"url(#clip8302)\" points=\"\n",
       "403.403,1009.83 403.403,1341.98 538.376,1341.98 538.376,1009.83 403.403,1009.83 403.403,1009.83 \n",
       "  \" fill=\"#009af9\" fill-rule=\"evenodd\" fill-opacity=\"1\"/>\n",
       "<polyline clip-path=\"url(#clip8302)\" style=\"stroke:#000000; stroke-width:5; stroke-opacity:1; fill:none\" points=\"\n",
       "  403.403,1009.83 403.403,1341.98 538.376,1341.98 538.376,1009.83 403.403,1009.83 \n",
       "  \"/>\n",
       "<polygon clip-path=\"url(#clip8302)\" points=\"\n",
       "538.376,1217.42 538.376,1341.98 673.349,1341.98 673.349,1217.42 538.376,1217.42 538.376,1217.42 \n",
       "  \" fill=\"#009af9\" fill-rule=\"evenodd\" fill-opacity=\"1\"/>\n",
       "<polyline clip-path=\"url(#clip8302)\" style=\"stroke:#000000; stroke-width:5; stroke-opacity:1; fill:none\" points=\"\n",
       "  538.376,1217.42 538.376,1341.98 673.349,1341.98 673.349,1217.42 538.376,1217.42 \n",
       "  \"/>\n",
       "<polygon clip-path=\"url(#clip8302)\" points=\"\n",
       "673.349,719.199 673.349,1341.98 808.322,1341.98 808.322,719.199 673.349,719.199 673.349,719.199 \n",
       "  \" fill=\"#009af9\" fill-rule=\"evenodd\" fill-opacity=\"1\"/>\n",
       "<polyline clip-path=\"url(#clip8302)\" style=\"stroke:#000000; stroke-width:5; stroke-opacity:1; fill:none\" points=\"\n",
       "  673.349,719.199 673.349,1341.98 808.322,1341.98 808.322,719.199 673.349,719.199 \n",
       "  \"/>\n",
       "<polygon clip-path=\"url(#clip8302)\" points=\"\n",
       "808.322,96.4218 808.322,1341.98 943.295,1341.98 943.295,96.4218 808.322,96.4218 808.322,96.4218 \n",
       "  \" fill=\"#009af9\" fill-rule=\"evenodd\" fill-opacity=\"1\"/>\n",
       "<polyline clip-path=\"url(#clip8302)\" style=\"stroke:#000000; stroke-width:5; stroke-opacity:1; fill:none\" points=\"\n",
       "  808.322,96.4218 808.322,1341.98 943.295,1341.98 943.295,96.4218 808.322,96.4218 \n",
       "  \"/>\n",
       "<polygon clip-path=\"url(#clip8302)\" points=\"\n",
       "943.295,885.273 943.295,1341.98 1078.27,1341.98 1078.27,885.273 943.295,885.273 943.295,885.273 \n",
       "  \" fill=\"#009af9\" fill-rule=\"evenodd\" fill-opacity=\"1\"/>\n",
       "<polyline clip-path=\"url(#clip8302)\" style=\"stroke:#000000; stroke-width:5; stroke-opacity:1; fill:none\" points=\"\n",
       "  943.295,885.273 943.295,1341.98 1078.27,1341.98 1078.27,885.273 943.295,885.273 \n",
       "  \"/>\n",
       "<polygon clip-path=\"url(#clip8302)\" points=\"\n",
       "1078.27,885.273 1078.27,1341.98 1213.24,1341.98 1213.24,885.273 1078.27,885.273 1078.27,885.273 \n",
       "  \" fill=\"#009af9\" fill-rule=\"evenodd\" fill-opacity=\"1\"/>\n",
       "<polyline clip-path=\"url(#clip8302)\" style=\"stroke:#000000; stroke-width:5; stroke-opacity:1; fill:none\" points=\"\n",
       "  1078.27,885.273 1078.27,1341.98 1213.24,1341.98 1213.24,885.273 1078.27,885.273 \n",
       "  \"/>\n",
       "<polygon clip-path=\"url(#clip8302)\" points=\"\n",
       "1213.24,760.718 1213.24,1341.98 1348.22,1341.98 1348.22,760.718 1213.24,760.718 1213.24,760.718 \n",
       "  \" fill=\"#009af9\" fill-rule=\"evenodd\" fill-opacity=\"1\"/>\n",
       "<polyline clip-path=\"url(#clip8302)\" style=\"stroke:#000000; stroke-width:5; stroke-opacity:1; fill:none\" points=\"\n",
       "  1213.24,760.718 1213.24,1341.98 1348.22,1341.98 1348.22,760.718 1213.24,760.718 \n",
       "  \"/>\n",
       "<polygon clip-path=\"url(#clip8302)\" points=\"\n",
       "1348.22,1175.9 1348.22,1341.98 1483.19,1341.98 1483.19,1175.9 1348.22,1175.9 1348.22,1175.9 \n",
       "  \" fill=\"#009af9\" fill-rule=\"evenodd\" fill-opacity=\"1\"/>\n",
       "<polyline clip-path=\"url(#clip8302)\" style=\"stroke:#000000; stroke-width:5; stroke-opacity:1; fill:none\" points=\"\n",
       "  1348.22,1175.9 1348.22,1341.98 1483.19,1341.98 1483.19,1175.9 1348.22,1175.9 \n",
       "  \"/>\n",
       "<polygon clip-path=\"url(#clip8302)\" points=\"\n",
       "1483.19,1300.46 1483.19,1341.98 1618.16,1341.98 1618.16,1300.46 1483.19,1300.46 1483.19,1300.46 \n",
       "  \" fill=\"#009af9\" fill-rule=\"evenodd\" fill-opacity=\"1\"/>\n",
       "<polyline clip-path=\"url(#clip8302)\" style=\"stroke:#000000; stroke-width:5; stroke-opacity:1; fill:none\" points=\"\n",
       "  1483.19,1300.46 1483.19,1341.98 1618.16,1341.98 1618.16,1300.46 1483.19,1300.46 \n",
       "  \"/>\n",
       "</svg>\n"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "N = 100\n",
    "y = rand(Normal(),N,1)\n",
    "histogram(y,\n",
    "    thickness_scaling=5,\n",
    "    size=(1800,1500),\n",
    "    legend=false\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Imagine your training set only contains the $y$-values, and they are shown above.\n",
    "- How can you predict a new $y$?\n",
    "- Reasonable idea: predict the mean value."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "-0.07391725221350744"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "mutable struct MeanPredictor\n",
    "    predict\n",
    "end\n",
    "function initializeMeanPredictor(x)\n",
    "    function predict()\n",
    "        return mean(x)\n",
    "    end\n",
    "    return MeanPredictor(predict)\n",
    "end\n",
    "mp = initializeMeanPredictor(y)\n",
    "mp.predict()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now imagine we have one feature $x$. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "100×1 Array{Float64,2}:\n",
       "  0.8831635512739546  \n",
       "  0.8751047181922496  \n",
       " -0.4720764222393786  \n",
       " -0.51812664538706    \n",
       "  0.3591094161675397  \n",
       "  0.7445285485029494  \n",
       " -0.9345408501260282  \n",
       " -1.0085789859263723  \n",
       "  0.06409746757617311 \n",
       "  0.4068747768671814  \n",
       "  0.4034029096924158  \n",
       " -0.0956580327050674  \n",
       " -1.0838238181257316  \n",
       "  ⋮                   \n",
       " -0.018448639194782207\n",
       " -0.4327510004459679  \n",
       "  0.052918241021443366\n",
       " -0.18648778922064535 \n",
       " -0.02602198889291544 \n",
       " -0.1619346534420139  \n",
       "  0.5220184414789591  \n",
       "  0.20071055103419677 \n",
       "  0.6282703036333255  \n",
       " -0.36024150458944937 \n",
       " -0.02862664924272551 \n",
       "  0.14748064695014118 "
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x = y ./ 2 + rand(Normal(),N,1) ./ 3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"utf-8\"?>\n",
       "<svg xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\" width=\"450\" height=\"375\" viewBox=\"0 0 1800 1500\">\n",
       "<defs>\n",
       "  <clipPath id=\"clip9900\">\n",
       "    <rect x=\"0\" y=\"0\" width=\"1800\" height=\"1500\"/>\n",
       "  </clipPath>\n",
       "</defs>\n",
       "<polygon clip-path=\"url(#clip9900)\" points=\"\n",
       "0,1500 1800,1500 1800,0 0,0 \n",
       "  \" fill=\"#ffffff\" fill-rule=\"evenodd\" fill-opacity=\"1\"/>\n",
       "<defs>\n",
       "  <clipPath id=\"clip9901\">\n",
       "    <rect x=\"360\" y=\"0\" width=\"1261\" height=\"1261\"/>\n",
       "  </clipPath>\n",
       "</defs>\n",
       "<polygon clip-path=\"url(#clip9900)\" points=\"\n",
       "461.063,1159.28 1701.57,1159.28 1701.57,59.0551 461.063,59.0551 \n",
       "  \" fill=\"#ffffff\" fill-rule=\"evenodd\" fill-opacity=\"1\"/>\n",
       "<defs>\n",
       "  <clipPath id=\"clip9902\">\n",
       "    <rect x=\"461\" y=\"59\" width=\"1242\" height=\"1101\"/>\n",
       "  </clipPath>\n",
       "</defs>\n",
       "<polyline clip-path=\"url(#clip9902)\" style=\"stroke:#000000; stroke-width:2.5; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  671.766,1159.28 671.766,59.0551 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip9902)\" style=\"stroke:#000000; stroke-width:2.5; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  892.128,1159.28 892.128,59.0551 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip9902)\" style=\"stroke:#000000; stroke-width:2.5; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  1112.49,1159.28 1112.49,59.0551 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip9902)\" style=\"stroke:#000000; stroke-width:2.5; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  1332.85,1159.28 1332.85,59.0551 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip9902)\" style=\"stroke:#000000; stroke-width:2.5; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  1553.21,1159.28 1553.21,59.0551 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip9902)\" style=\"stroke:#000000; stroke-width:2.5; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  461.063,1064.72 1701.57,1064.72 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip9902)\" style=\"stroke:#000000; stroke-width:2.5; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  461.063,844.329 1701.57,844.329 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip9902)\" style=\"stroke:#000000; stroke-width:2.5; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  461.063,623.94 1701.57,623.94 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip9902)\" style=\"stroke:#000000; stroke-width:2.5; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  461.063,403.55 1701.57,403.55 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip9902)\" style=\"stroke:#000000; stroke-width:2.5; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  461.063,183.161 1701.57,183.161 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip9900)\" style=\"stroke:#000000; stroke-width:5; stroke-opacity:1; fill:none\" points=\"\n",
       "  461.063,1159.28 1701.57,1159.28 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip9900)\" style=\"stroke:#000000; stroke-width:5; stroke-opacity:1; fill:none\" points=\"\n",
       "  461.063,1159.28 461.063,59.0551 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip9900)\" style=\"stroke:#000000; stroke-width:5; stroke-opacity:1; fill:none\" points=\"\n",
       "  671.766,1159.28 671.766,1142.78 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip9900)\" style=\"stroke:#000000; stroke-width:5; stroke-opacity:1; fill:none\" points=\"\n",
       "  892.128,1159.28 892.128,1142.78 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip9900)\" style=\"stroke:#000000; stroke-width:5; stroke-opacity:1; fill:none\" points=\"\n",
       "  1112.49,1159.28 1112.49,1142.78 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip9900)\" style=\"stroke:#000000; stroke-width:5; stroke-opacity:1; fill:none\" points=\"\n",
       "  1332.85,1159.28 1332.85,1142.78 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip9900)\" style=\"stroke:#000000; stroke-width:5; stroke-opacity:1; fill:none\" points=\"\n",
       "  1553.21,1159.28 1553.21,1142.78 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip9900)\" style=\"stroke:#000000; stroke-width:5; stroke-opacity:1; fill:none\" points=\"\n",
       "  461.063,1064.72 479.671,1064.72 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip9900)\" style=\"stroke:#000000; stroke-width:5; stroke-opacity:1; fill:none\" points=\"\n",
       "  461.063,844.329 479.671,844.329 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip9900)\" style=\"stroke:#000000; stroke-width:5; stroke-opacity:1; fill:none\" points=\"\n",
       "  461.063,623.94 479.671,623.94 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip9900)\" style=\"stroke:#000000; stroke-width:5; stroke-opacity:1; fill:none\" points=\"\n",
       "  461.063,403.55 479.671,403.55 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip9900)\" style=\"stroke:#000000; stroke-width:5; stroke-opacity:1; fill:none\" points=\"\n",
       "  461.063,183.161 479.671,183.161 \n",
       "  \"/>\n",
       "<g clip-path=\"url(#clip9900)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:59px; text-anchor:middle;\" transform=\"rotate(0, 671.766, 1219.88)\" x=\"671.766\" y=\"1219.88\">-1.0</text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip9900)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:59px; text-anchor:middle;\" transform=\"rotate(0, 892.128, 1219.88)\" x=\"892.128\" y=\"1219.88\">-0.5</text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip9900)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:59px; text-anchor:middle;\" transform=\"rotate(0, 1112.49, 1219.88)\" x=\"1112.49\" y=\"1219.88\">0.0</text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip9900)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:59px; text-anchor:middle;\" transform=\"rotate(0, 1332.85, 1219.88)\" x=\"1332.85\" y=\"1219.88\">0.5</text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip9900)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:59px; text-anchor:middle;\" transform=\"rotate(0, 1553.21, 1219.88)\" x=\"1553.21\" y=\"1219.88\">1.0</text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip9900)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:59px; text-anchor:end;\" transform=\"rotate(0, 443.063, 1086.22)\" x=\"443.063\" y=\"1086.22\">-2</text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip9900)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:59px; text-anchor:end;\" transform=\"rotate(0, 443.063, 865.829)\" x=\"443.063\" y=\"865.829\">-1</text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip9900)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:59px; text-anchor:end;\" transform=\"rotate(0, 443.063, 645.44)\" x=\"443.063\" y=\"645.44\">0</text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip9900)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:59px; text-anchor:end;\" transform=\"rotate(0, 443.063, 425.05)\" x=\"443.063\" y=\"425.05\">1</text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip9900)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:59px; text-anchor:end;\" transform=\"rotate(0, 443.063, 204.661)\" x=\"443.063\" y=\"204.661\">2</text>\n",
       "</g>\n",
       "<circle clip-path=\"url(#clip9902)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1501.72\" cy=\"373.306\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9902)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1501.72\" cy=\"373.306\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9902)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1498.17\" cy=\"261.206\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9902)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1498.17\" cy=\"261.206\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9902)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"904.434\" cy=\"695.313\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9902)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"904.434\" cy=\"695.313\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9902)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"884.139\" cy=\"713.049\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9902)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"884.139\" cy=\"713.049\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9902)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1270.76\" cy=\"301.927\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9902)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1270.76\" cy=\"301.927\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9902)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1440.62\" cy=\"345.535\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9902)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1440.62\" cy=\"345.535\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9902)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"700.615\" cy=\"1100.35\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9902)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"700.615\" cy=\"1100.35\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9902)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"667.985\" cy=\"960.433\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9902)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"667.985\" cy=\"960.433\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9902)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1140.74\" cy=\"633.481\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9902)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1140.74\" cy=\"633.481\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9902)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1291.81\" cy=\"403.253\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9902)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1291.81\" cy=\"403.253\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9902)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1290.28\" cy=\"412.492\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9902)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1290.28\" cy=\"412.492\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9902)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1070.33\" cy=\"584.604\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9902)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1070.33\" cy=\"584.604\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9902)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"634.823\" cy=\"1067.66\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9902)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"634.823\" cy=\"1067.66\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9902)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"998.144\" cy=\"760.064\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9902)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"998.144\" cy=\"760.064\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9902)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1425.95\" cy=\"640.989\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9902)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1425.95\" cy=\"640.989\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9902)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"996.385\" cy=\"733.838\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9902)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"996.385\" cy=\"733.838\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9902)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"924.232\" cy=\"789.792\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9902)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"924.232\" cy=\"789.792\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9902)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1048.55\" cy=\"707.268\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9902)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1048.55\" cy=\"707.268\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9902)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1282.69\" cy=\"389.861\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9902)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1282.69\" cy=\"389.861\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9902)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1280.68\" cy=\"273.011\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9902)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1280.68\" cy=\"273.011\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9902)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"726.352\" cy=\"841.201\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9902)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"726.352\" cy=\"841.201\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9902)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1244.44\" cy=\"723.772\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9902)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1244.44\" cy=\"723.772\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9902)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"832.186\" cy=\"778.31\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9902)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"832.186\" cy=\"778.31\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9902)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1574.58\" cy=\"310.898\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9902)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1574.58\" cy=\"310.898\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9902)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"717.43\" cy=\"1059.86\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9902)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"717.43\" cy=\"1059.86\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9902)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1292.91\" cy=\"399.33\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9902)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1292.91\" cy=\"399.33\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9902)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"949.401\" cy=\"734.371\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9902)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"949.401\" cy=\"734.371\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9902)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1013.32\" cy=\"686.742\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9902)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1013.32\" cy=\"686.742\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9902)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1311.92\" cy=\"372.322\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9902)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1311.92\" cy=\"372.322\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9902)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1315.01\" cy=\"714.114\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9902)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1315.01\" cy=\"714.114\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9902)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1059.85\" cy=\"778.109\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9902)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1059.85\" cy=\"778.109\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9902)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"823.346\" cy=\"775.344\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9902)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"823.346\" cy=\"775.344\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9902)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"596.849\" cy=\"1128.14\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9902)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"596.849\" cy=\"1128.14\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9902)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1159.69\" cy=\"472.073\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9902)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1159.69\" cy=\"472.073\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9902)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1573.09\" cy=\"361.513\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9902)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1573.09\" cy=\"361.513\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9902)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"975.663\" cy=\"667.407\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9902)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"975.663\" cy=\"667.407\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9902)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"813.189\" cy=\"812.75\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9902)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"813.189\" cy=\"812.75\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9902)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1218.86\" cy=\"520.218\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9902)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1218.86\" cy=\"520.218\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9902)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1646.16\" cy=\"505.774\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9902)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1646.16\" cy=\"505.774\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9902)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1495.37\" cy=\"503.83\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9902)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1495.37\" cy=\"503.83\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9902)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"774.613\" cy=\"916.92\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9902)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"774.613\" cy=\"916.92\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9902)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"496.172\" cy=\"987.789\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9902)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"496.172\" cy=\"987.789\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9902)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1033.28\" cy=\"660.445\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9902)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1033.28\" cy=\"660.445\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9902)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1175.42\" cy=\"640.812\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9902)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1175.42\" cy=\"640.812\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9902)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1034.76\" cy=\"788.966\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9902)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1034.76\" cy=\"788.966\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9902)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1107.18\" cy=\"350.923\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9902)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1107.18\" cy=\"350.923\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9902)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"866.135\" cy=\"745.351\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9902)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"866.135\" cy=\"745.351\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9902)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"833.412\" cy=\"657.664\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9902)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"833.412\" cy=\"657.664\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9902)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1037.84\" cy=\"688.534\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9902)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1037.84\" cy=\"688.534\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9902)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"697.359\" cy=\"827.25\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9902)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"697.359\" cy=\"827.25\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9902)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1102.63\" cy=\"629.163\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9902)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1102.63\" cy=\"629.163\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9902)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1246.59\" cy=\"531.845\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9902)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1246.59\" cy=\"531.845\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9902)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1101.14\" cy=\"598.294\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9902)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1101.14\" cy=\"598.294\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9902)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"972.802\" cy=\"650.803\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9902)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"972.802\" cy=\"650.803\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9902)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1281.22\" cy=\"609.042\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9902)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1281.22\" cy=\"609.042\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9902)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1131.32\" cy=\"439.684\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9902)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1131.32\" cy=\"439.684\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9902)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"921.543\" cy=\"649.695\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9902)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"921.543\" cy=\"649.695\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9902)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1086.69\" cy=\"757.471\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9902)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1086.69\" cy=\"757.471\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9902)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"496.274\" cy=\"1043.9\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9902)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"496.274\" cy=\"1043.9\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9902)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1245.22\" cy=\"359.638\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9902)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1245.22\" cy=\"359.638\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9902)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1002.61\" cy=\"711.683\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9902)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1002.61\" cy=\"711.683\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9902)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1310\" cy=\"707.221\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9902)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1310\" cy=\"707.221\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9902)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1020.84\" cy=\"673.186\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9902)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1020.84\" cy=\"673.186\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9902)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"789.012\" cy=\"901.138\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9902)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"789.012\" cy=\"901.138\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9902)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1506.47\" cy=\"415.752\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9902)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1506.47\" cy=\"415.752\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9902)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1133.11\" cy=\"652.964\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9902)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1133.11\" cy=\"652.964\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9902)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"834.844\" cy=\"812.741\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9902)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"834.844\" cy=\"812.741\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9902)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"886.659\" cy=\"690.04\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9902)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"886.659\" cy=\"690.04\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9902)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"912.212\" cy=\"653.684\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9902)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"912.212\" cy=\"653.684\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9902)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1303.23\" cy=\"549.75\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9902)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1303.23\" cy=\"549.75\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9902)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1370.61\" cy=\"330.767\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9902)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1370.61\" cy=\"330.767\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9902)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1303.28\" cy=\"268.257\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9902)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1303.28\" cy=\"268.257\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9902)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1202.4\" cy=\"528.042\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9902)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1202.4\" cy=\"528.042\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9902)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"738.769\" cy=\"1014.53\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9902)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"738.769\" cy=\"1014.53\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9902)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1411.25\" cy=\"504.806\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9902)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1411.25\" cy=\"504.806\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9902)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1205.25\" cy=\"475.386\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9902)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1205.25\" cy=\"475.386\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9902)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"773.193\" cy=\"987.619\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9902)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"773.193\" cy=\"987.619\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9902)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"930.092\" cy=\"955.487\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9902)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"930.092\" cy=\"955.487\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9902)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1490.7\" cy=\"365.518\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9902)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1490.7\" cy=\"365.518\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9902)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1105.22\" cy=\"438.535\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9902)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1105.22\" cy=\"438.535\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9902)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"801.184\" cy=\"1050.52\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9902)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"801.184\" cy=\"1050.52\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9902)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1005.89\" cy=\"842.622\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9902)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1005.89\" cy=\"842.622\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9902)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"874.717\" cy=\"703.402\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9902)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"874.717\" cy=\"703.402\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9902)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"759.406\" cy=\"837.568\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9902)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"759.406\" cy=\"837.568\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9902)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"898.237\" cy=\"953.926\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9902)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"898.237\" cy=\"953.926\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9902)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"982.346\" cy=\"679.977\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9902)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"982.346\" cy=\"679.977\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9902)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1151.26\" cy=\"711.216\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9902)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1151.26\" cy=\"711.216\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9902)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1666.47\" cy=\"90.1936\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9902)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1666.47\" cy=\"90.1936\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9902)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1104.36\" cy=\"733.815\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9902)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1104.36\" cy=\"733.815\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9902)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"921.766\" cy=\"699.363\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9902)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"921.766\" cy=\"699.363\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9902)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1135.81\" cy=\"558.248\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9902)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1135.81\" cy=\"558.248\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9902)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1030.3\" cy=\"632.581\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9902)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1030.3\" cy=\"632.581\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9902)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1101.02\" cy=\"619.486\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9902)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1101.02\" cy=\"619.486\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9902)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1041.12\" cy=\"621.477\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9902)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1041.12\" cy=\"621.477\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9902)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1342.56\" cy=\"426.972\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9902)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1342.56\" cy=\"426.972\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9902)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1200.95\" cy=\"492.278\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9902)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1200.95\" cy=\"492.278\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9902)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1389.38\" cy=\"605.465\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9902)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1389.38\" cy=\"605.465\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9902)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"953.723\" cy=\"637.894\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9902)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"953.723\" cy=\"637.894\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9902)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1099.87\" cy=\"288.536\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9902)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1099.87\" cy=\"288.536\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9902)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1177.49\" cy=\"372.68\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip9902)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1177.49\" cy=\"372.68\" r=\"18\"/>\n",
       "<polyline clip-path=\"url(#clip9902)\" style=\"stroke:#e26f46; stroke-width:25; stroke-opacity:1; fill:none\" points=\"\n",
       "  496.172,640.23 1666.47,640.23 \n",
       "  \"/>\n",
       "<polygon clip-path=\"url(#clip9900)\" points=\"\n",
       "515.063,318.255 1168.28,318.255 1168.28,91.4551 515.063,91.4551 \n",
       "  \" fill=\"#ffffff\" fill-rule=\"evenodd\" fill-opacity=\"1\"/>\n",
       "<polyline clip-path=\"url(#clip9900)\" style=\"stroke:#000000; stroke-width:5; stroke-opacity:1; fill:none\" points=\"\n",
       "  515.063,318.255 1168.28,318.255 1168.28,91.4551 515.063,91.4551 515.063,318.255 \n",
       "  \"/>\n",
       "<circle clip-path=\"url(#clip9900)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"596.063\" cy=\"167.055\" r=\"6\"/>\n",
       "<circle clip-path=\"url(#clip9900)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"596.063\" cy=\"167.055\" r=\"5\"/>\n",
       "<g clip-path=\"url(#clip9900)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:59px; text-anchor:start;\" transform=\"rotate(0, 659.063, 188.555)\" x=\"659.063\" y=\"188.555\">training data</text>\n",
       "</g>\n",
       "<polyline clip-path=\"url(#clip9900)\" style=\"stroke:#e26f46; stroke-width:25; stroke-opacity:1; fill:none\" points=\"\n",
       "  533.063,242.655 641.063,242.655 \n",
       "  \"/>\n",
       "<g clip-path=\"url(#clip9900)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:59px; text-anchor:start;\" transform=\"rotate(0, 659.063, 264.155)\" x=\"659.063\" y=\"264.155\">mean of y-values</text>\n",
       "</g>\n",
       "</svg>\n"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "scatter(x,y,label=\"training data\",\n",
    "    markersize=20,\n",
    "    thickness_scaling=5,\n",
    "    size=(1800,1500),\n",
    "    legend=:topleft\n",
    ")\n",
    "m = mean(y)\n",
    "plot!([minimum(x),maximum(x)],[m,m], linewidth=5, label=\"mean of y-values\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- So, what we were doing was sort of like linear regression with $w=0$.\n",
    "- Ok, but, why not use this feature?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"utf-8\"?>\n",
       "<svg xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\" width=\"450\" height=\"375\" viewBox=\"0 0 1800 1500\">\n",
       "<defs>\n",
       "  <clipPath id=\"clip0700\">\n",
       "    <rect x=\"0\" y=\"0\" width=\"1800\" height=\"1500\"/>\n",
       "  </clipPath>\n",
       "</defs>\n",
       "<polygon clip-path=\"url(#clip0700)\" points=\"\n",
       "0,1500 1800,1500 1800,0 0,0 \n",
       "  \" fill=\"#ffffff\" fill-rule=\"evenodd\" fill-opacity=\"1\"/>\n",
       "<defs>\n",
       "  <clipPath id=\"clip0701\">\n",
       "    <rect x=\"360\" y=\"0\" width=\"1261\" height=\"1261\"/>\n",
       "  </clipPath>\n",
       "</defs>\n",
       "<polygon clip-path=\"url(#clip0700)\" points=\"\n",
       "461.063,1159.28 1701.57,1159.28 1701.57,59.0551 461.063,59.0551 \n",
       "  \" fill=\"#ffffff\" fill-rule=\"evenodd\" fill-opacity=\"1\"/>\n",
       "<defs>\n",
       "  <clipPath id=\"clip0702\">\n",
       "    <rect x=\"461\" y=\"59\" width=\"1242\" height=\"1101\"/>\n",
       "  </clipPath>\n",
       "</defs>\n",
       "<polyline clip-path=\"url(#clip0702)\" style=\"stroke:#000000; stroke-width:2.5; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  671.766,1159.28 671.766,59.0551 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip0702)\" style=\"stroke:#000000; stroke-width:2.5; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  892.128,1159.28 892.128,59.0551 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip0702)\" style=\"stroke:#000000; stroke-width:2.5; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  1112.49,1159.28 1112.49,59.0551 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip0702)\" style=\"stroke:#000000; stroke-width:2.5; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  1332.85,1159.28 1332.85,59.0551 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip0702)\" style=\"stroke:#000000; stroke-width:2.5; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  1553.21,1159.28 1553.21,59.0551 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip0702)\" style=\"stroke:#000000; stroke-width:2.5; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  461.063,1064.72 1701.57,1064.72 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip0702)\" style=\"stroke:#000000; stroke-width:2.5; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  461.063,844.329 1701.57,844.329 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip0702)\" style=\"stroke:#000000; stroke-width:2.5; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  461.063,623.94 1701.57,623.94 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip0702)\" style=\"stroke:#000000; stroke-width:2.5; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  461.063,403.55 1701.57,403.55 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip0702)\" style=\"stroke:#000000; stroke-width:2.5; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  461.063,183.161 1701.57,183.161 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip0700)\" style=\"stroke:#000000; stroke-width:5; stroke-opacity:1; fill:none\" points=\"\n",
       "  461.063,1159.28 1701.57,1159.28 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip0700)\" style=\"stroke:#000000; stroke-width:5; stroke-opacity:1; fill:none\" points=\"\n",
       "  461.063,1159.28 461.063,59.0551 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip0700)\" style=\"stroke:#000000; stroke-width:5; stroke-opacity:1; fill:none\" points=\"\n",
       "  671.766,1159.28 671.766,1142.78 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip0700)\" style=\"stroke:#000000; stroke-width:5; stroke-opacity:1; fill:none\" points=\"\n",
       "  892.128,1159.28 892.128,1142.78 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip0700)\" style=\"stroke:#000000; stroke-width:5; stroke-opacity:1; fill:none\" points=\"\n",
       "  1112.49,1159.28 1112.49,1142.78 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip0700)\" style=\"stroke:#000000; stroke-width:5; stroke-opacity:1; fill:none\" points=\"\n",
       "  1332.85,1159.28 1332.85,1142.78 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip0700)\" style=\"stroke:#000000; stroke-width:5; stroke-opacity:1; fill:none\" points=\"\n",
       "  1553.21,1159.28 1553.21,1142.78 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip0700)\" style=\"stroke:#000000; stroke-width:5; stroke-opacity:1; fill:none\" points=\"\n",
       "  461.063,1064.72 479.671,1064.72 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip0700)\" style=\"stroke:#000000; stroke-width:5; stroke-opacity:1; fill:none\" points=\"\n",
       "  461.063,844.329 479.671,844.329 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip0700)\" style=\"stroke:#000000; stroke-width:5; stroke-opacity:1; fill:none\" points=\"\n",
       "  461.063,623.94 479.671,623.94 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip0700)\" style=\"stroke:#000000; stroke-width:5; stroke-opacity:1; fill:none\" points=\"\n",
       "  461.063,403.55 479.671,403.55 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip0700)\" style=\"stroke:#000000; stroke-width:5; stroke-opacity:1; fill:none\" points=\"\n",
       "  461.063,183.161 479.671,183.161 \n",
       "  \"/>\n",
       "<g clip-path=\"url(#clip0700)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:59px; text-anchor:middle;\" transform=\"rotate(0, 671.766, 1219.88)\" x=\"671.766\" y=\"1219.88\">-1.0</text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip0700)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:59px; text-anchor:middle;\" transform=\"rotate(0, 892.128, 1219.88)\" x=\"892.128\" y=\"1219.88\">-0.5</text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip0700)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:59px; text-anchor:middle;\" transform=\"rotate(0, 1112.49, 1219.88)\" x=\"1112.49\" y=\"1219.88\">0.0</text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip0700)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:59px; text-anchor:middle;\" transform=\"rotate(0, 1332.85, 1219.88)\" x=\"1332.85\" y=\"1219.88\">0.5</text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip0700)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:59px; text-anchor:middle;\" transform=\"rotate(0, 1553.21, 1219.88)\" x=\"1553.21\" y=\"1219.88\">1.0</text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip0700)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:59px; text-anchor:end;\" transform=\"rotate(0, 443.063, 1086.22)\" x=\"443.063\" y=\"1086.22\">-2</text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip0700)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:59px; text-anchor:end;\" transform=\"rotate(0, 443.063, 865.829)\" x=\"443.063\" y=\"865.829\">-1</text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip0700)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:59px; text-anchor:end;\" transform=\"rotate(0, 443.063, 645.44)\" x=\"443.063\" y=\"645.44\">0</text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip0700)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:59px; text-anchor:end;\" transform=\"rotate(0, 443.063, 425.05)\" x=\"443.063\" y=\"425.05\">1</text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip0700)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:59px; text-anchor:end;\" transform=\"rotate(0, 443.063, 204.661)\" x=\"443.063\" y=\"204.661\">2</text>\n",
       "</g>\n",
       "<circle clip-path=\"url(#clip0702)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1501.72\" cy=\"373.306\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip0702)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1501.72\" cy=\"373.306\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip0702)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1498.17\" cy=\"261.206\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip0702)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1498.17\" cy=\"261.206\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip0702)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"904.434\" cy=\"695.313\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip0702)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"904.434\" cy=\"695.313\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip0702)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"884.139\" cy=\"713.049\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip0702)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"884.139\" cy=\"713.049\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip0702)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1270.76\" cy=\"301.927\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip0702)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1270.76\" cy=\"301.927\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip0702)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1440.62\" cy=\"345.535\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip0702)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1440.62\" cy=\"345.535\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip0702)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"700.615\" cy=\"1100.35\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip0702)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"700.615\" cy=\"1100.35\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip0702)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"667.985\" cy=\"960.433\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip0702)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"667.985\" cy=\"960.433\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip0702)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1140.74\" cy=\"633.481\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip0702)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1140.74\" cy=\"633.481\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip0702)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1291.81\" cy=\"403.253\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip0702)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1291.81\" cy=\"403.253\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip0702)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1290.28\" cy=\"412.492\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip0702)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1290.28\" cy=\"412.492\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip0702)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1070.33\" cy=\"584.604\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip0702)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1070.33\" cy=\"584.604\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip0702)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"634.823\" cy=\"1067.66\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip0702)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"634.823\" cy=\"1067.66\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip0702)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"998.144\" cy=\"760.064\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip0702)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"998.144\" cy=\"760.064\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip0702)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1425.95\" cy=\"640.989\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip0702)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1425.95\" cy=\"640.989\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip0702)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"996.385\" cy=\"733.838\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip0702)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"996.385\" cy=\"733.838\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip0702)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"924.232\" cy=\"789.792\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip0702)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"924.232\" cy=\"789.792\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip0702)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1048.55\" cy=\"707.268\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip0702)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1048.55\" cy=\"707.268\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip0702)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1282.69\" cy=\"389.861\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip0702)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1282.69\" cy=\"389.861\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip0702)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1280.68\" cy=\"273.011\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip0702)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1280.68\" cy=\"273.011\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip0702)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"726.352\" cy=\"841.201\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip0702)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"726.352\" cy=\"841.201\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip0702)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1244.44\" cy=\"723.772\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip0702)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1244.44\" cy=\"723.772\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip0702)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"832.186\" cy=\"778.31\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip0702)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"832.186\" cy=\"778.31\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip0702)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1574.58\" cy=\"310.898\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip0702)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1574.58\" cy=\"310.898\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip0702)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"717.43\" cy=\"1059.86\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip0702)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"717.43\" cy=\"1059.86\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip0702)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1292.91\" cy=\"399.33\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip0702)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1292.91\" cy=\"399.33\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip0702)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"949.401\" cy=\"734.371\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip0702)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"949.401\" cy=\"734.371\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip0702)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1013.32\" cy=\"686.742\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip0702)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1013.32\" cy=\"686.742\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip0702)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1311.92\" cy=\"372.322\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip0702)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1311.92\" cy=\"372.322\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip0702)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1315.01\" cy=\"714.114\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip0702)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1315.01\" cy=\"714.114\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip0702)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1059.85\" cy=\"778.109\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip0702)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1059.85\" cy=\"778.109\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip0702)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"823.346\" cy=\"775.344\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip0702)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"823.346\" cy=\"775.344\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip0702)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"596.849\" cy=\"1128.14\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip0702)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"596.849\" cy=\"1128.14\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip0702)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1159.69\" cy=\"472.073\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip0702)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1159.69\" cy=\"472.073\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip0702)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1573.09\" cy=\"361.513\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip0702)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1573.09\" cy=\"361.513\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip0702)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"975.663\" cy=\"667.407\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip0702)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"975.663\" cy=\"667.407\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip0702)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"813.189\" cy=\"812.75\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip0702)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"813.189\" cy=\"812.75\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip0702)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1218.86\" cy=\"520.218\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip0702)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1218.86\" cy=\"520.218\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip0702)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1646.16\" cy=\"505.774\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip0702)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1646.16\" cy=\"505.774\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip0702)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1495.37\" cy=\"503.83\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip0702)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1495.37\" cy=\"503.83\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip0702)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"774.613\" cy=\"916.92\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip0702)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"774.613\" cy=\"916.92\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip0702)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"496.172\" cy=\"987.789\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip0702)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"496.172\" cy=\"987.789\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip0702)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1033.28\" cy=\"660.445\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip0702)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1033.28\" cy=\"660.445\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip0702)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1175.42\" cy=\"640.812\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip0702)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1175.42\" cy=\"640.812\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip0702)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1034.76\" cy=\"788.966\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip0702)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1034.76\" cy=\"788.966\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip0702)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1107.18\" cy=\"350.923\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip0702)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1107.18\" cy=\"350.923\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip0702)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"866.135\" cy=\"745.351\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip0702)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"866.135\" cy=\"745.351\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip0702)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"833.412\" cy=\"657.664\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip0702)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"833.412\" cy=\"657.664\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip0702)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1037.84\" cy=\"688.534\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip0702)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1037.84\" cy=\"688.534\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip0702)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"697.359\" cy=\"827.25\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip0702)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"697.359\" cy=\"827.25\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip0702)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1102.63\" cy=\"629.163\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip0702)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1102.63\" cy=\"629.163\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip0702)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1246.59\" cy=\"531.845\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip0702)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1246.59\" cy=\"531.845\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip0702)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1101.14\" cy=\"598.294\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip0702)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1101.14\" cy=\"598.294\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip0702)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"972.802\" cy=\"650.803\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip0702)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"972.802\" cy=\"650.803\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip0702)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1281.22\" cy=\"609.042\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip0702)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1281.22\" cy=\"609.042\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip0702)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1131.32\" cy=\"439.684\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip0702)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1131.32\" cy=\"439.684\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip0702)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"921.543\" cy=\"649.695\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip0702)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"921.543\" cy=\"649.695\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip0702)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1086.69\" cy=\"757.471\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip0702)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1086.69\" cy=\"757.471\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip0702)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"496.274\" cy=\"1043.9\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip0702)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"496.274\" cy=\"1043.9\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip0702)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1245.22\" cy=\"359.638\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip0702)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1245.22\" cy=\"359.638\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip0702)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1002.61\" cy=\"711.683\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip0702)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1002.61\" cy=\"711.683\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip0702)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1310\" cy=\"707.221\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip0702)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1310\" cy=\"707.221\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip0702)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1020.84\" cy=\"673.186\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip0702)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1020.84\" cy=\"673.186\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip0702)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"789.012\" cy=\"901.138\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip0702)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"789.012\" cy=\"901.138\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip0702)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1506.47\" cy=\"415.752\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip0702)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1506.47\" cy=\"415.752\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip0702)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1133.11\" cy=\"652.964\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip0702)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1133.11\" cy=\"652.964\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip0702)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"834.844\" cy=\"812.741\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip0702)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"834.844\" cy=\"812.741\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip0702)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"886.659\" cy=\"690.04\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip0702)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"886.659\" cy=\"690.04\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip0702)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"912.212\" cy=\"653.684\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip0702)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"912.212\" cy=\"653.684\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip0702)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1303.23\" cy=\"549.75\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip0702)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1303.23\" cy=\"549.75\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip0702)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1370.61\" cy=\"330.767\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip0702)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1370.61\" cy=\"330.767\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip0702)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1303.28\" cy=\"268.257\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip0702)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1303.28\" cy=\"268.257\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip0702)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1202.4\" cy=\"528.042\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip0702)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1202.4\" cy=\"528.042\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip0702)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"738.769\" cy=\"1014.53\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip0702)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"738.769\" cy=\"1014.53\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip0702)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1411.25\" cy=\"504.806\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip0702)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1411.25\" cy=\"504.806\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip0702)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1205.25\" cy=\"475.386\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip0702)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1205.25\" cy=\"475.386\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip0702)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"773.193\" cy=\"987.619\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip0702)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"773.193\" cy=\"987.619\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip0702)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"930.092\" cy=\"955.487\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip0702)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"930.092\" cy=\"955.487\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip0702)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1490.7\" cy=\"365.518\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip0702)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1490.7\" cy=\"365.518\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip0702)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1105.22\" cy=\"438.535\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip0702)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1105.22\" cy=\"438.535\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip0702)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"801.184\" cy=\"1050.52\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip0702)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"801.184\" cy=\"1050.52\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip0702)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1005.89\" cy=\"842.622\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip0702)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1005.89\" cy=\"842.622\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip0702)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"874.717\" cy=\"703.402\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip0702)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"874.717\" cy=\"703.402\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip0702)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"759.406\" cy=\"837.568\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip0702)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"759.406\" cy=\"837.568\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip0702)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"898.237\" cy=\"953.926\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip0702)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"898.237\" cy=\"953.926\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip0702)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"982.346\" cy=\"679.977\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip0702)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"982.346\" cy=\"679.977\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip0702)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1151.26\" cy=\"711.216\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip0702)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1151.26\" cy=\"711.216\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip0702)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1666.47\" cy=\"90.1936\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip0702)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1666.47\" cy=\"90.1936\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip0702)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1104.36\" cy=\"733.815\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip0702)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1104.36\" cy=\"733.815\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip0702)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"921.766\" cy=\"699.363\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip0702)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"921.766\" cy=\"699.363\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip0702)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1135.81\" cy=\"558.248\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip0702)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1135.81\" cy=\"558.248\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip0702)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1030.3\" cy=\"632.581\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip0702)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1030.3\" cy=\"632.581\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip0702)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1101.02\" cy=\"619.486\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip0702)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1101.02\" cy=\"619.486\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip0702)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1041.12\" cy=\"621.477\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip0702)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1041.12\" cy=\"621.477\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip0702)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1342.56\" cy=\"426.972\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip0702)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1342.56\" cy=\"426.972\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip0702)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1200.95\" cy=\"492.278\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip0702)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1200.95\" cy=\"492.278\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip0702)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1389.38\" cy=\"605.465\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip0702)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1389.38\" cy=\"605.465\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip0702)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"953.723\" cy=\"637.894\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip0702)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"953.723\" cy=\"637.894\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip0702)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1099.87\" cy=\"288.536\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip0702)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1099.87\" cy=\"288.536\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip0702)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1177.49\" cy=\"372.68\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip0702)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1177.49\" cy=\"372.68\" r=\"18\"/>\n",
       "<polyline clip-path=\"url(#clip0702)\" style=\"stroke:#e26f46; stroke-width:25; stroke-opacity:1; fill:none\" points=\"\n",
       "  1501.72,346.812 1498.17,349.341 904.434,772.072 884.139,786.522 1270.76,511.255 1440.62,390.314 700.615,917.189 667.985,940.421 1140.74,603.827 1291.81,496.267 \n",
       "  1290.28,497.356 1070.33,653.956 634.823,964.032 998.144,705.352 1425.95,400.76 996.385,706.605 924.232,757.976 1048.55,669.466 1282.69,502.762 1280.68,504.189 \n",
       "  726.352,898.865 1244.44,529.994 832.186,823.512 1574.58,294.935 717.43,905.217 1292.91,495.48 949.401,740.057 1013.32,694.545 1311.92,481.946 1315.01,479.748 \n",
       "  1059.85,661.419 823.346,829.806 596.849,991.069 1159.69,590.336 1573.09,295.999 975.663,721.358 813.189,837.038 1218.86,548.204 1646.16,243.976 1495.37,351.335 \n",
       "  774.613,864.503 496.172,1062.75 1033.28,680.335 1175.42,579.135 1034.76,679.281 1107.18,627.723 866.135,799.341 833.412,822.639 1037.84,677.089 697.359,919.507 \n",
       "  1102.63,630.963 1246.59,528.459 1101.14,632.021 972.802,723.396 1281.22,503.807 1131.32,610.531 921.543,759.891 1086.69,642.312 496.274,1062.68 1245.22,529.441 \n",
       "  1002.61,702.174 1310,483.318 1020.84,689.195 789.012,854.252 1506.47,343.431 1133.11,609.26 834.844,821.62 886.659,784.728 912.212,766.535 1303.23,488.136 \n",
       "  1370.61,440.16 1303.28,488.098 1202.4,559.923 738.769,890.024 1411.25,411.227 1205.25,557.896 773.193,865.514 930.092,753.804 1490.7,354.657 1105.22,629.116 \n",
       "  801.184,845.585 1005.89,699.834 874.717,793.231 759.406,875.331 898.237,776.485 982.346,716.6 1151.26,596.336 1666.47,229.516 1104.36,629.729 921.766,759.732 \n",
       "  1135.81,607.334 1030.3,682.458 1101.02,632.105 1041.12,674.753 1342.56,460.136 1200.95,560.959 1389.38,426.795 953.723,736.98 1099.87,632.922 1177.49,577.662 \n",
       "  \n",
       "  \"/>\n",
       "<polygon clip-path=\"url(#clip0700)\" points=\"\n",
       "515.063,318.255 1116.41,318.255 1116.41,91.4551 515.063,91.4551 \n",
       "  \" fill=\"#ffffff\" fill-rule=\"evenodd\" fill-opacity=\"1\"/>\n",
       "<polyline clip-path=\"url(#clip0700)\" style=\"stroke:#000000; stroke-width:5; stroke-opacity:1; fill:none\" points=\"\n",
       "  515.063,318.255 1116.41,318.255 1116.41,91.4551 515.063,91.4551 515.063,318.255 \n",
       "  \"/>\n",
       "<circle clip-path=\"url(#clip0700)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"596.063\" cy=\"167.055\" r=\"6\"/>\n",
       "<circle clip-path=\"url(#clip0700)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"596.063\" cy=\"167.055\" r=\"5\"/>\n",
       "<g clip-path=\"url(#clip0700)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:59px; text-anchor:start;\" transform=\"rotate(0, 659.063, 188.555)\" x=\"659.063\" y=\"188.555\">training data</text>\n",
       "</g>\n",
       "<polyline clip-path=\"url(#clip0700)\" style=\"stroke:#e26f46; stroke-width:25; stroke-opacity:1; fill:none\" points=\"\n",
       "  533.063,242.655 641.063,242.655 \n",
       "  \"/>\n",
       "<g clip-path=\"url(#clip0700)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:59px; text-anchor:start;\" transform=\"rotate(0, 659.063, 264.155)\" x=\"659.063\" y=\"264.155\">least squares fit</text>\n",
       "</g>\n",
       "</svg>\n"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "scatter(x,y,\n",
    "    markersize=20,\n",
    "    label=\"training data\",\n",
    "    thickness_scaling=5,\n",
    "    size=(1800,1500),\n",
    "    legend=:topleft\n",
    ")\n",
    "w = sum(x .* y) / sum(x .* x)\n",
    "plot!(x,w .* x,\n",
    "    linewidth=5,\n",
    "    label=\"least squares fit\"\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- But what if we don't trust this feature that much?\n",
    "- Let's take a compromise between our prior model (mean value) and our new model.\n",
    "- _We can achieve this by penalizing large values of $w$_.\n",
    "- The loss becomes\n",
    "\n",
    "$$\\sum_{i=1}^n (wx_i-y_i)^2 + \\lambda w^2$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"utf-8\"?>\n",
       "<svg xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\" width=\"450\" height=\"375\" viewBox=\"0 0 1800 1500\">\n",
       "<defs>\n",
       "  <clipPath id=\"clip1100\">\n",
       "    <rect x=\"0\" y=\"0\" width=\"1800\" height=\"1500\"/>\n",
       "  </clipPath>\n",
       "</defs>\n",
       "<polygon clip-path=\"url(#clip1100)\" points=\"\n",
       "0,1500 1800,1500 1800,0 0,0 \n",
       "  \" fill=\"#ffffff\" fill-rule=\"evenodd\" fill-opacity=\"1\"/>\n",
       "<defs>\n",
       "  <clipPath id=\"clip1101\">\n",
       "    <rect x=\"360\" y=\"0\" width=\"1261\" height=\"1261\"/>\n",
       "  </clipPath>\n",
       "</defs>\n",
       "<polygon clip-path=\"url(#clip1100)\" points=\"\n",
       "461.063,1159.28 1701.57,1159.28 1701.57,59.0551 461.063,59.0551 \n",
       "  \" fill=\"#ffffff\" fill-rule=\"evenodd\" fill-opacity=\"1\"/>\n",
       "<defs>\n",
       "  <clipPath id=\"clip1102\">\n",
       "    <rect x=\"461\" y=\"59\" width=\"1242\" height=\"1101\"/>\n",
       "  </clipPath>\n",
       "</defs>\n",
       "<polyline clip-path=\"url(#clip1102)\" style=\"stroke:#000000; stroke-width:2.5; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  671.766,1159.28 671.766,59.0551 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip1102)\" style=\"stroke:#000000; stroke-width:2.5; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  892.128,1159.28 892.128,59.0551 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip1102)\" style=\"stroke:#000000; stroke-width:2.5; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  1112.49,1159.28 1112.49,59.0551 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip1102)\" style=\"stroke:#000000; stroke-width:2.5; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  1332.85,1159.28 1332.85,59.0551 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip1102)\" style=\"stroke:#000000; stroke-width:2.5; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  1553.21,1159.28 1553.21,59.0551 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip1102)\" style=\"stroke:#000000; stroke-width:2.5; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  461.063,1064.72 1701.57,1064.72 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip1102)\" style=\"stroke:#000000; stroke-width:2.5; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  461.063,844.329 1701.57,844.329 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip1102)\" style=\"stroke:#000000; stroke-width:2.5; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  461.063,623.94 1701.57,623.94 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip1102)\" style=\"stroke:#000000; stroke-width:2.5; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  461.063,403.55 1701.57,403.55 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip1102)\" style=\"stroke:#000000; stroke-width:2.5; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  461.063,183.161 1701.57,183.161 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip1100)\" style=\"stroke:#000000; stroke-width:5; stroke-opacity:1; fill:none\" points=\"\n",
       "  461.063,1159.28 1701.57,1159.28 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip1100)\" style=\"stroke:#000000; stroke-width:5; stroke-opacity:1; fill:none\" points=\"\n",
       "  461.063,1159.28 461.063,59.0551 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip1100)\" style=\"stroke:#000000; stroke-width:5; stroke-opacity:1; fill:none\" points=\"\n",
       "  671.766,1159.28 671.766,1142.78 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip1100)\" style=\"stroke:#000000; stroke-width:5; stroke-opacity:1; fill:none\" points=\"\n",
       "  892.128,1159.28 892.128,1142.78 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip1100)\" style=\"stroke:#000000; stroke-width:5; stroke-opacity:1; fill:none\" points=\"\n",
       "  1112.49,1159.28 1112.49,1142.78 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip1100)\" style=\"stroke:#000000; stroke-width:5; stroke-opacity:1; fill:none\" points=\"\n",
       "  1332.85,1159.28 1332.85,1142.78 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip1100)\" style=\"stroke:#000000; stroke-width:5; stroke-opacity:1; fill:none\" points=\"\n",
       "  1553.21,1159.28 1553.21,1142.78 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip1100)\" style=\"stroke:#000000; stroke-width:5; stroke-opacity:1; fill:none\" points=\"\n",
       "  461.063,1064.72 479.671,1064.72 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip1100)\" style=\"stroke:#000000; stroke-width:5; stroke-opacity:1; fill:none\" points=\"\n",
       "  461.063,844.329 479.671,844.329 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip1100)\" style=\"stroke:#000000; stroke-width:5; stroke-opacity:1; fill:none\" points=\"\n",
       "  461.063,623.94 479.671,623.94 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip1100)\" style=\"stroke:#000000; stroke-width:5; stroke-opacity:1; fill:none\" points=\"\n",
       "  461.063,403.55 479.671,403.55 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip1100)\" style=\"stroke:#000000; stroke-width:5; stroke-opacity:1; fill:none\" points=\"\n",
       "  461.063,183.161 479.671,183.161 \n",
       "  \"/>\n",
       "<g clip-path=\"url(#clip1100)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:59px; text-anchor:middle;\" transform=\"rotate(0, 671.766, 1219.88)\" x=\"671.766\" y=\"1219.88\">-1.0</text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip1100)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:59px; text-anchor:middle;\" transform=\"rotate(0, 892.128, 1219.88)\" x=\"892.128\" y=\"1219.88\">-0.5</text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip1100)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:59px; text-anchor:middle;\" transform=\"rotate(0, 1112.49, 1219.88)\" x=\"1112.49\" y=\"1219.88\">0.0</text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip1100)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:59px; text-anchor:middle;\" transform=\"rotate(0, 1332.85, 1219.88)\" x=\"1332.85\" y=\"1219.88\">0.5</text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip1100)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:59px; text-anchor:middle;\" transform=\"rotate(0, 1553.21, 1219.88)\" x=\"1553.21\" y=\"1219.88\">1.0</text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip1100)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:59px; text-anchor:end;\" transform=\"rotate(0, 443.063, 1086.22)\" x=\"443.063\" y=\"1086.22\">-2</text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip1100)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:59px; text-anchor:end;\" transform=\"rotate(0, 443.063, 865.829)\" x=\"443.063\" y=\"865.829\">-1</text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip1100)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:59px; text-anchor:end;\" transform=\"rotate(0, 443.063, 645.44)\" x=\"443.063\" y=\"645.44\">0</text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip1100)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:59px; text-anchor:end;\" transform=\"rotate(0, 443.063, 425.05)\" x=\"443.063\" y=\"425.05\">1</text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip1100)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:59px; text-anchor:end;\" transform=\"rotate(0, 443.063, 204.661)\" x=\"443.063\" y=\"204.661\">2</text>\n",
       "</g>\n",
       "<circle clip-path=\"url(#clip1102)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1501.72\" cy=\"373.306\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip1102)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1501.72\" cy=\"373.306\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip1102)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1498.17\" cy=\"261.206\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip1102)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1498.17\" cy=\"261.206\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip1102)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"904.434\" cy=\"695.313\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip1102)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"904.434\" cy=\"695.313\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip1102)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"884.139\" cy=\"713.049\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip1102)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"884.139\" cy=\"713.049\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip1102)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1270.76\" cy=\"301.927\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip1102)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1270.76\" cy=\"301.927\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip1102)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1440.62\" cy=\"345.535\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip1102)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1440.62\" cy=\"345.535\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip1102)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"700.615\" cy=\"1100.35\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip1102)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"700.615\" cy=\"1100.35\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip1102)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"667.985\" cy=\"960.433\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip1102)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"667.985\" cy=\"960.433\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip1102)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1140.74\" cy=\"633.481\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip1102)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1140.74\" cy=\"633.481\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip1102)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1291.81\" cy=\"403.253\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip1102)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1291.81\" cy=\"403.253\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip1102)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1290.28\" cy=\"412.492\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip1102)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1290.28\" cy=\"412.492\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip1102)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1070.33\" cy=\"584.604\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip1102)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1070.33\" cy=\"584.604\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip1102)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"634.823\" cy=\"1067.66\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip1102)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"634.823\" cy=\"1067.66\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip1102)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"998.144\" cy=\"760.064\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip1102)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"998.144\" cy=\"760.064\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip1102)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1425.95\" cy=\"640.989\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip1102)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1425.95\" cy=\"640.989\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip1102)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"996.385\" cy=\"733.838\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip1102)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"996.385\" cy=\"733.838\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip1102)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"924.232\" cy=\"789.792\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip1102)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"924.232\" cy=\"789.792\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip1102)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1048.55\" cy=\"707.268\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip1102)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1048.55\" cy=\"707.268\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip1102)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1282.69\" cy=\"389.861\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip1102)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1282.69\" cy=\"389.861\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip1102)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1280.68\" cy=\"273.011\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip1102)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1280.68\" cy=\"273.011\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip1102)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"726.352\" cy=\"841.201\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip1102)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"726.352\" cy=\"841.201\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip1102)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1244.44\" cy=\"723.772\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip1102)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1244.44\" cy=\"723.772\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip1102)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"832.186\" cy=\"778.31\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip1102)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"832.186\" cy=\"778.31\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip1102)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1574.58\" cy=\"310.898\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip1102)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1574.58\" cy=\"310.898\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip1102)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"717.43\" cy=\"1059.86\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip1102)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"717.43\" cy=\"1059.86\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip1102)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1292.91\" cy=\"399.33\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip1102)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1292.91\" cy=\"399.33\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip1102)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"949.401\" cy=\"734.371\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip1102)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"949.401\" cy=\"734.371\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip1102)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1013.32\" cy=\"686.742\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip1102)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1013.32\" cy=\"686.742\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip1102)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1311.92\" cy=\"372.322\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip1102)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1311.92\" cy=\"372.322\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip1102)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1315.01\" cy=\"714.114\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip1102)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1315.01\" cy=\"714.114\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip1102)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1059.85\" cy=\"778.109\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip1102)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1059.85\" cy=\"778.109\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip1102)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"823.346\" cy=\"775.344\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip1102)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"823.346\" cy=\"775.344\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip1102)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"596.849\" cy=\"1128.14\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip1102)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"596.849\" cy=\"1128.14\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip1102)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1159.69\" cy=\"472.073\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip1102)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1159.69\" cy=\"472.073\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip1102)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1573.09\" cy=\"361.513\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip1102)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1573.09\" cy=\"361.513\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip1102)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"975.663\" cy=\"667.407\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip1102)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"975.663\" cy=\"667.407\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip1102)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"813.189\" cy=\"812.75\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip1102)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"813.189\" cy=\"812.75\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip1102)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1218.86\" cy=\"520.218\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip1102)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1218.86\" cy=\"520.218\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip1102)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1646.16\" cy=\"505.774\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip1102)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1646.16\" cy=\"505.774\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip1102)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1495.37\" cy=\"503.83\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip1102)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1495.37\" cy=\"503.83\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip1102)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"774.613\" cy=\"916.92\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip1102)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"774.613\" cy=\"916.92\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip1102)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"496.172\" cy=\"987.789\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip1102)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"496.172\" cy=\"987.789\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip1102)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1033.28\" cy=\"660.445\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip1102)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1033.28\" cy=\"660.445\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip1102)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1175.42\" cy=\"640.812\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip1102)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1175.42\" cy=\"640.812\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip1102)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1034.76\" cy=\"788.966\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip1102)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1034.76\" cy=\"788.966\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip1102)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1107.18\" cy=\"350.923\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip1102)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1107.18\" cy=\"350.923\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip1102)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"866.135\" cy=\"745.351\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip1102)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"866.135\" cy=\"745.351\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip1102)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"833.412\" cy=\"657.664\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip1102)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"833.412\" cy=\"657.664\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip1102)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1037.84\" cy=\"688.534\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip1102)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1037.84\" cy=\"688.534\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip1102)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"697.359\" cy=\"827.25\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip1102)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"697.359\" cy=\"827.25\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip1102)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1102.63\" cy=\"629.163\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip1102)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1102.63\" cy=\"629.163\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip1102)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1246.59\" cy=\"531.845\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip1102)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1246.59\" cy=\"531.845\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip1102)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1101.14\" cy=\"598.294\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip1102)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1101.14\" cy=\"598.294\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip1102)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"972.802\" cy=\"650.803\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip1102)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"972.802\" cy=\"650.803\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip1102)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1281.22\" cy=\"609.042\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip1102)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1281.22\" cy=\"609.042\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip1102)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1131.32\" cy=\"439.684\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip1102)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1131.32\" cy=\"439.684\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip1102)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"921.543\" cy=\"649.695\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip1102)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"921.543\" cy=\"649.695\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip1102)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1086.69\" cy=\"757.471\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip1102)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1086.69\" cy=\"757.471\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip1102)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"496.274\" cy=\"1043.9\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip1102)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"496.274\" cy=\"1043.9\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip1102)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1245.22\" cy=\"359.638\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip1102)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1245.22\" cy=\"359.638\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip1102)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1002.61\" cy=\"711.683\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip1102)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1002.61\" cy=\"711.683\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip1102)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1310\" cy=\"707.221\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip1102)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1310\" cy=\"707.221\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip1102)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1020.84\" cy=\"673.186\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip1102)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1020.84\" cy=\"673.186\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip1102)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"789.012\" cy=\"901.138\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip1102)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"789.012\" cy=\"901.138\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip1102)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1506.47\" cy=\"415.752\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip1102)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1506.47\" cy=\"415.752\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip1102)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1133.11\" cy=\"652.964\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip1102)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1133.11\" cy=\"652.964\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip1102)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"834.844\" cy=\"812.741\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip1102)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"834.844\" cy=\"812.741\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip1102)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"886.659\" cy=\"690.04\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip1102)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"886.659\" cy=\"690.04\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip1102)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"912.212\" cy=\"653.684\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip1102)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"912.212\" cy=\"653.684\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip1102)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1303.23\" cy=\"549.75\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip1102)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1303.23\" cy=\"549.75\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip1102)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1370.61\" cy=\"330.767\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip1102)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1370.61\" cy=\"330.767\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip1102)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1303.28\" cy=\"268.257\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip1102)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1303.28\" cy=\"268.257\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip1102)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1202.4\" cy=\"528.042\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip1102)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1202.4\" cy=\"528.042\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip1102)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"738.769\" cy=\"1014.53\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip1102)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"738.769\" cy=\"1014.53\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip1102)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1411.25\" cy=\"504.806\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip1102)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1411.25\" cy=\"504.806\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip1102)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1205.25\" cy=\"475.386\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip1102)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1205.25\" cy=\"475.386\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip1102)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"773.193\" cy=\"987.619\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip1102)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"773.193\" cy=\"987.619\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip1102)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"930.092\" cy=\"955.487\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip1102)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"930.092\" cy=\"955.487\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip1102)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1490.7\" cy=\"365.518\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip1102)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1490.7\" cy=\"365.518\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip1102)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1105.22\" cy=\"438.535\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip1102)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1105.22\" cy=\"438.535\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip1102)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"801.184\" cy=\"1050.52\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip1102)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"801.184\" cy=\"1050.52\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip1102)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1005.89\" cy=\"842.622\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip1102)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1005.89\" cy=\"842.622\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip1102)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"874.717\" cy=\"703.402\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip1102)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"874.717\" cy=\"703.402\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip1102)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"759.406\" cy=\"837.568\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip1102)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"759.406\" cy=\"837.568\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip1102)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"898.237\" cy=\"953.926\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip1102)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"898.237\" cy=\"953.926\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip1102)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"982.346\" cy=\"679.977\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip1102)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"982.346\" cy=\"679.977\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip1102)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1151.26\" cy=\"711.216\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip1102)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1151.26\" cy=\"711.216\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip1102)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1666.47\" cy=\"90.1936\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip1102)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1666.47\" cy=\"90.1936\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip1102)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1104.36\" cy=\"733.815\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip1102)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1104.36\" cy=\"733.815\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip1102)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"921.766\" cy=\"699.363\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip1102)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"921.766\" cy=\"699.363\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip1102)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1135.81\" cy=\"558.248\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip1102)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1135.81\" cy=\"558.248\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip1102)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1030.3\" cy=\"632.581\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip1102)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1030.3\" cy=\"632.581\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip1102)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1101.02\" cy=\"619.486\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip1102)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1101.02\" cy=\"619.486\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip1102)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1041.12\" cy=\"621.477\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip1102)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1041.12\" cy=\"621.477\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip1102)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1342.56\" cy=\"426.972\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip1102)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1342.56\" cy=\"426.972\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip1102)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1200.95\" cy=\"492.278\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip1102)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1200.95\" cy=\"492.278\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip1102)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1389.38\" cy=\"605.465\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip1102)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1389.38\" cy=\"605.465\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip1102)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"953.723\" cy=\"637.894\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip1102)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"953.723\" cy=\"637.894\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip1102)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1099.87\" cy=\"288.536\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip1102)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1099.87\" cy=\"288.536\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip1102)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1177.49\" cy=\"372.68\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip1102)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1177.49\" cy=\"372.68\" r=\"18\"/>\n",
       "<polyline clip-path=\"url(#clip1102)\" style=\"stroke:#e26f46; stroke-width:25; stroke-opacity:1; fill:none\" points=\"\n",
       "  1501.72,553.158 1498.17,553.804 904.434,661.774 884.139,665.465 1270.76,595.159 1440.62,564.269 700.615,698.839 667.985,704.773 1140.74,618.803 1291.81,591.33 \n",
       "  1290.28,591.609 1070.33,631.606 634.823,710.803 998.144,644.733 1425.95,566.937 996.385,645.053 924.232,658.174 1048.55,635.568 1282.69,592.989 1280.68,593.354 \n",
       "  726.352,694.159 1244.44,599.945 832.186,674.913 1574.58,539.908 717.43,695.781 1292.91,591.13 949.401,653.597 1013.32,641.973 1311.92,587.673 1315.01,587.111 \n",
       "  1059.85,633.512 823.346,676.52 596.849,717.709 1159.69,615.357 1573.09,540.18 975.663,648.821 813.189,678.367 1218.86,604.596 1646.16,526.893 1495.37,554.313 \n",
       "  774.613,685.382 496.172,736.017 1033.28,638.344 1175.42,612.496 1034.76,638.075 1107.18,624.906 866.135,668.739 833.412,674.69 1037.84,637.515 697.359,699.431 \n",
       "  1102.63,625.733 1246.59,599.553 1101.14,626.004 972.802,649.342 1281.22,593.256 1131.32,620.515 921.543,658.663 1086.69,628.632 496.274,735.998 1245.22,599.803 \n",
       "  1002.61,643.922 1310,588.023 1020.84,640.607 789.012,682.764 1506.47,552.294 1133.11,620.19 834.844,674.43 886.659,665.007 912.212,660.36 1303.23,589.254 \n",
       "  1370.61,577 1303.28,589.244 1202.4,607.589 738.769,691.901 1411.25,569.61 1205.25,607.071 773.193,685.641 930.092,657.109 1490.7,555.162 1105.22,625.262 \n",
       "  801.184,680.55 1005.89,643.324 874.717,667.179 759.406,688.148 898.237,662.901 982.346,647.606 1151.26,616.889 1666.47,523.199 1104.36,625.418 921.766,658.623 \n",
       "  1135.81,619.699 1030.3,638.886 1101.02,626.025 1041.12,636.918 1342.56,582.102 1200.95,607.854 1389.38,573.587 953.723,652.811 1099.87,626.234 1177.49,612.12 \n",
       "  \n",
       "  \"/>\n",
       "<polygon clip-path=\"url(#clip1100)\" points=\"\n",
       "515.063,318.255 1116.41,318.255 1116.41,91.4551 515.063,91.4551 \n",
       "  \" fill=\"#ffffff\" fill-rule=\"evenodd\" fill-opacity=\"1\"/>\n",
       "<polyline clip-path=\"url(#clip1100)\" style=\"stroke:#000000; stroke-width:5; stroke-opacity:1; fill:none\" points=\"\n",
       "  515.063,318.255 1116.41,318.255 1116.41,91.4551 515.063,91.4551 515.063,318.255 \n",
       "  \"/>\n",
       "<circle clip-path=\"url(#clip1100)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"596.063\" cy=\"167.055\" r=\"6\"/>\n",
       "<circle clip-path=\"url(#clip1100)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"596.063\" cy=\"167.055\" r=\"5\"/>\n",
       "<g clip-path=\"url(#clip1100)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:59px; text-anchor:start;\" transform=\"rotate(0, 659.063, 188.555)\" x=\"659.063\" y=\"188.555\">training data</text>\n",
       "</g>\n",
       "<polyline clip-path=\"url(#clip1100)\" style=\"stroke:#e26f46; stroke-width:25; stroke-opacity:1; fill:none\" points=\"\n",
       "  533.063,242.655 641.063,242.655 \n",
       "  \"/>\n",
       "<g clip-path=\"url(#clip1100)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:59px; text-anchor:start;\" transform=\"rotate(0, 659.063, 264.155)\" x=\"659.063\" y=\"264.155\">least squares fit</text>\n",
       "</g>\n",
       "</svg>\n"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "scatter(x,y,\n",
    "    markersize=20,\n",
    "    label=\"training data\",\n",
    "    thickness_scaling=5,\n",
    "    size=(1800,1500),\n",
    "    legend=:topleft\n",
    ")\n",
    "w = sum(x .* y)/ (sum(x .* x)+100)\n",
    "plot!(x,w .* x, linewidth=5,label=\"least squares fit\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Makes sense in light of the intuition: more features makes linear regression overfit.\n",
    "- This is an alternative to the hard decisions of feature selection."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Regularization demo"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Generate some synthetic data in 1-d"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"utf-8\"?>\n",
       "<svg xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\" width=\"450\" height=\"375\" viewBox=\"0 0 1800 1500\">\n",
       "<defs>\n",
       "  <clipPath id=\"clip1900\">\n",
       "    <rect x=\"0\" y=\"0\" width=\"1800\" height=\"1500\"/>\n",
       "  </clipPath>\n",
       "</defs>\n",
       "<polygon clip-path=\"url(#clip1900)\" points=\"\n",
       "0,1500 1800,1500 1800,0 0,0 \n",
       "  \" fill=\"#ffffff\" fill-rule=\"evenodd\" fill-opacity=\"1\"/>\n",
       "<defs>\n",
       "  <clipPath id=\"clip1901\">\n",
       "    <rect x=\"360\" y=\"0\" width=\"1261\" height=\"1261\"/>\n",
       "  </clipPath>\n",
       "</defs>\n",
       "<polygon clip-path=\"url(#clip1900)\" points=\"\n",
       "461.063,1159.28 1701.57,1159.28 1701.57,59.0551 461.063,59.0551 \n",
       "  \" fill=\"#ffffff\" fill-rule=\"evenodd\" fill-opacity=\"1\"/>\n",
       "<defs>\n",
       "  <clipPath id=\"clip1902\">\n",
       "    <rect x=\"461\" y=\"59\" width=\"1242\" height=\"1101\"/>\n",
       "  </clipPath>\n",
       "</defs>\n",
       "<polyline clip-path=\"url(#clip1902)\" style=\"stroke:#000000; stroke-width:2.5; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  485.854,1159.28 485.854,59.0551 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip1902)\" style=\"stroke:#000000; stroke-width:2.5; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  780.36,1159.28 780.36,59.0551 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip1902)\" style=\"stroke:#000000; stroke-width:2.5; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  1074.87,1159.28 1074.87,59.0551 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip1902)\" style=\"stroke:#000000; stroke-width:2.5; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  1369.37,1159.28 1369.37,59.0551 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip1902)\" style=\"stroke:#000000; stroke-width:2.5; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  1663.88,1159.28 1663.88,59.0551 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip1902)\" style=\"stroke:#000000; stroke-width:2.5; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  461.063,1069.83 1701.57,1069.83 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip1902)\" style=\"stroke:#000000; stroke-width:2.5; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  461.063,910.709 1701.57,910.709 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip1902)\" style=\"stroke:#000000; stroke-width:2.5; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  461.063,751.586 1701.57,751.586 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip1902)\" style=\"stroke:#000000; stroke-width:2.5; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  461.063,592.462 1701.57,592.462 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip1902)\" style=\"stroke:#000000; stroke-width:2.5; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  461.063,433.338 1701.57,433.338 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip1902)\" style=\"stroke:#000000; stroke-width:2.5; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  461.063,274.215 1701.57,274.215 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip1902)\" style=\"stroke:#000000; stroke-width:2.5; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  461.063,115.091 1701.57,115.091 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip1900)\" style=\"stroke:#000000; stroke-width:5; stroke-opacity:1; fill:none\" points=\"\n",
       "  461.063,1159.28 1701.57,1159.28 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip1900)\" style=\"stroke:#000000; stroke-width:5; stroke-opacity:1; fill:none\" points=\"\n",
       "  461.063,1159.28 461.063,59.0551 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip1900)\" style=\"stroke:#000000; stroke-width:5; stroke-opacity:1; fill:none\" points=\"\n",
       "  485.854,1159.28 485.854,1142.78 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip1900)\" style=\"stroke:#000000; stroke-width:5; stroke-opacity:1; fill:none\" points=\"\n",
       "  780.36,1159.28 780.36,1142.78 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip1900)\" style=\"stroke:#000000; stroke-width:5; stroke-opacity:1; fill:none\" points=\"\n",
       "  1074.87,1159.28 1074.87,1142.78 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip1900)\" style=\"stroke:#000000; stroke-width:5; stroke-opacity:1; fill:none\" points=\"\n",
       "  1369.37,1159.28 1369.37,1142.78 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip1900)\" style=\"stroke:#000000; stroke-width:5; stroke-opacity:1; fill:none\" points=\"\n",
       "  1663.88,1159.28 1663.88,1142.78 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip1900)\" style=\"stroke:#000000; stroke-width:5; stroke-opacity:1; fill:none\" points=\"\n",
       "  461.063,1069.83 479.671,1069.83 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip1900)\" style=\"stroke:#000000; stroke-width:5; stroke-opacity:1; fill:none\" points=\"\n",
       "  461.063,910.709 479.671,910.709 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip1900)\" style=\"stroke:#000000; stroke-width:5; stroke-opacity:1; fill:none\" points=\"\n",
       "  461.063,751.586 479.671,751.586 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip1900)\" style=\"stroke:#000000; stroke-width:5; stroke-opacity:1; fill:none\" points=\"\n",
       "  461.063,592.462 479.671,592.462 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip1900)\" style=\"stroke:#000000; stroke-width:5; stroke-opacity:1; fill:none\" points=\"\n",
       "  461.063,433.338 479.671,433.338 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip1900)\" style=\"stroke:#000000; stroke-width:5; stroke-opacity:1; fill:none\" points=\"\n",
       "  461.063,274.215 479.671,274.215 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip1900)\" style=\"stroke:#000000; stroke-width:5; stroke-opacity:1; fill:none\" points=\"\n",
       "  461.063,115.091 479.671,115.091 \n",
       "  \"/>\n",
       "<g clip-path=\"url(#clip1900)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:59px; text-anchor:middle;\" transform=\"rotate(0, 485.854, 1219.88)\" x=\"485.854\" y=\"1219.88\">-1.0</text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip1900)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:59px; text-anchor:middle;\" transform=\"rotate(0, 780.36, 1219.88)\" x=\"780.36\" y=\"1219.88\">-0.5</text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip1900)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:59px; text-anchor:middle;\" transform=\"rotate(0, 1074.87, 1219.88)\" x=\"1074.87\" y=\"1219.88\">0.0</text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip1900)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:59px; text-anchor:middle;\" transform=\"rotate(0, 1369.37, 1219.88)\" x=\"1369.37\" y=\"1219.88\">0.5</text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip1900)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:59px; text-anchor:middle;\" transform=\"rotate(0, 1663.88, 1219.88)\" x=\"1663.88\" y=\"1219.88\">1.0</text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip1900)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:59px; text-anchor:end;\" transform=\"rotate(0, 443.063, 1091.33)\" x=\"443.063\" y=\"1091.33\">-2</text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip1900)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:59px; text-anchor:end;\" transform=\"rotate(0, 443.063, 932.209)\" x=\"443.063\" y=\"932.209\">-1</text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip1900)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:59px; text-anchor:end;\" transform=\"rotate(0, 443.063, 773.086)\" x=\"443.063\" y=\"773.086\">0</text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip1900)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:59px; text-anchor:end;\" transform=\"rotate(0, 443.063, 613.962)\" x=\"443.063\" y=\"613.962\">1</text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip1900)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:59px; text-anchor:end;\" transform=\"rotate(0, 443.063, 454.838)\" x=\"443.063\" y=\"454.838\">2</text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip1900)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:59px; text-anchor:end;\" transform=\"rotate(0, 443.063, 295.715)\" x=\"443.063\" y=\"295.715\">3</text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip1900)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:59px; text-anchor:end;\" transform=\"rotate(0, 443.063, 136.591)\" x=\"443.063\" y=\"136.591\">4</text>\n",
       "</g>\n",
       "<circle clip-path=\"url(#clip1902)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"496.172\" cy=\"375.74\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip1902)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"496.172\" cy=\"375.74\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip1902)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"543.688\" cy=\"268.046\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip1902)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"543.688\" cy=\"268.046\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip1902)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"609.089\" cy=\"188.707\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip1902)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"609.089\" cy=\"188.707\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip1902)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"667.886\" cy=\"277.503\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip1902)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"667.886\" cy=\"277.503\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip1902)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"729.09\" cy=\"785.85\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip1902)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"729.09\" cy=\"785.85\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip1902)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"789.029\" cy=\"483.973\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip1902)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"789.029\" cy=\"483.973\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip1902)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"861.041\" cy=\"649.55\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip1902)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"861.041\" cy=\"649.55\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip1902)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"918.754\" cy=\"1128.14\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip1902)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"918.754\" cy=\"1128.14\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip1902)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"984.908\" cy=\"827.938\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip1902)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"984.908\" cy=\"827.938\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip1902)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1046.7\" cy=\"754.897\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip1902)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1046.7\" cy=\"754.897\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip1902)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1104.56\" cy=\"305.347\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip1902)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1104.56\" cy=\"305.347\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip1902)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1165.05\" cy=\"520.643\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip1902)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1165.05\" cy=\"520.643\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip1902)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1228.63\" cy=\"471.783\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip1902)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1228.63\" cy=\"471.783\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip1902)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1289.13\" cy=\"879.196\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip1902)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1289.13\" cy=\"879.196\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip1902)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1352.16\" cy=\"601.878\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip1902)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1352.16\" cy=\"601.878\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip1902)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1404.86\" cy=\"639.945\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip1902)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1404.86\" cy=\"639.945\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip1902)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1477.59\" cy=\"425.222\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip1902)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1477.59\" cy=\"425.222\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip1902)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1535.21\" cy=\"237.612\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip1902)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1535.21\" cy=\"237.612\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip1902)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1602.94\" cy=\"538.58\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip1902)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1602.94\" cy=\"538.58\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip1902)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1666.47\" cy=\"90.1936\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip1902)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1666.47\" cy=\"90.1936\" r=\"18\"/>\n",
       "</svg>\n"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "N = 20\n",
    "X = range(-1,stop=1,length=N)+rand(Normal(),N).*0.01\n",
    "X=reshape(X,length(X),1)\n",
    "y = rand(Normal(),N,1) + 3 .* X .^ 2\n",
    "\n",
    "plot(X,y,\n",
    "    seriestype=:scatter,\n",
    "    markersize=20,\n",
    "    thickness_scaling=5,\n",
    "    size=(1800,1500),\n",
    "    legend=false\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Fit (interpolate) with degree $N-1$ polynomial"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"utf-8\"?>\n",
       "<svg xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\" width=\"450\" height=\"375\" viewBox=\"0 0 1800 1500\">\n",
       "<defs>\n",
       "  <clipPath id=\"clip4700\">\n",
       "    <rect x=\"0\" y=\"0\" width=\"1800\" height=\"1500\"/>\n",
       "  </clipPath>\n",
       "</defs>\n",
       "<polygon clip-path=\"url(#clip4700)\" points=\"\n",
       "0,1500 1800,1500 1800,0 0,0 \n",
       "  \" fill=\"#ffffff\" fill-rule=\"evenodd\" fill-opacity=\"1\"/>\n",
       "<defs>\n",
       "  <clipPath id=\"clip4701\">\n",
       "    <rect x=\"360\" y=\"0\" width=\"1261\" height=\"1261\"/>\n",
       "  </clipPath>\n",
       "</defs>\n",
       "<polygon clip-path=\"url(#clip4700)\" points=\"\n",
       "279.205,1159.28 1701.57,1159.28 1701.57,59.0551 279.205,59.0551 \n",
       "  \" fill=\"#ffffff\" fill-rule=\"evenodd\" fill-opacity=\"1\"/>\n",
       "<defs>\n",
       "  <clipPath id=\"clip4702\">\n",
       "    <rect x=\"279\" y=\"59\" width=\"1423\" height=\"1101\"/>\n",
       "  </clipPath>\n",
       "</defs>\n",
       "<polyline clip-path=\"url(#clip4702)\" style=\"stroke:#000000; stroke-width:2.5; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  307.631,1159.28 307.631,59.0551 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip4702)\" style=\"stroke:#000000; stroke-width:2.5; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  645.31,1159.28 645.31,59.0551 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip4702)\" style=\"stroke:#000000; stroke-width:2.5; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  982.99,1159.28 982.99,59.0551 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip4702)\" style=\"stroke:#000000; stroke-width:2.5; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  1320.67,1159.28 1320.67,59.0551 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip4702)\" style=\"stroke:#000000; stroke-width:2.5; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  1658.35,1159.28 1658.35,59.0551 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip4702)\" style=\"stroke:#000000; stroke-width:2.5; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  279.205,1109.43 1701.57,1109.43 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip4702)\" style=\"stroke:#000000; stroke-width:2.5; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  279.205,884.18 1701.57,884.18 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip4702)\" style=\"stroke:#000000; stroke-width:2.5; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  279.205,658.931 1701.57,658.931 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip4702)\" style=\"stroke:#000000; stroke-width:2.5; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  279.205,433.683 1701.57,433.683 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip4702)\" style=\"stroke:#000000; stroke-width:2.5; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  279.205,208.435 1701.57,208.435 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip4700)\" style=\"stroke:#000000; stroke-width:5; stroke-opacity:1; fill:none\" points=\"\n",
       "  279.205,1159.28 1701.57,1159.28 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip4700)\" style=\"stroke:#000000; stroke-width:5; stroke-opacity:1; fill:none\" points=\"\n",
       "  279.205,1159.28 279.205,59.0551 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip4700)\" style=\"stroke:#000000; stroke-width:5; stroke-opacity:1; fill:none\" points=\"\n",
       "  307.631,1159.28 307.631,1142.78 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip4700)\" style=\"stroke:#000000; stroke-width:5; stroke-opacity:1; fill:none\" points=\"\n",
       "  645.31,1159.28 645.31,1142.78 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip4700)\" style=\"stroke:#000000; stroke-width:5; stroke-opacity:1; fill:none\" points=\"\n",
       "  982.99,1159.28 982.99,1142.78 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip4700)\" style=\"stroke:#000000; stroke-width:5; stroke-opacity:1; fill:none\" points=\"\n",
       "  1320.67,1159.28 1320.67,1142.78 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip4700)\" style=\"stroke:#000000; stroke-width:5; stroke-opacity:1; fill:none\" points=\"\n",
       "  1658.35,1159.28 1658.35,1142.78 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip4700)\" style=\"stroke:#000000; stroke-width:5; stroke-opacity:1; fill:none\" points=\"\n",
       "  279.205,1109.43 300.541,1109.43 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip4700)\" style=\"stroke:#000000; stroke-width:5; stroke-opacity:1; fill:none\" points=\"\n",
       "  279.205,884.18 300.541,884.18 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip4700)\" style=\"stroke:#000000; stroke-width:5; stroke-opacity:1; fill:none\" points=\"\n",
       "  279.205,658.931 300.541,658.931 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip4700)\" style=\"stroke:#000000; stroke-width:5; stroke-opacity:1; fill:none\" points=\"\n",
       "  279.205,433.683 300.541,433.683 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip4700)\" style=\"stroke:#000000; stroke-width:5; stroke-opacity:1; fill:none\" points=\"\n",
       "  279.205,208.435 300.541,208.435 \n",
       "  \"/>\n",
       "<g clip-path=\"url(#clip4700)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:59px; text-anchor:middle;\" transform=\"rotate(0, 307.631, 1219.88)\" x=\"307.631\" y=\"1219.88\">-1.0</text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip4700)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:59px; text-anchor:middle;\" transform=\"rotate(0, 645.31, 1219.88)\" x=\"645.31\" y=\"1219.88\">-0.5</text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip4700)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:59px; text-anchor:middle;\" transform=\"rotate(0, 982.99, 1219.88)\" x=\"982.99\" y=\"1219.88\">0.0</text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip4700)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:59px; text-anchor:middle;\" transform=\"rotate(0, 1320.67, 1219.88)\" x=\"1320.67\" y=\"1219.88\">0.5</text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip4700)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:59px; text-anchor:middle;\" transform=\"rotate(0, 1658.35, 1219.88)\" x=\"1658.35\" y=\"1219.88\">1.0</text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip4700)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:59px; text-anchor:start;\" transform=\"rotate(0, 63.3907, 1139.09)\" x=\"63.3907\" y=\"1139.09\">-</text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip4700)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:59px; text-anchor:start;\" transform=\"rotate(0, 98.5283, 1139.09)\" x=\"98.5283\" y=\"1139.09\">2×10</text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip4700)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:48px; text-anchor:start;\" transform=\"rotate(0, 234.025, 1104.82)\" x=\"234.025\" y=\"1104.82\">4</text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip4700)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:59px; text-anchor:end;\" transform=\"rotate(0, 261.205, 905.68)\" x=\"261.205\" y=\"905.68\">0</text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip4700)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:59px; text-anchor:start;\" transform=\"rotate(0, 98.5283, 688.591)\" x=\"98.5283\" y=\"688.591\">2×10</text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip4700)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:48px; text-anchor:start;\" transform=\"rotate(0, 234.025, 654.328)\" x=\"234.025\" y=\"654.328\">4</text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip4700)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:59px; text-anchor:start;\" transform=\"rotate(0, 98.5283, 463.343)\" x=\"98.5283\" y=\"463.343\">4×10</text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip4700)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:48px; text-anchor:start;\" transform=\"rotate(0, 234.025, 429.08)\" x=\"234.025\" y=\"429.08\">4</text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip4700)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:59px; text-anchor:start;\" transform=\"rotate(0, 98.5283, 238.094)\" x=\"98.5283\" y=\"238.094\">6×10</text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip4700)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:48px; text-anchor:start;\" transform=\"rotate(0, 234.025, 203.831)\" x=\"234.025\" y=\"203.831\">4</text>\n",
       "</g>\n",
       "<circle clip-path=\"url(#clip4702)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"319.461\" cy=\"884.153\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip4702)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"319.461\" cy=\"884.153\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip4702)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"373.943\" cy=\"884.145\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip4702)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"373.943\" cy=\"884.145\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip4702)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"448.931\" cy=\"884.14\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip4702)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"448.931\" cy=\"884.14\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip4702)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"516.348\" cy=\"884.146\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip4702)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"516.348\" cy=\"884.146\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip4702)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"586.525\" cy=\"884.182\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip4702)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"586.525\" cy=\"884.182\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip4702)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"655.251\" cy=\"884.161\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip4702)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"655.251\" cy=\"884.161\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip4702)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"737.82\" cy=\"884.172\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip4702)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"737.82\" cy=\"884.172\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip4702)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"803.994\" cy=\"884.206\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip4702)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"803.994\" cy=\"884.206\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip4702)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"879.845\" cy=\"884.185\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip4702)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"879.845\" cy=\"884.185\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip4702)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"950.694\" cy=\"884.18\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip4702)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"950.694\" cy=\"884.18\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip4702)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1017.03\" cy=\"884.148\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip4702)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1017.03\" cy=\"884.148\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip4702)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1086.4\" cy=\"884.163\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip4702)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1086.4\" cy=\"884.163\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip4702)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1159.29\" cy=\"884.16\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip4702)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1159.29\" cy=\"884.16\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip4702)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1228.66\" cy=\"884.189\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip4702)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1228.66\" cy=\"884.189\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip4702)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1300.94\" cy=\"884.169\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip4702)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1300.94\" cy=\"884.169\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip4702)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1361.36\" cy=\"884.172\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip4702)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1361.36\" cy=\"884.172\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip4702)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1444.75\" cy=\"884.157\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip4702)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1444.75\" cy=\"884.157\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip4702)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1510.82\" cy=\"884.143\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip4702)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1510.82\" cy=\"884.143\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip4702)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1588.48\" cy=\"884.165\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip4702)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1588.48\" cy=\"884.165\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip4702)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1661.32\" cy=\"884.133\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip4702)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1661.32\" cy=\"884.133\" r=\"18\"/>\n",
       "<polyline clip-path=\"url(#clip4702)\" style=\"stroke:#e26f46; stroke-width:25; stroke-opacity:1; fill:none\" points=\"\n",
       "  319.461,884.153 320.804,933.006 322.147,974.855 323.49,1010.34 324.834,1040.04 326.177,1064.53 327.52,1084.3 328.863,1099.83 330.206,1111.56 331.55,1119.9 \n",
       "  332.893,1125.21 334.236,1127.86 335.579,1128.14 336.923,1126.37 338.266,1122.8 339.609,1117.68 340.952,1111.23 342.295,1103.68 343.639,1095.19 344.982,1085.94 \n",
       "  346.325,1076.08 347.668,1065.76 349.011,1055.1 350.355,1044.21 351.698,1033.2 353.041,1022.14 354.384,1011.13 355.727,1000.23 357.071,989.507 358.414,979.008 \n",
       "  359.757,968.782 361.1,958.868 362.443,949.298 363.787,940.101 365.13,931.298 366.473,922.906 367.816,914.938 369.159,907.404 370.503,900.309 371.846,893.655 \n",
       "  373.189,887.442 374.532,881.668 375.875,876.327 377.219,871.412 378.562,866.914 379.905,862.824 381.248,859.129 382.591,855.817 383.935,852.874 385.278,850.286 \n",
       "  386.621,848.039 387.964,846.116 389.307,844.502 390.651,843.182 391.994,842.139 393.337,841.356 394.68,840.819 396.023,840.511 397.367,840.416 398.71,840.519 \n",
       "  400.053,840.806 401.396,841.26 402.739,841.867 404.083,842.615 405.426,843.489 406.769,844.475 408.112,845.563 409.455,846.739 410.799,847.992 412.142,849.311 \n",
       "  413.485,850.686 414.828,852.106 416.171,853.563 417.515,855.048 418.858,856.552 420.201,858.068 421.544,859.588 422.887,861.106 424.231,862.615 425.574,864.111 \n",
       "  426.917,865.586 428.26,867.038 429.603,868.461 430.947,869.851 432.29,871.205 433.633,872.52 434.976,873.793 436.319,875.022 437.663,876.204 439.006,877.338 \n",
       "  440.349,878.423 441.692,879.458 443.035,880.441 444.379,881.372 445.722,882.251 447.065,883.077 448.408,883.852 449.751,884.575 451.095,885.246 452.438,885.867 \n",
       "  453.781,886.437 455.124,886.959 456.467,887.433 457.811,887.861 459.154,888.243 460.497,888.582 461.84,888.878 463.183,889.133 464.527,889.349 465.87,889.527 \n",
       "  467.213,889.67 468.556,889.778 469.899,889.854 471.243,889.899 472.586,889.916 473.929,889.905 475.272,889.869 476.615,889.809 477.959,889.727 479.302,889.625 \n",
       "  480.645,889.504 481.988,889.367 483.331,889.214 484.675,889.047 486.018,888.868 487.361,888.678 488.704,888.479 490.047,888.272 491.391,888.058 492.734,887.839 \n",
       "  494.077,887.616 495.42,887.39 496.763,887.161 498.107,886.932 499.45,886.703 500.793,886.475 502.136,886.249 503.479,886.025 504.823,885.805 506.166,885.59 \n",
       "  507.509,885.379 508.852,885.173 510.195,884.973 511.539,884.78 512.882,884.594 514.225,884.415 515.568,884.243 516.911,884.08 518.255,883.924 519.598,883.777 \n",
       "  520.941,883.638 522.284,883.508 523.627,883.387 524.971,883.274 526.314,883.171 527.657,883.075 529,882.989 530.344,882.911 531.687,882.842 533.03,882.781 \n",
       "  534.373,882.728 535.716,882.683 537.06,882.646 538.403,882.617 539.746,882.594 541.089,882.58 542.432,882.571 543.776,882.57 545.119,882.575 546.462,882.585 \n",
       "  547.805,882.602 549.148,882.624 550.492,882.65 551.835,882.682 553.178,882.718 554.521,882.758 555.864,882.802 557.208,882.85 558.551,882.9 559.894,882.953 \n",
       "  561.237,883.009 562.58,883.067 563.924,883.127 565.267,883.189 566.61,883.252 567.953,883.316 569.296,883.381 570.64,883.446 571.983,883.511 573.326,883.577 \n",
       "  574.669,883.642 576.012,883.707 577.356,883.771 578.699,883.835 580.042,883.897 581.385,883.958 582.728,884.018 584.072,884.076 585.415,884.132 586.758,884.186 \n",
       "  588.101,884.239 589.444,884.289 590.788,884.337 592.131,884.383 593.474,884.427 594.817,884.468 596.16,884.507 597.504,884.543 598.847,884.576 600.19,884.607 \n",
       "  601.533,884.635 602.876,884.661 604.22,884.684 605.563,884.704 606.906,884.722 608.249,884.737 609.592,884.75 610.936,884.76 612.279,884.767 613.622,884.772 \n",
       "  614.965,884.775 616.308,884.775 617.652,884.773 618.995,884.769 620.338,884.763 621.681,884.755 623.024,884.745 624.368,884.733 625.711,884.719 627.054,884.704 \n",
       "  628.397,884.687 629.74,884.669 631.084,884.649 632.427,884.628 633.77,884.606 635.113,884.582 636.456,884.558 637.8,884.533 639.143,884.507 640.486,884.481 \n",
       "  641.829,884.454 643.172,884.426 644.516,884.398 645.859,884.37 647.202,884.342 648.545,884.313 649.888,884.285 651.232,884.257 652.575,884.228 653.918,884.201 \n",
       "  655.261,884.173 656.604,884.146 657.948,884.119 659.291,884.093 660.634,884.068 661.977,884.043 663.32,884.019 664.664,883.996 666.007,883.973 667.35,883.952 \n",
       "  668.693,883.931 670.036,883.912 671.38,883.893 672.723,883.876 674.066,883.859 675.409,883.844 676.752,883.83 678.096,883.817 679.439,883.805 680.782,883.794 \n",
       "  682.125,883.784 683.468,883.776 684.812,883.769 686.155,883.763 687.498,883.758 688.841,883.754 690.184,883.752 691.528,883.75 692.871,883.75 694.214,883.75 \n",
       "  695.557,883.752 696.9,883.755 698.244,883.759 699.587,883.764 700.93,883.769 702.273,883.776 703.616,883.783 704.96,883.792 706.303,883.801 707.646,883.811 \n",
       "  708.989,883.822 710.332,883.833 711.676,883.845 713.019,883.857 714.362,883.871 715.705,883.884 717.048,883.898 718.392,883.913 719.735,883.927 721.078,883.943 \n",
       "  722.421,883.958 723.764,883.974 725.108,883.989 726.451,884.005 727.794,884.021 729.137,884.037 730.481,884.053 731.824,884.07 733.167,884.085 734.51,884.101 \n",
       "  735.853,884.117 737.197,884.132 738.54,884.148 739.883,884.163 741.226,884.177 742.569,884.191 743.913,884.205 745.256,884.219 746.599,884.232 747.942,884.245 \n",
       "  749.285,884.257 750.629,884.268 751.972,884.28 753.315,884.29 754.658,884.3 756.001,884.31 757.345,884.319 758.688,884.327 760.031,884.334 761.374,884.341 \n",
       "  762.717,884.348 764.061,884.354 765.404,884.359 766.747,884.363 768.09,884.367 769.433,884.37 770.777,884.373 772.12,884.375 773.463,884.376 774.806,884.377 \n",
       "  776.149,884.377 777.493,884.376 778.836,884.375 780.179,884.373 781.522,884.371 782.865,884.368 784.209,884.365 785.552,884.361 786.895,884.356 788.238,884.352 \n",
       "  789.581,884.346 790.925,884.341 792.268,884.335 793.611,884.328 794.954,884.321 796.297,884.314 797.641,884.306 798.984,884.299 800.327,884.291 801.67,884.282 \n",
       "  803.013,884.274 804.357,884.265 805.7,884.256 807.043,884.247 808.386,884.238 809.729,884.229 811.073,884.22 812.416,884.211 813.759,884.202 815.102,884.192 \n",
       "  816.445,884.183 817.789,884.174 819.132,884.165 820.475,884.157 821.818,884.148 823.161,884.139 824.505,884.131 825.848,884.123 827.191,884.115 828.534,884.107 \n",
       "  829.877,884.1 831.221,884.093 832.564,884.086 833.907,884.08 835.25,884.074 836.593,884.068 837.937,884.062 839.28,884.057 840.623,884.053 841.966,884.048 \n",
       "  843.309,884.045 844.653,884.041 845.996,884.038 847.339,884.035 848.682,884.033 850.025,884.031 851.369,884.03 852.712,884.029 854.055,884.028 855.398,884.028 \n",
       "  856.741,884.029 858.085,884.029 859.428,884.03 860.771,884.032 862.114,884.034 863.457,884.036 864.801,884.039 866.144,884.042 867.487,884.045 868.83,884.049 \n",
       "  870.173,884.053 871.517,884.058 872.86,884.062 874.203,884.067 875.546,884.073 876.889,884.078 878.233,884.084 879.576,884.09 880.919,884.097 882.262,884.103 \n",
       "  883.605,884.11 884.949,884.117 886.292,884.124 887.635,884.131 888.978,884.138 890.321,884.145 891.665,884.153 893.008,884.16 894.351,884.168 895.694,884.175 \n",
       "  897.037,884.183 898.381,884.19 899.724,884.198 901.067,884.205 902.41,884.212 903.753,884.219 905.097,884.226 906.44,884.233 907.783,884.24 909.126,884.247 \n",
       "  910.469,884.253 911.813,884.259 913.156,884.265 914.499,884.271 915.842,884.277 917.185,884.282 918.529,884.287 919.872,884.292 921.215,884.296 922.558,884.3 \n",
       "  923.902,884.304 925.245,884.307 926.588,884.311 927.931,884.313 929.274,884.316 930.618,884.318 931.961,884.32 933.304,884.321 934.647,884.322 935.99,884.323 \n",
       "  937.334,884.323 938.677,884.323 940.02,884.323 941.363,884.322 942.706,884.321 944.05,884.319 945.393,884.317 946.736,884.315 948.079,884.312 949.422,884.309 \n",
       "  950.766,884.306 952.109,884.303 953.452,884.299 954.795,884.294 956.138,884.29 957.482,884.285 958.825,884.28 960.168,884.275 961.511,884.269 962.854,884.263 \n",
       "  964.198,884.257 965.541,884.251 966.884,884.245 968.227,884.238 969.57,884.231 970.914,884.224 972.257,884.217 973.6,884.21 974.943,884.203 976.286,884.195 \n",
       "  977.63,884.188 978.973,884.18 980.316,884.173 981.659,884.165 983.002,884.158 984.346,884.15 985.689,884.143 987.032,884.136 988.375,884.128 989.718,884.121 \n",
       "  991.062,884.114 992.405,884.107 993.748,884.1 995.091,884.093 996.434,884.087 997.778,884.08 999.121,884.074 1000.46,884.068 1001.81,884.063 1003.15,884.057 \n",
       "  1004.49,884.052 1005.84,884.047 1007.18,884.043 1008.52,884.038 1009.87,884.034 1011.21,884.031 1012.55,884.027 1013.9,884.024 1015.24,884.022 1016.58,884.019 \n",
       "  1017.93,884.017 1019.27,884.015 1020.61,884.014 1021.96,884.013 1023.3,884.013 1024.64,884.012 1025.98,884.013 1027.33,884.013 1028.67,884.014 1030.01,884.015 \n",
       "  1031.36,884.017 1032.7,884.019 1034.04,884.021 1035.39,884.024 1036.73,884.027 1038.07,884.03 1039.42,884.034 1040.76,884.038 1042.1,884.042 1043.45,884.047 \n",
       "  1044.79,884.052 1046.13,884.057 1047.48,884.062 1048.82,884.068 1050.16,884.074 1051.51,884.08 1052.85,884.087 1054.19,884.093 1055.54,884.1 1056.88,884.107 \n",
       "  1058.22,884.114 1059.56,884.121 1060.91,884.128 1062.25,884.136 1063.59,884.143 1064.94,884.151 1066.28,884.158 1067.62,884.166 1068.97,884.173 1070.31,884.181 \n",
       "  1071.65,884.188 1073,884.196 1074.34,884.203 1075.68,884.21 1077.03,884.218 1078.37,884.225 1079.71,884.232 1081.06,884.238 1082.4,884.245 1083.74,884.251 \n",
       "  1085.09,884.257 1086.43,884.263 1087.77,884.269 1089.12,884.274 1090.46,884.279 1091.8,884.284 1093.14,884.289 1094.49,884.293 1095.83,884.297 1097.17,884.3 \n",
       "  1098.52,884.303 1099.86,884.306 1101.2,884.309 1102.55,884.311 1103.89,884.312 1105.23,884.314 1106.58,884.314 1107.92,884.315 1109.26,884.315 1110.61,884.315 \n",
       "  1111.95,884.314 1113.29,884.313 1114.64,884.311 1115.98,884.309 1117.32,884.307 1118.67,884.304 1120.01,884.301 1121.35,884.297 1122.7,884.293 1124.04,884.289 \n",
       "  1125.38,884.285 1126.72,884.28 1128.07,884.274 1129.41,884.269 1130.75,884.263 1132.1,884.256 1133.44,884.25 1134.78,884.243 1136.13,884.236 1137.47,884.229 \n",
       "  1138.81,884.221 1140.16,884.213 1141.5,884.205 1142.84,884.197 1144.19,884.189 1145.53,884.181 1146.87,884.173 1148.22,884.164 1149.56,884.156 1150.9,884.147 \n",
       "  1152.25,884.139 1153.59,884.13 1154.93,884.122 1156.28,884.113 1157.62,884.105 1158.96,884.097 1160.3,884.089 1161.65,884.081 1162.99,884.073 1164.33,884.066 \n",
       "  1165.68,884.059 1167.02,884.052 1168.36,884.045 1169.71,884.039 1171.05,884.033 1172.39,884.028 1173.74,884.022 1175.08,884.018 1176.42,884.013 1177.77,884.009 \n",
       "  1179.11,884.006 1180.45,884.003 1181.8,884 1183.14,883.998 1184.48,883.996 1185.83,883.995 1187.17,883.995 1188.51,883.995 1189.86,883.995 1191.2,883.996 \n",
       "  1192.54,883.998 1193.88,884 1195.23,884.003 1196.57,884.006 1197.91,884.01 1199.26,884.014 1200.6,884.019 1201.94,884.025 1203.29,884.031 1204.63,884.038 \n",
       "  1205.97,884.045 1207.32,884.052 1208.66,884.06 1210,884.069 1211.35,884.078 1212.69,884.087 1214.03,884.097 1215.38,884.108 1216.72,884.118 1218.06,884.129 \n",
       "  1219.41,884.141 1220.75,884.152 1222.09,884.164 1223.44,884.176 1224.78,884.188 1226.12,884.201 1227.47,884.213 1228.81,884.226 1230.15,884.239 1231.49,884.252 \n",
       "  1232.84,884.264 1234.18,884.277 1235.52,884.29 1236.87,884.302 1238.21,884.314 1239.55,884.326 1240.9,884.338 1242.24,884.35 1243.58,884.361 1244.93,884.372 \n",
       "  1246.27,884.382 1247.61,884.392 1248.96,884.401 1250.3,884.41 1251.64,884.419 1252.99,884.426 1254.33,884.434 1255.67,884.44 1257.02,884.446 1258.36,884.451 \n",
       "  1259.7,884.455 1261.05,884.458 1262.39,884.461 1263.73,884.463 1265.07,884.463 1266.42,884.463 1267.76,884.462 1269.1,884.46 1270.45,884.458 1271.79,884.454 \n",
       "  1273.13,884.449 1274.48,884.443 1275.82,884.437 1277.16,884.429 1278.51,884.42 1279.85,884.411 1281.19,884.4 1282.54,884.389 1283.88,884.376 1285.22,884.363 \n",
       "  1286.57,884.349 1287.91,884.334 1289.25,884.318 1290.6,884.302 1291.94,884.284 1293.28,884.266 1294.63,884.248 1295.97,884.228 1297.31,884.209 1298.65,884.188 \n",
       "  1300,884.168 1301.34,884.147 1302.68,884.125 1304.03,884.104 1305.37,884.082 1306.71,884.06 1308.06,884.038 1309.4,884.017 1310.74,883.995 1312.09,883.974 \n",
       "  1313.43,883.953 1314.77,883.932 1316.12,883.912 1317.46,883.893 1318.8,883.874 1320.15,883.856 1321.49,883.839 1322.83,883.823 1324.18,883.809 1325.52,883.795 \n",
       "  1326.86,883.783 1328.21,883.772 1329.55,883.762 1330.89,883.755 1332.23,883.749 1333.58,883.744 1334.92,883.742 1336.26,883.741 1337.61,883.743 1338.95,883.747 \n",
       "  1340.29,883.753 1341.64,883.761 1342.98,883.772 1344.32,883.785 1345.67,883.8 1347.01,883.818 1348.35,883.839 1349.7,883.862 1351.04,883.888 1352.38,883.917 \n",
       "  1353.73,883.948 1355.07,883.982 1356.41,884.019 1357.76,884.058 1359.1,884.1 1360.44,884.145 1361.79,884.192 1363.13,884.242 1364.47,884.295 1365.81,884.35 \n",
       "  1367.16,884.407 1368.5,884.467 1369.84,884.529 1371.19,884.593 1372.53,884.659 1373.87,884.726 1375.22,884.796 1376.56,884.867 1377.9,884.939 1379.25,885.012 \n",
       "  1380.59,885.087 1381.93,885.162 1383.28,885.237 1384.62,885.313 1385.96,885.389 1387.31,885.465 1388.65,885.54 1389.99,885.615 1391.34,885.688 1392.68,885.761 \n",
       "  1394.02,885.831 1395.37,885.9 1396.71,885.966 1398.05,886.03 1399.39,886.091 1400.74,886.148 1402.08,886.202 1403.42,886.252 1404.77,886.297 1406.11,886.338 \n",
       "  1407.45,886.374 1408.8,886.404 1410.14,886.429 1411.48,886.447 1412.83,886.459 1414.17,886.464 1415.51,886.462 1416.86,886.452 1418.2,886.435 1419.54,886.409 \n",
       "  1420.89,886.375 1422.23,886.333 1423.57,886.281 1424.92,886.22 1426.26,886.149 1427.6,886.069 1428.95,885.979 1430.29,885.879 1431.63,885.769 1432.97,885.648 \n",
       "  1434.32,885.518 1435.66,885.377 1437,885.225 1438.35,885.064 1439.69,884.892 1441.03,884.71 1442.38,884.518 1443.72,884.316 1445.06,884.106 1446.41,883.886 \n",
       "  1447.75,883.657 1449.09,883.42 1450.44,883.175 1451.78,882.924 1453.12,882.665 1454.47,882.401 1455.81,882.131 1457.15,881.857 1458.5,881.579 1459.84,881.299 \n",
       "  1461.18,881.017 1462.53,880.734 1463.87,880.452 1465.21,880.172 1466.55,879.895 1467.9,879.622 1469.24,879.355 1470.58,879.096 1471.93,878.845 1473.27,878.605 \n",
       "  1474.61,878.377 1475.96,878.163 1477.3,877.964 1478.64,877.783 1479.99,877.622 1481.33,877.482 1482.67,877.366 1484.02,877.275 1485.36,877.213 1486.7,877.18 \n",
       "  1488.05,877.179 1489.39,877.212 1490.73,877.282 1492.08,877.391 1493.42,877.54 1494.76,877.733 1496.11,877.971 1497.45,878.257 1498.79,878.593 1500.13,878.981 \n",
       "  1501.48,879.423 1502.82,879.92 1504.16,880.476 1505.51,881.091 1506.85,881.768 1508.19,882.509 1509.54,883.313 1510.88,884.184 1512.22,885.122 1513.57,886.127 \n",
       "  1514.91,887.202 1516.25,888.346 1517.6,889.559 1518.94,890.842 1520.28,892.195 1521.63,893.616 1522.97,895.105 1524.31,896.662 1525.66,898.283 1527,899.968 \n",
       "  1528.34,901.713 1529.69,903.517 1531.03,905.375 1532.37,907.284 1533.71,909.24 1535.06,911.238 1536.4,913.272 1537.74,915.336 1539.09,917.425 1540.43,919.529 \n",
       "  1541.77,921.643 1543.12,923.756 1544.46,925.861 1545.8,927.945 1547.15,930 1548.49,932.014 1549.83,933.973 1551.18,935.865 1552.52,937.676 1553.86,939.391 \n",
       "  1555.21,940.995 1556.55,942.47 1557.89,943.8 1559.24,944.966 1560.58,945.948 1561.92,946.728 1563.27,947.284 1564.61,947.594 1565.95,947.636 1567.29,947.386 \n",
       "  1568.64,946.821 1569.98,945.914 1571.32,944.642 1572.67,942.976 1574.01,940.89 1575.35,938.357 1576.7,935.349 1578.04,931.837 1579.38,927.793 1580.73,923.187 \n",
       "  1582.07,917.99 1583.41,912.174 1584.76,905.708 1586.1,898.566 1587.44,890.718 1588.79,882.136 1590.13,872.795 1591.47,862.669 1592.82,851.732 1594.16,839.962 \n",
       "  1595.5,827.339 1596.85,813.842 1598.19,799.457 1599.53,784.167 1600.87,767.964 1602.22,750.84 1603.56,732.792 1604.9,713.821 1606.25,693.934 1607.59,673.141 \n",
       "  1608.93,651.461 1610.28,628.918 1611.62,605.543 1612.96,581.376 1614.31,556.466 1615.65,530.868 1616.99,504.653 1618.34,477.898 1619.68,450.694 1621.02,423.146 \n",
       "  1622.37,395.372 1623.71,367.505 1625.05,339.696 1626.4,312.112 1627.74,284.939 1629.08,258.385 1630.43,232.678 1631.77,208.071 1633.11,184.841 1634.46,163.292 \n",
       "  1635.8,143.757 1637.14,126.599 1638.48,112.214 1639.83,101.033 1641.17,93.5241 1642.51,90.1936 1643.86,91.5909 1645.2,98.3096 1646.54,110.99 1647.89,130.325 \n",
       "  1649.23,157.057 1650.57,191.987 1651.92,235.976 1653.26,289.949 1654.6,354.896 1655.95,431.879 1657.29,522.035 1658.63,626.581 1659.98,746.818 1661.32,884.133 \n",
       "  \n",
       "  \"/>\n",
       "</svg>\n"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pf = initializePolynomialFeatures(X,degree=N-1)\n",
    "Z = pf.transform(X)\n",
    "v = Z'*Z \\ Z'*y\n",
    "grid = reshape(range(minimum(X),stop=maximum(X),length=1000),1000,1)\n",
    "grid_poly = initializePolynomialFeatures(grid,degree=N-1).transform(grid)\n",
    "\n",
    "plot(X,y,\n",
    "    seriestype=:scatter,\n",
    "    markersize=20,\n",
    "    thickness_scaling=5,\n",
    "    size=(1800,1500),\n",
    "    left_margin=-200px,\n",
    "    legend=false\n",
    ")\n",
    "plot!(grid, grid_poly*v,\n",
    "    linewidth=5\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[1.92798; 335.352; -260.078; -33089.8; 12792.0; 9.53521e5; -250968.0; -1.21062e7; 2.33334e6; 8.0313e7; -1.12552e7; -3.02086e8; 2.94971e7; 6.63771e8; -4.19789e7; -8.39162e8; 3.0187e7; 5.62708e8; -8.49439e6; -1.54384e8]\n",
      "839162053"
     ]
    }
   ],
   "source": [
    "Printf.@printf(\"%s\\n\",v)\n",
    "Printf.@printf(\"%d\",maximum(broadcast(abs,v)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Problem: this results are crazy (overfitting). Interpretation: many large weights are cancelling each other out at the data points.\n",
    "\n",
    "### Solution 1: lower degree polynomial"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"utf-8\"?>\n",
       "<svg xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\" width=\"450\" height=\"375\" viewBox=\"0 0 1800 1500\">\n",
       "<defs>\n",
       "  <clipPath id=\"clip5100\">\n",
       "    <rect x=\"0\" y=\"0\" width=\"1800\" height=\"1500\"/>\n",
       "  </clipPath>\n",
       "</defs>\n",
       "<polygon clip-path=\"url(#clip5100)\" points=\"\n",
       "0,1500 1800,1500 1800,0 0,0 \n",
       "  \" fill=\"#ffffff\" fill-rule=\"evenodd\" fill-opacity=\"1\"/>\n",
       "<defs>\n",
       "  <clipPath id=\"clip5101\">\n",
       "    <rect x=\"360\" y=\"0\" width=\"1261\" height=\"1261\"/>\n",
       "  </clipPath>\n",
       "</defs>\n",
       "<polygon clip-path=\"url(#clip5100)\" points=\"\n",
       "461.063,1159.28 1701.57,1159.28 1701.57,59.0551 461.063,59.0551 \n",
       "  \" fill=\"#ffffff\" fill-rule=\"evenodd\" fill-opacity=\"1\"/>\n",
       "<defs>\n",
       "  <clipPath id=\"clip5102\">\n",
       "    <rect x=\"461\" y=\"59\" width=\"1242\" height=\"1101\"/>\n",
       "  </clipPath>\n",
       "</defs>\n",
       "<polyline clip-path=\"url(#clip5102)\" style=\"stroke:#000000; stroke-width:2.5; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  485.854,1159.28 485.854,59.0551 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip5102)\" style=\"stroke:#000000; stroke-width:2.5; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  780.36,1159.28 780.36,59.0551 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip5102)\" style=\"stroke:#000000; stroke-width:2.5; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  1074.87,1159.28 1074.87,59.0551 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip5102)\" style=\"stroke:#000000; stroke-width:2.5; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  1369.37,1159.28 1369.37,59.0551 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip5102)\" style=\"stroke:#000000; stroke-width:2.5; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  1663.88,1159.28 1663.88,59.0551 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip5102)\" style=\"stroke:#000000; stroke-width:2.5; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  461.063,1069.83 1701.57,1069.83 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip5102)\" style=\"stroke:#000000; stroke-width:2.5; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  461.063,910.709 1701.57,910.709 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip5102)\" style=\"stroke:#000000; stroke-width:2.5; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  461.063,751.586 1701.57,751.586 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip5102)\" style=\"stroke:#000000; stroke-width:2.5; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  461.063,592.462 1701.57,592.462 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip5102)\" style=\"stroke:#000000; stroke-width:2.5; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  461.063,433.338 1701.57,433.338 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip5102)\" style=\"stroke:#000000; stroke-width:2.5; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  461.063,274.215 1701.57,274.215 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip5102)\" style=\"stroke:#000000; stroke-width:2.5; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  461.063,115.091 1701.57,115.091 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip5100)\" style=\"stroke:#000000; stroke-width:5; stroke-opacity:1; fill:none\" points=\"\n",
       "  461.063,1159.28 1701.57,1159.28 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip5100)\" style=\"stroke:#000000; stroke-width:5; stroke-opacity:1; fill:none\" points=\"\n",
       "  461.063,1159.28 461.063,59.0551 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip5100)\" style=\"stroke:#000000; stroke-width:5; stroke-opacity:1; fill:none\" points=\"\n",
       "  485.854,1159.28 485.854,1142.78 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip5100)\" style=\"stroke:#000000; stroke-width:5; stroke-opacity:1; fill:none\" points=\"\n",
       "  780.36,1159.28 780.36,1142.78 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip5100)\" style=\"stroke:#000000; stroke-width:5; stroke-opacity:1; fill:none\" points=\"\n",
       "  1074.87,1159.28 1074.87,1142.78 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip5100)\" style=\"stroke:#000000; stroke-width:5; stroke-opacity:1; fill:none\" points=\"\n",
       "  1369.37,1159.28 1369.37,1142.78 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip5100)\" style=\"stroke:#000000; stroke-width:5; stroke-opacity:1; fill:none\" points=\"\n",
       "  1663.88,1159.28 1663.88,1142.78 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip5100)\" style=\"stroke:#000000; stroke-width:5; stroke-opacity:1; fill:none\" points=\"\n",
       "  461.063,1069.83 479.671,1069.83 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip5100)\" style=\"stroke:#000000; stroke-width:5; stroke-opacity:1; fill:none\" points=\"\n",
       "  461.063,910.709 479.671,910.709 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip5100)\" style=\"stroke:#000000; stroke-width:5; stroke-opacity:1; fill:none\" points=\"\n",
       "  461.063,751.586 479.671,751.586 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip5100)\" style=\"stroke:#000000; stroke-width:5; stroke-opacity:1; fill:none\" points=\"\n",
       "  461.063,592.462 479.671,592.462 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip5100)\" style=\"stroke:#000000; stroke-width:5; stroke-opacity:1; fill:none\" points=\"\n",
       "  461.063,433.338 479.671,433.338 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip5100)\" style=\"stroke:#000000; stroke-width:5; stroke-opacity:1; fill:none\" points=\"\n",
       "  461.063,274.215 479.671,274.215 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip5100)\" style=\"stroke:#000000; stroke-width:5; stroke-opacity:1; fill:none\" points=\"\n",
       "  461.063,115.091 479.671,115.091 \n",
       "  \"/>\n",
       "<g clip-path=\"url(#clip5100)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:59px; text-anchor:middle;\" transform=\"rotate(0, 485.854, 1219.88)\" x=\"485.854\" y=\"1219.88\">-1.0</text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip5100)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:59px; text-anchor:middle;\" transform=\"rotate(0, 780.36, 1219.88)\" x=\"780.36\" y=\"1219.88\">-0.5</text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip5100)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:59px; text-anchor:middle;\" transform=\"rotate(0, 1074.87, 1219.88)\" x=\"1074.87\" y=\"1219.88\">0.0</text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip5100)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:59px; text-anchor:middle;\" transform=\"rotate(0, 1369.37, 1219.88)\" x=\"1369.37\" y=\"1219.88\">0.5</text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip5100)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:59px; text-anchor:middle;\" transform=\"rotate(0, 1663.88, 1219.88)\" x=\"1663.88\" y=\"1219.88\">1.0</text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip5100)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:59px; text-anchor:end;\" transform=\"rotate(0, 443.063, 1091.33)\" x=\"443.063\" y=\"1091.33\">-2</text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip5100)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:59px; text-anchor:end;\" transform=\"rotate(0, 443.063, 932.209)\" x=\"443.063\" y=\"932.209\">-1</text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip5100)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:59px; text-anchor:end;\" transform=\"rotate(0, 443.063, 773.086)\" x=\"443.063\" y=\"773.086\">0</text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip5100)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:59px; text-anchor:end;\" transform=\"rotate(0, 443.063, 613.962)\" x=\"443.063\" y=\"613.962\">1</text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip5100)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:59px; text-anchor:end;\" transform=\"rotate(0, 443.063, 454.838)\" x=\"443.063\" y=\"454.838\">2</text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip5100)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:59px; text-anchor:end;\" transform=\"rotate(0, 443.063, 295.715)\" x=\"443.063\" y=\"295.715\">3</text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip5100)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:59px; text-anchor:end;\" transform=\"rotate(0, 443.063, 136.591)\" x=\"443.063\" y=\"136.591\">4</text>\n",
       "</g>\n",
       "<circle clip-path=\"url(#clip5102)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"496.172\" cy=\"375.74\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip5102)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"496.172\" cy=\"375.74\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip5102)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"543.688\" cy=\"268.046\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip5102)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"543.688\" cy=\"268.046\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip5102)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"609.089\" cy=\"188.707\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip5102)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"609.089\" cy=\"188.707\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip5102)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"667.886\" cy=\"277.503\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip5102)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"667.886\" cy=\"277.503\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip5102)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"729.09\" cy=\"785.85\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip5102)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"729.09\" cy=\"785.85\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip5102)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"789.029\" cy=\"483.973\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip5102)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"789.029\" cy=\"483.973\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip5102)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"861.041\" cy=\"649.55\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip5102)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"861.041\" cy=\"649.55\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip5102)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"918.754\" cy=\"1128.14\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip5102)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"918.754\" cy=\"1128.14\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip5102)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"984.908\" cy=\"827.938\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip5102)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"984.908\" cy=\"827.938\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip5102)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1046.7\" cy=\"754.897\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip5102)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1046.7\" cy=\"754.897\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip5102)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1104.56\" cy=\"305.347\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip5102)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1104.56\" cy=\"305.347\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip5102)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1165.05\" cy=\"520.643\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip5102)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1165.05\" cy=\"520.643\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip5102)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1228.63\" cy=\"471.783\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip5102)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1228.63\" cy=\"471.783\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip5102)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1289.13\" cy=\"879.196\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip5102)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1289.13\" cy=\"879.196\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip5102)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1352.16\" cy=\"601.878\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip5102)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1352.16\" cy=\"601.878\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip5102)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1404.86\" cy=\"639.945\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip5102)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1404.86\" cy=\"639.945\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip5102)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1477.59\" cy=\"425.222\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip5102)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1477.59\" cy=\"425.222\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip5102)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1535.21\" cy=\"237.612\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip5102)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1535.21\" cy=\"237.612\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip5102)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1602.94\" cy=\"538.58\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip5102)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1602.94\" cy=\"538.58\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip5102)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1666.47\" cy=\"90.1936\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip5102)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1666.47\" cy=\"90.1936\" r=\"18\"/>\n",
       "<polyline clip-path=\"url(#clip5102)\" style=\"stroke:#e26f46; stroke-width:25; stroke-opacity:1; fill:none\" points=\"\n",
       "  496.172,235.477 497.343,237.406 498.515,239.332 499.686,241.253 500.858,243.17 502.029,245.083 503.201,246.992 504.372,248.898 505.543,250.799 506.715,252.696 \n",
       "  507.886,254.59 509.058,256.479 510.229,258.365 511.401,260.246 512.572,262.124 513.744,263.997 514.915,265.867 516.087,267.733 517.258,269.594 518.43,271.452 \n",
       "  519.601,273.306 520.772,275.156 521.944,277.002 523.115,278.843 524.287,280.681 525.458,282.515 526.63,284.345 527.801,286.172 528.973,287.994 530.144,289.812 \n",
       "  531.316,291.626 532.487,293.436 533.659,295.243 534.83,297.045 536.002,298.843 537.173,300.638 538.344,302.428 539.516,304.215 540.687,305.997 541.859,307.776 \n",
       "  543.03,309.55 544.202,311.321 545.373,313.088 546.545,314.85 547.716,316.609 548.888,318.364 550.059,320.115 551.231,321.862 552.402,323.605 553.574,325.344 \n",
       "  554.745,327.079 555.916,328.81 557.088,330.537 558.259,332.26 559.431,333.979 560.602,335.694 561.774,337.405 562.945,339.113 564.117,340.816 565.288,342.515 \n",
       "  566.46,344.211 567.631,345.902 568.803,347.59 569.974,349.273 571.146,350.953 572.317,352.629 573.488,354.3 574.66,355.968 575.831,357.632 577.003,359.291 \n",
       "  578.174,360.947 579.346,362.599 580.517,364.247 581.689,365.891 582.86,367.531 584.032,369.167 585.203,370.799 586.375,372.427 587.546,374.051 588.718,375.671 \n",
       "  589.889,377.288 591.06,378.9 592.232,380.508 593.403,382.113 594.575,383.713 595.746,385.309 596.918,386.902 598.089,388.49 599.261,390.075 600.432,391.655 \n",
       "  601.604,393.232 602.775,394.805 603.947,396.373 605.118,397.938 606.289,399.499 607.461,401.056 608.632,402.609 609.804,404.158 610.975,405.703 612.147,407.243 \n",
       "  613.318,408.781 614.49,410.314 615.661,411.843 616.833,413.368 618.004,414.889 619.176,416.406 620.347,417.92 621.519,419.429 622.69,420.934 623.861,422.436 \n",
       "  625.033,423.933 626.204,425.426 627.376,426.916 628.547,428.402 629.719,429.883 630.89,431.361 632.062,432.834 633.233,434.304 634.405,435.77 635.576,437.232 \n",
       "  636.748,438.69 637.919,440.143 639.091,441.593 640.262,443.039 641.433,444.481 642.605,445.919 643.776,447.353 644.948,448.784 646.119,450.21 647.291,451.632 \n",
       "  648.462,453.05 649.634,454.464 650.805,455.875 651.977,457.281 653.148,458.683 654.32,460.082 655.491,461.476 656.663,462.867 657.834,464.253 659.005,465.636 \n",
       "  660.177,467.015 661.348,468.389 662.52,469.76 663.691,471.127 664.863,472.49 666.034,473.848 667.206,475.203 668.377,476.554 669.549,477.901 670.72,479.244 \n",
       "  671.892,480.583 673.063,481.918 674.235,483.25 675.406,484.577 676.577,485.9 677.749,487.219 678.92,488.534 680.092,489.846 681.263,491.153 682.435,492.457 \n",
       "  683.606,493.756 684.778,495.051 685.949,496.343 687.121,497.631 688.292,498.914 689.464,500.194 690.635,501.47 691.806,502.741 692.978,504.009 694.149,505.273 \n",
       "  695.321,506.533 696.492,507.789 697.664,509.04 698.835,510.288 700.007,511.532 701.178,512.773 702.35,514.009 703.521,515.241 704.693,516.469 705.864,517.693 \n",
       "  707.036,518.913 708.207,520.13 709.378,521.342 710.55,522.55 711.721,523.755 712.893,524.955 714.064,526.152 715.236,527.344 716.407,528.533 717.579,529.717 \n",
       "  718.75,530.898 719.922,532.075 721.093,533.248 722.265,534.416 723.436,535.581 724.608,536.742 725.779,537.899 726.95,539.052 728.122,540.201 729.293,541.346 \n",
       "  730.465,542.487 731.636,543.624 732.808,544.757 733.979,545.886 735.151,547.012 736.322,548.133 737.494,549.25 738.665,550.363 739.837,551.473 741.008,552.578 \n",
       "  742.18,553.68 743.351,554.777 744.522,555.871 745.694,556.96 746.865,558.046 748.037,559.128 749.208,560.205 750.38,561.279 751.551,562.349 752.723,563.415 \n",
       "  753.894,564.477 755.066,565.535 756.237,566.588 757.409,567.638 758.58,568.685 759.751,569.727 760.923,570.765 762.094,571.799 763.266,572.829 764.437,573.855 \n",
       "  765.609,574.878 766.78,575.896 767.952,576.91 769.123,577.921 770.295,578.927 771.466,579.929 772.638,580.928 773.809,581.923 774.981,582.913 776.152,583.9 \n",
       "  777.323,584.882 778.495,585.861 779.666,586.836 780.838,587.807 782.009,588.774 783.181,589.736 784.352,590.695 785.524,591.65 786.695,592.601 787.867,593.548 \n",
       "  789.038,594.491 790.21,595.431 791.381,596.366 792.553,597.297 793.724,598.224 794.895,599.147 796.067,600.067 797.238,600.982 798.41,601.893 799.581,602.801 \n",
       "  800.753,603.704 801.924,604.604 803.096,605.499 804.267,606.391 805.439,607.279 806.61,608.162 807.782,609.042 808.953,609.918 810.125,610.79 811.296,611.658 \n",
       "  812.467,612.521 813.639,613.381 814.81,614.237 815.982,615.089 817.153,615.937 818.325,616.781 819.496,617.622 820.668,618.458 821.839,619.29 823.011,620.118 \n",
       "  824.182,620.942 825.354,621.763 826.525,622.579 827.697,623.392 828.868,624.2 830.039,625.005 831.211,625.805 832.382,626.602 833.554,627.394 834.725,628.183 \n",
       "  835.897,628.968 837.068,629.748 838.24,630.525 839.411,631.298 840.583,632.067 841.754,632.832 842.926,633.593 844.097,634.35 845.268,635.103 846.44,635.852 \n",
       "  847.611,636.597 848.783,637.338 849.954,638.075 851.126,638.808 852.297,639.537 853.469,640.263 854.64,640.984 855.812,641.702 856.983,642.415 858.155,643.124 \n",
       "  859.326,643.83 860.498,644.531 861.669,645.229 862.84,645.923 864.012,646.612 865.183,647.298 866.355,647.98 867.526,648.657 868.698,649.331 869.869,650.001 \n",
       "  871.041,650.667 872.212,651.329 873.384,651.987 874.555,652.641 875.727,653.291 876.898,653.937 878.07,654.579 879.241,655.218 880.412,655.852 881.584,656.482 \n",
       "  882.755,657.108 883.927,657.731 885.098,658.349 886.27,658.963 887.441,659.574 888.613,660.18 889.784,660.783 890.956,661.382 892.127,661.976 893.299,662.567 \n",
       "  894.47,663.154 895.642,663.736 896.813,664.315 897.984,664.89 899.156,665.461 900.327,666.028 901.499,666.591 902.67,667.15 903.842,667.705 905.013,668.256 \n",
       "  906.185,668.803 907.356,669.346 908.528,669.885 909.699,670.42 910.871,670.952 912.042,671.479 913.214,672.002 914.385,672.522 915.556,673.037 916.728,673.549 \n",
       "  917.899,674.056 919.071,674.56 920.242,675.059 921.414,675.555 922.585,676.047 923.757,676.534 924.928,677.018 926.1,677.498 927.271,677.974 928.443,678.446 \n",
       "  929.614,678.914 930.785,679.378 931.957,679.838 933.128,680.294 934.3,680.746 935.471,681.194 936.643,681.638 937.814,682.078 938.986,682.514 940.157,682.947 \n",
       "  941.329,683.375 942.5,683.799 943.672,684.22 944.843,684.636 946.015,685.049 947.186,685.457 948.357,685.862 949.529,686.262 950.7,686.659 951.872,687.052 \n",
       "  953.043,687.44 954.215,687.825 955.386,688.206 956.558,688.583 957.729,688.956 958.901,689.325 960.072,689.69 961.244,690.051 962.415,690.408 963.587,690.761 \n",
       "  964.758,691.11 965.929,691.455 967.101,691.796 968.272,692.133 969.444,692.467 970.615,692.796 971.787,693.121 972.958,693.443 974.13,693.76 975.301,694.074 \n",
       "  976.473,694.383 977.644,694.689 978.816,694.991 979.987,695.288 981.159,695.582 982.33,695.872 983.501,696.157 984.673,696.439 985.844,696.717 987.016,696.991 \n",
       "  988.187,697.261 989.359,697.527 990.53,697.789 991.702,698.047 992.873,698.301 994.045,698.551 995.216,698.797 996.388,699.04 997.559,699.278 998.731,699.512 \n",
       "  999.902,699.742 1001.07,699.969 1002.24,700.191 1003.42,700.41 1004.59,700.624 1005.76,700.835 1006.93,701.041 1008.1,701.244 1009.27,701.443 1010.45,701.637 \n",
       "  1011.62,701.828 1012.79,702.015 1013.96,702.198 1015.13,702.376 1016.3,702.551 1017.47,702.722 1018.65,702.889 1019.82,703.052 1020.99,703.211 1022.16,703.366 \n",
       "  1023.33,703.518 1024.5,703.665 1025.67,703.808 1026.85,703.947 1028.02,704.083 1029.19,704.214 1030.36,704.341 1031.53,704.465 1032.7,704.584 1033.87,704.7 \n",
       "  1035.05,704.811 1036.22,704.919 1037.39,705.023 1038.56,705.122 1039.73,705.218 1040.9,705.31 1042.07,705.397 1043.25,705.481 1044.42,705.561 1045.59,705.637 \n",
       "  1046.76,705.709 1047.93,705.777 1049.1,705.841 1050.28,705.901 1051.45,705.957 1052.62,706.009 1053.79,706.058 1054.96,706.102 1056.13,706.142 1057.3,706.178 \n",
       "  1058.48,706.211 1059.65,706.239 1060.82,706.264 1061.99,706.284 1063.16,706.301 1064.33,706.313 1065.5,706.322 1066.68,706.326 1067.85,706.327 1069.02,706.324 \n",
       "  1070.19,706.316 1071.36,706.305 1072.53,706.29 1073.7,706.271 1074.88,706.248 1076.05,706.221 1077.22,706.19 1078.39,706.155 1079.56,706.116 1080.73,706.073 \n",
       "  1081.9,706.026 1083.08,705.975 1084.25,705.921 1085.42,705.862 1086.59,705.799 1087.76,705.733 1088.93,705.662 1090.1,705.588 1091.28,705.509 1092.45,705.427 \n",
       "  1093.62,705.34 1094.79,705.25 1095.96,705.155 1097.13,705.057 1098.31,704.955 1099.48,704.849 1100.65,704.738 1101.82,704.624 1102.99,704.506 1104.16,704.384 \n",
       "  1105.33,704.258 1106.51,704.128 1107.68,703.994 1108.85,703.856 1110.02,703.714 1111.19,703.568 1112.36,703.419 1113.53,703.265 1114.71,703.107 1115.88,702.946 \n",
       "  1117.05,702.78 1118.22,702.61 1119.39,702.437 1120.56,702.259 1121.73,702.078 1122.91,701.892 1124.08,701.703 1125.25,701.51 1126.42,701.312 1127.59,701.111 \n",
       "  1128.76,700.906 1129.93,700.697 1131.11,700.483 1132.28,700.266 1133.45,700.045 1134.62,699.82 1135.79,699.591 1136.96,699.358 1138.13,699.121 1139.31,698.881 \n",
       "  1140.48,698.636 1141.65,698.387 1142.82,698.134 1143.99,697.878 1145.16,697.617 1146.34,697.352 1147.51,697.084 1148.68,696.811 1149.85,696.535 1151.02,696.254 \n",
       "  1152.19,695.97 1153.36,695.681 1154.54,695.389 1155.71,695.093 1156.88,694.792 1158.05,694.488 1159.22,694.18 1160.39,693.868 1161.56,693.552 1162.74,693.232 \n",
       "  1163.91,692.908 1165.08,692.58 1166.25,692.248 1167.42,691.912 1168.59,691.572 1169.76,691.228 1170.94,690.881 1172.11,690.529 1173.28,690.173 1174.45,689.813 \n",
       "  1175.62,689.45 1176.79,689.082 1177.96,688.711 1179.14,688.335 1180.31,687.956 1181.48,687.572 1182.65,687.185 1183.82,686.794 1184.99,686.398 1186.17,685.999 \n",
       "  1187.34,685.596 1188.51,685.189 1189.68,684.778 1190.85,684.363 1192.02,683.944 1193.19,683.521 1194.37,683.094 1195.54,682.663 1196.71,682.228 1197.88,681.789 \n",
       "  1199.05,681.346 1200.22,680.899 1201.39,680.449 1202.57,679.994 1203.74,679.535 1204.91,679.073 1206.08,678.606 1207.25,678.136 1208.42,677.661 1209.59,677.183 \n",
       "  1210.77,676.7 1211.94,676.214 1213.11,675.724 1214.28,675.229 1215.45,674.731 1216.62,674.229 1217.79,673.723 1218.97,673.213 1220.14,672.698 1221.31,672.18 \n",
       "  1222.48,671.658 1223.65,671.132 1224.82,670.603 1225.99,670.069 1227.17,669.531 1228.34,668.989 1229.51,668.443 1230.68,667.894 1231.85,667.34 1233.02,666.782 \n",
       "  1234.2,666.221 1235.37,665.655 1236.54,665.086 1237.71,664.512 1238.88,663.935 1240.05,663.353 1241.22,662.768 1242.4,662.179 1243.57,661.585 1244.74,660.988 \n",
       "  1245.91,660.387 1247.08,659.782 1248.25,659.173 1249.42,658.56 1250.6,657.943 1251.77,657.322 1252.94,656.697 1254.11,656.068 1255.28,655.435 1256.45,654.798 \n",
       "  1257.62,654.157 1258.8,653.513 1259.97,652.864 1261.14,652.211 1262.31,651.554 1263.48,650.894 1264.65,650.229 1265.82,649.561 1267,648.888 1268.17,648.212 \n",
       "  1269.34,647.532 1270.51,646.847 1271.68,646.159 1272.85,645.467 1274.02,644.77 1275.2,644.07 1276.37,643.366 1277.54,642.658 1278.71,641.946 1279.88,641.23 \n",
       "  1281.05,640.51 1282.23,639.786 1283.4,639.058 1284.57,638.326 1285.74,637.59 1286.91,636.851 1288.08,636.107 1289.25,635.359 1290.43,634.608 1291.6,633.852 \n",
       "  1292.77,633.092 1293.94,632.329 1295.11,631.561 1296.28,630.79 1297.45,630.014 1298.63,629.235 1299.8,628.452 1300.97,627.664 1302.14,626.873 1303.31,626.078 \n",
       "  1304.48,625.279 1305.65,624.476 1306.83,623.669 1308,622.857 1309.17,622.042 1310.34,621.224 1311.51,620.401 1312.68,619.574 1313.85,618.743 1315.03,617.908 \n",
       "  1316.2,617.069 1317.37,616.227 1318.54,615.38 1319.71,614.529 1320.88,613.675 1322.06,612.816 1323.23,611.953 1324.4,611.087 1325.57,610.217 1326.74,609.342 \n",
       "  1327.91,608.464 1329.08,607.581 1330.26,606.695 1331.43,605.805 1332.6,604.911 1333.77,604.013 1334.94,603.11 1336.11,602.204 1337.28,601.294 1338.46,600.38 \n",
       "  1339.63,599.462 1340.8,598.54 1341.97,597.614 1343.14,596.685 1344.31,595.751 1345.48,594.813 1346.66,593.871 1347.83,592.926 1349,591.976 1350.17,591.022 \n",
       "  1351.34,590.065 1352.51,589.103 1353.68,588.138 1354.86,587.168 1356.03,586.195 1357.2,585.218 1358.37,584.236 1359.54,583.251 1360.71,582.262 1361.88,581.269 \n",
       "  1363.06,580.271 1364.23,579.27 1365.4,578.265 1366.57,577.256 1367.74,576.243 1368.91,575.226 1370.09,574.205 1371.26,573.18 1372.43,572.152 1373.6,571.119 \n",
       "  1374.77,570.082 1375.94,569.041 1377.11,567.997 1378.29,566.948 1379.46,565.895 1380.63,564.839 1381.8,563.778 1382.97,562.714 1384.14,561.645 1385.31,560.573 \n",
       "  1386.49,559.497 1387.66,558.416 1388.83,557.332 1390,556.244 1391.17,555.152 1392.34,554.056 1393.51,552.955 1394.69,551.851 1395.86,550.743 1397.03,549.631 \n",
       "  1398.2,548.515 1399.37,547.396 1400.54,546.272 1401.71,545.144 1402.89,544.012 1404.06,542.876 1405.23,541.737 1406.4,540.593 1407.57,539.445 1408.74,538.294 \n",
       "  1409.91,537.138 1411.09,535.979 1412.26,534.815 1413.43,533.648 1414.6,532.476 1415.77,531.301 1416.94,530.122 1418.12,528.938 1419.29,527.751 1420.46,526.56 \n",
       "  1421.63,525.365 1422.8,524.166 1423.97,522.963 1425.14,521.756 1426.32,520.545 1427.49,519.33 1428.66,518.111 1429.83,516.888 1431,515.661 1432.17,514.43 \n",
       "  1433.34,513.196 1434.52,511.957 1435.69,510.714 1436.86,509.468 1438.03,508.217 1439.2,506.963 1440.37,505.704 1441.54,504.442 1442.72,503.175 1443.89,501.905 \n",
       "  1445.06,500.631 1446.23,499.352 1447.4,498.07 1448.57,496.784 1449.74,495.494 1450.92,494.199 1452.09,492.901 1453.26,491.599 1454.43,490.293 1455.6,488.983 \n",
       "  1456.77,487.669 1457.95,486.352 1459.12,485.03 1460.29,483.704 1461.46,482.374 1462.63,481.04 1463.8,479.703 1464.97,478.361 1466.15,477.015 1467.32,475.666 \n",
       "  1468.49,474.312 1469.66,472.955 1470.83,471.593 1472,470.228 1473.17,468.859 1474.35,467.485 1475.52,466.108 1476.69,464.727 1477.86,463.342 1479.03,461.952 \n",
       "  1480.2,460.559 1481.37,459.162 1482.55,457.761 1483.72,456.356 1484.89,454.947 1486.06,453.534 1487.23,452.117 1488.4,450.697 1489.57,449.272 1490.75,447.843 \n",
       "  1491.92,446.41 1493.09,444.974 1494.26,443.533 1495.43,442.088 1496.6,440.64 1497.77,439.187 1498.95,437.731 1500.12,436.27 1501.29,434.806 1502.46,433.338 \n",
       "  1503.63,431.865 1504.8,430.389 1505.98,428.909 1507.15,427.425 1508.32,425.936 1509.49,424.444 1510.66,422.948 1511.83,421.448 1513,419.944 1514.18,418.436 \n",
       "  1515.35,416.924 1516.52,415.408 1517.69,413.889 1518.86,412.365 1520.03,410.837 1521.2,409.305 1522.38,407.77 1523.55,406.23 1524.72,404.686 1525.89,403.139 \n",
       "  1527.06,401.587 1528.23,400.032 1529.4,398.472 1530.58,396.909 1531.75,395.342 1532.92,393.77 1534.09,392.195 1535.26,390.616 1536.43,389.033 1537.6,387.446 \n",
       "  1538.78,385.854 1539.95,384.259 1541.12,382.66 1542.29,381.057 1543.46,379.45 1544.63,377.839 1545.81,376.225 1546.98,374.606 1548.15,372.983 1549.32,371.356 \n",
       "  1550.49,369.726 1551.66,368.091 1552.83,366.452 1554.01,364.81 1555.18,363.163 1556.35,361.513 1557.52,359.858 1558.69,358.2 1559.86,356.537 1561.03,354.871 \n",
       "  1562.21,353.201 1563.38,351.526 1564.55,349.848 1565.72,348.166 1566.89,346.48 1568.06,344.79 1569.23,343.096 1570.41,341.398 1571.58,339.696 1572.75,337.99 \n",
       "  1573.92,336.28 1575.09,334.566 1576.26,332.848 1577.43,331.126 1578.61,329.401 1579.78,327.671 1580.95,325.937 1582.12,324.2 1583.29,322.458 1584.46,320.713 \n",
       "  1585.63,318.963 1586.81,317.21 1587.98,315.452 1589.15,313.691 1590.32,311.926 1591.49,310.156 1592.66,308.383 1593.84,306.606 1595.01,304.825 1596.18,303.04 \n",
       "  1597.35,301.25 1598.52,299.457 1599.69,297.66 1600.86,295.859 1602.04,294.054 1603.21,292.246 1604.38,290.433 1605.55,288.616 1606.72,286.795 1607.89,284.97 \n",
       "  1609.06,283.142 1610.24,281.309 1611.41,279.472 1612.58,277.632 1613.75,275.787 1614.92,273.939 1616.09,272.086 1617.26,270.23 1618.44,268.37 1619.61,266.505 \n",
       "  1620.78,264.637 1621.95,262.765 1623.12,260.889 1624.29,259.008 1625.46,257.124 1626.64,255.236 1627.81,253.344 1628.98,251.448 1630.15,249.548 1631.32,247.644 \n",
       "  1632.49,245.736 1633.66,243.825 1634.84,241.909 1636.01,239.989 1637.18,238.065 1638.35,236.138 1639.52,234.206 1640.69,232.27 1641.87,230.331 1643.04,228.387 \n",
       "  1644.21,226.44 1645.38,224.488 1646.55,222.533 1647.72,220.574 1648.89,218.61 1650.07,216.643 1651.24,214.672 1652.41,212.697 1653.58,210.717 1654.75,208.734 \n",
       "  1655.92,206.747 1657.09,204.756 1658.27,202.761 1659.44,200.762 1660.61,198.759 1661.78,196.753 1662.95,194.742 1664.12,192.727 1665.29,190.708 1666.47,188.685 \n",
       "  \n",
       "  \"/>\n",
       "</svg>\n"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pf = initializePolynomialFeatures(X,degree=2)\n",
    "Z = pf.transform(X)\n",
    "\n",
    "v = Z'*Z \\ Z'*y\n",
    "\n",
    "plot(X,y,\n",
    "    seriestype=:scatter,\n",
    "    markersize=20,\n",
    "    thickness_scaling=5,\n",
    "    size=(1800,1500),\n",
    "    legend=false\n",
    ")\n",
    "grid_poly = pf.transform(grid)\n",
    "plot!(grid, grid_poly*v,\n",
    "    linewidth=5\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[0.28492; 0.0790719; 3.14544]\n",
      "3"
     ]
    }
   ],
   "source": [
    "Printf.@printf(\"%s\\n\",v)\n",
    "Printf.@printf(\"%d\",maximum(broadcast(abs,v)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Discussion:\n",
    "- The lower degree polynomial looks good. \n",
    "- But if the true relationship really was complicated? Then if we restricted the degree of the polynomial, we'd miss out on it. \n",
    "\n",
    "### Solution 2: add L2-regularization\n",
    "\n",
    "- Let's let the data decide how complex our model needs to be. \n",
    "- This is just like the complexity penalty from before, but now we can still optimize the loss easily."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"utf-8\"?>\n",
       "<svg xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\" width=\"450\" height=\"375\" viewBox=\"0 0 1800 1500\">\n",
       "<defs>\n",
       "  <clipPath id=\"clip5500\">\n",
       "    <rect x=\"0\" y=\"0\" width=\"1800\" height=\"1500\"/>\n",
       "  </clipPath>\n",
       "</defs>\n",
       "<polygon clip-path=\"url(#clip5500)\" points=\"\n",
       "0,1500 1800,1500 1800,0 0,0 \n",
       "  \" fill=\"#ffffff\" fill-rule=\"evenodd\" fill-opacity=\"1\"/>\n",
       "<defs>\n",
       "  <clipPath id=\"clip5501\">\n",
       "    <rect x=\"360\" y=\"0\" width=\"1261\" height=\"1261\"/>\n",
       "  </clipPath>\n",
       "</defs>\n",
       "<polygon clip-path=\"url(#clip5500)\" points=\"\n",
       "461.063,1159.28 1701.57,1159.28 1701.57,59.0551 461.063,59.0551 \n",
       "  \" fill=\"#ffffff\" fill-rule=\"evenodd\" fill-opacity=\"1\"/>\n",
       "<defs>\n",
       "  <clipPath id=\"clip5502\">\n",
       "    <rect x=\"461\" y=\"59\" width=\"1242\" height=\"1101\"/>\n",
       "  </clipPath>\n",
       "</defs>\n",
       "<polyline clip-path=\"url(#clip5502)\" style=\"stroke:#000000; stroke-width:2.5; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  485.854,1159.28 485.854,59.0551 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip5502)\" style=\"stroke:#000000; stroke-width:2.5; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  780.36,1159.28 780.36,59.0551 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip5502)\" style=\"stroke:#000000; stroke-width:2.5; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  1074.87,1159.28 1074.87,59.0551 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip5502)\" style=\"stroke:#000000; stroke-width:2.5; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  1369.37,1159.28 1369.37,59.0551 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip5502)\" style=\"stroke:#000000; stroke-width:2.5; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  1663.88,1159.28 1663.88,59.0551 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip5502)\" style=\"stroke:#000000; stroke-width:2.5; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  461.063,1069.83 1701.57,1069.83 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip5502)\" style=\"stroke:#000000; stroke-width:2.5; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  461.063,910.709 1701.57,910.709 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip5502)\" style=\"stroke:#000000; stroke-width:2.5; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  461.063,751.586 1701.57,751.586 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip5502)\" style=\"stroke:#000000; stroke-width:2.5; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  461.063,592.462 1701.57,592.462 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip5502)\" style=\"stroke:#000000; stroke-width:2.5; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  461.063,433.338 1701.57,433.338 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip5502)\" style=\"stroke:#000000; stroke-width:2.5; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  461.063,274.215 1701.57,274.215 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip5502)\" style=\"stroke:#000000; stroke-width:2.5; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  461.063,115.091 1701.57,115.091 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip5500)\" style=\"stroke:#000000; stroke-width:5; stroke-opacity:1; fill:none\" points=\"\n",
       "  461.063,1159.28 1701.57,1159.28 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip5500)\" style=\"stroke:#000000; stroke-width:5; stroke-opacity:1; fill:none\" points=\"\n",
       "  461.063,1159.28 461.063,59.0551 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip5500)\" style=\"stroke:#000000; stroke-width:5; stroke-opacity:1; fill:none\" points=\"\n",
       "  485.854,1159.28 485.854,1142.78 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip5500)\" style=\"stroke:#000000; stroke-width:5; stroke-opacity:1; fill:none\" points=\"\n",
       "  780.36,1159.28 780.36,1142.78 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip5500)\" style=\"stroke:#000000; stroke-width:5; stroke-opacity:1; fill:none\" points=\"\n",
       "  1074.87,1159.28 1074.87,1142.78 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip5500)\" style=\"stroke:#000000; stroke-width:5; stroke-opacity:1; fill:none\" points=\"\n",
       "  1369.37,1159.28 1369.37,1142.78 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip5500)\" style=\"stroke:#000000; stroke-width:5; stroke-opacity:1; fill:none\" points=\"\n",
       "  1663.88,1159.28 1663.88,1142.78 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip5500)\" style=\"stroke:#000000; stroke-width:5; stroke-opacity:1; fill:none\" points=\"\n",
       "  461.063,1069.83 479.671,1069.83 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip5500)\" style=\"stroke:#000000; stroke-width:5; stroke-opacity:1; fill:none\" points=\"\n",
       "  461.063,910.709 479.671,910.709 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip5500)\" style=\"stroke:#000000; stroke-width:5; stroke-opacity:1; fill:none\" points=\"\n",
       "  461.063,751.586 479.671,751.586 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip5500)\" style=\"stroke:#000000; stroke-width:5; stroke-opacity:1; fill:none\" points=\"\n",
       "  461.063,592.462 479.671,592.462 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip5500)\" style=\"stroke:#000000; stroke-width:5; stroke-opacity:1; fill:none\" points=\"\n",
       "  461.063,433.338 479.671,433.338 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip5500)\" style=\"stroke:#000000; stroke-width:5; stroke-opacity:1; fill:none\" points=\"\n",
       "  461.063,274.215 479.671,274.215 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip5500)\" style=\"stroke:#000000; stroke-width:5; stroke-opacity:1; fill:none\" points=\"\n",
       "  461.063,115.091 479.671,115.091 \n",
       "  \"/>\n",
       "<g clip-path=\"url(#clip5500)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:59px; text-anchor:middle;\" transform=\"rotate(0, 485.854, 1219.88)\" x=\"485.854\" y=\"1219.88\">-1.0</text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip5500)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:59px; text-anchor:middle;\" transform=\"rotate(0, 780.36, 1219.88)\" x=\"780.36\" y=\"1219.88\">-0.5</text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip5500)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:59px; text-anchor:middle;\" transform=\"rotate(0, 1074.87, 1219.88)\" x=\"1074.87\" y=\"1219.88\">0.0</text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip5500)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:59px; text-anchor:middle;\" transform=\"rotate(0, 1369.37, 1219.88)\" x=\"1369.37\" y=\"1219.88\">0.5</text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip5500)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:59px; text-anchor:middle;\" transform=\"rotate(0, 1663.88, 1219.88)\" x=\"1663.88\" y=\"1219.88\">1.0</text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip5500)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:59px; text-anchor:end;\" transform=\"rotate(0, 443.063, 1091.33)\" x=\"443.063\" y=\"1091.33\">-2</text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip5500)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:59px; text-anchor:end;\" transform=\"rotate(0, 443.063, 932.209)\" x=\"443.063\" y=\"932.209\">-1</text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip5500)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:59px; text-anchor:end;\" transform=\"rotate(0, 443.063, 773.086)\" x=\"443.063\" y=\"773.086\">0</text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip5500)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:59px; text-anchor:end;\" transform=\"rotate(0, 443.063, 613.962)\" x=\"443.063\" y=\"613.962\">1</text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip5500)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:59px; text-anchor:end;\" transform=\"rotate(0, 443.063, 454.838)\" x=\"443.063\" y=\"454.838\">2</text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip5500)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:59px; text-anchor:end;\" transform=\"rotate(0, 443.063, 295.715)\" x=\"443.063\" y=\"295.715\">3</text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip5500)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:59px; text-anchor:end;\" transform=\"rotate(0, 443.063, 136.591)\" x=\"443.063\" y=\"136.591\">4</text>\n",
       "</g>\n",
       "<circle clip-path=\"url(#clip5502)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"496.172\" cy=\"375.74\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip5502)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"496.172\" cy=\"375.74\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip5502)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"543.688\" cy=\"268.046\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip5502)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"543.688\" cy=\"268.046\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip5502)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"609.089\" cy=\"188.707\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip5502)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"609.089\" cy=\"188.707\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip5502)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"667.886\" cy=\"277.503\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip5502)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"667.886\" cy=\"277.503\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip5502)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"729.09\" cy=\"785.85\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip5502)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"729.09\" cy=\"785.85\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip5502)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"789.029\" cy=\"483.973\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip5502)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"789.029\" cy=\"483.973\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip5502)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"861.041\" cy=\"649.55\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip5502)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"861.041\" cy=\"649.55\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip5502)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"918.754\" cy=\"1128.14\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip5502)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"918.754\" cy=\"1128.14\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip5502)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"984.908\" cy=\"827.938\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip5502)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"984.908\" cy=\"827.938\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip5502)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1046.7\" cy=\"754.897\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip5502)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1046.7\" cy=\"754.897\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip5502)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1104.56\" cy=\"305.347\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip5502)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1104.56\" cy=\"305.347\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip5502)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1165.05\" cy=\"520.643\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip5502)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1165.05\" cy=\"520.643\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip5502)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1228.63\" cy=\"471.783\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip5502)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1228.63\" cy=\"471.783\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip5502)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1289.13\" cy=\"879.196\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip5502)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1289.13\" cy=\"879.196\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip5502)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1352.16\" cy=\"601.878\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip5502)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1352.16\" cy=\"601.878\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip5502)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1404.86\" cy=\"639.945\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip5502)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1404.86\" cy=\"639.945\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip5502)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1477.59\" cy=\"425.222\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip5502)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1477.59\" cy=\"425.222\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip5502)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1535.21\" cy=\"237.612\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip5502)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1535.21\" cy=\"237.612\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip5502)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1602.94\" cy=\"538.58\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip5502)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1602.94\" cy=\"538.58\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip5502)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1666.47\" cy=\"90.1936\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip5502)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1666.47\" cy=\"90.1936\" r=\"18\"/>\n",
       "<polyline clip-path=\"url(#clip5502)\" style=\"stroke:#e26f46; stroke-width:5; stroke-opacity:1; fill:none\" points=\"\n",
       "  496.172,315.535 497.343,313.879 498.515,312.365 499.686,310.987 500.858,309.741 502.029,308.623 503.201,307.627 504.372,306.751 505.543,305.99 506.715,305.34 \n",
       "  507.886,304.797 509.058,304.358 510.229,304.018 511.401,303.774 512.572,303.624 513.744,303.563 514.915,303.588 516.087,303.697 517.258,303.886 518.43,304.152 \n",
       "  519.601,304.492 520.772,304.904 521.944,305.385 523.115,305.932 524.287,306.544 525.458,307.216 526.63,307.948 527.801,308.736 528.973,309.579 530.144,310.474 \n",
       "  531.316,311.419 532.487,312.412 533.659,313.452 534.83,314.536 536.002,315.662 537.173,316.829 538.344,318.034 539.516,319.277 540.687,320.556 541.859,321.868 \n",
       "  543.03,323.213 544.202,324.589 545.373,325.994 546.545,327.427 547.716,328.887 548.888,330.373 550.059,331.883 551.231,333.415 552.402,334.97 553.574,336.545 \n",
       "  554.745,338.139 555.916,339.752 557.088,341.383 558.259,343.03 559.431,344.692 560.602,346.369 561.774,348.059 562.945,349.762 564.117,351.477 565.288,353.203 \n",
       "  566.46,354.939 567.631,356.684 568.803,358.439 569.974,360.201 571.146,361.971 572.317,363.747 573.488,365.529 574.66,367.317 575.831,369.109 577.003,370.906 \n",
       "  578.174,372.706 579.346,374.509 580.517,376.315 581.689,378.123 582.86,379.933 584.032,381.743 585.203,383.555 586.375,385.366 587.546,387.178 588.718,388.989 \n",
       "  589.889,390.799 591.06,392.607 592.232,394.414 593.403,396.219 594.575,398.022 595.746,399.822 596.918,401.619 598.089,403.412 599.261,405.202 600.432,406.989 \n",
       "  601.604,408.771 602.775,410.549 603.947,412.322 605.118,414.091 606.289,415.854 607.461,417.612 608.632,419.365 609.804,421.112 610.975,422.854 612.147,424.589 \n",
       "  613.318,426.319 614.49,428.042 615.661,429.758 616.833,431.468 618.004,433.172 619.176,434.868 620.347,436.557 621.519,438.24 622.69,439.915 623.861,441.583 \n",
       "  625.033,443.243 626.204,444.896 627.376,446.541 628.547,448.178 629.719,449.808 630.89,451.43 632.062,453.044 633.233,454.65 634.405,456.248 635.576,457.838 \n",
       "  636.748,459.419 637.919,460.993 639.091,462.558 640.262,464.115 641.433,465.663 642.605,467.203 643.776,468.735 644.948,470.259 646.119,471.773 647.291,473.28 \n",
       "  648.462,474.778 649.634,476.267 650.805,477.748 651.977,479.22 653.148,480.684 654.32,482.139 655.491,483.586 656.663,485.024 657.834,486.453 659.005,487.874 \n",
       "  660.177,489.286 661.348,490.69 662.52,492.085 663.691,493.471 664.863,494.849 666.034,496.219 667.206,497.58 668.377,498.932 669.549,500.276 670.72,501.611 \n",
       "  671.892,502.938 673.063,504.257 674.235,505.567 675.406,506.868 676.577,508.162 677.749,509.447 678.92,510.723 680.092,511.991 681.263,513.251 682.435,514.503 \n",
       "  683.606,515.747 684.778,516.982 685.949,518.209 687.121,519.428 688.292,520.639 689.464,521.841 690.635,523.036 691.806,524.223 692.978,525.401 694.149,526.572 \n",
       "  695.321,527.735 696.492,528.889 697.664,530.036 698.835,531.176 700.007,532.307 701.178,533.431 702.35,534.546 703.521,535.655 704.693,536.755 705.864,537.848 \n",
       "  707.036,538.934 708.207,540.011 709.378,541.082 710.55,542.145 711.721,543.2 712.893,544.248 714.064,545.289 715.236,546.322 716.407,547.348 717.579,548.367 \n",
       "  718.75,549.379 719.922,550.384 721.093,551.381 722.265,552.371 723.436,553.355 724.608,554.331 725.779,555.3 726.95,556.263 728.122,557.218 729.293,558.167 \n",
       "  730.465,559.108 731.636,560.043 732.808,560.972 733.979,561.893 735.151,562.808 736.322,563.716 737.494,564.618 738.665,565.513 739.837,566.402 741.008,567.284 \n",
       "  742.18,568.16 743.351,569.029 744.522,569.892 745.694,570.749 746.865,571.599 748.037,572.443 749.208,573.281 750.38,574.113 751.551,574.938 752.723,575.758 \n",
       "  753.894,576.571 755.066,577.378 756.237,578.18 757.409,578.975 758.58,579.764 759.751,580.548 760.923,581.326 762.094,582.097 763.266,582.863 764.437,583.624 \n",
       "  765.609,584.378 766.78,585.127 767.952,585.87 769.123,586.608 770.295,587.34 771.466,588.067 772.638,588.788 773.809,589.503 774.981,590.213 776.152,590.918 \n",
       "  777.323,591.617 778.495,592.311 779.666,592.999 780.838,593.683 782.009,594.361 783.181,595.034 784.352,595.701 785.524,596.364 786.695,597.021 787.867,597.673 \n",
       "  789.038,598.32 790.21,598.962 791.381,599.599 792.553,600.231 793.724,600.859 794.895,601.481 796.067,602.098 797.238,602.711 798.41,603.318 799.581,603.921 \n",
       "  800.753,604.519 801.924,605.113 803.096,605.701 804.267,606.285 805.439,606.864 806.61,607.439 807.782,608.009 808.953,608.575 810.125,609.135 811.296,609.692 \n",
       "  812.467,610.244 813.639,610.791 814.81,611.334 815.982,611.873 817.153,612.407 818.325,612.937 819.496,613.462 820.668,613.983 821.839,614.5 823.011,615.012 \n",
       "  824.182,615.521 825.354,616.025 826.525,616.525 827.697,617.021 828.868,617.512 830.039,618 831.211,618.483 832.382,618.963 833.554,619.438 834.725,619.909 \n",
       "  835.897,620.376 837.068,620.84 838.24,621.299 839.411,621.754 840.583,622.206 841.754,622.654 842.926,623.097 844.097,623.537 845.268,623.973 846.44,624.406 \n",
       "  847.611,624.834 848.783,625.259 849.954,625.68 851.126,626.097 852.297,626.511 853.469,626.921 854.64,627.327 855.812,627.73 856.983,628.129 858.155,628.524 \n",
       "  859.326,628.916 860.498,629.305 861.669,629.689 862.84,630.071 864.012,630.449 865.183,630.823 866.355,631.194 867.526,631.561 868.698,631.925 869.869,632.286 \n",
       "  871.041,632.643 872.212,632.997 873.384,633.348 874.555,633.695 875.727,634.039 876.898,634.38 878.07,634.717 879.241,635.051 880.412,635.382 881.584,635.71 \n",
       "  882.755,636.034 883.927,636.355 885.098,636.674 886.27,636.989 887.441,637.3 888.613,637.609 889.784,637.915 890.956,638.217 892.127,638.517 893.299,638.813 \n",
       "  894.47,639.106 895.642,639.397 896.813,639.684 897.984,639.969 899.156,640.25 900.327,640.528 901.499,640.804 902.67,641.076 903.842,641.346 905.013,641.613 \n",
       "  906.185,641.877 907.356,642.138 908.528,642.396 909.699,642.651 910.871,642.903 912.042,643.153 913.214,643.4 914.385,643.644 915.556,643.885 916.728,644.124 \n",
       "  917.899,644.359 919.071,644.592 920.242,644.823 921.414,645.05 922.585,645.275 923.757,645.497 924.928,645.717 926.1,645.934 927.271,646.148 928.443,646.36 \n",
       "  929.614,646.569 930.785,646.775 931.957,646.979 933.128,647.18 934.3,647.379 935.471,647.575 936.643,647.769 937.814,647.96 938.986,648.148 940.157,648.334 \n",
       "  941.329,648.518 942.5,648.699 943.672,648.877 944.843,649.053 946.015,649.227 947.186,649.398 948.357,649.567 949.529,649.733 950.7,649.897 951.872,650.059 \n",
       "  953.043,650.218 954.215,650.375 955.386,650.529 956.558,650.681 957.729,650.831 958.901,650.978 960.072,651.123 961.244,651.266 962.415,651.406 963.587,651.544 \n",
       "  964.758,651.68 965.929,651.814 967.101,651.945 968.272,652.074 969.444,652.201 970.615,652.325 971.787,652.447 972.958,652.567 974.13,652.685 975.301,652.801 \n",
       "  976.473,652.914 977.644,653.026 978.816,653.135 979.987,653.242 981.159,653.346 982.33,653.449 983.501,653.549 984.673,653.648 985.844,653.744 987.016,653.838 \n",
       "  988.187,653.93 989.359,654.02 990.53,654.107 991.702,654.193 992.873,654.277 994.045,654.358 995.216,654.438 996.388,654.515 997.559,654.59 998.731,654.664 \n",
       "  999.902,654.735 1001.07,654.804 1002.24,654.871 1003.42,654.937 1004.59,655 1005.76,655.061 1006.93,655.12 1008.1,655.177 1009.27,655.233 1010.45,655.286 \n",
       "  1011.62,655.337 1012.79,655.387 1013.96,655.434 1015.13,655.48 1016.3,655.523 1017.47,655.565 1018.65,655.604 1019.82,655.642 1020.99,655.678 1022.16,655.712 \n",
       "  1023.33,655.744 1024.5,655.774 1025.67,655.802 1026.85,655.829 1028.02,655.853 1029.19,655.876 1030.36,655.896 1031.53,655.915 1032.7,655.932 1033.87,655.947 \n",
       "  1035.05,655.961 1036.22,655.972 1037.39,655.982 1038.56,655.99 1039.73,655.996 1040.9,656 1042.07,656.002 1043.25,656.003 1044.42,656.002 1045.59,655.999 \n",
       "  1046.76,655.994 1047.93,655.987 1049.1,655.979 1050.28,655.969 1051.45,655.957 1052.62,655.943 1053.79,655.928 1054.96,655.911 1056.13,655.892 1057.3,655.871 \n",
       "  1058.48,655.849 1059.65,655.824 1060.82,655.798 1061.99,655.771 1063.16,655.741 1064.33,655.71 1065.5,655.677 1066.68,655.643 1067.85,655.607 1069.02,655.569 \n",
       "  1070.19,655.529 1071.36,655.487 1072.53,655.444 1073.7,655.4 1074.88,655.353 1076.05,655.305 1077.22,655.255 1078.39,655.204 1079.56,655.15 1080.73,655.096 \n",
       "  1081.9,655.039 1083.08,654.981 1084.25,654.921 1085.42,654.859 1086.59,654.796 1087.76,654.731 1088.93,654.665 1090.1,654.597 1091.28,654.527 1092.45,654.455 \n",
       "  1093.62,654.382 1094.79,654.308 1095.96,654.231 1097.13,654.153 1098.31,654.074 1099.48,653.992 1100.65,653.91 1101.82,653.825 1102.99,653.739 1104.16,653.651 \n",
       "  1105.33,653.562 1106.51,653.471 1107.68,653.378 1108.85,653.284 1110.02,653.189 1111.19,653.091 1112.36,652.992 1113.53,652.892 1114.71,652.789 1115.88,652.686 \n",
       "  1117.05,652.58 1118.22,652.473 1119.39,652.365 1120.56,652.255 1121.73,652.143 1122.91,652.03 1124.08,651.915 1125.25,651.798 1126.42,651.68 1127.59,651.561 \n",
       "  1128.76,651.439 1129.93,651.317 1131.11,651.192 1132.28,651.066 1133.45,650.939 1134.62,650.81 1135.79,650.679 1136.96,650.547 1138.13,650.413 1139.31,650.277 \n",
       "  1140.48,650.14 1141.65,650.002 1142.82,649.862 1143.99,649.72 1145.16,649.577 1146.34,649.432 1147.51,649.286 1148.68,649.138 1149.85,648.988 1151.02,648.837 \n",
       "  1152.19,648.685 1153.36,648.531 1154.54,648.375 1155.71,648.218 1156.88,648.059 1158.05,647.898 1159.22,647.736 1160.39,647.573 1161.56,647.408 1162.74,647.241 \n",
       "  1163.91,647.073 1165.08,646.903 1166.25,646.732 1167.42,646.559 1168.59,646.384 1169.76,646.208 1170.94,646.031 1172.11,645.852 1173.28,645.671 1174.45,645.489 \n",
       "  1175.62,645.305 1176.79,645.119 1177.96,644.932 1179.14,644.744 1180.31,644.554 1181.48,644.362 1182.65,644.169 1183.82,643.974 1184.99,643.778 1186.17,643.58 \n",
       "  1187.34,643.38 1188.51,643.179 1189.68,642.976 1190.85,642.772 1192.02,642.566 1193.19,642.359 1194.37,642.15 1195.54,641.939 1196.71,641.727 1197.88,641.513 \n",
       "  1199.05,641.298 1200.22,641.081 1201.39,640.862 1202.57,640.642 1203.74,640.42 1204.91,640.197 1206.08,639.972 1207.25,639.746 1208.42,639.518 1209.59,639.288 \n",
       "  1210.77,639.057 1211.94,638.824 1213.11,638.589 1214.28,638.353 1215.45,638.115 1216.62,637.876 1217.79,637.635 1218.97,637.392 1220.14,637.148 1221.31,636.902 \n",
       "  1222.48,636.655 1223.65,636.406 1224.82,636.155 1225.99,635.902 1227.17,635.648 1228.34,635.393 1229.51,635.135 1230.68,634.876 1231.85,634.616 1233.02,634.353 \n",
       "  1234.2,634.089 1235.37,633.824 1236.54,633.557 1237.71,633.288 1238.88,633.017 1240.05,632.745 1241.22,632.471 1242.4,632.195 1243.57,631.918 1244.74,631.639 \n",
       "  1245.91,631.358 1247.08,631.075 1248.25,630.791 1249.42,630.505 1250.6,630.218 1251.77,629.929 1252.94,629.638 1254.11,629.345 1255.28,629.051 1256.45,628.754 \n",
       "  1257.62,628.456 1258.8,628.157 1259.97,627.855 1261.14,627.552 1262.31,627.247 1263.48,626.941 1264.65,626.632 1265.82,626.322 1267,626.01 1268.17,625.696 \n",
       "  1269.34,625.381 1270.51,625.064 1271.68,624.744 1272.85,624.424 1274.02,624.101 1275.2,623.776 1276.37,623.45 1277.54,623.122 1278.71,622.792 1279.88,622.46 \n",
       "  1281.05,622.126 1282.23,621.791 1283.4,621.454 1284.57,621.114 1285.74,620.773 1286.91,620.43 1288.08,620.085 1289.25,619.739 1290.43,619.39 1291.6,619.04 \n",
       "  1292.77,618.687 1293.94,618.333 1295.11,617.977 1296.28,617.619 1297.45,617.259 1298.63,616.897 1299.8,616.533 1300.97,616.167 1302.14,615.799 1303.31,615.429 \n",
       "  1304.48,615.057 1305.65,614.683 1306.83,614.308 1308,613.93 1309.17,613.55 1310.34,613.168 1311.51,612.785 1312.68,612.399 1313.85,612.011 1315.03,611.621 \n",
       "  1316.2,611.229 1317.37,610.835 1318.54,610.439 1319.71,610.041 1320.88,609.641 1322.06,609.238 1323.23,608.834 1324.4,608.427 1325.57,608.019 1326.74,607.608 \n",
       "  1327.91,607.195 1329.08,606.78 1330.26,606.362 1331.43,605.943 1332.6,605.521 1333.77,605.097 1334.94,604.671 1336.11,604.243 1337.28,603.813 1338.46,603.38 \n",
       "  1339.63,602.945 1340.8,602.508 1341.97,602.068 1343.14,601.627 1344.31,601.183 1345.48,600.736 1346.66,600.288 1347.83,599.837 1349,599.383 1350.17,598.928 \n",
       "  1351.34,598.47 1352.51,598.009 1353.68,597.546 1354.86,597.081 1356.03,596.614 1357.2,596.144 1358.37,595.671 1359.54,595.196 1360.71,594.719 1361.88,594.239 \n",
       "  1363.06,593.757 1364.23,593.272 1365.4,592.785 1366.57,592.295 1367.74,591.803 1368.91,591.308 1370.09,590.81 1371.26,590.31 1372.43,589.808 1373.6,589.302 \n",
       "  1374.77,588.794 1375.94,588.284 1377.11,587.771 1378.29,587.255 1379.46,586.736 1380.63,586.215 1381.8,585.691 1382.97,585.164 1384.14,584.634 1385.31,584.102 \n",
       "  1386.49,583.567 1387.66,583.029 1388.83,582.488 1390,581.944 1391.17,581.398 1392.34,580.848 1393.51,580.296 1394.69,579.741 1395.86,579.183 1397.03,578.622 \n",
       "  1398.2,578.058 1399.37,577.49 1400.54,576.92 1401.71,576.347 1402.89,575.771 1404.06,575.192 1405.23,574.609 1406.4,574.023 1407.57,573.435 1408.74,572.843 \n",
       "  1409.91,572.248 1411.09,571.649 1412.26,571.048 1413.43,570.443 1414.6,569.835 1415.77,569.223 1416.94,568.608 1418.12,567.99 1419.29,567.369 1420.46,566.744 \n",
       "  1421.63,566.115 1422.8,565.483 1423.97,564.848 1425.14,564.209 1426.32,563.566 1427.49,562.92 1428.66,562.271 1429.83,561.617 1431,560.96 1432.17,560.299 \n",
       "  1433.34,559.635 1434.52,558.966 1435.69,558.294 1436.86,557.618 1438.03,556.939 1439.2,556.255 1440.37,555.567 1441.54,554.876 1442.72,554.18 1443.89,553.48 \n",
       "  1445.06,552.776 1446.23,552.069 1447.4,551.356 1448.57,550.64 1449.74,549.92 1450.92,549.195 1452.09,548.466 1453.26,547.732 1454.43,546.994 1455.6,546.252 \n",
       "  1456.77,545.505 1457.95,544.754 1459.12,543.998 1460.29,543.237 1461.46,542.472 1462.63,541.701 1463.8,540.927 1464.97,540.147 1466.15,539.362 1467.32,538.573 \n",
       "  1468.49,537.778 1469.66,536.978 1470.83,536.174 1472,535.364 1473.17,534.549 1474.35,533.728 1475.52,532.902 1476.69,532.071 1477.86,531.234 1479.03,530.392 \n",
       "  1480.2,529.544 1481.37,528.691 1482.55,527.831 1483.72,526.966 1484.89,526.095 1486.06,525.218 1487.23,524.335 1488.4,523.446 1489.57,522.55 1490.75,521.649 \n",
       "  1491.92,520.74 1493.09,519.826 1494.26,518.905 1495.43,517.977 1496.6,517.042 1497.77,516.101 1498.95,515.152 1500.12,514.197 1501.29,513.234 1502.46,512.265 \n",
       "  1503.63,511.287 1504.8,510.303 1505.98,509.311 1507.15,508.311 1508.32,507.303 1509.49,506.287 1510.66,505.264 1511.83,504.232 1513,503.192 1514.18,502.143 \n",
       "  1515.35,501.086 1516.52,500.02 1517.69,498.945 1518.86,497.861 1520.03,496.768 1521.2,495.666 1522.38,494.554 1523.55,493.432 1524.72,492.301 1525.89,491.16 \n",
       "  1527.06,490.008 1528.23,488.846 1529.4,487.674 1530.58,486.491 1531.75,485.297 1532.92,484.091 1534.09,482.875 1535.26,481.647 1536.43,480.407 1537.6,479.155 \n",
       "  1538.78,477.891 1539.95,476.614 1541.12,475.325 1542.29,474.022 1543.46,472.707 1544.63,471.378 1545.81,470.035 1546.98,468.678 1548.15,467.306 1549.32,465.921 \n",
       "  1550.49,464.52 1551.66,463.104 1552.83,461.672 1554.01,460.224 1555.18,458.76 1556.35,457.28 1557.52,455.782 1558.69,454.268 1559.86,452.735 1561.03,451.184 \n",
       "  1562.21,449.615 1563.38,448.027 1564.55,446.42 1565.72,444.793 1566.89,443.145 1568.06,441.477 1569.23,439.788 1570.41,438.077 1571.58,436.344 1572.75,434.588 \n",
       "  1573.92,432.809 1575.09,431.006 1576.26,429.179 1577.43,427.327 1578.61,425.449 1579.78,423.546 1580.95,421.615 1582.12,419.657 1583.29,417.671 1584.46,415.656 \n",
       "  1585.63,413.612 1586.81,411.538 1587.98,409.432 1589.15,407.295 1590.32,405.126 1591.49,402.923 1592.66,400.685 1593.84,398.413 1595.01,396.105 1596.18,393.76 \n",
       "  1597.35,391.377 1598.52,388.956 1599.69,386.494 1600.86,383.992 1602.04,381.448 1603.21,378.862 1604.38,376.231 1605.55,373.555 1606.72,370.832 1607.89,368.062 \n",
       "  1609.06,365.243 1610.24,362.374 1611.41,359.453 1612.58,356.48 1613.75,353.452 1614.92,350.369 1616.09,347.228 1617.26,344.029 1618.44,340.77 1619.61,337.448 \n",
       "  1620.78,334.063 1621.95,330.613 1623.12,327.095 1624.29,323.509 1625.46,319.852 1626.64,316.123 1627.81,312.318 1628.98,308.437 1630.15,304.478 1631.32,300.437 \n",
       "  1632.49,296.313 1633.66,292.104 1634.84,287.807 1636.01,283.42 1637.18,278.94 1638.35,274.365 1639.52,269.692 1640.69,264.918 1641.87,260.042 1643.04,255.059 \n",
       "  1644.21,249.967 1645.38,244.762 1646.55,239.443 1647.72,234.005 1648.89,228.445 1650.07,222.76 1651.24,216.947 1652.41,211.001 1653.58,204.92 1654.75,198.699 \n",
       "  1655.92,192.334 1657.09,185.822 1658.27,179.158 1659.44,172.338 1660.61,165.358 1661.78,158.213 1662.95,150.898 1664.12,143.41 1665.29,135.743 1666.47,127.891 \n",
       "  \n",
       "  \"/>\n",
       "</svg>\n"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pf = initializePolynomialFeatures(X,degree=19)\n",
    "Z = pf.transform(X)\n",
    "\n",
    "λ = 1\n",
    "v = (Z'*Z + λ*eye(size(Z)[2])) \\ Z'*y\n",
    "\n",
    "plot(X,y,\n",
    "    seriestype=:scatter,\n",
    "    markersize=20,\n",
    "    thickness_scaling=5,\n",
    "    size=(1800,1500),\n",
    "    legend=false\n",
    ")\n",
    "grid_poly = pf.transform(grid)\n",
    "plot!(grid, grid_poly*v)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(20, 20)"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "size(Z)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[0.604763; 0.149463; 1.34012; -0.468081; 0.865257; -0.268357; 0.430103; -0.101371; 0.170485; 0.0277239; 0.0310185; 0.128077; -0.0369899; 0.206066; -0.0643567; 0.266599; -0.0693268; 0.313502; -0.0624236; 0.349736]\n",
      "1"
     ]
    }
   ],
   "source": [
    "Printf.@printf(\"%s\\n\",v)\n",
    "Printf.@printf(\"%d\",maximum(broadcast(abs,v)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Discussion:\n",
    "- Regularization helped! Even though we used a degree $N-1$ polynomial, we didn't end up with a crazy model.\n",
    "- We can add regularization to many models, not just least squares with a polynomial basis."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### More regularization thoughts (to read after class):\n",
    "\n",
    "Imagine minimizing $f(w)=(w-1)^2+\\lambda w^2$ with respect to $w$. The left-hand term wants to set $w=1$ to be minimized (equal to zero, in this case, since something squared can't be negative). The right-hand term wants to set $w=0$ to be minimized (again, equal to zero). What if we set $w=0$? Then we get a penalty of $1$ from the left-hand term and a penalty of zero from the right-hand term, for a total of $f(0)=1$. Now what if we set $w=1$? Then the left-hand term gives a penalty of zero and the right-hand term equals $\\lambda$, so $f(1)=\\lambda$. Then if $\\lambda=1$ these two solutions are equally desirable, but if $\\lambda=100$ then it becomes much better to set $w=0$ than $w=1$ and on the other hand if $\\lambda=0.01$ then the opposite is true. So as you can see $\\lambda$ controls the relative strength or \"importance\" of the two terms in our objective function.\n",
    " \n",
    "In this particular case, because I picked such a simple example, we could also do the math by hand:\n",
    " \n",
    "$$f(w)=(w-1)^2+\\lambda w^2=(\\lambda+1)w^2-2w+1$$\n",
    " \n",
    "Setting the derivative to zero:\n",
    " \n",
    "$$2(\\lambda+1)w-2=0$$\n",
    " \n",
    "yields\n",
    " \n",
    "$$w=\\frac{1}{1+\\lambda}$$\n",
    " \n",
    "So now we see it very clearly: if $\\lambda=1$ then the solution lies at $0.5$ which is sensible, since the two terms are weighted equally. But as $\\lambda$ grows the solution moves towards $w=0$, and as $\\lambda$ shrinks towards zero the solutions goes towards $w=1$. \n",
    " \n",
    "All this is fairly analogous to linear regression situation except that\n",
    "\n",
    "- $w$ is $d$-dimensional instead of $1$-dimensional\n",
    "- the minimizer of the left-hand side is the $w$ that minimizes training error\n",
    " \n",
    "So $\\lambda$ controls the relative importance of training error vs. regularization."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "@webio": {
   "lastCommId": null,
   "lastKernelId": null
  },
  "kernelspec": {
   "display_name": "Julia 0.7.0",
   "language": "julia",
   "name": "julia-0.7"
  },
  "language_info": {
   "file_extension": ".jl",
   "mimetype": "application/julia",
   "name": "julia",
   "version": "0.7.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}