lecture_04_pre_numpy.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "ddc4f8e7",
   "metadata": {},
   "source": [
    "# Machine learning is just linear algebra and vector calculus\n",
    "\n",
    "A generic ML algorithm\n",
    "1. define an equation with parameters you can set, such that optimizing the equation (picking the best parameters) means being better at the task you want to solve\n",
    "1. [optional] transform your data from its raw form to make it easier to learn with\n",
    "1. do one of two things to make the ML system learn on your training data:\n",
    "    - if the equation has a single unique minimum that you can solve for analytically (e.g. OLS) then do so. \n",
    "    - otherwise find the gradient of the equation with respect to the parameters, and iteratively move closer to a better solution by changing the parameters a bit along the direction in the gradient that makes you better at the task\n",
    "1. once you have the best/good enough parameters, you can predict with the ML system by using the parameters on new inputs (validation or testing data)\n",
    "\n",
    "All steps involve matrix algebra, step 3 involves vector caculus \n",
    "\n",
    "# NumPy is how you do linear algebra in python\n",
    "- the fundamental package for scientific computing with Python\n",
    "- numpy is highly optimized (fast + memory effecient)\n",
    "- numpy is used to build many commonly used scientific tools in python:\n",
    "    - scipy: fundamental algorithms for scientific computing in Python (numeric integration, interpolation, transforms, statistics, etc)\n",
    "    - scikit-learn: the \"default\" library for machine learning\n",
    "    - scikit-image: as above, but for image processing\n",
    "    - pandas: as above, but for data wrangling\n",
    "    - statsmodels: as above, but for frequentist statistical modeling\n",
    "    - pystan: as above, but for Bayesian statistical modeling\n",
    "    - jax: Google's high performance ML library that includes automatic differentiation (vector calculus) of functions you write, compilation to runon GPU/TPU, and parallelization\n",
    "  \n",
    "    \n",
    "In this class we will sometimes have you do things using raw numpy and writing your own algorithms the first time, so you can understand how things work at a very low level.\n",
    "\n",
    "However, when you really want to get things done in ML you will use a higher level library like scikit-learn. These libraries help you with\n",
    "- prebuilt algorithms that are well-debugged and fast\n",
    "- all kinds of extras beyond the base algorithm (e.g., regularization to prevent overfitting of your data)\n",
    "- multiple solvers (different ways to implement the algorithm, often with different extras and speed tradeoffs)\n",
    "\n",
    "\n",
    "# NumPy tutorials you should check out\n",
    "\n",
    "- [NumPy: the absolute basics for beginners](https://numpy.org/devdocs/user/absolute_beginners.html)\n",
    "\n",
    "- [NumPy Illustrated: The Visual Guide to NumPy](https://betterprogramming.pub/3b1d4976de1d?sk=57b908a77aa44075a49293fa1631dd9b)\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6cba7888",
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "data = np.array([1, 2, 3])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "487ee9b8",
   "metadata": {},
   "source": [
    "You can visualize your array this way:\n",
    "![numpy array 1,2,3](https://numpy.org/devdocs/_images/np_array.png)\n",
    "\n",
    "# Indexing and slicing\n",
    "You can index and slice NumPy arrays in the same ways you can slice Python lists.\n",
    "\n",
    "NOTE 1: Everything is 0-indexed!\n",
    "NOTE 2: Arrays use the same indexing as Python's list, which roughly speaking works like `[start here: stop before here : step size going from start to stop]` and features the ability to use negative numbers and has default behavior if you omit any of those 3 slots from the indexer.\n",
    "\n",
    "![numpy array slicing](https://numpy.org/devdocs/_images/np_indexing.png)\n",
    "\n",
    "Please take a look at all of the indexing operations below... predict what you will see before you execute them, then see if the predictions match reality. If you can't predict what they will do then please review list/array indexing tutorials on the web \n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "60ee1f71",
   "metadata": {},
   "outputs": [],
   "source": [
    "print( data[1] )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "38be483c",
   "metadata": {},
   "outputs": [],
   "source": [
    "print( data[:2] )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c1201111",
   "metadata": {},
   "outputs": [],
   "source": [
    "print( data[1:] )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "edcb36fb",
   "metadata": {},
   "outputs": [],
   "source": [
    "print( data[-2::] )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0efd89bb",
   "metadata": {},
   "outputs": [],
   "source": [
    "print( data[::2] )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "decf740d",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Your code here... \n",
    "# how would you reverse the order of the array?\n",
    "raise NotImplementedError"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c5e79cc3",
   "metadata": {},
   "source": [
    "# Why NumPy and not just lists?\n",
    "\n",
    "What's the time it take to do Hadamard product on a large vector?  \n",
    "\n",
    "You know what Hadamard product is yeah? ;)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ddc24e32",
   "metadata": {},
   "outputs": [],
   "source": [
    "size = 1000000\n",
    "\n",
    "# declaring big lists\n",
    "list1 = range(size)\n",
    "list2 = range(size)\n",
    "  \n",
    "# declaring big arrays with the same numbers\n",
    "array1 = np.arange(size)\n",
    "array2 = np.arange(size)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0396f20b",
   "metadata": {},
   "outputs": [],
   "source": [
    "# here we use python's zip \n",
    "# to do element wise multiplication on the lists\n",
    "%timeit [(a * b) for a, b in zip(list1, list2)]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1a993e66",
   "metadata": {},
   "outputs": [],
   "source": [
    "# in numpy arrays, * is overloaded to do Hadamard product\n",
    "# that is element wise multiplication on the arrays\n",
    "%timeit array1 * array2"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "beae825f",
   "metadata": {},
   "source": [
    "WOW! numpy is more than 60x faster! But is the output identical? \n",
    "\n",
    "BTW, checking if two arrays/lists have exactly the same contents is the kind of task you will probably have to do quite often... so maybe this is a useful prototype for you"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b6e23c3a",
   "metadata": {},
   "outputs": [],
   "source": [
    "# matched is an array the same size as the others\n",
    "# it contains bools for every array element, True indicating a match\n",
    "matched = (array1 * array2) == [(a * b) for a, b in zip(list1, list2)]\n",
    "print('shape:',matched.shape, 'sample:', matched)\n",
    "\n",
    "# np.all returns True if all elements are True\n",
    "print('is every element identical?',np.all( matched ))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3b52d7cc",
   "metadata": {},
   "source": [
    "# Multidimensional arrays (i.e, matrixes)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b0047eda",
   "metadata": {},
   "outputs": [],
   "source": [
    "array_example = np.array([    [[0, 1, 2, 3],\n",
    "                               [4, 5, 6, 7]],\n",
    "   \n",
    "                              [[1, 2, 3, 4],\n",
    "                               [5, 6, 7, 8]],\n",
    "   \n",
    "                              [[2 ,3, 4, 5],\n",
    "                               [6, 7, 8, 9]]    ])\n",
    "\n",
    "array_example"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7473fd99",
   "metadata": {},
   "outputs": [],
   "source": [
    "array_example.ndim"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fdfcaa90",
   "metadata": {},
   "outputs": [],
   "source": [
    "array_example.size"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "30ab2a7d",
   "metadata": {},
   "outputs": [],
   "source": [
    "array_example.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bf3f2d8b",
   "metadata": {},
   "outputs": [],
   "source": [
    "array_example[0,1,2]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "dbfab18a",
   "metadata": {},
   "outputs": [],
   "source": [
    "array_example[:,1,2]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c797241b",
   "metadata": {},
   "outputs": [],
   "source": [
    "array_example[1,:,2]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4f0a73d1",
   "metadata": {},
   "outputs": [],
   "source": [
    "array_example[0,1,:]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d489f1d3",
   "metadata": {},
   "outputs": [],
   "source": [
    "a = np.arange(6)\n",
    "print(a)\n",
    "b = a.reshape(3, 2)\n",
    "print(b)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bd68fec4",
   "metadata": {},
   "outputs": [],
   "source": [
    "array_example.reshape(24,)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "437ac4a8",
   "metadata": {},
   "outputs": [],
   "source": [
    "array_example.flatten()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e19e3e7a",
   "metadata": {},
   "outputs": [],
   "source": [
    "np.zeros(4)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "737e4b30",
   "metadata": {},
   "outputs": [],
   "source": [
    "np.ones(4)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "28d57d12",
   "metadata": {},
   "outputs": [],
   "source": [
    "np.ones((2,4))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6daa07dd",
   "metadata": {},
   "outputs": [],
   "source": [
    "np.zeros((3,2,4))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8eff20a5",
   "metadata": {},
   "outputs": [],
   "source": [
    "np.ones_like(array_example)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e9f39e69",
   "metadata": {},
   "source": [
    "# Useful NumPy operations\n",
    "\n",
    "## finding things out about data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b90fe157",
   "metadata": {},
   "outputs": [],
   "source": [
    "data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1bdd2c2c",
   "metadata": {},
   "outputs": [],
   "source": [
    "data.min(), data.max(), data.mean()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "36c34186",
   "metadata": {},
   "outputs": [],
   "source": [
    "array_example"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2c1e28ea",
   "metadata": {},
   "outputs": [],
   "source": [
    "array_example.min(), array_example.max()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d59d77c3",
   "metadata": {},
   "outputs": [],
   "source": [
    "array_example.argmin(), array_example.argmax()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "072f7c35",
   "metadata": {},
   "outputs": [],
   "source": [
    "array_example.flatten()[ array_example.argmin() ], array_example.flatten()[ array_example.argmax() ]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b8460cd2",
   "metadata": {},
   "outputs": [],
   "source": [
    "array_example.min(axis=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2ae419d9",
   "metadata": {},
   "outputs": [],
   "source": [
    "array_example.min(axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a391c2c8",
   "metadata": {},
   "outputs": [],
   "source": [
    "array_example.min(axis=2)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6927e61a",
   "metadata": {},
   "source": [
    "## doing linear algebra"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9d9161d8",
   "metadata": {},
   "outputs": [],
   "source": [
    "# taking 2 vectors of same dimensionality\n",
    "v1 = np.array([2 , 6])\n",
    "v2 = np.array([5 , 7])\n",
    "\n",
    "v3 = v1 + v2 # addition\n",
    "print('v2 + v1 =',v3)\n",
    "\n",
    "v4 = v2 - v1 # subtraction\n",
    "print('v2 - v1 =',v4)\n",
    "\n",
    "print('0.7 * (v2 - v1) = ', 0.7*v4) # scalar product\n",
    "\n",
    "# two ways to get Hadamard multiplication\n",
    "print('element-wise mult method #1', np.multiply(v1, v2))\n",
    "print('element-wise mult method #2', v1 * v2)\n",
    "\n",
    "# four ways to get the dot product\n",
    "print('dot prod method #1', np.sum( v1 * v2 ))  # what a dot product is \n",
    "print('dot prod method #2', np.dot(v1,v2))      # expliclty dot product\n",
    "print('dot prod method #3', v1.dot(v2))         # expliclty dot product\n",
    "print('dot prod method #4', np.matmul(v1,v2))   # for 1D vectors, matrix mult is a dot product\n",
    "\n",
    "#  cross product: theta is angle between v1,v2; n is the vector normal to them both\n",
    "print('cross product v1 x v2 = ||v1|| ||v2|| |sin theta| n =', np.cross(v1,v2)) \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7563cc18",
   "metadata": {},
   "outputs": [],
   "source": [
    "mat_b = np.array([[1, 2, 3],\n",
    "                  [4, 5, 6]])\n",
    "mat_a = np.array([[1, 1, 1],\n",
    "                  [2, 2, 2]])\n",
    "\n",
    "mat_a"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f97fbfe5",
   "metadata": {},
   "outputs": [],
   "source": [
    "# transpose of a matrix\n",
    "mat_b.T"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c5937068",
   "metadata": {},
   "outputs": [],
   "source": [
    "# in matrix multiplication\n",
    "# 1st matrix (mxn) determines number of rows (m)\n",
    "# 2nd matrix (nxp) determines number of columns (p)\n",
    "# note inner size (n) must match!\n",
    "# outputs are (mxp)\n",
    "print('matrix mult method #1\\n', mat_a @ mat_b.T)\n",
    "print('matrix mult method #2\\n', np.matmul(mat_a, mat_b.T))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ff27bfe6",
   "metadata": {},
   "outputs": [],
   "source": [
    "mat_c = np.array([[1., 2.], [3., 4.]])\n",
    "# a matrix inverse is like a reciprocal for a scalar\n",
    "# the original matrix times its inverse equals I (diagonal ones)\n",
    "# not every matrix has an inverse (must be square & full rank) \n",
    "# but if it does then you can find it like this\n",
    "inv_c = np.linalg.inv(mat_c)\n",
    "close_to_I = mat_c @ inv_c\n",
    "print(mat_c, '\\n x \\n', inv_c, '\\n = \\n', close_to_I)\n",
    "\n",
    "# note that c @ inv_c will probably be not quite the same as I\n",
    "# closeness-but-not-exact can be checked if you desire\n",
    "np.allclose(np.dot(mat_c, inv_c), np.eye(2))\n",
    "\n",
    "# why is it close but not exact?\n",
    "# the inverse solver (LIKE MOST NUMERIC COMPUTATIONS!) is not \n",
    "# exact.. it could go forever (or just very very long) if it tried\n",
    "# to be exact.  So it calculates using an iterative algorithm\n",
    "# until succesive iterations add less than a fixed tolerance \n",
    "# change to the answer"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e4ef3fab",
   "metadata": {},
   "source": [
    "# Using matrix algebra in ML\n",
    "\n",
    "What if we have a 2D image, where the values of `input` are simply the brighness of the pixels.\n",
    "\n",
    "Our ML system is setup to produce a prediction from the image. For instance, what if it's trying to detect a vertical bright line in the center of the image.  In our toy example here we will pretend that learning is done. We only want to use our ML system to detect vertical lines in the input.\n",
    "\n",
    "We will predict using the following linear algebra\n",
    "\n",
    "$ y = w^T x$ \n",
    "\n",
    "where $y$ is the output prediction, $w$ is a column vector of parameters (weights learned by the ML system to do the task), and the inputs $x$ are a column vector. Because $w,x$ are 1D vectors the same length the output prediction $y$ is scalar number.\n",
    "\n",
    "To turn our 2D image into a 1D vector we will reshape it using numpy.  We will define a weight vector by hand that produces larger $y$ values for a vertical line in the center, and smaller values for other kinds of lines "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8a7a0295",
   "metadata": {},
   "outputs": [],
   "source": [
    "# here's an \"image\" with a bright vertical line against a dark background\n",
    "inputs = np.array([[0, 1, 0],\n",
    "                   [0, 1, 0],\n",
    "                   [0, 1, 0]])\n",
    "    \n",
    "print(inputs)\n",
    "print('shape:',inputs.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "937afb89",
   "metadata": {},
   "outputs": [],
   "source": [
    "inputs.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "eafbbff3",
   "metadata": {},
   "outputs": [],
   "source": [
    "# flatten isn't quite what we need to make the image a column vector\n",
    "# this is a 1-D vector, not a column!\n",
    "print(inputs.flatten())\n",
    "print('shape:',inputs.flatten().shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d2a7cefd",
   "metadata": {},
   "outputs": [],
   "source": [
    "columnar_inputs = inputs.reshape(9,1)\n",
    "print(columnar_inputs)\n",
    "print('shape:',columnar_inputs.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "91f30d5d",
   "metadata": {},
   "outputs": [],
   "source": [
    "# ok let's make a quick function to columnarize any 2d array/image\n",
    "def columnar(image):\n",
    "    n_els = np.prod(image.shape)\n",
    "    return image.reshape(n_els,1)\n",
    "\n",
    "col_ins = columnar(inputs)\n",
    "print(col_ins)\n",
    "print('shape:', col_ins.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bc2ec096",
   "metadata": {},
   "outputs": [],
   "source": [
    "# here are some weights I hand designed to detect VERTICAL lines\n",
    "# in the CENTER of a 3x3 image\n",
    "weights = np.array([[-1],[+1],[-1],[-1],[+1],[-1],[-1],[+1],[-1]])\n",
    "print(weights)\n",
    "print('shape:',weights.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4e0c5064",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Your code here... \n",
    "# how would you use weights, inputs, and columnar() \n",
    "# and matrix multiplication to predict \n",
    "# whether the original image has a vertical line\n",
    "def predict(img, wts):\n",
    "    '''\n",
    "    inputs: \n",
    "        img - input image, 2d array\n",
    "        wts - weights of predictor, columnar array\n",
    "    returns:\n",
    "        bool describing if img contains a vertical line\n",
    "    uses:\n",
    "        matrix multiplication of inputs, you must determine how\n",
    "        and columnar()\n",
    "    '''\n",
    "    raise NotImplementedError"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6c1a3bbd",
   "metadata": {},
   "outputs": [],
   "source": [
    "# to test your function above here's some images\n",
    "# one with a bright VERTICAL CENTERED line against a dark background\n",
    "# this is positive class (predictor should say TRUE)\n",
    "inputs_vc = np.array([[0, 1, 0],\n",
    "                      [0, 1, 0],\n",
    "                      [0, 1, 0]])\n",
    "\n",
    "# one with a bright HORIZONTAL CENTERED line against a dark background\n",
    "# this is negative class (predictor should say FALSE)\n",
    "inputs_hc = np.array([[0, 0, 0],\n",
    "                      [1, 1, 1],\n",
    "                      [0, 0, 0]])\n",
    "\n",
    "# also negative class (VERTICAL LEFT)\n",
    "inputs_vl = np.array([[1, 0, 0],\n",
    "                      [1, 0, 0],\n",
    "                      [1, 0, 0]])\n",
    "\n",
    "# also negative class (DIAGONAL)\n",
    "inputs_diag = np.eye(3)\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b470434f",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2321e530",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}