pytorch/building_the_model_layer.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## What is a neural network?\n",
    "\n",
    "Neural network is a collection of neurons that are connected by layers. Each neuron is small computing until that performs simple calculations to collectively solve a problem. They are organized in layers. There are 3 types of layers: \n",
    "* input layer\n",
    "* hidden layer\n",
    "* output layer\n",
    "\n",
    "Each layer contains a number of neurons, except for the input layer. Neural networks mimic the way a human brain processes information."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Components of a neural network\n",
    "\n",
    "#### Activation function\n",
    "`Activation function determines whether a neuron should be activated or not`. The computations that happen in a neural network include applying an activation function. `If a neuron activates, then it means the input is important`. They are different kinds of activation functions. `The choice of which activation funciton to use dependes on what you want the output to be`. Another important role of an activation function is to `add non-linearity to the model`.\n",
    "  * *Binary* used to set an output node to 1 if function result is positive and 0 if the function result is negative\n",
    "\n",
    "$$\n",
    "f(x)= {\\small \\begin{cases} 0, & \\text{if } x < 0\\\\ 1, & \\text{if } x\\geq 0\\\\ \\end{cases}}\n",
    "$$\n",
    "\n",
    "  * *Sigmod* is used to predict the probability of an output node being between 0 and 1\n",
    "\n",
    "$$\n",
    "f(x) = {\\large \\frac{1}{1+e^{-x}}}\n",
    "$$\n",
    "\n",
    "  * *Tanh* is used to predict the probability of an output node being between -1 and 1. Used in classification use cases.\n",
    "\n",
    "$$\n",
    "f(x) = {\\large \\frac{e^{x} - e^{-x}}{e^{x} + e^{-x}}}\n",
    "$$\n",
    "\n",
    "  * *ReLU(Rectified Linear Unit)* used to set the output node to 0 if function result is negative and keeps the result value if the result is a postive values.\n",
    "\n",
    "$$\n",
    "f(x)= {\\small \\begin{cases} 0, & \\text{if } x < 0\\\\ x, & \\text{if } x\\geq 0\\\\ \\end{cases}}\n",
    "$$\n",
    "\n",
    "#### Weights\n",
    "`Weights` influence how well the output of our network will come close to the expected output value. As an input enter the neuron, it gets multiplied by a weight value and the resulting output is either observed, or passed to the next layer in the neural network. Weights for all neurons in a layer are organize into one tensor.\n",
    "\n",
    "#### Bias\n",
    "`Bias` makes up the difference between the activation function's output and its intended output. A low bias suggest that the network is making more assumptions about the form of the output, whereas a high bias value makes less assumptions about the form of the output.\n",
    "\n",
    "\n",
    "We can say that output y of a neural nework layer with weights W and bias b is computed as summation of the inputs multiply by the weights plus the bias\n",
    "\n",
    "$$\n",
    "x = \\sum{(weights * inputs) + bias}\n",
    "$$\n",
    "\n",
    "where f(x) is the activation function."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Build a neural network\n",
    "\n",
    "Neural networks are comprised of layer/modules that perform operations on data. The `torch.nn` namespace provides all the building blocks you need to build your own neural network. Every module in PyTorch subclasses the `nn.Module`. A neural network is a module itself that consists of other modules (layers). This nested structure allows for building and managing complex architectures easily.\n",
    "\n",
    "In the following sections, we will build a neural network to classify images in the FashionMNIST dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu118"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "import torch\n",
    "from torch import nn\n",
    "from torch.utils.data import DataLoader\n",
    "from torchvision import datasets, transforms"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Get a hardware device for training"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# check the paltform, Apple Silicon or Linux\n",
    "import os, platform\n",
    "\n",
    "torch_device=\"cpu\"\n",
    "\n",
    "if 'kaggle' in os.environ.get('KAGGLE_URL_BASE','localhost'):\n",
    "    torch_device = 'cuda'\n",
    "else:\n",
    "    torch_device = 'mps' if platform.system() == 'Darwin' else 'cpu'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "os.environ['PYTORCH_ENABLE_MPS_FALLBACK'] = '1'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "torch_device"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Define the class\n",
    "\n",
    "We define our neural network by subclassing `nn.Module`, and initialize the neural network layers in `__init__`. Every `nn.Module` subclass implements the operations on input data in the `forward` method.\n",
    "\n",
    "Our neural network are composed of the following:\n",
    "\n",
    "* The input layer with 28x28 or 784 features/pixels\n",
    "* The first linear module takes the input 784 features and transforms it to a hidden layer with 512 features\n",
    "* The ReLU activation function will be applied in the transformation\n",
    "* The second linear module take 512 features as input from the first hidden layer and transforms it to the next hidden layer with 512 features\n",
    "* The ReLU activation function will be applied in the transformation\n",
    "* The third linear module take 512 features as input form the second hidden layer and transforms it  to the output layer with 10, which is the number of classes\n",
    "* The ReLU activation function will be applied in the transformation\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "class NeuralNetwork(nn.Module):\n",
    "    def __init__(self) -> None:\n",
    "        super(NeuralNetwork, self).__init__()\n",
    "        self.flatten = nn.Flatten()\n",
    "        self.linear_relu_stack = nn.Sequential(\n",
    "            nn.Linear(28*28, 512),\n",
    "            nn.ReLU(),\n",
    "            nn.Linear(512, 512),\n",
    "            nn.ReLU(),\n",
    "            nn.Linear(512, 10),\n",
    "            nn.ReLU()\n",
    "        )\n",
    "    def forward(self,x):\n",
    "        x =self.flatten(x)\n",
    "        logits = self.linear_relu_stack(x)\n",
    "        return logits"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Move the Neural Network to the device we defined earlier and print the model's architecture."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = NeuralNetwork().to(torch_device)\n",
    "print(model)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To use the model, we pass it the input data. This executes the models `forward`, along with some background operation. However, do not call `model.forward()` directly. Calling the model on the input returns a 10-dimensional tensor with raw predicted values for each class. We get the prediction densities by passing it through an instance of the `nn.Softmax`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X =torch.rand(1,28,28, device=torch_device)\n",
    "logits = model(X)\n",
    "pred_probab = nn.Softmax(dim=1)(logits)\n",
    "y_pred = pred_probab.argmax(1)\n",
    "print(f\"Predicted class: {y_pred}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Weight and Bias\n",
    "\n",
    "The `nn.linear` module randomly initialized the *weights* and *bias* for each layer and internally stores the values in Tensors."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(f'First Linear weights: {model.linear_relu_stack[0].weight} \\n')\n",
    "print(f'First Linear bias: {model.linear_relu_stack[0].bias} \\n')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Model layers\n",
    "\n",
    "Let's break down the layers in the FashionMNIST model. To illustrate it, we will take a sample minibatch of 3 images of size 28x28 and see what happens to it as we pass it through the network."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "input_sample = torch.rand(3,28,28)\n",
    "print(input_sample.size())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### nn.Flatten\n",
    "\n",
    "We initialize the `nn.Flatten` layer to convert each 2D 28x28 image into a contiguous array of 784 pixel values (the minibatch dimension (at dim=0) is maintained). Each of the pixels are pass to the input layer of the neural network."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "flatten = nn.Flatten()\n",
    "flat_image = flatten(input_sample)\n",
    "print(flat_image.size())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### nn.Linear\n",
    "\n",
    "The linear layer is a module that applies a linear transformation on the input using it's stored weights and biases. The gayscale value of each pixel in the input layer will be connected to neurons in the hidden layer for calculation. The calculation used for the transformation is `weight` * `input` + `bias`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "layer1 = nn.Linear(in_features=28*28, out_features=20)\n",
    "hidden1 = layer1(flat_image)\n",
    "print(hidden1.size())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### nn.ReLU\n",
    "\n",
    "Non-linear activations are what create the complex mappings between the model's input and output. They are applied after linear tranformations to introduce `nonlinearity`, helping neural networks learn a wide variety of phenomena. In this model, we use `nn.ReLU` between our linear layers, but there's other activations to introduce non-linearity in your model.\n",
    "\n",
    "The ReLU activation function takes the output from the linear layer calculation and replaces the negative values with zeros.\n",
    "\n",
    "Linear output: ${ x = {weight * input + bias}}$\n",
    "\n",
    "ReLU: $f(x)= {\\small \\begin{cases} 0, & \\text{if } x < 0\\\\ x, & \\text{if } x\\geq 0\\\\ \\end{cases}}$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(f'Before ReLU: {hidden1}\\n\\n')\n",
    "hidden1 = nn.ReLU()(hidden1)\n",
    "print(f'After ReLU: {hidden1}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### nn.Sequential\n",
    "\n",
    "`nn.Sequential` is an ordered container of modules. The data is passed through all the modulers in the same order as defined. You can use sequential containers to put together a quick network like `seq_modules`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "seq_modules = nn.Sequential(\n",
    "    flatten,\n",
    "    layer1, nn.ReLU(),\n",
    "    nn.Linear(20,10)\n",
    ")\n",
    "input_image = torch.rand(3,28,28)\n",
    "logits = seq_modules(input_image)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### nn.Softmax\n",
    "\n",
    "The last linear layer of the neural network returns `logits` - raw values in [-infty, infty], which are passed to the `nn.Softmax` module. The Softmax activation function is used to calcualte the probability of the output from the neural network. It is only used on output layer  of a neural network. the results are scaled to values [0,1] representing the model's predicted densities for each class. `dim` parameter indicates the dimension along which the result values must sum to 1. The node with the highest probability predicts the desired output."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "softmax = nn.Softmax(dim=1)\n",
    "pred_probab = softmax(logits)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Model parameters\n",
    "\n",
    "Many layers inside a neural network are `parameterized`, i.e. have associated weights and biases that are optimized during training. Subclassing `nn.Module` automatically tracks all fields defined inside your model object, and makes all parameters accessible using your model's `parameters()` or `named_parameters()` methods."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print('Model structure:', model, '\\n\\n')\n",
    "\n",
    "for name, param in model.named_parameters():\n",
    "    print(f'Layer: {name} | Size: {param.size()} | Values: {param[:2]} \\n')"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "diffusers",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "name": "python",
   "version": "3.11.4"
  },
  "orig_nbformat": 4
 },
 "nbformat": 4,
 "nbformat_minor": 2
}